The question I pose below is now academic, but any proposed
answers could be of use in future scenarios.

We recently experienced the effects of a slight routing protocol
misconfiguration. Suffice it to say the routing tables on our routers
were consequently not quite correct and we thus had difficulties with
traffic going between two subnets, which I will refer to as subnets A
and B. There are multiple paths via intervening subnets between subnets
A and B. The problem occurred on a production subnet and has since
been corrected.

All is now fine except for the presence of one lingering mystery:

While there was absolutely no problem with conventional ICMP pings
between subnets A and B in either direction, TCP and UDP would not
route correctly. The general question is:

"Under what circumstances in a routed environment would ICMP pings
succeed, while TCP and UDP would consistently fail?"

Here are some further facts:
- Pings performed consistently and reliably between A and B in either
direction. There were no unusual ICMP packet losses observed during
the several hours the problem persisted.
- I did not have the opportunity to determine if any TCP or UDP at all
could get across from A to B, but it didn't appear that any was
getting across. Simple telnet, SMTP, FTP and DNS, among many other
protocols all failed consistently.
- Traffic between subnets A and B normally passes through either subnet
C or subnet D. There was no discernable problem with TCP or UDP
between subnets A and C, A and D, B and C, or B and D, in either
direction.
- The size of the ICMP packets made no difference. 16KB packets worked
just as well as 64 byte packets.
- Network congestion was not an issue. Interfaces were at less than
10% of capacity and no QOS policies were active.
- There are no firewalls, NAT, or filter rules of any sort in place
between subnets A and B.

I would have done more detective work, including copious sniffing, had
I had the opportunity, but this occurred on a production network, and I
needed to get things working ASAP.
It has been said often enough that ICMP pings can be misleading, and
this is quite true, but false negatives are far more common than false
positives; this is usually being due to ICMP being given second-class
treatment, being discarded, etc. I find it odder and more mysterious
when TCP and UDP fail but ICMP prevails.

"Ben Strauss" <imttis@bellsouth.net> writes:
> "Under what circumstances in a routed environment would ICMP pings
> succeed, while TCP and UDP would consistently fail?"
[...]
> - There are no firewalls, NAT, or filter rules of any sort in place
> between subnets A and B.

That's the most likely explanation -- a filter somewhere between these
two. Most routers can be configured to filter traffic, and it's at
least possible that someone forgot about a rule configured somewhere.

Another possible problem would be bugs in one of the implementations.
There've been occasional cases on some platforms where (for instance)
hardware checksum features in the network adapters cause regular
transports like TCP and UDP to be mangled while ICMP (which often
isn't on the "fast path") is spared the indignity.

There may be other more exotic possibilities, such as QoS rules.

Since you haven't gathered debug information on the network while the
problem was happening, it's going to be hard to say what causes the
effects you see. Unless it can be reproduced and more information
gathered, I'd just list it as a mystery.

--
James Carlson, KISS Network <james.d.carlson@sun.com>
Sun Microsystems / 1 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677

I've reviewed the configs and I'm quite sure there are no filters in
place, and if there were it would be quite unusual to allow ICMP but
not TCP or UDP.

I wonder if anyone else has experienced this behavior and found it to
be anything other than user-configurable filtering or QOS.
Theoretical possibilities might include:
(a) Some undocumented/nonobvious filtering behavior.
(b) Some inherent difference in the way certain IP stacks treat ICMP
echos.

An example of such an oddity is one I once encountered in a low-end
network device from a now-defunct company, which when attempting to
send TCP to its default gateway router would properly observe any ICMP
redirects it received and would readdress and resend the packets
accordingly. When faced with replying to an ICMP echo it would totally
ignore redirects and just keep sending them to the gateway.

I have no reason to believe that this is the case here. The devices
involved were: Cisco IOS Routers, Cisco VPN Concentrators (subnet B is
connected to subnets C and D via IPSec tunnels as no frame-relay
connection is feasible), and Windows machines (involved in much
pinging).

I've reviewed the configs and I'm quite sure there are no filters in
place, and if there were it would be quite unusual to allow ICMP but
not TCP or UDP.

I wonder if anyone else has experienced this behavior and found it to
be anything other than user-configurable filtering or QOS.
Theoretical possibilities might include:
(a) Some undocumented/nonobvious filtering behavior.
(b) Some inherent difference in the way certain IP stacks treat ICMP
echos.

An example of such an oddity is one I once encountered in a low-end
network device from a now-defunct company, which when attempting to
send TCP to its default gateway router would properly observe any ICMP
redirects it received and would readdress and resend the packets
accordingly. When faced with replying to an ICMP echo it would totally
ignore redirects and just keep sending them to the gateway.

I have no reason to believe that this is the case here. The devices
involved were: Cisco IOS Routers, Cisco VPN Concentrators (subnet B is
connected to subnets C and D via IPSec tunnels as no frame-relay
connection is feasible), and Windows machines (involved in much
pinging).