Can someone explain what's going on during the trace shown below? Why
doesn't the trace stop when it reaches the target in step 6?

The sample below was done using XP (behind a NAT'ed router), but similar
behavior is observed with 'traceroute' on Linux.


C:\Documents and Settings\benh>tracert cr1.ge1-1-0.osl255.netcom.no

Tracing route to cr1.ge1-1-0.osl255.netcom.no [212.169.116.109]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms amilo2.storsandmo [192.168.2.95]
2 996 ms 957 ms 1017 ms 212.169.98.1
3 1283 ms 1023 ms 751 ms 10.240.239.5
4 758 ms 992 ms 441 ms 212.45.186.194
5 1078 ms 1222 ms 1222 ms 212.45.186.94
6 554 ms 803 ms 558 ms cr1.ge1-1-0.osl255.netcom.no
[212.169.116.109]
7 1501 ms 920 ms 944 ms osl200-gw01.teliamobile.net [192.36.252.162]
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 ^C
C:\Documents and Settings\benh>

--
-+-Ben-+-

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ben Hetland wrote:
> Can someone explain what's going on during the trace shown below? Why
> doesn't the trace stop when it reaches the target in step 6?
>
> The sample below was done using XP (behind a NAT'ed router), but similar
> behavior is observed with 'traceroute' on Linux.
>
>
> C:\Documents and Settings\benh>tracert cr1.ge1-1-0.osl255.netcom.no
>
> Tracing route to cr1.ge1-1-0.osl255.netcom.no [212.169.116.109]
> over a maximum of 30 hops:
>
> 1 <1 ms <1 ms <1 ms amilo2.storsandmo [192.168.2.95]
> 2 996 ms 957 ms 1017 ms 212.169.98.1
> 3 1283 ms 1023 ms 751 ms 10.240.239.5
> 4 758 ms 992 ms 441 ms 212.45.186.194
> 5 1078 ms 1222 ms 1222 ms 212.45.186.94
> 6 554 ms 803 ms 558 ms cr1.ge1-1-0.osl255.netcom.no
> [212.169.116.109]
> 7 1501 ms 920 ms 944 ms osl200-gw01.teliamobile.net [192.36.252.162]
> 8 * * * Request timed out.
> 9 * * * Request timed out.
> 10 * * * Request timed out.
> 11 * * * Request timed out.
> 12 * * * Request timed out.
> 13 * * * Request timed out.
> 14 ^C
> C:\Documents and Settings\benh>

My best guess is that somewhere between osl200-gw01.teliamobile.net and
the next hop, someone has implemented a filter to drop ICMP ECHO REPLY
packets. Thus, you only get ping replies (ICMP ECHO REPLY) from those
hops up to osl200..., and the ping replies from anywhere past that point
are not returned to you.

- --

Lew Pitcher, IT Specialist, Corporate Technology Solutions,
Enterprise Technology Solutions, TD Bank Financial Group

(Opinions expressed here are my own, not my employer's)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEULdiagVFX4UWr64RAj6SAJ45tf+bPWSsd4pel+Lv7s Z2hWqejgCgqPPF
Be59GUa8so0Myf9jlHkg5TE=
=sJ7i
-----END PGP SIGNATURE-----

In article <4450B390.7080700@sintef.no>,
Ben Hetland <ben.a.hetland@sintef.no> wrote:

> Can someone explain what's going on during the trace shown below? Why
> doesn't the trace stop when it reaches the target in step 6?
>
> The sample below was done using XP (behind a NAT'ed router), but similar
> behavior is observed with 'traceroute' on Linux.

Tracert sends ICMP Echo messages. It expects to receive ICMP TTL
Exceeded messages from the routers in the path, and ICMP Echo Reply
messages from the target, and I presume it doesn't stop until it gets
the Echo Reply.

My guess is that the device in step 6 is actually a NATting router, so
it normally passes packets for 212.169.116.109 through rather than
respond to them itself. But during the traceroute it responds with TTL
Exceeded when the TTL doesn't allow it to pass the packets.

Run Ethereal while you trace and see what comes back from that address.

>
>
> C:\Documents and Settings\benh>tracert cr1.ge1-1-0.osl255.netcom.no
>
> Tracing route to cr1.ge1-1-0.osl255.netcom.no [212.169.116.109]
> over a maximum of 30 hops:
>
> 1 <1 ms <1 ms <1 ms amilo2.storsandmo [192.168.2.95]
> 2 996 ms 957 ms 1017 ms 212.169.98.1
> 3 1283 ms 1023 ms 751 ms 10.240.239.5
> 4 758 ms 992 ms 441 ms 212.45.186.194
> 5 1078 ms 1222 ms 1222 ms 212.45.186.94
> 6 554 ms 803 ms 558 ms cr1.ge1-1-0.osl255.netcom.no
> [212.169.116.109]
> 7 1501 ms 920 ms 944 ms osl200-gw01.teliamobile.net [192.36.252.162]
> 8 * * * Request timed out.
> 9 * * * Request timed out.
> 10 * * * Request timed out.
> 11 * * * Request timed out.
> 12 * * * Request timed out.
> 13 * * * Request timed out.
> 14 ^C
> C:\Documents and Settings\benh>

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

In article <CH24g.3614$fx.324132@news20.bellglobal.com>,
Lew Pitcher <Lew.Pitcher@tdsecurities.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ben Hetland wrote:
> > Can someone explain what's going on during the trace shown below? Why
> > doesn't the trace stop when it reaches the target in step 6?
> >
> > The sample below was done using XP (behind a NAT'ed router), but similar
> > behavior is observed with 'traceroute' on Linux.
> >
> >
> > C:\Documents and Settings\benh>tracert cr1.ge1-1-0.osl255.netcom.no
> >
> > Tracing route to cr1.ge1-1-0.osl255.netcom.no [212.169.116.109]
> > over a maximum of 30 hops:
> >
> > 1 <1 ms <1 ms <1 ms amilo2.storsandmo [192.168.2.95]
> > 2 996 ms 957 ms 1017 ms 212.169.98.1
> > 3 1283 ms 1023 ms 751 ms 10.240.239.5
> > 4 758 ms 992 ms 441 ms 212.45.186.194
> > 5 1078 ms 1222 ms 1222 ms 212.45.186.94
> > 6 554 ms 803 ms 558 ms cr1.ge1-1-0.osl255.netcom.no
> > [212.169.116.109]
> > 7 1501 ms 920 ms 944 ms osl200-gw01.teliamobile.net [192.36.252.162]
> > 8 * * * Request timed out.
> > 9 * * * Request timed out.
> > 10 * * * Request timed out.
> > 11 * * * Request timed out.
> > 12 * * * Request timed out.
> > 13 * * * Request timed out.
> > 14 ^C
> > C:\Documents and Settings\benh>
>
> My best guess is that somewhere between osl200-gw01.teliamobile.net and
> the next hop, someone has implemented a filter to drop ICMP ECHO REPLY
> packets. Thus, you only get ping replies (ICMP ECHO REPLY) from those
> hops up to osl200..., and the ping replies from anywhere past that point
> are not returned to you.

How does that explain that it went *past* the destination, which was the
point of his question? You seem to be trying to explain the "Request
timed out" lines starting at hop 8, when his question is about hop 6.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Barry Margolin wrote:
>
> Tracert sends ICMP Echo messages.

Whatever they are called, according to Ethereal it seems traceroute at
least (not tracert, but maybe it does the same) sends UDP messages with
10 bytes of payload data by default. The returned ICMP "TTL Exceeded"
msg seems to embed this UDP msg as its payload.


> It expects to receive ICMP TTL
> Exceeded messages from the routers in the path, and ICMP Echo Reply
> messages from the target, and I presume it doesn't stop until it gets
> the Echo Reply.

It sends 3 messages (UDP packets) for each TTL value, then increases the
TTL for the next set of 3 messages. The * that we see indicate a missing
ICMP response (timeout). Since it increases the TTL nevertheless, we can
"see" the routers even behind those that don't send an ICMP back.


> My guess is that the device in step 6 is actually a NATting router, so
> it normally passes packets for 212.169.116.109 through rather than
> respond to them itself. But during the traceroute it responds with TTL
> Exceeded when the TTL doesn't allow it to pass the packets.

That might be so, and it means that the packed is passed from the
"internal" to the "external" side of the NAT router, the TTL isn't
decremented. Otherwise I should see both of these interfaces on the
router. Fair enough, since that seems to be the "normal" way to do it
anyway, so that a box only appears once in the path.


>> 5 1078 ms 1222 ms 1222 ms 212.45.186.94
>> 6 554 ms 803 ms 558 ms cr1.ge1-1-0.osl255.netcom.no
>> [212.169.116.109]
>> 7 1501 ms 920 ms 944 ms osl200-gw01.teliamobile.net [192.36.252.162]

But doesn't this mean that the "osl255" gateway is very stupid since it
passes a packet addressed to itself on to a different gw on another
segment? What appears to happen after that is that the packet just flies
off into the big void (and eventually dropped as TTL=0).

What could be a good reason for doing that? I reckon it could maybe be
reasonable, and maybe even common to do that, but I just can't think of
a very good reason for it myself. Any ideas?

--
-+-Ben-+-

>
> How does that explain that it went *past* the destination, which was the
> point of his question? You seem to be trying to explain the "Request
> timed out" lines starting at hop 8, when his question is about hop 6.
>
> --
> Barry Margolin, barmar@alum.mit.edu
> Arlington, MA
> *** PLEASE post questions in newsgroups, not directly to me ***
> *** PLEASE don't copy me on replies, I'll read them in the group ***

Hi Barry,

Could this be a possible explanation ?

1. The tracert program included with Windows XP sends out ICMP messages
- it looks for ICMP ECHOREPLY only.

2. The traceroutes on Linux can do ICMP or UDP - it looks for ICMP
ECHOREPLY (icmp) , UNREACH_PORT (udp) or UNREACH_PROTOCOL (udp). I
quickly checked the traceroute 1.4 source code on Linux and this
appears to be the case !!

3. If the destination returned any other ICMP code, tracert assumes it
has not reached the target and continues plodding on. It doesnt check
the source ip address in the ICMP packet for a match. That was quite
surprising to me - I thought any ICMP reply from the target is
sufficient reason to conclude that we are done.

4. His tracert output shows a lot of swings in the Round trip times, so
packet 7 (which appears to be past the target) could actually be
another interface on the target or even another router.

I agree that a ethereal trace is in order.

Regs,
Vivek

VivekRajan wrote:
> 3. If the destination returned any other ICMP code, tracert assumes it
> has not reached the target and continues plodding on. It doesnt check
> the source ip address in the ICMP packet for a match. That was quite
> surprising to me - I thought any ICMP reply from the target is
> sufficient reason to conclude that we are done.

Yes, surprising to me too, so that's why I got so curious about it.

On first thought I'd say that inspecting the source address of the reply
should be sufficient, but then I see that it doesn't really prove that
we have reached the target! For instance, what about port forwarding
along the path to the target (or kinda "reverse NATting" if one might
do something like that)? What if the packets take an extra round-trip
before it hits the target the "right" way (i.e. so it will respond). I
could also imagine that a packet is traveling several times through the
same gateway, for instance based on packet type and source/destination
addresses or incoming interface at the gw.

However, as I see it *can* be done, I'm still curious to hear of
real-life reasons why sysadmins would want to do this. I would believe
that normally such configurations would only tend to increase system
loads. I'd be glad if someone could shed some light on this!


> 4. His tracert output shows a lot of swings in the Round trip times, so
> packet 7 (which appears to be past the target) could actually be
> another interface on the target or even another router.

Those "swings" are probably due to the first hop (between step 1 and 2)
being a wireless EDGE mobile connection. Traffic tends to affect it, but
the strange trace was "discovered" while I was trying to find out why it
seems to introduce a penalty of at least 400-500 ms even on an otherwise
idle connection. (And the same happens if connected over UMTS.)

So if anyone has experience with GPRS/EDGE/UMTS and associated
latencies, I'd like to know if this is really normal!


> I agree that a ethereal trace is in order.

Done; see other posting! ;-)

--
-+-Ben-+-

In article <e2rkfq$cpr$1@orkan.itea.ntnu.no>,
Ben Hetland <ben.a.hetland@sintef.no> wrote:

> Barry Margolin wrote:
> >
> > Tracert sends ICMP Echo messages.
>
> Whatever they are called, according to Ethereal it seems traceroute at
> least (not tracert, but maybe it does the same) sends UDP messages with
> 10 bytes of payload data by default. The returned ICMP "TTL Exceeded"
> msg seems to embed this UDP msg as its payload.

Traceroute sends UDP, tracert sends ICMP. Since the OP was using
tracert, I described its behavior. I don't see how what traceroute does
is relevant.

>
> >> 5 1078 ms 1222 ms 1222 ms 212.45.186.94
> >> 6 554 ms 803 ms 558 ms cr1.ge1-1-0.osl255.netcom.no
> >> [212.169.116.109]
> >> 7 1501 ms 920 ms 944 ms osl200-gw01.teliamobile.net
> >> [192.36.252.162]
>
> But doesn't this mean that the "osl255" gateway is very stupid since it
> passes a packet addressed to itself on to a different gw on another
> segment? What appears to happen after that is that the packet just flies
> off into the big void (and eventually dropped as TTL=0).

When a NAT router has port forwarding enabled, it's normal for it to
pass packets addressed to itself through. Many NAT routers also allow
you to configure an inside address as an "exposed host" (aka "DMZ
host"), which causes all traffic for the router's address to be passed
through to this host (except ports that have explicit port forwarding to
another address).

So I suspect something like this may be going on with that device.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

In article <e2spg3$k2$1@orkan.itea.ntnu.no>,
Ben Hetland <ben.a.hetland@sintef.no> wrote:

> VivekRajan wrote:
> > 3. If the destination returned any other ICMP code, tracert assumes it
> > has not reached the target and continues plodding on. It doesnt check
> > the source ip address in the ICMP packet for a match. That was quite
> > surprising to me - I thought any ICMP reply from the target is
> > sufficient reason to conclude that we are done.
>
> Yes, surprising to me too, so that's why I got so curious about it.
>
> On first thought I'd say that inspecting the source address of the reply
> should be sufficient, but then I see that it doesn't really prove that
> we have reached the target! For instance, what about port forwarding
> along the path to the target (or kinda "reverse NATting" if one might
> do something like that)? What if the packets take an extra round-trip
> before it hits the target the "right" way (i.e. so it will respond). I
> could also imagine that a packet is traveling several times through the
> same gateway, for instance based on packet type and source/destination
> addresses or incoming interface at the gw.

Also, there are times when the final response comes from a *different*
address than the destination. This happens when the target host has
multiple addresses, and it sends the response from one of its other
addresses.

So the most reliable way to tell that you've reached the final
destination is to look for a response other than ICMP TTL Exceeded.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

>So the most reliable way to tell that you've reached the final
>destination is to look for a response other than ICMP TTL Exceeded.

What if we get a response from the actual target ip address?
Should we care what the ICMP code is ?

In article <1146547907.150747.123160@v46g2000cwv.googlegroups .com>,
"VivekRajan" <vivek_rajagopal@yahoo.com> wrote:

> >So the most reliable way to tell that you've reached the final
> >destination is to look for a response other than ICMP TTL Exceeded.
>
> What if we get a response from the actual target ip address?
> Should we care what the ICMP code is ?

Yes. If the response is TTL Exceeded, then it's apparently a router
with port forwarding enabled, so you should keep going. Which seems to
be preciely what was happening in the original post.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Barry Margolin wrote:
> Yes. If the response is TTL Exceeded, then it's apparently a router
> with port forwarding enabled, so you should keep going.

Yes, and it doesn't necessarily need to have port forwarding enabled
either. It could simply (for reasons unknown) have been configured to
forward all traffic for itself to another box, or could for instance
distinguish between the different protocols.


> Which seems to be preciely what was happening in the original post.

Obviously! And I've found that the same thing happens with several other
destination addresses too. However, I start to think that maybe this
could be a poor or erroneous configuration in the specific case, because
once packets have "escaped" past the destination they seem to be lost
forever (or dropped). Anyway it makes diagnozing connections a tad
harder, which is bad for the good guys and good for the bad guys ...

--
-+-Ben-+-