We are having FTP performance problems when taking a certain IP route
through our network.

A packet capture shows an Active FTP session riddled with tons of TCP
Duplicate Acks by the receiver and subsequent Retransmissions by the
sender (CCC).... no wonder the user experience was very poor. However,
although we have been able to identify the symptoms, the root cause has
eluded us. Our analysis shows that the sender appears to be properly
transmitting his FTP Data packets, but the receiver is sending a normal
ACK followed immediately by a DUP ACK. This ongoing "paranoid" behavior
by the receiver results in Retransmissions by the sender, thus clogging
up the flow with a snowballing effect.

The strange thing here is that according to the timestamps, there is no
need for the receiver to be sending a DUP ACK because there appears to
be no delay whatsoever. So what else would cause the receiver to feel
the need to send a DUP ACK if there was no obvious delay ?

tmcgov05 wrote:

> A packet capture shows an Active FTP session riddled with tons of TCP
> Duplicate Acks by the receiver and subsequent Retransmissions by the
> sender (CCC).... no wonder the user experience was very poor. However,
> although we have been able to identify the symptoms, the root cause has
> eluded us

It sounds like the root cause is packet loss. Where is your sniffer
located?

> Our analysis shows that the sender appears to be properly
> transmitting his FTP Data packets, but the receiver is sending a normal
> ACK followed immediately by a DUP ACK.

A dupACK is a normal ACK. It's just *another* normal ACK.

The fact that you're seeing the data sent correctly indicates that
nothing is wrong with the sender, nor with the network equipment up to
the point of the sniffer, but that packets (segments) aren't correctly
arriving at the receiver. Either the network isn't delivering them, or
the receiver can't hear them.

> This ongoing "paranoid" behavior
> by the receiver results in Retransmissions by the sender, thus clogging
> up the flow with a snowballing effect.

The receiver isn't paranoid. He's telling you that data isn't showing
up in order.

Congestion control on the sender should prevent a snowball effect. The
overall packet rate should be very low when dupACKs are rolling in.

> The strange thing here is that according to the timestamps, there is no
> need for the receiver to be sending a DUP ACK because there appears to
> be no delay whatsoever. So what else would cause the receiver to feel
> the need to send a DUP ACK if there was no obvious delay ?

dupACKs are not a product of delay. They're a product of
missing/misordered segments.

Try deploying two sniffers: one very close to the server, the other
very close to the client.

See if all client-bound packets observed by the server sniffer are
observed by the client sniffer.

/chris

Thanks Chris,

The Sniffer is monitoring the router interface closest to the FTP
server (sender). I totally agree about getting a capture at the other
end but unfortunately the receiver is another company and they are not
cooperating very well. The strange thing is this: the problem goes
away and I get a perfectly ordered packet capture when changing a
static route on the aformentioned router from the HSRP virtual address
to the physical address of WAN router 2. Note that WAN router 1&2 are
connected with a /30 ethernet segment running OSPF and all traffic is
designed to get routed over WAN router2. So when the virtual next-hop
is used, there is that extra hop... which is obviously causing the
problem... just can't figure out why. It doesn't that we dont have
access to the WAN routers (provider managed).

.3 (Primary)--------------WAN router 1
---------------->
My Router------.5 (virtual HSRP)
.4 (Secondary)--------------WAN router 2
----------------->

tmcgov05 wrote:
> Thanks Chris,
>
> The Sniffer is monitoring the router interface closest to the FTP
> server (sender). I totally agree about getting a capture at the other
> end but unfortunately the receiver is another company and they are not
> cooperating very well. The strange thing is this: the problem goes
> away and I get a perfectly ordered packet capture when changing a
> static route on the aformentioned router from the HSRP virtual address
> to the physical address of WAN router 2. Note that WAN router 1&2 are
> connected with a /30 ethernet segment running OSPF and all traffic is
> designed to get routed over WAN router2. So when the virtual next-hop
> is used, there is that extra hop... which is obviously causing the
> problem... just can't figure out why. It doesn't that we dont have
> access to the WAN routers (provider managed).
>
> .3 (Primary)--------------WAN router 1
> ---------------->
> My Router------.5 (virtual HSRP)
> .4 (Secondary)--------------WAN router 2
> ----------------->

Interesting.

I presume you mean that the routers are connected with an *additional*
ethernet segment to the one where the HSRP router lives? The setup
you've described doesn't fit in a /30. Sounds like there is a
crossover cable between the WAN routers.

My guess:

You have a duplex mismatch on the crossover interfaces of the WAN
routers.

What would I do?

I'd get an ethernet tap on the crossover, and listen to the CDP
messages being advertised by each router. Those messages should tell
you the operating modes of each router's transceiver.

You can build a tap for just a few dollars:
http://www.snort.org/docs/tap/

/chris

If part of the "bad" path is a trunked/bonded/aggregated link and
doing something foolish like round-robin packet scheduling across the
links the segments could be getting reordered, which would cause the
reciever to send an immediate ACK upon reciept of each out-of-order
segment. That ACK would be for the last in-order segment received and
would therefore look like a duplicate.

If you can get cooperation from the receiver, have them take a
snapshot of TCP statistics on the FTP server from two points during
your FTP transfer and run them (if possible) through the likes of
beforeafter:

ftp://ftp.cup.hp.com/dist/networking/tools/

and see if the receiver is reporting lots of out-of-order segments.
Also see if they are receiving lots (for some definition of "lots"
which would be as a percentage of total traffic) of completely
duplicate packets.

Lots of out-of-order and dups suggest either a fubar sender, or packet
reordering in the network triggering false rtx.

Lots of out-of-order without lots of dups suggest honest-to-goodness
packet loss in the network.

It might be a real pain, but one other way to see if it is reordering
rather than packet loss is to "cripple" the "send window" on your FTP
client - say by shrinking his send socket buffer to something no more
than about 3 or 4x the TCP MSS between your client and the server.
The idea is to keep there from being enough TCP segments outstanding
on the connection to trigger fast retransmit. Run that on both the
"good" and the "bad" config and be prepared for it to be rather
slower. If your dup ACKs and retrans go away it means either the link
was saturated, or it was doing that round-robin thing. Not
conclusive, but it would be another data point that didn't require
cooperation from either the network provider or the FTP server admin.

rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

right - I am referring to an additional segment between the routers.
However, I have eliminated that as a cause because additional captures
verify that destinations that use the Primary HSRP router's WAN link -
WAN router 1 in the diagram - (meaning they never cross that segment)
also suffer from the exact same problem of tons of Dup ACKS and
Retransmissions. I really suspect that the provider's HSRP
implementation is bad (even though comfigs look good). I am going to
further test this by changing the static route of one of the
aformentioned destinations to be the physical IP address of the Primary
HSRP router instead of the virtual, do another packet capture, and see
if TCP stabilizes.

Thanks for all of your insight.

Thanks Rick - this is interesting stuff. I just may have to give that a
try.

Rick Jones wrote:
> If part of the "bad" path is a trunked/bonded/aggregated link and
> doing something foolish like round-robin packet scheduling across the
> links the segments could be getting reordered, which would cause the
> reciever to send an immediate ACK upon reciept of each out-of-order
> segment. That ACK would be for the last in-order segment received and
> would therefore look like a duplicate.

Very insightful, Rick. The OP mentioned seeing TCP retransmissions,
but it's possible that the connection is experiencing minor packet
loss, with consistent reordering.

That is to say, maybe packets are showing up like this: 2,1,4,3,6,5,8,7

That would result in a lot of dupACKs, but not many retransmissions,
which I think is the scenario you're describing.

> If you can get cooperation from the receiver, have them take a
> snapshot of TCP statistics on the FTP server from two points during
> your FTP transfer and run them (if possible) through the likes of
> beforeafter:
>
> ftp://ftp.cup.hp.com/dist/networking/tools/
>
> and see if the receiver is reporting lots of out-of-order segments.
> Also see if they are receiving lots (for some definition of "lots"
> which would be as a percentage of total traffic) of completely
> duplicate packets.

The OP might be able to learn something by looking only at the sender's
TCP stats: If the number of TCPinDupAcks (or somesuch) is increasing
rapidly, but the number of TCPRetransmissions (or somesuch) is not,
then it sounds like a misordering problem.

> Lots of out-of-order and dups suggest either a fubar sender, or packet
> reordering in the network triggering false rtx.
>
> Lots of out-of-order without lots of dups suggest honest-to-goodness
> packet loss in the network.

I think Rick means dup *segments* in those statements, not dupACKs.

I've thought of another scenario: How sure are you that the traffic is
only going across one of your WAN links? Perhaps your HSRP primary is
sending some of the traffic to the WAN, and some of the traffic to the
other WAN router?

Can you pull the WAN cable from the WAN router 2 (.3)?

/chris

tmcgov05 wrote:
> right - I am referring to an additional segment between the routers.
> However, I have eliminated that as a cause because additional captures
> verify that destinations that use the Primary HSRP router's WAN link -
> WAN router 1 in the diagram - (meaning they never cross that segment)

Please clarify. I don't understand how you know that they're not using
the crossover segment.

All kinds of goofy things can be done on an IRL.

/chris

"tmcgov05" <tmcgov@gmail.com> wrote in message
news:1143048021.894578.297240@z34g2000cwc.googlegr oups.com...
> Thanks Chris,
>
> The Sniffer is monitoring the router interface closest to the FTP
> server (sender). I totally agree about getting a capture at the other
> end but unfortunately the receiver is another company and they are not
> cooperating very well. The strange thing is this: the problem goes
> away and I get a perfectly ordered packet capture when changing a
> static route on the aformentioned router from the HSRP virtual address
> to the physical address of WAN router 2. Note that WAN router 1&2 are
> connected with a /30 ethernet segment running OSPF and all traffic is
> designed to get routed over WAN router2. So when the virtual next-hop
> is used, there is that extra hop... which is obviously causing the
> problem... just can't figure out why. It doesn't that we dont have
> access to the WAN routers (provider managed).

i have seen a similar effect when there is something wrong at layer 2
between the 2 HSRP router interfaces - if the local LAN doesnt forward the
HSRP keepalive traffic properly, then you can end up with both router ports
as "HSRP active" - the net effect is that 2 copies of the packet gets
forwarded, 1 by each router.

this would explain why pointing the static route at a physical IP makes the
problem vanish....
>
> .3 (Primary)--------------WAN router 1
> ---------------->
> My Router------.5 (virtual HSRP)
> .4 (Secondary)--------------WAN router 2
> ----------------->
--
Regards

stephen_hope@xyzworld.com - replace xyz with ntl

Yes - Exactly what I am now investigating. Did you recall what you did
to correct this ? An IOS upgrade perhaps ?

>> Lots of out-of-order and dups suggest either a fubar sender, or packet
>> reordering in the network triggering false rtx.
>>
>> Lots of out-of-order without lots of dups suggest honest-to-goodness
>> packet loss in the network.

> I think Rick means dup *segments* in those statements, not dupACKs.

That is correct, I did mean data segments, not ACKs.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

btw - This is a Token Ring interface ugh!

"tmcgov05" <tmcgov@gmail.com> wrote in message
news:1143059120.058133.66560@e56g2000cwe.googlegro ups.com...
> Yes - Exactly what I am now investigating. Did you recall what you did
> to correct this ? An IOS upgrade perhaps ?

IOS could cause something like this, but i suspect it is in the underlying
switch(es) connecting the 2 routers rather than a router - although if you
have the same MAC address on both router LAN interfaces i think you can make
this happen.

a 1 way link where a packet can get to the router from the LAN, but HSRP
keepalives cannot get back can also cause it.

i 1st saw this on a shared network (FDDI ring - long time ago) - that one
was an IOS bug.
>
--
Regards

stephen_hope@xyzworld.com - replace xyz with ntl

In article <1143059120.058133.66560@e56g2000cwe.googlegroups. com>,
"tmcgov05" <tmcgov@gmail.com> wrote:

> Yes - Exactly what I am now investigating. Did you recall what you did
> to correct this ? An IOS upgrade perhaps ?

Which of the many replies are you referring to?

You're using Google, so please read AND OBEY
<http://cfaj.freeshell.org/google/>.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***