We've been chasing a strange performance problem and I'm looking for
ideas. I don't understand this one at all.

The German office of our company recently installed a new E1 circuit
using the same ISP that we're using in the US. We thought this would
improve performance for VPN traffic between our offices. It improved
latency, but our actual application performance over the VPN got
*worse* instead of better.

We've been doing a lot of testing to understand what's going wrong.
In this testing, we've seen some serious asymmetry in FTP performance
when we cross the ocean. For example, I can download a 10MB file from
the ISP's ftp server in Germany (trying to get our company's circuit
in Germany out of the equation) and get apx. 345 KBytes/sec--much
better than I was expecting. However, when I put that same 10MB file,
I get 145 KBytes/sec. From our German office they run a very similar
test to an FTP server in the US (not mine, so my circuit is out of the
equation) and see asymmetric performance. Here's the kicker--
performance is always bad when the data is going from the US to
Germany and always good when going from Germany to the US. In other
words, we see great performance when GETting in the US from Germany
and PUTting in Germany to the US. We see slow performance when we
PUTting in the US to Germany or GETting in Germany from the US.

Through all of this, latency and packet loss on pings look fine.

The ISP is suggesting we tune the TCP parameters for a high latency
link. While I think this may a little, it doesn't begin to
explain why performance sucks when the data flows east across the
ocean. Nor does it say anything about why we weren't having this
performance problem when our German office was using a different ISP.
I don't have a traceroute for proof, but I'm pretty sure that when our
German office was using the old ISP, data between our sites crossed
the Atlantic on the other ISP's circuits. Now that we're all on the
same ISP, we're crossing the Atlantic on that ISP's circuits.

What kind of a problem gives you good latency numbers but poor
application performance? What kind of problem only shows up when the
big data flows move east?

I'd appreciate any ideas or suggestions.

Thanks

Pat

On Sat, 05 May 2007 19:14:55 -0700, patrick.peters wrote:

> We've been chasing a strange performance problem and I'm looking for
> ideas. I don't understand this one at all.

(snip)

> and see asymmetric performance. Here's the kicker-- performance is
> always bad when the data is going from the US to Germany and always good
> when going from Germany to the US. In other words, we see great
> performance when GETting in the US from Germany and PUTting in Germany
> to the US. We see slow performance when we PUTting in the US to Germany
> or GETting in Germany from the US.

(snip)

> What kind of a problem gives you good latency numbers but poor
> application performance? What kind of problem only shows up when the
> big data flows move east?

Some random thoughts:

Check for pmtud blackholes. If the clients are Windows, they try to work
around pmtud blackholes, giving bad performance on startup.

Capture the problematic and non-problematic sessions with wireshark and
look for differences.

HTH,
M4

patrick.peters@orametrix.com writes:
> What kind of a problem gives you good latency numbers but poor
> application performance? What kind of problem only shows up when the
> big data flows move east?

routing doesn't have to be symmetrical, bandwidth/traffic shaping
doesn't have to be symmetrical, physical links may not be symmetrical,
contention may not be symmetrical (aka link may be symmetrical but
dominant traffic flow may not be symmetrical, aka lots of traffic from
servers in the west to clients in the east).

traceroute from both ends may give whether the hops are the same
(symmetrical routing)

large packet traceroute (from both ends) may give whether the
transmission latency over each individual hop is symmetrical.

Small packets may give similar latency for similar hop ... even if
bandwidth/traffic of the links are significantly different (or different
in different directions). traceroute with different port numbers might
turn up port-number specific traffic shaping.

bandwidth/traffic shaping can occur at ip layer and may even be
udp/tcp/port specific ... and can be asymmetrical. asymmetrical
bandwidth can also occur at lower levels.

wiki page on traffic shaping
http://en.wikipedia.org/wiki/Traffic_shaping

other traffic shaping reference from search engine
http://www.cisco.com/univercd/cc/td/...rt4/qcfrts.htm

any of the possible characteristics might be asymmetrical for any
specific hop.

Are you certian the same TCP window size is used in each direction?
Not all FTP clients/servers use the same settings, and it is possible
to have assymetric performance as a result.

When you say the latency and packet losses on pings look fine, what is
your numeric defintion of "fine" here?-)

Can you reproduce the assymetric performance with something else
besides FTP? Naturally I would suggest netperf, but then I have a
built-in preference http://www.netperf.org/

rick jones
--
The computing industry isn't as much a game of "Follow The Leader" as
it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
- Rick Jones
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Thanks for the responses. Here's some additional information and
questions based on the responses from folks...

I'm kind of leaning toward the traffic shaping theory right now, but I
don't have a good way to prove it.

A point to clarify (or emphasize): If I connect to this ftp server in
Germany I can see the asymmetry between get and put performance. I am
always the client and I am connected to the same server. I would not
expect the routing to change between gets and puts in the same ftp
session. Is this a bad assumption?

I have run a traceroute from the US to Germany and also from Germany
to the US. The routes are effectively the same (looks like different
interfaces on the same routers).

How do I find out if there's a pmtud black hole in my path?

I ran a traceroute with 256 byte packets (the largest I could generate
with the tool I am using) and the transmission latency seems to be
symmetric.

I can see the same kind of asymmetric performance when doing file
transfers through an encrypted VPN between our US and German offices.
The actual content of the traffic (ftp vs. something else) would be
masked by the ipsec encryption. If someone is "shaping" me, they must
be shaping VPN traffic too.

On Saturday evening, round trip ping time (router to router) between
our US and German offices was about 140 msec. Judging from the
tracert, we took on about 80-90 msec of that time crossing the
ocean.

Does anyone know of a good testing tool that would use UDP packets?
I've got Cisco routers on each end and Windows clients on our
networks.

Thanks for the .

Pat

patrick.peters@orametrix.com wrote:
> I'm kind of leaning toward the traffic shaping theory right now, but
> I don't have a good way to prove it.

If the traffic shaping is tweaking on things being FTP, then if you
were to use something other than FTP, presumably the traffic shaping
wouldn't tweak on it, or would tweak on it in a different way.

> A point to clarify (or emphasize): If I connect to this ftp server
> in Germany I can see the asymmetry between get and put performance.
> I am always the client and I am connected to the same server. I
> would not expect the routing to change between gets and puts in the
> same ftp session. Is this a bad assumption?

Some FTP bits will remember to set SO_RCVBUF, but not SO_SNDBUF and so
can have different performance in different directions, particularly
as the path latency increases. Hence why it would be a Really Good
Thing to get a packet trace of both the get and the put, showing the
connection establishement (aka SYN) segments in each case.

> How do I find out if there's a pmtud black hole in my path?

Send an IP datagram of a size such that it would require fragmentation
somewhere along the way, and with the DF bit set in the IP header. If
you get ICMP Destination Unreachable, datagram too big messages in
response, you don't have a Path MTU Discovery black hole. If you
_don't_ get those messages in response, you _may_ have a black hole.

In theory you could try ping with a vastly increased datagram size
from the 64 byte default.

> I ran a traceroute with 256 byte packets (the largest I could generate
> with the tool I am using) and the transmission latency seems to be
> symmetric.

> I can see the same kind of asymmetric performance when doing file
> transfers through an encrypted VPN between our US and German offices.
> The actual content of the traffic (ftp vs. something else) would be
> masked by the ipsec encryption. If someone is "shaping" me, they must
> be shaping VPN traffic too.

> On Saturday evening, round trip ping time (router to router) between
> our US and German offices was about 140 msec. Judging from the
> tracert, we took on about 80-90 msec of that time crossing the
> ocean.

> Does anyone know of a good testing tool that would use UDP packets?
> I've got Cisco routers on each end and Windows clients on our
> networks.

There are both UDP_STREAM and UDP_RR tests in netperf, and netperf is
known to compile and run on Windows. I would suggest some
considerable caution running something like a UDP_STREAM test - there
is no flow control in UDP and you don't want to oversaturate the
network.

Of course, "traffic shapers" could also shape UDP traffic.

rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...