Hi all,

I recently discovered a TCP feature that I find quite unnerving.

First, let's consider what happens when a TCP connection is "broken" at
the lower layers, meaning that all IP packets are suddenly lost in one or
both directions. The TCP send buffers will fill up and TCP will diligently
retry. At that point, two things can happen: either the connection will be
re-established at some point (a transient break), or it won't (because of
a changed IP address, for example).

The application, having no way of knowing if the connection will come
back, will eventually time-out and close the socket. As far as it is
concerned, this TCP connection is dead and buried. No data will ever be
sent over it again, in any direction.

Now here's the catch: if the connection is re-established after the
application has closed its socket, TCP will flush the contents of the send
buffer to the other end before actually closing the connection down!
From the point of view of the application, it means this: when you timeout
on a connection and close a socket, there is some piece of data from the
stream you were sending that MAY reach the other end at SOME TIME in the
future.

That is a quite subtle and unexpected behaviour, IMHO. I've experienced
it while uploading files with a retrying FTP client: if a file upload was
interrupted, the client would time-out, close the socket and make a new
connection to restart the upload. But on the server side, the first
connection wasn't closed, so at some point during the new upload, the
contents of the old connection's buffer would pop up from the client's TCP
layer (with the client application unaware of this, since it had closed
its socket) and get mixed into the uploading file, thus messing it up.

I guess having a way of clearing the send buffer on the client side
before closing the socket whould , but then, there could be lingering
packets out there... I really don't know how to make a simple model of
this problem, or how and where to address it.

Any idea?


Mikael

P.S. Both ends are Linux 2.6. I suspect this is "normal" TCP behaviour,
but then, I could be mistaken.

Mikael Bouillot wrote:

> The application, having no way of knowing if the connection will come
> back, will eventually time-out and close the socket. As far as it is
> concerned, this TCP connection is dead and buried. No data will ever be
> sent over it again, in any direction.

Sure, but data that was already sent may or may not have made it to the
other end, you obviously can have no idea.

> Now here's the catch: if the connection is re-established after the
> application has closed its socket, TCP will flush the contents of the send
> buffer to the other end before actually closing the connection down!
> From the point of view of the application, it means this: when you timeout
> on a connection and close a socket, there is some piece of data from the
> stream you were sending that MAY reach the other end at SOME TIME in the
> future.

But that would be the case no matter what. That data might have been
sent already and then the cable cut. The other end will receive it
whenever it gets there, which can be an arbitrarily long amount of
time, and you can't be sure you can stop it since the cable is cut.

> That is a quite subtle and unexpected behaviour, IMHO.

Nonsense. There is nothing you can do about it. If a TCP connection
breaks, the other end might receive some data that was queued and there
is nothing you can do about it. You have no way to know whether the
connection broke before or after the data was sent, and even if you
knew it broke after, you still could not know whether or when the other
end might receive it.

> I've experienced
> it while uploading files with a retrying FTP client: if a file upload was
> interrupted, the client would time-out, close the socket and make a new
> connection to restart the upload. But on the server side, the first
> connection wasn't closed, so at some point during the new upload, the
> contents of the old connection's buffer would pop up from the client's TCP
> layer (with the client application unaware of this, since it had closed
> its socket) and get mixed into the uploading file, thus messing it up.

That's just part of the way networking works. If the network actually
breaks, and the client times out, it has no way to tell the server. Any
data that the client sent might or might not get to the server, and
there's no way you can know when or whether.

> I guess having a way of clearing the send buffer on the client side
> before closing the socket whould , but then, there could be lingering
> packets out there... I really don't know how to make a simple model of
> this problem, or how and where to address it.

Exactly. You can turn off "linger on close", google for SO_LINGER, but
the behavior can still happen, it just becomes less likely. The only
real solution is to design a protocol based on "all or nothing"
transactions where no permanent effects happen until the client send an
"I'm done, commit this" indication.

DS

"Mikael Bouillot" <vgmtuckc.remove-this@corbac.com> wrote in message
news:pan.2006.09.29.10.56.31.677369@corbac.com...
> Hi all,
>
> I recently discovered a TCP feature that I find quite unnerving.
>
> First, let's consider what happens when a TCP connection is "broken" at
> the lower layers, meaning that all IP packets are suddenly lost in one or
> both directions. The TCP send buffers will fill up and TCP will diligently
> retry. At that point, two things can happen: either the connection will be
> re-established at some point (a transient break), or it won't (because of
> a changed IP address, for example).

this is part of a wider issue - connections are not "known" to be in an "up"
state apart from when packets actually get transferred successfully.

this means that the state at each end of an idle TCP connection (and
anything in between that keeps state) may not stay in sync if the path gets
broken in some way.

we had a problem a while back with control connections for IP phones in a
call centre. With no activity (say overnight while the centre is closed) a
firewall would silently break a connection, but the phone still regarded it
as "up" in the morning - but the 1st call wouldnt work...

FWIW the fix was to implement keepalives.
>
> The application, having no way of knowing if the connection will come
> back, will eventually time-out and close the socket. As far as it is
> concerned, this TCP connection is dead and buried. No data will ever be
> sent over it again, in any direction.
>
> Now here's the catch: if the connection is re-established after the
> application has closed its socket, TCP will flush the contents of the send
> buffer to the other end before actually closing the connection down!
> From the point of view of the application, it means this: when you timeout
> on a connection and close a socket, there is some piece of data from the
> stream you were sending that MAY reach the other end at SOME TIME in the
> future.

Agreed.

Once you give data to the stack it is potentially going to be delivered, no
matter what is done to the connection afterwards.
>
> That is a quite subtle and unexpected behaviour, IMHO. I've experienced
> it while uploading files with a retrying FTP client: if a file upload was
> interrupted, the client would time-out, close the socket and make a new
> connection to restart the upload. But on the server side, the first
> connection wasn't closed, so at some point during the new upload, the
> contents of the old connection's buffer would pop up from the client's TCP
> layer (with the client application unaware of this, since it had closed
> its socket) and get mixed into the uploading file, thus messing it up.

maybe this is an FTP problem - should a file be committed to the file system
before it is complete?
Different operating systems and even apps do this differently.

eg on my w2k box firefox browser downloading a file gives it a temp name of
xxx.xxx.part (AFAIR) during transfer. Once complete it renames it to
xxx.xxx.

i take this to mean that a partial file is in building state so, you should
not treat it as a fixed entity?
>
> I guess having a way of clearing the send buffer on the client side
> before closing the socket whould , but then, there could be lingering
> packets out there... I really don't know how to make a simple model of
> this problem, or how and where to address it.
>
> Any idea?
>
>
> Mikael
>
> P.S. Both ends are Linux 2.6. I suspect this is "normal" TCP behaviour,
> but then, I could be mistaken.
--
Regards

stephen_hope@xyzworld.com - replace xyz with ntl