I'm ing out with an excessive streams memory usage problem we're
seeing on certain SCO 5.0.5 boxes under our care. Said boxes are at the
latest patch level. No, upgrading them is not an option at this time.

The machines in question "grow" pages of sockets stuck in the SYN_SENT
state; only a reboot will clear them. Notice how each port is used
twice. Example:

(The host name has been changed to protect the innocent)

f1c0ee00 tcp 0 0 HOST.3654 *.*
SYN_SENT
f1c0d600 tcp 0 0 HOST.3654 *.*
SYN_SENT
f23a5a00 tcp 0 0 HOST.3642 *.*
SYN_SENT
f23a4000 tcp 0 0 HOST.3642 *.*
SYN_SENT
f1c0ea00 tcp 0 0 HOST.3630 *.*
SYN_SENT
f1c0d200 tcp 0 0 HOST.3630 *.*
SYN_SENT
f1c0e400 tcp 0 0 HOST.3619 *.*
SYN_SENT
f18ebe00 tcp 0 0 HOST.3619 *.*
SYN_SENT
f1c0dc00 tcp 0 0 HOST.3608 *.*
SYN_SENT
f1c0d800 tcp 0 0 HOST.3608 *.*
SYN_SENT
f18ebc00 tcp 0 0 HOST.3596 *.*
SYN_SENT

SCO "netstat" is not particularly full featured and "lsof -i
tcp:[TROUBLE PORT]" (ver. 4.51) gives no output; it's almost as if
something is trying to initiate a connection without defining a
destination address. I'm having a devil of a time identifying what
process is causing these hung connections. These are production
machines so I cannot tinker with them much. Of course I can't reproduce
the problem in our lab.


Additionally, I'm seeing the host machine connect to itself on
duplicate port numbers. My understanding is that there's nothing
outright egregious in the following, but something seems fishy about
having dozens of similar pairs open.

f1eaaa00 tcp 0 0 HOST.1037 HOST.1121
ESTABLISHED
f1eaac00 tcp 0 0 HOST.1121 HOST.1037
ESTABLISHED
f1eaa600 tcp 0 0 HOST.1056 HOST.1116
ESTABLISHED
f1eaa800 tcp 0 0 HOST.1116 HOST.1056
ESTABLISHED
f1319200 tcp 0 0 HOST.1061 HOST.1112
ESTABLISHED
f1319400 tcp 0 0 HOST.1112 HOST.1061
ESTABLISHED

Sometimes the port numbers are in clean pairs as in the above and other
times I'll see a particular port in use on many multiple connections.
In the latter case, "lsof" reveals a Java process is involved and the
machine will have a tangled web of self referential ESTABLISHED
connections.

It's been a long time since I've messed with Unix sockets and I feel
like I'm overlooking something painfully obvious.

Thanks for any suggestions.

In article <1162306990.563572.18040@m7g2000cwm.googlegroups.c om>,
phreon@gmail.com wrote:

> I'm ing out with an excessive streams memory usage problem we're
> seeing on certain SCO 5.0.5 boxes under our care. Said boxes are at the
> latest patch level. No, upgrading them is not an option at this time.
>
> The machines in question "grow" pages of sockets stuck in the SYN_SENT
> state; only a reboot will clear them. Notice how each port is used
> twice. Example:
>
> (The host name has been changed to protect the innocent)
>
> f1c0ee00 tcp 0 0 HOST.3654 *.*
> SYN_SENT
> f1c0d600 tcp 0 0 HOST.3654 *.*
> SYN_SENT
> f23a5a00 tcp 0 0 HOST.3642 *.*
> SYN_SENT
> f23a4000 tcp 0 0 HOST.3642 *.*
> SYN_SENT
> f1c0ea00 tcp 0 0 HOST.3630 *.*
> SYN_SENT
> f1c0d200 tcp 0 0 HOST.3630 *.*
> SYN_SENT
> f1c0e400 tcp 0 0 HOST.3619 *.*
> SYN_SENT
> f18ebe00 tcp 0 0 HOST.3619 *.*
> SYN_SENT
> f1c0dc00 tcp 0 0 HOST.3608 *.*
> SYN_SENT
> f1c0d800 tcp 0 0 HOST.3608 *.*
> SYN_SENT
> f18ebc00 tcp 0 0 HOST.3596 *.*
> SYN_SENT

This doesn't make much sense to me. How can you send a SYN to an
unspecified remote address and port? The foreign address can only be
*.* on a LISTENING socket. This smells like an OS bug to me. Have you
talked to the vendor?

>
> SCO "netstat" is not particularly full featured and "lsof -i
> tcp:[TROUBLE PORT]" (ver. 4.51) gives no output; it's almost as if
> something is trying to initiate a connection without defining a
> destination address. I'm having a devil of a time identifying what
> process is causing these hung connections. These are production
> machines so I cannot tinker with them much. Of course I can't reproduce
> the problem in our lab.
>
>
> Additionally, I'm seeing the host machine connect to itself on
> duplicate port numbers. My understanding is that there's nothing
> outright egregious in the following, but something seems fishy about
> having dozens of similar pairs open.

When a computer connects to itself, it's normal for them to appear in
pairs like this. There's one socket in the client process, and another
socket with the opposite port numbers in the server process. All these
random port numbers are probably RPC clients and servers.

>
> f1eaaa00 tcp 0 0 HOST.1037 HOST.1121
> ESTABLISHED
> f1eaac00 tcp 0 0 HOST.1121 HOST.1037
> ESTABLISHED
> f1eaa600 tcp 0 0 HOST.1056 HOST.1116
> ESTABLISHED
> f1eaa800 tcp 0 0 HOST.1116 HOST.1056
> ESTABLISHED
> f1319200 tcp 0 0 HOST.1061 HOST.1112
> ESTABLISHED
> f1319400 tcp 0 0 HOST.1112 HOST.1061
> ESTABLISHED
>
> Sometimes the port numbers are in clean pairs as in the above and other
> times I'll see a particular port in use on many multiple connections.
> In the latter case, "lsof" reveals a Java process is involved and the
> machine will have a tangled web of self referential ESTABLISHED
> connections.
>
> It's been a long time since I've messed with Unix sockets and I feel
> like I'm overlooking something painfully obvious.
>
> Thanks for any suggestions.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Barry Margolin wrote:
> In article <1162306990.563572.18040@m7g2000cwm.googlegroups.c om>,
> phreon@gmail.com wrote:

<CUT>
> > SYN_SENT
> > f1c0e400 tcp 0 0 HOST.3619 *.*
> > SYN_SENT
> > f18ebe00 tcp 0 0 HOST.3619 *.*
> > SYN_SENT
> > f1c0dc00 tcp 0 0 HOST.3608 *.*
> > SYN_SENT
> > f1c0d800 tcp 0 0 HOST.3608 *.*
> > SYN_SENT
> > f18ebc00 tcp 0 0 HOST.3596 *.*
> > SYN_SENT
>
> This doesn't make much sense to me. How can you send a SYN to an
> unspecified remote address and port? The foreign address can only be
> *.* on a LISTENING socket. This smells like an OS bug to me. Have you
> talked to the vendor?
>
> >

Thanks for the corroboration. I thought the above was an example of a
technically impossible state; I like to verify the incredible before
proceeding too far.

Perhaps this is the manifestation of an OS bug, but I haven't gone to
SCO yet. 5.0.5 has been superceded and their standard answer is
usually, "upgrade". If I was talking about a handful of machines, it
would be a "no brainer", but upgrading several hundred 24x7 boxes
scattered across the country is non-trivial. I'm trying to gather as
much ammunition as possible before invoking their tedious support
process.

I know the standard answer should be, "the operating system won't allow
this", but could I be dealing with a server process so poorly written
that it can request a socket for a connection to an unspecified
destination addresses?


<CUT>
>
> When a computer connects to itself, it's normal for them to appear in
> pairs like this. There's one socket in the client process, and another
> socket with the opposite port numbers in the server process. All these
> random port numbers are probably RPC clients and servers.
>

I suspected as much. I still harbor a suspicion because there *appears*
to be a loose correlation between the number of these ESTABLISHED pairs
(and/or the load on the software subsystem they're attached to) and how
quickly our troublesome SYN_SENT sockets grow.

In the short term, we've increased the streams memory on the worst
offenders, but I view this "solution" as akin to slapping a bandage
over a bleeding shotgun wound.

Thanks

I can't believe I used "worst". Where's a spel chequer wen u kneed won?