PHWinfo - Afficher un message

23/03/2006, 15h49

Rick Jones wrote:
> fell.anthony@gmail.com wrote:
> > A dump from the inetstatShow command lists multiple connections in
> > the FIN_WAIT_1 state, some of which have data in their Send-Q.
> > These connections were previously ESTABLISHED connections between a
> > CORBA middle-ware client and server. They remian in the FIN_WAIT_1
> > state for hours.
>
> Interesting. One would indeed expect those connections to "retransmit
> timeout" - are you sure it is the same data in the Send-Q you see each
> time, and is indeed the same connection in FIN_WAIT_1? You might try
> to get a packet trace of traffic to/from those connections to see if
> there are indeed ACKs coming back that may be keeping the connection
> alive. Otherwise, it may be a bug in the TCP stack.
>

First, the OS is VxWorks and its network stack is based on BSD4.3. I
don't know if the data is the same, but the size of the data in the
Send-Q is the same. Yes, its the same connection that is stuck in the
FIN_WAIT_1 state. See partial output from inetstatShow command at
times 17:06, 17:50 and 20:43, respectively.

fe2e73c TCP 0 0 172.23.92.232.1106 172.20.37.21.43091
FIN_WAIT_1
fe2e6b8 TCP 0 4096 172.23.92.232.1105 172.20.60.44.33456
FIN_WAIT_1
fe2e3a0 TCP 0 0 172.23.92.232.1103 172.20.60.44.33456
FIN_WAIT_1
fe2e4a8 TCP 0 0 172.23.92.232.1099 172.20.60.44.33456
FIN_WAIT_1
fe2e190 TCP 0 2048 172.23.92.232.1097 172.20.37.21.43091
FIN_WAIT_1

fe2e73c TCP 0 0 172.23.92.232.1106 172.20.37.21.43091
FIN_WAIT_1
fe2e6b8 TCP 0 4096 172.23.92.232.1105 172.20.60.44.33456
FIN_WAIT_1
fe2e3a0 TCP 0 0 172.23.92.232.1103 172.20.60.44.33456
FIN_WAIT_1
fe2e4a8 TCP 0 0 172.23.92.232.1099 172.20.60.44.33456
FIN_WAIT_1
fe2e190 TCP 0 2048 172.23.92.232.1097 172.20.37.21.43091
FIN_WAIT_1

fe2e73c TCP 0 0 172.23.92.232.1106 172.20.37.21.43091
FIN_WAIT_1
fe2e6b8 TCP 0 4096 172.23.92.232.1105 172.20.60.44.33456
FIN_WAIT_1
fe2e3a0 TCP 0 0 172.23.92.232.1103 172.20.60.44.33456
FIN_WAIT_1
fe2e4a8 TCP 0 0 172.23.92.232.1099 172.20.60.44.33456
FIN_WAIT_1
fe2e190 TCP 0 2048 172.23.92.232.1097 172.20.37.21.43091
FIN_WAIT_1

Wouldn't an ACK send the connections to FIN_WAIT_2 state?

If the receiving end Recv-Q was full, and the FIN packet was stuck
behind other data on the Send-Q, would the protocol still try to
retransmit, or would it recognized the initial FIN was still not sent
and not retransmit?

> > The dump also shows some ESTABLISHED connections with data in the
> > Recv-Q. The amount of data in the Recv-Q does not change over a
> > period of hours, again, connections related to CORBA middle-ware.
>
> That one sounds like a middleware/app issue - either not checking for
> data available, or ignoring data available indications.
>
> > Finally, there is a large amount of data in the Recv-Q of a UDP echo
> > server. The amount of data fluctuates over time, but the Queue is
> > never emptied.
>
> That one sounds a triffle sinister. Unless you know you have
> applications that really need to reach-out and touch the echo service
> you should probably disable it, or at least restrict it to your
> organization's IPs. Particularly a UDP echo service since that one
> would be more vulnerable to IP source address spoofing, allowing an
> attacker to get you to send data to some other poor slob out on the
> network.
>
> > All these partially terminated connections and non-empty data queues
> > exhaust the avialable mbuf pool.
>
> Just how many of these things do you have and how small is your mbuf
> pool?
>

Could the lack of mbufs prevent the echo service (used on restricted
network only) and the CORBA app from taking data off the Recv-Q and
processing it (or in the case of the echo server, only processing a
little at a time as mbufs were freed slowly)?

Looks like the 64 byte clusters (400 allocated) are being exhausted
first. In normal or even heavy operations, the data clusters are very
conservative - only in this bizarre case does it become a problem.

Note - mbufShow dump lists

number of times failed to find space: 215141
number of times waited for space: 0
number of times drained protocols for space: 187039.

What happens when the protocols are drained for space?

Thanks for any info.

> rick jones
> --
> a wide gulf separates "what if" from "if only"
> these opinions are mine, all mine; HP might not want them anyway...

> feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

23/03/2006, 15h49	#3
fell.anthony@gmail.com Messages: n/a Hébergeur:	Re: Network stack question Rick Jones wrote: > fell.anthony@gmail.com wrote: > > A dump from the inetstatShow command lists multiple connections in > > the FIN_WAIT_1 state, some of which have data in their Send-Q. > > These connections were previously ESTABLISHED connections between a > > CORBA middle-ware client and server. They remian in the FIN_WAIT_1 > > state for hours. > > Interesting. One would indeed expect those connections to "retransmit > timeout" - are you sure it is the same data in the Send-Q you see each > time, and is indeed the same connection in FIN_WAIT_1? You might try > to get a packet trace of traffic to/from those connections to see if > there are indeed ACKs coming back that may be keeping the connection > alive. Otherwise, it may be a bug in the TCP stack. > First, the OS is VxWorks and its network stack is based on BSD4.3. I don't know if the data is the same, but the size of the data in the Send-Q is the same. Yes, its the same connection that is stuck in the FIN_WAIT_1 state. See partial output from inetstatShow command at times 17:06, 17:50 and 20:43, respectively. fe2e73c TCP 0 0 172.23.92.232.1106 172.20.37.21.43091 FIN_WAIT_1 fe2e6b8 TCP 0 4096 172.23.92.232.1105 172.20.60.44.33456 FIN_WAIT_1 fe2e3a0 TCP 0 0 172.23.92.232.1103 172.20.60.44.33456 FIN_WAIT_1 fe2e4a8 TCP 0 0 172.23.92.232.1099 172.20.60.44.33456 FIN_WAIT_1 fe2e190 TCP 0 2048 172.23.92.232.1097 172.20.37.21.43091 FIN_WAIT_1 fe2e73c TCP 0 0 172.23.92.232.1106 172.20.37.21.43091 FIN_WAIT_1 fe2e6b8 TCP 0 4096 172.23.92.232.1105 172.20.60.44.33456 FIN_WAIT_1 fe2e3a0 TCP 0 0 172.23.92.232.1103 172.20.60.44.33456 FIN_WAIT_1 fe2e4a8 TCP 0 0 172.23.92.232.1099 172.20.60.44.33456 FIN_WAIT_1 fe2e190 TCP 0 2048 172.23.92.232.1097 172.20.37.21.43091 FIN_WAIT_1 fe2e73c TCP 0 0 172.23.92.232.1106 172.20.37.21.43091 FIN_WAIT_1 fe2e6b8 TCP 0 4096 172.23.92.232.1105 172.20.60.44.33456 FIN_WAIT_1 fe2e3a0 TCP 0 0 172.23.92.232.1103 172.20.60.44.33456 FIN_WAIT_1 fe2e4a8 TCP 0 0 172.23.92.232.1099 172.20.60.44.33456 FIN_WAIT_1 fe2e190 TCP 0 2048 172.23.92.232.1097 172.20.37.21.43091 FIN_WAIT_1 Wouldn't an ACK send the connections to FIN_WAIT_2 state? If the receiving end Recv-Q was full, and the FIN packet was stuck behind other data on the Send-Q, would the protocol still try to retransmit, or would it recognized the initial FIN was still not sent and not retransmit? > > The dump also shows some ESTABLISHED connections with data in the > > Recv-Q. The amount of data in the Recv-Q does not change over a > > period of hours, again, connections related to CORBA middle-ware. > > That one sounds like a middleware/app issue - either not checking for > data available, or ignoring data available indications. > > > Finally, there is a large amount of data in the Recv-Q of a UDP echo > > server. The amount of data fluctuates over time, but the Queue is > > never emptied. > > That one sounds a triffle sinister. Unless you know you have > applications that really need to reach-out and touch the echo service > you should probably disable it, or at least restrict it to your > organization's IPs. Particularly a UDP echo service since that one > would be more vulnerable to IP source address spoofing, allowing an > attacker to get you to send data to some other poor slob out on the > network. > > > All these partially terminated connections and non-empty data queues > > exhaust the avialable mbuf pool. > > Just how many of these things do you have and how small is your mbuf > pool? > Could the lack of mbufs prevent the echo service (used on restricted network only) and the CORBA app from taking data off the Recv-Q and processing it (or in the case of the echo server, only processing a little at a time as mbufs were freed slowly)? Looks like the 64 byte clusters (400 allocated) are being exhausted first. In normal or even heavy operations, the data clusters are very conservative - only in this bizarre case does it become a problem. Note - mbufShow dump lists number of times failed to find space: 215141 number of times waited for space: 0 number of times drained protocols for space: 187039. What happens when the protocols are drained for space? Thanks for any info. > rick jones > -- > a wide gulf separates "what if" from "if only" > these opinions are mine, all mine; HP might not want them anyway... > feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...