I came across some strangeness while working with libpcap on linux and
I'm wondering if the explanation deals with the way libpcap works or
the way the kernel's IP stack works. Hopefully someone can lend some
insight...

When I load tcpdump on machine-A and send a set number (400) of UDP
packets from machine-B, tcpdump only sees about 392 or 393 of the
packets -- it only prints out information for 392 or 393 of the
packets, and also the statistics that print out when I close tcpdump
say that it only captured that many and that 0 packets were dropped by
the kernel. The same number of packets are lost whether I use
tcpdump's filter or use no filter and later use grep to see how many of
the packets that I sent were reported.

The first explanation to come to mind is that machine-B is simply
sending the packets too quickly for tcpdump to keep up, but when I slow
down the rate that machine-B sends the packets, the same number and
often times even more packets are lost!

When I observed the behavior above, machine-A did not have any program
listening for those UDP packets other than tcpdump. When I loaded a
program that listened for UDP packets, tcpdump suddenly began acting as
I would expect it to -- it would often report on all 400 packets, and
the few times when it did not report on all 400, the number of packets
that it did not report on were accurately presented in the statistics
that tcpdump prints at the end (ie. the number of packets captured +
the number of packets droped by the kernel always equalled 400).

I noticed that the same behavior can be observed in a perl script that
I wrote that uses Net::Pcap to look for packets. Since both Net::Pcap
and tcpdump use libpcap, I'm assuming that the explanation either has
something to do with the internal workings of pcap or the internal
workings of the linux kernel.

I have tested this on linux kernel 2.6.15-1.1833_FC4 and 2.6.9-5.EL as
well as on different network cards (which use different drivers) and I
get the same results. The common denominator is libpcap and the linux
ip stack (which I don't believe has changed much between 2.6.9 and
2.6.15).

Does anyone have any idea what an explanation for this strange behavior
might be?