Hello everyone,

I have a question regarding the following paper:

"Measuring Interactions Between Transport Protocols and Middleboxes"
http://www.imconf.net/imc-2004/papers/p336-medina.pdf

The authors state:

IP packets may contain options to encode additional information at
the end of IP headers. A number of concerns have been raised regarding
the use of IP options. One concern is that the use of IP options may
significantly increase the overhead in routers, because in some cases
packets with IP options are processed on the slow path of the forwarding
engine. A second concern is that receiving IP packets with malformed IP
options may trigger alignment problems on many architectures and OS
versions. Solutions to this problem range from patching the OS, to
blocking access to packets using unknown IP options or using IP options
in general. A third concern is that of possible denial of service
attacks that may be caused by packets with invalid IP options going to
network routers. These concerns, together with the fact that the
generation and processing of IP options is non-mandatory at both the
routers and the end hosts, have led routers, hosts, and middleboxes to
simply drop packets with unknown IP options, or even to drop packets
with standard and properly formed options.

I'm confused by their claim that "the generation and processing of IP
options is non-mandatory at both the routers and the end hosts".

while RFC 791 clearly states:

http://tools.ietf.org/html/rfc791#page-15

The options may appear or not in datagrams. They must be implemented
by all IP modules (host and gateways). What is optional is their
transmission in any particular datagram, not their implementation.

Did an ulterior RFC change that requirement?

I found another paper which seems to corroborate the first paper.
The Need for an Alternative to IPv4-Options
http://www.sm.luth.se/~pierre/main.pdf

I'm playing with an experimental IP option (option number 94).
http://www.iana.org/assignments/ip-parameters

I've noticed that a Cisco Catalyst 2960 Switch starts dropping packets
like mad when handling packets with the 4-byte IPv4 option at rates
above 38 Mbit/s (all is fine at lower rates).

(Perhaps there is a way to configure the switch differently?)

Regards.

Spoon wrote:

> I have a question regarding the following paper:
>
> "Measuring Interactions Between Transport Protocols and Middleboxes"
> http://www.imconf.net/imc-2004/papers/p336-medina.pdf
>
> The authors state:
>
> IP packets may contain options to encode additional information at the
> end of IP headers. A number of concerns have been raised regarding the
> use of IP options. One concern is that the use of IP options may
> significantly increase the overhead in routers, because in some cases
> packets with IP options are processed on the slow path of the forwarding
> engine. A second concern is that receiving IP packets with malformed IP
> options may trigger alignment problems on many architectures and OS
> versions. Solutions to this problem range from patching the OS, to
> blocking access to packets using unknown IP options or using IP options
> in general. A third concern is that of possible denial of service
> attacks that may be caused by packets with invalid IP options going to
> network routers. These concerns, together with the fact that the
> generation and processing of IP options is non-mandatory at both the
> routers and the end hosts, have led routers, hosts, and middleboxes to
> simply drop packets with unknown IP options, or even to drop packets
> with standard and properly formed options.
>
> I'm confused by their claim that "the generation and processing of IP
> options is non-mandatory at both the routers and the end hosts".
>
> while RFC 791 clearly states:
>
> http://tools.ietf.org/html/rfc791#page-15
>
> The options may appear or not in datagrams. They must be implemented
> by all IP modules (host and gateways). What is optional is their
> transmission in any particular datagram, not their implementation.
>
> Did an ulterior RFC change that requirement?

RFC 1812 adds:
Requirements for IP Version 4 Routers
http://tools.ietf.org/html/rfc1812#section-5.3.13.1

5.3.13.1 Unrecognized Options

Unrecognized IP options in forwarded packets MUST be passed
through unchanged.

I take this to mean the packet should not be dropped. Or do I misunderstand?

Regards.

Spoon wrote:

> I've noticed that a Cisco Catalyst 2960 Switch starts dropping packets
> like mad when handling packets with the 4-byte IPv4 option at rates
> above 38 Mbit/s (all is fine at lower rates).

Doh! I meant 3750G not 2960. And it starts dropping packets only when
it is routing, not when it merely switching.

In article <4638a952$0$7190$426a74cc@news.free.fr>, Spoon <devnull@localhost.com> writes:
> Spoon wrote:
>
>> I have a question regarding the following paper:
>>
>> "Measuring Interactions Between Transport Protocols and Middleboxes"
>> http://www.imconf.net/imc-2004/papers/p336-medina.pdf
>>
>> The authors state:
>>
>> IP packets may contain options to encode additional information at the
>> end of IP headers. A number of concerns have been raised regarding the
>> use of IP options. One concern is that the use of IP options may
>> significantly increase the overhead in routers, because in some cases
>> packets with IP options are processed on the slow path of the forwarding
>> engine. A second concern is that receiving IP packets with malformed IP
>> options may trigger alignment problems on many architectures and OS
>> versions. Solutions to this problem range from patching the OS, to
>> blocking access to packets using unknown IP options or using IP options
>> in general. A third concern is that of possible denial of service
>> attacks that may be caused by packets with invalid IP options going to
>> network routers. These concerns, together with the fact that the
>> generation and processing of IP options is non-mandatory at both the
>> routers and the end hosts, have led routers, hosts, and middleboxes to
>> simply drop packets with unknown IP options, or even to drop packets
>> with standard and properly formed options.
>>
>> I'm confused by their claim that "the generation and processing of IP
>> options is non-mandatory at both the routers and the end hosts".
>>
>> while RFC 791 clearly states:
>>
>> http://tools.ietf.org/html/rfc791#page-15
>>
>> The options may appear or not in datagrams. They must be implemented
>> by all IP modules (host and gateways). What is optional is their
>> transmission in any particular datagram, not their implementation.
>>
>> Did an ulterior RFC change that requirement?
>
> RFC 1812 adds:
> Requirements for IP Version 4 Routers
> http://tools.ietf.org/html/rfc1812#section-5.3.13.1
>
> 5.3.13.1 Unrecognized Options
>
> Unrecognized IP options in forwarded packets MUST be passed
> through unchanged.
>
> I take this to mean the packet should not be dropped. Or do I misunderstand?

A packet that is dropped will not be forwarded. A packet that is
not forwarded satisfies the above clause vacuously.

What you're not allowed to do is to strip out or modify IP options
that you don't understand and then forward the resulting packet.

If you don't forward the packet you are free to mess around with
the options before dropping it on the floor. (No one much cares
what you do in the privacy of your own bit bucket).

briggs wrote:
> In article <4638a952$0$7190$426a74cc@news.free.fr>, Spoon <devnull@localhost.com> writes:
>> Spoon wrote:
>>
>>> I have a question regarding the following paper:
>>>
>>> "Measuring Interactions Between Transport Protocols and Middleboxes"
>>> http://www.imconf.net/imc-2004/papers/p336-medina.pdf
>>>
>>> The authors state:
>>>
>>> IP packets may contain options to encode additional information at the
>>> end of IP headers. A number of concerns have been raised regarding the
>>> use of IP options. One concern is that the use of IP options may
>>> significantly increase the overhead in routers, because in some cases
>>> packets with IP options are processed on the slow path of the forwarding
>>> engine. A second concern is that receiving IP packets with malformed IP
>>> options may trigger alignment problems on many architectures and OS
>>> versions. Solutions to this problem range from patching the OS, to
>>> blocking access to packets using unknown IP options or using IP options
>>> in general. A third concern is that of possible denial of service
>>> attacks that may be caused by packets with invalid IP options going to
>>> network routers. These concerns, together with the fact that the
>>> generation and processing of IP options is non-mandatory at both the
>>> routers and the end hosts, have led routers, hosts, and middleboxes to
>>> simply drop packets with unknown IP options, or even to drop packets
>>> with standard and properly formed options.
>>>
>>> I'm confused by their claim that "the generation and processing of IP
>>> options is non-mandatory at both the routers and the end hosts".
>>>
>>> while RFC 791 clearly states:
>>>
>>> http://tools.ietf.org/html/rfc791#page-15
>>>
>>> The options may appear or not in datagrams. They must be implemented
>>> by all IP modules (host and gateways). What is optional is their
>>> transmission in any particular datagram, not their implementation.
>>>
>>> Did an ulterior RFC change that requirement?
>> RFC 1812 adds:
>> Requirements for IP Version 4 Routers
>> http://tools.ietf.org/html/rfc1812#section-5.3.13.1
>>
>> 5.3.13.1 Unrecognized Options
>>
>> Unrecognized IP options in forwarded packets MUST be passed
>> through unchanged.
>>
>> I take this to mean the packet should not be dropped. Or do I misunderstand?
>
> A packet that is dropped will not be forwarded. A packet that is
> not forwarded satisfies the above clause vacuously.
>
> What you're not allowed to do is to strip out or modify IP options
> that you don't understand and then forward the resulting packet.
>
> If you don't forward the packet you are free to mess around with
> the options before dropping it on the floor. (No one much cares
> what you do in the privacy of your own bit bucket).

Thanks for pointing that out.

The RFC has a "MUST NOT drop" provision in a different case:

The IP header contains two reserved bits: one in the Type of Service
byte and the other in the Flags field. A router MUST NOT set either
of these bits to one in datagrams originated by the router. A router
MUST NOT drop (refuse to receive or forward) a packet merely because
one or more of these reserved bits has a non-zero value; i.e., the
router MUST NOT check the values of thes bits.

In article <4638c381$0$28538$426a74cc@news.free.fr>, Spoon <devnull@localhost.com> writes:
> briggs wrote:
>> If you don't forward the packet you are free to mess around with
>> the options before dropping it on the floor. (No one much cares
>> what you do in the privacy of your own bit bucket).
>
> Thanks for pointing that out.
>
> The RFC has a "MUST NOT drop" provision in a different case:
>
> The IP header contains two reserved bits: one in the Type of Service
> byte and the other in the Flags field. A router MUST NOT set either
> of these bits to one in datagrams originated by the router. A router
> MUST NOT drop (refuse to receive or forward) a packet merely because
> one or more of these reserved bits has a non-zero value; i.e., the
> router MUST NOT check the values of thes bits.

That's a rather mild constraint. A router is free to drop any or all
packets. The above clause simply requires it to give equal
[mis-]treatment to packets in which the reserved bits are set.

In particular, this rule does not forbid a router to drop all packets
containing a particular IP option. Some of us run routers that do
exactly that.

In article <wFuW7opAMomj@eisner.encompasserve.org>,
<briggs@encompasserve.org> wrote:

>That's a rather mild constraint. A router is free to drop any or all
>packets. The above clause simply requires it to give equal
>[mis-]treatment to packets in which the reserved bits are set.
>
>In particular, this rule does not forbid a router to drop all packets
>containing a particular IP option. Some of us run routers that do
>exactly that.

Yes, but the reason that is true has nothing to do with any RFC. It
is instead because there are no real Net Cops.

The desired and good reality is that RFCs are only about increasing
the likelihood of interoperating. You are free to do anything you
want no matter how stupid anyone else says it is or how many RFCs
say "MUST NOT," provided you don't trigger too many security or
other active abuse alarms at your provider or your peers.

That is demonstrated by the many people who drop all ICMP packets
to create PMTU Discovery blackholes and then whine about the consequences.
It is also demonstrated by almost all installations that drop IP
packets with IP options. For example, source-routing is between
harmless and useful unless you are running ancient hosts with
insecure applications that shouldn't be exposed naked to the modern
Internet under any circumstances.


Vernon Schryver vjs@rhyolite.com

briggs wrote:

> Spoon wrote:
>
>> briggs wrote:
>>
>>> If you don't forward the packet you are free to mess around with
>>> the options before dropping it on the floor. (No one much cares
>>> what you do in the privacy of your own bit bucket).
>>
>> Thanks for pointing that out.
>>
>> The RFC has a "MUST NOT drop" provision in a different case:
>>
>> The IP header contains two reserved bits: one in the Type of Service
>> byte and the other in the Flags field. A router MUST NOT set either
>> of these bits to one in datagrams originated by the router. A router
>> MUST NOT drop (refuse to receive or forward) a packet merely because
>> one or more of these reserved bits has a non-zero value; i.e., the
>> router MUST NOT check the values of thes bits.
>
> That's a rather mild constraint. A router is free to drop any or all
> packets. The above clause simply requires it to give equal
> [mis-]treatment to packets in which the reserved bits are set.
>
> In particular, this rule does not forbid a router to drop all packets
> containing a particular IP option. Some of us run routers that do
> exactly that.

In your experience, what would these routers do with UDP packets that
carry a single 4-byte IP option defined as:

option-type = 94
i.e. not copied, debugging and measurement
option number 30 (RFC3692-style experiment)
as recommended in RFC 3692 and RFC 4727
option-length = 4
My option's 16-bit "payload" is, in fact, a sequence number.

Would I improve my chances if I set class to 0 (control) i.e. set
option-type to 30?

Since my intention is to carry a 16-bit sequence number, if I can't use
IP options, I thought I could use the identification field in the IP
header. I would set the DF bit. Is this a better alternative?

Regards.

"Spoon" <devnull@localhost.com> wrote in message
news:4638b38d$0$27872$426a74cc@news.free.fr...
> Spoon wrote:
>
> > I've noticed that a Cisco Catalyst 2960 Switch starts dropping packets
> > like mad when handling packets with the 4-byte IPv4 option at rates
> > above 38 Mbit/s (all is fine at lower rates).
>
> Doh! I meant 3750G not 2960. And it starts dropping packets only when
> it is routing, not when it merely switching.

perf of various cisco boxes for routing:
http://www.cisco.com/warp/public/765...erformance.pdf

and the same for switches:
http://www.cisco.com/warp/public/765...erformance.pdf

so looks like you are doing software forwarding, since the 3750 should be
able to route a fair bit faster.

--
Regards

stephen_hope@xyzworld.com - replace xyz with ntl

vjs@calcite.rhyolite.com (Vernon Schryver) writes:
> packets with IP options. For example, source-routing is between
> harmless and useful unless you are running ancient hosts with
> insecure applications that shouldn't be exposed naked to the modern
> Internet under any circumstances.

Y'know, I read this message right after catching up on the food-fight
thread over the IPv6 RT0 header now taking place on a number of
mailing lists. Thanks for a bit of relief. ;-}

--
James Carlson, Solaris Networking <james.d.carlson@sun.com>
Sun Microsystems / 1 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677

stephen wrote:

> Spoon wrote:
>
>> Spoon wrote:
>>
>>> I've noticed that a Cisco Catalyst 2960 Switch starts dropping packets
>>> like mad when handling packets with the 4-byte IPv4 option at rates
>>> above 38 Mbit/s (all is fine at lower rates).
>>
>> Doh! I meant 3750G not 2960. And it starts dropping packets only when
>> it is routing, not when it merely switching.
>
> perf of various cisco boxes for routing:
> http://www.cisco.com/warp/public/765...erformance.pdf
>
> and the same for switches:
> http://www.cisco.com/warp/public/765...erformance.pdf
>
> so looks like you are doing software forwarding, since the 3750 should be
> able to route a fair bit faster.

Is there any way to convince the Cisco 3750G to use the hardware path
when handling IPv4 packets with IP options?

At the moment, I cannot push more than 3300 packets per second. I've
tried with small and large packets (100 bytes and 1450 bytes).

Regards.

"Spoon" <devnull@localhost.com> wrote in message
news:463ae47a$0$3217$426a34cc@news.free.fr...
> stephen wrote:
>
> > Spoon wrote:
> >
> >> Spoon wrote:
> >>
> >>> I've noticed that a Cisco Catalyst 2960 Switch starts dropping packets
> >>> like mad when handling packets with the 4-byte IPv4 option at rates
> >>> above 38 Mbit/s (all is fine at lower rates).
> >>
> >> Doh! I meant 3750G not 2960. And it starts dropping packets only when
> >> it is routing, not when it merely switching.
> >
> > perf of various cisco boxes for routing:
> >
http://www.cisco.com/warp/public/765...erformance.pdf
> >
> > and the same for switches:
> >
http://www.cisco.com/warp/public/765...erformance.pdf
> >
> > so looks like you are doing software forwarding, since the 3750 should
be
> > able to route a fair bit faster.
>
> Is there any way to convince the Cisco 3750G to use the hardware path
> when handling IPv4 packets with IP options?
>
> At the moment, I cannot push more than 3300 packets per second. I've
> tried with small and large packets (100 bytes and 1450 bytes).

i think you have hit the buffers and will struggle.

most hardware acceleration is optimised to do the most common jobs
efficiently, and drop back to software for complex unusual stuff.

processing options in hardware is unlikely to make a box much faster for
typical traffic flows, and doesnt feature in many perforamce tests - so it
looks like the cisco designers have used their available chip real estate on
simple packet forwarding.

however - how often is your Options stream likely to be all packets thru a
specific box?

1 of the design assumptions implied by the Cisco design is that option
packets are rare (and AFAICT it is true for other hardware as well). it
must be an assumption that users dont generally have a problem with, or it
the kit would not be so common....

when you break those kinds of widely held assumptions then you are likely to
have problems.

All you have now is an upper limit to the number of packets with options you
can get thru a 3750G - so maybe you need to think about "seeding" your
options as a small %age of packets into non option traffic if you need high
thruput?
>
> Regards.
--
Regards

stephen_hope@xyzworld.com - replace xyz with ntl

stephen wrote:

> Spoon wrote:
>
>> Is there any way to convince the Cisco 3750G to use the hardware path
>> when handling IPv4 packets with IP options?
>>
>> At the moment, I cannot push more than 3300 packets per second. I've
>> tried with small and large packets (100 bytes and 1450 bytes).
>
> i think you have hit the buffers and will struggle.
>
> most hardware acceleration is optimised to do the most common jobs
> efficiently, and drop back to software for complex unusual stuff.
>
> processing options in hardware is unlikely to make a box much faster for
> typical traffic flows, and doesnt feature in many perforamce tests - so it
> looks like the cisco designers have used their available chip real estate on
> simple packet forwarding.

In my tests, the IP options consist of 4 or 8 NOPs (option type 1).
Thus there are no alignment issues and there is nothing for the router
to process.

> however - how often is your Options stream likely to be all packets thru a
> specific box?
>
> 1 of the design assumptions implied by the Cisco design is that option
> packets are rare (and AFAICT it is true for other hardware as well). it
> must be an assumption that users dont generally have a problem with, or it
> the kit would not be so common....

I'm trying to push 3800 packets per second. They all carry an IP option.

> when you break those kinds of widely held assumptions then you are likely to
> have problems.

OK. I hadn't anticipated IP options would be so problematic.
Time for Plan B, it seems :-)

> All you have now is an upper limit to the number of packets with options you
> can get thru a 3750G - so maybe you need to think about "seeding" your
> options as a small %age of packets into non option traffic if you need high
> thruput?

All my packets carry an IP option: the payload is a sequence number. The
intent was to provide this information in a side-channel, so that the
stream would remain compatible with legacy apps, but new apps could have
the additional information.

In article <463b0fd0$0$3278$426a74cc@news.free.fr>,
Spoon <devnull@localhost.com> wrote:

> In my tests, the IP options consist of 4 or 8 NOPs (option type 1).
> Thus there are no alignment issues and there is nothing for the router
> to process.

But the router doesn't know that until it starts processing them, which
is too late.

Are you expecting there to be an option that will tell the router to
assume that no options need processing? I suspect you're the first one
to ever have a need for such a thing, which is why they've never
implemented it.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Barry Margolin wrote:

> Spoon wrote:
>
>> In my tests, the IP options consist of 4 or 8 NOPs (option type 1).
>> Thus there are no alignment issues and there is nothing for the router
>> to process.
>
> But the router doesn't know that until it starts processing them, which
> is too late.
>
> Are you expecting there to be an option that will tell the router to
> assume that no options need processing? I suspect you're the first one
> to ever have a need for such a thing, which is why they've never
> implemented it.

Your suspicion is not quite correct ;-)

cf. http://www.sm.luth.se/~pierre/main.pdf

"To enable faster implementation of IP options we propose
arranging the option types in a table immediately following
the IP header. The reserved bit in the IP header is used to
indicate the existence of a table of IP-options directly
after the IP header."