I'm about to start implementing an NNTP client. I'm considering having it
read and process one full "line" (by which I mean "series of bytes
terminated by CR LF") at a time.

My question, which I am unable to answer even after reading RFC 977, is:
Does NNTP guarantee a maximum size for each of these "lines"?

tjb <tjb@invalid.invalid> wrote:
> My question, which I am unable to answer even after reading RFC 977, is:
> Does NNTP guarantee a maximum size for each of these "lines"?

I ask because I would like to be able to use a buffer whose size is /the
maximum size/ (assuming the concept exists), rather than having to resize
the buffer whenever I encounter a line that's "too long".

In article <hb99i9sve65w.dlg@tjb.invalid.invalid>,
tjb <tjb@invalid.invalid> wrote:

> I'm about to start implementing an NNTP client. I'm considering having it
> read and process one full "line" (by which I mean "series of bytes
> terminated by CR LF") at a time.
>
> My question, which I am unable to answer even after reading RFC 977, is:
> Does NNTP guarantee a maximum size for each of these "lines"?

RFC 850 specifies the format of Usenet messages, and this format is
based on RFC 822, the format of Internet mail messages.

RFC 822 doesn't specify a maximum line length, but its replacement, RFC
2822, says that the maximum is 998, not including the CRLF. Also, RFC
821, which specifies the SMTP protocol used to transmit Internet mail,
says that the maximum text line length is 1000 bytes including the CRLF,
so this effectively put a limit on RFC 822 messages.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Barry Margolin <barmar@alum.mit.edu> wrote:
>> I'm about to start implementing an NNTP client. I'm considering having it
>> read and process one full "line" (by which I mean "series of bytes
>> terminated by CR LF") at a time.
>>
>> My question, which I am unable to answer even after reading RFC 977, is:
>> Does NNTP guarantee a maximum size for each of these "lines"?
>
> RFC 850 specifies the format of Usenet messages, and this format is
> based on RFC 822, the format of Internet mail messages.
>
> RFC 822 doesn't specify a maximum line length, but its replacement, RFC
> 2822, says that the maximum is 998, not including the CRLF. Also, RFC
> 821, which specifies the SMTP protocol used to transmit Internet mail,
> says that the maximum text line length is 1000 bytes including the CRLF,
> so this effectively put a limit on RFC 822 messages.

I'm not sure if I'm being clear on what I mean by "line" here. What I mean
is a series of bytes (terminated by CRLF) in an NNTP server's response.
For example: "200 news.mozilla\r\n". Or "211 29291 2 29292
mozilla.support.firefox\r\n". These are examples of what I mean by "line"
here. Sorry for any confusion.

To put the question differently: In an NNTP client, what buffer size
should one use to store sequences of bytes terminated by CRLF read in from
an NNTP server?

Thank you.

In article <1tln2khwc0c7f$.dlg@tjb.invalid.invalid>,
tjb <tjb@invalid.invalid> wrote:

> tjb <tjb@invalid.invalid> wrote:
> > My question, which I am unable to answer even after reading RFC 977, is:
> > Does NNTP guarantee a maximum size for each of these "lines"?
>
> I ask because I would like to be able to use a buffer whose size is /the
> maximum size/ (assuming the concept exists), rather than having to resize
> the buffer whenever I encounter a line that's "too long".

What do you plan to do if a server doesn't obey the line length limit.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Barry Margolin <barmar@alum.mit.edu> wrote:
>>> My question, which I am unable to answer even after reading RFC 977, is:
>>> Does NNTP guarantee a maximum size for each of these "lines"?
>>
>> I ask because I would like to be able to use a buffer whose size is /the
>> maximum size/ (assuming the concept exists), rather than having to resize
>> the buffer whenever I encounter a line that's "too long".
>
> What do you plan to do if a server doesn't obey the line length limit.

Drop the line and perhaps warn the user, I guess. After all, the client
code must do this at some point, right? Otherwise, a server could send an
indefinitely long line -- perhaps as a result of a bug, or perhaps in a
malicious attempt to crash clients -- and the client would have to deal
with it somehow.

I'm not sure though. I don't know how big a buffer to use here.

In article <uugehdf58su5$.dlg@tjb.invalid.invalid>,
tjb <tjb@invalid.invalid> wrote:

> I'm not sure if I'm being clear on what I mean by "line" here. What I mean
> is a series of bytes (terminated by CRLF) in an NNTP server's response.
> For example: "200 news.mozilla\r\n". Or "211 29291 2 29292
> mozilla.support.firefox\r\n". These are examples of what I mean by "line"
> here. Sorry for any confusion.
>
> To put the question differently: In an NNTP client, what buffer size
> should one use to store sequences of bytes terminated by CRLF read in from
> an NNTP server?
>
> Thank you.

It doesn't look like there's any limit for response lines, only command
lines (512 bytes, including the CRLF).

However, as a practical matter the only part of a response that is
likely to be very long is the newsgroup name. This has to be at most
504 bytes, so that "GROUP <groupname>CRLF" will fit in a 512-byte
command line. If you then add to this the maximum sizes of the other
fields in that 211 response, you'll arrive at the maximum practical line
length.

If you make your buffer 1000 bytes, I don't think you should run into a
problem.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Barry Margolin <barmar@alum.mit.edu> wrote:
> It doesn't look like there's any limit for response lines, only command
> lines (512 bytes, including the CRLF).
>
> However, as a practical matter the only part of a response that is
> likely to be very long is the newsgroup name. This has to be at most
> 504 bytes, so that "GROUP <groupname>CRLF" will fit in a 512-byte
> command line. If you then add to this the maximum sizes of the other
> fields in that 211 response, you'll arrive at the maximum practical line
> length.
>
> If you make your buffer 1000 bytes, I don't think you should run into a
> problem.

Thanks very much.

Just to point out that I'm quite surprised that a maximum "response line"
length isn't strictly defined in NNTP. It all seems far too lenient to me.
Hmm.

tjb <tjb@invalid.invalid> wrote:
> Just to point out that I'm quite surprised that a maximum "response
> line" length isn't strictly defined in NNTP. It all seems far too
> lenient to me.

Many RFC's were written before the massive expansion of the "Internet"
brought-on by "the web," back when malicious intent was much less of a
concern.

rick jones
--
Wisdom Teeth are impacted, people are affected by the effects of events.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

In article <cHOZg.1341$8M6.1132@news.cpqcorp.net>,
Rick Jones <rick.jones2@hp.com> wrote:

>> Just to point out that I'm quite surprised that a maximum "response
>> line" length isn't strictly defined in NNTP. It all seems far too
>> lenient to me.
>
>Many RFC's were written before the massive expansion of the "Internet"
>brought-on by "the web," back when malicious intent was much less of a
>concern.

I don't understand the connection between those two statements. No
matter what anyone writes in any standard or specification, bad code
will do what it does, and good code must cope. For example, the 1982
definition on page 43 of RFC 821 of the maximum line length for SMTP

text line
The maximum total length of a text line including the
<CRLF> is 1000 characters (but not counting the leading
dot duplicated for transparency).

does not constrain junk SMTP clients. It also does not relieve competently
written SMTP servers from doing something reasonable with messages with
lines that are too long.

There is also the the truth of "never attribute to malice that which
can be adequately explained by stupidity." That there is a lot more
email today with excessively long lines than 20 years ago is not due
to malicious intent. It's not that the junk spamware that sends mail
with lines too long is trying to be malicious (in that respect), but
that the authors are stupid.

Talk about how things have darkened since the innocent good ol' days
of the Internet is usually off the mark. Recall the Morris Worm.
Remember contemporary hassles with anonymous FTP servers or oh so cute
netnews (what would now be called) DoS attackers at universities.


Please note in passing that I disagree with other messages in this thread:

- The maximum SMTP line length has been defined since at least RFC 821/822.

- I would not drop drop NNTP or SMTP lines that are too long, but
pretend they end correctly, such as by inserting a NUL that would
be removed when I passed the message along. If I couldn't do that,
I'd reject the whole message, because it is Wrong, Evil, Nasty,
and Wrong for a transport system to modify passing messages.
If I couldn't reject the message (in some protocol other than NNTP
or SMTP), I'd insert <CR><LF> as the smallest damage to the message.
If that were impossible, I'd give up, because dropping a 1000
character line would be over the top.

- "Warning the user" can make sense for an NNTP or other netnews
server checking a submission, but is dangerously wrong when applied
to incoming, flooded messages. It is impossible to reach "the
user" who wrote a non-conforming flooded message. Trying is
likely to get your blacklisted or worse. A server can at most
reject or discard incoming, non-conforming articles.


Vernon Schryver vjs@rhyolite.com

> Talk about how things have darkened since the innocent good ol' days
> of the Internet is usually off the mark. Recall the Morris Worm.
> Remember contemporary hassles with anonymous FTP servers or oh so
> cute netnews (what would now be called) DoS attackers at
> universities.

Maybe I have rose-colored looking-back glasses, but while I will agree
that "the good ol' days" were not completely sweetness and light, I do
think that "today" is more bitter and dark.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Thu, 19 Oct 2006, Rick Jones wrote:
> Maybe I have rose-colored looking-back glasses, but while I will agree
> that "the good ol' days" were not completely sweetness and light, I do
> think that "today" is more bitter and dark.

I agree, but I think that something very important is being overlooked.
In the "good old days", the network had a (largely benevolent) military
dictatorship which could (and did!) exert absolute authority to punish
miscreants and bring the network into line.

The authority of that dictatorship was widely resented, even as the
dictator became civilian instead of military. It was rather fashionable
at the time to advocate anarchy for the network. The last civilian
dictator surrendered power in the early years of the Clinton
administration, and anarchy was left to florish.

That's what we have today; anarchy. And, as has always happens when
anarchy comes about, the result is brigands and warlords, plundered
peasant villages, and walled cities.

We got what we asked for. It's entirely too late to get the old regime
back. And, as it turns out, creating a democracy to establish law and
order turns out to be a damnably hard that usually fails.

The only rational course is to work on your city walls. The brigands and
warlords aren't going to go away any time soon.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

In article <alpine.OSX.0.7.0610191141470.17547@pangtzu.panda. com>,
Mark Crispin <mrc@CAC.Washington.EDU> wrote:
>On Thu, 19 Oct 2006, Rick Jones wrote:
>> Maybe I have rose-colored looking-back glasses, but while I will agree
>> that "the good ol' days" were not completely sweetness and light, I do
>> think that "today" is more bitter and dark.

The attitudes are certainly darker and more bitter, but I don't
think the threats are so much different. Some attacks are more
frequent, but that doesn't change the real dangers. Something I forgot
to mention was the infamous packet sniffing to steal passwords in the Bay Area.
Perhaps a major change is that in the good ol' days the bad news travelled
by word of mouth instead of trade rag espurts and security snake oil
salesslime every flogging real and plenty not so real dangers.

And then there is the Microsoft effect. I don't reall whether (or that)
Microsoft was connected at all in the 1980's, but I'm sure we didn't
have Microsoft Botnets.

>I agree, but I think that something very important is being overlooked.
>In the "good old days", the network had a (largely benevolent) military
>dictatorship which could (and did!) exert absolute authority to punish
>miscreants and bring the network into line.

Did the Dictator ever really zap anyone? I vaguely recall hearing a
very little talk of action, but most of what I saw were threats made
by third parties (like me) to local users in the name of the Dictator,
along the lines of "If you don't stop that, we'll lose our connection
and then you'll really be in trouble!" I also recall some flagrant
violations of the letter of AUP that the Dictator carefully ignored
because they didn't really violate the spirit of the "in support of"
clause.

>The authority of that dictatorship was widely resented, even as the
>dictator became civilian instead of military. It was rather fashionable
>at the time to advocate anarchy for the network. The last civilian
>dictator surrendered power in the early years of the Clinton
>administration, and anarchy was left to florish.
>
>That's what we have today; anarchy. And, as has always happens when
>anarchy comes about, the result is brigands and warlords, plundered
>peasant villages, and walled cities.
>
>We got what we asked for. It's entirely too late to get the old regime
>back.

The irony is the continuing blather from those that variously don't
know better or should (and probably do) about annointing the United
Nations or the ITU as the New Dictator, as if that wouldn't be far worse
than the current or any likely anarchy.

> And, as it turns out, creating a democracy to establish law and
>order turns out to be a damnably hard that usually fails.
>
>The only rational course is to work on your city walls. The brigands and
>warlords aren't going to go away any time soon.

yes.

The only real difference between now and then is that the brigands
and warlords are more likely members of Fortune 500 than foolish
college students. By the old rules, consumer ISPs are responsible
for stopping the use of their networks by botnets as well as vanilla
spammers--er--legitimate unsolicited newsletter operators. Their
10+ years of disinterest makes them brigand warloards.


Vernon Schryver vjs@rhyolite.com

On Thu, 19 Oct 2006, Vernon Schryver wrote:
> The attitudes are certainly darker and more bitter, but I don't
> think the threats are so much different.

I disagree. Some of today's threats (e.g., DDOS) were simply not feasible
to undertake in the past. Bandwidth increased far faster than the ability
of our hardware and software to handle it.

> Some attacks are more
> frequent, but that doesn't change the real dangers. Something I forgot
> to mention was the infamous packet sniffing to steal passwords in the Bay Area.

That too is a comparatively recent development (past 25 or so years). You
couldn't do that on the ARPAnet.

> And then there is the Microsoft effect. I don't reall whether (or that)
> Microsoft was connected at all in the 1980's, but I'm sure we didn't
> have Microsoft Botnets.

For the most part, the only PCs that were connected in the 1980s ran
third-party TCP stacks. Also, this was DOS, not Windows.

> Did the Dictator ever really zap anyone?

Yes.

More common, however, was the mandatory flight to Washington DC to explain
why the order to disconnect should be rescinded. It was a sufficiently
unpleasant experience that the site involved generally did not have a
repeat performance.

> I also recall some flagrant
> violations of the letter of AUP that the Dictator carefully ignored
> because they didn't really violate the spirit of the "in support of"
> clause.

Indeed.

> The irony is the continuing blather from those that variously don't
> know better or should (and probably do) about annointing the United
> Nations or the ITU as the New Dictator, as if that wouldn't be far worse
> than the current or any likely anarchy.

Indeed. But hey, we have kooky kids (especially in the Pacific Northwest)
who think that anarchy is a good thing. So it isn't surprising that the
somewhat less kooky are willing to turn over control to "the international
community" (meaning the UN, ITU, whatever).

> The only real difference between now and then is that the brigands
> and warlords are more likely members of Fortune 500 than foolish
> college students.

Warlords, yes.

Brigands are a different matter; there is quite a bit of organized crime
involvement in the garbage that's going on today. Presently, we will
crown one of the warlords as King, in return for his protection against
the brigands and other warlords.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

In article <Pine.WNT.4.65.0610191635410.420@Tomobiki-Cho.CAC.Washington.EDU>,
Mark Crispin <MRC@CAC.Washington.EDU> wrote:

>> The attitudes are certainly darker and more bitter, but I don't
>> think the threats are so much different.
>
>I disagree. Some of today's threats (e.g., DDOS) were simply not feasible
>to undertake in the past. Bandwidth increased far faster than the ability
>of our hardware and software to handle it.

Maybe so, but I don't see the functional difference between a DoS that
kills a 56 Kbit/sec (or 9600 b/s) line back then and a DoS that kills
routers or hosts connected to Gbit lines today.
Besides, if you consider the Morris Worm a DoS attack, then bandwidth
was not the problem but host CPU cycles.


>> Some attacks are more
>> frequent, but that doesn't change the real dangers. Something I forgot
>> to mention was the infamous packet sniffing to steal passwords in the
>Bay Area.
>
>That too is a comparatively recent development (past 25 or so years). You
>couldn't do that on the ARPAnet.

Yes, but 25 years goes back to 1981 or before. I've claimed to have
met the Internet on the console of TIP-25 (DOCB) in 1972, but when
pressed, I admit that Arpanet wasn't the same as the Internet. It was
too secret, private, and more like Digital's XNS network than the
Internet. I think a real birthday is around CSNET's hooking up lots of
sites in the mid-1980s.


>Brigands are a different matter; there is quite a bit of organized crime
>involvement in the garbage that's going on today. Presently, we will
>crown one of the warlords as King, in return for his protection against
>the brigands and other warlords.

thereby recapitulating the evolution from tribes to frontier anarchy
to feudalism, with no hope of anything better. oh, well.


Vernon Schryver vjs@rhyolite.com

On Thu, 19 Oct 2006, Vernon Schryver wrote:
> >I disagree. Some of today's threats (e.g., DDOS) were simply not feasible
> >to undertake in the past. Bandwidth increased far faster than the ability
> >of our hardware and software to handle it.
> Maybe so, but I don't see the functional difference between a DoS that
> kills a 56 Kbit/sec (or 9600 b/s) line back then and a DoS that kills
> routers or hosts connected to Gbit lines today.

First, a DDOS wasn't feasible; there just weren't enough nodes.

Second, some systems were fast enough that they could handle continuous
maximum bandwidth network input and shrug off the load.

Third, the ARPAnet load-balanced and load-limited sources in ways that
modern networks do not do.

> Besides, if you consider the Morris Worm a DoS attack, then bandwidth
> was not the problem but host CPU cycles.

That was a bug in the broken UNIX systems at the time.

The only impact to the non-UNIX systems (TOPS-20, Multics, ITS) on the net
was the paniced responses of system and network administrators who didn't
realize that non-UNIX systems were utterly immune to the worm.

> >That too is a comparatively recent development (past 25 or so years). You
> >couldn't do that on the ARPAnet.
> Yes, but 25 years goes back to 1981 or before. I've claimed to have
> met the Internet on the console of TIP-25 (DOCB) in 1972, but when
> pressed, I admit that Arpanet wasn't the same as the Internet.

1972 was also when I first met ARPAnet.

> thereby recapitulating the evolution from tribes to frontier anarchy
> to feudalism, with no hope of anything better. oh, well.

Indeed.

However, "frontier anarchy" is more in myth than reality. The history of
the frontier shows that law and order went with the very first settlers.
The notion of an anarchistic nirvana before those nasty lawmen arrived is
not historical reality.

Even tribes had codes of law and order; there were taboos and social more
that the members were obliged to follow.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

In article <alpine.OSX.0.7.0610192213040.29088@pangtzu.panda. com>,
Mark Crispin <mrc@CAC.Washington.EDU> wrote:

>> >I disagree. Some of today's threats (e.g., DDOS) were simply not feasible
>> >to undertake in the past. Bandwidth increased far faster than the ability
>> >of our hardware and software to handle it.

>> Maybe so, but I don't see the functional difference between a DoS that
>> kills a 56 Kbit/sec (or 9600 b/s) line back then and a DoS that kills
>> routers or hosts connected to Gbit lines today.
>
>First, a DDOS wasn't feasible; there just weren't enough nodes.

which is why I wrote "DoS" instead of "DDOS"

>Second, some systems were fast enough that they could handle continuous
>maximum bandwidth network input and shrug off the load.
>
>Third, the ARPAnet load-balanced and load-limited sources in ways that
>modern networks do not do.

yes, but some of that was based on things that even then were best
not discussed in polite society---on the other hand, MPLS can be
seen as x.25.

>> Besides, if you consider the Morris Worm a DoS attack, then bandwidth
>> was not the problem but host CPU cycles.
>
>That was a bug in the broken UNIX systems at the time.

only on the UNIX systems that were broken, only VAX and Sun as I recall,
and none of the other commercial UNIX vendors.

>The only impact to the non-UNIX systems (TOPS-20, Multics, ITS) on the net
>was the paniced responses of system and network administrators who didn't
>realize that non-UNIX systems were utterly immune to the worm.

Ditto for the UNIX brand I cared about...well, I noticed a little
more strange email traffic than usual sent to the corporate gateway
and that some outside sites seemed to be having troubles. I wondered
in passing about a large power outage in the S.F. Bay Area that
somehow wasn't affecting us, but didn't think much of it until the
human hue and cry finally reached us.


>> thereby recapitulating the evolution from tribes to frontier anarchy
>> to feudalism, with no hope of anything better. oh, well.
>
>Indeed.
>
>However, "frontier anarchy" is more in myth than reality. The history of
>the frontier shows that law and order went with the very first settlers.
>The notion of an anarchistic nirvana before those nasty lawmen arrived is
>not historical reality.
>
>Even tribes had codes of law and order; there were taboos and social more
>that the members were obliged to follow.

I was thinking of the taboos, social mores, totems, and so on of NSFNet
and previous when I wrote "tribes"

Perhaps better than "frontier anarchy" would be "urban gangs." I'm
not sure whether the implications of that perspective are more or
less discouraging than the prospect of making the corporate warlords
into royalty.


Vernon Schryver vjs@rhyolite.com

Vernon Schryver <vjs@calcite.rhyolite.com>:
<snip>
> text line
> The maximum total length of a text line including the
> <CRLF> is 1000 characters (but not counting the leading
> dot duplicated for transparency).
</snip>

(As a side-note: Doesn't this mean, then, that I should in fact make my
buffer 1001 bytes in size?)

<snip>
> - I would not drop drop NNTP or SMTP lines that are too long, but
> pretend they end correctly, such as by inserting a NUL that would
> be removed when I passed the message along. If I couldn't do that,
> I'd reject the whole message, because it is Wrong, Evil, Nasty,
> and Wrong for a transport system to modify passing messages.
> If I couldn't reject the message (in some protocol other than NNTP
> or SMTP), I'd insert <CR><LF> as the smallest damage to the message.
> If that were impossible, I'd give up, because dropping a 1000
> character line would be over the top.
</snip>

But what if the server, perhaps as a result of a bug, sends an indefinitely
long line? The processing must stop somewhere, surely. (Or maybe I've
misunderstood you. I'm not 100% certain.)

And couldn't arbitrarily pretending that a CRLF was present potentially
cause problems? For example, how about adding a CRLF between an existing
CRLF (after the CR and before the LF)?

<snip>
> - "Warning the user" can make sense for an NNTP or other netnews
> server checking a submission, but is dangerously wrong when applied
> to incoming, flooded messages. It is impossible to reach "the
> user" who wrote a non-conforming flooded message. Trying is
> likely to get your blacklisted or worse. A server can at most
> reject or discard incoming, non-conforming articles.
</snip>

What I meant by "warning the user" was "warning the user of my news client
(perhaps in an error log) that some of the communication with the NNTP
server was problematic".

Sorry for any confusion. I should have been more specific initially by
pointing out that I'm implementing a *newsreader*.

Rick Jones <rick.jones2@hp.com> wrote:
>> Just to point out that I'm quite surprised that a maximum "response
>> line" length isn't strictly defined in NNTP. It all seems far too
>> lenient to me.
>
> Many RFC's were written before the massive expansion of the "Internet"
> brought-on by "the web," back when malicious intent was much less of a
> concern.

But what if a server decides it'll limit things at 2000 characters? Then
if my client is expecting a 1000-character limit, there'll be problems.

tjb <tjb@invalid.invalid> wrote:
>>> Just to point out that I'm quite surprised that a maximum "response
>>> line" length isn't strictly defined in NNTP. It all seems far too
>>> lenient to me.
>>
>> Many RFC's were written before the massive expansion of the "Internet"
>> brought-on by "the web," back when malicious intent was much less of a
>> concern.
>
> But what if a server decides it'll limit things at 2000 characters? Then
> if my client is expecting a 1000-character limit, there'll be problems.

Well, not necessarily, but potentially.

In article <1uplfyyr3tir5$.dlg@tjb.invalid.invalid>,
tjb <tjb@invalid.invalid> wrote:

>> text line
>> The maximum total length of a text line including the
>> <CRLF> is 1000 characters (but not counting the leading
>> dot duplicated for transparency).

>(As a side-note: Doesn't this mean, then, that I should in fact make my
>buffer 1001 bytes in size?)

What language are you using? In C I would use a buffer of at least 1002
bytes to cover not only the <CR> and the <LF>, but also trailing NUL
that I might add if I used the standard string library routines.

However, no matter what language you are using, if you really need to
ask such a question, please consider learning a little more about the
craft before diving into writing an NNTP server.


>> - I would not drop drop NNTP or SMTP lines that are too long, but
>> pretend they end correctly, such as by inserting a NUL that would
>> be removed when I passed the message along. If I couldn't do that,
>> I'd reject the whole message, because it is Wrong, Evil, Nasty,
>> and Wrong for a transport system to modify passing messages.
>> If I couldn't reject the message (in some protocol other than NNTP
>> or SMTP), I'd insert <CR><LF> as the smallest damage to the message.
>> If that were impossible, I'd give up, because dropping a 1000
>> character line would be over the top.

>But what if the server, perhaps as a result of a bug, sends an indefinitely
>long line? The processing must stop somewhere, surely. (Or maybe I've
>misunderstood you. I'm not 100% certain.)

If you are referring to my first choice of breaking long lines as if
they ended properly, then of course processing stops somewhere. It
stops at the assumed end of line at 1000 bytes. I'd treat the nominally
obligatory <CR><LF> as optional.

>And couldn't arbitrarily pretending that a CRLF was present potentially
>cause problems? For example, how about adding a CRLF between an existing
>CRLF (after the CR and before the LF)?

Yes, those cases must be handled. Treating <CR> and <LF> as optional
might do the trick.

What are you going to do if the server hiccups and inserts an isolated
<CR> or <LF> in into the stream?

That question also suggests that a smaller project would be a better
choice. It might have been better if someone has made such a suggestion
to me decades ago. My first effort at programming was something to
play checkers. It got to 500 cards of Fortran before I got distracted.
The most important thing I didn't know was that I didn't (and still
don't) have enough clues, and that it's good to look at what other
people have found before inventing my own clues.


>Sorry for any confusion. I should have been more specific initially by
>pointing out that I'm implementing a *newsreader*.

Beware that the major problems solved by a useful newsreader are not
in dealing with a server that babbles, forgets <CR><LF>, or anything
similar. Dealing with the server is the easiest, most trivial part of
the exercise. If I needed to name that tune, I'd start by looking at
one of the many open source newsreaders, not particularly to borrow
code but to understand the problems and solutions of other people.


Vernon Schryver vjs@rhyolite.com

Vernon Schryver <vjs@calcite.rhyolite.com> wrote:
>>> text line
>>> The maximum total length of a text line including the
>>> <CRLF> is 1000 characters (but not counting the leading
>>> dot duplicated for transparency).
>
>>(As a side-note: Doesn't this mean, then, that I should in fact make my
>>buffer 1001 bytes in size?)
>
> What language are you using? In C I would use a buffer of at least 1002
> bytes to cover not only the <CR> and the <LF>, but also trailing NUL
> that I might add if I used the standard string library routines.

By "buffer", I'm speaking more of the abstract concept than any concrete
implementation. I'm speaking of a language-independing thing really.

The 1000 includes the CRLF, so I take it you're allowing the extra two
bytes for the possible duplicate dot (and for the NUL), which effectively
answers my question with "yes" as far as I can see.

> However, no matter what language you are using, if you really need to
> ask such a question, please consider learning a little more about the
> craft before diving into writing an NNTP server.

My question is based on someone earlier saying that a 1000-byte buffer
should suffice. But the RFC you quoted, understood along the lines of the
discussion from earlier in this thread, actually seems to suggest 1001.
Thus, I was asking, "Is the 1000 figure *really* correct?"

>>> - I would not drop drop NNTP or SMTP lines that are too long, but
>>> pretend they end correctly, such as by inserting a NUL that would
>>> be removed when I passed the message along. If I couldn't do that,
>>> I'd reject the whole message, because it is Wrong, Evil, Nasty,
>>> and Wrong for a transport system to modify passing messages.
>>> If I couldn't reject the message (in some protocol other than NNTP
>>> or SMTP), I'd insert <CR><LF> as the smallest damage to the message.
>>> If that were impossible, I'd give up, because dropping a 1000
>>> character line would be over the top.
>
>>But what if the server, perhaps as a result of a bug, sends an indefinitely
>>long line? The processing must stop somewhere, surely. (Or maybe I've
>>misunderstood you. I'm not 100% certain.)
>
> If you are referring to my first choice of breaking long lines as if
> they ended properly, then of course processing stops somewhere. It
> stops at the assumed end of line at 1000 bytes. I'd treat the nominally
> obligatory <CR><LF> as optional.

I'm talking more about having to draw the line somewhere, in the
theoretical case of a server sending you an indefinitely long line.

>>And couldn't arbitrarily pretending that a CRLF was present potentially
>>cause problems? For example, how about adding a CRLF between an existing
>>CRLF (after the CR and before the LF)?
>
> Yes, those cases must be handled. Treating <CR> and <LF> as optional
> might do the trick.
>
> What are you going to do if the server hiccups and inserts an isolated
> <CR> or <LF> in into the stream?
>
> That question also suggests that a smaller project would be a better
> choice. It might have been better if someone has made such a suggestion
> to me decades ago. My first effort at programming was something to
> play checkers. It got to 500 cards of Fortran before I got distracted.
> The most important thing I didn't know was that I didn't (and still
> don't) have enough clues, and that it's good to look at what other
> people have found before inventing my own clues.

My statement is more of "is that really a good idea, 'cause there are some
exceptions that cast doubt on it in my mind". I'll give it some more
thought though.

>>Sorry for any confusion. I should have been more specific initially by
>>pointing out that I'm implementing a *newsreader*.
>
> Beware that the major problems solved by a useful newsreader are not
> in dealing with a server that babbles, forgets <CR><LF>, or anything
> similar. Dealing with the server is the easiest, most trivial part of
> the exercise.

Oh, no doubt.

> If I needed to name that tune, I'd start by looking at
> one of the many open source newsreaders, not particularly to borrow
> code but to understand the problems and solutions of other people.

Yes, I have, but I won't assume they're right. Sure, they might seem to
work, but at the end of the day, I'd rather have a comprehensive RFC
explicitly telling all that it should, like the largest a response line
could ever be in an NNTP implementation.

On Fri, 20 Oct 2006, Vernon Schryver wrote:
>>> Besides, if you consider the Morris Worm a DoS attack, then bandwidth
>>> was not the problem but host CPU cycles.
>> That was a bug in the broken UNIX systems at the time.
> only on the UNIX systems that were broken, only VAX and Sun as I recall,
> and none of the other commercial UNIX vendors.

Right.

>> The only impact to the non-UNIX systems (TOPS-20, Multics, ITS) on the net
>> was the paniced responses of system and network administrators who didn't
>> realize that non-UNIX systems were utterly immune to the worm.
> Ditto for the UNIX brand I cared about...well, I noticed a little
> more strange email traffic than usual sent to the corporate gateway
> and that some outside sites seemed to be having troubles. I wondered
> in passing about a large power outage in the S.F. Bay Area that
> somehow wasn't affecting us, but didn't think much of it until