I'm hoping someone can clear up these few issues for me...

1. If one were to implement an IMAP server that allowed concurrent
connections to the same mailbox (across multiple servers across NFS),
and one of those connections performed an EXPUNGE command, what would
the (ideal) consequences be for the other connections?

2. Section 5.2 states that a server is "NOT permitted to send an EXISTS
response that would reduce the number of messages in the mailbox". If
another connection expunged messages, then there would certainly be
less.

3. Also, if an expunge occurred, there would be some discrepancy
between the message sequence numbers (in memory) of the currently
selected mailbox and the actual (on disk) mailbox. How would this be
resolved in respect to non-UID based requests?

4. How is the APPENDing of 8-bit data best handled? Should it be
stored as raw 8-bit data, and then converted into base64 when FETCHed,
or base64 encoded on the fly when appending?

Do IMAP clients often APPEND 8-bit data?

Thanks

ondrugs@gmail.com writes:

> I'm hoping someone can clear up these few issues for me...
>
> 1. If one were to implement an IMAP server that allowed concurrent
> connections to the same mailbox (across multiple servers across NFS),
> and one of those connections performed an EXPUNGE command, what would
> the (ideal) consequences be for the other connections?

The other connections would send untagged * EXPUNGE at the first available
opportunity; until then the FETCHs of the expunged messages get rejected
with a suitable error message, and the expunged messages get silently
skipped over by other commands (SEARCH, COPY, et al).

> 2. Section 5.2 states that a server is "NOT permitted to send an EXISTS
> response that would reduce the number of messages in the mailbox". If
> another connection expunged messages, then there would certainly be
> less.

That's what the EXPUNGE untagged message is for. A single EXPUNGE
automatically decreases the number of messages in the mailbox by one.

> 3. Also, if an expunge occurred, there would be some discrepancy
> between the message sequence numbers (in memory) of the currently
> selected mailbox and the actual (on disk) mailbox. How would this be
> resolved in respect to non-UID based requests?

You are assuming that just because IMAP uses consecutive message numbers,
the IMAP server implementation would have to use some kind of an equivalent
scheme of using a consecutive counter for the individual messages, in the
mailbox.

Your assumption is unwarranted.

A particular IMAP server might use a separate file for each message in a
mailbox, with arbitrary filenames, and internally translate between IMAP
message sequence numbers, and the corresponding filenames. Obviously,
removing a file under these circumstances has no impact on the consecutive
message sequence numbers, other than the unavailability of the corresponding
message.

How the server goes about translating messags sequence numbers to actual
messages is out of scope for IMAP. Nothing in IMAP requires a server to
work in any particular way, internally. IMAP only defines an external
protocol between the client and server. How the server goes about doing
what needs to be done, internally, has nothing to do with IMAP.

> 4. How is the APPENDing of 8-bit data best handled? Should it be
> stored as raw 8-bit data, and then converted into base64 when FETCHed,
> or base64 encoded on the fly when appending?

It's implementation-defined. IMAP servers are free to reject the command,
ignore the appended message, or instead fill the mailbox with properly
MIME-encoded porn, when that happens.

> Do IMAP clients often APPEND 8-bit data?

All the time. MIME "Content-Transfer-Encoding: 8bit" often contains 8-bit
characters.

You are confusing "8-bit data" with "raw binary data".

>
> Thanks
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.4 (GNU/Linux)

iD8DBQBE0Vf7x9p3GYHlUOIRAoImAJ4kDY69JS3yODDnEm08MS trPANDMgCfdGgo
p20cfDTbJwN9hqbemIpRkrg=
=B72V
-----END PGP SIGNATURE-----

On Wed, 2 Aug 2006, ondrugs@gmail.com wrote:
> 1. If one were to implement an IMAP server that allowed concurrent
> connections to the same mailbox (across multiple servers across NFS),
> and one of those connections performed an EXPUNGE command, what would
> the (ideal) consequences be for the other connections?

The other connections will receive untagged EXPUNGE events when permitted
in the session. Until an untagged EXPUNGE is permitted, the other
connections can still access the expunged message data as described in RFC
2180 section 4.1.1.

If that turns out to be unimplementable due to server implementation
details, disallow the shared EXPUNGE as described in RFC 2180 section
4.1.4.

Disregard RFC 2180 section 4.1.2 and 4.1.3; those techniques have been
proven to cause problems for clients. I predicted as much, and opposed
those sections being part of RFC 2180, but some people wanted to try those
techniques to see if they were viable. They are not.

> 2. Section 5.2 states that a server is "NOT permitted to send an EXISTS
> response that would reduce the number of messages in the mailbox". If
> another connection expunged messages, then there would certainly be
> less.

That is correct. Only an EXPUNGE response can reduce the number of
messages in the mailbox.

> 3. Also, if an expunge occurred, there would be some discrepancy
> between the message sequence numbers (in memory) of the currently
> selected mailbox and the actual (on disk) mailbox. How would this be
> resolved in respect to non-UID based requests?

That is the purpose of the EXPUNGE response. The instant that an EXPUNGE
response is received, the in-memory copy of the associated message
sequence number is removed, and the message sequence numbers of all
subsequent in-memory messages are immediately decremented by 1.

> 4. How is the APPENDing of 8-bit data best handled? Should it be
> stored as raw 8-bit data, and then converted into base64 when FETCHed,
> or base64 encoded on the fly when appending?

As the ability to APPEND raw 8-bit data is server dependent, and some
servers will refuse raw 8-bit data (or even 8-bit textual data), a prudent
client will convert 8-bit data to 7-bit (either as QUOTED-PRINTABLE or
BASE64) prior to appending.

> Do IMAP clients often APPEND 8-bit data?

IMAP clients should not attempt to append raw 8-bit data to a server which
does not advertise the BINARY extension.

Append of 8-bit textual data, in a context that is properly charset
tagged, is less likely to cause problems; but it is still within a
server's discretion to reject it.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Mark Crispin wrote:
> On Wed, 2 Aug 2006, ondrugs@gmail.com wrote:
> > 1. If one were to implement an IMAP server that allowed concurrent
> > connections to the same mailbox (across multiple servers across NFS),
> > and one of those connections performed an EXPUNGE command, what would
> > the (ideal) consequences be for the other connections?

> The other connections will receive untagged EXPUNGE events when permitted
> in the session. Until an untagged EXPUNGE is permitted, the other
> connections can still access the expunged message data as described in RFC
> 2180 section 4.1.1.

So untagged EXPUNGE responses are permitted at any time? A client
could be sitting idle and suddenly receive a pile of untagged EXPUNGE
responses followed by an untagged EXISTS response when the server
detected that another server had expunged messages, or can these
untagged responses only occur during certain situations?

I had missed RFC 2180... I'll have to read it now. Thanks for pointing
it out.


> > Do IMAP clients often APPEND 8-bit data?

> IMAP clients should not attempt to append raw 8-bit data to a server which
> does not advertise the BINARY extension.

Ahh, sounds like a good compromise for now

Thanks for the quick response.

-Derek

Sam wrote:
> ondrugs@gmail.com writes:
> > 3. Also, if an expunge occurred, there would be some discrepancy
> > between the message sequence numbers (in memory) of the currently
> > selected mailbox and the actual (on disk) mailbox. How would this be
> > resolved in respect to non-UID based requests?

> You are assuming that just because IMAP uses consecutive message numbers,
> the IMAP server implementation would have to use some kind of an equivalent
> scheme of using a consecutive counter for the individual messages, in the
> mailbox.

Well, it would need to keep some sort of mapping between the
consecutive message sequence numbers and the actual messages. If some
in-memory index is kept, it would make sense to have a direct mapping
between the sequence number and the index i.e. like an array
index[i-1]. A translation between this and what's on disk would have
to occur. I expect that using UIDs would be a workable solution.

> How the server goes about translating messags sequence numbers to actual
> messages is out of scope for IMAP. Nothing in IMAP requires a server to
> work in any particular way, internally. IMAP only defines an external
> protocol between the client and server. How the server goes about doing
> what needs to be done, internally, has nothing to do with IMAP.

I understand that, I was just musing about the best solution. The IMAP
implementation I am developing uses an index file of records containing
the UID, flags as well as sizes and offsets into the actual message
file. If records from this index file were expunged by another
process, I'd best be careful when making updates


> > Do IMAP clients often APPEND 8-bit data?

> All the time. MIME "Content-Transfer-Encoding: 8bit" often contains 8-bit
> characters.

> You are confusing "8-bit data" with "raw binary data".

So 8-bit data would still contain linefeeds?

On Wed, 2 Aug 2006, ondrugs@gmail.com wrote:
>> The other connections will receive untagged EXPUNGE events when permitted
>> in the session. Until an untagged EXPUNGE is permitted, the other
>> connections can still access the expunged message data as described in RFC
>> 2180 section 4.1.1.
> So untagged EXPUNGE responses are permitted at any time?

I don't see how that conclusion follows from what I said, particularly
given that I used such phrasing as:
"when permitted"
"Until an untagged EXPUNGE is permitted"

Furthermore, RFC 3501 sections 5.3, 5.5, and 7.4.1 all discuss the fact
that an untagged EXPUNGE response is permitted only at certain points in
the session, and only at those times.

> A client
> could be sitting idle and suddenly receive a pile of untagged EXPUNGE
> responses followed by an untagged EXISTS response when the server
> detected that another server had expunged messages, or can these
> untagged responses only occur during certain situations?

From RFC 3501, section 7.4.1:

An EXPUNGE response MUST NOT be sent when no command is in
progress, nor while responding to a FETCH, STORE, or SEARCH
command.

The corrolary to this, which is also found in RFC 3501 section 5.5, is
that an EXPUNGE response MAY be sent when any command other than FETCH
STORE or SEARCH is in progress.

Remember that, unless specifically prohibited (such as with the EXPUNGE
response), ANY untagged response can be sent at ANY time.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Wed, 2 Aug 2006, ondrugs@gmail.com wrote:
> The IMAP
> implementation I am developing uses an index file of records containing
> the UID, flags as well as sizes and offsets into the actual message
> file. If records from this index file were expunged by another
> process, I'd best be careful when making updates

Indeed, any shared access environment must handle flag updates and
expunges very carefully in order to do the right thing. It is possible to
do if you follow IMAP's rules carefully; but this in turn requires a very
careful reading of the specification. If there is something that you do
not understand, it is better to ask (as you have done, thank you!!)
because it's likely to be an important point.

You may want to look into existing implementations of IMAP server that do
this. For example, the mix format in the imap-2006 version of UW imapd
uses an index file that is very much like what you discuss. If you look
at the mix format driver, you'll see that there's quite a bit to lock and
synchronize updates between multiple sessions.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Mark Crispin wrote:
|
| You may want to look into existing implementations of IMAP server that
| do this. For example, the mix format in the imap-2006 version of UW
| imapd uses an index file that is very much like what you discuss. If
| you look at the mix format driver, you'll see that there's quite a bit
| to lock and synchronize updates between multiple sessions.


I've been a bit out of touch on UW IMAP lately, but mix sounds awfully good.
We're still seeing occasional MBX corruption at ARL:UT on Solaris, running
imap-2004g. It's not too frequent (perhaps once every three months on average
over our circa 1000 users), but enough to make me really edgy.

Do you have an ETA on a non-dev-snapshot release of 2006 with mix support? I
suppose that we could again run in a hybrid mode where certain users have their
mail delivered into mix boxes and the rest stick with their current MBX?

Excitedly,

Jon

| -- Mark --
|
| http://panda.com/mrc
| Democracy is two wolves and a sheep deciding what to eat for lunch.
| Liberty is a well-armed sheep contesting the vote.

--
-------------------------------------------------------------------------------
Jonathan Abbey jonabbey@arlut.utexas.edu
Applied Research Laboratories The University of Texas at Austin
GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg

On Mon, 11 Sep 2006, Jonathan Abbey wrote:
> I've been a bit out of touch on UW IMAP lately, but mix sounds awfully good.
> We're still seeing occasional MBX corruption at ARL:UT on Solaris, running
> imap-2004g. It's not too frequent (perhaps once every three months on
> average over our circa 1000 users), but enough to make me really edgy.

What kind of corruption have you been seeing? Has it been primarily
incomplete expunges -- that is, after the end of a message is some partial
message, then some set of earlier messages?

Or have you been seeing things like flag updates in the wrong place?
Supposedly, 2004g exterminated these.

> Do you have an ETA on a non-dev-snapshot release of 2006 with mix support?

I was originally planning to release in conjunction with Alpine (which is
what Pine is morphing into). However, it now appears that there will be
an IMAP-only release this month.

Actually, I've already determined/scheduled a release date; I just don't
want to say it publicly in case it has to slip.

So, "stay tuned"!!

> I
> suppose that we could again run in a hybrid mode where certain users have
> their mail delivered into mix boxes and the rest stick with their current
> MBX?

Yes.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On 2006-09-11, Mark Crispin <MRC@CAC.Washington.EDU> wrote:
|
| What kind of corruption have you been seeing? Has it been primarily
| incomplete expunges -- that is, after the end of a message is some partial
| message, then some set of earlier messages?

Yes, that's it. I've discussed it with you a couple of times in the
past; you had theorized that it was due to the buggy semantics of
fctnl() locking in Solaris, and recommended we move to Linux or
FreeBSD.

We haven't come to the end of our depreciation cycle for our
UltraSparc IMAP server hardware yet, so we've just kept on with
Solaris, performing occasional scans of our server for corrupted mbx
mailbox files using a custom Perl script.

| I was originally planning to release in conjunction with Alpine (which is
| what Pine is morphing into). However, it now appears that there will be
| an IMAP-only release this month.
|
| Actually, I've already determined/scheduled a release date; I just don't
| want to say it publicly in case it has to slip.
|
| So, "stay tuned"!!

Fantastic, I'm looking forward to it. Mix sounds very nice.

|> I suppose that we could again run in a hybrid mode where certain users have
|> their mail delivered into mix boxes and the rest stick with their current
|> MBX?
|
| Yes.

Perfect.

I'll be keeping an eye out for a new release, and will try to get some
users running on mix as soon as possible for a controlled test.

Jon

| -- Mark --

--
-------------------------------------------------------------------------------
Jonathan Abbey jonabbey@arlut.utexas.edu
Applied Research Laboratories The University of Texas at Austin
GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg

On Tue, 12 Sep 2006, Jonathan Abbey wrote:
> Fantastic, I'm looking forward to it. Mix sounds very nice.

There is an imap-2006 development snapshot available now with mix support:
ftp://ftp.cac.washington.edu/mail/imap-2006.DEV.tar.Z

You probably should start experimenting with mix format now. The problem
with waiting for a release version is that if everybody waits for a
release version, then there isn't much external beta testing...

We have been using it in-house within the UW email development team for
several months, but it would be useful if other sites tested it as well.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Mark Crispin wrote:
| On Tue, 12 Sep 2006, Jonathan Abbey wrote:
|> Fantastic, I'm looking forward to it. Mix sounds very nice.
|
| There is an imap-2006 development snapshot available now with mix support:
| ftp://ftp.cac.washington.edu/mail/imap-2006.DEV.tar.Z

Right, I've pulled that down, and read up on mix.

| You probably should start experimenting with mix format now. The
| problem with waiting for a release version is that if everybody waits
| for a release version, then there isn't much external beta testing...

Understood.

| We have been using it in-house within the UW email development team for
| several months, but it would be useful if other sites tested it as well.

Okay, I'll see if I can't arrange that.

Right now we're running with a solitary IMAP server system, so the easiest/
most natural way to go forward would be to upgrade to the snapshot and then
configure a few accounts to use the mix format.

I presume that the rest of the code has seen relatively little disruptive
changes, such that the risk wouldn't be too great for our /var/mail mbox and
/home/user mbx users while we're doing the mix testing?

Jon

| -- Mark --
|
| http://panda.com/mrc
| Democracy is two wolves and a sheep deciding what to eat for lunch.
| Liberty is a well-armed sheep contesting the vote.

--
-------------------------------------------------------------------------------
Jonathan Abbey jonabbey@arlut.utexas.edu
Applied Research Laboratories The University of Texas at Austin
GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg

On Tue, 12 Sep 2006, Jonathan Abbey wrote:
> I presume that the rest of the code has seen relatively little disruptive
> changes, such that the risk wouldn't be too great for our /var/mail mbox and
> /home/user mbx users while we're doing the mix testing?

Unfortunately, that's not a safe presumption.

imap-2006 is a major release, and quite a bit has changed internally since
imap-2004g. One change that impacts everything (including mbox and mbx
format users) is the addition of UIDPLUS support.

Ideally, traditional UNIX and mbx format users should not see any changed
behaviors, and there may even be slightly improved imapd performance for
mbx format with large messages. However, the addition of UIDPLUS was
disruptive; but it isn't known yet how disruptive.

Unlike traditional UNIX and mbx formats, mix format was designed with
UIDPLUS handling in mind from the onset.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Mark Crispin wrote:
| On Tue, 12 Sep 2006, Jonathan Abbey wrote:
|> I presume that the rest of the code has seen relatively little disruptive
|> changes, such that the risk wouldn't be too great for our /var/mail
|> mbox and
|> /home/user mbx users while we're doing the mix testing?
|
| Unfortunately, that's not a safe presumption.
|
| imap-2006 is a major release, and quite a bit has changed internally
| since imap-2004g. One change that impacts everything (including mbox
| and mbx format users) is the addition of UIDPLUS support.
|
| Ideally, traditional UNIX and mbx format users should not see any
| changed behaviors, and there may even be slightly improved imapd
| performance for mbx format with large messages. However, the addition
| of UIDPLUS was disruptive; but it isn't known yet how disruptive.
|
| Unlike traditional UNIX and mbx formats, mix format was designed with
| UIDPLUS handling in mind from the onset.

Got you. I'll set up some users with the new code and see how it goes over the
next couple of weeks.

| -- Mark --

--
-------------------------------------------------------------------------------
Jonathan Abbey jonabbey@arlut.utexas.edu
Applied Research Laboratories The University of Texas at Austin
GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg