(crossposted to comp.mail.pine and comp.mail.imap since maybe
someone on c.m.i will know the answer to the questions I ask
below)

Sur 2006-03-13, Mark Crispin skribis:
> On Mon, 13 Mar 2006, Harald Schoen wrote:
>>
>> - Pine says there are a number of commonly used keywords.
>> What are these and what do they mean?
>
> I don't know if there is an RFC out yet; but there was a draft
> document floating around that listed some. All of these were
> names that started with $ and were specially recognized by
> software.
>
> Keywords that do not begin with $ are all for the human user's
> exclusive use.

I didn't know that that was what the $ means -- is this
documented somewhere? If so, I'll mention it on my pages where I
discuss IMAP keywords and change the keywords that I use, which
I list here:

<http://www.ii.com/internet/messaging/pine/sharing/#keywords>

The draft that Mark refers to used to be:

Common IMAP Keywords
<http://www.ietf.org/internet-drafts/draft-melnikov-imap-keywords-03.txt>

But it's not available at the moment -- does anyone know what's
going on with that?


>> - Where's a Keyword stored when it's set on a message? I once
>> set a keyword on a message and viewed the message using the
>> "full header" command but couldn't find any difference.
>
> It is stored in the message's metadata, not in the message
> itself.

Note that some mailbox formats, for example mbox when used with a
UW IMAP server, store the metadata in the headers. I discuss this
here:

Procmail Quick Start: Setting Keywords or Labels
<http://www.ii.com/internet/robots/procmail/qs/#keywords>


>> - Related to the above: How are keywords transmitted along
>> with a message? Is the Keyword concept a generally accepted
>> standard?
>
> Keywords are part of the IMAP standard. They are NOT
> transmitted in a message; they are strictly tags that the user
> receiving the message can attach to the message.

Note that keywords can be "transmitted" with a message if the
message is transported via the IMAP protocol, e.g., from one IMAP
mailbox to another.

Also note that delivery agents, such as Procmail and Sieve, can
be used to set keywords. I do this extensively with Sieve's
'addflag' command. Details are here:

SIEVE Email Filtering: IMAP flag Extension
<http://www.faqs.org/ftp/pub/internet-drafts/draft-ietf-sieve-imapflags-03.txt>

and an example is here:

Using a MaybeSpam Mailbox
<http://deflexion.com/2006/01/using-maybespam-mailbox>


>> - Is there any correspondence between keywords and special
>> tags used by Outlook (e.g. categories, flags, follow-ups)?
>> I.e. can I use Pine to check and process these special
>> features?
>
> I don't know. Keywords are a standard part of IMAP, but who
> knows what Outlook uses.

Whenever I need to investigate something like this, which I
recently did with the Opera 9 Preview, I use the client (e.g.,
Opera or Outlook) to access an mbox-formatted mailbox on a UW
IMAP server and set each tag/category/flag on a separate message.
Then I quit the client and ssh to the UW IMAP server and look at
the raw mbox (e.g., using 'less mboxname'. If IMAP keywords are
being used, they should be in X-Keywords headers.

I've written more about IMAP keywords in the following.

Power Pine: Setting Up Keywords (Labels)
<http://www.ii.com/internet/messaging/pine/pc/#keywords>

IMAP Service Providers section
<http://www.ii.com/internet/messaging/imap/isps/#keywords>

Hope this s,
Nancy

--
Nancy McGough ~ <http://www.ii.com> ~ <http://deflexion.com>

On Tue, 14 Mar 2006, NM Public wrote:
>> Keywords that do not begin with $ are all for the human user's exclusive
>> use.
> I didn't know that that was what the $ means -- is this documented somewhere?

It's a convention, and not enforced in the protocol. The idea was that
all keywords which would have special meaning to some software would be
start with $, and be registered someplace. Similarly, all keywords which
do not start with $ would be solely for humans.

Alexey's document was supposed to detail this. I'm distressed to see that
it's gone. I'll ask Alexey about it when I see him in a couple of weeks.

> Note that some mailbox formats, for example mbox when used with a UW IMAP
> server, store the metadata in the headers.

That doesn't matter. As with Status: and other metadata headers, it is
removed from the header that is presented to the user. It may appear in
the header in the file, but the user's question was what he sees when
viewing full headers. As far as the user is concerned, it doesn't exist
in the header.

> Note that keywords can be "transmitted" with a message if the message is
> transported via the IMAP protocol, e.g., from one IMAP mailbox to another.

That wasn't the user's question.

> Also note that delivery agents, such as Procmail and Sieve, can be used to
> set keywords. I do this extensively with Sieve's 'addflag' command.

That's only if the delivery agent knows how to set metadata, and/or
communicates with a delivery agent that knows. That is not always the
case, and in many cases is not the case.

I would definitely NOT depend upon a procmail rule that manually adds the
header line to a traditional UNIX mailbox file, because that can get
filtered out as part of a system manager reconfiguration of delivery.

Going through Sieve is (much) better, but that assumes that Sieve is
installed. That's a big assumption.

I'm glad that it works for you (it was certainly intended to!); but you
really need to emphasize the "power user" nature of this in most
installations. Only on a few installations, which have done the necessary
technical groundwork, is the capability available in a reliable
user-accessible way. On most installations, it is either unavailable or
available only via kludge.

> Whenever I need to investigate something like this, which I recently did with
> the Opera 9 Preview, I use the client (e.g., Opera or Outlook) to access an
> mbox-formatted mailbox on a UW IMAP server and set each tag/category/flag on
> a separate message. Then I quit the client and ssh to the UW IMAP server and
> look at the raw mbox (e.g., using 'less mboxname'. If IMAP keywords are being
> used, they should be in X-Keywords headers.

This is one of those kludges. It assumes that the site offers shell
access to the IMAP server (a big assumption) and that the site won't
change mailbox formats.

Don't get me wrong. I urge you to continue passing on this information;
it's good and valuable information for the community. You're doing a
fantastic service too! [And in case you don't hear it enough, "Thank
you!!!!"]

But please remember to have suitable cautionary notes. What works for
you, a power user with the technical skills to handle a curve ball tossed
your way, may not work as easily for a novice.

I don't want people to get the idea that they can copy a procmail script
that insert X-Keywords into a header and it will just work as
advertised...

The good news is that with greater use for keywords, there's greater
demand. Thus, we can expect that IMAP server implementors and site
managers will be more likely to put Sieve and other official mechanisms in
place for keywords to be set during delivery.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Mark Crispin wrote:
> On Tue, 14 Mar 2006, NM Public wrote:
>
>>Note that keywords can be "transmitted" with a message if the message is
>>transported via the IMAP protocol, e.g., from one IMAP mailbox to another.
>
> That wasn't the user's question.

Actually, I'm not sure.

Does "transmitting" mean "sending via SMTP" ?

Couldn't keywords be sent with a X-Keyword header and eventually a
content-disposition header ?

Or perhaps thanks to http://www.ietf.org/rfc/rfc3458.txt ?

Alexey once mentioned : "There is currently no standard for forwarding
keywords over SMTP, however some servers like Cyrus LMTP support an
extension for doing that" -
http://joseph.randomnetworks.com/arc.../#comment-2220

There's some demand to send mails with keywords : "But when am I going
to be able to do this with email, in a way that is recognized by the
recipient and the sender?" -
http://www.choiresicha.com/archives/000578.html and that kind of tool
might be very useful in a corporate/biz environment.

--
kael

kael writes:

> Mark Crispin wrote:
>> On Tue, 14 Mar 2006, NM Public wrote:
>>
>>>Note that keywords can be "transmitted" with a message if the message is
>>>transported via the IMAP protocol, e.g., from one IMAP mailbox to another.
>>
>> That wasn't the user's question.
>
> Actually, I'm not sure.
>
> Does "transmitting" mean "sending via SMTP" ?

No, it means "copied or uploaded via IMAP, to some mail folder".

> Couldn't keywords be sent with a X-Keyword header and eventually a
> content-disposition header ?

You may include whatever header pleases you, as part of a message that you
wish to send via SMTP. When the message gets delivered to the recipient,
if this header remains intact, and the message is opened by the UW-IMAP
server, it's likely that this header will be processed accordingly, and the
message will be marked with the given keywords.

However, if the message is opened by some other mail handling software, the
X-Keyword: header will likely to be ignored completely.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQBEF0x1x9p3GYHlUOIRAuZtAJ0SmTnIFgWrlVUj+iQgc4 tUmXsLagCfXIJo
vLDdc71Qo/ee3V+wt1V9Oc0=
=hop3
-----END PGP SIGNATURE-----

Sam wrote:
> kael writes:
>>Mark Crispin wrote:
>>>On Tue, 14 Mar 2006, NM Public wrote:
>>>
>>>>Note that keywords can be "transmitted" with a message if the message is
>>>>transported via the IMAP protocol, e.g., from one IMAP mailbox to another.
>>>
>>>That wasn't the user's question.
>>
>>Actually, I'm not sure.
>>
>>Does "transmitting" mean "sending via SMTP" ?
>
> No, it means "copied or uploaded via IMAP, to some mail folder".

The original question is slightly ambiguous.

>>Couldn't keywords be sent with a X-Keyword header and eventually a
>>content-disposition header ?
>
> You may include whatever header pleases you, as part of a message that you
> wish to send via SMTP. When the message gets delivered to the recipient,
> if this header remains intact, and the message is opened by the UW-IMAP
> server, it's likely that this header will be processed accordingly, and the
> message will be marked with the given keywords.
>
> However, if the message is opened by some other mail handling software, the
> X-Keyword: header will likely to be ignored completely.

Couldn't headers for keywords be comparable to those from mailing-lists
as defined by http://www.ietf.org/rfc/rfc2369.txt ?

Don't IMAP client deal with those headers in a specific manner thanks to
the ENVELOPE structure ?

--
kael

On Tue, 14 Mar 2006, Sam wrote:
> > Couldn't keywords be sent with a X-Keyword header and eventually a
> > content-disposition header ?
> You may include whatever header pleases you, as part of a message that you
> wish to send via SMTP. When the message gets delivered to the recipient, if
> this header remains intact, and the message is opened by the UW-IMAP server,
> it's likely that this header will be processed accordingly, and the message
> will be marked with the given keywords.

UW imapd's affiliated mail delivery tools, and several unrelated mail
delivery tools, go to some length (and in the future will do more) to
discard metadata headers in SMTP-incoming messages that are destined for
traditional UNIX format messages.

Although it is still possible to sneak a metadata header today on some
sites, it likely will not still work tommorrow. There are too many
attacks caused by allowing sender-specified metadata. Sites are
increasingly deploying mail delivery tools that block such headers.

Nor do you want to allow metadata headers to be transmitted via SMTP. If
that happened, then anybody could do all sorts of things to your mail
viewing environment just by sending you a message.

Note that this problem is specific to traditional UNIX format files, which
by their nature store metadata in the headers. Most IMAP servers,
including UW imapd, support other formats which do not store metadata in
the headers. Some IMAP servers do not support traditional UNIX format at
all.

> However, if the message is opened by some other mail handling software,
> the X-Keyword: header will likely to be ignored completely.

It's not a software issue; it's a mailbox format issue.

Thus, a more accurate statement is:

However, unless the message is delivered to a traditional UNIX format
mailbox file, the X-Keyword header will likely be ignored complete.
Other formats, which do not include metadata in headers, are completely
immune from this attack.

The bottom line is that playing with metadata headers such as X-Keyword is
like playing with black magic. It won't always work, it's likely to
backfire on you, and it may incur the wrath of the sorcerer... ;-)

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

kael writes:

> Sam wrote:
>
>>>Couldn't keywords be sent with a X-Keyword header and eventually a
>>>content-disposition header ?
>>
>> You may include whatever header pleases you, as part of a message that you
>> wish to send via SMTP. When the message gets delivered to the recipient,
>> if this header remains intact, and the message is opened by the UW-IMAP
>> server, it's likely that this header will be processed accordingly, and the
>> message will be marked with the given keywords.
>>
>> However, if the message is opened by some other mail handling software, the
>> X-Keyword: header will likely to be ignored completely.
>
> Couldn't headers for keywords be comparable to those from mailing-lists
> as defined by http://www.ietf.org/rfc/rfc2369.txt ?

Show me a meaningful RFC that defines a header for holding keywords.

> Don't IMAP client deal with those headers in a specific manner thanks to
> the ENVELOPE structure ?

The RFC2369 headers are not returned by ENVELOPE. Nor BODYSTRUCTURE.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQBEF2MKx9p3GYHlUOIRAh76AJ9Nmcr5+vHEpOeSXa/w5hEzET8cagCeJeFi
06jORUTyanlolO2tlhrvNrA=
=RCDi
-----END PGP SIGNATURE-----

On Wed, 15 Mar 2006, kael wrote:
> Couldn't headers for keywords be comparable to those from mailing-lists as
> defined by http://www.ietf.org/rfc/rfc2369.txt ?

IMAP keywords have nothing in common with message headers. It is simply
an artifact of traditional UNIX mailbox format that they are stored in the
header at all.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Sur 2006-03-14, Mark Crispin skribis:
> On Tue, 14 Mar 2006, NM Public wrote:
>>
>>> Keywords that do not begin with $ are all for the human
>>> user's exclusive use.
>>>
>> I didn't know that that was what the $ means -- is this
>> documented somewhere?
>
> It's a convention, and not enforced in the protocol. The idea
> was that all keywords which would have special meaning to some
> software would be start with $, and be registered someplace.
> Similarly, all keywords which do not start with $ would be
> solely for humans.
>
> Alexey's document was supposed to detail this. I'm distressed
> to see that it's gone. I'll ask Alexey about it when I see him
> in a couple of weeks.

OK, thanks for the details about the leading '$' -- this explains
why I felt like Alexey and I were not communicating well when I
had email with him about the "Common IMAP Keywords" draft. The
reason I felt that way was because we in fact WERE
miscommunicating. I was making suggestions for keywords that
users (i.e. humans) would use, but that's not what those
leading-$ keywords are for!

Maybe we need two types of common IMAP keywords:

Common User IMAP Keywords
Common Software IMAP Keywords


>> Note that keywords can be "transmitted" with a message if the
>> message is transported via the IMAP protocol, e.g., from one
>> IMAP mailbox to another.
>
> That wasn't the user's question.

The user's question was this: "How are keywords transmitted along
with a message?"

To me, it's still not clear what he was asking and I was simply
trying to make it clear that IMAP keywords should be preserved
when a message is "transmitted" via the IMAP protocol. As long as
the target IMAP system supports IMAP keywords, of course.


>> Also note that delivery agents, such as Procmail and Sieve,
>> can be used to set keywords. I do this extensively with
>> Sieve's 'addflag' command.
>
> That's only if the delivery agent knows how to set metadata,
> and/or communicates with a delivery agent that knows. That is
> not always the case, and in many cases is not the case.
>
> I would definitely NOT depend upon a procmail rule that
> manually adds the header line to a traditional UNIX mailbox
> file, because that can get filtered out as part of a system
> manager reconfiguration of delivery.
>
> Going through Sieve is (much) better, but that assumes that
> Sieve is installed. That's a big assumption.
>
> I'm glad that it works for you (it was certainly intended to!);
> but you really need to emphasize the "power user" nature of
> this in most installations. Only on a few installations, which
> have done the necessary technical groundwork, is the capability
> available in a reliable user-accessible way. On most
> installations, it is either unavailable or available only via
> kludge.

OK, I'll update my Procmail page to make it clear that this is a
"power user" feature and won't necessarily work in future
versions of UW IMAP.

One of the reasons that I created my IMAP Service Providers page
was to get users to demand tools like Sieve and its 'addflag'
extension. If Sieve is available everywhere, more IMAP clients
will start supporting Sieve and the ManageSieve protocol, and
user's will be able to easily control their own server-side
filtering, especially greenlisting (aka whitelisting).

[deleted]

> Don't get me wrong. I urge you to continue passing on this
> information; it's good and valuable information for the
> community. You're doing a fantastic service too! [And in case
> you don't hear it enough, "Thank you!!!!"]

Thank YOU Mark for all the work you've done on IMAP. And thanks
for the thanks too!


> But please remember to have suitable cautionary notes. What
> works for you, a power user with the technical skills to handle
> a curve ball tossed your way, may not work as easily for a
> novice.
>
> I don't want people to get the idea that they can copy a
> procmail script that insert X-Keywords into a header and it
> will just work as advertised...
>
> The good news is that with greater use for keywords, there's
> greater demand. Thus, we can expect that IMAP server
> implementors and site managers will be more likely to put Sieve
> and other official mechanisms in place for keywords to be set
> during delivery.

Yes, yes, this is good news. To push it all forward, please
everyone demand support for IMAP keywords, Sieve, and ManageSieve
from your IMAP service providers, your IMAP desktop clients, and
your IMAP web-based clients!

Nancy
posted to c.m.p and c.m.i, and Bcc'd to Alexey Melnikov

--
Nancy McGough ~ <http://www.ii.com> ~ <http://deflexion.com>

On Wed, 15 Mar 2006, NM Public wrote:
> Maybe we need two types of common IMAP keywords:
> Common User IMAP Keywords
> Common Software IMAP Keywords

I fail to understand your point. If something is a "Common User IMAP
Keyword", what does that mean as opposed to, say, user keywords of
PurplePeopleEater or Blurdybloop?

By design, user keywords convey no meaning other than to the human user
who created them. If the keyword is listed in a document, it presumably
does so for the benefit of software which interprets it. An entire space
of keywords, starting with $, was grafted from the user keyword space for
that purpose (in addition to the IMAP protocol reserved space of keywords
that start with \).

I personally would be very unhappy if some MUA were to interpret my user
keywords and to take its own unilateral action based upon what it thought
should be done with them.

Since user keyword names are arbitrary and have no meaning to software,
the only thing that software can do with such names is present them to the
user as being associated with the message, and to allow users to set
(which may imply creating a new user keyword) and unset user keywords in a
message.

> OK, I'll update my Procmail page to make it clear that this is a "power user"
> feature and won't necessarily work in future versions of UW IMAP.

Just to be sure that you understand; the "power user" feature that is
likely to vanish is that of inserting an X-Keywords header into the
message. It assumes that the target mailbox is in traditional UNIX
mailbox format, and that the means of insertion successfully evaded the
code that defends against SMTP senders including metadata headers in the
message.

The ability to set keywords in a message during delivery via some
mechanism such as Sieve's addflag is also a "power user" feature, but is
NOT likely to go away. In fact, this is the mechanism that users SHOULD
use instead of hacking internal message metadata.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Sam wrote:
> kael writes:
>>Sam wrote:
>>
>>>>Couldn't keywords be sent with a X-Keyword header and eventually a
>>>>content-disposition header ?
>>>
>>>You may include whatever header pleases you, as part of a message that you
>>>wish to send via SMTP. When the message gets delivered to the recipient,
>>>if this header remains intact, and the message is opened by the UW-IMAP
>>>server, it's likely that this header will be processed accordingly, and the
>>>message will be marked with the given keywords.
>>>
>>>However, if the message is opened by some other mail handling software, the
>>>X-Keyword: header will likely to be ignored completely.
>>
>>Couldn't headers for keywords be comparable to those from mailing-lists
>>as defined by http://www.ietf.org/rfc/rfc2369.txt ?
>
> Show me a meaningful RFC that defines a header for holding keywords.

Well, it depends on the sense of 'keywords'.

According to http://www.ietf.org/rfc/rfc2076.txt (3.7), the 'keywords'
header field is made for "search keys for data base retrieval".

The RFC 2076 points to the RFC 822: 4.7.1 and RFC 1036: 2.2.9 ; and the
RFC 2822: 3.6.5 defines the 'keywords' header field as "contain[ing] a
comma-separated list of one or more words or quoted-strings".

But this field is "intended to have _only human-readable content_ with
information about the message" ; so, a contrario, it shouldn't be used
to convey machine-readable data, should it ?

When you say "a header for holding keywords", you probably mean a
machine-readable header-field, or more precisely a header field
explicitely machine-readable, don't you ?

>>Don't IMAP client deal with those headers in a specific manner thanks to
>>the ENVELOPE structure ?
>
> The RFC2369 headers are not returned by ENVELOPE. Nor BODYSTRUCTURE.

I thought RFC2369 headers were part of one macro for the FETCH command,
hence my confusion.

Do Cone and Pine deal with those header fields like the
_mailinglistheader thunderbird extension_
http://www.cweiske.de/misc_extension...linglistheader for example ?

--
kael

Mark Crispin wrote:
> On Tue, 14 Mar 2006, Sam wrote:
>
>>However, if the message is opened by some other mail handling software,
>>the X-Keyword: header will likely to be ignored completely.
>
> It's not a software issue; it's a mailbox format issue.
>
> Thus, a more accurate statement is:
>
> However, unless the message is delivered to a traditional UNIX format
> mailbox file, the X-Keyword header will likely be ignored complete.
> Other formats, which do not include metadata in headers, are completely
> immune from this attack.

Does it mean that for Unix mailbox format files, running EXAMINE or
SELECT makes UW-IMAP /converts/ this header field into KEYWORDs
(FLAGS/PERMANENTFLAGS) ?

Is there any example of those kind of file displaying IMAP KEYWORDs in
the header field ?

--
kael

On Thu, 16 Mar 2006, kael wrote:
> According to http://www.ietf.org/rfc/rfc2076.txt (3.7), the 'keywords' header
> field is made for "search keys for data base retrieval".
> The RFC 2076 points to the RFC 822: 4.7.1 and RFC 1036: 2.2.9 ; and the RFC
> 2822: 3.6.5 defines the 'keywords' header field as "contain[ing] a
> comma-separated list of one or more words or quoted-strings".

IMAP keywords have nothing to do with the Keywords header discussed in
RFCs [2]822, 1036, and 2076.

IMAP keywords are metadata flags attached to the message. They are
accessed in IMAP via FETCH FLAGS, searched via SEARCH KEYWORDS, and
altered/added/removed via STORE FLAGS.

The keywords discussed in RFCs [2]822, 1036, and 2076 are an email header
item. They can be accessed in IMAP via FETCH HEADER.FIELDS and searched
via SEARCH HEADER. However, as these are email header items, they are
static and thus can not be altered, added, or removed other than by making
a new copy of the message with a different Keywords: header.

I think that you are confused by the fact that in the traditional UNIX
mailbox format, metadata is stored in pseudo-header lines in the message
header. These pseudo-header fields are NOT part of any Internet standard
and should be considered to be an internal detail of a particular software
implementation.

> Do Cone and Pine deal with those header fields like the _mailinglistheader
> thunderbird extension_
> http://www.cweiske.de/misc_extension...linglistheader for example ?

Pine supports RFC 2369. However, it does not support the non-standard
header fields described in that URL.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Thu, 16 Mar 2006, kael wrote:
> Does it mean that for Unix mailbox format files, running EXAMINE or SELECT
> makes UW-IMAP /converts/ this header field into KEYWORDs
> (FLAGS/PERMANENTFLAGS) ?

"Conversion" is not the correct word. When a traditional UNIX mailbox
file is processed, lines in the header which define IMAP metadata are
loaded into the IMAP server's metadata, and are NOT included in any
message header fetches. It's as if they were never there.

> Is there any example of those kind of file displaying IMAP KEYWORDs in
> the header field ?

Why do you want this?

This is strictly an internal detail of how UW imapd implements IMAP
metadata with traditional UNIX mailbox files; and that is only because the
traditional UNIX format is so limited. These internal details are not
accessible through IMAP in any way. No other software should ever use,
inspect, or depend upon these internal details.

UW imapd sites that use a different mailbox format that does not have
these artifacts. Other IMAP server implementations probably don't have
these artifacts at all.

Both traditional UNIX mailbox format, and the various kludges that had to
be done to support metadata in that format, are best forgotten except as
an unhappy memory.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Sur 2006-03-15, Mark Crispin skribis:
>
> On Wed, 15 Mar 2006, NM Public wrote:
>> Maybe we need two types of common IMAP keywords:
>> Common User IMAP Keywords
>> Common Software IMAP Keywords
>
> I fail to understand your point. If something is a "Common
> User IMAP Keyword", what does that mean as opposed to, say,
> user keywords of PurplePeopleEater or Blurdybloop?
>
> By design, user keywords convey no meaning other than to the
> human user who created them. If the keyword is listed in a
> document, it presumably does so for the benefit of software
> which interprets it. An entire space of keywords, starting
> with $, was grafted from the user keyword space for that
> purpose (in addition to the IMAP protocol reserved space of
> keywords that start with \).
>
> I personally would be very unhappy if some MUA were to
> interpret my user keywords and to take its own unilateral
> action based upon what it thought should be done with them.
>
> Since user keyword names are arbitrary and have no meaning to
> software, the only thing that software can do with such names
> is present them to the user as being associated with the
> message, and to allow users to set (which may imply creating a
> new user keyword) and unset user keywords in a message.


What I meant, but did not make clear, is that it might make sense
to distinguish between common (standard) keywords that are:

* set by the user
* set by software (e.g., Sieve)

For example, consider a keyword $Green that means "this message
is from someone in my greenlist". There are a number of ways that
this keyword could be interpreted. It could mean:

1] Sieve set the $Green keyword because the sender is in my
greenlist.

2] I set the $Green keyword as a way to communicate to my
filtering software that this sender needs to be added to my
greenlist.

3] I set the $Green keyword because I want this message to show
up in my virtual mailbox that is a result of doing a search for
"all my messages that have keyword $Green."

I don't know what is the best way to distinguish these. Maybe all
of them would get the keyword $Green, but case 1 would also get
the keyword $AutoGreen and case 2 would also get the keyword
$TrainAsGreen.

Using the not-very-good terminology I suggested above, I would
call $Green a "common user IMAP keyword" and $AutoGreen and
$TrainAsGreen "common software IMAP keywords."

I'm just thinking out loud here. I don't know what the solution
is, but I do know that I was confused when Alexey and I were
emailing about common IMAP keywords and I think the root of my
conufusion is that it was not clear to me which keywords were:

set by user for the user
set by software for the user
set by user to communicate with software (e.g., filtering
software)

I hope this makes sense,
Nancy

--
Nancy McGough ~ <http://www.ii.com> ~ <http://deflexion.com>

On Thu, 16 Mar 2006, NM Public wrote:
> What I meant, but did not make clear, is that it might make sense to
> distinguish between common (standard) keywords that are:
> * set by the user
> * set by software (e.g., Sieve)

In my opinion, that difference is small compared to the difference between
keywords which:
. have meaning only to the user
. have meaning to software

I don't think that the world is made better by going higher than n=2 on
the 2^n + 1 scale on the types of message flags. We already have:
. system flags that have meaning only to the user (\Flagged)
. system flags that have meaning to software (\Deleted, \Seen, \Answered,
\Draft).
. special system flags (\Recent)
. optional flags that have meaning to software ($ prefix keywords)
. optional flags that have meaning only to the user (all other keywords)

A new vector of "set by user" and "set by software" will cause n=3. Ugh.

By the way, \Draft sticks out like a sore thumb as something that should
have been a $ prefix keyword. I opposed it, and a few applications ever
used it, and nobody really knows what it actually means, precisely. It's
a good example of what happens when a perceived need of the moment takes
precedence over design principles.

In your example with $Green, you mentioned three cases:

> 1] Sieve set the $Green keyword because the sender is in my greenlist.
> 2] I set the $Green keyword as a way to communicate to my filtering software
> that this sender needs to be added to my greenlist.
> 3] I set the $Green keyword because I want this message to show up in my
> virtual mailbox that is a result of doing a search for "all my messages that
> have keyword $Green."

Cases (1) and (2) are an example of overloading the semantics of $Green;
you're using the same name for two entirely difference purposes. Case (2)
also presupposes more about the semantics of IMAP flags than is supported
by the IMAP protocol.

Case (3) is just wrong. That's keyword Green, not $Green. User-defined
keywords that have meaning only to the user do not use the $ prefix.

Now, it may be that you might want to set a message $Green manually; and
you can do that. But that's just a manual setting of case (1), not a
separate case (3).

> I don't know what is the best way to distinguish these. Maybe all of them
> would get the keyword $Green, but case 1 would also get the keyword
> $AutoGreen and case 2 would also get the keyword $TrainAsGreen.
> Using the not-very-good terminology I suggested above, I would call $Green a
> "common user IMAP keyword" and $AutoGreen and $TrainAsGreen "common software
> IMAP keywords."

I understand what you're saying here; but you've gotten way too
complicated. As a practical matter, such that level of complexity will go
over too many users' heads, and thus collapse under its own weight.

> the root of my conufusion is that it was not clear to me
> which keywords were:
> set by user for the user
> set by software for the user
> set by user to communicate with software (e.g., filtering software)

set by user for the user: all flags
set by software for the user: all prefixed flags (\ and $)
set by user to communicate with software: all flags except \Recent

IMHO, this isn't particularly interesting compared to what the flags do:

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Mark Crispin wrote:
> On Thu, 16 Mar 2006, kael wrote:
>> According to http://www.ietf.org/rfc/rfc2076.txt (3.7), the 'keywords' header
>> field is made for "search keys for data base retrieval".
>> The RFC 2076 points to the RFC 822: 4.7.1 and RFC 1036: 2.2.9 ; and the RFC
>> 2822: 3.6.5 defines the 'keywords' header field as "contain[ing] a
>> comma-separated list of one or more words or quoted-strings".
>
> IMAP keywords have nothing to do with the Keywords header discussed in
> RFCs [2]822, 1036, and 2076.
>
> IMAP keywords are metadata flags attached to the message. They are
> accessed in IMAP via FETCH FLAGS, searched via SEARCH KEYWORDS, and
> altered/added/removed via STORE FLAGS.
>
> The keywords discussed in RFCs [2]822, 1036, and 2076 are an email header
> item. They can be accessed in IMAP via FETCH HEADER.FIELDS and searched
> via SEARCH HEADER. However, as these are email header items, they are
> static and thus can not be altered, added, or removed other than by making
> a new copy of the message with a different Keywords: header.
>
> I think that you are confused by the fact that in the traditional UNIX
> mailbox format, metadata is stored in pseudo-header lines in the message
> header. These pseudo-header fields are NOT part of any Internet standard
> and should be considered to be an internal detail of a particular software
> implementation.

Thanks for clarifying this point (it's probably time to learn the
history of mailbox formats).

>> Do Cone and Pine deal with those header fields like the _mailinglistheader
>> thunderbird extension_
>> http://www.cweiske.de/misc_extension...linglistheader for example ?
>
> Pine supports RFC 2369. However, it does not support the non-standard
> header fields described in that URL.

So systematically, Pine FETCHes a set of /default/ header fields, which
includes the RFC 2369 ones amongst them, doesn't it ?

If so, couldn't be this list of header fields considered as conventional
(at least for developers) or as _informal_ FETCH command macros ?

My point was to include a potential X-Keywords header field in this set
of default header fields. But this doesn't solve the problem of turning
X-Keywords header into KEYWORDs anyhow.

I think some IETF WG should really try to think about it - at least to
see if it's reasonable to envisage an architecture for conveying tags
via email and to make it compatible with IMAP.

Because there's so much demand for having tags persisting on emails.

Some are requesting a Thunderbird Tagging extension :

"We need a tagging extension for Thunderbird. Urgently. It’s just a
pain (and soooo “web 1.0″!) not being able to combine emails in an IMAP
folder by other means than making yet another folder" -
http://en.magenson.de/?p=45

Others discuss the ability to use tags in Thunderbird with IMAP and with
an X-Tag header :

"Tags are applied to an e-mail message by adding one or more X-Tag
headers to the message. Search is performed using the existing search
engine in Thunderbird or through IMAP by searching for an exact match
against the value of an X-Tag header" -
http://blogometer.com/2006/02/21/thu...ing-extension/

And some others want tags some much that they use _Tag the Bird_, a
third-party (!) Thunderbird extension to tag their emails
http://www.tagthe.net/blog/stories/1437718/.

Obviously, there's some material to work on.

So, assuming people would send tags with an (IETF defined) X-Keywords
header field, how could this header field be handled by the recipient
server and client ?

For POP retrieved mails, tags would simply be displayed according to a
list of strings.

But for IMAP retrieved mails, it becomes tricky.

Since no artifacts are allowed, only SIEVE could really :

- during the mail process delivery, the SIEVE server would extract tags
from the X-Keywords header field. But would it be possible ? Eventually
thanks to
http://www.ietf.org/internet-drafts/...e-regex-00.txt ?

- or SIEVE plugged to IMAP, thanks to
http://www.ietf.org/internet-drafts/...p-sieve-01.txt,
would automatically set the corresponding IMAP KEYWORDs according to the
X-Keywords header field ?

There's a lot of potential with emails and tags, even more with the
coming VIEW extension.

Oh, and some others are discussing a royal IMAP2XMPP hack
http://mailman.jabber.org/pipermail/...ry/009583.html
; but that is another story. ;-)

--
kael

Mark Crispin wrote:
> On Thu, 16 Mar 2006, kael wrote:
>> Is there any example of those kind of file displaying IMAP KEYWORDs in
>> the header field ?
>
> Why do you want this?

Just out of curiosity - learning by example. I thought it was a
conventional format.

--
kael

On Fri, 17 Mar 2006, kael wrote:
> So systematically, Pine FETCHes a set of /default/ header fields, which
> includes the RFC 2369 ones amongst them, doesn't it ?

Pine fetches IMAP envelopes, and certain named headers that aren't in the
IMAP envelope. IMAP envelopes will probably be extended in the future.

> If so, couldn't be this list of header fields considered as conventional (at
> least for developers) or as _informal_ FETCH command macros ?

The only macros in IMAP are the "FAST", "FULL", and "ALL" ones. They were
a mistake, and are not likely to be repeated. IMAP's chattiness problem
is in its responses, not in its commands.

> My point was to include a potential X-Keywords header field in this set of
> default header fields. But this doesn't solve the problem of turning
> X-Keywords header into KEYWORDs anyhow.

Not only is the plan doomed to failure, it's a bad idea; you do not want
people to add to your IMAP mailbox metadata just by sending you email.

> I think some IETF WG should really try to think about it - at least to see if
> it's reasonable to envisage an architecture for conveying tags via email and
> to make it compatible with IMAP.

Why not use the RFC [2]822 Keywords header, and use IMAP's SEARCH HEADER
facility?

> So, assuming people would send tags with an (IETF defined) X-Keywords header
> field, how could this header field be handled by the recipient server and
> client ?

There can be no IETF-defined X-Keywords header.

X-Keywords, by virtue of being an X- header, is undefinable.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Fri, 17 Mar 2006, kael wrote:
>>> Is there any example of those kind of file displaying IMAP KEYWORDs in the
>>> header field ?
>> Why do you want this?
> Just out of curiosity - learning by example. I thought it was a conventional
> format.

It's not conventional. It's an internal detail of how one particular
server implementation handles a deficient mailbox format. Metadata is not
kept in the headers at all with better formats.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Sur 2006-03-16, Mark Crispin skribis:
>
> set by user for the user: all flags
> set by software for the user: all prefixed flags (\ and $)
> set by user to communicate with software: all flags except \Recent
>
> IMHO, this isn't particularly