Hi,

I would like to use IMAP as a back-end for a bulletin board CGI.
Is it scalable? Can IMAP handle thousands of posts?
I will use the "THREAD" extension also.
What about perfos?

Thanks for any advice,
Greg

On Thu, 6 Apr 2006, sursum.corda@free.fr wrote:
> I would like to use IMAP as a back-end for a bulletin board CGI.
> Is it scalable? Can IMAP handle thousands of posts?
> I will use the "THREAD" extension also.
> What about perfos?

Sites use IMAP today to do this, with tens of thousands of messages in a
single mailbox. An in-progress extension to UW IMAP scales to hundreds of
thousands of messages.

The issue is not the protocol. It's the combination of the selection of
software, hardware, and overall system architecture. You can not look at
any one of these three things without considering the other two.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Regarding the protocol, is there a way to get the last ten threads
only? and then threads 11 to 20? ... ?
I did a prototype where I retrieve all threads, then I can pick the
first 10, but just wondering if I have thousands of threads: it may
hurt perfos!?

Thanks!

On Thu, 6 Apr 2006, sursum.corda@free.fr wrote:
> Regarding the protocol, is there a way to get the last ten threads
> only? and then threads 11 to 20? ... ?
> I did a prototype where I retrieve all threads, then I can pick the
> first 10, but just wondering if I have thousands of threads: it may
> hurt perfos!?

"Last 10 threads" or "threads 11 to 20" is a nebulous concept. Threads
are a tree, and the tree can change dramatically with the addition (or
removal) of just one message.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

I agree threads are trees, but what I mean is the last 10 roots of
trees.
They can change of course, as some people will reply to messages or
create new ones; but at one point in time, I want to see the last 10
topics, the last 10 trees' roots, the last 10 threads, ...
Is there a way to query IMAP for this? Or do I have to ask for ALL
threads and filter on client side?

Maybe it's not possible to do?

Thanks,
Greg

On Thu, 6 Apr 2006, sursum.corda@free.fr wrote:
> I agree threads are trees, but what I mean is the last 10 roots of
> trees.
> They can change of course, as some people will reply to messages or
> create new ones; but at one point in time, I want to see the last 10
> topics, the last 10 trees' roots, the last 10 threads, ...

For what purpose? If the intent is to scroll a view, then you don't want
to re-thread as the range changes.

This quickly gets complicated. It's only easy when you think of the
first, obvious, server actions for one particular client; not what what
some client other than yours wants to do with the data.

> Is there a way to query IMAP for this? Or do I have to ask for ALL
> threads and filter on client side?

You can filter the messages that are threaded on the server, but you can
not filter the threads themselves.

Unless you have a very slow link, getting all the threads isn't a problem,
especially as once you have the entire tree, the client side operations to
manage the view are trivial (there is no need to communicate to the server
what, precisely, the client expects in the view).

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Let me clarify again, so I'll be sure you understand me

I'm writting Bulletin Board as a CGI, this CGI is the client, and an
IMAP server will be the server.

If I have thousands of messages, and hundreds of threads packing these
messages, I can't show all these threads at once to the user, I have to
split into different web pages.

This CGI, the client, will ask "a003 THREAD REFERENCES UTF-8 ALL" each
time in order to get messages packed as threads.
>From there, this client is able to pick only the first ten threads for
example.

But if I have thousands of messages, the IMAP answer can be quite long,
right?
That's my main scalability concern.

Also, I can refine with a date instead of ALL, but then I may have
incomplete threads, am I right?

Again, thanks!
Greg

sursum.corda@free.fr writes:

> This CGI, the client, will ask "a003 THREAD REFERENCES UTF-8 ALL" each
> time in order to get messages packed as threads.
>>From there, this client is able to pick only the first ten threads for
> example.
>
> But if I have thousands of messages, the IMAP answer can be quite long,
> right?
> That's my main scalability concern.

The only thing you possibly could do is replace ALL with 1:1000, to thread
just the first 1000 messages in the folder; then use 1001:2000 to thread the
next thousand, etc. The only problem with this is that the threads that
span the boundaries will be shown as separate threads.

> Also, I can refine with a date instead of ALL, but then I may have
> incomplete threads, am I right?

If you use message sequence numbers, instead, you will not miss any
messages, but some threads will be pruned into two smaller halfs.

One idea that might be worth thinking about is to do overlapping ranges:
1:1000 first, then 501:1500, and so on. Also, keep in mind that it is
possible -- and it will happen eventually -- that a message with a later
(higher) sequence number will be threaded as a parent of a message with an
earlier (lower) sequence number. This will happen if mailing list messages
arrive out of order, although I don't think that's likely to happen in a
bulleting board environment.

But, yes, THREAD REFERENCES ALL, alone, does not scale.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQBENZs9x9p3GYHlUOIRAoNDAJ9O34vSMM6iQ3RCguJf+n +nOygt/ACfc00w
X6NhD4daMFH0dch5auhkNj4=
=P2d9
-----END PGP SIGNATURE-----

This is very ful, thank you!

Another last thing, I don't well understand the following:

> If you use message sequence numbers, instead, you will not miss any
> messages, but some threads will be pruned into two smaller halfs.

Do you mean that if I do a THREAD REFERENCES 1:1000
I will get all threads contained in this range, including messages with
a sequence number beyong 1000 ? Meaning that no messages are missing
for each thread?

Or do you mean that I will get part of a thread until 1000; and then I
can get the rest of this thread by asking for 1001:2000?

Going to test anyway ;-)

Thanks,
Greg

Did the test: the imap server first filter on given criteria, for
example 1:1000, and then build threads from this subset.

so... I have to find an algo for this, but I think the imap server
could have done a better job doing it, by for instance, maintaining a
threads list each time a new message is coming, ...

Greg

sursum.corda@free.fr writes:

> This is very ful, thank you!
>
> Another last thing, I don't well understand the following:
>
>> If you use message sequence numbers, instead, you will not miss any
>> messages, but some threads will be pruned into two smaller halfs.
>
> Do you mean that if I do a THREAD REFERENCES 1:1000
> I will get all threads contained in this range, including messages with
> a sequence number beyong 1000 ?

No, I mean exactly the opposite.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQBENbTsx9p3GYHlUOIRAmWAAJ9TIXD4r7XP0LmLt/MAh9yrRVukiACfQvI/
A3r1YcGZJ0ZrAHSIzI49/tE=
=ctTl
-----END PGP SIGNATURE-----