Our church has a 2.8Ghz Gentoo Linux (2.6.10) server w/1Gig RAM, running
Apache 2.0.52 . One of the features we offer is the ability to download
MP3 sermons off the site.

The other day I couldn't log on remotely to the server. Logging in at
the console, I discovered that the machine was in a death spiral of "Out
of memory - killing process 12842 (apache2)" errors. Since all activity
on the console was taking a painfully long time to process, I eventually
just rebooted.

Looking at the logs, I saw that a single client had tried downloading
over 140 MP3 sermons within the span of a few minutes. Undoubtedly that
was causing the "out of memory" errors.

So now I have two goals. First, I want to understand why the problem
happened. Second, I want to make sure it doesn't happen again.

1) I don't really quite understand how memory usage occurs with Apache2.
For instance, here's the current output of "ps aux" related to Apache:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 14071 0.0 0.3 7364 2896 ? Ss Mar06 0:00
/usr/sbin/apache2
apache 14078 0.0 0.3 7304 2824 ? S Mar06 0:00
/usr/sbin/apache2
apache 16564 0.0 0.3 7724 3536 ? S Mar06 0:00
/usr/sbin/apache2
apache 16761 0.0 6.7 85192 61424 ? S Mar06 0:00
/usr/sbin/apache2
apache 16766 0.0 8.3 104496 75436 ? S Mar06 0:00
/usr/sbin/apache2
apache 17146 0.0 0.4 7992 3648 ? S 01:19 0:00
/usr/sbin/apache2
apache 17234 0.0 0.3 7992 3592 ? S 01:38 0:00
/usr/sbin/apache2
apache 17235 0.0 0.4 7992 3632 ? S 01:38 0:00
/usr/sbin/apache2
apache 20959 0.0 3.8 49040 35016 ? S 07:28 0:01
/usr/sbin/apache2
apache 21241 0.0 0.3 7856 3436 ? S 08:31 0:00
/usr/sbin/apache2
apache 21969 0.0 0.3 7584 3288 ? S 10:48 0:00
/usr/sbin/apache2
apache 22254 0.0 0.3 7584 3292 ? S 11:28 0:00
/usr/sbin/apache2

Can I determine the total amount of memory used by Apache by adding up
the %MEM columns? (i.e. 0.3 + 0.3 + 0.3 + 6.7 etc. = 21.7% of total memory?

I'd like to understand what actually happens in terms of memory usage
when someone starts downloading an MP3 file. Does Linux simply spawn
another child process of Apache (additional memory requirement = 1
Apache process), or does Apache have to read the entire MP3 file into
memory as it delivers it? (additional memory requirement = 16M, the size
of the MP3 file). I'm guessing that's NOT the case, but when I
experimented by watching "top" while running a script that downloaded 5
MP3s at once, I was surprised to watch the memory usage climb... and
climb... and climb... and continue to climb until I finally aborted the
script.

2) So how can I stop the system from running out of memory? I thought
of maybe writing a script that will stop any given IP address from
downloading more than X files in a given period of time (say, no more
than 2 files every 10 minutes), but is there a better way? How do
larger file servers that could be serving hundreds or thousands of files
at any given time do it, without having a terabyte of RAM?

Thanks to anyone who can me get a handle on this!