John, thanks for your reply. I neglected to mention my swap space, but
yes, in addition to the 1gig RAM I also have 1Gig swap.

I hadn't thought about it, but I guess I do actually have a script
involved in my MP3-serving process (among other reasons, this is so I
can deliver a file to the user that may not necessarily be within the
Apache DocumentRoot). So the process goes something like this:

1) User clicks on http://example.com/download.cgi?file=2005-03-10-AM.rm

2) My Perl script (after performing security checks on the filename,
naturally) logs the hit and begins delivering the file, as shown in this
snippet:

my $filesize = (stat("$sermon_dir/$pathinfo"))[7];
print "Content-Disposition: inline;filename=$pathinfo\n";
print "Content-Length: $filesize\n";
print "Content-Type: application/octet-stream\n\n";
open(SERMONFILE, "<$sermon_dir/$pathinfo");
binmode(SERMONFILE);
binmode(STDOUT);
my $chunk;
while (read(SERMONFILE,$chunk,1024)) {
print $chunk;
}
close SERMONFILE;

So here's the question: does this require the Apache child processes to
grow to the size of the entire MP3 file as it delivers it? If that's
true, I could understand how a few simultaneous downloads could cause
the server to run low on memory.

If that IS true, is there a better way for a script to deliver a file to
the customer, without using up an obscene amount of memory? (I can't
just redirect to it, since the MP3 file may not be within DocumentRoot)

Again, thanks for your !


John Murtari wrote:
> George Adams <g_adams27@hotmail.SPAMBGONE.com> writes:
>
>
>>Our church has a 2.8Ghz Gentoo Linux (2.6.10) server w/1Gig RAM,
>>running Apache 2.0.52 . One of the features we offer is the ability
>>to download MP3 sermons off the site.
>>
>>
>>The other day I couldn't log on remotely to the server. Logging in at
>>the console, I discovered that the machine was in a death spiral of
>>"Out of memory - killing process 12842 (apache2)" errors. Since all
>>activity on the console was taking a painfully long time to process, I
>>eventually just rebooted.
>>
>>
>>Looking at the logs, I saw that a single client had tried downloading
>>over 140 MP3 sermons within the span of a few minutes. Undoubtedly
>>that was causing the "out of memory" errors.
>>
>>
>>So now I have two goals. First, I want to understand why the problem
>>happened. Second, I want to make sure it doesn't happen again.
>
>
> They may have used a utility that allows multiple simultaneous
> connections - can be hard to stop that. I'm assuming they are just
> downloading the file which should be that memory intensive for Apache.
> You may just be hurt by the overhead of more Apache processes running.
>
> You don't mention it, but make sure you have some swap space
> allocated for your machine, at least a Gigabyte -- that will give you
> more breathing room.
>
>>
>>1) I don't really quite understand how memory usage occurs with
>>Apache2. For instance, here's the current output of "ps aux" related
>>to Apache:
>>
>>
>>USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>>root 14071 0.0 0.3 7364 2896 ? Ss Mar06 0:00
>>/usr/sbin/apache2
>>
>>apache 14078 0.0 0.3 7304 2824 ? S Mar06 0:00
>
>
>>
>>Can I determine the total amount of memory used by Apache by adding up
>>the %MEM columns? (i.e. 0.3 + 0.3 + 0.3 + 6.7 etc. = 21.7% of total
>>memory?
>
>
> NO, you should either try 'top' and sort my Memory - OR,
> for command line use ps -elfm m Os
>
> Print out all processes and threads, show memory used,
> sorted by memory.
>
>
>>
>>I'd like to understand what actually happens in terms of memory usage
>>when someone starts downloading an MP3 file. Does Linux simply spawn
>>another child process of Apache (additional memory requirement = 1
>>Apache process), or does Apache have to read the entire MP3 file into
>>memory as it delivers it? (additional memory requirement = 16M, the
>>size of the MP3 file). I'm guessing that's NOT the case, but when I
>>experimented by watching "top" while running a script that downloaded
>>5 MP3s at once, I was surprised to watch the memory usage climb... and
>>climb... and climb... and continue to climb until I finally aborted
>>the script.
>
>
> Apache itself will start another child process to service a
> request if a spare child is not available, that is the Max/MinSpareServers
> stuff in the config file.
>
> You said you watched a 'script' execute. Above I was assuming
> you are just giving direct links to the MP3 files? If you are using
> a script, then something bizarre could be going on if the script attempt
> to read the whole file into memory.
>
>>
>>2) So how can I stop the system from running out of memory? I thought
>>of maybe writing a script that will stop any given IP address from
>>downloading more than X files in a given period of time (say, no more
>>than 2 files every 10 minutes), but is there a better way? How do
>>larger file servers that could be serving hundreds or thousands of
>>files at any given time do it, without having a terabyte of RAM?
>>
>>
>>Thanks to anyone who can me get a handle on this!
>
>
>
> Add the SWAP space, it is very easy to do. Here is a little
> cheat sheet that works on redhat. If you are using a script, you get
> put some tracking in there to block simul. connections from the same
> IP.
>
> -------------------
> 1. Create swap file:
>
> dd if=/dev/zero of=thebookswap bs=1024 count=200000
>
> -- this will create a 200 Meg area!
>
>
> 2. Convert to swap format
>
> mkswap thebookswap
>
>
> 3. Activate as swap, do a 'free' before and after each command
>
> swapon thebookswap
>
> -- you should see the new space
>
>
> -- To Turn it OFF
>
> swapoff thebookswap
>
> 5. To make it permanent add to /etc/fstab -- will be picked
> up when system gives swapon -a at boot automatically:
>
> /dev/sda9 swap swap defaults,noatime 0 0
> /export/tools2/system/swap/thebookswap swap swap defaults,noatime 0 0
>
> ---------------------------