Our church has a 2.8Ghz Gentoo Linux (2.6.10) server w/1Gig RAM, running
Apache 2.0.52 . One of the features we offer is the ability to download
MP3 sermons off the site.

The other day I couldn't log on remotely to the server. Logging in at
the console, I discovered that the machine was in a death spiral of "Out
of memory - killing process 12842 (apache2)" errors. Since all activity
on the console was taking a painfully long time to process, I eventually
just rebooted.

Looking at the logs, I saw that a single client had tried downloading
over 140 MP3 sermons within the span of a few minutes. Undoubtedly that
was causing the "out of memory" errors.

So now I have two goals. First, I want to understand why the problem
happened. Second, I want to make sure it doesn't happen again.

1) I don't really quite understand how memory usage occurs with Apache2.
For instance, here's the current output of "ps aux" related to Apache:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 14071 0.0 0.3 7364 2896 ? Ss Mar06 0:00
/usr/sbin/apache2
apache 14078 0.0 0.3 7304 2824 ? S Mar06 0:00
/usr/sbin/apache2
apache 16564 0.0 0.3 7724 3536 ? S Mar06 0:00
/usr/sbin/apache2
apache 16761 0.0 6.7 85192 61424 ? S Mar06 0:00
/usr/sbin/apache2
apache 16766 0.0 8.3 104496 75436 ? S Mar06 0:00
/usr/sbin/apache2
apache 17146 0.0 0.4 7992 3648 ? S 01:19 0:00
/usr/sbin/apache2
apache 17234 0.0 0.3 7992 3592 ? S 01:38 0:00
/usr/sbin/apache2
apache 17235 0.0 0.4 7992 3632 ? S 01:38 0:00
/usr/sbin/apache2
apache 20959 0.0 3.8 49040 35016 ? S 07:28 0:01
/usr/sbin/apache2
apache 21241 0.0 0.3 7856 3436 ? S 08:31 0:00
/usr/sbin/apache2
apache 21969 0.0 0.3 7584 3288 ? S 10:48 0:00
/usr/sbin/apache2
apache 22254 0.0 0.3 7584 3292 ? S 11:28 0:00
/usr/sbin/apache2

Can I determine the total amount of memory used by Apache by adding up
the %MEM columns? (i.e. 0.3 + 0.3 + 0.3 + 6.7 etc. = 21.7% of total memory?

I'd like to understand what actually happens in terms of memory usage
when someone starts downloading an MP3 file. Does Linux simply spawn
another child process of Apache (additional memory requirement = 1
Apache process), or does Apache have to read the entire MP3 file into
memory as it delivers it? (additional memory requirement = 16M, the size
of the MP3 file). I'm guessing that's NOT the case, but when I
experimented by watching "top" while running a script that downloaded 5
MP3s at once, I was surprised to watch the memory usage climb... and
climb... and climb... and continue to climb until I finally aborted the
script.

2) So how can I stop the system from running out of memory? I thought
of maybe writing a script that will stop any given IP address from
downloading more than X files in a given period of time (say, no more
than 2 files every 10 minutes), but is there a better way? How do
larger file servers that could be serving hundreds or thousands of files
at any given time do it, without having a terabyte of RAM?

Thanks to anyone who can me get a handle on this!

George Adams <g_adams27@hotmail.SPAMBGONE.com> writes:

> Our church has a 2.8Ghz Gentoo Linux (2.6.10) server w/1Gig RAM,
> running Apache 2.0.52 . One of the features we offer is the ability
> to download MP3 sermons off the site.
>
>
> The other day I couldn't log on remotely to the server. Logging in at
> the console, I discovered that the machine was in a death spiral of
> "Out of memory - killing process 12842 (apache2)" errors. Since all
> activity on the console was taking a painfully long time to process, I
> eventually just rebooted.
>
>
> Looking at the logs, I saw that a single client had tried downloading
> over 140 MP3 sermons within the span of a few minutes. Undoubtedly
> that was causing the "out of memory" errors.
>
>
> So now I have two goals. First, I want to understand why the problem
> happened. Second, I want to make sure it doesn't happen again.

They may have used a utility that allows multiple simultaneous
connections - can be hard to stop that. I'm assuming they are just
downloading the file which should be that memory intensive for Apache.
You may just be hurt by the overhead of more Apache processes running.

You don't mention it, but make sure you have some swap space
allocated for your machine, at least a Gigabyte -- that will give you
more breathing room.
>
>
> 1) I don't really quite understand how memory usage occurs with
> Apache2. For instance, here's the current output of "ps aux" related
> to Apache:
>
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 14071 0.0 0.3 7364 2896 ? Ss Mar06 0:00
> /usr/sbin/apache2
>
> apache 14078 0.0 0.3 7304 2824 ? S Mar06 0:00

>
>
> Can I determine the total amount of memory used by Apache by adding up
> the %MEM columns? (i.e. 0.3 + 0.3 + 0.3 + 6.7 etc. = 21.7% of total
> memory?

NO, you should either try 'top' and sort my Memory - OR,
for command line use ps -elfm m Os

Print out all processes and threads, show memory used,
sorted by memory.

>
>
> I'd like to understand what actually happens in terms of memory usage
> when someone starts downloading an MP3 file. Does Linux simply spawn
> another child process of Apache (additional memory requirement = 1
> Apache process), or does Apache have to read the entire MP3 file into
> memory as it delivers it? (additional memory requirement = 16M, the
> size of the MP3 file). I'm guessing that's NOT the case, but when I
> experimented by watching "top" while running a script that downloaded
> 5 MP3s at once, I was surprised to watch the memory usage climb... and
> climb... and climb... and continue to climb until I finally aborted
> the script.

Apache itself will start another child process to service a
request if a spare child is not available, that is the Max/MinSpareServers
stuff in the config file.

You said you watched a 'script' execute. Above I was assuming
you are just giving direct links to the MP3 files? If you are using
a script, then something bizarre could be going on if the script attempt
to read the whole file into memory.
>
>
> 2) So how can I stop the system from running out of memory? I thought
> of maybe writing a script that will stop any given IP address from
> downloading more than X files in a given period of time (say, no more
> than 2 files every 10 minutes), but is there a better way? How do
> larger file servers that could be serving hundreds or thousands of
> files at any given time do it, without having a terabyte of RAM?
>
>
> Thanks to anyone who can me get a handle on this!


Add the SWAP space, it is very easy to do. Here is a little
cheat sheet that works on redhat. If you are using a script, you get
put some tracking in there to block simul. connections from the same
IP.

-------------------
1. Create swap file:

dd if=/dev/zero of=thebookswap bs=1024 count=200000

-- this will create a 200 Meg area!


2. Convert to swap format

mkswap thebookswap


3. Activate as swap, do a 'free' before and after each command

swapon thebookswap

-- you should see the new space


-- To Turn it OFF

swapoff thebookswap

5. To make it permanent add to /etc/fstab -- will be picked
up when system gives swapon -a at boot automatically:

/dev/sda9 swap swap defaults,noatime 0 0
/export/tools2/system/swap/thebookswap swap swap defaults,noatime 0 0

---------------------------
--
John
__________________________________________________ _________________
John Murtari Software Workshop Inc.
jmurtari@following domain 315.635-1968(x-211) "TheBook.Com" (TM)
http://thebook.com/

John, thanks for your reply. I neglected to mention my swap space, but
yes, in addition to the 1gig RAM I also have 1Gig swap.

I hadn't thought about it, but I guess I do actually have a script
involved in my MP3-serving process (among other reasons, this is so I
can deliver a file to the user that may not necessarily be within the
Apache DocumentRoot). So the process goes something like this:

1) User clicks on http://example.com/download.cgi?file=2005-03-10-AM.rm

2) My Perl script (after performing security checks on the filename,
naturally) logs the hit and begins delivering the file, as shown in this
snippet:

my $filesize = (stat("$sermon_dir/$pathinfo"))[7];
print "Content-Disposition: inline;filename=$pathinfo\n";
print "Content-Length: $filesize\n";
print "Content-Type: application/octet-stream\n\n";
open(SERMONFILE, "<$sermon_dir/$pathinfo");
binmode(SERMONFILE);
binmode(STDOUT);
my $chunk;
while (read(SERMONFILE,$chunk,1024)) {
print $chunk;
}
close SERMONFILE;

So here's the question: does this require the Apache child processes to
grow to the size of the entire MP3 file as it delivers it? If that's
true, I could understand how a few simultaneous downloads could cause
the server to run low on memory.

If that IS true, is there a better way for a script to deliver a file to
the customer, without using up an obscene amount of memory? (I can't
just redirect to it, since the MP3 file may not be within DocumentRoot)

Again, thanks for your !


John Murtari wrote:
> George Adams <g_adams27@hotmail.SPAMBGONE.com> writes:
>
>
>>Our church has a 2.8Ghz Gentoo Linux (2.6.10) server w/1Gig RAM,
>>running Apache 2.0.52 . One of the features we offer is the ability
>>to download MP3 sermons off the site.
>>
>>
>>The other day I couldn't log on remotely to the server. Logging in at
>>the console, I discovered that the machine was in a death spiral of
>>"Out of memory - killing process 12842 (apache2)" errors. Since all
>>activity on the console was taking a painfully long time to process, I
>>eventually just rebooted.
>>
>>
>>Looking at the logs, I saw that a single client had tried downloading
>>over 140 MP3 sermons within the span of a few minutes. Undoubtedly
>>that was causing the "out of memory" errors.
>>
>>
>>So now I have two goals. First, I want to understand why the problem
>>happened. Second, I want to make sure it doesn't happen again.
>
>
> They may have used a utility that allows multiple simultaneous
> connections - can be hard to stop that. I'm assuming they are just
> downloading the file which should be that memory intensive for Apache.
> You may just be hurt by the overhead of more Apache processes running.
>
> You don't mention it, but make sure you have some swap space
> allocated for your machine, at least a Gigabyte -- that will give you
> more breathing room.
>
>>
>>1) I don't really quite understand how memory usage occurs with
>>Apache2. For instance, here's the current output of "ps aux" related
>>to Apache:
>>
>>
>>USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>>root 14071 0.0 0.3 7364 2896 ? Ss Mar06 0:00
>>/usr/sbin/apache2
>>
>>apache 14078 0.0 0.3 7304 2824 ? S Mar06 0:00
>
>
>>
>>Can I determine the total amount of memory used by Apache by adding up
>>the %MEM columns? (i.e. 0.3 + 0.3 + 0.3 + 6.7 etc. = 21.7% of total
>>memory?
>
>
> NO, you should either try 'top' and sort my Memory - OR,
> for command line use ps -elfm m Os
>
> Print out all processes and threads, show memory used,
> sorted by memory.
>
>
>>
>>I'd like to understand what actually happens in terms of memory usage
>>when someone starts downloading an MP3 file. Does Linux simply spawn
>>another child process of Apache (additional memory requirement = 1
>>Apache process), or does Apache have to read the entire MP3 file into
>>memory as it delivers it? (additional memory requirement = 16M, the
>>size of the MP3 file). I'm guessing that's NOT the case, but when I
>>experimented by watching "top" while running a script that downloaded
>>5 MP3s at once, I was surprised to watch the memory usage climb... and
>>climb... and climb... and continue to climb until I finally aborted
>>the script.
>
>
> Apache itself will start another child process to service a
> request if a spare child is not available, that is the Max/MinSpareServers
> stuff in the config file.
>
> You said you watched a 'script' execute. Above I was assuming
> you are just giving direct links to the MP3 files? If you are using
> a script, then something bizarre could be going on if the script attempt
> to read the whole file into memory.
>
>>
>>2) So how can I stop the system from running out of memory? I thought
>>of maybe writing a script that will stop any given IP address from
>>downloading more than X files in a given period of time (say, no more
>>than 2 files every 10 minutes), but is there a better way? How do
>>larger file servers that could be serving hundreds or thousands of
>>files at any given time do it, without having a terabyte of RAM?
>>
>>
>>Thanks to anyone who can me get a handle on this!
>
>
>
> Add the SWAP space, it is very easy to do. Here is a little
> cheat sheet that works on redhat. If you are using a script, you get
> put some tracking in there to block simul. connections from the same
> IP.
>
> -------------------
> 1. Create swap file:
>
> dd if=/dev/zero of=thebookswap bs=1024 count=200000
>
> -- this will create a 200 Meg area!
>
>
> 2. Convert to swap format
>
> mkswap thebookswap
>
>
> 3. Activate as swap, do a 'free' before and after each command
>
> swapon thebookswap
>
> -- you should see the new space
>
>
> -- To Turn it OFF
>
> swapoff thebookswap
>
> 5. To make it permanent add to /etc/fstab -- will be picked
> up when system gives swapon -a at boot automatically:
>
> /dev/sda9 swap swap defaults,noatime 0 0
> /export/tools2/system/swap/thebookswap swap swap defaults,noatime 0 0
>
> ---------------------------

George Adams <g_adams27@hotmail.SPAMBGONE.com> writes:

> John, thanks for your reply. I neglected to mention my swap space,
> but yes, in addition to the 1gig RAM I also have 1Gig swap.
>
>
> I hadn't thought about it, but I guess I do actually have a script
> involved in my MP3-serving process (among other reasons, this is so I
> can deliver a file to the user that may not necessarily be within the
> Apache DocumentRoot). So the process goes something like this:
>
>
> 1) User clicks on http://example.com/download.cgi?file=2005-03-10-AM.rm
>
> 2) My Perl script (after performing security checks on the filename,
> naturally) logs the hit and begins delivering the file, as shown in
> this snippet:
>
>
> my $filesize = (stat("$sermon_dir/$pathinfo"))[7];
> print "Content-Disposition: inline;filename=$pathinfo\n";
> print "Content-Length: $filesize\n";
> print "Content-Type: application/octet-stream\n\n";
> open(SERMONFILE, "<$sermon_dir/$pathinfo");
> binmode(SERMONFILE);
> binmode(STDOUT);
> my $chunk;
> while (read(SERMONFILE,$chunk,1024)) {
> print $chunk;
> }
> close SERMONFILE;
>
> So here's the question: does this require the Apache child processes
> to grow to the size of the entire MP3 file as it delivers it? If
> that's true, I could understand how a few simultaneous downloads could
> cause the server to run low on memory.


From what you have there. It should not be a problem at all.
I would recommend you run 'top' and have it refresh at 1 sec intervals
and try a few downloads just to watch the actual memory usage of the
Apache processes. If this is something that just happens at random
time you are probably going to need a cron job running 1/minute to check
swap free and if it falls below a threshhold then have it run a ps command
to print memory usage by process: ps -elfm Os m (that is an 'oh' not a zero)

You may also want to add something like this to your apache config
file to limit memory usage on each child process

RLimitMEM 30000000

You are going to have to pick a value that 'fits' you. Monitor
the error log, you will get killed process messages there.

Hope this s.


--
John
__________________________________________________ _________________
John Murtari Software Workshop Inc.
jmurtari@following domain 315.635-1968(x-211) "TheBook.Com" (TM)
http://thebook.com/