Hi,

I'm running Apache 1 (going to 2.2 soon) on multiple load balanced web
servers. The sites current get about 4,000,000 page requests/day spread
across 65 virtual hosts. I've been asked to find a log analysis tool
that does better than the home grown one that someone here put together
a few years ago.

I'm testing awstats, but I don't think it will work. It seems to insist
on keeping too much data in its monthly data file. After loading 2 days
of logs, I have some hosts where awstats takes 5+ minutes to process 30
minutes of log entries. Also, it appears to read its entire data file
in to update it which leads to a sizable memory footprint.

Does anyone have suggestions for tools and/or techniques to handle this
volume of log files in a timely fashion?

Thanks,
Jim

Jim Hayter wrote:
> Hi,
>
> I'm running Apache 1 (going to 2.2 soon) on multiple load balanced web
> servers. The sites current get about 4,000,000 page requests/day spread
> across 65 virtual hosts. I've been asked to find a log analysis tool
> that does better than the home grown one that someone here put together
> a few years ago.
>
> I'm testing awstats, but I don't think it will work. It seems to insist
> on keeping too much data in its monthly data file. After loading 2 days
> of logs, I have some hosts where awstats takes 5+ minutes to process 30
> minutes of log entries. Also, it appears to read its entire data file
> in to update it which leads to a sizable memory footprint.
>
> Does anyone have suggestions for tools and/or techniques to handle this
> volume of log files in a timely fashion?
>
> Thanks,
> Jim

I'm not sure how it'd cope with the volume but I use Analog
(www.analog.cx). Quite simple to set up using configuration
files and plays nicely with Apache Virtual Hosts.

Andy J.