About Logmonster

A web log utility for merging, sorting, and processing web server log files.

Logmonster is a tool to collect log files from one or many web servers, split them based on the virtual host they were served for, sort the logs into chronological order, and finally pipe the sorted logs to the log file analyzer of choice (webalizer, http-analyze, AWstats, etc).

Synopsis

logmonster.pl -i <interval> [-v] [-r] [-n] [-b N]

  Interval is one of:

      hour    (last hour)
      day     (yesterday)
      month   (last month)
   
  Optional:

     -v     verbose     - lots of status messages 
     -n     dry run     - do everything except feed the logs into the processor
     -r     report      - last periods hit counts
     -b N   back N days - use with -i day to process logs older than one day

Usage

To see what it will do without actually doing anything

  /usr/local/sbin/logmonster -i day -v -n

From cron:

  5 1 * * * /usr/local/sbin/logmonster -i day

From cron with a report of activity:

  5 1 * * * /usr/local/sbin/logmonster -i day -v

Install

FAQ

Changes

Sample Report

Features

Log Retrieval from one or many hosts
Apache and Lighttpd support (needs Apache ELF log format + vhost)
Discards referral spam!
Dynamic Configuration
- Generate vhost specific config files for awstats
- Automatic vhost detection - If using Apache, Logmonster will read your Apache config files to learn about your virtual hosts and their file system location. If your vhosts are dynamically configured (such as with lighttpd or Apache mass vhost options), you can specify a pattern and logmonster will dynamically detect your vhosts and sort the logs appropriately.
- Create missing stats directories if desired (see statsdir_policy in logmonster.conf).
- Alias domains - Understands and correctly deals with server aliases. It can learn them from Apache's ServerAlias directive or the file system (as with lighttpd vhost_simple).

Flexible
- Output to webalizer, http-analyze, and AWstats.
- Update intervals (monthly, daily, hourly)
- Settings configurable for each virtualhost - Outputs stats into each virtual domains stats dir, but only if that directory exists. This is an easy way to enable or disable stats for a virtual host. If "stats" exists, it will be updated. Otherwise it will not.
- Processes back logs (see the -b option)

Efficient
- compress remote log files before fetching them
- uses Compress::Zlib locally to read directly from .gz files (minimizes disk i/o)
- skips processing logs for vhosts with no $statsdir.
- skips sorting if you only have logs from one host.

Reporting - on disk activity report and email friendly report.
Reliable - ample error checking and useful error messages.

TODO

Add config file setting for the location of awstats.pl
Add support for analog.
Delete log files older than X days/month - super low priority, it's easy and low maintenance to manually delete a few months log files.
Do something with error logs (other than just compress)
Add support for individual webalizer.conf file for each domain (this will likely not happen until someone submits a diff or pays me to do it as I no longer use webalizer.

License and Copyright

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Neither the name of the The Network People, Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DIS CLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Logmonster

Contents