Posts Tagged ‘sysadmin’

Excellent Apache Log Analyser & Monitor

July 23rd, 2009

python-logo-master-v3-TM.pngI’ve been looking up hill and down dale for a good (Python-based – fanboy!! Woo!!) log analyser and monitor for Apache.

It looks like I’ve found one – and awesome it is too!

wtop is by far the best I’ve come across of the options available and I’m not just saying that because its Python based. It nicely formats based on a variety of rules you give to it.

I plan on using it to monitor when certain IP’s access my server then hooking it upto an SMS API to text me when it happens. You can use it for monitoring of dodgy behaviour such as accesses to Paypal folders and such.

Some important points though:

  1. Make sure your log format is configured properly. You need to add the %D option which the default “combined” format doesn’t have.
  2. After changing the formats and making sure the wtop config matches your new Apache log format you may need to delete or archive your old ‘badly’ formatted logs. As wtop won’t cope well with the new/old format mix.
  3. Install it – don’t think it’ll work straight off the bat.
  4. Read the instructions.
  5. To make sure you’ve got it working run a very basic command with it that does no filtering.
    logrep -o 'class,max(bytes),avg(msec)' access.log
  6. More info on using logrep for monitoring your Apache logs is here.

Some More Examples…

  • Show all 404 page errors in log with a count:
    logrep -m grep -o 'status,count(*),url' -f 'status=404' access.log
  • Show visits of a certain IP to the homepage of the site:
    logrep -f 'ip~192.29.29,class=home' -o hour,minute,ip,url access.log

Python Virtual Host Creator

December 10th, 2008

I just threw together a quick Python script which generates a virtual host file, enables it, creates the web directory and restarts the webserver.

It’s basic – but I found it useful.  Maybe someone else will too.

#!/usr/bin/python
import os, sys, getopt

def main(argv):

  try:
    opts, args = getopt.getopt(argv, "hd:a:", ["help", "domain="])
  except getopt.GetoptError:
    usage()
    sys.exit(2)

  if len(opts) < 1:
    usage()
    sys.exit(2)

  for o, a in opts:
    if o in ("-h", "--help"):
      usage()
      sys.exit()
    if o in ("-d", "--domain"):
      domain = a
    else:
      assert False, "unhandled option"

  directory = domain.split('.')[0]

  if not os.path.isdir("/var/www/%s" % directory):
    os.mkdir("/var/www/%s" % directory)

  if not os.path.isdir("/var/www/%s/htdocs" % directory):
    os.mkdir("/var/www/%s/htdocs" % directory)

  vhost_template = """
  
    ServerName www.__DOMAIN__
    ServerAlias __DOMAIN__
    DocumentRoot /var/www/__DIR__/htdocs
    CustomLog /var/log/apache2/__DOMAIN__-access.log combined
    ErrorLog /var/log/apache2/__DOMAIN__-error.log

    ErrorDocument 404 /404.html
    ErrorDocument 401 /401.html
    ErrorDocument 500 /500.html

    RewriteEngine On
    RewriteCond %{HTTP_HOST} ^__DOMAIN__
    RewriteRule (.*)$ http://www.__DOMAIN__$1 [R=301,L]

    
      Options +Indexes FollowSymlinks IncludesNoExec +Includes -MultiViews
      AllowOverride All
      Order Allow,Deny
      Allow from all
    
  
  """

  vhost = vhost_template.replace('__DIR__', directory).replace('__DOMAIN__', domain)
  open('/etc/apache2/sites-available/%s.conf' % directory, 'w').write(vhost)
  os.system('a2ensite %s.conf' % directory)
  os.system('/etc/init.d/apache2 restart')

def usage():
  print 'usage: newv.py [-d domain.com]'

if __name__=="__main__":
  main(sys.argv[1:])

Download It

Managing Cron Jobs – Without Steroids

November 7th, 2008

There’s a funky RubyGem available over on Rubyforge for managing your systems cron jobs.  When I first started to read about it I got confused because it says you add a cron job for the jobmanager which then runs your cron jobs.

Madness…not quite.

This jobmanager thingy actually provides a great set of tools that sit in between your cron and jobs that enables you to log events, email people on completion, manage timeouts and rotate logs.  So its kind of job-management-on-steroids-without-the-steroids. :)