Bot Troubles

Do you run a webserver? Ever seen lines like the following in your logs?

95.31.11.173 - - [06/Dec/2009:07:29:14 -0700] localhost:80 "POST http://yuamin.blog95.fc2.com/?no=2127&ul=96fa963c0da02340 HTTP/1.1" 200 6699
7
88.80.10.1 - - [06/Dec/2009:07:29:26 -0700] localhost:80 "CONNECT auctions.godaddy.com:443 HTTP/1.1" 301 -
7
74.63.225.45 - - [06/Dec/2009:07:29:40 -0700] localhost:80 "GET http://aanserver88.com/js/banner/banner.js HTTP/1.0" 200 11922
74.63.225.45 - - [06/Dec/2009:07:29:41 -0700] localhost:80 "GET http://c5.zedo.com/jsc/c5/fo.js HTTP/1.0" 200 3599
95.31.11.173 - - [06/Dec/2009:07:29:41 -0700] localhost:80 "GET http://seo.fc2.com/spam/ HTTP/1.1" 200 55738
74.63.225.45 - - [06/Dec/2009:07:29:42 -0700] localhost:80 "GET http://c7.zedo.com/bar/v15-202/c5/jsc/fm.js?c=2404/1729&f=&n=735&r=5&d=0&q=&s=80&z=0.3461414148545188 HTTP/1.0" 200 3505
7
...

I had been having this problem for a long time and couldn't think of a solution. These bot networks were almost constantly hammering my server with requests (I even got one log file that was over 12MB; rotated weekly). I hadn't a clue how to stop them. After lots experimentation with apache and research into its capabilities, I found a fairly simple solution. I found information about using the ext_filter module and added the following into my 'httpd.conf' file:

ExtFilterDefine bots mode=input cmd="/scripts/bot_filter.py"
SetInputFilter bots

This filters all input through a python script I made. Using the apache-set environment variables I have enough information to determine whether to block an ip. My script determines if the request is suspicious and, if so, entirely bans the ip address with my firewall of choice, iptables. Heres the script:

#!/usr/bin/env python

import os
from subprocess import Popen

ip = ""
meth = ""
req = ""

try:
    ip = str(os.environ["REMOTE_ADDR"])
    meth = str(os.environ["REQUEST_METHOD"])
    req = str(os.environ["REQUEST_URI"])
except:
    pass

#if req.find("http://") == 0
if req.find("/") != 0 or not (meth == "GET" or meth == "POST" or meth == "HEAD" or meth == "OPTIONS" or meth == "TRACE"):
    try:
        list = os.popen("sudo /sbin/iptables -n --list INPUT").read()
       
        if list.find(ip) < 0:
            Popen(['sudo', '/sbin/iptables', '-I', 'INPUT', '-p', 'all', '-s', ip, '-j', 'DROP'])
    except:
        f = open('/var/log/bot.log', 'a')
        f.write("failed\n")
        f.close()
        os.exit(0)
        pass

    f = open('/var/log/bot.log', 'a')
    f.write(ip + " success " + meth + " " + req + "\n")
    f.close()

# Allows POST data to be transparently passed on
import sys
sys.stdout.write(sys.stdin.read())

This prepends the ip address of the offending request onto the beginning of the iptables INPUT filter chain.

Now the first malicious request earns a well-deserved ban. I no longer have these malformed requests from bot networks hammering my server, but only legitimate request. Its working quite well so far.