White listing User Agents to combat Spam Bots and Scrapers

Posted 2008-03-21 in Spam by Johann.

IncrediBill mentioned white listing user agents to block spam bots and other types of abusers. I then tried out white listing as opposed to black listing as I did before.

Here is a short explanation of the difference between black listing and white listing.

Black listing

Black listing means rejecting all requests that fit into a certain pattern.

Black listing looks like this in my web server configuration:

$HTTP["useragent"] =~ "(bad_bot|comment spammer|spambot 1)" {
 url.access-deny = ( "" )
}

This means that all user agent strings containing bad_bot, comment spammer or spambot 1 are served a 403 Forbidden error message.

White listing

White listing means rejecting all request that do not fit into a certain pattern.

White listing looks as follows in my configuration:

$HTTP["useragent"] !~ "^(Mozilla|Opera)" {
 url.access-deny ( "" )
}

This means that only user agent strings starting with Mozilla or Opera are allowed, everything else is served a 403 Forbidden error message.

The downside of white listing is that the number of allowed user agents can be very large. As an example, user agents of Motorola cell phones start with Motorola-, but some also start with MOT-, MOTOROKR or even motorazr.

Right now, I have more than 100 rules in my white list regular expression.

Which one is right for me?

I recommend using black lists if you cannot spend much time reading log files and changing your web server’s configuration. Black listing, combined with IP blocking of known abusers, can still be effective in limiting bandwidth theft.

Black listing, however, will not prepare you against future bad bots and reincarnations of Russian email harvester outfits. This is where white listing is better.

4 comments