About referrer spam
Referrer spam requests are requests for some pages with a faked referrer string (where the user came from). Normally this is just annoying as it appears in your webstats. More info about referer spam you can read at wikipedia
But sites could be hit by many requests within short period of time (seconds) affecting server performance. The requests usually come from very different IP addresses, so blocking with iptables is not an option. For example, I’ve noticed many nasty activities including domains make-money-online.7makemoneyonline.com and buttons-for-website.com in the last few days but requests have came from hundreds IP addresses and many different countries.
But I wanted to block only these referrer domains (among many others) not whole countries and IPs.
So what to do?
The trick here is to block not IP addresses but http referrers instead. This tutorial is based on Apache web server and .htaccess file. There are two methods you can use: simpler, with .htaccess file only and another one with .htaccess file calling a file with black-listed domains.
1 method: Blocking referrer spam bots with htaccess only
1) Add into .htaccess:
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_REFERER} domain1
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} domain2
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} domain3
RewriteRule ^.* - [F]
Explanation
If there is a referer in the request and on of the words domain1, domain2 or domain3 are in the referrer, send them a 403 forbidden message. This way the server blocks the request early in the processing before any cgi-script is called. This saves CPU resources and bandwidth.
2 method: Blocking referrer spam bots with blacklist file
This method is useful when you blacklist increases too many nasty domains and it uses separate file just for keeping all of these bad sites.
1) Create file /etc/apache2/blacklist.txt and add:
domain1 - domain2 - domain3 -
The – at the end of the lines is important, but could be any character. If you use another character, modify the RewriteCond shown later.
2) Tell apache to load this file by adding the following to your /etc/apache2/apache.conf file:
Rewritemap refhashmap txt:/etc/apache2/blacklist.txt
3) Add the following for every virtual host you want to have the filter running:
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_REFERER} ^http://([^/]+)
RewriteCond ${refhashmap:%1} ^-$
RewriteRule ^.* - [F]
If you already have the RewriteEngine and RewriteBase lines, you don’t need to repeat them. If you changed the char in the blacklist file above, set the char in the RewriteCond ${refhashmap:%1} line into the ^-$ block.
This way you have one site-wide blacklist and cleaner configuration files. Bye bye referrer spammers!!!
