how to block a particular User Agent via .htaccess ?

Discussion in 'Hosting' started by paul, Mar 30, 2009.

  1. paul

    paul Ninja

    I am having problems with a spammer on one of my sites and I want to block the user agent in my .htaccess file
    I've tried a few things, but I don't think I am escaping things properly. Any tips ?
  2. louie

    louie New Member

    RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Zeus
    RewriteRule ^.* - [F,L]

    Block IP range

    #ban ip
    order allow,deny
    deny from 41.204.224. #part IP address
    deny from #full ip address
    allow from all
  3. paul

    paul Ninja

    Thanks louie, but I am wondering how I escape

    what worked was
    *silly forum won't allow the escape characters. I escaped everything ...
  4. louie

    louie New Member

    That is a very common UA and I wouldn't recommended it

    something like this

    RewriteCond %{HTTP_USER_AGENT} ^Mozilla\/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
    RewriteRule ^.* - [F,L]

    you might have to escape "\" some of the characters.
  5. paul

    paul Ninja

    I thought the UA was common alright, but right now the software doesn't have a really good way of spam protection. So I don't have much options. I spent 2 hours cleaning up the mess today, and if it means I loose 10% of the visitors and get no spam, then I will be a happy camper.
  6. ghost

    ghost New Member

    blocking User Agent

    Hi Paul
    I have had this UA on one of my sites sense before Christmas , from what I have read on the net its the AVG link checker , I get over 100 hits a day . needless to say I have blocked it .
    How to beat AVG's fake traffic spew ? The Register
  7. jmcc

    jmcc Active Member

    The UA in question could be part of a botnet rather than the AVG link checker. The common factor is that there is no referrer and you should block on the UA and the absence of a referrer. The majority of the hits will be from .BR boxes.
  8. ghost

    ghost New Member

    Its back

    I have this one back on one site for a few days ,
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

    I am sure I blocked it before with the following code but its not working for me now.

    RewriteCond %{HTTP_USER_AGENT} ^"Mozilla\/4\.0\ (compatible;\ MSIE\ 6\.0;\ Windows\ NT\ 5\.1;\ SV1)" [OR]

    I think its an escape problem, could some one take a look for me

    Thank you
  9. jmcc

    jmcc Active Member

    There shouldn't be any quotation marks " in the string. The '(' and ')' may also need to be escaped. It may be a very common user agent and it might be worth doing some kind of analysis on the IPs that are causing the problem rather than doing a simple user agent ban.

    This is one of the best references on bots/crawlers/maggots:

  10. ghost

    ghost New Member

    I have had this one on and off for about a year an its all Bad .
    this is a typical entry. - - [19/Feb/2010:13:02:34 +0000] "GET /sandtrap/index.php HTTP/1.1" 200 20 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

    The IP's are mostly East European, Asian or proxy and it ignores robots txt.

    Yesterday I created a link to a non existent page and now when it gets a 404 it immediately tries the same link with a new IP. This Bot Cloacks it self as IE-6 WIN-XP SP1 with no referrer , its been around for a while and its a known bad guy .
    One site with between 300 and 400 visitors a week has 73 hits from this today.
    I had tried leaving out the quotation marks and escaping the \( already.
    I am missing something simple here and just cant see it.

  11. jmcc

    jmcc Active Member

    One solution is to build a bot trap that blocks scrapers at the IP level.

    KLOTH.NET - Trap bad bots in a bot trap

    It is an old link but it should provide some ideas.

  12. ghost

    ghost New Member

    Windows NT 5.1; SV1)

    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

    Found a solution at last this one has been breaking my heart for ages. The 1813 was an older version of the same thing.

    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} "^(?:User-Agent: )?Mozilla/4.0 \(compatible; MSIE 6.0; Windows NT 5.1;(?: SV1|1813)\)$"
    RewriteRule ^.* - [F,L]

Share This Page