how to block a particular User Agent via .htaccess ?

Status
Not open for further replies.

paul

Ninja
I am having problems with a spammer on one of my sites and I want to block the user agent in my .htaccess file
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
I've tried a few things, but I don't think I am escaping things properly. Any tips ?
 

louie

New Member
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule ^.* - [F,L]

Block IP range

#ban ip
order allow,deny
deny from 41.204.224. #part IP address
deny from 193.120.116.180 #full ip address
allow from all
 

louie

New Member
That is a very common UA and I wouldn't recommended it

something like this

RewriteCond %{HTTP_USER_AGENT} ^Mozilla\/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
RewriteRule ^.* - [F,L]

you might have to escape "\" some of the characters.
 

paul

Ninja
I thought the UA was common alright, but right now the software doesn't have a really good way of spam protection. So I don't have much options. I spent 2 hours cleaning up the mess today, and if it means I loose 10% of the visitors and get no spam, then I will be a happy camper.
 

jmcc

Active Member
The UA in question could be part of a botnet rather than the AVG link checker. The common factor is that there is no referrer and you should block on the UA and the absence of a referrer. The majority of the hits will be from .BR boxes.
Regards...jmcc
 

ghost

New Member
Its back

I have this one back on one site for a few days ,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

I am sure I blocked it before with the following code but its not working for me now.

RewriteCond %{HTTP_USER_AGENT} ^"Mozilla\/4\.0\ (compatible;\ MSIE\ 6\.0;\ Windows\ NT\ 5\.1;\ SV1)" [OR]

I think its an escape problem, could some one take a look for me

Thank you
 

jmcc

Active Member
RewriteCond %{HTTP_USER_AGENT} ^"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" [OR]
There shouldn't be any quotation marks " in the string. The '(' and ')' may also need to be escaped. It may be a very common user agent and it might be worth doing some kind of analysis on the IPs that are causing the problem rather than doing a simple user agent ban.

This is one of the best references on bots/crawlers/maggots:
http://www.webmasterworld.com/search_engine_spiders/

Regards...jmcc
 

ghost

New Member
I have had this one on and off for about a year an its all Bad .
this is a typical entry.
68.68.104.34 - - [19/Feb/2010:13:02:34 +0000] "GET /sandtrap/index.php HTTP/1.1" 200 20 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

The IP's are mostly East European, Asian or proxy and it ignores robots txt.

Yesterday I created a link to a non existent page and now when it gets a 404 it immediately tries the same link with a new IP. This Bot Cloacks it self as IE-6 WIN-XP SP1 with no referrer , its been around for a while and its a known bad guy .
One site with between 300 and 400 visitors a week has 73 hits from this today.
I had tried leaving out the quotation marks and escaping the \( already.
I am missing something simple here and just cant see it.

Thanks
 

jmcc

Active Member
I have had this one on and off for about a year an its all Bad .
this is a typical entry.
68.68.104.34 - - [19/Feb/2010:13:02:34 +0000] "GET /sandtrap/index.php HTTP/1.1" 200 20 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

The IP's are mostly East European, Asian or proxy and it ignores robots txt.
One solution is to build a bot trap that blocks scrapers at the IP level.

KLOTH.NET - Trap bad bots in a bot trap

It is an old link but it should provide some ideas.

Regards...jmcc
 

ghost

New Member
Windows NT 5.1; SV1)

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

Found a solution at last this one has been breaking my heart for ages. The 1813 was an older version of the same thing.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "^(?:User-Agent: )?Mozilla/4.0 \(compatible; MSIE 6.0; Windows NT 5.1;(?: SV1|1813)\)$"
RewriteRule ^.* - [F,L]
 
Status
Not open for further replies.
Top