View Single Post

  #4 (permalink)  
Old 28-04-2008, 12:55 PM
jmcc's Avatar
jmcc jmcc is offline
Wannabe Geek
 
Join Date: Feb 2006
Posts: 335
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
jmcc is just really nicejmcc is just really nicejmcc is just really nicejmcc is just really nice
Default

Quote:
Originally Posted by bazzjazz View Post
Not sure if this is the correct board for this...
If the pages are php then you could use something like this:
http://www.modem-help.freeserve.co.u...-block.php.txt

Quote:
User-agent: Teoma
Not really a player any more.

Quote:
User-agent: Gigabot
A waste of bandwidth.

Quote:
User-agent: Scrubby
Never heard of this one. Sounds like a maggot/scraper.

Quote:
User-agent: Robozilla
Can't remember seeing this as a legit spider user agent.

There's a pile more that should be in there. And also look into including a scraper/ bad bot trap to block scrapers.

Keep an eye on this forum:
Search Engine Spider Identification:

Block all Chinese/Korean nets if necessary. Only allow users from the markets that the site is meant to serve if you really want to control access.

Regards...jmcc
Reply With Quote