View Single Post

  #3 (permalink)  
Old 28-04-2008, 12:40 PM
bazzjazz bazzjazz is offline
Frontpage User
 
Join Date: Oct 2007
Posts: 20
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
bazzjazz will become famous soon enough
Default

I have already disallowed all directories! Maybe the robots.txt file is not in the correct format. Here it is:

# Allows only major search engines and known friendly spiders
# Major Search Engines and Known Friendly Spiders (allowed)

User-agent: Googlebot
Disallow:
Crawl-delay: 10
Request-rate: 1/5
Visit-time: 0800-1000

User-agent: MSNBot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: Gigabot
Disallow:

User-agent: Scrubby
Disallow:

User-agent: Robozilla
Disallow:

# Everyone Else (NOT allowed)

User-agent: *
Disallow: /

#disallow any legit search engines from crawling the following directories

User-agent: *
Disallow: /stock
Disallow: /cgi-bin
Disallow: /cp
Disallow: /images
Disallow: /albums
Disallow: /tmp
Disallow: /admin
Disallow: /auth2
Disallow: /auth1
Disallow: /webalizer
Disallow: /graphics
Disallow: /covers
Disallow: /dict
Disallow: /dump
Disallow: /gallery
Disallow: /graphics
Disallow: /guest
Disallow: /includes
Disallow: /mailing_list
Disallow: /mp3
Disallow: /newcovers
Disallow: /order_form
Disallow: /ordering
Disallow: /photography
Disallow: /pictures
Disallow: /registration
Disallow: /scene
Disallow: /competition
Disallow: /specials
Reply With Quote