View Single Post

  #7 (permalink)  
Old 30-04-2008, 11:12 AM
bazzjazz bazzjazz is offline
Frontpage User
 
Join Date: Oct 2007
Posts: 20
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
bazzjazz will become famous soon enough
Default

You'll have to excuse my ignorance but I can only see my logs on a webpage!

Here is a link to a saved page with 50 most recent entries:
H-SPHERE roadrecs (basic unix)

I notice in one entry, that a googlebot is crawling the /stock directory but I have disallowed that in robots.txt with the following lines:

User-agent: *
Disallow: /stock
Disallow: /cgi-bin
Disallow: /cp
Disallow: /images
Disallow: /albums
Disallow: /tmp
Disallow: /admin

and more.....

The file name is 'robots.txt' and it is in the root folder but maybe I am doing something wrong with it as I am sure Google obeys robots.txt?

I haven't had a chance to investigate John's suggestions above yet but hope to get around to it asap.

Thanks for the help.

barry
Reply With Quote