View Single Post

  #9 (permalink)  
Old 30-04-2008, 12:49 PM
ghost's Avatar
ghost ghost is offline
Wannabe Geek
 
Join Date: Dec 2007
Location: Ennis
Posts: 176
Nominated 0 Times in 0 Posts
TOTW/F/M Award(s): 0
ghost will become famous soon enough
Default

Ran a quick dns check on the two googlebots from your logs and both seem genuine
66.249.65.101
66.249.70.234
Check here DNS Tools
One thing I noticed is googlebot and slurp are following what are possible empty links like this one
Basket is empty page
Road Records Shopping Section
I had a problem a few months back with excessive crawling by googlebot following empty links on an events calendar
googlebot was smart enough to pull out after a few identical pages but slurp needed a bit more persuasion.
fake googlebots gobbling my band width
Keep a check on your logs and see exactly what google is spidering and try to block it from spidering empty or irrelevant pages
also you should create an xml site map and include a link in your robots have a look at the two below.

http://www.coslia.com/robots.txt
http://www.coslia.com/sitemap.xml
Reply With Quote