Thread: fake googlebots gobbling my band width

Has any one got a solution for these creatures.
I have a googlebot in one site nearly every hour , yesterday it cost me 30.33 MB
I am sure its not the real googlebot as my google Webmaster tools show the last visit on the 7th jan.
How do others deal with this problem or do you just ignore it.
In addition to this the bot activity I am seeing now started just before Christmas ,
I all so use a gmail account on the site to keep spam pressure off my server , my Gmail spam has more then doubled
and the subjects have changed I always get porn invites + 1001 ways to enlarge my manhood now I have a whole new range of ads.
any thoughts.

Originally Posted by ghost

and the subjects have changed I always get porn invites + 1001 ways to enlarge my manhood now I have a whole new range of ads.
any thoughts.

In some cases it can work ... but you want to be careful and make sure its from a reputable seller


but seriously ... why do you think its googlebot ?

is 30mbs a day of bandwidth an issue ? ... can you implement an no follow rule ?

<meta name="robots" content="noindex,nofollow" />

or alternatively the robot.txt to exclude that specific bot ?

User-agent: BadBot
Disallow: /

where BadBot is the name of the bot ?

before you block it make sure is not Google.

Just because it shows in the Webmaster Central the date of last visit, it doesn't mean Google doesn't spider your website everyday.

30 megs a day is neglible.

In any case you could check the logs for the IP ranges and see if they belong to Google or not...

Originally Posted by Forbairt

In some cases it can work ... but you want to be careful and make sure its from a reputable seller

In situations where most men need to shake I have to kick mine

No problem with the band width unless its been wasted with no benefits to me I am just presuming its a fake google bot .this site has about 20 odd visitors a day with very few inward links the only changing content is a news feed with a cach time of 8 hours.
Is it usual for google to visit sites every hour or so if it is I am more them happy to accommodate it.
I have already tried to capitalize on it by adding links to other sites on the main pages.

Originally Posted by ghost

In situations where most men need to shake I have to kick mine

Ah well maybe its for you then .. sometimes things just need a good kick start

No problem with the band width unless its been wasted with no benefits to me I am just presuming its a fake google bot .

30*30days = less than a gb ... most plans these days are giving 30 ish gbs a month (warning may not be true) ... (most plans I've seen in the budgetish range at any rate). Unless it really starts to become an issue I wouldn't worry about it

this site has about 20 odd visitors a day with very few inward links the only changing content is a news feed with a cach time of 8 hours.

Is it possible google is seeing the news feed on every page ? and updating thinking your content is changing daily ? (and may increase the rate it indexes your site)

You could try changing the rate in google webmaster tools to something more reasonable.

Originally Posted by ghost

I am sure its not the real googlebot as my google Webmaster tools show the last visit on the 7th jan.

The date they show in GWT is something akin to the fairy tales of Ireland.

There is actually a way to identify real Googlebots. Cant find the code right now, but if you PM me I'll dig it out. Google it - you'll find out how to verify Googlebot.

Originally Posted by ghost

No problem with the band width unless its been wasted

Sorry band width was not an issue , I am just finding it hard to believe google is giving this site so much attention.
News feeds 2 on the home page only
Have a robots,txt
have a google xml sitemap
have a yahoo txt sitemap
Will check the ip later
Maybe I have done something right and hit pay dirt and should stop moaning.

Originally Posted by blacknight

In any case you could check the logs for the IP ranges and see if they belong to Google or not...

Yes the server logs revealed all , and a its genuine Googlebot eight visits to day.
The problem I am having
The site has an Events Calender on the side bar of each page (an include page)
also has a photo gallery page , Each gallery displays 10 pics with links to larger images
in each case google is following the link for the larger image also finding the calender on each page and chasing so many links deep in to the calender as well, ie /gallery.php?picture_ref=1&user_eventsDate=2007-09
so the possiblities are 10 picture links multiplied by 30 calender links on each page = not good for google or me as most of the events are empty.
I will have to rectify this.
Thanks for the Help.
Ghost

You can use the robots.txt to stop the googlebots from spidering certain sections of your site.

NB: Not all bots behave