It may be that the bots are downloading the entire db repeatedly over the month. (Each php page is effectively a new page to most bots especially if there is any date type information included.) Google is not too bad and understands 304s (unchanged). Yahoo's Slurp is a bit screwed up. Microsoft's bot is so far beyond screwed up that it is in another universe. The bots should not be accessing the shopping cart. The revised robots.txt with the images/shopping etc added may make a difference. However it is limited by the number of times that the bots recheck robots.txt.

Regards...jmcc