Automatic searching of Google

Status
Not open for further replies.

Fintan

New Member
Hi Guys

Im looking for a little advice.

I want to do a small project that will download in csv the top 10 results in google for a particular search term.

However I have approximately 500 search terms, so want to automate this as much as possible.

Are there software programmes out there that can do this?

I know there is google soap and I do have a key, however its outside my knowledge of programming.

All advice greatly appreciated.

Thanks
Fintan
 

Forbairt

Portfolio Building
What are your search terms currently in ? excel ? txt file ? paper :p ?

How do you want the output ?

[I don't specifically know of any program out there ... there are loads probably]
 

CiaranR

Weeno Ltd + Skimlinks.com
Easiest way to do it would be a simple php file that pulls your search terms from a mysql DB googles them and scrapes the top 10 results out and stores them in a second table.

Wouldn't take too long to write and would be the easiest for reporting on and storing the info.

Edit: run it on a cron every day obviously, to fully automate it.
 

Fintan

New Member
I can present the search terms in whatever format is needed thats no big deal.

The results I'd like to see are as follows:

Search Phrase

Position 1
URL

Position 2
URL

All the way down to 10

I've tried googling but am struggling try to find anything relevant.
 

Fintan

New Member
Easiest way to do it would be a simple php file that pulls your search terms from a mysql DB googles them and scrapes the top 10 results out and stores them in a second table.

Wouldn't take too long to write and would be the easiest for reporting on and storing the info.

Edit: run it on a cron every day obviously, to fully automate it.
Do you know of anything off the shelf that would do this? Or how I go about getting this set up or finding someone to do it for me?

Cheers
Fintan
 

Forbairt

Portfolio Building
I have some code here that'll partially do it for you ... but its rough very rough ... if I get a chance I'll pass it on later if anyone is interested ...

Not sure how much google would like getting 500 keyword requests though ...

there are probably better ways of doing it as well hmm...
 

Fintan

New Member
I'd appreciate any help, thank Forbairt.

Well the api allows 1000 searches per day, so Im not too worried about that.

The google soap api does allow automated searches, its just that I don't know enough to set it up to work the way I want.
 

RedCardinal

New Member
I have some code here that'll partially do it for you ... but its rough very rough ... if I get a chance I'll pass it on later if anyone is interested ...

Not sure how much google would like getting 500 keyword requests though ...

there are probably better ways of doing it as well hmm...
I imagine they'll be telling you you're spyware fairly quickly...
 

Forbairt

Portfolio Building
code was rather simple actually... parsing a page of its links ...

a loop could be dumped onto it and you could run it overnight but I won't be going that route ...


Took a look at the Search API ... with a small bit of work it should be possible but yeah I'd have to read up on the TOS on it which I haven't done.

Being allowed to run 1000 ? queries against the API a day may not mean you're allowed to run an automated query against it ? (though time delays and so on could be implemented...)
 

RedCardinal

New Member
Automated requests are no problem with API. You can hammer them until you hit 1k, then they tell you to go forth and prosper. Until tomorrow.
 

Forbairt

Portfolio Building
EDIT: no its not using the SOAP API key

I'm going to release this in pro chat for anyone who is interested
Its crap code ... its early beta .. spent about 2 hours this morning figuring out a few bits as I've not played with the API before so I don't have much to show for it
 

georgiecasey

New Member
i thought the google api was gone?

anyways, i've code that scrapes normal results if you want it. it goes pretty slow though as google is fairly good at blocking automated requests so you just have to have a few seconds between hits.
 

Forbairt

Portfolio Building
i thought the google api was gone?

anyways, i've code that scrapes normal results if you want it. it goes pretty slow though as google is fairly good at blocking automated requests so you just have to have a few seconds between hits.
google soap API is gone ... :) ish ... no new keys are being generated its still functional though to the best of my knowledge

What I had ended up using was google Ajax Search API ..


There are various bits of code floating around the net that will parse the google searches ... (still I prefer to a certain extent what I was doing as its something google technically allows people to do query their API ... to the limit .. where as RedCardinal said repeated requests will be seen as an attack / spam / whatever and get blocked ? (sorry if thats not what RedCardinal has said it was what I understood)
 
Status
Not open for further replies.
Award-winning Mac antivirus and Internet security software
Top