Where do I start with a web bot?

Jon Duvillier picture Jon Duvillier · Oct 21, 2009 · Viewed 23k times · Source

I simply want to create an automatic script that can run (preferably) on a web-server, and simply 'clicks' on an object of a web page. I am new to Python or whatever language this would be used for so I thought I would go here to ask where to start! This may seem like I want the script to scam advertisements or do something illegal, but it's simply to interact with another website.

Answer

FrustratedWithFormsDesigner picture FrustratedWithFormsDesigner · Oct 21, 2009

It doesn't have to be Python, I've seen it done in PHP and Perl, and you can probably do it in many other languages. The general approach is:

1) You give your app a URL and it makes an HTTP request to that URL. I think I have seen this done with php/wget. Probably many other ways to do it.

2) Scan the HTTP response for other URLs that you want to "click" (really, sending HTTP requests to them), and then send requests to those. Parsing the links usually requires some understanding of regular expressions (if you are not familiar with regular expressions, brush up on it - it's important stuff ;)).