Rotating Proxies for web scraping

python proxy screen-scraping web-crawler squid
Jacob picture Jacob · Dec 19, 2009 · Viewed 15.8k times · Source

I've got a python web crawler and I want to distribute the download requests among many different proxy servers, probably running squid (though I'm open to alternatives). For example, it could work in a round-robin fashion, where request1 goes to proxy1, request2 to proxy2, and eventually looping back around. Any idea how to set this up?

To make it harder, I'd also like to be able to dynamically change the list of available proxies, bring some down, and add others.

If it matters, IP addresses are assigned dynamically.

Thanks :)

Answer

sw. picture sw. · Dec 23, 2011

I've setted up rotating proxies using HAProxy + DeleGate + Multiple Tor Instances. With Tor you don't have good control of bandwidth and latency but it's useful for web scraping. I've just published an article on the subject: Running Your Own Anonymous Rotating Proxies

Related questions

Using pip behind a proxy with CNTLM

I am trying to use pip behind a proxy at work. One of the answers from this post suggested using CNTLM. I installed and configured it per this other post, but running cntlm.exe -c cntlm.ini -I -M http://…

Read More
How to get pip to work behind a proxy server

I am trying to use python package manager pip to install a package and it's dependencies from the internet. However I am behind a proxy in my college and have already set the http_proxy environment variable. But when I …

Read More
How to use pip on windows behind an authenticating proxy

My computer is running windows behind a proxy on a windows server (using active directory), and I can't figure out how to get through it with pip (in python3). I have tried using --proxy, but it still just timeouts. I …

Read More