What is the easiest way to run python scripts in a cloud server?

python cloud web-crawler virtual server
user1330691 picture user1330691 · Nov 13, 2014 · Viewed 21.2k times · Source

I have a web crawling python script that takes hours to complete, and is infeasible to run in its entirety on my local machine. Is there a convenient way to deploy this to a simple web server? The script basically downloads webpages into text files. How would this be best accomplished? Thanks!

Answer

alecxe picture alecxe · Nov 13, 2014

Since you said that performance is a problem and you are doing web-scraping, first thing to try is a Scrapy framework - it is a very fast and easy to use web-scraping framework. scrapyd tool would allow you to distribute the crawling - you can have multiple scrapyd services running on different servers and split the load between each. See:

There is also a Scrapy Cloud service out there:

Scrapy Cloud bridges the highly efficient Scrapy development environment with a robust, fully-featured production environment to deploy and run your crawls. It's like a Heroku for Scrapy, although other technologies will be supported in the near future. It runs on top of the Scrapinghub platform, which means your project can scale on demand, as needed.

Related questions

How can I access s3 files in Python using urls?

I want to write a Python script that will read and write files from s3 using their url's, eg:'s3:/mybucket/file'. It would need to run locally and in the cloud without any code changes. Is there a way …

Read More
Get Public URL for File - Google Cloud Storage - App Engine (Python)

Is there a python equivalent to the getPublicUrl PHP method? $public_url = CloudStorageTools::getPublicUrl("gs://my_bucket/some_file.txt", true); I am storing some files using the Google Cloud Client Library for Python, and I'm trying to figure out …

Read More
Heroku not recognized as an internal or external command (Windows)

Following this heroku tutorial to launch an app on heroku. But when I use the command heroku create, it says it's not recognized. I added C:\Program Files\Heroku to my PATH. How do I fix this?

Read More