python: [Errno 10054] An existing connection was forcibly closed by the remote host

python twitter web-crawler
Nama Keru picture Nama Keru · Jan 11, 2012 · Viewed 104k times · Source

I am writing python to crawl Twitter space using Twitter-py. I have set the crawler to sleep for a while (2 seconds) between each request to api.twitter.com. However, after some times of running (around 1), when the Twitter's rate limit not exceeded yet, I got this error.

[Errno 10054] An existing connection was forcibly closed by the remote host.

What are possible causes of this problem and how to solve this?

I have searched through and found that the Twitter server itself may force to close the connection due to many requests.

Thank you very much in advance.

Answer

David Schwartz picture David Schwartz · Jan 11, 2012

This can be caused by the two sides of the connection disagreeing over whether the connection timed out or not during a keepalive. (Your code tries to reused the connection just as the server is closing it because it has been idle for too long.) You should basically just retry the operation over a new connection. (I'm surprised your library doesn't do this automatically.)

Related questions

How to extract hours and minutes from a datetime.datetime object?

I am required to extract the time of the day from the datetime.datetime object returned by the created_at attribute. But I do not understand how to do that. This is my code for getting the datetime.datetime object. …

Read More
Pretty-Print JSON Data to a File using Python

A project for class involves parsing Twitter JSON data. I'm getting the data and setting it to the file without much trouble, but it's all in one line. This is fine for the data manipulation I'm trying to do, but …

Read More
syntaxError: 'continue' not properly in loop

I have been struggling with this error for a while now and there seems to be different opinions regarding why the interpreter complains about the 'continue'. So I would like to provide the erroneous code below. import tweepy import time …

Read More