Why do I constantly see "Resetting dropped connection" when uploading data to my database?

python http connection-pooling python-requests urllib3
Andrew Thompson picture Andrew Thompson · Mar 5, 2015 · Viewed 29.8k times · Source

I'm uploading hundreds of millions of items to my database via a REST API from a cloud server on Heroku to a database in AWS EC2. I'm using Python and I am constantly seeing the following INFO log message in the logs.

[requests.packages.urllib3.connectionpool] [INFO] Resetting dropped connection: <hostname>

This "resetting of the dropped connection" seems to take many seconds (sometimes 30+ sec) before my code continues to execute again.

  • Firstly what exactly is happening here and why?
  • Secondly is there a way to stop the connection from dropping so that I am able to upload data faster?

Thanks for your help. Andrew.

Answer

laike9m picture laike9m · Jan 23, 2016

Requests uses Keep-Alive by default. Resetting dropped connection, from my understanding, means a connection that should be alive was dropped somehow. Possible reasons are:

  1. Server doesn't support Keep-Alive.
  2. There's no data transfer in established connections for a while, so server drops connections.

See https://stackoverflow.com/a/25239947/2142577 for more details.

Related questions

http request with timeout, maximum size and connection pooling

I'm looking for a way in Python (2.7) to do HTTP requests with 3 requirements: timeout (for reliability) content maximum size (for security) connection pooling (for performance) I've checked quite every python HTTP librairies, but none of them meet my requirements. For …

Read More
How do I download a file over HTTP using Python?

I have a small utility that I use to download an MP3 file from a website on a schedule and then builds/updates a podcast XML file which I've added to iTunes. The text processing that creates/updates the XML …

Read More
What is the quickest way to HTTP GET in Python?

What is the quickest way to HTTP GET in Python if I know the content will be a string? I am searching the documentation for a quick one-liner like: contents = url.get("http://example.com/foo/bar") But all I …

Read More