urllib3 connectionpool - Connection pool is full, discarding connection

python connection pool urllib3
JavaFan picture JavaFan · Dec 13, 2018 · Viewed 20.8k times · Source

Does seeing the 

urllib3.connectionpool WARNING - Connection pool is full, discarding connection

mean that I am effectively loosing data (because of lost connection)
OR
Does it mean that connection is dropped (because pool is full); however, the same connection will be re-tried later on when connection pool becomes available?

Answer

shazow picture shazow · Dec 13, 2018

No data is being lost!

The connection is being discarded after the request is completed (because the pool is full, as mentioned). This means that this particular connection is not going to be re-used in the future.

Because a urllib3 PoolManager reuses connections, it will limit how many connections are allowed per host at any given time to avoid accumulating too many unused sockets.

If you're relying on concurrency, it could be a good idea to increase the size of the pool (maxsize) to be at least as large as the number of threads you're using, so that each thread effectively gets its own connection.

More details here: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#customizing-pool-behavior

Related questions

Python [Errno 98] Address already in use

In my Python socket program, I sometimes need to interrupt it with Ctrl-C. When I do this, it does close the connection using socket.close(). However, when I try to reopen it I have to wait what seems like a …

Read More
Connect to Exchange mailbox with Python

I need to connect to an Exchange mailbox in a Python script, without using any profile setup on the local machine (including using Outlook). If I use win32com to create a MAPI.Session I could logon (with the Logon() …

Read More
Python requests - Exception Type: ConnectionError - try: except does not work

I am using a webservice to retrieve some data but sometimes the url is not working and my site is not loading. Do you know how I can handle the following exception so there is no problem with the site …

Read More