In requests library, how can I avoid "HttpConnectionPool is full, discarding connection" warning?

Question

I'm using python requests library with sessions:

def _get_session(self):
    if not self.session:
        self.session = requests.Session()
    return self.session

And sometimes I'm getting this warning in my logs:

[2014/May/12 14:40:04 WARNING ] HttpConnectionPool is full, discarding connection: www.ebi.ac.uk

My question is: why this is warning and not an exception?

This is the code responsible for this (from http://pydoc.net/Python/requests/0.8.5/requests.packages.urllib3.connectionpool/):

def _put_conn(self, conn):
    try:
        self.pool.put(conn, block=False)
    except Full:
        # This should never happen if self.block == True
        log.warning("HttpConnectionPool is full, discarding connection: %s"
                    % self.host)

Why this exception is catched here? If it was reraised, I could handle this exception in my code, by creating new session and deleting the old one.

If it's only a warning, does it mean it doesn't affect my results in any way? Can I ignore it? If not, how can I handle this situation?

Answer 1

From Requests docs in http://docs.python-requests.org/en/latest/api/

 class requests.adapters.HTTPAdapter(pool_connections=10, pool_maxsize=10, max_retries=0, pool_block=False)

The built-in HTTP Adapter for urllib3.

Provides a general-case interface for Requests sessions to contact HTTP and HTTPS urls by implementing the Transport Adapter interface. This class will usually be created by the Session class under the covers.

Parameters:

• pool_connections – The number of urllib3 connection pools to cache.
• pool_maxsize – The maximum number of connections to save in the pool.
• max_retries (int) – The maximum number of retries each connection should attempt. Note, this applies only to failed connections and timeouts, never to requests where the server returns a response.
• pool_block – Whether the connection pool should block for connections.

and a little below, comes an example

import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=3)
s.mount('http://', a)

Try this

a = requests.adapters.HTTPAdapter(pool_connections = N, pool_maxsize = M)

Where N and M are suitable for your program.