Can I change the connection pool size for Python's "requests" module?

python multithreading request python-requests
Skip Huffman picture Skip Huffman · Aug 27, 2013 · Viewed 24.3k times · Source

(edit: Perhaps I am wrong in what this error means. Is this indicating that the connection pool at my CLIENT is full? or a connection pool at the SERVER is full and this is the error my client is being given?)

I am attempting to make a large number of http requests concurrently using the python threading and requests module. I am seeing this error in logs: 

WARNING:requests.packages.urllib3.connectionpool:HttpConnectionPool is full, discarding connection:

What can I do to increase the size of the connection pool for requests?

Answer

Jahaja picture Jahaja · Sep 17, 2013

This should do the trick:

import requests
sess = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=100, pool_maxsize=100)
sess.mount('http://', adapter)
resp = sess.get("/mypage")

Related questions

How can I use threading in Python?

I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them. How do you clearly show tasks being divided for multi-threading?

Read More
Is there any way to kill a Thread?

Is it possible to terminate a running thread without setting/checking any flags/semaphores/etc.?

Read More
Run certain code every n seconds

Is there a way to, for example, print Hello World! every n seconds? For example, the program would go through whatever code I had, then once it had been 5 seconds (with time.sleep()) it would execute that code. I would …

Read More