Python requests speed up using keep-alive

python http python-requests python-sockets
PaulOverflow picture PaulOverflow · Aug 11, 2014 · Viewed 52.9k times · Source

In the HTTP protocol you can send many requests in one socket using keep-alive and then receive the response from server at once, so that will significantly speed up whole process. Is there any way to do this in python requests lib? Or are there any other ways to speed this up that well using requests lib?

Answer

metatoaster picture metatoaster · Aug 11, 2014

Yes, there is. Use requests.Session and it will do keep-alive by default.

I guess I should include a quick example:

import logging
import requests

logging.basicConfig(level=logging.DEBUG)
s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
s.get('http://httpbin.org/cookies/set/anothercookie/123456789')
r = s.get("http://httpbin.org/cookies")
print(r.text)

You will note that these log message occur

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/sessioncookie/123456789 HTTP/1.1" 302 223
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 55
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/anothercookie/123456789 HTTP/1.1" 302 223
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90

If you wait a little while, and repeat the last get call

INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90

Note that it resets the dropped connection, i.e. reestablishing the connection to the server to make the new request.

Related questions

Python requests - print entire http request (raw)?

While using the requests module, is there any way to print the raw HTTP request? I don't want just the headers, I want the request line, headers, and content printout. Is it possible to see what ultimately is constructed from …

Read More
Python Requests library redirect new url

I've been looking through the Python Requests documentation but I cannot see any functionality for what I am trying to achieve. In my script I am setting allow_redirects=True. I would like to know if the page has been …

Read More
Python-Requests close http connection

I was wondering, how do you close a connection with Requests (python-requests.org)? With httplib it's HTTPConnection.close(), but how do I do the same with Requests? Code: r = requests.post("https://stream.twitter.com/1/statuses/filter.json", data={'track':…

Read More