Getting HEAD content with Python Requests

python head http-request python-requests
Yarin picture Yarin · Mar 4, 2012 · Viewed 27.2k times · Source

I'm trying to parse the result of a HEAD request done using the Python Requests library, but can't seem to access the response content.

According to the docs, I should be able to access the content from requests.Response.text. This works fine for me on GET requests, but returns None on HEAD requests.

GET request (works)

import requests
response = requests.get(url)
content = response.text

content = <html>...</html>

HEAD request (no content)

import requests
response = requests.head(url)
content = response.text

content = None

EDIT

OK I've quickly realized form the answers that the HEAD request is not supposed to return content- only headers. But does that mean that, to access things found IN the <head> tag of a page, like <link> and <meta> tags, that one must GET the whole document?

Answer

phihag picture phihag · Mar 4, 2012

By definition, the responses to HEAD requests do not contain a message-body.

Send a GET request if you want to, well, get a response body. Send a HEAD request iff you are only interested in the response status code and headers.

HTTP transfers arbitrary content; the HTTP term header is completely unrelated to an HTML <head>. However, HTTP can be advised to download only a part of the document. If you know the length of the HTML <head> code (or an upper boundary therefor), you can include an HTTP Range header in your request that advises the remote server to only return a certain number of bytes. If the remote server supports HTTP ranges, it will then serve the reduced answer.

Related questions

Pandas - Get first row value of a given column

This seems like a ridiculously easy question... but I'm not seeing the easy answer I was expecting. So, how do I get the value at an nth row of a given column in Pandas? (I am particularly interested in the …

Read More
How to read first N lines of a file?

We have a large raw data file that we would like to trim to a specified size. I am experienced in .net c#, however would like to do this in python to simplify things and out of interest. How would …

Read More
Get first element of Series without knowing the index

Is that any way that I can get first element of Seires without have information on index. For example,We have a Series import pandas as pd key='MCS096' SUBJECTS=pd.DataFrame({'ID':Series([146],index=[145]),\ 'study':Series(['MCS'],index=[145]),\ …

Read More