POST request works in Postman, but not in Python Requests (200 response with robot detection)
I have a POST request that works perfectly with both Postman an cURL (it returns a JSON blob of data). However, when I perform the exact same request with Python's Requests library, I get a 200 success response, but instead of my JSON blob, I get this:
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
I've used HTTP request bins to verify that the request from Postman/cURL is exactly the same as the one from Python Requests.
Here is my Postman request in cURL:
curl -X POST \
https:/someurl/bla/bla \
-H 'Content-Type: application/json' \
-H 'Postman-Token: 2488e914-531e-4ac7-ae8d-8490b2242396' \
-H 'Referer: https://www.host.com/bla/bla/' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:65.0) Gecko/20100101 Firefox/65.0' \
-H 'cache-control: no-cache' \
-d '{"json1":"blabla","etc":"etc"}'
...and here is my Python code:
payload = {
"json1": "blabla",
"etc": "etc",
}
headers = {
'Host': 'www.host.com',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',
'Accept': 'application/json, text/javascript, */*; q=0.01',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Referer': 'https://www.host.com/bla/bla/',
'Content-Type':'application/json',
'X-Requested-With': 'XMLHttpRequest',
'Connection': 'keep-alive',
'Origin': 'https://www.host.com',
}
s = requests.Session()
response_raw = s.post(url, json=payload, headers=headers)
print(response_raw)
print(response_raw.text)
I have verified that the payload and headers are correct and valid. Any help would be much appreciated; thanks!
Answer
You are getting a 200 success response but not JSON data in the response.
This means that is just a response object. It contains only response code
to extract blob information from the response, convert response object to json
simply json_resp = response_raw.json()
This json_resp contains your actual response details.