cURL request on a page requiring JavaScript support

javascript cookies curl web-scraping spoofing
user965748 picture user965748 · Sep 6, 2012 · Viewed 25.2k times · Source

I need to get the HTML source of pinnaclesports.com. The problem is it detects whether cookies and JS are enabled and if not, it just returns some page saying

This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your browser.

Is there any way how to spoof JS support when using cURL?

EDIT: I can use a headless browser that runs either as a Perl/Ruby module or is written in PHP

Answer

João Paulo Cercal picture João Paulo Cercal · Jul 30, 2014

Other sugestion is set the user agent, this solution works for me on parser of the Google Groups:

curl -L -v "https://groups.google.com/d/forum/<GROUP-NAME>" -A "Mozilla/5.0 (compatible;  MSIE 7.01; Windows NT 5.0)"

Related questions

How do I set/unset a cookie with jQuery?

How do I set and unset a cookie using jQuery, for example create a cookie named test and set the value to 1?

Read More
Set cookie and get cookie with JavaScript

I'm trying to set a cookie depending on which CSS file I choose in my HTML. I have a form with a list of options, and different CSS files as values. When I choose a file, it should be saved …

Read More
Get cookie by name

I have a getter to get the value from a cookie. Now I have 2 cookies by the name shares= and by the name obligations= . I want to make this getter only to get the values from the obligations cookie. How …

Read More