Can servers block curl requests?

www.amitpatil.me picture www.amitpatil.me · Feb 22, 2012 · Viewed 45.3k times · Source

I am working on ZOHO API and trying to update the record using cURL. I tried different cURL variations, but it always returns "false". But when I call the same URL using a browser, it works.

Is there any way they can block cURL requests? Is there any other way I can call that URL using a POST or maybe a GET request?

The cURL code I have tried is as below:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
curl_close($ch);

Answer

Tom Desp picture Tom Desp · Feb 22, 2012

Many web servers want to block HTTP requests forged by something else than a browser, to prevent bots abuses. If you want to simulate/pretend your request from a browser, you at least have to:

  1. Pass the exact same headers than your browsers (use ie Firebug to get them)

    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    
  2. Change the user agent (name of the browser)

    curl_setopt($ch, CURLOPT_USERAGENT, $agent);
    
  3. Enable cookies (for eg redirection and session handling)

    curl_setopt ($ch, CURLOPT_COOKIEJAR, $file);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
    
  4. Add referers

    curl_setopt($curl, CURLOPT_REFERER, 'http://www.google.com');
    curl_setopt($curl, CURLOPT_AUTOREFERER, true);
    

And pray you haven't missed anything!