How to download a file from a URL in C, as a browser would?

Question

I want to download a file from a URL, this one : http://download.finance.yahoo/d/quotes.csv?s=YHOO+GOOG+MSFT&f=sl1d1t1c1hgvbap2 When i go on my browser and input this URL in my browser, the file is automatically downloaded as it should. What I want is to download this file without going on my browser using a program in C language, I need this type of information for a financial project. I tried to download the file using libcurl but libcurl downloads the HTML page corresponding to this URL which is empty of course because the only thing this URL does is start a download. I suppose this URL is the way int a HTTP server of some kind but I am completely lost as to how to get this file.

Thank you all in advance for your time and help, please if you can help by explaining or even better with C code feel free to do so and don't be afraid to be too precise.

Answer 1

Use libcurl, and see this examples page.

If you want to get it to work, make it work with the command line curl too, and use the --libcurl option. I suspect the problem might be more to do with javascript, cookies, a login or whatever. All these are soluble, but play with the command line to get it to work. My diagnosis here is that your URL is missing .com after yahoo.

For instance:

curl --silent --libcurl /tmp/test.c 'http://download.finance.yahoo.com/d/quotes.csv?s=YHOO+GOOG+MSFT&f=sl1d1t1c1hgvbap2'

produces the output to screen:

"YHOO",51.04,"11/21/2014","4:00pm",-0.21,52.25,50.99,22226984,N/A,52.49,"-0.41%"
"GOOG",537.50,"11/21/2014","4:00pm",+2.67,542.14,536.56,2218249,N/A,575.00,"+0.50%"
"MSFT",47.98,"11/21/2014","4:00pm",-0.72,49.05,47.57,42884796,N/A,49.05,"-1.48%"

and produces the code:

/********* Sample code generated by the curl command line tool **********
 * All curl_easy_setopt() options are documented at:
 * http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
 ************************************************************************/
#include <curl/curl.h>

int
main (int argc, char *argv[])
{
  CURLcode ret;
  CURL *hnd;

  hnd = curl_easy_init ();
  curl_easy_setopt (hnd, CURLOPT_URL,
                    "http://download.finance.yahoo.com/d/quotes.csv?s=YHOO+GOOG+MSFT&f=sl1d1t1c1hgvbap2");
  curl_easy_setopt (hnd, CURLOPT_NOPROGRESS, 1L);
  curl_easy_setopt (hnd, CURLOPT_USERAGENT, "curl/7.35.0");
  curl_easy_setopt (hnd, CURLOPT_MAXREDIRS, 50L);
  curl_easy_setopt (hnd, CURLOPT_TCP_KEEPALIVE, 1L);

  /* Here is a list of options the curl code used that cannot get generated
     as source easily. You may select to either not use them or implement
     them yourself.

     CURLOPT_WRITEDATA set to a objectpointer
     CURLOPT_WRITEFUNCTION set to a functionpointer
     CURLOPT_READDATA set to a objectpointer
     CURLOPT_READFUNCTION set to a functionpointer
     CURLOPT_SEEKDATA set to a objectpointer
     CURLOPT_SEEKFUNCTION set to a functionpointer
     CURLOPT_ERRORBUFFER set to a objectpointer
     CURLOPT_STDERR set to a objectpointer
     CURLOPT_HEADERFUNCTION set to a functionpointer
     CURLOPT_HEADERDATA set to a objectpointer

   */

  ret = curl_easy_perform (hnd);

  curl_easy_cleanup (hnd);
  hnd = NULL;

  return (int) ret;
}

/**** End of sample code ****/