Downloading Binary Files With Wininet

Question

I am currently programming a simple program, I want to distribute to my friends. What I am trying to accomplish, is to write some external binary files to a buffer from the internet, upon starting the program. To do this, I am using windows internet(wininet). Currently, I am using InternetReadFile to write the file to a buffer which I use later in the program. However, the File is not read completely, as in, the resulting size is much smaller than the size of the file on the server, when it should be the same.

I would like to do this, without using any external libraries.

Any idea of what could solve my problem?

Thanks, Andrew

Answer 1

The documentation makes the following remarks:

InternetReadFile operates much like the base ReadFile function, with a few exceptions. Typically, InternetReadFile retrieves data from an HINTERNET handle as a sequential stream of bytes. The amount of data to be read for each call to InternetReadFile is specified by the dwNumberOfBytesToRead parameter and the data is returned in the lpBuffer parameter. A normal read retrieves the specified dwNumberOfBytesToRead for each call to InternetReadFile until the end of the file is reached. To ensure all data is retrieved, an application must continue to call the InternetReadFile function until the function returns TRUE and the lpdwNumberOfBytesRead parameter equals zero.

Basically, there is no guarantee that the function to read exactly dwNumberOfBytesToRead. Check out how many bytes were actually read using the lpdwNumberOfBytesRead parameter.

Moreover, as soon as the total file size is larger than dwNumberOfBytesToRead, you will need to invoke the call multiple times. Because it cannot read more than dwNumberOfBytesToRead at once.

If you have the total file size in advance, the loop takes the following form:

::DWORD error = ERROR_SUCCESS;
::BYTE data[SIZE]; // total file size.
::DWORD size = 0;
::DWORD read = 0;
do {
    ::BOOL result = ::InternetReadFile(stream, data+size, SIZE-size, &read);
    if ( result == FALSE ) {
        error = ::GetLastError();
    }
}
while ((error == ERROR_SUCCESS) && (read > 0) && ((size+=read) < SIZE));
  // check that `SIZE` was correct.
if (size != SIZE) {
}

If not, then you need to write the data in the buffer to another file instead of accumulating it.

EDIT (SAMPLE TEST PROGRAM):

Here's a complete program that fetches StackOverflow's front page. This downloads about 200K of HTML code in 1K chunks and the full page is retrieved. Can you run this and see if it works?

#include <Windows.h>
#include <Wininet.h>
#include <iostream>
#include <fstream>

namespace {

    ::HINTERNET netstart ()
    {
        const ::HINTERNET handle =
            ::InternetOpenW(0, INTERNET_OPEN_TYPE_DIRECT, 0, 0, 0);
        if ( handle == 0 )
        {
            const ::DWORD error = ::GetLastError();
            std::cerr
                << "InternetOpen(): " << error << "."
                << std::endl;
        }
        return (handle);
    }

    void netclose ( ::HINTERNET object )
    {
        const ::BOOL result = ::InternetCloseHandle(object);
        if ( result == FALSE )
        {
            const ::DWORD error = ::GetLastError();
            std::cerr
                << "InternetClose(): " << error << "."
                << std::endl;
        }
    }

    ::HINTERNET netopen ( ::HINTERNET session, ::LPCWSTR url )
    {
        const ::HINTERNET handle =
            ::InternetOpenUrlW(session, url, 0, 0, 0, 0);
        if ( handle == 0 )
        {
            const ::DWORD error = ::GetLastError();
            std::cerr
                << "InternetOpenUrl(): " << error << "."
                << std::endl;
        }
        return (handle);
    }

    void netfetch ( ::HINTERNET istream, std::ostream& ostream )
    {
        static const ::DWORD SIZE = 1024;
        ::DWORD error = ERROR_SUCCESS;
        ::BYTE data[SIZE];
        ::DWORD size = 0;
        do {
            ::BOOL result = ::InternetReadFile(istream, data, SIZE, &size);
            if ( result == FALSE )
            {
                error = ::GetLastError();
                std::cerr
                    << "InternetReadFile(): " << error << "."
                    << std::endl;
            }
            ostream.write((const char*)data, size);
        }
        while ((error == ERROR_SUCCESS) && (size > 0));
    }

}

int main ( int, char ** )
{
    const ::WCHAR URL[] = L"http://stackoverflow.com/";
    const ::HINTERNET session = ::netstart();
    if ( session != 0 )
    {
        const ::HINTERNET istream = ::netopen(session, URL);
        if ( istream != 0 )
        {
            std::ofstream ostream("output.txt", std::ios::binary);
            if ( ostream.is_open() ) {
                ::netfetch(istream, ostream);
            }
            else {
                std::cerr << "Could not open 'output.txt'." << std::endl;
            }
            ::netclose(istream);
        }
        ::netclose(session);
    }
}

#pragma comment ( lib, "Wininet.lib" )