How to read the response stream before the Http response completes

user434917 picture user434917 · May 16, 2010 · Viewed 30.1k times · Source

When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.

Answer

scherand picture scherand · May 18, 2010

I think this is very close to what @Zachary suggests. And it (seems to) work(s); actually I think using using as @Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.

In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.

You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?

The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.

namespace StreamReadWebRequest
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.Net;
    using System.IO;

    class Program
    {
        static void Main(string[] args)
        {
            HttpWebRequest req;
            HttpWebResponse res = null;

            try
            {
                req = (HttpWebRequest)WebRequest.Create(
                        "http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
                res = (HttpWebResponse)req.GetResponse();
                Stream stream = res.GetResponseStream();

                byte[] data = new byte[4096];
                int read;
                while ((read = stream.Read(data, 0, data.Length)) > 0)
                {
                    Process(data, read);
                }
            }
            finally
            {
                if (res != null)
                    res.Close();
            }
            Console.In.Read();
        }

        private static void Process(byte[] data, int read)
        {
            Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
        }
    }
}