GZipStream complains magic number in header is not correct

jocull picture jocull · May 5, 2015 · Viewed 13.4k times · Source

I'm attempting to use National Weather Service (U.S.) data, but something has changed recently and the GZip file no longer opens.

.NET 4.5 complains that...

Message=The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
Source=System
StackTrace:
   at System.IO.Compression.GZipDecoder.ReadHeader(InputBuffer input)
   at System.IO.Compression.Inflater.Decode()
   at System.IO.Compression.Inflater.Inflate(Byte[] bytes, Int32 offset, Int32 length)
   at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count)

I don't understand what has changed, but this is becoming a real show-stopper. Can anyone with GZip format experience tell me what has changed to make this stop working?

A file that works:

http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201504/20150404/nws_precip_2015040420.tar.gz

A file that doesn't work:

http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201505/20150505/nws_precip_2015050505.tar.gz

Update with sample code

const string url = "http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201505/20150505/nws_precip_2015050505.tar.gz";
string appPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
string downloadPath = Path.Combine(appPath, Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), "nws_precip_2015050505.tar.gz");
using (var wc = new WebClient())
{
    wc.DownloadFile(url, downloadPath);
}

string extractDirPath = Path.Combine(appPath, "Extracted");
if (!Directory.Exists(extractDirPath))
{
    Directory.CreateDirectory(extractDirPath);
}
string extractFilePath = Path.Combine(extractDirPath, "nws_precip_2015050505.tar");

using (var fsIn = new FileStream(downloadPath, FileMode.Open, FileAccess.Read))
using (var fsOut = new FileStream(extractFilePath, FileMode.Create, FileAccess.Write))
using (var gz = new GZipStream(fsIn, CompressionMode.Decompress, true))
{
    gz.CopyTo(fsOut);
}

Answer

jocull picture jocull · May 6, 2015

It appears that this service SOMETIMES returns tar format files disguised as .tar.gz. This is very confusing, but if you check that the first two bytes are 0x1F and 0x8B, you can detect if the file is a GZip by checking its magic numbers manually.

using (FileStream fs = new FileStream(downloadPath, FileMode.Open, FileAccess.Read))
{
    byte[] buffer = new byte[2];
    fs.Read(buffer, 0, buffer.Length);
    if (buffer[0] == 0x1F
        && buffer[1] == 0x8B)
    {
        // It's probably a GZip file
    }
    else
    {
        // It's probably not a GZip file
    }
}