Download Multiple Files from http using Powershell with proper names

Question

I have searched for something similar and I keep running across the FTP download answers. This is helpful information, but ultimately proving to be difficult to translate. I have found a powershell script and it works, but I am wondering if it can be tweaked for my needs. I don't have much experience with powershell scripting, but I'm trying to learn.

The need is this. I need to download and install a series of files to a remote machine, unattended. The files are distributed via email via tinyurls. I currently throw those into a .txt file, then have a powershell script read the list and download each file.

Requirements of the project and why I have turned to powershell (and not other utilities), is that these are very specialized machines. The only tools available are ones that are baked into Windows 7 embedded.

The difficulties I run into are: The files download one at the time. I would like to grab as many downloads at the same time that the web server will allow. (usually 6)

The current script creates file names based off the tinyurl. I need the actual file name from the webserver.

Thanks in advance for any suggestions.

Below is the script I’m currently using.

#  Copyright (C) 2011 by David Wright ([email protected])
#  All Rights Reserved.

#  Redistribution and use in source and binary forms, with or without
#  modification or permission, are permitted.

#  Additional information available at http://www.digitalwindfire.com.

$folder = "d:\downloads\"
$userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1"
$web = New-Object System.Net.WebClient
$web.Headers.Add("user-agent", $userAgent)


Get-Content "d:\downloads\files.txt" |
    Foreach-Object { 
        "Downloading " + $_
        try {
            $target = join-path $folder ([io.path]::getfilename($_))
            $web.DownloadFile($_, $target)
        } catch {
            $_.Exception.Message
        }
}

Answer 1

If you do the web request before you decide on file name you should be able to get the expanded path (otherwise you would have to make two web requests, one to get the extended path and one to download the file).

When I tried this, I found that the BaseResponse property of the Microsoft.PowerShell.Commands.HtmlWebResponseObject returned by the Invoke-WebRequest cmdlet had a ResponseUri property which was the extended path we are looking for.

If you get the correct response, just save the file using the name from the extended path, something like the following (this sample code does not look at HTTP response codes or similar, but expects everything to go well):

function Save-TinyUrlFile
{
    PARAM (
        $TinyUrl,
        $DestinationFolder
    )

    $response = Invoke-WebRequest -Uri $TinyUrl
    $filename = [System.IO.Path]::GetFileName($response.BaseResponse.ResponseUri.OriginalString)
    $filepath = [System.IO.Path]::Combine($DestinationFolder, $filename)
    try
    {
        $filestream = [System.IO.File]::Create($filepath)
        $response.RawContentStream.WriteTo($filestream)
        $filestream.Close()
    }
    finally
    {
        if ($filestream)
        {
            $filestream.Dispose();
        }
    }
}

This method could be called using something like the following, given that the $HOME\Documents\Temp folder exists:

Save-TinyUrlFile -TinyUrl http://tinyurl.com/ojt3lgz -DestinationFolder $HOME\Documents\Temp

On my computer, that saves a file called robots.txt, taken from a github repository, to my computer.

If you want to download many files at the same time, you could let PowerShell make this happen for you. Either use PowerShell workflows parallel functionality or simply start a Job for each url. Here's a sample on how you could do it using PowerShell Jobs:

Get-Content files.txt | Foreach {
    Start-Job {
        function Save-TinyUrlFile
        {
            PARAM (
                $TinyUrl,
                $DestinationFolder
            )

            $response = Invoke-WebRequest -Uri $TinyUrl
            $filename = [System.IO.Path]::GetFileName($response.BaseResponse.ResponseUri.OriginalString)
            $filepath = [System.IO.Path]::Combine($DestinationFolder, $filename)
            try
            {
                $filestream = [System.IO.File]::Create($filepath)
                $response.RawContentStream.WriteTo($filestream)
                $filestream.Close()
            }
            finally
            {
                if ($filestream)
                {
                    $filestream.Dispose();
                }
            }
        }

        Save-TinyUrlFile -TinyUrl $args[0] -DestinationFolder $args[1]
    } -ArgumentList $_, "$HOME\documents\temp"
}