Running simultaneous background jobs in PowerShell in pretty straightforward, but I cannot seem to get it working with the new (in v3) cmdlet Invoke-WebRequest.
I have several thousand files I am downloading scriptomagically via PowerShell. Works great, but doing them in series is taking days:
for($f=0;$f -lt $urlList.Count;$f++)
{
$remote = $urlList[$f] + $fileList[$f]
$local = 'C:\folder\' + $fileList[$f]
Invoke-WebRequest $remote -Method Get -OutFile $local -UserAgent FireFox
}
I've made several attempts at leveraging the the 'AsJob' method, but either they error-out or complete fine yet no local files get saved. Here's an example of the latter:
for($f=0;$f -lt $urlList.Count;$f++)
{
$remote = $urlList[$f] + $fileList[$f]
$local = 'C:\folder\' + $fileList[$f]
$command = "Invoke-WebRequest $remote -Method Get -OutFile $local -UserAgent FireFox"
Start-Job {Invoke-Expression -Command $command}
}
Get-Job|Wait-Job
Output examples:
Id Name PSJobTypeName State HasMoreData Location Command
-- ---- ------------- ----- ----------- -------- -------
339 Job339 BackgroundJob Running True localhost Invoke-Expression -Com...
341 Job341 BackgroundJob Running True localhost Invoke-Expression -Com...
343 Job343 BackgroundJob Running True localhost Invoke-Expression -Com...
339 Job355 BackgroundJob Completed True localhost Invoke-Expression -Com...
341 Job357 BackgroundJob Completed True localhost Invoke-Expression -Com...
343 Job359 BackgroundJob Completed True localhost Invoke-Expression -Com...
The weird thing is, the jobs above complete roughly as fast as it would take to download the associated file...so it seems that data is going somewhere -- just not to my save location. I assume it's going to memory but being dumped without being saved. Tried adding and removing the 'PassThru' option, but get the same results either way. Also tried piping it to Out-File, still no joy. Not sure why this particular cmdlet is being such a bugger.
You need to pass in the parameters in such as way that they actually make it into the job. In V3, you can use the $using: syntax to do this:
for($f=0;$f -lt $urlList.Count;$f++)
{
$remote = $urlList[$f] + $fileList[$f]
$local = 'C:\folder\' + $fileList[$f]
Start-Job {Invoke-WebRequest $using:remote -Method Get -OutFile $using:local -UserAgent FireFox}
}
Get-Job|Wait-Job
BTW you don't need to use Invoke-Expression. Just put the code you want to run in a job inside a scriptblock i.e. a set of curly braces {...}
.