How to run PowerShell from Azure Data Factory

Rahul Reddy Vudem picture Rahul Reddy Vudem · Oct 9, 2018 · Viewed 7.4k times · Source

I have PowerShell script which splits a complex CSV file to a smaller CSV file for every 1000 records. Here is the code:

$i=0;Get-Content C:\Users\dell\Desktop\Powershell\Input\bigsizeFile.csv -ReadCount 1000 | %{$i++; $_ | Out-File C:\Users\dell\Desktop\Powershell\Output\file$i.csv
}

Now I want to use this script in Azure PowerShell and I want to run this from Azure Data Factory. Can Someone please help with this.

Answer

user2197446 picture user2197446 · Nov 1, 2019

I was able to run a PowerShell script by loading the script as an application in the Batch account associated to the Batch Custom Activity. From there it was figuring out the correct syntactical sugar to use the environment variables to get the path to my script and run it. I did this like so:

powershell powershell -command (\"(Get-ChildItem Env:AZ_BATCH_APP_PACKAGE_powershellscripts#1.0).Value\" + '\\Powershell\\processFromAzure.ps1

If you just want to call from the task dir, this should work:

powershell powershell -command ("$env:AZ_BATCH_TASK_DIR" + '\wd\processFromAzure.ps1')

Hope this helps someone else out there!