Dynamically Append datetime to filename during copy activity or when specifying name in blob dataset

Question

I am saving a file to blob storage in Data factory V2, when I specify the location to save to I am calling the file (for example) file1 and it saves in blob as file1, no problem. But can I use the dynamic content feature to append the datetime to the filename so its something like file1_01-07-2019_14-30-00 ?(7th Jan 14:30:00 just in case its awkward to read). Alternatively, can I output the result (the filename) of the webhook activity to the next activity (the function)?

Thank you.

Answer 1

I couldn't get this to work without editing the copy pipeline JSON file directly (late 2018 - may not be needed anymore). You need dynamic code in the copy pipeline JSON and settings defined in the dataset for setting filename parameters.

In the dataset define 'Parameters' for folder path and/or filename (click '+ New' and give them any name you like) e.g. sourceFolderPath, sourceFileName.

Then in dataset under 'Connection' include the following in the 'File path' definition: @dataset().sourceFolderPath and @dataset().sourceFileName either side of the '/' (see screenshot below)

In the copy pipeline click on 'Code' in the upper right corner of pipeline window and look for the following code under the 'blob' object you want defined by a dynamic filename - it the 'parameters' code isn't included add it to the JSON and click the 'Finish' button - this code may be needed in 'inputs', 'outputs' or both depending on the dynamic files you are referencing in your flow - below is an example where the output includes the date parameter in both folder path and file name (the date is set by a Trigger parameter):

  "inputs": [
     {
        "referenceName": "tmpDataForImportParticipants",
        "type": "DatasetReference"
     }
  ],
  "outputs": [
      {
         "referenceName": "StgParticipants",
         "type": "DatasetReference",
         "parameters": {
              "sourceFolderPath": {
                   "value": <derived value of folder path>,
                   "type": "Expression"
               },
               "sourceFileName": {
                    "value": <derived file name>,
                    "type": "Expression"
               }
          }
      }
  ]

Derived value of folder path may be something like the following - this results in a folder path of yyyy/mm/dd within specified blobContainer:

"blobContainer/@{formatDateTime(pipeline().parameters.windowStart,'yyyy')}/@{formatDateTime(pipeline().parameters.windowStart,'MM')}/@{formatDateTime(pipeline().parameters.windowStart,'dd')}"

or it could be hardcoded e.g. "blobContainer/directoryPath" - don't include '/' at start or end of definition

Derived file name could be something like the following:

"@concat(string(pipeline().parameters.'_',formatDateTime(dataset().WindowStartTime, 'MM-dd-yyyy_hh-mm-ss'))>,'.txt')"

You can include any parameter set by the Trigger e.g. an ID value, account name, etc. by including pipeline().parameters.

Dynamic Dataset Parameters example

Dynamic Dataset Connection example