Moving - not copying - data in Azure Data Factory

Sam picture Sam · Jan 19, 2017 · Viewed 8.8k times · Source

I'd like to set up an Azure Data Factory pipeline which performs a move (i.e. copy, verify, delete) operation rather than just a copy operation between Blob Storage and a Data Lake Store. I cannot seem to find any detail on how to do this.

Answer

Alex KeySmith picture Alex KeySmith · Jun 5, 2019

Just to add a contemporary update for anyone coming across this.

Data Factory V2 has relatively released a dedicated Delete Activity

At the time of writing this supports:

  • Azure Blob storage
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • File System
  • FTP
  • SFTP
  • Amazon S3
{
    "name": "DeleteActivity",
    "type": "Delete",
    "typeProperties": {
        "dataset": {
            "referenceName": "<dataset name>",
            "type": "DatasetReference"
        },
        "recursive": true/false,
        "maxConcurrentConnections": <number>,
        "enableLogging": true/false,
        "logStorageSettings": {
            "linkedServiceName": {
                "referenceName": "<name of linked service>",
                "type": "LinkedServiceReference"
            },
            "path": "<path to save log file>"
        }
    }
}

Taken from: https://docs.microsoft.com/en-gb/azure/data-factory/delete-activity