Azure Data Factory get data from foreach value

MAK picture MAK · Sep 13, 2018 · Viewed 7.3k times · Source

I've a lookup activity with gets value from a container within a blob. I've a foreach activity which is connected to the lookup activity

enter image description here

and under settings it has the value:

@activity('LookupMessageType').output.value

I've another copy activity which runs within this foreach activity. It copies data from cosmos DB to Azure Data Lake.
enter image description here

This is the query in the source dataset:

select c.Tag, data.assetTag, data.timestamp, data.jsonVersion, data.messageType, data.speed from c join data in c.data
where (data.speed>  item().speed_Value) AND
(data.timestamp >= '@{formatDateTime(addhours(pipeline().TriggerTime, -1), 'yyyy-MM-ddTHH:mm:ssZ' )}' 
AND data.timestamp < '@{formatDateTime(pipeline().TriggerTime, 'yyyy-MM-ddTHH:mm:ssZ' )}')

I get an error as I run this pipeline:

{
    "errorCode": "2200",
    "message": "Failure happened on 'Source' side. ErrorCode=UserErrorDocumentDBReadError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=DocumentDb operation failed: Message: {\"errors\":[{\"severity\":\"Error\",\"location\":{\"start\":231,\"end\":235},\"code\":\"SC2005\",\"message\":\"'item' is not a recognized built-in function name.\"}]}\r\nActivityId: *redacted*, documentdb-dotnet-sdk/1.21.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0.,Source=Microsoft.DataTransfer.ClientLibrary.DocumentDb,''Type=Microsoft.Azure.Documents.BadRequestException,Message=Message: {\"errors\":[{\"severity\":\"Error\",\"location\":{\"start\":231,\"end\":235},\"code\":\"SC2005\",\"message\":\"'item' is not a recognized built-in function name.\"}]}\r\nActivityId: redacted, documentdb-dotnet-sdk/1.21.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0,Source=Microsoft.Azure.Documents.Client,''Type=System.Runtime.InteropServices.COMException,Message=Exception from HRESULT: 0x800A0B00,Source=,'",
    "failureType": "UserError",
    "target": "Copy Data1"
}

It is saying item is not a built in function name. I want to use value from the foreach activity to query the source.

Furthermore, here is my dynamic filepath content

@concat('test_', item().speed, '.csv')

I get the desired dynamic file structure with this expression, if I am using static values in the query like: data.speed> 500

Answer

MAK picture MAK · Sep 13, 2018

So, I figured out what was wrong with the query: instead of

where (data.speed> item().speed_Value)

I had to use

where (data.speed> @{item().speed_value})