DynamoDb : Scan query does not return all the data
I have a DynamoDb table with thousands of data. I am scanning the table using Scan function and I have applied "Between" FilterExpression. However , the query response only gives 3 records whereas it should return about 100 records.
I have created the Lambda function using Node js.
Answer
The other common issue could be whether the scan is executed until LastEvaluatedKey is empty.
If you are already doing this and still not getting all the items, please show your code to look at it in detail.
If the total number of scanned items exceeds the maximum data set size limit of 1 MB, the scan stops and results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation. The results also include the number of items exceeding the limit. A scan can result in no table data meeting the filter criteria.
If LastEvaluatedKey is empty, then the "last page" of results has been processed and there is no more data to be retrieved.
If LastEvaluatedKey is not empty, it does not necessarily mean that there is more data in the result set. The only way to know when you have reached the end of the result set is when LastEvaluatedKey is empty.