Amazon Cloud Watch Log - PutLogEventsRequest - The given sequenceToken is invalid

Question

I am building a small log tracker for my application using Amazon Cloud Watch service. The idea is not track log outputs on files and use search engine from aws console to find log information.

I am using:

• Eclipse as IDE
• Java 8
• Dependencies : aws-java-sdk-core / aws-java-sdk-cloudwatch V 1.10.49

In the other hand I have the following AWS configuration:

• Access and private keys
• Region : California
• Log group : demo1
• Log stream : stream1

I was writing the following code to make a simple functional test:

package com.test.pe.cloudwatch;

import java.text.ParseException;
import java.util.ArrayList;
import java.util.Calendar;

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.logs.AWSLogsClient;
import com.amazonaws.services.logs.model.InputLogEvent;
import com.amazonaws.services.logs.model.PutLogEventsRequest;
import com.amazonaws.services.logs.model.PutLogEventsResult;
import com.test.pe.base.CredentialBuilder; 

public class RegisterLog {
    private static String LOG_GROUP = "demo1";
    private static String LOG_STREAM = "stream1";

    public static void main(String[] args) throws ParseException {
        // building my credential and calendar instances
        AWSCredentials credential = CredentialBuilder.getCredential();
        Calendar calendar = Calendar.getInstance();
        // building a cloud watch log client
        AWSLogsClient cloudWatchlog = new AWSLogsClient(credential);
        cloudWatchlog.setRegion(Region.getRegion(Regions.US_WEST_1));
        // building a put request log
        PutLogEventsRequest request = new PutLogEventsRequest();
        request.setLogGroupName(LOG_GROUP);
        request.setLogStreamName(LOG_STREAM);
        // building my log event
        InputLogEvent log = new InputLogEvent();
        log.setMessage("Some message for a test");
        log.setTimestamp(calendar.getTimeInMillis());
        // building the array list log event
        ArrayList<InputLogEvent> logEvents = new ArrayList<InputLogEvent>();
        logEvents.add(log);
        // setting the error array list
        request.setLogEvents(logEvents);
        // make the request
        cloudWatchlog.putLogEvents(request);

        System.out.println("done!");
    }
}

When I run the code for the first time all is okey, the message is saved successfully.

However, when I execute the code for a second time I get the following exception:

Exception in thread "main" com.amazonaws.services.logs.model.InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: xxxxxxxxxxxxxxxxxxxxxxxxxxx (Service: AWSLogs; Status Code: 400; Error Code: InvalidSequenceTokenException; Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389)

XXXXXXXXXXX : Are token codes generated by amazon.

Reading the amazon documentation and I found the following info:

Request syntax:

{
    "LogEvents": [
        {
            "Message": "string",
            "Timestamp": number
        }
    ],
    "LogGroupName": "string",
    "LogStreamName": "string",
    "SequenceToken": "string"
}

SequenceToken

A string token that must be obtained from the response of the previous PutLogEvents request.

Type: String

Length constraints: Minimum length of 1.

Required: No

amazon documentation about cloud watch log REST API

And I decided to hard code the sequence token on my code as follows :

request.setSequenceToken("58523.......");

It works fine. And I made it just for test.

Finally, the only way that I found to get the sequence token was.

PutLogEventsResult response = cloudWatchlog.putLogEvents(request);
String token = response.getNextSequenceToken();

How can I validate and get the sequence code before make a request?. I can't find that on the documentation.

Answer 1

You normally get the nextToken when you do a call to putLogEvents (getNextSequenceToken). If there is more than one producer pushing to the stream, they are competing and only one of them can push at a time (ie: if you get a token and somebody else pushes it invalidates your token).

If this happens you need to describe the stream and get a new token: http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_DescribeLogStreams.html

So the pattern is: 1) If you don't have a valid token or don't have a token at all (you're just starting) describe the stream to find out the token. 2) Push using the token you've got. If the push is successful update the token 3) If the push is not successful go to 1), get a new token and try again. You may need to try multiple times (ie loop) if multiple producers.