How to store Kinesis stream to S3 storage in specific folder structure within S3 bucket

Question

I have event captured by Kinesis Stream.I want to put all events on specific folder structure on S3. I want to make a folder with date stamp like all events of 15th June should go in that folder and 16th june onwards the new folder should come to pick events and so on.

Being new to Kinesis i am just going with the documentation and i found there is connector framework where S3Emitter is used with configuration to pick the S3 location where data needs to be emitted.However can somebody please suggest me as how to maintain a folder structure to capture event date in a date wise folder ?

Answer 1

I found a way to solve this issue and have posted the answer here: https://github.com/awslabs/amazon-kinesis-connectors/issues/24

Here is the answer again:

It is easy to achieve with the following changes to the sample code:

In S3sample.properties:

createS3Bucket = true

In S3Emitter.java:

/* Add the required imports */

import java.text.SimpleDateFormat;
import java.util.Calendar;

public class S3Emitter implements IEmitter {

    //create date_bucket variable

    protected final String date_bucket = new SimpleDateFormat("yyyy_MM_dd_HH").format(Calendar.getInstance().getTime());

    public S3Emitter(KinesisConnectorConfiguration configuration) {
        s3Bucket = configuration.S3_BUCKET + "/" + date_bucket;
    }
}

Hope this helps!