How to check User Data status while launching the instance in aws

Question

I am trying to launch aws instance with User Data. My User Data is a server installation process and i have to check whether the user data scripts are executed properly. Is there any option to check if the status of User data is completed ? I need to know the status since from that launched instance i am taking another image. As off now, i explicitly used time.sleep(90) for my process completion.

Note: I am using Boto library.

Any solution on this would be greatly appreciated!

Answer 1

UPDATE

What I landed up doing was creating a marker file at the end of the user-data run. I had the node controller spawn one ssh session per ec2 node and run a simple busy-wait loop as a command on the other end, so it only returns when the file is created. I then just wait() for all the ssh sessions to exit or until the wait timeout occurs.

It's ugly, but it works. It's very frustrating that EC2 doesn't provide better facilities for signalling status from within instances.

Labels

One possible approach is to have the instance's user-data script add an additional label to the instance when it completes. You can poll the instance with update or do a describe-instances with a filter that includes only nodes with the tag you use to specify that user data has been updated.

This requires that you include a limited API key and secret in your user-data scripts when you send them. Don't use your regular api key and secret, make one with very limited IAM rights. Additionally the user-data script will probably want to delete its self when it's done.

SNS / SQS

I've also considered using the Simple Notification Service and/or SQS for this, but it seems like overkill.

Like setting tags it requires that the instance have its own EC2 credentials.

SNS is push-only, so you have to have an endpoint reachable by EC2. That's a pain. SQS is pull, but doesn't have message routing, so you need one queue per set of nodes you're bringing up. You have to pass the unique queue name into the instance or have the instance use EC2 credentials to query it from a tag, then have the instance use that particular queue.

So, yeah, a pain.

Console

Getting console output won't work, EC2 stops updating it shortly after the instance transitions to the 'running' state.

There doesn't appear to be any way, instance- or client-side, to force an update.

Marker file

When the cloud-init script finishes it can touch a marker file somewhere shell-accessible to the normal user. This is a bit annoying, as it requires ssh'ing into every node and then polling for the creation of the marker file. The pain of polling can be somewhat reduced by use of a loop like:

while ! test -e 'cloud-init-complete'
do
    inotifywait -qq -t 2 -e create -e moved_to . ||true
done

after the installation of the inotify-tools package. If you don't burn inotify-tools into your AMIs you'll want to replace inotifywait with a simple sleep and accept the extra latency, or do:

while ! test -e 'cloud-init-complete'
do
    if test -x /usr/bin/inotifywait; then
        inotifywait -qq -t 2 -e create -e moved_to . ||true
    else
        sleep 2
    fi
done

This still requires an ssh connection to each server, though, and that's a pain to monitor and poll.

Something smarter?

My dream solution is being able to send an additional request to the EC2 metadata service to set a special instance tag or custom extra status field.