Starting multiple upstart instances automatically

Question

We use PHP gearman workers to run various tasks in parallel. Everything works just fine, and I have silly little shell script to spin them up when I want them. Being a programmer (and therefore lazy), I wanted to see if I could spin these up via an upstart script.

I figured out how to use the instance stanza, so I could start them with an instance number:

description "Async insert workers"
author      "Mike Grunder"

env SCRIPT_PATH="/path/to/my/script"

instance $N

script
    php $SCRIPT_PATH/worker.php
end script

And this works great, to start them like so:

sudo start async-worker N=1
sudo start async-worker N=2

The way I want to use these workers is to spin up some number of them (maybe one per core, etc), and I would like to do this on startup. To be clear, I don't need the upstart script to detect the number of cores. I'm happy to just say "do 8 instances", but that's why I want multiple running. Is there a way for me to use the "start on" clause in an upstart script to do this automatically?

For example, start instance 1, 2, 3, 4? Then have them exit on shutdown properly?

I suppose I could hook this into an init.d script, but I was wondering if upstart can handle something like this, or if anyone has figured out this issue.

Cheers guys!

Answer 1

What you need is a bootstrap task that runs on startup and iterates over all your worker jobs, starting each one.

#/etc/init/async-workers-all.conf

start on runlevel [2345]

task

env NUM_WORKERS=8

script
  for i in `seq 1 $NUM_WORKERS`
  do
    start async-worker N=$i
  done
end script

The key is to make this a task, which tells upstart to let the task run to completion before emitting any events for it. See http://upstart.ubuntu.com/cookbook/#task and http://upstart.ubuntu.com/cookbook/#instance