Can upstart expect/respawn be used on processes that fork more than twice?

johnjamesmiller picture johnjamesmiller · Aug 30, 2012 · Viewed 8k times · Source

I am using upstart to start/stop/automatically restart daemons. One of the daemons forks 4 times. The upstart cookbook states that it only supports forking twice. Is there a workaround?

How it fails

If I try to use expect daemon or expect fork, upstart uses the pid of the second fork. When I try to stop the job, nobody responds to upstarts SIGKILL signal and it hangs until you exhaust the pid space and loop back around. It gets worse if you add respawn. Upstart thinks the job died and immediately starts another one.

Bug acknowledged by upstream

A bug has been entered for upstart. The solutions presented are stick with the old sysvinit, rewrite your daemon, or wait for a re-write. RHEL is close to 2 years behind the latest upstart package, so by the time the rewrite is released and we get updated the wait will probably be 4 years. The daemon is written by a subcontractor of a subcontractor of a contractor so it will not be fixed any time soon either.

Answer

johnjamesmiller picture johnjamesmiller · Aug 31, 2012

I came up with an ugly hack to make this work. It works for my application on my system. YMMV.

  1. start the application in the pre-start section
  2. in the script section run a script that runs as long as the application runs. The pid of this script is what upstart will track.
  3. in the post-stop section kill the application

example

env DAEMON=/usr/bin/forky-application

pre-start script
    su -s /bin/sh -c "$DAEMON" joeuseraccount
end script

script 
    sleepWhileAppIsUp(){
        while pidof $1 >/dev/null; do
            sleep 1
        done
    }

    sleepWhileAppIsUp $DAEMON
end script

post-stop script
    if pidof $DAEMON;
    then
        kill `pidof $DAEMON`
        #pkill  $DAEMON # post-stop process (19300) terminated with status 1
    fi
end script

a similar approach could be taken with pid files.