How to design a distributed job scheduler?

cron distributed cron-task system-design
coderz picture coderz · Nov 12, 2014 · Viewed 18.5k times · Source

I want to design a job scheduler cluster, which contains several hosts to do cron job scheduling. For example, a job which needs run every 5 minutes is submitted to the cluster, the cluster should point out which host to fire next run, making sure:

  1. Disaster tolerance: if not all of the hosts are down, the job should be fired successfully.
  2. Validity: only one host to fire next job run.

Due to disaster tolerance, job cannot bind to a specific host. One way is all the hosts polling a DB table(certainly with lock), this guaranteed only one host gets the next job run. Since it often locks table, is there any better design?

Answer

Stefan picture Stefan · Nov 12, 2014

Use the Quartz framework for that. It has a cron like syntax, can be clustered and only one of the hosts in the cluster will do one job at a time. If a host or job fails, another host will retry the pending job.

Related questions

How do I list all cron jobs for all users?

Is there a command or an existing script that will let me view all of a *NIX system's scheduled cron jobs at once? I'd like it to include all of the user crontabs, as well as /etc/crontab, and whatever's …

Read More
How do I schedule jobs in Jenkins?

I added a new job in Jenkins, which I want to schedule periodically. From Configure job, I am checking the "Build Periodically" checkbox and in the Schedule text field added the expression: 15 13 * * * But it does not run at the scheduled …

Read More
Running a cron every 30 seconds

Ok so I have a cron that I need to run every 30 seconds. Here is what I have: */30 * * * * /bin/bash -l -c 'cd /srv/last_song/releases/20120308133159 && script/rails runner -e production '\''Song.insert_latest'\''' It …

Read More