How to run containers sequentially as a Kubernetes job?

k-kawa picture k-kawa · Nov 21, 2016 · Viewed 24.3k times · Source

I'm trying to replace my legacy job scheduler with Kubernetes job and wondering how to write sequential jobs as a Kubernetes job.

First, I wrote the following script to execute job1 and job2 in the written order but it didn't work as I expected.

apiVersion: batch/v1
kind: Job
metadata:
  name: sequential
spec:
  activeDeadlineSeconds: 100
  template:
    metadata:
      name: sequential_jobs
    spec:
      containers:
      - name: job1
        image: image1
      - name: job2
        image: image2
      restartPolicy: Never

The job described above seems to run job1 and job2 in parallel. Is there any good way to run job1 and job2 in the written order?

Appended.

I recently found https://github.com/argoproj/argo very good for my usecase.

Answer

Bloodysock picture Bloodysock · Oct 23, 2017

After a few attempts, I did this and that solved the basic problem (similar to what the OP has posted). This configuration ensures that job-1 completes before job-2 begins. If job-1 fails, job-2 container is not started. I still need to work on the retries and failure handling, but the basics works. Hopefully, this will help others:

apiVersion: v1
kind: Pod
metadata:
  name: sequential-job
spec:
  initContainers:
  - name: job-1
    image: busybox
    # runs for 15 seconds; echoes job name and timestamp
    command: ['sh', '-c', 'for i in 1 2 3; do echo "job-1 `date`" && sleep 5s; done;']
  - name: job-2
    image: busybox
    # runs for 15 seconds; echoes job name and timestamp
    command: ['sh', '-c', 'for i in 1 2 3; do echo "job-2 `date`" && sleep 5s; done;']
  # I don't really need the 'containers', but syntax requires 
  # it so, I'm using it as a place where I can report the 
  # completion status
  containers:
  - name: job-done
    image: busybox
    command: ['sh', '-c', 'echo "job-1 and job-2 completed"']
  restartPolicy: Never

Update

The same configuration as above also works inside a Job spec:

apiVersion: batch/v1
kind: Job
metadata:
  name: sequential-jobs
spec:
  template:
    metadata:
      name: sequential-job
    spec:
      initContainers:
      - name: job-1
        image: busybox
        command: ['sh', '-c', 'for i in 1 2 3; do echo "job-1 `date`" && sleep 5s; done;']
      - name: job-2
        image: busybox
        command: ['sh', '-c', 'for i in 1 2 3; do echo "job-2 `date`" && sleep 5s; done;']
      containers:
      - name: job-done
        image: busybox
        command: ['sh', '-c', 'echo "job-1 and job-2 completed"']
      restartPolicy: Never