How can I run a list of commands in parallel?

bash parallel-processing system systems-programming
monkeyking picture monkeyking · Jul 15, 2012 · Viewed 19.6k times · Source

I have a file containing command lines that I want to run. This file contains around 2,000 lines.

I have 8 cores available. Is it possible to parse the file and start 8 processes, then execute another one from the file whenever one of the programs finishes? I want this to continue until the end of file is reached.

Answer

Steve picture Steve · Jul 15, 2012

Use GNU parallel. It's an incredibly powerful tool and official packages exist for about 20 or so linux distros. What's that? You have an excuse as to why you can't use it? Here's a simple example showing how to run a list or file of commands in parallel:

Contents of jobs.txt:

sleep 1; echo "a"
sleep 3; echo "b"
sleep 2; echo "c"

Command:

time parallel :::: jobs.txt

Results:

a
c
b

real    0m3.332s
user    0m0.170s
sys     0m0.037s

Notes:

If you wish to keep the order the same as the input, pass the -k flag to GNU parallel.

If you have more than eight cores and only wish to process with eight cores, add -j 8 to the args list.

The man page is a good read, but if you haven't already read this tutorial I would highly recommend the time investment.

Related questions

How do you run multiple programs in parallel from a bash script?

I am trying to write a .sh file that runs many programs simultaneously I tried this prog1 prog2 But that runs prog1 then waits until prog1 ends and then starts prog2... So how can I run them in parallel?

Read More
Running shell script in parallel

I have a shell script which shuffles a large text file (6 million rows and 6 columns) sorts the file based the first column outputs 1000 files So the pseudocode looks like this file1.sh #!/bin/bash for i in $(seq 1 1000) do Generating …

Read More
Running programs in parallel using xargs

I currently have the current script. #!/bin/bash # script.sh for i in {0..99}; do script-to-run.sh input/ output/ $i done I wish to run it in parallel using xargs. I have tried script.sh | xargs -P8 But doing the above …

Read More