Running several scripts in parallel bash script

bash parallel-processing
Bdar picture Bdar · Mar 26, 2013 · Viewed 62.7k times · Source

I have a bash script that contains other scripts inside that are run in series. However, it takes a decent amount of time to run them all. Is there a way to run these scripts in parallel to improve overall perfomance? They are independent of each other.

It looks similar to:

#!/bin/bash

#some code here
cppcheck.sh
churn.sh
run.sh

Update:

**git log --pretty=format: --numstat | perl -ane'$c{$F[2]} += abs($F[0]+$F[1]) 
if $F[2];END {print "$_\t$c{$_}\n" for sort keys %c}' > ${OUTPUT_DIR}/churn.txt**
sed -i -e '/deps/d;/build/d;/translations/d;/tests/d' -e 30q ${OUTPUT_DIR}/churn.txt
sort -r -n -t$'\t' -k2 ${OUTPUT_DIR}/churn.txt -o ${OUTPUT_DIR}/churn.txt
echo "set term canvas size 1200, 800; set output '${OUTPUT_DIR}/output.html'; 
unset key; set bmargin at screen 0.4; set xtics rotate by -90 scale 0,0; 
set ylabel 'Number of lines changed (total)'; set title 'Files with high churn 
level';set boxwidth 0.7; set style fill solid; set noborder; 
plot '${OUTPUT_DIR}/churn.txt' using 2:xticlabels(1) with boxes" | gnuplot
echo "finished running churn.sh!"

This is the code inside churn.sh. The bold command takes 40 or so secs to implement. If in a main script I put ampersand after churn.sh &, it throws an error saying sed can't read churn.txt file (since it's not created yet). It seems that it doesn't wait till the output is saved in a file. I inserted wait after that command but it doesn't help.

Answer

Anirudh Ramanathan picture Anirudh Ramanathan · Mar 26, 2013

Using the & to run it in the background will do the trick

cppcheck.sh &
churn.sh &
run.sh &

wait
echo "All 3 complete"

It will fork off a new process for each of them.

The bash wait will also come in handy as stated in the comments, if you have something to be run on the parent script, after these three finish.

Without an argument it will wait for all child processes to complete, and then resume execution of the parent script.

The issues you are facing seem to be directly related to this. Variables set are only visible to the sub-shell in which they are defined. So, if you have OUT_DIR specified in the parent script, it won't be visible to the child script when it forks off. The right thing to do in this case would be to export the variable as an environment variable.

Related questions

How do you run multiple programs in parallel from a bash script?

I am trying to write a .sh file that runs many programs simultaneously I tried this prog1 prog2 But that runs prog1 then waits until prog1 ends and then starts prog2... So how can I run them in parallel?

Read More
Running shell script in parallel

I have a shell script which shuffles a large text file (6 million rows and 6 columns) sorts the file based the first column outputs 1000 files So the pseudocode looks like this file1.sh #!/bin/bash for i in $(seq 1 1000) do Generating …

Read More
Running programs in parallel using xargs

I currently have the current script. #!/bin/bash # script.sh for i in {0..99}; do script-to-run.sh input/ output/ $i done I wish to run it in parallel using xargs. I have tried script.sh | xargs -P8 But doing the above …

Read More