Splitting command line args with GNU parallel

bash file-processing gnu-parallel
drhodes picture drhodes · Jun 6, 2011 · Viewed 13k times · Source

Using GNU parallel: http://www.gnu.org/software/parallel/

I have a program that takes two arguments, e.g.

$ ./prog file1 file2
$ ./prog file2 file3
...
$ ./prog file23456 file23457

I'm using a script that generates the file name pairs, however this poses a problem because the result of the script is a single string - not a pair. like:

$ ./prog "file1 file2"

GNU parallel seems to have a slew of tricks up its sleeves, I wonder if there's one for splitting text around separators:

$ generate_file_pairs | parallel ./prog ?  
  # where ? is text under consideration, like "file1 file2"

The easy work around is to split the args manually in prog, but I'd like to know if it's possible in GNU parallel.

Answer

Ole Tange picture Ole Tange · Jun 6, 2011

You are probably looking for --colsep.

generate_file_pairs | parallel --colsep ' ' ./prog {1} {2}

Read man parallel for more. And watch the intro video if you have not already done so http://www.youtube.com/watch?v=OpaiGYxkSuQ

Related questions

Read words in a specific line in a text file using shell script

In my Bash shell script, I would like to read a specific line from a file; that is delimited by : and assign each section to a variable for processing later. For example I want to read the words found on …

Read More
How to concatenate string variables in Bash

In PHP, strings are concatenated together as follows: $foo = "Hello"; $foo .= " World"; Here, $foo becomes "Hello World". How is this accomplished in Bash?

Read More
How can I check if a directory exists in a Bash shell script?

What command can be used to check if a directory exists or not, within a Bash shell script?

Read More