GNU parallel output progress while output to file

bash parallel-processing gnu-parallel
Alvin Das picture Alvin Das · Sep 16, 2013 · Viewed 8.7k times · Source

I have a simple bash script to run:

cat full_path.csv | parallel --progress -j +0 'echo -n {},; pdfgrep -c [^_] {};' > path_count.csv

Parallel's progress indicator "--progress", writes into the file path_count.csv. I only want echo {} and pdfgrep {} to write to the file, while showing --progress output to screen.

If I do : 

cat full_path.csv | parallel --progress -j +0  'echo -n {},>>path_count.csv; pdfgrep -c [^_] {}>>path_count.csv;'

the file path_count is still garbled with progress.

Any help is appreciated. Thanks Alvin

Answer

Ole Tange picture Ole Tange · Sep 17, 2013

The behaviour you see is not what GNU Parallel is designed to do: --progress is normally sent to STDERR and not to STDOUT for exactly that reason:

$ seq 3 | bin/parallel --progress echo {} >/tmp/out

Computers / CPU cores / Max jobs to run
1:local / 8 / 3

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
local:0/3/100%/0.0s
$ cat /tmp/out
1
2
3

Has there been local modifications of GNU Parallel? Can you reproduce the issue on other systems?

PS: instead of 'echo -n' why to try: --tag

Related questions

"find" and "ls" with GNU parallel

I'm trying to use GNU parallel to post a lot of files to a web server. In my directory, I have some files: file1.xml file2.xml and I have a shell script that looks like this: #! /usr/bin/env …

Read More
parallel call multiple bash functions

I have read the example at http://www.gnu.org/software/parallel/man.html#example__calling_bash_functions however, is it possible to use gnu parallel to call 2 functions which do not have any variables you pass to them ? example …

Read More
Parallelizing a while loop with arrays read from a file in bash

I have a while loop in Bash handled like this: while IFS=$'\t' read -r -a line; do myprogram ${line[0]} ${line[1]} ${line[0]}_vs_${line[1]}.result; done < fileinput It reads from a file with this structure, for reference: foo …

Read More