Gzip with all cores

linux bash gzip
User1 picture User1 · Dec 3, 2010 · Viewed 49.1k times · Source

I have a set of servers filled each with a bunch of files that can be gzipped. The servers all have different numbers of cores. How can I write a bash script to launch a gzip for each core and make sure the gzips are not zipping the same file?

Answer

David Yaw picture David Yaw · Dec 3, 2010

There is an implementation of gzip that is multithreaded, pigz. Since it is compressing one file on multiple threads, it should be able to read from disk more efficiently, compared to compressing multiple files at once.

Related questions

How to gzip all files in all sub-directories into one compressed file in bash

Possible Duplicate: gzipping up a set of directories and creating a tar compressed file This post describes how to gzip each file individually within a directory structure. However, I need to do something slightly different. I need to produce one …

Read More
gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now

I have a bash script that creates a tar.gz and encrypts then sends to drive. However I cannot open the .tar.gz afterwards. Here is my process... Bash Script that encrypts. #!/bin/sh # tar the automysqlbackup directory tar -zcf "…

Read More
GZip every file separately

How can we GZip every file separately? I don't want to have all of the files in a big tar.

Read More