pigz

Introduction:

PIGZ is a program for the lossless compression or uncompression of data. The compression operation can be done in parallel. PIGZ uses the ZLIB and PTHREAD libraries. By default, it compresses files to the GZIP format, but can instead use the ZLIB format.

Web site:

http://zlib.net/pigz/

Reference:

Usage:

To check the latest information on available versions of PIGZ on any ARC cluster, type "module spider pigz". PIGZ requires that the appropriate modules be loaded before it can be run. One version of the appropriate commands for use on NewRiver is:

module purge
module load pigz

Examples:

In the following batch file, we compress and decompress a file, and compare its size to the original, and the required time using 1, 2 and 4 processors.

#! /bin/bash
#
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=4
#PBS -W group_list=cascades
#PBS -q open_q
#PBS -j oe

cd $PBS_O_WORKDIR

module purge
module load pigz

ls -l robinson_crusoe.txt

cp robinson_crusoe.txt robinson_crusoe2.txt
time pigz -p 1 robinson_crusoe2.txt

rm robinson_crusoe2.txt.gz
cp robinson_crusoe.txt robinson_crusoe2.txt
time pigz -p 2 robinson_crusoe2.txt

rm robinson_crusoe2.txt.gz
cp robinson_crusoe.txt robinson_crusoe2.txt
time pigz -p 4 robinson_crusoe2.txt

ls -l robinson_crusoe2.txt.gz
unpigz robinson_crusoe2.txt.gz

ls -l robinson_crusoe2.txt

A complete set of files to carry out a similar process are available in a tar file