szip

Introduction:

SZIP is an implementation of the extended-Rice lossless compression algorithm, which has been integrated into the HDF4 and HDF5 hierarchical data file standard.

The Consultative Committee on Space Data Systems (CCSDS) has adopted the extended-Rice algorithm for international standards for space applications. SZIP is reported to provide fast and effective compression, specifically for the EOS data generated by the NASA Earth Observatory System. SZIP was developed at the University of New Mexico (UNM) and integrated with HDF by UNM researchers and developers.

The project web site is https://www.hdfgroup.org/doc_resource/SZIP/.

Reference:

  • Pen-Shu Yeh, Wei Xia-Serafino, Lowell Miles, Ben Kobler, Daniel Menasce,
    Implementation of CCSDS Lossless Data Compression in HDF,
    Earth Science Technology Conference–2002,
    11–13 June 2002, Pasadena, California.

Availability:

Version 2.1 of SZIP is installed on Cascades, BlueRidge, DragonsTooth, HokieOne, HokieSpeed, and NewRiver. To check the latest information on available versions on any ARC cluster, type "module spider szip".

Usage:

SZIP requires that several modules be loaded before it can be run. Assuming that the HDF5 library is to be used as well, one version of the appropriate commands for use on NewRiver is:

module purge
module load gcc/5.2.0
module load hdf5/1.8.16
module load szip/2.1

Examples:

In the following batch file, an example user program is compiled and executed, which creates a file using the HDF5 standard, with SZIP compression.

#! /bin/bash
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=1
#PBS -W group_list=newriver
#PBS -q open_q
#PBS -j oe

cd $PBS_O_WORKDIR

module load gcc/5.2.0
module load hdf5/1.8.16
module load szip/2.1

h5cc -c szip_test.c 
h5cc -o szip_test szip_test.o
./szip_test

A complete set of files to carry out a similar process are available in a tar file