Numcodecs is a Python package providing buffer compression and transformation codecs for use in data storage and communication applications. These include:
- Compression codecs, e.g., Zlib, BZ2, LZMA, ZFPY and Blosc.
- Pre-compression filters, e.g., Delta, Quantize, FixedScaleOffset, PackBits, Categorize.
- Integrity checks, e.g., CRC32, Adler32.
All codecs implement the same API, allowing codecs to be organized into pipelines in a variety of ways.
If you have a question, find a bug, would like to make a suggestion or contribute code, please raise an issue on GitHub.
Numcodecs depends on NumPy. It is generally best to install NumPy first using whatever method is most appropriate for you operating system and Python distribution.
Install from PyPI:
$ pip install numcodecs
Alternatively, install via conda:
$ conda install -c conda-forge numcodecs
Numcodecs includes a C extension providing integration with the Blosc library. Wheels are available for most platforms.
Installing a wheel or via conda will install a pre-compiled binary distribution. However, if you have a newer CPU that supports the AVX2 instruction set (e.g., Intel Haswell, Broadwell or Skylake) then installing via pip is preferable, because this will compile the Blosc library from source with optimisations for AVX2.
Note that if you compile the C extensions on a machine with AVX2 support you probably then cannot use the same binaries on a machine without AVX2. To disable compilation with AVX2 support regardless of the machine architecture:
$ export DISABLE_NUMCODECS_AVX2= $ pip install -v --no-cache-dir --no-binary numcodecs numcodecs
To work with Numcodecs source code in development, install from GitHub:
$ git clone --recursive https://github.com/zarr-developers/numcodecs.git $ cd numcodecs $ python setup.py install
To verify that Numcodecs has been fully installed (including the Blosc extension) run the test suite:
$ pip install nose $ python -m nose -v numcodecs
- Codec API
- Codec registry
- 32-bit checksums
- Codecs for variable-length objects
- Release notes
- Contributing to NumCodecs
The following people have contributed to the development of NumCodecs by contributing code, documentation, code reviews, comments and/or ideas:
- Francesc Alted
- Prakhar Goel
- Jerome Kelleher
- John Kirkham
- Alistair Miles
- Jeff Reback
- Trevor Manz
- Grzegorz Bokota
- Josh Moore
- Martin Durant
- Paul Branson
Numcodecs bundles the c-blosc library.
Development of this package is supported by the MRC Centre for Genomics and Global Health.