Welcome to the h5py_wrapper

This is the documentation of the h5py_wrapper.

Introduction

h5py_wrapper

Wrapper to conveniently store arbitrarily nested python dictionaries to hdf5 files.

The dictionaries are stored in an hdf5 file by creating groups for every level and a dataset for the value in the lowest level.

Functions

  • save : store nested dictionary in hdf5 file
  • load : load nested dictionary from hdf5 file

API reference

API reference

h5py_wrapper.save(filename, d, write_mode='a', overwrite_dataset=False, resize=False, path=None, dict_label='', compression=None)[source]

Save a dictionary to an hdf5 file.

filename : string
The file name of the hdf5 file.
d : dict
The dictionary to be stored.
write_mode : {‘a’, ‘w’}, optional
Analog to normal file handling in python. Defaults to ‘a’.
overwrite_dataset : bool, optional
Whether datasets should be overwritten if already existing. Defaults to False.
resize : bool, optional
If True, the hdf5 file is resized after writing all data, may reduce file size. Uses h5repack (see https://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Repack). Caution: slows down writing. Defaults to False.
path : string, optional
If not empty, the dictionary is stored under the given path in the hdf5 file, with levels separated by ‘/’. For instance, path=’test/trial/spiketrains’. Defaults to None.
compression : {‘gzip’, ‘szip’,’lzf’, 0,…,10}, optional
Compression strategy to reduce file size. An integer >0, <=10 leads to usage of gzip,indicating the level of compression. ‘gzip’ is recommended. See http://docs.h5py.org/en/latest/high/dataset.html for details. Caution: This slows down writing and loading of data. Attention: Will be ignored for scalar data.

None

>>> d = {}
>>> d['a'] = {'a1': [1, 2, 3], 'a2': 4., 'a3': {'a31': 'Test'}}
>>> d['b'] = 'string'
>>> import h5py_wrapper as h5w
>>> h5w.save('example.h5', d)
h5py_wrapper.load(filename, path='', lazy=False)[source]

Loads a dictionary from an hdf5 file.

filename : string
The file name of the hdf5 file.
path : string, optional
If not empty, specifies a path to access deeper levels in the hdf5 file.
lazy : boolean, optional
If True, only keys from all levels of the dictionary are loaded with values. Defaults to False.
dictionary : dict
Dictionary from the hdf5 file.
>>> d = {}
>>> d['a'] = {'a1': [1, 2, 3], 'a2': 4., 'a3': {'a31': 'Test'}}
>>> d['b'] = 'string'
>>> import h5py_wrapper as h5w
>>> h5w.save('example_load.h5', d, overwrite_dataset=True)
>>> h5w.load('example_load.h5')
{u'a': {u'a1': array([1, 2, 3]), u'a3': {u'a31': 'Test'}, u'a2': 4.0}, u'b': 'string'}

Supported data types

Supported data types

The wrapper stores the original data types of values of the dictionary in the produced hdf5 file. The following data types are supported:

  • float

  • int

  • str

  • tuple

  • numpy.array

  • numpy.int64 and numpy.float64

  • list

  • bool

  • quantities.Quantity (see https://pypi.python.org/pypi/quantities)

  • Lists, tuples and numpy.arrays up to arbitrary depths if all dimensions are uniform, e.g.

    l = numpy.ones((3,3,3))

  • Lists and tuples are required to contain the contain equal data types across one dimension. For instance, this raises an error

    l = [[1,2], 'a']

  • Lists, tuples and numpy.arrays with unequal dimensions with maximal depth 1.

    l = [[1,2], [1]]

Conversion from old releases

Conversion from old release versions

Release version 1.0.1 introduced some changes to the file format (see Release notes).

Conversion script to convert files from a previous
release version to the current version.

Usage: convert_h5file [-h|--help] [<files>...] [--save-backup] [-v|--verbose] [--release=<version>]

Options:
    <files>  List of files to be converted, can be a file pattern
    --save-backup  save backup version of file in old file format [default: False].
    --release=<version>   release version used to create the file to be converted [default: 0.0.1]
    -v, --verbose  print informative output to screen
    -h, --help     print this text

Release Notes

Release notes

1.1.0

https://github.com/INM-6/h5py_wrapper/releases/tag/1.1.0

This release adds Python3 support to the wrapper. This transition required major changes to the code base, particularly the handling of strings in the wrapper was revised.

Furthermore, we added:

  • support for complex numbers
  • support for numpy 64bit data types.
  • improved the error handling and warnings.

This release contains 48 commits, 11 closed issues and 18 merged pull-requests.

API changes

The deprecated functions add_to_h5 and load_h5 were removed.

Contributors

  • Maximilian Schmidt
  • Jakob Jordan
  • Julia Sprenger

1.0.1

https://github.com/INM-6/h5py_wrapper/releases/tag/1.0.1

API changes

The old function names load_h5 and add_to_h5 have been replaced by load and save. The old function names are still available, but will raise a DeprecationWarning and be removed in one of the next releases. New features and functionality

The wrapper now explicitly stores the types of values of the dictionary in the hdf5 file and is able to retrieve the original value types. A list of supported value types is provided in the documentation.

This has the effect that old files created with release version 0.0.1 cannot be loaded with 1.0.1. To remedy this, we provide a conversion script that converts old files to the new format. The user can provide

  • single file names,
  • lists of file names
  • files containing file names
  • input from stdin.

Contributors

  • Maximilian Schmidt
  • Jakob Jordan

v0.0.1

https://github.com/INM-6/h5py_wrapper/releases/tag/v0.0.1

This release contains 16 commits since the migration to github. The wrapper provides the two basic functions wrapper.add_to_h5 and wrapper.load_h5. Furthermore, the user can load h5 files (load_h5_old) that were created with a deprecated version of the wrapper and transform these files into the current format (transform_h5).

Contributors

(in alphabetical order)

  • Hannah Bos
  • Michael Denker
  • Jakob Jordan
  • Maximilian Schmidt
  • Jannis Schuecker

Indices and tables