magni.utils.multiprocessing._util module

Module providing the public class of the magni.utils.multiprocessing subpackage.

class magni.utils.multiprocessing._util.File(*args, **kwargs)[source]

Bases: object

Control pytables access to hdf5 files when using multiprocessing.

File retains the interface of tables.open_file and should only be used in ‘with’ statements (see Examples).

Parameters:
  • args (tuple) – The arguments that are passed to ‘tables.open_file’.
  • kwargs (dict) – The keyword arguments that are passed to ‘tables.open_file’.

See also

tables.open_file
The wrapped function.

Notes

Internally the module uses a global lock which is shared amongst all files. This solution is simple and does not entail significant overhead. However, the wait time introduced when using multiple files at the same time can be significant.

Examples

The class is used in the following way:

>>> from magni.utils.multiprocessing._util import File
>>> with File('database.hdf5', 'a') as f:
...     pass # execute something involving the opened file
__enter__()[source]

Acquire the global lock before opening and returning the file.

Returns:file (tables.File) – The file specified in the call to __init__.
__exit__(type, value, traceback)[source]

Release the global lock after closing the file.

Parameters:
  • type (type) – The type of the exception raised, if any.
  • value (Exception) – The exception rasied, if any.
  • traceback (traceback) – The traceback of the exception raised, if any.