Package hdf5pickle
[hide private]
[frames] | no frames]

Source Code for Package hdf5pickle

  1  # Copyright (c) 2006 Pauli Virtanen <pav@iki.fi> 
  2  r""" 
  3   
  4  ========== 
  5  hdf5pickle 
  6  ========== 
  7   
  8  :author: Pauli Virtanen <pav@iki.fi> 
  9   
 10   
 11  Create easily interoperable representations of Python objects in HDF5_ 
 12  files. The aim of this module is to provide both 
 13   
 14  (1) convenient Python object persistence 
 15  (2) compatibility with non-Python applications 
 16   
 17  Point 2 is useful, for example, if results from numerical 
 18  calculations should be easily transferable for example to a non-Python 
 19  visualization program, such as Octave_. Having a serialized object 
 20  format that is directly readable saves some hassle in writing custom 
 21  data dumping routines for each object. 
 22   
 23  Of course, if your data does not fit into memory, you still need to 
 24  use full features of PyTables_. But, you can still use hdf5pickle for 
 25  other parts of the data. 
 26   
 27  This module implements `dump` and `load` methods analogous to those in 
 28  Python's pickle module. The programming interface corresponds to 
 29  pickle protocol 2, although the data is not serialized but saved in 
 30  HDF5 files. Additional methods, `dump_many` and `load_many`, are 
 31  provided for loading multiple objects at once, to preserve references. 
 32   
 33   
 34  :warning: 
 35      Although this module passes all relevant pickle unit tests from 
 36      Python2.4 plus additional tests, it is still in early stages of 
 37      development. 
 38   
 39   
 40  .. _HDF5: http://hdf.ncsa.uiuc.edu/HDF5/ 
 41  .. _Octave: http://www.octave.org/ 
 42  .. _PyTables: http://www.pytables.org/ 
 43   
 44   
 45  Data format 
 46  =========== 
 47   
 48  The layout of a python object saved to a HDF5 node is described below. 
 49  The notation is roughly:: 
 50   
 51      type-of-hdf5-node [(array shape), array type)] = what's in it 
 52         .attribute-of-node = what's in it 
 53         child-node 
 54   
 55  The structure of a node corresponding to a Python object varies, 
 56  depending on the type of the Python object. 
 57   
 58  * Basic types (``None``, ``bool``, ``int``, ``float``, ``complex``):: 
 59   
 60      array [(1,), int/float] = NONE/BOOL/INT/FLOAT/COMPLEX 
 61          .pickletype         = PICKLE_TYPE 
 62   
 63  * Basic stream types (``long``, ``str``, ``unicode``). 
 64    Longs and unicodes are converted to strings (`pickle.encode_long`, utf-8), 
 65    and strings are converted to arrays of unsigned 8-bit integers::  
 66   
 67      array [(n,), uint8] = DATA 
 68          .pickletype    = LONG/STR/UNICODE 
 69          .empty     = 1 #if len(DATA) == 0 
 70       
 71    :bug: 
 72       At present strings are not stored as HDF5 strings, 
 73       as PyTables appears to chop them off at '\\x00' characters. 
 74   
 75  * dicts:: 
 76   
 77      group 
 78          .pickletype  = DICT 
 79   
 80          #for KEY, VALUE in DICT: 
 81           #if KEY is a string and a valid python variable name 
 82          KEY          = node for VALUE 
 83           #else 
 84          SURROGATE    = node for VALUE 
 85          __/SURROGATE = node for KEY 
 86           #end if 
 87          #end for 
 88       
 89  * instances:: 
 90   
 91      group 
 92          .pickletype         = INST/REDUCE 
 93       
 94          #if through __reduce__ / new-style class 
 95          .has_reduce_content = 1 if state present 
 96          __/args             = arguments for class.__new__ or func 
 97          __/func             = creation func 
 98          __/cls              = creation class 
 99          #else 
100          __/args             = arguments for class.__init__ 
101          __/cls              = creation class 
102          #endif 
103       
104          #if state is dict 
105          insert entries of dict here as in dict 
106          #else 
107          __/content          = node for content 
108          #endif 
109   
110  * globals (classes, etc):: 
111   
112      array [as for strings] = GLOBAL/EXT4 data locator, as in pickle 
113          .pickletype        = GLOBAL/EXT4 
114   
115  * reference to an object elsewhere:: 
116   
117      group 
118          .pickletype        = REF 
119          .target            = abs. path to the referred object in this file 
120   
121  """ 
122   
123  from base import * 
124   
125  __docformat__ = "restructuredtext en" 
126  __version__ = "0.2" 
127