1
2 r"""
3
4 ==========
5 hdf5pickle
6 ==========
7
8 :author: Pauli Virtanen <pav@iki.fi>
9
10
11 Create easily interoperable representations of Python objects in HDF5_
12 files. The aim of this module is to provide both
13
14 (1) convenient Python object persistence
15 (2) compatibility with non-Python applications
16
17 Point 2 is useful, for example, if results from numerical
18 calculations should be easily transferable for example to a non-Python
19 visualization program, such as Octave_. Having a serialized object
20 format that is directly readable saves some hassle in writing custom
21 data dumping routines for each object.
22
23 Of course, if your data does not fit into memory, you still need to
24 use full features of PyTables_. But, you can still use hdf5pickle for
25 other parts of the data.
26
27 This module implements `dump` and `load` methods analogous to those in
28 Python's pickle module. The programming interface corresponds to
29 pickle protocol 2, although the data is not serialized but saved in
30 HDF5 files. Additional methods, `dump_many` and `load_many`, are
31 provided for loading multiple objects at once, to preserve references.
32
33
34 :warning:
35 Although this module passes all relevant pickle unit tests from
36 Python2.4 plus additional tests, it is still in early stages of
37 development.
38
39
40 .. _HDF5: http://hdf.ncsa.uiuc.edu/HDF5/
41 .. _Octave: http://www.octave.org/
42 .. _PyTables: http://www.pytables.org/
43
44
45 Data format
46 ===========
47
48 The layout of a python object saved to a HDF5 node is described below.
49 The notation is roughly::
50
51 type-of-hdf5-node [(array shape), array type)] = what's in it
52 .attribute-of-node = what's in it
53 child-node
54
55 The structure of a node corresponding to a Python object varies,
56 depending on the type of the Python object.
57
58 * Basic types (``None``, ``bool``, ``int``, ``float``, ``complex``)::
59
60 array [(1,), int/float] = NONE/BOOL/INT/FLOAT/COMPLEX
61 .pickletype = PICKLE_TYPE
62
63 * Basic stream types (``long``, ``str``, ``unicode``).
64 Longs and unicodes are converted to strings (`pickle.encode_long`, utf-8),
65 and strings are converted to arrays of unsigned 8-bit integers::
66
67 array [(n,), uint8] = DATA
68 .pickletype = LONG/STR/UNICODE
69 .empty = 1 #if len(DATA) == 0
70
71 :bug:
72 At present strings are not stored as HDF5 strings,
73 as PyTables appears to chop them off at '\\x00' characters.
74
75 * dicts::
76
77 group
78 .pickletype = DICT
79
80 #for KEY, VALUE in DICT:
81 #if KEY is a string and a valid python variable name
82 KEY = node for VALUE
83 #else
84 SURROGATE = node for VALUE
85 __/SURROGATE = node for KEY
86 #end if
87 #end for
88
89 * instances::
90
91 group
92 .pickletype = INST/REDUCE
93
94 #if through __reduce__ / new-style class
95 .has_reduce_content = 1 if state present
96 __/args = arguments for class.__new__ or func
97 __/func = creation func
98 __/cls = creation class
99 #else
100 __/args = arguments for class.__init__
101 __/cls = creation class
102 #endif
103
104 #if state is dict
105 insert entries of dict here as in dict
106 #else
107 __/content = node for content
108 #endif
109
110 * globals (classes, etc)::
111
112 array [as for strings] = GLOBAL/EXT4 data locator, as in pickle
113 .pickletype = GLOBAL/EXT4
114
115 * reference to an object elsewhere::
116
117 group
118 .pickletype = REF
119 .target = abs. path to the referred object in this file
120
121 """
122
123 from base import *
124
125 __docformat__ = "restructuredtext en"
126 __version__ = "0.2"
127