Feature/tuples as column names

Description

This MR contains two commits, we have to decide which one we want to follow:

• one with partial implementation of column names tuplization: ('FIELD NAME', slot, link, position)
• second with extension of the tuple including block id and sub block: ('FIELD NAME', 'BLOCK ID', 'SUB BLOCK', slot, link, position)

Since the performance drop is not significant (and anyways it is an improvement over the f-strings), I'd go for the second option as this would greatly simplify the analysis. The end goal for analysis would be to convert the pandas.DataFrame returned by unpacker to pandas.MultiIndex and then it can easily be sliced or cross-sectioned to select required subset of data (e.g. select all VFATs, or all VFATs from AMC N GEB M etc.)

Note: the documentation will require an update. I do not include it here for two reasons:

• First we need to decide on the tuple complexity
• There will be separate MR on the documentation build

Related Issue

Closes #11 (closed)

How Has This Been Tested?

Tested by unpacking a test dataset. Performance result on MBP 13" Fall 2014:

python -m timeit -n 1 "from gdh import unpacker; unpacker.run('./doc/notebooks/test_data/test_sdram.dat','sdram')"
1 loop, best of 5: 3.39 sec per loop

For comparison, previous result was 3.9s

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

gdh/formats/amc.py

@@ -6,67 +6,68 @@ class AMC(GenericBlock):
     def __init__(self):
         super().__init__()
         self.slot = 0
+        self.block_id = 'AMC'
         self.geb = GEB()
 
     def setup(self):
         # first 72 bits of 192-long header
-        self.header_1_fields = {'AMC:$:H:PAD'      :'u4',
-                                'AMC:$:H:AMC N'    :'u4',
-                                'AMC:$:H:L1A ID'   :'u24',
-                                'AMC:$:H:BX N'     :'u12',
-                                'AMC:$:H:DATA LGTH':'u20'}
+        self.header_1_fields = {'PAD'      :'u4',
+                                'AMC N'    :'u4',
+                                'L1A ID'   :'u24',
+                                'BX N'     :'u12',
+                                'DATA LGTH':'u20'}
 
-        self.header_2_start_fields = {'AMC:$:H:FV'      :'u4',
-                                      'AMC:$:H:RUN TYPE':'u4'}
+        self.header_2_start_fields = {'FV'      :'u4',
+                                      'RUN TYPE':'u4'}
     
         # 24 bits optional run parameters
-        self.run_parameters = ({'AMC:$:RP:PAD'          :'u24'},#Transition
-                               {'AMC:$:RP:PAD'          :'u11', #Physics
-                                'AMC:$:RP:PULSE STRETCH':'u3',
-                                'AMC:$:RP:LATENCY'      :'u10'},
-                               {'AMC:$:RP:PAD'          :'u1',  #Latency
-                                'AMC:$:RP:EXT TRG'      :'u1',
-                                'AMC:$:RP:IS C'         :'u1',
-                                'AMC:$:RP:CAL DAC'      :'u8',
-                                'AMC:$:RP:PULSE STRETCH':'u3',
-                                'AMC:$:RP:LATENCY'      :'u10'},
-                               {'AMC:$:RP:PAD'          :'u1',  #Threshold
-                                'AMC:$:RP:SE IC'        :'u2',
-                                'AMC:$:RP:THR ARM DAC'  :'u8',
-                                'AMC:$:RP:PULSE STRETCH':'u3',
-                                'AMC:$:RP:THR ZCC DAC'  :'u10'},
-                               {'AMC:$:RP:PAD'          :'u2',  #S-curve
-                                'AMC:$:RP:IS C'         :'u1',
-                                'AMC:$:RP:CAL DAC'      :'u8',
-                                'AMC:$:RP:PULSE STRETCH':'u3',
-                                'AMC:$:RP:THR ARM DAC'  :'u10'})
+        self.run_parameters = ({'RP:PAD'          :'u24'},#Transition
+                               {'RP:PAD'          :'u11', #Physics
+                                'RP:PULSE STRETCH':'u3',
+                                'RP:LATENCY'      :'u10'},
+                               {'RP:PAD'          :'u1',  #Latency
+                                'RP:EXT TRG'      :'u1',
+                                'RP:IS C'         :'u1',
+                                'RP:CAL DAC'      :'u8',
+                                'RP:PULSE STRETCH':'u3',
+                                'RP:LATENCY'      :'u10'},
+                               {'RP:PAD'          :'u1',  #Threshold
+                                'RP:SE IC'        :'u2',
+                                'RP:THR ARM DAC'  :'u8',
+                                'RP:PULSE STRETCH':'u3',
+                                'RP:THR ZCC DAC'  :'u10'},
+                               {'RP:PAD'          :'u2',  #S-curve
+                                'RP:IS C'         :'u1',
+                                'RP:CAL DAC'      :'u8',
+                                'RP:PULSE STRETCH':'u3',
+                                'RP:THR ARM DAC'  :'u10'})
     
         # last 96 bits of header
-        self.header_2_end_fields = {'AMC:$:H:OR N'    :'u16',
-                                    'AMC:$:H:BOARD ID':'u16'}
+        self.header_2_end_fields = {'OR N'    :'u16',
+                                    'BOARD ID':'u16'}
 
-        self.header_3_fields = {'AMC:$:H:DAV LIST'    :'u24',
-                                'AMC:$:H:BUF STATUS'  :'u24',
-                                'AMC:$:H:DAV CNT'     :'u5',
-                                'AMC:$:H:PAYLOAD VAR' :'u3',
-                                'AMC:$:H:PAYLOAD TYPE':'u4',
-                                'AMC:$:H:TTS'         :'u4'}
+        self.header_3_fields = {'DAV LIST'    :'u24',
+                                'BUF STATUS'  :'u24',
+                                'DAV CNT'     :'u5',
+                                'PAYLOAD VAR' :'u3',
+                                'PAYLOAD TYPE':'u4',
+                                'TTS'         :'u4'}
     
         # 128 bit trailer
-        self.trailer_1_fields = {'AMC:$:T:LINK TO'  :'u24',
-                                 'AMC:$:T:OOS'      :'u1',
-                                 'AMC:$:T:PAD1'     :'u31',
-                                 'AMC:$:T:BP'       :'u1',
-                                 'AMC:$:T:ML'       :'u1',
-                                 'AMC:$:T:CL'       :'u1',
-                                 'AMC:$:T:DR'       :'u1',
-                                 'AMC:$:T:BCL'      :'u1',
-                                 'AMC:$:T:PAD2'     :'u3'}
+        self.trailer_1_fields = {'LINK TO'  :'u24',
+                                 'OOS'      :'u1',
+                                 'PAD1'     :'u31',
+                                 'BP'       :'u1',
+                                 'ML'       :'u1',
+                                 'CL'       :'u1',
+                                 'DR'       :'u1',
+                                 'BCL'      :'u1',
+                                 'PAD2'     :'u3'}
 
-        self.trailer_2_fields = {'AMC:$:T:CRC32'    :'u32',
-                                 'AMC:$:T:L1A ID'   :'u8',
-                                 'AMC:$:T:PAD3'     :'u4',
-                                 'AMC:$:T:DATA LGTH':'u20'}
+        self.trailer_2_fields = {'CRC32'    :'u32',
+                                 'L1A ID'   :'u8',
+                                 'PAD3'     :'u4',
+                                 'DATA LGTH':'u20'}
 
     def assemble_header_2(self, rp):
         tmp = {}
@@ -89,27 +90,27 @@ class AMC(GenericBlock):
         self.trailer_2_compiled = bs.compile(''.join(list(self.trailer_2_fields.values())), list(self.trailer_2_fields.keys()))
 
     def unpack(self, stream):
-        event = {f'{k[:4]}{self.slot}{k[5:]}': v 
+        event = {self.update_key(k, 'HEADER'): v 
             for k, v in self.unpack_word(stream, self.header_1_compiled).items()}
         temp_dict = self.unpack_word(stream, self.header_2_start_compiled)
         stream.pos = stream.pos - 1
-        run_type = temp_dict['AMC:$:H:RUN TYPE']
+        run_type = temp_dict['RUN TYPE']
         #self.logger.debug("Run type: %d" %run_type)
-        event.update({f'{k[:4]}{self.slot}{k[5:]}': v 
+        event.update({self.update_key(k, 'HEADER'): v 
             for k, v in self.unpack_word(stream, self.header_2_array_compiled[run_type]).items()})
-        event.update({f'{k[:4]}{self.slot}{k[5:]}': v 
+        event.update({self.update_key(k, 'HEADER'): v 
             for k, v in self.unpack_word(stream, self.header_3_compiled).items()})
         # Retrieve active optical links and retreieve payloads
-        dav_list = event[f'AMC:{self.slot}:H:DAV LIST']
+        dav_list = event[('DAV LIST', self.block_id, 'HEADER', self.slot, self.link, self.pos)]
         for link in range(24):
             if ((dav_list >> link) & 0x1):
                 self.geb.slot = self.slot
                 self.geb.link = link
                 event.update(self.geb.unpack(stream))
         #self.logger.debug("all GEBs payload reading done. Stream position %d" %(stream.pos))
-        event.update({f'{k[:4]}{self.slot}{k[5:]}': v 
+        event.update({self.update_key(k, 'TRAILER'): v 
             for k, v in self.unpack_word(stream, self.trailer_1_compiled).items()})
-        event.update({f'{k[:4]}{self.slot}{k[5:]}': v 
+        event.update({self.update_key(k, 'TRAILER'): v 
             for k, v in self.unpack_word(stream, self.trailer_2_compiled).items()})
         #self.logger.debug("AMC:%d payload reading done. Stream position %d" %(self.slot, stream.pos))
         return event