External trigger scan finishes without setting off (FMC) BUSY signal

When performed an external trigger scan (python scan_ext_trigger.py), with the KC705+FMC+SCC + TLU for trigger (LEMO connector, simple handshake mode), development branch (33690f29); when finish, the BUSY signal is still present, therefore the TLU is sending triggers. This is not the expected behaviour as after the scan is finished, the BUSY signal should be down, stopping the TLU triggers (as is in handshake mode). Esther and myself have (provisionally) fixed this behaviour by applying a TLU reset after the scan stop, but we are not sure if this is the proper or better way to do that, so please take a look a let us know. You can find the path in our fork: sifca/bdaq53@874b93ab

assigned to @ydieter

As I remember the busy signal is a trigger veto to prevent the TLU to send new triggers and this is done on purpose like this (@ydieter)? We use the TLU as a simple run control and do not want the TLU to send triggers before all devices are ready. Thus we veto at the beginning and after the end of scan, the std. setting when not data taking is busy high.

added Discussion label

Coder of the TLU/Trigger module here: The TLU Busy signal is definitely not high in idle mode. See here: https://github.com/SiLab-Bonn/basil/blob/07b7cf30d4607caa5df677098d9207d0ec0b1c61/firmware/modules/tlu/tlu_controller_fsm.v#L328 The TLU Veto signal (TLU CLock high) is not used/enabled here (disabled by default) and I do not recommend using/enabling it (because I've never seen EUDAQ using it in their implementations, not tested, etc). The behavior you're describing is because the TLU/Trigger FSM of the BDAQ53 firmware is still running (and actively accepting triggers). The issue with the busy high is related to DAQ software issue and not related to the firmware. Usually the the external trigger scan is interrupted after a certain amount of time or certain amount of triggers. The trigger has to be actively stopped by the DAQ software. Can you check if

self.stop_readout()

is called after the scan has finished? Maybe add a print statement to the stop_readout function. Especially self.chip['cmd'].set_ext_trigger(False) needs to be called.

The maximum triggers can be set up with:

if kwargs['max_triggers']:
    self.chip['tlu']['MAX_TRIGGERS'] = kwargs['max_triggers']

Then you get the exact number of triggers. Take this as a feature request :-)

Thank you for the explanation!

Thank you Jens!

self.stop_readout() is called after the scan finished. And of course self.chip['cmd'].set_ext_trigger(False) is called here within the self.stop_readout() function :)

max_triggers is exactly set up like this here :)

Thanks for the details. If the TLU Busy signal is not high in idle mode, then probably the TLU is not in idle mode after the scan is finished (how do you put it back in idle mode?). Let me check if the self.stop_readout() is actually called after the scan is finished, anyway.

Exactly! Please additionally check if self['tlu']['TRIGGER_ENABLE'] is 0 at the end of def scan(self, **kwargs).

@ydieter Thanks for making this clear.

I can confirm you that trigger enable is set to 1 at the end of scan function (just before the logger messages)

2018-05-28 08:42:11,752 - [ExtTriggerScan ] - INFO Total amount of triggers accepted: 20114 2018-05-28 08:42:11,753 - [ExtTriggerScan ] - INFO Total amount of triggers collected: 20031 2018-05-28 08:42:11,753 - [ExtTriggerScan ] - INFO Scan finished self.chip['tlu']['TRIGGER_ENABLE']= 1

Also, the self.stop_readout() function (the one in the scan_ext_trigger.py) is not called (I put a print, and to be sure a raise, but nothing happended)

If I call inside the scan method the function stop_readout() (although probably this is not the proper way to do it):

        self.logger.info('Scan finished')

        print "calling stop_readout"
        self.stop_readout()
        print "called stop_readout"
        print "self.chip['tlu']['TRIGGER_EANBLE']=",self.chip['tlu']['TRIGGER_ENABLE']

Using this prints:

   def stop_readout(self, timeout=10.0):
        print "XXXXXXXXXXXXXXXXXXXX"
        # disable TLU module
        self.chip.set_tlu_module(False)
        print "XXXXXXXXXXXXXXXXXXXX -after set tlu"
        # disable external trigger
        self.chip['cmd'].set_ext_trigger(False)
        print "XXXXXXXXXXXXXXXXXXXX -after set ext trigger"

        print "XXXXXXXXXXXXXXXXXXXX -before scan_timeout set ext trigger"
        if self.scan_timeout:
            self.scan_timeout_timer.cancel()
        print "XXXXXXXXXXXXXXXXXXXX -after fifo_timeout "
        self.fifo_readout.stop(timeout=timeout)
        print "XXXXXXXXXXXXXXXXXXXX -after fifo_timeout "

Then, the busy line is effectively off, but the program stops long before is supposed, just after fifo_readout.stop, not doing nothing else (there is an analyze method and some prints that they are not shown):

2018-05-28 09:01:38,111 - [ExtTriggerScan ] - INFO    Scan finished
calling stop_readout
XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXX -after set tlu
XXXXXXXXXXXXXXXXXXXX -after set ext trigger
XXXXXXXXXXXXXXXXXXXX -before scan_timeout set ext trigger
XXXXXXXXXXXXXXXXXXXX -after fifo_timeout 
Closing remaining open files:/bdaq53/bdaq53/bdaq53/scans/output_data/20180528_090115_ext_trigger_scan.h5...done

So, by the moment, the self.chip['tlu']['RESET']=1 line we added at the end of the scan function is doing what we expect. But, again, probably this is not the proper way to do it, so let us know...

You're correct, stop_readout() is not called but it should be. I had a closer look at the code and the ScanBase class has start_readout() implemented but no stop_readout(). See here: https://gitlab.cern.ch/silab/bdaq53/blob/ba7d1c11f83ad676b86b563f99151e9c34f5f1bb/bdaq53/scan_base.py#L501

stop_readout() needs to be called after the yield statement (instead of self.fifo_readout.stop(timeout=timeout)). This is how it is in pyBAR and also should be done here in a similar fashion:

@contextmanager
def readout(self, *args, **kwargs):
    timeout = kwargs.pop('timeout', 10.0)
    self.start_readout(*args, **kwargs)
    try:
        yield
        self.stop_readout(timeout=timeout)
    finally:
        # in case something fails, call this on last resort
        if self.fifo_readout.is_running:
            self.fifo_readout.stop(timeout=0.0)

def start_readout(self, *args, **kwargs):
    ...

def stop_readout(self, timeout=10.0):
    self.fifo_readout.stop(timeout=timeout)

star_readout() / stop_readout() is then overwritten by the individual scan implementation and functionality is added if necessary.

: The contextmanager is worthless when there is no

try:
    ...
finally:
    ...

statement. Otherwise the cleanup (in this case stop_readout()) will not happen in case of an exception.

I'll let do the others doing the bugfix because I have no test setup on my desk.

Thanks Jens and the others for the quick reaction! So, we will keep our dummy patch to go on with our tests until you implement your proper fix. Please, let us known when is done, so we will change to your commit.

Thank you Jens and Jordi for pointing this out ;) I am very sure that at some point stop_readout() was called, maybe the code was changed. I will figure this out!

@duarte, thank you for spotting this bug! I fixed now the stop_readout function. Can you test again with this fix? I made a new branch.

@ydieter we tested and worked perfectly. Thank you very much for the quick fix!

Cool! No problem, it was very good that you spotted this bug!

mentioned in merge request !136 (merged)

added Bug label

mentioned in commit 0603ba58

The fix is now merged into development, thus I close this issue.

closed