The commands module is deprecated, but also wasn't properly surfacing an error
in the eos command. Using shell=True in submodule is necessary at least with
the current eos command string.

With recent athena releases the version of xrootd set up is incompatible with
the eos executable and causes symbol not found errors. As a workaround,
explicitly override the library path for the duration of its invocation.
Ideally the executable will handle the situation better in future.

The offline beamspot account atlidbs has a release set up at ~atlidbs/cron
which is responsible for running our cron jobs. I'm updating it to use release
21 built with cmake from git, and so the setup script needs to change to
reflect the new workflow.

I have also tweaked the debug messages and used the canonical shebang line.

Changes taken from these SVN tags:
 BCM_G4_SD-00-07-00
 PixelG4_SD-00-09-00
 SCT_G4_SD-00-08-00
 TRT_G4_SD-00-09-01

Fixes ATLASSIM-3009

I don't see how the old behaviour ever worked, since the test string could
never be equal to the empty string so that if statement would always be true.
However since the code was always run in environments with a patch version the
bug was never noticed.

With the newer cmake+git releases this buggy code path is triggered, so
I instead remove it. Also `os.path.expandvars()` was really being abused here
since if the variable is not set it returns the variable name literally in the
result whereas I think the author expected it to return an empty string.
Instead `os.getenv()` is more appropriate.

Updates to DCS conditions data have been silently ignored. This was
caused by using insert rather than an assignment to update existing
data.

Add protection against invalid hashId. Remove unnecessary initialization. Use is_valid method as a standard way.

By default the offline tracking does not check for dead pixel front-ends. Added a flag
to enable this check, but switched off by default to preserve FT0.
Enables the check by default for the VtxLumi tracking setup.

Vakhtang Tsulaia authored 7 years ago and Jovan Mitrevski committed 7 years ago

Fix a binning issue in InDet::TRT_ElectronPidToolRun2::StorePIDinfo::GetBin

See merge request !1852

For reasons I don't fully understand, when I submitted beamspot jobs using
release 21 they fail to submit unless the shell is passed explicitly to the
bsub command. With this change jobs can again be submitted normally.

This will be used more uniformly throughout the file in future.

This should have gone into part 2 but I missed it when I was splitting the
changes into separate branches. It just prevents the following debug statement
from being dead code.

This just allows a glob-like syntax for restricting the import command to only
run on particular tasks/datasets.

Similar to 9ebe7e4d, remove misleading version lines which are no longer
updated. Once I confirm that the __version__ variables are not being used
elsewhere I will remove these as a more permanent solution.

Similar to the change in beamspotman.py from 2428a685

The previous commit introduced a new interface to TaskManager where using it as
a context manager cleans up the database connection at the end of the block.
This commit switches all current uses of the TaskManager to this new interface,
in some cases re-ordering parts of the code locally so as to make the block
with the database connection as minimal as possible.

The majority of lines touched here are only because they needed to be indented
to sit inside a new context block.

We ran into a compound issue when running beamspot jobs on the batch system.
The underlying issue was that there is a ROOT bug triggered when the end of the
python script is reached and ROOT enters a cleanup routine. It would crash and
then cause the scripts to hang.

The problem was that we were opening an Oracle connection to the beamspot task
database but deferring the closure of the connection until __del__. Because of
the ROOT bug python never got around to cleaning the TaskManager and so the
connection would remain open. Other jobs in the same batch would then hang
waiting for the database lock to be released causing a deadlock. The jobs would
fill up all available execution slots on the batch queue. To make matters even
worse the default batch queue was atlasb1 which is used for time-critical jobs.

This change instead makes TaskManager a python context manager so that it can
be called using a pattern like:

    with TaskManager(...) as taskman:
      # do something ...

and the connection will be closed at the end of the block. A variable is added
to keep track of cases when this pattern is not being used so that a warning
can be printed. This should be sufficient to minimise deadlock issues like we
observed before.

This is not done consistently throughout the file, but will be superseded by
a better interface in the near future so it's not worth changing the other
commands as well.

This change was part of svn changeset 786385:

  https://svnweb.cern.ch/trac/atlasoff/changeset/786385/InnerDetector/InDetExample/InDetBeamSpotExample/trunk/bin

Improvements include:
 - improved readability
 - command-line options to choose straws to mask
 - possibility to mask only full boards
 - updates to reference and format of input/ output
   for compatibility with other scripts

The __version__ variable in each file is used in log messages to explicitly
note the source version running. Since these are no longer updated in the git
world I have given them dummy values for now so as not to be misleading. In the
future this feature may be removed.

The postprocessing step would be triggered after jobs were in the SUBMITTED
state, but this was too early. Postponing until RUNNING prevents the
postprocessing from beginning too early and failing.

This change was part of svn changeset 786208:

  https://svnweb.cern.ch/trac/atlasoff/changeset/786208/InnerDetector/InDetExample/InDetBeamSpotExample/trunk/bin

Sometimes the list of postprocessing steps was comma-separated and sometimes it
was space-separated. The argument as passed in would then be stored in the
beamspot task database without validation, implicitly assuming that spaces were
delimiting the steps. Commas would silently be accepted but then when the
string is read out of the database later it is split only on spaces, causing
the job to crash with an unknown step name.

With this change the option parser is extended with a new field that
understands both comma and space delimiting so that the option is stored as
a python list. This list is then uniformly serialised with spaces when passed
to the database logic.

This is part of svn changeset 786208:

  https://svnweb.cern.ch/trac/atlasoff/changeset/786208/InnerDetector/InDetExample/InDetBeamSpotExample/trunk/bin

While a context-specific default queue normally does the right thing, this flag
allows it to be overwritten if needed. The default queue will be the sensible
one for normal tasks and this is unchanged.

This change corresponds to the svn changeset 785931:

  https://svnweb.cern.ch/trac/atlasoff/changeset/785931/InnerDetector/InDetExample/InDetBeamSpotExample/trunk/bin

This is part of svn changeset 786212:

  https://svnweb.cern.ch/trac/atlasoff/changeset/786212/InnerDetector/InDetExample/InDetBeamSpotExample/trunk/bin