Dynamically generated child pipeline for athena and logParser jobs

Description

Major restructuring of pipeline to use the feature of dynamically generated child pipelines.

The run_athena and check_logParser jobs are now not fixed but the corresponding CI jobs are generated dynamically by scripts/generateYaml.py which is run by the setup_athena job in the parent pipeline. This will create a .run_athena.yml file which is used in run_child_pipeline in order to trigger the child pipeline jobs.

The dynamic feature of the child pipeline allows to

• run multiple athena jobs without extra time costs (since the jobs run in parallel) - see #12 (closed)
• run any setup in the jobs, in particular switch between images on the fly. This allows in particular to seamlessly integrate running jobs on CC7 and SLC6 (see #130 (closed)) (also CI jobs in the same pipeline can run using different images, e.g. one job running on CC7 and another on SLC6)

Parent pipeline

The run_child_pipeline job will wait until the child pipeline is finished:

If the child pipeline fails (e.g. athena or logParser fail) the pipeline will stop at run_child_pipeline:

The last job remove_logs will delete all log.generate.short files.

Child pipeline

The child pipeline will contain up to 5 pairs of (run_athena_DSID - check_logParser_DSID) jobs, that run in a directed acyclic graph (as it can be seen below, each pair of jobs is independent of the other jobs, e.g. once run_athena_700000 has finished check_logParser_700000 can run, without waiting for the rest of the run_athena_DSID jobs to finish)

Some things to note:

• Currently there will be a maximum of 5 pairs of athena-logParser jobs
• If there are 0 jO committed, a dummy CI job will be created in the child pipeline which will always succeed (see below)
• If there are <5 jO committed in a commit, there will be as many athena-logParser jobs as the number of jO changed in the commit
• The previous rules apply, i.e. the run_athena_DSID jobs will run only if a log.generate.short file is present in the DSID directory

Issues resolved

Closes #130 (closed) #12 (closed)