BLM NXCALS Property Pull

Property pull is a Python project for the Cern's BLM section. It get precise data property for diamond devices by NXCALS and store it in lighter format on a MYSQL DB. In order to be used in other application. It is connected to a nodeJs API to communicate with the database.

A routine script have been done in order to get the beamMode data from pytimber and pull the data accordingly.

The final purpose is that it run daily on a server with a cronJob.

An API may be develop in the future for precise data pull without DB writing. Meaning an instant return (JSON format) of data required.

Installation

For now, it only work on linux OS and Windows Professional, and not tested on MacOS.

At the time of writing, python3.6 is required to run the CERN's Spark bundle propertly.

In order to run it, you have to Download the CERN's Spark bundle and set up a working CERN's NXCALS environment. See on NXCALS Documentation.

You need then to export 2 environment variables for Spark & Python.

For linux (Ubuntu tested)

export SPARK_HOME=<location_of_the_bundle>/spark-<version>-bin-hadoop<version>
export PYTHONPATH=$SPARK_HOME/nxcals-python3-env/lib/python3.6/site-packages 

For windows
Into shell with admin rights

setx SPARK_HOME <location_of_the_bundle>/spark-<version>-bin-hadoop<version>
setx PYTHON_PATH %SPARK_HOME%/nxcals-python3-env/lib/python3.6/site-packages

Then install pyspark packages on python if not already done.

pip install pyspark

Maybe you would have to replace the nxcals-jars directory with the one you can find into the bundle at <bundle_path>/nxcals-jars and also the nxcals-hdoop-pro-config.jar that you can find at <bundle_path>/jars/nxcals-hadoop-pro-config-<version>.jar.

Know that all the links and directory path are up to date at the time of writing. It may have been some updates since.

You can ask this info by creating an issue on this repo gitLab or by sending a mail to a manager of this project with your name, section and why you want to use the project.

Usage

The routine is made to pull the lasts fills from NXCALS (accordingly to the intermediate Database). It use the property_pull script to get each 5 types of data (histogram, integral, integral distribution, raw distribution and turnloss) for each BeamMode of the fill for each devices. Just for information, it means 5nbBeamModenbDevices different calls on NXCALS.

To run the routine or the property-pull script, it is common tu use the spark-submit command, that allow to use the CERN's spark bundle to submit application to the cluster. So tu run the routine on the cluster for example, do

spark-submit --master yarn routine_pull

Left to do

There is some improvement left to do.

First, clean the project and create a package for the property_pull script so it can be reused easyly. It is more tricky than it's look as it need the right configuration to be executed.

Then, create an API to do single request to pull only precise data ()