ETL Worker
Docker image for a Dashboard ETL worker - contains software performing NXCALS extractions and ETL transformations (using SciPy, Pandas etc...)
How to use
- Generate a keytab with :
cern-get-keytab --user --keytab nxcals.keytab- Provide Influxdb connectivity env variables
- Provide parameters to your extraction script
- Run :
docker run --net=host -e KPRINCIPAL=$USER -v `pwd`/nxcals.keytab:/auth/private.keytab -v `pwd`/myscript.py:/opt/nxcals-spark/work/script.py etlworker- You can also mount
/opt/nxcals-spark/workas a persistent volume if you wish to collect the output of your build. - You can also specify a password instead of a keytab via environment variable KPASSWORD :
# Read your password into local env variable (no echo)
read -s YOUR_PASSWORD
docker run --net=host -e KPRINCIPAL=$USER -e KPASSWORD=$YOUR_PASSWORD -v `pwd`/myscript.py:/opt/nxcals-spark/work/script.py etlworker- You can also run a bash session in the container :
docker run -it --net=host -e KPRINCIPAL=$USER -v ~/nxcals.keytab:/auth/private.keytab -v `pwd`/scripts:/opt/nxcals-spark/work:z etlworker bashDevelopment-related instructions
How to release
First, start a gitflow release branch and update the version to a non-SNAPSHOT:
export NEW_VERSION=<new version>
git flow release start $NEW_VERSION
mvn versions:set -DnewVersion=$NEW_VERSION
git commit -a -m "Preparing version $NEW_VERSION"Then, refine the release as needed. When you are ready :
git flow release finish $NEW_VERSION
git push --tags originThe release will be automatically deployed by Gitlab CI.
Once back on the develop branch, update the version and git push
mvn versions:set -DnewVersion=<new SNAPSHOT version>
git commit -a -m "Preparing next SNAPSHOT" && git pushHow to build manually
docker build --build-arg FROM="cern/cc8-base" -t etlworker .