Skip to content

WIP: NXCALS-1095 Livy ansible installation

Grigorios Avgitidis requested to merge feature-NXCALS-1085-livy-spark-ansible-installation into develop

#Merge request template @acc-logging-team - please have a look for approval.

This MR introduces a new set of hostgroups, the 'extractors', responsible for accepting installations for modules that extract data from NXCALS system.

Moreover, after the revival of this MR from the NXCALS limbo requests, a new addition has been introduced. Now, ansible, common-startup-scripts role can work with non in-house developed modules as well! To make this happen, there was a need to modify the monit process invocation script, as well as the start.sh shell script to provide a functionality to optionally generate a PID for the underling application process.

Key ingredients of this set of commits are the ansible roles, for installing Apache's Livy and Spark. As well as, the modification of the hadoop role to also install Livy specific jars on HDFS side.

The fist will use the other, in order to spawn spark sessions remotely from it's ReST API.

For this series of commits, I request your attention mostly on the configuration of inventories and configuration file properties.

BTW, you can already play with the new Livy module installed on my environment...

Edited by Grigorios Avgitidis

Merge request reports