Skip to content

NXCALS-1095 Spark ansible installation with NXCALS sources

#Merge request template @acc-logging-team - please have a look for approval.

This merge request, is part of the separation that derived out of the effort initially introduced in the livy-spark MR. Link to parent MR: !129 (closed)

The configured spark could be invoked seamlessly as follows (example with spark-shell):

[gavgitid@nxcals-gavgitid4 spark-home]$ spark-2.2.1-bin-hadoop2.6/bin/spark-shell <-- no need for args, it's configured completely via spark config files :: loading settings :: file = /opt/nxcals/spark-home/ivy_settings/ivysettings.xml Ivy Default Cache set to: /opt/nxcals/spark-home/.ivy2/cache The jars for the packages stored in: /opt/nxcals/spark-home/.ivy2/jars :: loading settings :: url = jar:file:/opt/nxcals/spark-home/spark-2.2.1-bin-hadoop2.6/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml cern.accsoft.nxcals#accsoft-nxcals-data-access added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found cern.accsoft.nxcals#accsoft-nxcals-data-access;0.1.53-SNAPSHOT in m2-local found cern.accsoft.nxcals#accsoft-nxcals-common;0.1.53-SNAPSHOT in m2-local found org.apache.avro#avro;1.8.2 in http://artifactory.cern.ch/ds-release found org.codehaus.jackson#jackson-core-asl;1.9.13 in http://artifactory.cern.ch/ds-release found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in http://artifactory.cern.ch/ds-release found com.thoughtworks.paranamer#paranamer;2.7 in http://artifactory.cern.ch/ds-release found org.xerial.snappy#snappy-java;1.1.1.3 in http://artifactory.cern.ch/ds-release found org.apache.commons#commons-compress;1.8.1 in http://artifactory.cern.ch/ds-release found org.tukaani#xz;1.5 in http://artifactory.cern.ch/ds-release found org.slf4j#slf4j-api;1.7.7 in http://artifactory.cern.ch/ds-release found com.google.guava#guava;16.0 in http://artifactory.cern.ch/ds-release found commons-codec#commons-codec;1.10 in http://artifactory.cern.ch/ds-release

Edited by Grigorios Avgitidis

Merge request reports