Skip to content

GitLab

Commit dd3915ae authored by Domenico Giordano's avatar Domenico Giordano
Browse files

Update... 

Update control_room/airflow-compose/swarm_step_1_pass_credentials_to_cluster.sh, control_room/airflow-compose/swarm_step2_start_swarm.sh, control_room/airflow-compose/README.md files
parent 71ac7fbf
Hide whitespace changes
Inline Side-by-side
README.md
View file @ dd3915ae
......@@ -39,66 +39,9 @@ This is a guide map of the repository:
Implement anomaly detection Models, based on pyOD, traditional ML and DL methods
1. Docker image definition ([link](docker-images))<br>
Dockerfiles for images used in this project
1. Airflow-based Anomaly Detection System ([link](control_room))<br>
Setup and run the Anomaly Detection System
## From where to start
Detail procedures for newcomers (W.I.P.)
## Installation Procedure
Start the VM (**tested on CentOS 7**) with the following port forwarding:
```
ssh -L 5003:localhost:5003 -L 8080:localhost:8080 root@cloud-anomaly-deploy
```
Run this script on your system and it will download all the necessary files in the folder **/opt/ad_system/** of your current machine.
```
curl https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/raw/qa/install_for_centos.sh | sh
```
Substitute *your_username* with the actual service username in kerberos, substitute *your_password* with the actual password.
```
curl https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/raw/qa/initialize_account.sh | sh -s your_username your_password
```
Start the system with the two following commands:
```
cd /opt/control_ad_system/airflow-compose
./start_ad_system.sh
```
**Congratulation!** You just complete the full installation of your Anomaly Detection System.
## Getting started with your first Anomaly Detection DAG
Now that Airflow is up and running we can test the Anomaly Detection System and
its algorithms on a demo scenario.
Follow these steps:
1. Open the File Browser http://localhost:5003/ and login (username = admin, pass = admin), Navigate to the folder **/airflow-compose/dags** and open the file
**config_variables.py**. There you have to change comments on the deploy section:
```
# DEPLOY
SYSTEM_FOLDER = "..."
DATALAKE_FOLDER = "..."
TMP_CONFIG = "..."
IMAGE_NAME = "..."
```
and comment the developement section:
```
# DEVELOPEMENT
# SYSTEM_FOLDER = "..."
# DATALAKE_FOLDER = "..."
# TMP_CONFIG = "..."
# IMAGE_NAME = "..."
```
1. Open the Airflow UI: http://localhost:8080/
1. Search for the dag named **dag_ad_demo** and click on its name.
1. Click on the *graph view* tab to see the interconnection between different tasks
1. Click on the **on/off switch** nex to the header *DAG: dag_ad_demo*.
**Congratulation!** You just started your first Anomaly Detection pipeline. Check the its successful termination via the *graph view*, when all the boxes are dark green the pipeline is completed.
> **_NOTE:_** The file browser is used to create new Airflow DAG (Direct Acyclic Graphs) and to modify the configuration files. Access it from here http://localhost:5003/ with username = admin, pass = admin.
control_room/README.md 0 → 100644
View file @ dd3915ae
# Control Room
Control room contains the procedures to deploy the Airflow setup and automate the Anomaly Detection task.
The folder includes
1. Installation scripts ([link](install_AD.sh))<br>
To be run once when a new machine needs to be configured
1. Docker-compose configuration ([link](airflow-compose))<br>
To setup the Airflow system
1. Configuration files ([link](config_file))<br>
Configuration files used for .... #FIXME
Steps:
1. Login to the VM (**tested on CentOS 7**) with the following port forwarding:
```
VM=your_vm_name
ssh -L 5003:localhost:5003 -L 8080:localhost:8080 root@$VM
```
Notice that if running from outside CERN you could need to double hop to get port forwarding
```
VM=your_vm_name
ssh -L 9999:$VM:22 lxtunnel.cern.ch
ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa -L 8080:localhost:8080 -L 5003:localhost:5003 localhost -p 9999 -l root
```
2. When starting from a new VM, few packages need to be installed, if already not available in the VM.
For instance docker-compose, and the data-analytics package itself.
In addition, to enable the connection to the Spark cluster with `kinit`, the `secret` credentials have to be made available
and firewall rules to be setup.
Does the VM require initial installation?
* **No**: go to next step
* **YES**: Run the [install_AD.sh](install_AD.sh) script.
Run the script on your system and it will download all the necessary files in the folder **/opt/ad_system/** of your current machine.
In general the branch should be **master** (default) or a given gitlab **tag**, but any other branch can be configured, changing the env variable branch
```
export branch=master
curl https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/raw/$branch/control_room/install_AD.sh -O
. ./install_AD.sh
install_all
```
3. Start the docker compose of the Airflow-based Anomaly Detection System with the following command:
```
/opt/control_ad_system/airflow-compose/start_ad_system.sh
```
**Congratulation!** You just complete the full installation of your Anomaly Detection System.
## Getting started with your first Anomaly Detection DAG
Now that Airflow is up and running we can test the Anomaly Detection System and
its algorithms on a demo scenario.
Follow these steps:
1. Open the File Browser http://localhost:5003/ and login (username = admin, pass = admin), Navigate to the folder **/airflow-compose/dags** and open the file
**config_variables.py**. There you have to change comments on the deploy section:
```
# DEPLOY
SYSTEM_FOLDER = "..."
DATALAKE_FOLDER = "..."
TMP_CONFIG = "..."
IMAGE_NAME = "..."
```
and comment the developement section:
```
# DEVELOPEMENT
# SYSTEM_FOLDER = "..."
# DATALAKE_FOLDER = "..."
# TMP_CONFIG = "..."
# IMAGE_NAME = "..."
```
1. Open the Airflow UI: http://localhost:8080/
1. Search for the dag named **dag_ad_demo** and click on its name.
1. Click on the *graph view* tab to see the interconnection between different tasks
1. Click on the **on/off switch** nex to the header *DAG: dag_ad_demo*.
**Congratulation!** You just started your first Anomaly Detection pipeline. Check the its successful termination via the *graph view*, when all the boxes are dark green the pipeline is completed.
> **_NOTE:_** The file browser is used to create new Airflow DAG (Direct Acyclic Graphs) and to modify the configuration files. Access it from here http://localhost:5003/ with username = admin, pass = admin.
control_room/airflow-compose/README.md
View file @ dd3915ae
# Anomaly Detection System driven by Airflow
## NB:
**NB**:
In these examples there are dummy passwords, in the secret.sh file
those passwords are here only as example for a simple local test.
!!! Do not commit real production passwords !!!
## Start Docker Compose
Initial example of docker-compose for airflow
This is heavily based on examples found in https://github.com/puckel/docker-airflow
The Anomaly Detection System driven by Airflow can be started in different ways
1. using Docker compose on a given VM
1. using Docker Swarm (requires a Swarm cluster is already up)
1. using Kubernetes (w.i.p)
To start a docker-compose example of airflow, run **for the first time** the script (for login to the docker registry and to enable the EOS client)
Details are in the following paragraphs.
## Secrets setup (TO BE REVIEWED - REMOVED)
To use the Spark cluster in the container spawn by Airflow you need to pass the credentials during Airflow startup.
This is done by creating a file named **local_service_password.sh** in the folder:
```
./one_time_start.sh
/root/local_service_password.sh
```
And put this in it, consisting in the real credentials of your (Kerberos) account in it:
```
export KRB_USER=your_username_in_cern_single_sign_on
export KRB_USER_PASSW=your_pass_in_cern_single_sign_on
```
Finally, it must be executable so you need to run
```
chmod 711 local_service_password.sh
```
then, run this script (for starting the docker-compose)
## Start Docker Compose
Initial example of docker-compose for airflow
This is heavily based on examples found in https://github.com/puckel/docker-airflow
To start the docker-compose of the Airflow-based Anomaly Detection System
make sure that all the needed packages and configurations have been installed (see [../README.md](../README.md))
run the following script (for starting the docker-compose)
```
./start_ad_system.sh
......@@ -26,7 +45,7 @@ When it is started, assuming you are in a VM, do double hop to get port forwardi
```
VM=your-cern-vm
ssh -L 9999:$VM:22 giordano@lxtunnel.cern.ch
ssh -L 9999:$VM:22 lxtunnel.cern.ch
ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa -L 8080:localhost:8080 localhost -p 9999 -l root
```
......@@ -43,8 +62,8 @@ docker run -it -v `pwd`:`pwd` -v /var/run/docker.sock:/var/run/docker.sock --ne
In case of a docker swarm, run to start
```
./0_prepare_swarm_cluster.sh
./1_start_swarm.sh
./swarm_step1_pass_credentials_to_cluster.sh
./swarm_step2_start_swarm.sh
```
### Docker Operator
......@@ -65,18 +84,3 @@ Documentation can be found at
- https://airflow.apache.org/docs/stable/kubernetes.html
# Secrets setup
To use the Spark cluster in the container spawn by Airflow you need to pass the credentials during Airflow startup. This is done by creating a file named **local_service_password.sh** in the folder:
```
/root/local_service_password.sh
```
And put this in it, consisting in the real credentials of your (Kerberos) account in it:
```
export KRB_USER=your_username_in_cern_single_sign_on
export KRB_USER_PASSW=your_pass_in_cern_single_sign_on
```
Finally, it must be executable so you need to run
```
chmod 711 local_service_password.sh
```
control_room/airflow-compose/dags/README.md
View file @ dd3915ae
# Available Airflow DAGs (or Pipelines)
Date: 23 March 2021 - Author: Matteo Paltenghi
We have four main types of DAGs representing the three steps in the benchmark procedure plus a combination suitable for continuous monitoring:
1. batch_1_etl / shared_1_etl : it downloads the data periodically from HDFS, aggregate them and save them in EOS. You control what to download via the two configuration yaml files: CONFIG_TEMPLATE_FILE_NAME_TEST and CONFIG_TEMPLATE_FILE_NAME_TRAIN. Note that this is done in Spark and the number of concurrent DAG you run influences the memory of the local VM, so be careful not to run too many of them concurrently (max_active_runs parameter of the dag).
......
control_room/airflow-compose/docker-compose.yml
View file @ dd3915ae
......@@ -87,7 +87,7 @@ services:
- pgdata:/var/lib/postgresql/data/pgdata
srv_airflow_web:
image: gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:matteo-thesis-2020 #airflowlocal # gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:${AIRFLOW_IMAGE}
image: gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:${AIRFLOW_IMAGE}
command: bash -c "echo $KRB_USER_PASSW | kinit $KRB_USER@CERN.CH; ls /eos/project/i/it-cloud-data-analytics/; /entrypoint.sh webserver"
restart: always
depends_on:
......@@ -97,6 +97,7 @@ services:
- LOAD_EX=n
- AIRFLOW__CORE__FERNET_KEY=${FERNET_KEY}
- secret_key=${FERNET_KEY}
- AIRFLOW__WEBSERVER__SECRET_KEY=${FERNET_KEY}
- FERNET_KEY=${FERNET_KEY}
- EXECUTOR=Celery
- POSTGRES_HOST=srv_postgres
......@@ -126,7 +127,7 @@ services:
retries: 3
srv_flower:
image: gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:matteo-thesis-2020 #airflowlocal # gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:${AIRFLOW_IMAGE}
image: gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:${AIRFLOW_IMAGE}
command: bash -c "echo $KRB_USER_PASSW | kinit $KRB_USER@CERN.CH; ls /eos/project/i/it-cloud-data-analytics/; /entrypoint.sh flower"
restart: always
depends_on:
......@@ -153,7 +154,7 @@ services:
- "6555:6555"
srv_scheduler:
image: gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:matteo-thesis-2020 #airflowlocal # gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:${AIRFLOW_IMAGE}
image: gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:${AIRFLOW_IMAGE}
command: bash -c "echo $KRB_USER_PASSW | kinit $KRB_USER@CERN.CH; ls /eos/project/i/it-cloud-data-analytics/; /entrypoint.sh scheduler"
restart: always
depends_on:
......@@ -182,7 +183,7 @@ services:
- REDIS_PASSWORD=${REDIS_PASSWORD}
srv_worker:
image: gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:matteo-thesis-2020 #airflowlocal # gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:${AIRFLOW_IMAGE}
image: gitlab-registry.cern.ch/cloud-infrastructure/data-analytics/docker-airflow:${AIRFLOW_IMAGE}
command: bash -c "echo $KRB_USER_PASSW | kinit $KRB_USER@CERN.CH; ls /eos/project/i/it-cloud-data-analytics/; /entrypoint.sh worker"
restart: always
privileged: true
......
control_room/airflow-compose/one_time_start.sh deleted 100755 → 0
View file @ 71ac7fbf
#!/bin/bash
# run only for the first configuration
# login to the gitlab registry to make sure that you can download
# the docker images used by the system
docker login gitlab-registry.cern.ch
# set up EOS
# or better make sure that we have all the required files
locmap --enable eosclient; locmap --configure eosclient
\ No newline at end of file
control_room/airflow-compose/secret.sh
View file @ dd3915ae
......@@ -6,11 +6,11 @@
# !!! Do not commit real production passwords !!!
export FLUENTD_IMAGE_TAG=v0.1
export SPARKNB_IMAGE_TAG=v0.1
export FLUENTD_IMAGE_TAG=v0.3
export SPARKNB_IMAGE_TAG=v0.3
export CVMFS_IMAGE_TAG=qa
export DAG_PATH=`pwd`/dags
export AIRFLOW_IMAGE=v0.1
export DAG_PATH=${SOURCE_DIR}/dags
export AIRFLOW_IMAGE=v0.3
export KRB_USER=this_is_not_the_user
......@@ -20,7 +20,7 @@ export KRB_USER_PASSW=this_is_not_the_password
# Secret key to save connection passwords in the db.
# To generate a key look at # https://bcb.github.io/airflow/fernet-key
export FERNET_KEY=YYWRfGG7GpTkbCYUuSEv5ycpY5Rz_a2PC_vlTOInfkg= # This is FAKE!!! Change it
export secret_key=${FERNET_KEY} #FIXME not needed here
# Postgres details
......@@ -71,4 +71,4 @@ export REDIS_DBNUM="1"
# SMALL CLUSTER
# export KUBECONFIG=/opt/data_repo_volume/repositories/k8s/config
# BIG CLUSTER
export KUBECONFIG=/opt/data_repo_volume/repositories/k8s/big-anomaly/config
\ No newline at end of file
export KUBECONFIG=/opt/data_repo_volume/repositories/k8s/big-anomaly/config
control_room/airflow-compose/start_ad_system.sh
View file @ dd3915ae
#!/bin/bash
# This script drives the start of docker-compose services
# for the Anomaly Detection System based on Airflow
SOURCE_SCRIPT=$(readlink -f $0)
export SOURCE_DIR=$(readlink -f $(dirname $0))
echo "SOURCE_DIR $SOURCE_DIR"
# Example to start docker compose
# this should be done from an instance having docker-compose
......@@ -8,127 +15,28 @@
# get superuser priviledges to run iptables and setup eos
. ./secret.sh
# LOCAL FILE WITH MY PASSWORD
. /opt/local_service_user_password.sh
if [ `iptables-save | grep DOCKER-USER | grep -c '\-\-dport 6555 -j DROP'` -eq 0 ];
then
echo "defining firewall rule to close port 6555 used by airflow and give only localhost access"
iptables -I DOCKER-USER ! -s 127.0.0.1/32 -i eth0 -p tcp -m tcp --dport 6555 -j DROP
/sbin/service iptables save
fi
if [ `iptables-save | grep DOCKER-USER | grep -c '\-\-dport 8080 -j DROP'` -eq 0 ];
then
echo "defining firewall rule to close port 8080 used by airflow Web UI and give only localhost access"
iptables -I DOCKER-USER ! -s 127.0.0.1/32 -i eth0 -p tcp -m tcp --dport 8080 -j DROP
/sbin/service iptables save
fi
if [ `iptables-save | grep DOCKER-USER | grep -c '\-\-dport 8793 -j DROP'` -eq 0 ];
then
echo "defining firewall rule to close port 8793 used by airflow and give only localhost access"
iptables -I DOCKER-USER ! -s 127.0.0.1/32 -i eth0 -p tcp -m tcp --dport 8793 -j DROP
/sbin/service iptables save
fi
if [ `iptables-save | grep DOCKER-USER | grep -c '\-\-dport 24224 -j DROP'` -eq 0 ];
then
echo "defining firewall rule to close port 24224 used by fluentd and give only localhost access"
iptables -I DOCKER-USER ! -s 127.0.0.1/32 -i eth0 -p tcp -m tcp --dport 24224 -j DROP
/sbin/service iptables save
fi
if [ `iptables-save | grep INPUT | grep -c '\-p tcp \-m multiport \-\-dports 5001:6000 \-m comment \-\-comment "00200 firewall for hadoop jobs" \-j ACCEPT'` -eq 0 ];
then
echo "defining firewall rule to open the Spark ports"
iptables -I INPUT -p tcp -m multiport --dports 5001:6000 -m comment --comment "00200 firewall for hadoop jobs" -j ACCEPT
/sbin/service iptables save
fi
if [ `iptables-save | grep DOCKER-USER | grep -c '\-p tcp \-m multiport \-\-dports 5001:6000 \-m comment \-\-comment "00200 firewall for hadoop jobs" \-j ACCEPT'` -eq 0 ];
then
echo "defining firewall rule to make the Spark ports accessible for docker"
iptables -I DOCKER-USER -p tcp -m multiport --dports 5001:6000 -m comment --comment "00200 firewall for hadoop jobs" -j ACCEPT
/sbin/service iptables save
fi
iptables -L DOCKER-USER
useradd airflow
usermod -aG docker airflow
## uncomment the commented part to delete all the airflow history
# MANUAL CONFIGURATION OF EOS ON THE VM
# https://cern.service-now.com/service-portal?id=kb_article&n=KB0003846
# STEP 1
# locmap --enable eosclient; locmap --configure eosclient;
# STEP 2 - COPY FILE MANUALLY FROM LXPLUS VIA SCP INTO THIS VM
# /etc/autofs.conf
# /etc/auto.master
# /etc/auto.eos
# /etc/eos/fuse.*
# /etc/sysconfig/eos.* <---- not present in lxplus
# STEP 3 - UMOUNT EOS
# umount /eos/project
# umount /eos/project-<letter>
# STEP 4 - RESTART THE SERVICE
# systemctl restart autofs
# STEP 5 - DONE
# Now you can access your data via /eos/project/i/it-cloud....
# TROUBLESHOOTING
# use this command to inspect processes that are using eos and stopping your
# from using umount (the second column gives their PID):
# lsof | grep eos
# than kill them with the number of the PID:
# kill -9 12345 <--- random example of PID
#umount /eos/project
# umount /eos/project-<letter>
# kill -9 $(pgrep -f eos)
#systemctl restart autofs
. ${SOURCE_DIR}/secret.sh
# LOCAL FILE WITH KRB PASSWORD
LOCAL_FILE_KRB_AUTH=/opt/ad_system/etc/local_service_user_password.sh
[ -e ${LOCAL_FILE_KRB_AUTH} ] && . ${LOCAL_FILE_KRB_AUTH}
echo -e "\nStop previous docker-compose...\n"
docker-compose -f docker-compose.yml down --remove-orphans # --volumes
docker-compose -f docker-compose.yml rm
docker-compose -f ${SOURCE_DIR}/docker-compose.yml down --remove-orphans # --volumes
docker-compose -f ${SOURCE_DIR}/docker-compose.yml rm
echo -e "\nKill processing still using cvmfs\n"
kill -9 $(pgrep -f cvmfs)
echo -e "\nKill processes still using cvmfs\n"
[ `pgrep -f cvmfs -c` -gt 0 ] && kill -9 $(pgrep -f cvmfs)
echo -e "\nRunning kinit...\n"
(echo $KRB_USER_PASSW | kinit $KRB_USER@CERN.CH) || fail 'kinit'
echo -e "\nCopy EOS config files from LXPLUS...\n"
mkdir /etc/eos/
(echo $KRB_USER_PASSW | scp $KRB_USER@lxplus.cern.ch:/etc/autofs.conf /etc/autofs.conf) || fail 'scp'
(echo $KRB_USER_PASSW | scp $KRB_USER@lxplus.cern.ch:/etc/auto.eos /etc/auto.eos) || fail 'scp'
(echo $KRB_USER_PASSW | scp $KRB_USER@lxplus.cern.ch:/etc/auto.master /etc/auto.master) || fail 'scp'
(echo $KRB_USER_PASSW | scp $KRB_USER@lxplus.cern.ch:/etc/eos/fuse.* /etc/eos/) || fail 'scp'
echo -e "\nStart EOS\n"
umount /eos/project
umount /eos/project-i
systemctl restart autofs
#kill -9 $(pgrep -f eos)
#echo '{"name":"project","hostport":"eosproject-fuse.cern.ch","remotemountdir":"/eos/project/","rm-rf-protect-levels":1}' >> /etc/eos/fuse.project.conf
#eosxd -ofsname=project /eos/project
#eosxd -ofsname=project-i /eos/project-i
echo -e "\nTest EOS access...\n"
ls /eos/project/i/it-cloud-data-analytics/
echo -e "\nStart new docker-compose...\n"
docker-compose -f docker-compose.yml up -d --remove-orphans --renew-anon-volumes # --abort-on-container-exit # --force-recreate
docker-compose -f ${SOURCE_DIR}/docker-compose.yml up -d --remove-orphans --renew-anon-volumes # --abort-on-container-exit # --force-recreate
control_room/airflow-compose/troubleshooting.md
View file @ dd3915ae
......@@ -10,4 +10,66 @@ ERROR: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables fa
systemctl restart docker
systemctl restart iptables
iptables -t filter -N DOCKER
```
\ No newline at end of file
```
# EOS problems
## Try manual installation
```
# MANUAL CONFIGURATION OF EOS ON THE VM
# https://cern.service-now.com/service-portal?id=kb_article&n=KB0003846
# STEP 1
# locmap --enable eosclient; locmap --configure eosclient;
# STEP 2 - COPY FILE MANUALLY FROM LXPLUS VIA SCP INTO THIS VM
# /etc/autofs.conf
# /etc/auto.master
# /etc/auto.eos
# /etc/eos/fuse.*
# /etc/sysconfig/eos.* <---- not present in lxplus
# STEP 3 - UMOUNT EOS
# umount /eos/project
# umount /eos/project-<letter>
# STEP 4 - RESTART THE SERVICE
# systemctl restart autofs
# STEP 5 - DONE
# Now you can access your data via /eos/project/i/it-cloud....
# TROUBLESHOOTING
# use this command to inspect processes that are using eos and stopping your
# from using umount (the second column gives their PID):
# lsof | grep eos
# than kill them with the number of the PID:
# kill -9 12345 <--- random example of PID
#umount /eos/project
# umount /eos/project-<letter>
# kill -9 $(pgrep -f eos)
#systemctl restart autofs
```
## Try copy from lxplus (to be avoided)
```
echo -e "\nCopy EOS config files from LXPLUS...\n"
mkdir /etc/eos/
(echo $KRB_USER_PASSW | scp $KRB_USER@lxplus.cern.ch:/etc/autofs.conf /etc/autofs.conf) || fail 'scp'
(echo $KRB_USER_PASSW | scp $KRB_USER@lxplus.cern.ch:/etc/auto.eos /etc/auto.eos) || fail 'scp'
(echo $KRB_USER_PASSW | scp $KRB_USER@lxplus.cern.ch:/etc/auto.master /etc/auto.master) || fail 'scp'
(echo $KRB_USER_PASSW | scp $KRB_USER@lxplus.cern.ch:/etc/eos/fuse.* /etc/eos/) || fail 'scp'
echo -e "\nStart EOS\n"
umount /eos/project
umount /eos/project-i
systemctl restart autofs
#kill -9 $(pgrep -f eos)
#echo '{"name":"project","hostport":"eosproject-fuse.cern.ch","remotemountdir":"/eos/project/","rm-rf-protect-levels":1}' >> /etc/eos/fuse.project.conf
#eosxd -ofsname=project /eos/project
#eosxd -ofsname=project-i /eos/project-i
```
\ No newline at end of file
control_room/install_AD.sh 0 → 100755
View file @ dd3915ae
# Script to install pkgs and configure the Anomaly Detection System
# To be run as root
function uninstall(){
# remove the system
rm -R /opt/ad_system/
# remove soft link to control room
rm /opt/control_ad_system
}
function install_centos7() {
# INSTALL WGET
yum -y install wget
# INSTALL DOCKER
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce docker-ce-cli containerd.io
install_eos
}
function install_eos(){
# INSTALL EOS
# https://cern.service-now.com/service-portal?id=kb_article&n=KB0003846
# For CERN CentOS 7 and CentOS8 desktops, please be sure that the EPEL repository is enabled and then use (as root):
echo -e "install EOS: this can require few minutes"
locmap --enable eosclient
locmap --configure eosclient
}
function install_common(){
# INSTALL DOCKER COMPOSE
# https://docs.docker.com/compose/install/
curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
[ ! -e /usr/bin/docker-compose ] && ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version
# PREPARE MAIN FOLDER
# main folder with files of the system
[ ! -e /opt/ad_system/ ] && mkdir -p /opt/ad_system/
# go to the main folder
# download the repository file FROM QA
# and create a folder
cd /opt/ad_system/
wget https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/archive/$branch/data-analytics-$branch.tar.gz -O - | tar -xz
# create the symlink to give a standard directory for the dags
ln -s /opt/ad_system/data-analytics-$branch/control_room/ /opt/control_ad_system
}
function set_firewall(){
# Set firewall rules to close some ports
# and open the port to communicate with Spark Cluster