README.md

# Anomaly Detection Pipeline based on Airflow

The Anomaly Detection task can also be run in an automatic way. 
For doing that we rely on [Apache Airflow](https://airflow.apache.org/).
To have an easy to use environment we encapsulated all the required blocks (Airflow included) in Docker Containers that can be run thanks to *Docker compose*.
The Airflow Docker compose is heavily based on examples found in https://github.com/puckel/docker-airflow


This area is called `Control room` and  contains the procedures to deploy the Airflow setup and automate the Anomaly Detection task.

The folder includes

1. Installation scripts ([install_AD.sh](install_AD.sh))<br>
   To be run once when a new machine needs to be configured  
1. Docker-compose configuration ([airflow-compose](airflow-compose))<br>
   To setup the Airflow system
1. Docker-swarm configuration - w.i.p. ([docker-swarm](docker-swarm))<br>

   

## Getting started

The set of components that will be deployed with the following procedure is described in this image 
<br><img src="documentation/images/AD_components_deployed.png" width="70%"><br>

We suggest to run on a dedicated virtual machine VM that can be provisioned on the [OpenStack CERN Platform](https://openstack.cern.ch/). 
<br> For initial test we suggest to start with a flavor providing at least 7 GB of RAM.


1. Login to the VM (**tested on CentOS 7**) with the following port forwarding:
```
VM=your_vm_name
ssh -L 5003:localhost:5003 -L 8080:localhost:8080 root@$VM
```

Notice that if running from outside CERN you could need to double hop to get  port forwarding

```
VM=your_vm_name
ssh -L 9999:$VM:22 lxtunnel.cern.ch
ssh  -o StrictHostKeyChecking=no  -i ~/.ssh/id_rsa -L 8080:localhost:8080 -L 5003:localhost:5003 localhost -p 9999  -l root
```

2. When starting from a new VM, few packages need to be installed, if already not available in the VM.
For instance docker-compose, and the data-analytics package itself.
In addition, to enable the connection to the Spark cluster with `kinit`, the `secret` credentials have to be made available
and firewall rules to be setup.

Does the VM require initial installation?
   * **No**: go to next step 
   * **YES**: Run the [install_AD.sh](install_AD.sh)  script.

Run the script on your system and it will download all the necessary files in the folder **/opt/ad_system/** of your current machine.
In general the branch should be **master** (default) or a given gitlab **tag**, but any other branch can be configured, changing the env variable branch 
```
export branch=master
curl https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/raw/$branch/deploy_AD/install_AD.sh -O 
. ./install_AD.sh
install_all
```

Follow then the instructions printed by the install_AD.sh script to finalise the setup 

3. Start the docker compose of the Airflow-based Anomaly Detection System with the following command:
```
sudo -u airflow /opt/ad_system/control_ad_system/start_ad_system.sh
```

NB: the script `/opt/ad_system/control_ad_system/start_ad_system.sh` can also be sourced, to easily delete the running docker-compose setup

**Congratulation!** You just complete the full installation of your Anomaly Detection System.


### Getting started with Anomaly Detection DAG  

Now that Airflow is up and running we can test the Anomaly Detection System and
its algorithms on a demo scenario.

1. Open the Airflow UI: http://localhost:8080/
1. Select the *DAGs* tab from the Airflow menu.
1. Go to the *Graph View* tab to see the interconnection between different tasks
1. Click on the **on/off switch** next to the header *DAG: <dag_name>* to enable it.

**Congratulation!** You just started your first Anomaly Detection pipeline. Check then its successful termination via the *Tree view*, 
when the last box is dark green the pipeline is completed successfully.

## Additional Documentation

The Anomaly Detection System driven by Airflow can be started not only with Docker compose (our implemented choice) but also with
Docker Swarm (requires a Swarm cluster is already up) or using Kubernetes. These two methods are still work in progress.

- In case of a docker swarm, to continue the work look at the scripts in folder [docker-swarm](./docker-swarm)
- For Kubernetes documentation can be found at
   - https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/
   - https://airflow.apache.org/docs/stable/kubernetes.html