Commit fb8f5082 authored by Domenico Giordano's avatar Domenico Giordano
Browse files

improve documentation

parent 187f9ad3
......@@ -2,7 +2,7 @@
The Anomaly Detection task can also be run in an automatic way.
For doing that we rely on [Apache Airflow](https://airflow.apache.org/).
To have an easy to use environment we encapsulated all the required blocks (Airflow included) in Docker Containers that can be run thanks to Docker compose.
To have an easy to use environment we encapsulated all the required blocks (Airflow included) in Docker Containers that can be run thanks to *Docker compose*.
The Airflow Docker compose is heavily based on examples found in https://github.com/puckel/docker-airflow
......@@ -10,11 +10,11 @@ This area is called `Control room` and contains the procedures to deploy the Ai
The folder includes
1. Installation scripts ([link](install_AD.sh))<br>
1. Installation scripts ([install_AD.sh](install_AD.sh))<br>
To be run once when a new machine needs to be configured
1. Docker-compose configuration ([link](airflow-compose))<br>
1. Docker-compose configuration ([airflow-compose](airflow-compose))<br>
To setup the Airflow system
1. Docker-swarm configuration - w.i.p. ([link](docker-swarm))<br>
1. Docker-swarm configuration - w.i.p. ([docker-swarm](docker-swarm))<br>
......@@ -73,12 +73,12 @@ Now that Airflow is up and running we can test the Anomaly Detection System and
its algorithms on a demo scenario.
1. Open the Airflow UI: http://localhost:8080/
1. Search for the dag named **dag_ad_demo** and click on its name.
1. Click on the *graph view* tab to see the interconnection between different tasks
1. Click on the **on/off switch** next to the header *DAG: dag_ad_demo*.
1. Select the *DAGs* tab from the Airflow menu.
1. Go to the *Graph View* tab to see the interconnection between different tasks
1. Click on the **on/off switch** next to the header *DAG: <dag_name>* to enable it.
**Congratulation!** You just started your first Anomaly Detection pipeline. Check then its successful termination via the *graph view*,
when all the boxes are dark green the pipeline is completed.
**Congratulation!** You just started your first Anomaly Detection pipeline. Check then its successful termination via the *Tree view*,
when the last box is dark green the pipeline is completed successfully.
## Additional Documentation
......
# Anomaly Detection System driven by Airflow
**NB**:
In these examples there are dummy passwords stored in the secret.sh file.
In these examples there are dummy passwords stored in the **secret.sh** file.
Those passwords are here only as example for a simple local test.
**!!! Do not commit real production passwords !!!**
## Docker Operator
The Airflow Operators usable in our use case are BashOperator, DockerOperator, Docker Swarm Operator or kubernetes Operator.
Documentation about docker operator can be found at
- https://airflow.apache.org/docs/stable/_modules/airflow/operators/docker_operator.html#DockerOperator
- https://marclamberti.com/blog/how-to-use-dockeroperator-apache-airflow/
Docker swarm operator
- https://airflow.apache.org/docs/stable/_modules/airflow/contrib/operators/docker_swarm_operator.html#DockerSwarmOperator._run_image
To start/stop Airflow docker-compose use the following commands:
1. Initialize: `source ./start_ad_system.sh`
1. Reset (!!!CAVEAT: this removes all stored DAG executions and logs!!!): `start_compose_AD prune`
1. Stop AD Airflow: `stop_compose_AD`
1. Stop AD Airflow and remove volumes (same effect as in 2.): `stop_compose_AD --volumes`
1. Start/Re-start AD Airflow (w/o removing volumes): `start_compose_AD`
## Troubleshooting
......
......@@ -21,6 +21,15 @@ the **--log-driver** Docker attribute is not supported. We use the log-driver to
In order to guarantee the authentication to the **Cern Spark cluster** and to **EOS** the Kerberos ticket cache is percolated from the Airflow docker-compose to the running *sparknotebook* container.
Note:
- Documentation about docker operator can be found at
- https://airflow.apache.org/docs/stable/_modules/airflow/operators/docker_operator.html#DockerOperator
- https://marclamberti.com/blog/how-to-use-dockeroperator-apache-airflow/
- Docker swarm operator
- https://airflow.apache.org/docs/stable/_modules/airflow/contrib/operators/docker_swarm_operator.html#DockerSwarmOperator._run_image
# Run on Kubernetes (THIS PART REFERS TO OLD APPROACH. TO BE REVIEWED)
Steps to run algorithm train on the Kubernetes (K8s) cluster are described here
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment