Commit dc7594df authored by Domenico Giordano's avatar Domenico Giordano
Browse files

improved documentation

parent 0bc43fd7
# Data Analytics # Data Analytics
| qa | master | | [qa-v0.4](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/commits/qa-v0.4) | [v0.3](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/commits/v0.3) |
| :-: | :-: | | :-: | :-: |
|[![pipeline status qa](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/badges/qa/pipeline.svg)](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/commits/qa) |[![pipeline status master](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/badges/master/pipeline.svg)](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/commits/master)| |![](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/badges/qa-v0.4/pipeline.svg) |![](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/badges/v0.3/pipeline.svg)|
|[![coverage report qa](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/badges/qa/coverage.svg)](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/commits/qa)|[![coverage report master](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/badges/master/coverage.svg)](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/commits/master)| |![](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/badges/qa-v0.4/coverage.svg) |![](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/badges/v0.3/coverage.svg)|
The project contains a suite of tools to run data analytics pipelines on the monitoring data of the CERN Cloud Infrastructure. The project contains a suite of tools to run data analytics pipelines on the monitoring data of the CERN Cloud Infrastructure.
...@@ -26,25 +26,27 @@ The CI/CD of this project is used to: ...@@ -26,25 +26,27 @@ The CI/CD of this project is used to:
1. Build Docker images with pre-installed libraries needed for the project's scope 1. Build Docker images with pre-installed libraries needed for the project's scope
1. Run functional tests of the Data Analytics' pipeline, and its components 1. Run functional tests of the Data Analytics' pipeline, and its components
The repository contains extensive documentation of each subfolder in the README file included in the specific subfolder.<br> The repository contains extensive documentation of each subfolder in the README file included in each subfolder.<br>
This is a guide map of the repository: This is a guide map of the repository:
1. ETL libraries ([etl](etl))<br> 1. ETL libraries ([etl](etl))<br>
Implement the extraction of data from the different monitoring databases: InfluxDB, ElasticSearch, HDFS Implement the extraction of data from the different monitoring databases: InfluxDB, ElasticSearch, HDFS
1. Tests suite ([tests](tests))<br>
Unit tests of the ETL libraries, test pipelines' components
1. Javascript Grafana extension ([grafana_extension](grafana_extension))<br>
Implement an extension of the Grafana Annotation panel, modifying the Grafana JS code
1. Anomaly detection libraries ([adcern](adcern))<br> 1. Anomaly detection libraries ([adcern](adcern))<br>
Implement anomaly detection Models, based on pyOD, traditional ML and DL methods Implement anomaly detection Models, based on pyOD, traditional ML and DL methods
1. Tests suite ([tests](tests))<br>
Unit tests of the ETL libraries, test pipelines' components
1. Docker image definition ([docker-images](docker-images))<br> 1. Docker image definition ([docker-images](docker-images))<br>
Dockerfiles for images used in this project Dockerfiles for images used in this project
1. Airflow-based Anomaly Detection System ([control_room](control_room))<br> 1. Airflow-based Anomaly Detection System ([control_room](control_room))<br>
Setup and run the Anomaly Detection System Setup and run the Anomaly Detection System
1. Javascript Grafana extension ([grafana_extension](grafana_extension))<br>
Implement an extension of the Grafana Annotation panel, modifying the Grafana JS code
All these components are needed to deploy the Anomaly Detection System described in the figure
<br><img src="documentation/images/AD_system_technologies.png" width="70%"><br>
## From where to start ## From where to start
1. For a general introduction on this activity see the [ITTF seminar](https://indico.cern.ch/event/1012703/) 1. For a general introduction on this activity see the [ITTF seminar](https://indico.cern.ch/event/1012703/)
1. For interactive examples see [examples](examples) 1. For interactive examples see [examples](examples)
1. For Airflow deployment see [control_room](control_room)) 1. For Airflow deployment see [control_room](control_room)
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment