Commit fb13df94 authored by Domenico Giordano's avatar Domenico Giordano
Browse files

Update control_room/airflow-compose/dags/README.md,...

Update control_room/airflow-compose/dags/README.md, control_room/airflow-compose/config_files/publish_fluentd_always_on/batch_etl_test.yaml, control_room/airflow-compose/config_files/README.md, control_room/airflow-compose/docker-swarm/swarm-docker-compose.yml, control_room/airflow-compose/docker-swarm/swarm_step2_start_swarm.sh, control_room/airflow-compose/docker-swarm/swarm_step_1_pass_credentials_to_cluster.sh, control_room/airflow-compose/README.md, control_room/README.md, control_room/airflow-compose/config_files/always_on/analysis_eos.yaml, control_room/airflow-compose/config_files/always_on/etl_test_eos.yaml, control_room/airflow-compose/config_files/always_on/etl_train_eos.yaml, control_room/airflow-compose/config_files/benchmark/essential_metrics/batch_etl_test.yaml, control_room/airflow-compose/config_files/benchmark/essential_metrics/batch_etl_train.yaml, control_room/airflow-compose/config_files/benchmark/essential_metrics/batch_experiment.yaml, control_room/airflow-compose/config_files/benchmark/essential_metrics/shared_etl_test.yaml, control_room/airflow-compose/config_files/benchmark/essential_metrics/shared_etl_train.yaml, control_room/airflow-compose/config_files/benchmark/essential_metrics/shared_experiment.yaml, control_room/airflow-compose/config_files/benchmark/improve_metrics/batch_etl_test.yaml, control_room/airflow-compose/config_files/benchmark/improve_metrics/batch_etl_train.yaml, control_room/airflow-compose/config_files/benchmark/improve_metrics/shared_etl_test.yaml, control_room/airflow-compose/config_files/benchmark/improve_metrics/shared_etl_train.yaml, control_room/airflow-compose/config_files/benchmark/noise_metrics/batch_etl_test.yaml, control_room/airflow-compose/config_files/benchmark/noise_metrics/batch_etl_train.yaml, control_room/airflow-compose/config_files/benchmark/noise_metrics/shared_etl_test.yaml, control_room/airflow-compose/config_files/benchmark/noise_metrics/shared_etl_train.yaml, control_room/airflow-compose/config_files/benchmark/batch_etl_test.yaml, control_room/airflow-compose/config_files/benchmark/batch_etl_train.yaml, control_room/airflow-compose/config_files/benchmark/batch_experiment.yaml, control_room/airflow-compose/config_files/benchmark/etl_test_shared.yaml, control_room/airflow-compose/config_files/benchmark/etl_train_shared.yaml, control_room/airflow-compose/config_files/benchmark/experiment_shared.yaml, control_room/airflow-compose/config_files/benchmark/experiment_shared_lof.yaml, control_room/airflow-compose/config_files/demo/analysis_eos.yaml, control_room/airflow-compose/config_files/demo/etl_test.yaml, control_room/airflow-compose/config_files/demo/etl_test_eos.yaml, control_room/airflow-compose/config_files/demo/etl_train.yaml, control_room/airflow-compose/config_files/demo/etl_train_eos.yaml, control_room/airflow-compose/config_files/demo/template_analysis.yaml, control_room/airflow-compose/config_files/publish_fluentd/batch_etl_test.yaml, control_room/airflow-compose/config_files/publish_fluentd/batch_etl_train.yaml, control_room/airflow-compose/config_files/publish_fluentd/batch_experiment.yaml, control_room/airflow-compose/config_files/publish_fluentd_always_on/batch_etl_train.yaml, control_room/airflow-compose/config_files/publish_fluentd_always_on/batch_experiment.yaml files
Deleted control_room/airflow-compose/troubleshooting.md
parent de23bf17
......@@ -193,4 +193,4 @@ selected_plugins:
# type_instance: blocked
# plugin_name: processes
...
\ No newline at end of file
...
......@@ -4,4 +4,4 @@ We have four main types of DAGs representing the three steps in the benchmark pr
1. batch_1_etl / shared_1_etl : it downloads the data periodically from HDFS, aggregate them and save them in EOS. You control what to download via the two configuration yaml files: CONFIG_TEMPLATE_FILE_NAME_TEST and CONFIG_TEMPLATE_FILE_NAME_TRAIN. Note that this is done in Spark and the number of concurrent DAG you run influences the memory of the local VM, so be careful not to run too many of them concurrently (max_active_runs parameter of the dag).
1. batch_2_experiment / shared_2_experiment : it reads the EOS data and run the predefined algorithms in the analysis.yaml file (refer to CONFIG_TEMPLATE_ANALYSIS variable). This step can be on the current VM or on the K8s cluster by using the Airflow operators, DockerToPythonOperator and PythonInK8sOperator respectively. Note that this step publish data both in the MONIT infrastructure (via fluentd) and in the score folder (typically in EOS) as specified in the analysis.yaml file (refer to CONFIG_TEMPLATE_ANALYSIS variable).
1. batch_3_evaluation / shared_3_evaluation : it read the locally produced scores (in EOS), downloads the labeled data from Grafana and compute the performance of every algorithm in the benchmark period. It produces the results in the form of diagrams in the relative result folder defined in the analysis.yaml file (refer to CONFIG_TEMPLATE_ANALYSIS variable).
1. batch_4_always_on / shared_4_always_on: One additional kind of pipeline combines the previous two steps (ETL + production of scores in the MONIT) and run continuously.
\ No newline at end of file
1. batch_4_always_on / shared_4_always_on: One additional kind of pipeline combines the previous two steps (ETL + production of scores in the MONIT) and run continuously.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment