Dashboard ETL
Extraction, Transformation and Loading (ETL) python module library (uses NXCALS, Pandas, influxdb)
How to use the module
First, you will need to provide INFLUXDB credentials, you can do so in a .env file located
in the same folder as your script or any folder above.
INFLUXDB_HOST=host
INFLUXDB_PORT=port
INFLUXDB_USERNAME=username
INFLUXDB_PASSWORD=password
INFLUXDB_DATABASE=database_nameYou can also provide these environment variable directly in the environment.
Then you need a python script like so :
from dashetl import etl
# Initiate a spark session
etl.create_spark_session('etl-example')
# This will extract the list of given variables and send the data to the specified INFLUXDB instance
# Variable names become Influxdb metrics
etl.extraction(measurements=["VARIABLE_1","VARIABLE_2"], \
start_date='2020-07-12 00:00:00.000', \
end_date='2020-07-12 04:00:00.000')