swan_installation.md 1.76 KB
Newer Older
Matteo Paltenghi's avatar
Matteo Paltenghi committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
## Getting Started
You can use this module in two environments:
1. in the SWAN notebook
1. in your local machine
1. in a [CERN VM for Automatic Anomaly Detection](documentation/ad-pipeline-airflow.md)

### Prerequisites

You need to:
1. be able to authenticate via kerberos for the Spark access
```
kinit username@CERN.CH
```
2. have a Grafana token, to access the InfluxDB and ElasticSearch data.


### Installing - SWAN (https://swan.web.cern.ch/)
1. Make sure you start Swan connecting to the right Spark cluster, that in general is `the General Purpose (Analytix)`
2. Paste this in you notebook:
``` python
# necessary to download the authenticate
import getpass
import os, sys

print("Please enter your kerberos password")
ret = os.system("echo \"%s\" | kinit" % getpass.getpass())

if ret == 0: print("Credentials created successfully")
else:        sys.stderr.write('Error creating credentials, return code: %s\n' % ret)
```
2. Execute the cell and insert your passwork. Now you are logged in.
3. Paste this and execute:
```
! pip install --user git+https://:@gitlab.cern.ch:8443/cloud-infrastructure/data-analytics.git
```
4. Instal the package that you want (e.g):
``` python
from etl.spark_etl import cluster_utils
```

### Installing - Local machine
 1. You can directly login to your shell with the command:
 ```
 kinit username@CERN.CH
 ```
 2. Insert you pasword.
 3. Directly install the package from gitlab
 ```
 ! pip install --user git+https://:@gitlab.cern.ch:8443/cloud-infrastructure/data-analytics.git
 ```

## Possible use cases

![use-case-diagram](documentation/diagrams/use-case-data-analytics.png)


## Example Notebooks

You can refer to the [notebook folder](https://gitlab.cern.ch/cloud-infrastructure/data-analytics/-/tree/master/notebooks) to explore some example of usage.