README.md 2.7 KB
Newer Older
1
2
# Aggregator
The tool aggregates data in a format wich is suitable for generating reports.
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
3

4
5
6
7
8
9
*Currently also contains the script to export `jobinfo` data into DB from the post execution script*.

## Requirements
The aggregator needs particular software packages to be installed along with data available in DB.

-  Python 3.
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
10
-  InfluxDB. Currently the aggregator works only with InfluxDB, so it should be installed and the aggregator should be configured as shown in configuration section below.
11
12
-  DB should contain all required data specified in [DB cpec](https://gibraltar.chi.uni-hannover.de/profit-hpc/ProfiT-HPC/blob/d9a21af233ab373bf90420e3f0f0c05e1c65aef8/Internals/DB/InfluxDBspec.md).

Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
13
14
15
16
17
18
You should also install Python dependencies with:

```bash
python3 -m pip install -r requirements.txt
```

19
20
21
22
23
24
25
26
## Configuration
Samples for configurations can be stored in the repository, please rename corresponding templates with an extension `.sample` to `.py` and change placeholder values accordingly.

Real configs should be ignored in `.gitignore` file.

Sample configs must not be used in the code.

**Example**: `influxdb.sample` -> `influxdb.py`
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
27
28

## Usage
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
29
The main executable of the aggregator module is `data.py`. You can type `./data.py -h` for more help.
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
30
31

```
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
32
usage: data.py [-h] [-s] [-t {text,pdf}] JOBID
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
33

Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
34
Gets the job information required for generating text or PDF reports and
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
35
36
37
38
39
40
41
42
outputs it in JSON format. Optionaly it stores the job info in DB.

positional arguments:
  JOBID                 job ID used in the batch system

optional arguments:
  -h, --help            show this help message and exit
  -s, --save            save batch info of the job in DB (default: False)
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
43
  -t {text,pdf}, --type {text,pdf}
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
44

Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
45
                        type of the output (default: text)
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
46
47
48
```

## Get test output
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
49
In order to get `json` output from the test data, located at `test/data`, a docker container with InfluxDB instanse and imported data should be running. To run the InfluxDB container, simply execute:
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
```
test/docker/influxdb/run_influxdb.sh
```
After the influxDB instance is up you need to configure the aggregator to use it, by copying `influxdb.sample` -> `influxdb.py` and editing it as:

```
IDB = {
    "username": "",
    "password": "",
    "database": "pfit",
    "api_url": "http://localhost:58086",
    "ssl": False,
}
```

Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
65
As an example the following command will output the test data in the `json` format for a `pdf` report:
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
66
67
68
69
70
71

```
  ./data.py -t pdf 2368599
```

## Export job info
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
72
If the `data.py` is called with `-s`(`--save`) flag as `data.py -s JOBID`, then the aggregator will gather a job information with ID `JOBID` from the batch system configured in `/conf/config.py` and save it into the configured database as a `pfit-jobinfo` measurement.