README.md 2.6 KB
Newer Older
1
2
# Aggregator
The tool aggregates data in a format wich is suitable for generating reports.
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
3

4
5
6
7
8
9
*Currently also contains the script to export `jobinfo` data into DB from the post execution script*.

## Requirements
The aggregator needs particular software packages to be installed along with data available in DB.

-  Python 3.
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
10
-  InfluxDB. Currently the aggregator works only with InfluxDB, so it should be installed and the aggregator should be configured as shown in configuration section below.
11
12
13
14
15
16
17
18
19
20
-  DB should contain all required data specified in [DB cpec](https://gibraltar.chi.uni-hannover.de/profit-hpc/ProfiT-HPC/blob/d9a21af233ab373bf90420e3f0f0c05e1c65aef8/Internals/DB/InfluxDBspec.md).

## Configuration
Samples for configurations can be stored in the repository, please rename corresponding templates with an extension `.sample` to `.py` and change placeholder values accordingly.

Real configs should be ignored in `.gitignore` file.

Sample configs must not be used in the code.

**Example**: `influxdb.sample` -> `influxdb.py`
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
21
22

## Usage
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
23
The main executable of the aggregator module is `data.py`. You can type `./data.py -h` for more help.
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

```
usage: data.py [-h] [-s] [-t {ascii,pdf}] JOBID

Gets the job information required for generating ASCII or PDF reports and
outputs it in JSON format. Optionaly it stores the job info in DB.

positional arguments:
  JOBID                 job ID used in the batch system

optional arguments:
  -h, --help            show this help message and exit
  -s, --save            save batch info of the job in DB (default: False)
  -t {ascii,pdf}, --type {ascii,pdf}

                        type of the output (default: ascii)
```

## Get test output
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
43
In order to get `json` output from the test data, located at `test/data`, a docker container with InfluxDB instanse and imported data should be running. To run the InfluxDB container, simply execute:
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
```
test/docker/influxdb/run_influxdb.sh
```
After the influxDB instance is up you need to configure the aggregator to use it, by copying `influxdb.sample` -> `influxdb.py` and editing it as:

```
IDB = {
    "username": "",
    "password": "",
    "database": "pfit",
    "api_url": "http://localhost:58086",
    "ssl": False,
}
```

Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
59
As an example the following command will output the test data in the `json` format for a `pdf` report:
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
60
61
62
63
64
65

```
  ./data.py -t pdf 2368599
```

## Export job info
Azat Khuziyakhmetov's avatar
Azat Khuziyakhmetov committed
66
If the `data.py` is called with `-s`(`--save`) flag as `data.py -s JOBID`, then the aggregator will gather a job information with ID `JOBID` from the batch system configured in `/conf/config.py` and save it into the configured database as a `pfit-jobinfo` measurement.