sensor2graph issueshttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues2021-03-08T16:07:41Zhttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/6Ability to turn off individual csv stages2021-03-08T16:07:41ZJakob Leonard HördtAbility to turn off individual csv stagesI miss this during development, especially because I don't have info on how to setup a development PID registry.
Probably a good idea to control this with another config setting.I miss this during development, especially because I don't have info on how to setup a development PID registry.
Probably a good idea to control this with another config setting.Jakob Leonard HördtJakob Leonard Hördthttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/10calculate first- / last_msg2021-03-07T18:32:56ZJakob Leonard Hördtcalculate first- / last_msgcurrently the last MeasurementLocation is taken for both last/first
Three options come to mind:
#### Track last_first msg in Sensor node and use that to update pid:
probably pretty easy to implement, danger of inconsistency between Measu...currently the last MeasurementLocation is taken for both last/first
Three options come to mind:
#### Track last_first msg in Sensor node and use that to update pid:
probably pretty easy to implement, danger of inconsistency between Measurement and Sensor Nodes, redundant info also requires slightly more storage
#### Calculate minmax from all MeasurementLocations when updating pid:
should be fast enough, no redundancy, robust against any database change
#### Assume we create the PID when processing the oldest Measurement and that the current Measurement is always the newest. Only update `last_msg` without comparison.
This is simply not robust. Assumptions not met.https://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/9cannot coerce NULL to LocalDate2021-03-07T18:32:56ZJakob Leonard Hördtcannot coerce NULL to LocalDate`head 2020-09-30_sds011_sensor_622.csv`:
```
sensor_id;sensor_type;location;lat;lon;timestamp;P1;durP1;ratioP1;P2;durP2;ratioP2
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:01:44;21.57;;;18.43;;
622;SDS011;25927;51.03122679;13...`head 2020-09-30_sds011_sensor_622.csv`:
```
sensor_id;sensor_type;location;lat;lon;timestamp;P1;durP1;ratioP1;P2;durP2;ratioP2
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:01:44;21.57;;;18.43;;
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:04:10;22.80;;;19.27;;
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:06:36;21.33;;;19.10;;
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:09:02;19.30;;;17.17;;
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:11:28;19.30;;;16.80;;
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:13:54;17.67;;;15.87;;
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:16:20;19.83;;;17.17;;
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:18:45;20.30;;;16.13;;
622;SDS011;25927;51.03122679;13.73194456;2020-09-30T00:21:11;19.33;;;16.90;;
```
```
SEVERE: sensor https://archive.sensor.community/2020-09-30/2020-09-30_sds011_sensor_622.csv not uploaded because: Cannot coerce NULL to LocalDate
org.neo4j.driver.exceptions.value.Uncoercible: Cannot coerce NULL to LocalDate
at org.neo4j.driver.internal.value.ValueAdapter.asLocalDate(ValueAdapter.java:317)
at sensor2graph.Sensor.request_to_registry(Sensor.java:71)
at sensor2graph.Sensor.register_new_pid(Sensor.java:55)
at sensor2graph.Sensor.process(Sensor.java:41)
at sensor2graph.DayUploader.lambda$make_uploading_task$0(DayUploader.java:47)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
```
needs to be investigatedhttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/25certificate expired2023-01-25T20:31:25ZJakob Leonard Hördtcertificate expiredhttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/2Complete PID registry registration implementation2021-10-13T15:09:32ZJakob Leonard HördtComplete PID registry registration implementationexample implementation: https://hdl.handle.net/21.T11998/0000-001A-3905-F?noredirect
**bold** -> Mandatory
*italic* -> part of SensorType
- [x] **Identifier**
what is this supposed to be exactly? How do we get the PID while creat...example implementation: https://hdl.handle.net/21.T11998/0000-001A-3905-F?noredirect
**bold** -> Mandatory
*italic* -> part of SensorType
- [x] **Identifier**
what is this supposed to be exactly? How do we get the PID while creating it? handle identifier-ASCII
- [x] **LandingPage**
can temporarily implement with sensor.community, later sensordata.open-forecast.eu/devices/{PID}
- [x] _**Name**_
implemented as `sensor_type`. Maybe this should be sensor_id instead.
- [x] _Description_
should contain link to datasheet and maybe description taken from datasheet
- [x] _**Manufacturers**_
dtr reads (https://dtr-test.pidconsortium.eu/#objects/21.T11148/7adfcd13b3b01de0d875):
> The instrument's manufacturer(s) or developer. This may also be the owner for custom build instruments
Unfortunately mandatory, could be derived from `sensor_type`. adapt graphdb model first. Also derive from sensor_type
- [x] _Model_
> Name of the model or type of device as attributed by the manufacturer.
we could implement this with `sensor_type` as well, see if we have Identifiers for some.
- [x] **Owners**
> Institution(s) responsible for the management of the instrument. This may include the legal owner, the operator, or an institute providing access to the instrument.
- ownerName: Sensor Community
ownerContact: contact@open-forecast.eu
Form to acquire:
- ownerName: Scientist: ...
ownerContact: ...
which email?
- [x] _InstrumentType_
Low Priority, maybe later categorize our sensor types. Just a unicode string
- [x] _MeasuredVariables_
Again, adapt graphdb model first and derive from `sensor_type` / csv
- [x] Dates
implemented with `first_msg`, `last_msg`
- [x] ~~RelatedIdentifiers~~
~~See Thesis, some uses. isSibling will be the only implemented relationship.~~
Is getting redesigned
- [x] ~~AlternateIdentifiers~~
unused
~~implemented as `sensor_id`
change to other-identifier~~
Implemented in graphdb as part of SensorType:
- Name
- Description
- InstrumentType
- MeasuredVariables
- Manufacturers
<details><summary>Descriptions</summary>
- https://www.bosch-sensortec.com/products/environmental-sensors/humidity-sensors-bme280/
- https://www.mouser.de/ProductDetail/Bosch-Sensortec/BMP180?qs=d72FGnIDsgTlLIC5YM2WKA==
- https://www.bosch-sensortec.com/products/environmental-sensors/pressure-sensors/bmp280/
- https://www.mikrocontroller-elektronik.de/wp-content/uploads/delightful-downloads/2015/05/DHT22.pdf
- https://www.maximintegrated.com/en/products/sensors/DS18B20.html?intcid=para
- https://www.mouser.de/datasheet/2/187/honeywell-sensing-particulate-hpm-series-datasheet-1568364.pdf
- https://www.mouser.de/datasheet/2/418/5/NG_DS_HPC199_6_A1-1128627.pdf
- http://www.plantower.com/en/list/?118_1.html
- https://www.researchgate.net/publication/324171671_Understanding_the_Shinyei_PPD42NS_low-cost_dust_sensor
- https://rhelectronics.net/store/sbm20-geiger-muller-counter-tube.html
- http://www.sovtube.com/content/2-geiger-counter
- https://multigeiger.readthedocs.io/en/latest/setup.html
- https://learn.watterott.com/de/sensors/sds011/sds011.pdf
- https://www.sensirion.com/de/umweltsensoren/feinstaubsensoren-pm25/
</details>Jakob Leonard HördtJakob Leonard Hördthttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/17days without measurements don't get tar'd2021-09-06T20:37:03ZJakob Leonard Hördtdays without measurements don't get tar'dbecause then the folder is never created and tar can't find itbecause then the folder is never created and tar can't find ithttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/13delete csv folders after archiving2021-11-15T17:15:32ZJakob Leonard Hördtdelete csv folders after archivinghttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/12detached java/gradle version in CI2021-03-05T21:22:13ZJakob Leonard Hördtdetached java/gradle version in CIhttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/20duplicate measured variables in pids2021-10-05T15:58:51ZJakob Leonard Hördtduplicate measured variables in pidshttps://hdl.handle.net/api/handles/21.11138/a6442237-381e-4e97-92f7-74b7749350da?noredirect
at 2021-09-29 04:04 PM UTC:
```json
{"responseCode":1,"handle":"21.11138/a6442237-381e-4e97-92f7-74b7749350da","values":[{"index":1,"type":"21.T1...https://hdl.handle.net/api/handles/21.11138/a6442237-381e-4e97-92f7-74b7749350da?noredirect
at 2021-09-29 04:04 PM UTC:
```json
{"responseCode":1,"handle":"21.11138/a6442237-381e-4e97-92f7-74b7749350da","values":[{"index":1,"type":"21.T11148/8eb858ee0b12e8e463a5","data":{"format":"string","value":"{\"identifier-general-with-type\":{\"identifierValue\":\"21.11138/a6442237-381e-4e97-92f7-74b7749350da\", \"identifierType\":\"Handle\"}}"},"ttl":86400,"timestamp":"2021-04-19T10:06:49Z"},{"index":2,"type":"21.T11148/22c62082a4d2d9ae2602","data":{"format":"string","value":"{\"Dates\":[{\"date\":{\"dateValue\":\"2017-03-25\",\"dateType\":\"Commissioned\"}},{\"date\":{\"dateValue\":\"2021-09-28\"}}]}"},"ttl":86400,"timestamp":"2021-09-29T04:14:35Z"},{"index":3,"type":"21.T11148/709a23220f2c3d64d1e1","data":{"format":"string","value":"\"1337\""},"ttl":86400,"timestamp":"2021-04-18T16:22:43Z"},{"index":4,"type":"21.T11148/c1a0ec5ad347427f25d6","data":{"format":"string","value":"{\"Model\":{\"modelName\":\"null\", \"modelIdentifier\":{\"modelIdentifierValue\":\"null\"}}}"},"ttl":86400,"timestamp":"2021-09-29T04:14:35Z"},{"index":5,"type":"21.T11148/9a15a4735d4bda329d80","data":{"format":"string","value":"\"https://sensor.community\""},"ttl":86400,"timestamp":"2021-04-18T16:22:43Z"},{"index":6,"type":"21.T11148/4eaec4bc0f1df68ab2a7","data":{"format":"string","value":"{\"Owners\":[{\"owner\":{\"ownerName\":\"SensorCommunity\",\"ownerContact\":\"contact@open-forecast.eu\"}}]}"},"ttl":86400,"timestamp":"2021-09-16T17:53:05Z"},{"index":7,"type":"21.T11148/1f3e82ddf0697a497432","data":{"format":"string","value":"{\"Manufacturers\":[{\"manufacturer\":{\"manufacturerName\":\"null\"}}]}"},"ttl":86400,"timestamp":"2021-09-29T04:14:35Z"},{"index":9,"type":"URL","data":{"format":"string","value":"https://sensordata.open-forecast.eu/"},"ttl":86400,"timestamp":"2021-04-19T10:06:49Z"},{"index":10,"type":"21.T11148/72928b84e060d491ee41","data":{"format":"string","value":"{\"MeasuredVariables\":[{\"measuredVariable\":\"temperature\"},{\"measuredVariable\":\"temperature\"},{\"measuredVariable\":\"humidity\"},{\"measuredVariable\":\"humidity\"}]}"},"ttl":86400,"timestamp":"2021-09-29T04:14:35Z"},{"index":100,"type":"HS_ADMIN","data":{"format":"admin","value":{"handle":"21.11138/USER01","index":300,"permissions":"110011111111"}},"ttl":86400,"timestamp":"2021-04-18T16:22:43Z"}]}
```https://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/16figure out more descriptive names for measured variables in PIDs2021-10-06T10:50:34ZJakob Leonard Hördtfigure out more descriptive names for measured variables in PIDshttps://github.com/opendata-stuttgart/meta/wiki/APIs:
> PMS1003 - PMS 7003: value_types: P1 (PM10), P2 (PM2.5); X-Pin: 1
|value reported in CSV |determined meaning |
|------ |--------- |
|N05 ...https://github.com/opendata-stuttgart/meta/wiki/APIs:
> PMS1003 - PMS 7003: value_types: P1 (PM10), P2 (PM2.5); X-Pin: 1
|value reported in CSV |determined meaning |
|------ |--------- |
|N05 | |
|N1 | |
|N10 | |
|N25 | |
|N4 | |
|P1 |PM10 |
|P4 | |
|P2 |PM2.5 |
|P0 | |
|TS | |
|altitude | |
|counts | |
|counts_per_minute | |
|durP1 | |
|durP2 | |
|humidity | |
|hv_pulses | |
|noise_LA01 | |
|noise_LA95 | |
|noise_LA_max | |
|noise_LA_min | |
|noise_LAeq | |
|pressure | |
|pressure_sealevel | |
|ratioP1 | |
|ratioP2 | |
|sample_time_ms | |
|temperature | |
|tube | |https://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/21import csv into influxdb2021-12-15T16:45:56ZJakob Leonard Hördtimport csv into influxdb## implementation options:
### invoke data-importer with each single uploaded csv after it has been saved
- easy to do, requires almost no changes to data-importer
- needs to establish influxdb connection for every csv file -> inefficien...## implementation options:
### invoke data-importer with each single uploaded csv after it has been saved
- easy to do, requires almost no changes to data-importer
- needs to establish influxdb connection for every csv file -> inefficient
### invoke data-importer with an entire day and when uploading failed files
- a little harder to do business logic wise, currently mainly using for object do operation paradigm
- conserves db connection much better
### implement with influxdb java driver
- conserves db connection perfectly
- significant development effort, especially because java api is much worse than python's. Only allows line protocol or single points afaict https://github.com/influxdata/influxdb-client-java/tree/master/client#synchronous-blocking-api.
- unified error handling
### run data-importer with cron without any connection to sensor2graph
- would need to implement getting date
- easily leaks days if there are ever problems with sensor2graphhttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/14improve archive completeness2021-03-07T20:08:25ZJakob Leonard Hördtimprove archive completenesshttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/3Improve GraphDB model by splitting off MeasurementLocation from Sensor nodes2021-03-07T20:07:04ZJakob Leonard HördtImprove GraphDB model by splitting off MeasurementLocation from Sensor nodes## Current:
![current model where Sensor node stores coordinates and times](/uploads/d97213e72cffd84851386a1ef72bce89/actualv2.svg "current model")
## Desired:
![desired model where Sensor nodes MEASURED_IN MeasurementLocation](/upload...## Current:
![current model where Sensor node stores coordinates and times](/uploads/d97213e72cffd84851386a1ef72bce89/actualv2.svg "current model")
## Desired:
![desired model where Sensor nodes MEASURED_IN MeasurementLocation](/uploads/ae80b5d2b29fce6e29f76ae616e5cc8d/graphdb-schema-pid-per-sensor.svg "desired model")Jakob Leonard HördtJakob Leonard Hördthttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/11kaniko does not get upgraded2021-05-27T11:46:29ZJakob Leonard Hördtkaniko does not get upgradedhttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/8neo4j transactions failing2021-03-03T15:11:12ZJakob Leonard Hördtneo4j transactions failinglocally as welllocally as wellhttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/26new sensortypes like nextpm unsupported2023-01-25T20:32:36ZJakob Leonard Hördtnew sensortypes like nextpm unsupportedhttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/18no matching files in CI since directory structure change2021-09-29T14:51:39ZJakob Leonard Hördtno matching files in CI since directory structure changehttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/19nulls in pids2021-09-29T16:26:51ZJakob Leonard Hördtnulls in pidshttps://hdl.handle.net/api/handles/21.11138/a6442237-381e-4e97-92f7-74b7749350da?noredirect
at 2021-09-29 04:04 PM UTC:
```json
{"responseCode":1,"handle":"21.11138/a6442237-381e-4e97-92f7-74b7749350da","values":[{"index":1,"type":"21.T1...https://hdl.handle.net/api/handles/21.11138/a6442237-381e-4e97-92f7-74b7749350da?noredirect
at 2021-09-29 04:04 PM UTC:
```json
{"responseCode":1,"handle":"21.11138/a6442237-381e-4e97-92f7-74b7749350da","values":[{"index":1,"type":"21.T11148/8eb858ee0b12e8e463a5","data":{"format":"string","value":"{\"identifier-general-with-type\":{\"identifierValue\":\"21.11138/a6442237-381e-4e97-92f7-74b7749350da\", \"identifierType\":\"Handle\"}}"},"ttl":86400,"timestamp":"2021-04-19T10:06:49Z"},{"index":2,"type":"21.T11148/22c62082a4d2d9ae2602","data":{"format":"string","value":"{\"Dates\":[{\"date\":{\"dateValue\":\"2017-03-25\",\"dateType\":\"Commissioned\"}},{\"date\":{\"dateValue\":\"2021-09-28\"}}]}"},"ttl":86400,"timestamp":"2021-09-29T04:14:35Z"},{"index":3,"type":"21.T11148/709a23220f2c3d64d1e1","data":{"format":"string","value":"\"1337\""},"ttl":86400,"timestamp":"2021-04-18T16:22:43Z"},{"index":4,"type":"21.T11148/c1a0ec5ad347427f25d6","data":{"format":"string","value":"{\"Model\":{\"modelName\":\"null\", \"modelIdentifier\":{\"modelIdentifierValue\":\"null\"}}}"},"ttl":86400,"timestamp":"2021-09-29T04:14:35Z"},{"index":5,"type":"21.T11148/9a15a4735d4bda329d80","data":{"format":"string","value":"\"https://sensor.community\""},"ttl":86400,"timestamp":"2021-04-18T16:22:43Z"},{"index":6,"type":"21.T11148/4eaec4bc0f1df68ab2a7","data":{"format":"string","value":"{\"Owners\":[{\"owner\":{\"ownerName\":\"SensorCommunity\",\"ownerContact\":\"contact@open-forecast.eu\"}}]}"},"ttl":86400,"timestamp":"2021-09-16T17:53:05Z"},{"index":7,"type":"21.T11148/1f3e82ddf0697a497432","data":{"format":"string","value":"{\"Manufacturers\":[{\"manufacturer\":{\"manufacturerName\":\"null\"}}]}"},"ttl":86400,"timestamp":"2021-09-29T04:14:35Z"},{"index":9,"type":"URL","data":{"format":"string","value":"https://sensordata.open-forecast.eu/"},"ttl":86400,"timestamp":"2021-04-19T10:06:49Z"},{"index":10,"type":"21.T11148/72928b84e060d491ee41","data":{"format":"string","value":"{\"MeasuredVariables\":[{\"measuredVariable\":\"temperature\"},{\"measuredVariable\":\"temperature\"},{\"measuredVariable\":\"humidity\"},{\"measuredVariable\":\"humidity\"}]}"},"ttl":86400,"timestamp":"2021-09-29T04:14:35Z"},{"index":100,"type":"HS_ADMIN","data":{"format":"admin","value":{"handle":"21.11138/USER01","index":300,"permissions":"110011111111"}},"ttl":86400,"timestamp":"2021-04-18T16:22:43Z"}]}
```https://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/24program never recovers when terminated unexpectedly2023-01-22T21:05:10ZJakob Leonard Hördtprogram never recovers when terminated unexpectedlyjust do what the program already suggests: load failed files from backupjust do what the program already suggests: load failed files from backuphttps://gitlab.gwdg.de/j.hoerdt/sensor2graph/-/issues/5Refactor to improve readability to an acceptable level2021-10-06T10:50:53ZJakob Leonard HördtRefactor to improve readability to an acceptable levelCode quality was not a priority during the initial development and will now have to get some attention.
I imagine it must be possible to have a piece of code similar to this:
```java
public void process_csv(URL url) {
var raw_conten...Code quality was not a priority during the initial development and will now have to get some attention.
I imagine it must be possible to have a piece of code similar to this:
```java
public void process_csv(URL url) {
var raw_content = download_csv(url);
save_to_disc(url);
var parsed_content = parse_csv(raw_content);
var info_from_graphdb = update_graphdb(head(parsed_content));
update_pid_registry(info_from_graphdb);
// as soon as influx import is merged:
update_influxdb(parsed_content);
}
```Jakob Leonard HördtJakob Leonard Hördt