Skip to content

GitLab

Last edited by domenico.velotto
Page history

AMMOD Data portal documentation

Portal production and staging systems

The AMMOD data portal provides:

  • access to the (meta-)data produced in the pilot phase of the project
  • visualisation of the deployed stations, incl. their status and telemetry data

The data portal consists of two separate systems, a production system and a staging system. Both run the same code base and offer the same functionality with some significant differences, described below.

Make sure to use the correct system according to your needs - if in doubt, please contact Domenico Velotto.

Production System

  • data.ammod.de
  • A dedicated storage space of 5TB attached to the production system is replicated in two distinct locations (Gießen & Göttingen).
  • The production system is dedicated to managing the data collected by the deployed AMMOD devices.
  • The system accepts only uploads of data produced by a pre-registered device in the sensor registry. In case of doubt, please refer to the Wiki page Use cases workflow and have a look at Initial registration of a Sensor.
  • All devices are managed via the production system of the AWI sensor registry. Please note, the production and staging system use different instances of the sensor registry, therefore the deviceIDs will differ.
  • IMPORTANT: DO NOT use the production system to test the upload service. Use the staging system instead.

Staging System

  • IMPORTANT: DO NOT use the staging system to upload valuable data! Data there is not backed up and can be deleted anytime.
  • ammod.gfbio.dev
  • The storage space is significantly smaller (100GB) and there is no backup! In addition, due to space constraints, development tasks, etc. data might be deleted at any time.
  • The staging system is used for internal development and testing the automated upload (by the base station) or manual upload (by users with upload permission).
  • All devices are managed via the staging system of the AWI sensor registry. Please note, the AMMOD production and staging systems use different instances of the sensor registry, therefore the deviceIDs will differ.
  • List of deviceID per location and sensor type currently used for testing uploads to the staging system:
  • Base station Melbgarten
    • Smellscape -> deviceID 7979
    • StereoCam -> deviceID 7980
    • Bioacoustic (inactive) -> deviceID 7975
  • Base station Britz
    • 4Ch bioacoustic -> deviceID 7977
    • Ultrasound bioacoustic -> deviceID 7978
  • Get in touch with Domenico Velotto to request additional deviceIDs should your configuration be different from the list above, e.g. smellscape at "Energieberg".

Functionality description

The following describes the functionality of the AMMOD portal. The access level is described as follows:

  • Public: no account or login required
  • Ammod-members: login and extra permission required to search, view, and download data (see “Login and access”)
  • Ammod-uploaders: login and extra permission required to upload data (see “Login and access”)

There are 4 main categories:

  • Home [public]: it provides the geographical map of the AMMOD base stations network. The service automatically fetches the metadata of the base stations that have been previously stored in the sensor registry and assigned to the collection AMMOD. By clicking the pin of the base station, users get a list of installed sensors and a direct link to their metadata. In addition, a useful shortcut search button allows the users to retrieve all data collected by the selected base station.
  • Telemetry [public]: in the sub-menu Dashboard, users can create widgets to plot and visualize a specific parameter of the uploaded telemetry data.
  • Data and Metadata [ammod-members]: it gives access to the API UI for the Search and Upload services.
    • Search [ammod-members]: By default is listing all the data available in the database with pagination of 20 results per page. The list of results can be refined by providing user-defined values for the parameters in the filter page: time frame, location (base station based), data type (raw, processed) and sensors. With the advanced filter, users can specify a specific key value pair to search for. By default all entries matching the query are selected and the total size of the download is displayed. By clicking the download button, the service starts to pack the selected data in a zip archive. If the packaging time is in the order of 20 seconds the user will be directly prompted with the zip data file, otherwise a notification email will inform the user when the zip file is ready to download. Clicking the share icon will copy the filter setting and the string will be formatted as a shareable URL.
    • Upload [ammod-uploaders]: users can upload data collected manually using this UI instead of the command line. The UI accepts single uploads (either one metadata file + data file or one telemetry file) or a folder containing multiple uploads, e.g. a series of measurements). In the latter case, files that do not belong to an upload will be automatically discarded. It is important to note that both metadata and telemetry uploads via the UI have to be compliant with the schemas (see the schema and example repository). The backend of the UI is based on the same API used for automated data transfer. To have more information on the possible responses, please visit the API documentation in the section Tools. IMPORTANT NOTE: The permission to upload data is handled separately from the permissions to search, view and download data. Please contact Domenico Velotto to get the upload permission.
  • Tools [public]: the section is dedicated to grouping useful tools and documentation.
    • JSON Validation [public]: This is a practical UI to validate metadata and telemetry data against the implemented schemas. Alternative methods to validate JSON files, like using the API in the command line or a python script, are described in the schema and example repository.
    • API Documentation [public]: The API documentation is streamlined using the Swagger UI. Each service lists the implemented HTTP verbs, contains a short description, the list of parameters and the dedicated response code. Users are encouraged to get familiar with the AMMOD services by testing the API using the interactive console accessible with the “Try it out” button available in each HTTP verb.

Log in and access

Both the production and stage systems use the GFBio SSO service for authentication. You can login using:

  • An existing GFBio account (‘Login with your GFBio Account’)
  • An existing GWDG account (‘Login with your GFBio Account’)
  • Your institutional account, provided your institution is part of the DFN AAI (‘Login using DFN AAI’)
  • Create a new GFBio account (follow the link in the SSO window)

IMPORTANT:

  • If you already have a GFBio account, DO NOT use the option ‘Import your existing GFBio account’. This functionality is deprecated and will be removed soon.
  • After logging in, your account does not have permissions to search, view, download or upload data. Please contact Domenico Velotto to get access. The portal will also automatically detect if you don’t have permissions to interact with the data (you should see a lock on the services where access is not granted) and clicking on it pop-up a message to request access from there. Please note that granting permissions might require an authorization by an AMMOD PI.
  • You can log in to both systems (production and staging) with the same account, however the permissions mentioned above are managed separately. Therefore, you need to log in to both systems and request them separately.

Providing Feedback

Please report any problems by sending an email to the support