feat(ci): data import via GitLab CI Job Service Container

This MR allows for using CI Jobs to update data on all instances. Therefore it introduces two new CI Stages: data_import and data_upload. The import runs in parallel based on a defined list of language aggregations (currently 5). This will allow for checking possible error messages during import and also download and check import results, as they are now stored as job artifact. the data upload stage then will send all the data to respective target instances.

Todo: It is still to define when to start a complete import and what instance(s) to target.

It leaves the app as it is with small changes in two sections sensitive to the presences of an env var CI_SERVICE.

post installation

During post-install a cleanup and non-recursive import of main and lang editions is triggered. These resources are required for the instance to provide complete data, esp. via TextAPI (not used so far, but considered).

document trigger

The trigger is only active when the env var CI_SERVICE is present to detect an ephemeral instance. This allows for running instances (targets) to receive the data without re-rendering the resources.

smaller changes

To not reach the limit of CI Job logs (4194304 bytes.) the still useful stdout output of commons:debug#0 is only returned when memory usage is more than half of available maximum.

minor typos.

ci-scripts/resource-downloader.sh

+#!/bin/bash
+# load all transformation results from service (mainly the GitLab CI job's service container)
+# PUT the results to a target instance
+
+# parsing incoming parameter, see: https://stackoverflow.com/a/14203146
+POSITIONAL_ARGS=()
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    -s|--service)
+      SERVICE="$2"
+      REST="${SERVICE}/exist/rest/data/textgrid"
+      shift # past argument
+      shift # past value
+      ;;
+    -uri|--textgrid-uri)
+      TG_URI="$2"
+      shift # past argument
+      shift # past value
+      ;;
+    -t|--target)
+      TARGET="$2"
+      shift # past argument
+      shift # past value
+      ;;
+    -p|--target-password)
+      EXIST_ADMIN_PW="$2"
+      shift # past argument
+      shift # past value
+      ;;
+    --help)
+      printf ' 🚀 import a textgrid edition and pass all import artifacts to a target instance \n\n parameter:\n -s|--service where to start the import\n -uri|--textgrid-uri the object to import recursively\n -t|--target where to PUT the data\n -p|--target-password the admin password for the target instance'
+      exit 0
+      ;;
+    -*|--*)
+      printf " 🧨\tUnknown option $1\n\trun with --help for more usage instructions"
+      exit 1
+      ;;
+    *)
+      POSITIONAL_ARGS+=("$1") # save positional arg
+      shift # past argument
+      ;;
+  esac
+done
+
+# end script on failing command (exit code)
+set -e
+
+TOKEN="${APP_DEPLOY_TOKEN:-1234}"
+
+printf " ⏱️ waiting for service to be up and running…"
+until (curl -I -s "${SERVICE}"); do printf "…" && sleep 1; done
+curl --silent "${SERVICE}/exist/restxq/import-data/${TG_URI}?token=${TOKEN}&cleanup=false&recursive=true"
+
+# since the service starts with empty transformation results collections, we can load all resources
+
+# prepare
+TMPDIR="./export/${TG_URI/textgrid:/}"
+mkdir --parents ${TMPDIR}
+cd ${TMPDIR} && echo ${PWD}
+
+# load all resources form tmp, data and html
+COLLECTIONS=( "agg" "data" "html" "meta" "json" "tmp" )
+for collection in ${COLLECTIONS[@]}; do
+    mkdir ${collection}
+    for i in $(curl --silent ${REST}/${collection} | xmlstarlet sel -t -v "//exist:resource/@name"); do
+    curl --silent ${REST}/${collection}/${i} -o ${collection}/${i}
+    done
+done