Deployment issueshttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues2021-12-15T15:16:06Zhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/64Debian forwards mail to root to default user2021-12-15T15:16:06ZIlka SchulzDebian forwards mail to root to default userThe Debian installer lets you create a user and then sets this `/etc/aliases`:
```text
# /etc/aliases
mailer-daemon: postmaster
postmaster: root
nobody: root
hostmaster: root
usenet: root
news: root
webmaster: root
www: root
ftp: root
ab...The Debian installer lets you create a user and then sets this `/etc/aliases`:
```text
# /etc/aliases
mailer-daemon: postmaster
postmaster: root
nobody: root
hostmaster: root
usenet: root
news: root
webmaster: root
www: root
ftp: root
abuse: root
noc: root
security: root
root: ilka
logcheck: root
```
Obviously, mail to `root` shall not be forwarded to that user but to a specified one.
###### Definition of Done:
- [ ] adjust `prepare-debian.sh` to remove that line
###### Hints:
- none...Ilka SchulzIlka Schulzhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/53ecdsa-sha2-nistp256 used for deployment2021-10-11T09:51:58ZIlka Schulzecdsa-sha2-nistp256 used for deployment> NIST P-curves are possibly back-doored by the U.S. National Security Agency.
sshaudit.com
###### Definition of Done:
- [ ] replace by secure algorithm
###### Hints:
- That SSH server was set up by GWDG Cloud provider and is hard to r...> NIST P-curves are possibly back-doored by the U.S. National Security Agency.
sshaudit.com
###### Definition of Done:
- [ ] replace by secure algorithm
###### Hints:
- That SSH server was set up by GWDG Cloud provider and is hard to reconfigure. However, the client may choose a secure algorithm anyways.
- several occurences in https://gitlab.rz.htw-berlin.de/snet-asclepios-demo/snet-asclepios-deployment/-/blob/deploy-to-gwdg-vps/.gitlab-ci.yml#L42https://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/52passwordless sudo on GWDG VMs2021-10-11T09:36:10ZIlka Schulzpasswordless sudo on GWDG VMsOur CI pipeline currently assumes passwordless sudo on both our GWDG Cloud Server and our GWDG Virtual Server which is bad practice.
```
$ sudo cat /etc/sudoers.d/gitlab-ci
gitlab ALL=(ALL) NOPASSWD:ALL
```
###### Definition of Done:
-...Our CI pipeline currently assumes passwordless sudo on both our GWDG Cloud Server and our GWDG Virtual Server which is bad practice.
```
$ sudo cat /etc/sudoers.d/gitlab-ci
gitlab ALL=(ALL) NOPASSWD:ALL
```
###### Definition of Done:
- [ ] allow only certain commands/scripts to run passwordless
- [ ] put all necessary commands into those scripts
- [ ] remove passwordless sudo configuration from all machines
- [ ] test
###### Hints:
- none...https://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/50cpabe-server: no such file: cpabe/demo/cpabe/decReceivedSSEencKey2021-10-05T17:27:31ZIlka Schulzcpabe-server: no such file: cpabe/demo/cpabe/decReceivedSSEencKey<details>
<summary>Logs from GWDG Cloud Server</summary>
```text
2021-09-29 10:16:07.448 INFO 1 --- [io-8084-exec-10] e.u.a.service.impl.SSEServiceImpl : Request to fetch userid
2021-09-29 10:16:07.448 INFO 1 --- [io-8084-exec...<details>
<summary>Logs from GWDG Cloud Server</summary>
```text
2021-09-29 10:16:07.448 INFO 1 --- [io-8084-exec-10] e.u.a.service.impl.SSEServiceImpl : Request to fetch userid
2021-09-29 10:16:07.448 INFO 1 --- [io-8084-exec-10] e.u.a.service.impl.SSEServiceImpl : Fetching access token first
2021-09-29 10:16:07.490 INFO 1 --- [nio-8084-exec-2] e.u.a.service.impl.SSEServiceImpl : Request to fetch userid
2021-09-29 10:16:07.490 INFO 1 --- [nio-8084-exec-2] e.u.a.service.impl.SSEServiceImpl : Fetching access token first
2021-09-29 10:16:07.563 INFO 1 --- [io-8084-exec-10] e.u.a.service.impl.SSEServiceImpl : Token received successfully
2021-09-29 10:16:07.568 INFO 1 --- [nio-8084-exec-2] e.u.a.service.impl.SSEServiceImpl : Token received successfully
2021-09-29 10:16:07.638 INFO 1 --- [io-8084-exec-10] e.u.a.service.impl.SSEServiceImpl : user-id fetched successfully
2021-09-29 10:16:07.642 INFO 1 --- [nio-8084-exec-2] e.u.a.service.impl.SSEServiceImpl : user-id fetched successfully
//start to dec SSEencKey
//start to dec SSEencKey
e = {x=7524289778303053449634668001685356177848404444903472387128100972881745276882455215946622448428025255996828617167576523337754566531709551195647878622006149,y=7217027477341853557016189860819350455832729611283231474320101234689347536872129297306517507226913097498811930839624929293359609411546529421400679560416051}
//end to dec SSEencKey
//start to dec SSEverKey
e = {x=7524289778303053449634668001685356177848404444903472387128100972881745276882455215946622448428025255996828617167576523337754566531709551195647878622006149,y=7217027477341853557016189860819350455832729611283231474320101234689347536872129297306517507226913097498811930839624929293359609411546529421400679560416051}
//end to dec SSEencKey
//start to dec SSEverKey
e = {x=5973482489080380463466626190761019541335443540031266541809045977161740753267763975624188951638571576155628178661431826061265926957375910487741839834908697,y=301237108240397097734535776598554794874643397666057899786728150189997280555478518031972994261312500418449000999341639878319967521235415212154919154406955}
//end to dec SSEverKey
e = {x=5973482489080380463466626190761019541335443540031266541809045977161740753267763975624188951638571576155628178661431826061265926957375910487741839834908697,y=301237108240397097734535776598554794874643397666057899786728150189997280555478518031972994261312500418449000999341639878319967521235415212154919154406955}
//end to dec SSEverKey
java.nio.file.NoSuchFileException: cpabe/demo/cpabe/decReceivedSSEencKey
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at java.nio.file.Files.newBufferedReader(Files.java:2784)
at java.nio.file.Files.lines(Files.java:3744)
at co.junwei.cpabe.Demo.readFile(Demo.java:201)
at co.junwei.cpabe.Demo.tryToDecrypt(Demo.java:122)
at eu.ubitech.asclepios.service.impl.SSEServiceImpl.getSSEKeys(SSEServiceImpl.java:218)
at eu.ubitech.asclepios.rest.SSERestController.getKeys(SSERestController.java:72)
at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticatedActionsFilter.doFilter(KeycloakAuthenticatedActionsFilter.java:57)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.keycloak.adapters.springsecurity.filter.KeycloakSecurityContextRequestFilter.doFilter(KeycloakSecurityContextRequestFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.keycloak.adapters.springsecurity.filter.KeycloakPreAuthActionsFilter.doFilter(KeycloakPreAuthActionsFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticationProcessingFilter.successfulAuthentication(KeycloakAuthenticationProcessingFilter.java:214)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:240)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticatedActionsFilter.doFilter(KeycloakAuthenticatedActionsFilter.java:74)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.keycloak.adapters.springsecurity.filter.KeycloakSecurityContextRequestFilter.doFilter(KeycloakSecurityContextRequestFilter.java:92)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticationProcessingFilter.successfulAuthentication(KeycloakAuthenticationProcessingFilter.java:214)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:240)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.keycloak.adapters.springsecurity.filter.KeycloakPreAuthActionsFilter.doFilter(KeycloakPreAuthActionsFilter.java:96)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:92)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.keycloak.adapters.tomcat.AbstractAuthenticatedActionsValve.invoke(AbstractAuthenticatedActionsValve.java:67)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.keycloak.adapters.tomcat.AbstractKeycloakAuthenticatorValve.invoke(AbstractKeycloakAuthenticatorValve.java:181)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
```
</details>
###### Definition of Done:
- [ ] identify source
- [ ] fix it
- [ ] test
###### Hints:
- Log is an excerpt from `docker logs cpabe-server` on `141.5.102.110`.https://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/49evaluate generated asymmetric keys2021-09-17T10:19:13ZIlka Schulzevaluate generated asymmetric keysKey generation in `bin/generate-certs` is very optimistic:
```sh
KEY_ALG=RSA
KEY_SIZE=2048
VALIDITY=3650
```
###### Definition of Done:
- [ ] fix `bin/generate-certs`
- [ ] find other certificates with bad choices
- [ ] generate furthe...Key generation in `bin/generate-certs` is very optimistic:
```sh
KEY_ALG=RSA
KEY_SIZE=2048
VALIDITY=3650
```
###### Definition of Done:
- [ ] fix `bin/generate-certs`
- [ ] find other certificates with bad choices
- [ ] generate further issues / add tasks
###### Hints:
- CPABE is a whole topic on its own and does not fit into this issue (see https://gitlab.com/asclepios-project/cpabe-cli/-/issues/2)Ilka SchulzIlka Schulzhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/48Run `test-gwdg` in docker container2021-09-08T08:10:08ZIlka SchulzRun `test-gwdg` in docker containere2e tests require the following dependencies
* python3
* python3-dotenv
We should either build or find an suitable image containing these dependencies already so that we run our tests within a docker container.
###### Definition of D...e2e tests require the following dependencies
* python3
* python3-dotenv
We should either build or find an suitable image containing these dependencies already so that we run our tests within a docker container.
###### Definition of Done:
IDEA 1, (more work, more maintenance, runs faster)
- [ ] either build or find a light weight docker image containing python3 and python3-dotenv
- [ ] run the content of the test-gwdg gitlab-ci job inside a docker container using that image
IDEA 2: (less work, less maintenance, slow because deps install on every run) (blocked by #39)
- [ ] move the content of test-gwdg into it's own script in `/bin`
- [ ] add dependency installation to the new script
- [ ] run that script in a generic docker containing using a generic debian image
##### hints:
Maybe we should just maintain one build/configure/deploy docker image with all our dependencies, then run each stage of the deployment with that imageCloud Deploymenthttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/47run `./generate-env` script in docker container2021-09-08T09:26:57ZIlka Schulzrun `./generate-env` script in docker containerThe `generate-env` script has a number of dependencies.
We should either build or find an suitable image containing these dependencies already so that we run `generate-env` in a docker container
We do have a solution for this. See: htt...The `generate-env` script has a number of dependencies.
We should either build or find an suitable image containing these dependencies already so that we run `generate-env` in a docker container
We do have a solution for this. See: https://gitlab.rz.htw-berlin.de/snet-asclepios-demo/snet-asclepios-deployment/-/blob/master/configure#L56
But we are not using it (by disabling it with the `.configure.hostgen` feature flag.) due to issue #39
###### Definition of Done:
IDEA 1, (more work, more maintenance, runs faster)
- [ ] either build or find a light weight docker image containing dependencies
- [ ] run `./generate-env` inside a docker container using that image
IDEA 2: (less work, less maintenance, slow because deps install on every run) (blocked by #39)
- [x] move the content of `generate-env` into it's own script in `/bin`
- [x] add dependency installation to the new script
- [ ] run that script in a generic docker containing using a generic debian image
##### hints:
Maybe we should just maintain one build/configure/deploy docker image with all our dependencies, then run each stage of the deployment with that imageCloud DeploymentIlka SchulzIlka Schulzhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/45/usr/local/snet-asclepios-deployment/micado/_settings must be manually copied...2021-09-07T09:44:18ZIlka Schulz/usr/local/snet-asclepios-deployment/micado/_settings must be manually copied to GWDG serverThis file contains the settings for the MiCADO API client scripts. It also contains secrets (micado username/password), so it can not be included in the repository.
even worse, it is a very insecure password
###### Definition of Done:
...This file contains the settings for the MiCADO API client scripts. It also contains secrets (micado username/password), so it can not be included in the repository.
even worse, it is a very insecure password
###### Definition of Done:
- [ ] find out if it is possible to set our own MiCADO api credentials
- [ ] find a way to store these credentials securely
- [ ] generate the _settings file on-the-fly during CICloud Deploymenthttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/43run 'start-UoW' job in docker container2021-11-17T10:17:09ZIlka Schulzrun 'start-UoW' job in docker containerUnfortunately the MiCADO management scripts `1-submit-adt.sh`, `2-delete-beast.sh`, `3-check-status.sh` have some dependencies that are not always installed on hosts. Specifically:
* curl
* jq
We should either build or find an suitab...Unfortunately the MiCADO management scripts `1-submit-adt.sh`, `2-delete-beast.sh`, `3-check-status.sh` have some dependencies that are not always installed on hosts. Specifically:
* curl
* jq
We should either build or find an suitable image containing these dependencies already so that we run our MiCADO orchestration within a docker container.
Fortunately, for now we are still installing build, configure and deploy dependencies (including JQ and curl) on the host due to #39 so this is not urgent
###### Definition of Done:
IDEA 1, (more work, more maintenance, runs faster)
- [ ] either build or find a light weight docker image containing curl and jq
- [ ] run the content on start-UoW gitlab-ci job inside a docker container using that image
IDEA 2: (less work, less maintenance, slow because deps install on every run) (blocked by #39)
- [ ] move the content of start-UoW into it's own script in `/bin`
- [ ] add dependency installation to the new script
- [ ] run that script in a generic docker containing using a generic debian image
##### hints:
Maybe we should just maintain one build/configure/deploy docker image with all our dependencies, then run each stage of the deployment with that imageCloud DeploymentIlka SchulzIlka Schulzhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/42UoW MiCADO master uses a self-signed TLS certificate causing an NET::ERR_CERT...2021-09-06T02:02:38ZIlka SchulzUoW MiCADO master uses a self-signed TLS certificate causing an NET::ERR_CERT_AUTHORITY_INVALID# summary
UoW MiCADO master uses a self-signed TLS certificate causing an `NET::ERR_CERT_AUTHORITY_INVALID` in firefox and forcing as to use the `--insecure` flag with curl to interact with the MiCADO API:
eg here: https://gitlab.rz.ht...# summary
UoW MiCADO master uses a self-signed TLS certificate causing an `NET::ERR_CERT_AUTHORITY_INVALID` in firefox and forcing as to use the `--insecure` flag with curl to interact with the MiCADO API:
eg here: https://gitlab.rz.htw-berlin.de/snet-asclepios-demo/snet-asclepios-deployment/-/blob/master/micado/1-submit-adt.sh#L28
# cert
```
NET::ERR_CERT_AUTHORITY_INVALID
Subject: micado-master: Self-signed certificate
Issuer: micado-master: Self-signed certificate
Expires on: 27 Jul 2031
Current date: 6 Sept 2021
PEM encoded chain:
-----BEGIN CERTIFICATE-----
MIIFVjCCAz6gAwIBAgIUK9+cNVclidc0jPnQS1onEONNqAswDQYJKoZIhvcNAQEL
BQAwUzEgMB4GA1UECgwXU2VsZi1zaWduZWQgY2VydGlmaWNhdGUxLzAtBgNVBAMM
Jm1pY2Fkby1tYXN0ZXI6IFNlbGYtc2lnbmVkIGNlcnRpZmljYXRlMB4XDTIxMDcy
OTExMDIyNFoXDTMxMDcyNzExMDIyNFowUzEgMB4GA1UECgwXU2VsZi1zaWduZWQg
Y2VydGlmaWNhdGUxLzAtBgNVBAMMJm1pY2Fkby1tYXN0ZXI6IFNlbGYtc2lnbmVk
IGNlcnRpZmljYXRlMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvpbR
h0KiJGeSD9hAQGH+KzJInnxjr6cI5GPMYfNgxDhuETuQVd3RgLTldyt9/ipli7sS
LHt5951hYmY7EQgHE90J/5eApSnVt5Nvtf1J6484yqmrKueBJZN+d18XCT24hy9F
vOi4HfrTvI5Mtx1ZnQojE7ZPLL0DxFYTRwrFhKBEos4CP+JyQCfQG0MF3CaSeD1/
NsUpP+S7Rvh5b1kOkz9ch+ovu2dlE9JporTZnXH6I7bRuUcjVvXdLApCBmOZuaLn
NiCUvYl5BFOpfokn4GHcFirjbFaVKnxtJmFXRVy1sndPk+AsBLXtr+bbds5nzfaB
dXDEEdSoRFQV6srOFrk9reFNRMjv3LQ0w+JHF4/J1vlQnGLe6rrJnIx6jUCXI0Ml
3RO0qPhPDF3gI4t2e6AmhfiNhyQvGtmX7/K0bEMqMP3Y3V+Ly91Dyn1A/AouA/w+
H05jIp5Vth3vNT5wZVGcU8FqawJ/d3Rfc4kyIPKlpiBNIA4ygModKBLG3Pb0T3w6
33p1ICOMwFHoXkmfx3SnwQroGTd3HQ03XWi2+1TpyeWBRFx90gUDygaRQq9U4hlI
6sVEVfUnYUzeD+dEEKaT0EYalncOfhe5KyIT9xBp+2w9kH6MEJIkjHJtPibHi2Z3
lzXpr4CKI7Sq+6MWGxiafIvsmjc5/GmlUJSDxTUCAwEAAaMiMCAwHgYDVR0RBBcw
FYcEoUofYIINbWljYWRvLW1hc3RlcjANBgkqhkiG9w0BAQsFAAOCAgEAdc8nojIX
2sgS9o7Wi9FYlNXTdVob3Bod6ZtFhH3EoTHSR1C/5r31OGnWdJgsHD+yAJ8Q14vE
Rcdd+2YMdyGu2WaOT4ETHOP6tofUzuCNgIpcKJOmwxjFXYsJhXQxSxUEChqf0nIY
mKie0Y3FbNfF7WpuHjDjgRqT/iZVyhLq8sayuOP1nEkM6NOggsqBSceGuq6NgN2n
9XGLEVspgk5DJjPfyiRzKNhttLp1HDN8hW2fMGIBZ1iADTBh1a1e0VMeWk8bQ9Ht
2R9cVDilhqwDAU4zQzapTGGO+0+1Pr2Ae7lnqnHNXmKOWgIybPH9Dn4Zm8Q43fnO
UCqNKtuwo5GAfDpNKS0L6n/NbDaoK9GSILtCC4c69ATfDDu3AjtPv/d4ivvSWJP3
zUMpF49OX/azmjaywnyvy+dSe01uwjxCkOkZFI0rpi5QkTPFZE9m5ZdQsEDu5ajV
41Ku5C8XjjAnK43XQpqramHSV83QKx16FBEIp/QnPQFWyJ+29XmgfsP7VkEaGWcw
jY9fqg2qj4fmpd6wF0EYdkaDMwY25a4ff3yJrEUADxSdVrmt6A6KYRkwxytc2L7/
Z9loNI9jJtqEQansjQtHUPVSPs1CbvSR6Pofh8xp+09LE0A0KOxPBen+IFI+hh8U
wjeqQoZlNLv3RfLYeTRTh21zsZTaR2EH5bM=
-----END CERTIFICATE-----
```
###### Definition of Done:
- [ ] ask them to use LetsEncrypt like everyone else
###### hints
* more info on this server here: https://gitlab.rz.htw-berlin.de/snet-asclepios-demo/snet-asclepios-deployment/-/wikis/UoW-MiCADO-Masterhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/41Write Performance Tests2021-09-06T01:52:17ZIlka SchulzWrite Performance TestsTODO: introduction
###### Definition of Done:
- [ ] TODO
###### Hints:
- none...TODO: introduction
###### Definition of Done:
- [ ] TODO
###### Hints:
- none...https://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/40Keycloak issues many warnings and on start up2021-09-03T18:19:05ZIlka SchulzKeycloak issues many warnings and on start up## Find out why keycloak complains so much
We previously had many open issues (which I am about to close) for individual warnings issued by keycloak. I am going to organise this under one issue for a few reasons:
1) They were opened a ...## Find out why keycloak complains so much
We previously had many open issues (which I am about to close) for individual warnings issued by keycloak. I am going to organise this under one issue for a few reasons:
1) They were opened a long time ago, and the deployment project has undergone many changes since then
2) I think many of the more serious errors where actually caused by missing environment variables in the test environment, probably due to bad set up instructions
3) Warnings and error messages are less useful without the full context
3) I like seeing the issue counter go down ;)
###### Definition of Done:
- [ ] Analyse output of `docker logs keycloak` and identify any log lines with the level `WARN` or higher
- (I have attached a boot log of a working local keycloak instance below, but you may want to also analyse the server side logs for comparison)
- [ ] Understand the meaning of each warning or error
- [ ] Decide which warnings or errors effects us in any meaningful way, and need to be fixed
- [ ] Extend issue or create new issues for each error
###### Hints:
[keycloak.boot.log](/uploads/242165795505a2b938807603f41f4832/keycloak.boot.log)https://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/39Job Failed #9076 Debian bullseye repos seem to be unreachable from GWDG cloud...2021-09-08T09:20:06ZIlka SchulzJob Failed #9076 Debian bullseye repos seem to be unreachable from GWDG cloud server# preamble
I have been working towards running the entire configuration process inside docker containers, in order to make moving the deployment between hosts and operating systems easier, and allowing us to use standard debian as a run...# preamble
I have been working towards running the entire configuration process inside docker containers, in order to make moving the deployment between hosts and operating systems easier, and allowing us to use standard debian as a runtime environment for most of the bootstrapping tool-chain.
But I have run into the following error while trying to update the containers package list and install packages using the official Debian bullseye repositories,
```
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Err:1 http://security.debian.org/debian-security bullseye-security InRelease
Connection timed out [IP: 151.101.130.132 80]
Err:2 http://deb.debian.org/debian bullseye InRelease
Connection failed [IP: 199.232.138.132 80]
Err:3 http://deb.debian.org/debian bullseye-updates InRelease
Connection failed [IP: 199.232.138.132 80]
Reading package lists...
```
# side note
the Ubuntu (18.04) docker host GWDG Cloud instance seems to be using these unofficial repositoroies
```
http://ftp.gwdg.de/pub/linux/debian/ubuntu bionic InRelease
http://ftp.gwdg.de/pub/linux/debian/ubuntu bionic-updates InRelease
http://ftp.gwdg.de/pub/linux/debian/ubuntu bionic-backports InRelease
http://ftp.gwdg.de/pub/linux/debian/ubuntu bionic-security InRelease
https://download.docker.com/linux/ubuntu bionic InRelease
```
possibly they provide their own repos for bullseye?
# actual failing job
Job [#9092](https://gitlab.rz.htw-berlin.de/snet-asclepios-demo/snet-asclepios-deployment/-/jobs/9092) failed for c07993c5c709a511b6673df46caac4589569577b:
##### hints:
Maybe we should just maintain one build/configure/deploy docker image with all our dependencies, then run each stage of the deployment with that imageIlka SchulzIlka Schulzhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/36configure prints secrets to console even in CD2021-09-15T11:40:02ZIlka Schulzconfigure prints secrets to console even in CD`configure` should not reveal secrets to the GitLab runner and GitLab.`configure` should not reveal secrets to the GitLab runner and GitLab.Cloud DeploymentIlka SchulzIlka Schulzhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/33Dependency Dashboard2022-03-31T04:07:07ZIlka SchulzDependency DashboardThis issue provides visibility into Renovate updates and their statuses. [Learn more](https://docs.renovatebot.com/key-concepts/dashboard/)
## Open
These updates have all been created already. Click a checkbox below to force a retry/re...This issue provides visibility into Renovate updates and their statuses. [Learn more](https://docs.renovatebot.com/key-concepts/dashboard/)
## Open
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
- [ ] <!-- rebase-branch=renovate/registry.gitlab.com-asclepios-project-registration-authority-cpabe -->[Update registry.gitlab.com/asclepios-project/registration-authority-cpabe Docker digest to 1eca65c](!220)
- [ ] <!-- rebase-branch=renovate/registry.gitlab.com-asclepios-project-cpabe_server-2.x -->[Update registry.gitlab.com/asclepios-project/cpabe_server Docker tag to v2.0.7_https](!105)
- [ ] <!-- rebase-branch=renovate/mysql-5.x -->[Update mysql Docker tag to v5.7](!23)
- [ ] <!-- rebase-branch=renovate/postgres-12.x -->[Update postgres Docker tag to v12.10](!24)
- [ ] <!-- rebase-branch=renovate/registry.gitlab.com-asclepios-project-abac-authorization-abac-zuul-proxy-3.x -->[Update registry.gitlab.com/asclepios-project/abac-authorization/abac-zuul-proxy Docker tag to v3.1.0](!33)
- [ ] <!-- rebase-branch=renovate/jboss-keycloak-16.x -->[Update jboss/keycloak Docker tag to v16](!32)
- [ ] <!-- rebase-branch=renovate/mysql-8.x -->[Update mysql Docker tag to v8](!26)
- [ ] <!-- rebase-branch=renovate/postgres-14.x -->[Update postgres Docker tag](!27)
- [ ] <!-- rebase-all-open-prs -->**Click on this checkbox to rebase all open MRs at once**https://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/28Write Functional Tests2021-09-06T01:48:25ZIlka SchulzWrite Functional TestsWe need some basic tests before and after running deployment (no staging server :( )
###### Definition of Done:
- [ ] develop concept for testing submodules
- [x] develop concept for e2e API tests
- [ ] develop concept for browser appli...We need some basic tests before and after running deployment (no staging server :( )
###### Definition of Done:
- [ ] develop concept for testing submodules
- [x] develop concept for e2e API tests
- [ ] develop concept for browser application tests
- [ ] extend issue or write new issues for submodules, etc.
###### Hints:
- Running tests on the submodules would be really nice to early identify bugs, e.g. incompatible renovate upgrades.
- sse-client has a test suit, i just never managed to make it run: https://gitlab.rz.htw-berlin.de/snet-asclepios-demo/asclepios-sse-client-nodeCloud DeploymentIlka SchulzIlka Schulzhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/25very high RAM usage2021-09-15T11:40:36ZIlka Schulzvery high RAM usageThe current RAM usage of the software stack is somewhere around 4.5 GiB which can probably be reduced in manifolds. It might become very important to reduce resource consumption for the purpose of automated tests.
###### Definition of D...The current RAM usage of the software stack is somewhere around 4.5 GiB which can probably be reduced in manifolds. It might become very important to reduce resource consumption for the purpose of automated tests.
###### Definition of Done:
- [ ] identify memory hogs
- [ ] improve
- [ ] test
###### Hints:
- none...Cloud DeploymentIlka SchulzIlka Schulzhttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/22magic IP address in conf-tpl/xnat.sql.tpl2021-06-15T06:46:26ZIlka Schulzmagic IP address in conf-tpl/xnat.sql.tplThere is a hard-coded local IP address in `/conf.tpl/xnat.sql.tpl`:
```text
3 2021-05-31 12:35:02.76 1970-01-01 00:00:00 t 2021-05-31 12:35:02.76 192.168.32.12\n0:0:0:0:0:0:0:1\n127.0.0.1
```
###### Definition of Done:
- [ ] find out wh...There is a hard-coded local IP address in `/conf.tpl/xnat.sql.tpl`:
```text
3 2021-05-31 12:35:02.76 1970-01-01 00:00:00 t 2021-05-31 12:35:02.76 192.168.32.12\n0:0:0:0:0:0:0:1\n127.0.0.1
```
###### Definition of Done:
- [ ] find out what it is for
- [ ] decide how to fix it (at least remove the magic string)
###### Hints:
- There may be many more occasions but this is the only illegal occurence of `192.168` in the project.Automated Testshttps://gitlab.gwdg.de/snet-asclepios-demo/snet-asclepios-deployment/-/issues/8concept: continuous deployment2021-09-15T11:54:07ZIlka Schulzconcept: continuous deploymentWe need to make some conceptual decisions on the continuous deployment part.
**definition of done:**
- [x] decide on access mechanism --> `git push` via SSH and run custom commands per CD job via SSH
- [x] decide on credential storage ...We need to make some conceptual decisions on the continuous deployment part.
**definition of done:**
- [x] decide on access mechanism --> `git push` via SSH and run custom commands per CD job via SSH
- [x] decide on credential storage / signature validation --> SSH password as GitLab CI variable
- [ ] decide on prod/test environments and deployment strategy
- [ ] decide on schedule
- [x] decide on tool (Micado / Ansible / Docker compose / etc.) --> currently Docker compose but may switch to Micado
- [ ] decide on monitoringCloud DeploymentIlka SchulzIlka Schulz