Skip to content
  • Clemens Wolff's avatar
    Switch Azure deployment to Kubernetes (#1301) · e630eb7e
    Clemens Wolff authored
    Previously we were hosting the ok-server, ok-worker, autopy-manager and
    autopy-worker in AppService for containers. However, we were seeing some
    issues with this approach as workers were killed without death
    notifications and rq/Redis was getting confused. After this change, we
    now deploy the compute (workers and servers) to Azure Kubernetes
    service.
    
    In order to make it easier to orchestrate the setup of the various
    resources in Azure as well as the Kubernetes deployments, we're using
    bash scripts. First, we create a bunch of resources using `setup.sh` and
    then we configure them using `install.sh`. The setup scripts are wrapped
    in a Docker container so that the environment in which the scripts
    execute is well known, e.g. Azure CLI is installed, helm is available,
    kubectl is available, etc.
    
    Most of the setup scripts fall-back to ARM templates to create the
    resources in Azure. The ARM templates can be modified to configure
    parameters such as resource SKUs, number of VMs to spin up for the
    cluster, type of VMs to use in the cluster, etc.
    
    The install scripts perform operations such as creating the database
    schema, deploying the application to Kubernetes via Helm and setting up
    the DNS for the Kubernetes ingress via the Azure CLI so that kube-lego
    can configure a TLS certificate for the ingress via Let's Encrypt.
    
    Note that the ingress to the applications running in the Kubernetes
    cluster are configured as just one entrypoint so that a single TLS
    certificate and DNS entry can be used (Azure public IP resource
    currently only support a single DNS label per IP). To solve routing, we
    thus need to mount the two sub-applications (OKpy and AutoPY) at
    sub-urls of the ingress which requires some minor application changes
    which are submitted in a separate pull request:
    https://github.com/okpy/ok/pull/1300
    
    The setup scripts can be run either via the "Deploy to Azure" button (in
    which case the setup Docker container is run in an Azure Container
    Instance) or via a local `docker run` command. In the latter case, it's
    advised to also include a volume mount `docker run -v $PWD:/app/secrets`
    so that the access keys for the resources created during the deployment
    will be made available to the deployer. Note that in order to skip
    deploying specific resources (e.g. to re-use existing resources), you
    can place a file into the secrets folder that contains the access keys
    for your existing resources in which case the setup scripts will skip
    deploying that step.
    e630eb7e