diff --git a/.envs/.local/.django b/.envs/.local/.django index bcde257af73ace06252c947f4adfa4814dfe8d4e..3496f64b10e5adf58b2603b9c73692a5d971e160 100644 --- a/.envs/.local/.django +++ b/.envs/.local/.django @@ -1,4 +1,3 @@ # General # ------------------------------------------------------------------------------ -USE_DOCKER=yes IPYTHONDIR=/app/.ipython diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index abe0dedd1057dc8c5ba11b0b486452023c61c4cf..760e4445f719c4621956004ffc14b5fea30b5e2c 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,7 +1,7 @@ workflow: rules: - if: $CI_MERGE_REQUEST_IID - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH image: docker:19.03 @@ -27,6 +27,7 @@ variables: PIP_REQUIREMENTS_FILE: requirements/production.txt DS_DISABLE_DIND: "true" + stages: - build - test @@ -137,6 +138,19 @@ license_scanning: reports: license_scanning: gl-license-scanning-report.json +container_scanning: + variables: + CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE + CI_APPLICATION_TAG: latest + DOCKERFILE_PATH: compose/production/django/Dockerfile + GIT_STRATEGY: fetch + artifacts: + reports: + container_scanning: gl-container-scanning-report.json + rules: + - if: '$CI_COMMIT_BRANCH == "master"' + when: always + create_release: image: node:10 stage: release diff --git a/README.rst b/README.rst index 3924c29bee39f2bdb492352e6869aba1cfed2aed..6e02f18dcca126f6728126a744755481100c012d 100644 --- a/README.rst +++ b/README.rst @@ -4,21 +4,6 @@ Discuss Data Open Platform for the Interactive Discussion of Research Data Quality (on the example of area studies on the post-Soviet region) -.. image:: https://img.shields.io/badge/built%20with-Cookiecutter%20Django-ff69b4.svg - :target: https://github.com/pydanny/cookiecutter-django/ - :alt: Built with Cookiecutter Django -.. image:: https://img.shields.io/badge/code%20style-black-000000.svg - :target: https://github.com/ambv/black - :alt: Black code style -.. image:: https://img.shields.io/badge/commitizen-friendly-brightgreen.svg - :target: http://commitizen.github.io/cz-cli/ - :alt: Commitizen friendly -.. image:: https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg - :target: https://github.com/semantic-release/semantic-release - :alt: semantic-release -.. image:: https://img.shields.io/badge/License-GPLv3-blue.svg - :target: https://www.gnu.org/licenses/gpl-3.0 - :alt: License: GPL v3 Documentation ------------- @@ -30,183 +15,117 @@ Documentation .. _`General documentation`: https://discuss-data.pages.gwdg.de/discuss-data/docs/ .. _`Coverage Report`: https://discuss-data.pages.gwdg.de/discuss-data/coverage/ -Settings --------- - -Settings are read from environment configuration files located in ``.envs/``. -Bear in mind that no settings for the production environment are provided. -You could easily create your own environment settings from ``.envs/.local`` -and from the documentation on `Configuring the Environment`_. - -.. _`Configuring the Environment`: https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html#configuring-the-environment - -Basic Commands --------------- +Preparing the development environment +------------------------------------- -Type checks -^^^^^^^^^^^ +Provided, you have Docker, Python3 and npm installed on your system, prepare your local environment by installing node modules and python dependencies into a virtual environment. -Running type checks with mypy: +.. code:: bash -:: - - $ mypy discuss_data + npm install + python3 -m venv venv + . venv/bin/activate + python -m pip install -U pip + pip install -r requirements/local.txt -Django unit tests -^^^^^^^^^^^^^^^^^ -:: +Build the docker stack and install fixtures as well as test data to fill up the database: - $ docker-compose -f local.yml run django python manage.py test +.. code:: bash -Test coverage -^^^^^^^^^^^^^ + ./init -To run the tests, check your test coverage, and generate an HTML coverage report:: - - $ docker-compose -f local.yml run --rm django ./run_pytest - -Django template check -^^^^^^^^^^^^^^^^^^^^^ -:: - - $ docker-compose -f local.yml run django python manage.py templatecheck - - -I18N -^^^^ - -For further information see `Localization`_. - -.. _`Localization`: https://docs.djangoproject.com/en/3.0/topics/i18n/translation/#localization-how-to-create-language-files - - -Email Server -^^^^^^^^^^^^ - -The local SMTP server `MailHog`_ with a web interface is available as docker -container. It will start automatically when composing the "local"-setup. - -With MailHog running, to view messages that are sent by your application, open your browser and go to ``http://127.0.0.1:8025`` - -.. _mailhog: https://github.com/mailhog/MailHog - -Search -^^^^^^ - -To initialize or rebuild the elasticsearch index run: - - $ docker-compose -f local.yml run --rm django python manage.py search_index --rebuild - -Print the whole index in development by accessing: - ``http://localhost:9200/users/_search?pretty`` -Test Data -^^^^^^^^^ - -To provide initial data for manual or automated testing, fixtures are provided. - -Load fixtures for users and datasets by calling - - $ docker-compose -f local.yml run --rm django python manage.py loaddata ddpublications.json ddusers.json dddatasets.json core.json +Settings +-------- +Settings are read from environment configuration files located in ``.envs/``. +Bear in mind that the settings provided ``.envs/.local`` are not meant to be used in a production environment. +To create your own environment settings, read the documentation on `Configuring the Environment`_. +.. _`Configuring the Environment`: https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html#configuring-the-environment -Deployment ----------- +Testing +------- -The following details how to deploy this application to the different -environments. +Run type checks with mypy: +:: + mypy discuss_data -Development (w/ Docker) -^^^^^^^^^^^^^^^^^^^^^^^ +Run Django unit tests: -This project is configured to use Docker for development. -Build images: :: - $ docker-compose -f local.yml build - -Run container: -:: + docker-compose -f local.yml run django python manage.py test - $ docker-compose -f local.yml up +To run the tests, check your test coverage, and generate an HTML coverage report: -Make migrations: :: - $ docker-compose -f local.yml run --rm django python manage.py makemigrations - -Run migrations: -:: + docker-compose -f local.yml run --rm django ./run_pytest - $ docker-compose -f local.yml run --rm django python manage.py migrate +Run Django template checks -Create new empty database: :: - $ docker-compose -f local.yml run --rm django sh - /app # psql -U debug -h postgres -p 5432 postgres + docker-compose -f local.yml run django python manage.py templatecheck - postgres=# CREATE DATABASE discuss_data_empty; - postgres=# GRANT ALL PRIVILEGES ON DATABASE discuss_data_empty to debug; - postgres=# \q -Reference the new database in `.envs/.local/.postgres` and restart the stack using the new db. +Localization and Internationalization +------------------------------------- +For further information see `Localization`_. -Consult `cookiecutter-django Docker documentation`_ for further details. +.. _`Localization`: https://docs.djangoproject.com/en/3.0/topics/i18n/translation/#localization-how-to-create-language-files -.. _`cookiecutter-django Docker documentation`: http://cookiecutter-django.readthedocs.io/en/latest/deployment-with-docker.html +Email Server +------------ -Development (w/o Docker) -^^^^^^^^^^^^^^^^^^^^^^^^ -This is not recommended. Since this project is configured to use Docker you -have to override the Docker-setting with an environment variable: +The ``local`` Docker stack comprises the SMTP server `MailHog`_ with a web interface available under ``http://127.0.0.1:8025``. -:: +.. _mailhog: https://github.com/mailhog/MailHog - $ export USE_DOCKER=false +Search +------ -Also, set your local Database url: +To initialize or rebuild the elasticsearch index run: :: - $ export DATABASE_URL=postgres://:@127.0.0.1:5432/ + docker-compose -f local.yml run --rm django python manage.py search_index --rebuild -After having installed the node dependencies with ``npm install`` the project -can be run with ``npm run dev`` with Live reloading and Sass CSS compilation. +Print the whole index in development by accessing: +:: -Add new node dependencies via ``npm install --save-dev``. + http://localhost:9200/users/_search?pretty -Production -^^^^^^^^^^ +Deploy to production +-------------------- -To deploy to the production environment see the `Discuss Data Docker`_ project. +For building and testing the production environment a ``production`` compose setup is provided with this repo. +For an example on how to actually deploy to a production environment see the `Discuss Data Docker`_ project. .. _`Discuss Data Docker`: https://gitlab.gwdg.de/discuss-data/discuss-data-docker -Backup ------- - -run backup: -:: - - $ docker-compose -f production.yml exec postgres backup - -show backups: -:: - - $ docker-compose -f production.yml exec postgres backups - -check backup mount: -:: - $ docker volume inspect discussdata_production_postgres_data_backups -current backup mount: -:: +Badges +------ - $ /var/lib/docker/volumes/discussdata_production_postgres_data_backups/_data +.. image:: https://img.shields.io/badge/built%20with-Cookiecutter%20Django-ff69b4.svg + :target: https://github.com/pydanny/cookiecutter-django/ + :alt: Built with Cookiecutter Django +.. image:: https://img.shields.io/badge/code%20style-black-000000.svg + :target: https://github.com/ambv/black + :alt: Black code style +.. image:: https://img.shields.io/badge/commitizen-friendly-brightgreen.svg + :target: http://commitizen.github.io/cz-cli/ + :alt: Commitizen friendly +.. image:: https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg + :target: https://github.com/semantic-release/semantic-release + :alt: semantic-release +.. image:: https://img.shields.io/badge/License-GPLv3-blue.svg + :target: https://www.gnu.org/licenses/gpl-3.0 + :alt: License: GPL v3 diff --git a/config/settings/local.py b/config/settings/local.py index 0f2cc6ddcb465fbb4d8d9943650c8a544f41a744..4d827444aa95a68b860d151fb47c8843d18e1413 100644 --- a/config/settings/local.py +++ b/config/settings/local.py @@ -1,4 +1,6 @@ -from .base import * # noqa +import socket + +from .base import * # pylint: disable=W0401,W0614 from .base import env # GENERAL @@ -48,12 +50,11 @@ DEBUG_TOOLBAR_CONFIG = { "SHOW_TEMPLATE_CONTEXT": True, } # https://django-debug-toolbar.readthedocs.io/en/latest/installation.html#internal-ips -INTERNAL_IPS = ["127.0.0.1", "0.0.0.0" "10.0.2.2"] -if env("USE_DOCKER") == "yes": - import socket +INTERNAL_IPS = ["127.0.0.1", "0.0.0.0", "10.0.2.2"] - hostname, _, ips = socket.gethostbyname_ex(socket.gethostname()) - INTERNAL_IPS += [ip[:-1] + "1" for ip in ips] +# DOCKER +hostname, _, ips = socket.gethostbyname_ex(socket.gethostname()) +INTERNAL_IPS += [ip[:-1] + "1" for ip in ips] # django-extensions # ------------------------------------------------------------------------------ diff --git a/config/urls.py b/config/urls.py index 10251234e1367e1ae4b87f53035f5df8a325cd0f..47c1138460da1125b3971dc156f4959571a056ee 100644 --- a/config/urls.py +++ b/config/urls.py @@ -1,17 +1,15 @@ from django.conf import settings -from django.urls import include, path from django.conf.urls.static import static from django.contrib import admin +from django.urls import include, path from django.views import defaults as default_views -from discuss_data.ddusers.views import following_act_feed -import discuss_data.dddatasets - from wagtail.admin import urls as wagtailadmin_urls from wagtail.documents import urls as wagtaildocs_urls from wagtail.core import urls as wagtail_urls from discuss_data.core.views import landing_page +from discuss_data.ddusers.views import following_act_feed urlpatterns = [ diff --git a/discuss_data/core/views.py b/discuss_data/core/views.py index 3da0e7f7c3ef895d51412d80ff695aea346097e2..c5ddb6b7dcaac448a297c150e7b91853ec6305a3 100644 --- a/discuss_data/core/views.py +++ b/discuss_data/core/views.py @@ -11,6 +11,7 @@ from django.shortcuts import ( from elasticsearch_dsl import Q from guardian.core import ObjectPermissionChecker +from sentry_sdk import last_event_id from discuss_data.core.documents import ( UserDocument, @@ -257,10 +258,8 @@ def handler404(request): return response -def handler500(request): - response = render(request, "500.html") - response.status_code = 500 - return response +def handler500(request, *args, **argv): + return render(request, "500.html", {"sentry_event_id": last_event_id()}, status=500) def get_keyword_tags(request): diff --git a/discuss_data/templates/core/_object_edit_panel.html b/discuss_data/templates/core/_object_edit_panel.html index 8df0f90a7d6db02202b41a7aff55ed765a6eeb12..b48c1b867de78b1695e30ee01b7b9fa01c0873b4 100644 --- a/discuss_data/templates/core/_object_edit_panel.html +++ b/discuss_data/templates/core/_object_edit_panel.html @@ -8,7 +8,7 @@ {% endif %} -
+
{% include "core/_get_user_objects.html" with user=oep.user objects=oep.objects ds=ds edit_url=oep.edit_url add_url=oep.add_url delete_url=oep.delete_url object_type=oep.object_type object_text=oep.object_text download_url=oep.download_url ictarget='#ds-content' %}
diff --git a/discuss_data/templates/core/_tags.html b/discuss_data/templates/core/_tags.html index ec9100873b77415b4f92efb540ef5f082e6b66c5..63cf9aa95e73875490afffe700600557f087fcf6 100644 --- a/discuss_data/templates/core/_tags.html +++ b/discuss_data/templates/core/_tags.html @@ -1,7 +1,7 @@ {% load static i18n %} {% block content %} - [{% for tag in qs %}"{{ tag }}"{% if forloop.last %}{% else %},{% endif %}{% endfor %}] + [{% for tag in qs %}"{{ tag }}"{% if forloop.last %}{% else %},{% endif %}{% endfor %}] {% if error_msg %}

{{ error_msg }}

{% endif %} diff --git a/discuss_data/templates/dddatasets/detail.html b/discuss_data/templates/dddatasets/detail.html index 140f2276300cddc322243b4f042d7e5aecec7c61..f797ce9dea38832996a2f8acd4fd26457ac7a8d1 100644 --- a/discuss_data/templates/dddatasets/detail.html +++ b/discuss_data/templates/dddatasets/detail.html @@ -16,7 +16,7 @@

{% trans "Countries" %}

{% for country in ds.countries.all %} - {{ country }} + {{ country }} {% endfor %}

{% endif %} @@ -34,11 +34,11 @@

{% trans "Language of data" %}

{% for language in ds.get_language_tags.all %} - {{ language }} + {{ language }} {% endfor %}

{% endif %} - + {% if ds.get_methods_of_data_collection_tags.all %}

{% trans "Methods of data collection" %}

@@ -46,7 +46,7 @@ {{ mdc }} {% endfor %}

- {% endif %} + {% endif %} {% if ds.get_methods_of_data_analysis_tags.all %}

{% trans "Methods of data analysis" %}

@@ -57,7 +57,7 @@

{% endif %} {# dataset content end #} -
+
{% endblock ic-content %} diff --git a/discuss_data/templates/dddatasets/metadata.html b/discuss_data/templates/dddatasets/metadata.html index 2dcab6844a3e9e89725427e82748435f72454eb6..e2e2beb7fe5ce0816e311d1e60de5c27eb7588d9 100644 --- a/discuss_data/templates/dddatasets/metadata.html +++ b/discuss_data/templates/dddatasets/metadata.html @@ -3,9 +3,9 @@ {% block ic-content %} -
+
{% include 'dddatasets/_dataset_header.html' with ds=ds type='metadata' %} - + {# ds Metadata Tab #}

{% trans "Metadata" %}

diff --git a/discuss_data/templates/dddatasets/prep/_object_tags_list.html b/discuss_data/templates/dddatasets/prep/_object_tags_list.html index 166dda720244605d18b6408677181bbac266b53d..abb059a0a8aa0121d293a75ee2f25fac7034bb46 100644 --- a/discuss_data/templates/dddatasets/prep/_object_tags_list.html +++ b/discuss_data/templates/dddatasets/prep/_object_tags_list.html @@ -1,6 +1,6 @@ {% load i18n %} -[{% for tag in qs %}"{{ tag }}"{% if forloop.last %}{% else %},{% endif %}{% endfor %}] +[{% for tag in qs %}"{{ tag }}"{% if forloop.last %}{% else %},{% endif %}{% endfor %}] {% if error_msg %}

{{ error_msg }}

diff --git a/discuss_data/templates/dddatasets/prep/listing.html b/discuss_data/templates/dddatasets/prep/listing.html index b7eaeb81eb700fa9bd4d15138b5f776cff9d2211..69d4cfd74e659570ec6ca8de3887c6dc2cd005a0 100644 --- a/discuss_data/templates/dddatasets/prep/listing.html +++ b/discuss_data/templates/dddatasets/prep/listing.html @@ -1,4 +1,4 @@ -{% extends 'base.html' %} +{% extends 'base.html' %} {% load static i18n %} {% block title %} @@ -18,11 +18,11 @@

-
- {% if dsmo_own %} -

{% trans "Your Datasets" %}

+
+ {% if dsmo_own %} +

{% trans "Your Datasets" %}

{% for dsmo in dsmo_own %} - {% include "dddatasets/_user_dataset.html" with dataset=dsmo.get_top_version_dataset prep=True %} + {% include "dddatasets/_user_dataset.html" with dataset=dsmo.get_top_version_dataset prep=True %} {% endfor %} {% endif %} @@ -33,7 +33,7 @@ {% endfor %} {% endif %} - {% if dsmo_edit %} + {% if dsmo_edit %}

{% trans "Datasets you can edit" %}

{% for dsmo in dsmo_edit %} {% include "dddatasets/_user_dataset.html" with dataset=dsmo.get_top_version_dataset prep=True %} diff --git a/discuss_data/templates/dddatasets/versions.html b/discuss_data/templates/dddatasets/versions.html index 22d7544debab93fae04f1951b5c06fbc1f8f8ce8..b8d8c555b6b534c26eacfbc03b867ed731beb6d3 100644 --- a/discuss_data/templates/dddatasets/versions.html +++ b/discuss_data/templates/dddatasets/versions.html @@ -11,18 +11,18 @@ {% for ds in versions %}
-
-
-
+
+
+
-
{{ ds.title }}
+
{{ ds.title }}
{{ ds.version }}
-
+
{% endfor %} diff --git a/discuss_data/templates/ddusers/_modal.html b/discuss_data/templates/ddusers/_modal.html index d8ed8bc1ca6816f94fb440a82a7f12b266f03900..0605d7784ce4ebe74284a5a3fc3d043b9afd10e1 100644 --- a/discuss_data/templates/ddusers/_modal.html +++ b/discuss_data/templates/ddusers/_modal.html @@ -7,7 +7,7 @@

{{ modaltitle }}

- +
diff --git a/discuss_data/templates/ddusers/_user_search_list.html b/discuss_data/templates/ddusers/_user_search_list.html index 6fa1ab8804f820a267a2a266646a833690f08368..d5b36f2993313c9d50a040fa14abfc84d1c3e23b 100644 --- a/discuss_data/templates/ddusers/_user_search_list.html +++ b/discuss_data/templates/ddusers/_user_search_list.html @@ -4,7 +4,7 @@
{% if users %} - {% for object in users %} + {% for object in users %}
@@ -26,13 +26,13 @@
{% csrf_token %} - -

-
+
@@ -40,5 +40,5 @@ {% else %}
No users found with query {{ query }}.
{% endif %} -
+ diff --git a/discuss_data/templates/ddusers/change_password.html b/discuss_data/templates/ddusers/change_password.html index eafbd639efe0c619b62480bf09b31e83ae5dc4af..17f2da860c63618e184c9e33e851d62b427b96fd 100644 --- a/discuss_data/templates/ddusers/change_password.html +++ b/discuss_data/templates/ddusers/change_password.html @@ -3,7 +3,7 @@ {% block content %} {% if updated %} {% endif %} {% if form %} @@ -13,7 +13,7 @@
{% for field in form %}
- {{ field.label_tag }} + {{ field.label_tag }} {% for error in field.errors %}
{{ error }}
{% endfor %} diff --git a/discuss_data/templates/ddusers/list.html b/discuss_data/templates/ddusers/list.html index 52b4ac67246a4305829ed10715dd258f77138479..ba1adffd7adecb4e8938fae8cd1caa8c0e725115 100644 --- a/discuss_data/templates/ddusers/list.html +++ b/discuss_data/templates/ddusers/list.html @@ -1,4 +1,4 @@ -{% extends 'base.html' %} +{% extends 'base.html' %} {% load static i18n %} {% block title %}{% trans "Researchers" %}{% endblock %} @@ -7,19 +7,19 @@
- {% if users %} -
+ {% if users %} +

{% trans "Researchers" %}

{% for object in users %} - {% include 'ddusers/_user_detail_card.html' with object=object image_url=objectimage.url %} + {% include 'ddusers/_user_detail_card.html' with object=object image_url=objectimage.url %} {% endfor %}
-

Filtered by

- {% include 'core/_search_filter.html' with object=object image_url=objectimage.url %} -
+

Filtered by

+ {% include 'core/_search_filter.html' with object=object image_url=objectimage.url %} +
{% else %} {% trans "No Resarchers found" %}. {% endif %} diff --git a/discuss_data/templates/pages/blocks/manual_topic_block.html b/discuss_data/templates/pages/blocks/manual_topic_block.html index 13498d04a191f63a5e7f12c16cdf8cb4ba4d3e77..43a50292e1e206407372e9006e9624be75bc32f4 100644 --- a/discuss_data/templates/pages/blocks/manual_topic_block.html +++ b/discuss_data/templates/pages/blocks/manual_topic_block.html @@ -1,7 +1,7 @@ {% load wagtailcore_tags wagtailimages_tags %} -
- {% image value.image width-800 %} +
+ {% image value.image width-800 %}

{{ value.heading }}

{{ value.paragraph|richtext }}
\ No newline at end of file diff --git a/discuss_data/templates/pages/team_member_index_page.html b/discuss_data/templates/pages/team_member_index_page.html index dab0dfcc6b7c862aa26d7d9c1df337c5a323cea5..5bf597f7dbd6610c7496bb953daf3a93504b0619 100644 --- a/discuss_data/templates/pages/team_member_index_page.html +++ b/discuss_data/templates/pages/team_member_index_page.html @@ -9,8 +9,8 @@
{{ page.intro|richtext }}
- {% for member in page.get_children %} -

{{ member.title }}

+ {% for member in page.get_children %} +

{{ member.title }}

{% endfor %} {% endblock %} \ No newline at end of file diff --git a/discuss_data/templates/pages/team_member_page.html b/discuss_data/templates/pages/team_member_page.html index 4d4aab0fbdea40d1e0a20bf828186168df7a51a9..27c713d69377538b9f5a5c22cfbaf5e2c7336587 100644 --- a/discuss_data/templates/pages/team_member_page.html +++ b/discuss_data/templates/pages/team_member_page.html @@ -4,14 +4,14 @@ {% block body_class %}template-teammemberpage{% endblock %} -{% block content %} +{% block content %}
{% if page.image %} {% image page.image fill-320x240 class='mr-3' %} {% endif %}
{{ page.title }}
- {{ page.text|richtext }} + {{ page.text|richtext }}
diff --git a/init b/init new file mode 100755 index 0000000000000000000000000000000000000000..a82b01c75807417b22c19f2db01d10f063d1f0f4 --- /dev/null +++ b/init @@ -0,0 +1,11 @@ +#!/bin/sh + +docker-compose -f local.yml build + +docker-compose -f local.yml run --rm django python manage.py migrate + +docker-compose -f local.yml run --rm django python manage.py loaddata ddpublications.json ddusers.json dddatasets.json core.json + +docker-compose -f local.yml run --rm django python manage.py runscript initial_data + +docker-compose -f local.yml run --rm django python manage.py createsuperuser