rdd-technical-reference.md



title:  'SUB RDD Technical Reference'


About This Document
Author: Software Quality Working Group


Audience: Developers of the Research and Development Department of the Göttingen State and University Library.


Purpose: This guideline should help you getting started a new software development project (or improving an existing one!) in the Research and Development Department of the Göttingen State and University Library.
Our goal is to establish better software quality by following standards the developer team has mutually agreed upon. Roughly basing on the EURISE Network Technical Reference, these standards are discussed, worked out, and decided in the Software Quality Working Group, which meets biweekly on Tuesdays at 13:00--14:00. However, they aren't cast in stone, so in case you have a good idea for a better standard, feel free to contribute!


Status: This document is a living document and will be extended as soon as the Software Quality Working Group has agreed upon a new standard for software projects in RDD.
TODOs and addenda of this document are maintained here.

Explanatory Notes
CESSDA's Software Maturity Levels (SML): Several organizations already have put a lot of work into developing metrics for good software. Since we didn't want to reinvent the wheel, we decided to adapt (and modify if need be) a metric which suits our needs. CESSDA is one of the ERICs, and although it focuses on Social Sciences it has similar requirements regarding its software.
Throughout the document you will find sections with the heading "CESSDA's Software Maturity Levels in RDD" in which we describe which of the SMLs we aim for and how we want to implement it.

FE-Onboarding
This section focuses on the specific needs and requirements of developers new
to the department. Please also refer to any of the general information on
onboarding within the department, as well as the departmental mentoring
program. This includes skills and tools developers  as well as a checklist of
the basic repos and groups new developers should be granted access to. A list
of relevant and useful mailing lists new developers ought to consider
subscribing to is provided at the end of the section.

Skills & Tools
Here you will find a list of skills and tools that a developer will likely
need at one point or another during their tenure at FE. We assume a basic
familiarity with basal technologies such as XML.

Ant
GitLab CI/CD
Implementation of DARIAH-AAI
Puppet
Docker
Exist
(Lucene-)Indices
SADE


Repos & Groups

The departmental group at github: https://github.com/subugoe/teams/fe

The departmental group at GitLab.gwdg.de: https://gitlab.gwdg.de/fe

The SUB-wide group at GitLab.gwdg.de: https://gitlab.gwdg.de/subugoe


DARIAH-wikispace SUB-GWDG-IT
(probably further project-related wikispaces)
Kanboard-user group FE


Mailing Lists

eXist-db (eXist open mailing list)
de-RSE
RfII
relevant TEI-lists


Further Reading

RDD Technical Reference: https://github.com/subugoe/rdd-technical-reference

DH-FE-Zoterobibliothek
Heise, Golem, etc.


Style Guides

General
The basic definitions are given by our EditorConfig file, .editorconfig,
i.e. Unix line breaks and 2 space indentation.

For Specific Programming Languages
For the more prominent programming languages we have formatting and general style guides we ask you to follow:


Java: The Java style guide can be found here. It's based on the Google style guide for Java with some minor RDD specific settings. You can configure Eclipse to use it automatically at Eclipse > Preferences > Java > Code Style > Formatter. Just load the RDD Eclipse Java Google Style in the formatter preferences and use it in your RDD projects.


JavaScript: For JS we use the Airbnb JavaScript Style Guide.


HTML/CSS: For HTML/CSS we agreed upon the Google HTML/CSS Style Guide.


XQuery: We use the xqdoc style guide with the following addenda:

use double quotes instead of single quotes (for easy escaping)
use four spaces for a TAB (because eXide switching the preferences in eXide's setting isn't permanent)


XSLT: Since there is no official style guide for XSLT, we decided to write
our own, resulting from common best practices and own experiences within
the department.


Python: For Python PEP 8 should be used, Django has a style guide based on PEP-8 with some exceptions: Django-Styleguide. There are linters and tools like flake-8 and pep-8 available as support.


SPARQL: For SPARQL there is not really any official style guide and there is no possibility to simply include any code style automatically using a code style file. We just collect some advice how to format and use SPARQL code here.


Documentation

Doc Sprints
To ensure the best documentation of our code we meet on a weekly base for a code sprint to document everything we have coded throughout the week and haven't been able to document properly yet. The meeting takes place in the meeting room at 1pm. Cookies may be provided.

General Notions about Documentation


Don't document a language's specifics, e.g. operators.

Example: Use of => in the XQuery expression replace("qbc", "q", "b") => substring(1, 2) MUST NOT be explained in a comment.


It is best to use a language's structure to document.


Write the best documentation you can.


Documentation and variable language is American English.


Docs should be as close to the code as possible.

This refers both to the documentation in a repository and in the code itself, e.g. inline documentation.


Every code repository MUST have:

a README.md file according to this template that contains

link to original repository (if the software is forked or otherwise based on preexisting software)
short introduction on what the repository is about
a guide how to get the software running (if makes sense)
link to demo instance
example or demo installation
link to license file
contribution guide
link to style guide
link to bug tracker/project management system
known issues
badges to CI status


a LICENSE file


Developer Documentation

Software Architecture
Each software project should be documented using an architecture diagram that helps understanding its basic functionality (even though using tools to generate diagrams such as UML class diagrams doesn't seem to be possible in every case).\
Examples:

Generating call graphs in eXist-db

Call diagrams can be useful to follow code and service calls and should be existing for every API method.

API Documentation


The docs should comprise used parameters, author and @since annotations


Example for Java


Links to callers must not be listed in the documentation, because this info will be deprecated soon. It is strongly recommended to use call stacks of tools like Eclipse (Java) and/or Call Graph Module (SADE).


Document REST-APIs using openAPI if possible. OpenAPI docs should be located at /doc/api on servers.


CESSDA's Software Maturity Levels in RDD (CA1.3)

MUST
MUST be SML2, which is defined as follows:

There is external documentation that describes public API functionality and is sufficient to be used by an experienced developer. If available, source code is consistently and clearly commented. Source code naming conventions are adhered to and consistent.


Actions to Be Taken in RDD

provide a fully documented public API, e.g. by using OpenAPI
naming conventions still have to be discussed --> style guide?
reference to style guide used in the CONTRIBUTING/README file?


Admin Documentation

The docs should comprise how to install the software, how to run and/or restart it, how to test the installation, ...


CESSDA's Software Maturity Levels in RDD (CA1.2)

MUST
MUST be SML3, which is defined as follows:

There is a deployment and configuration manual that can guide an experienced operational user through deployment, management and configuration of the software. Exception and failure messages are explained, but descriptions of solutions are not available. Documentation is consistent with current version of the software.


Actions to Be Taken in RDD

deployment: short deployment descriptions can be provided in the README, more detailed explanations should be kept separately (such as INSTALL(.md), linked from README)
exception and failure messages are described in doc strings/function annotations
function documentation should be generated automatically and made available/searchable in the web (such as readthedocs, javadoc html, etc. pp.)


Server Documentation
This type of documentation is provided and maintained in our DARIAH wiki space.\
We have a template encompassing all information necessary: To create a wiki page for a new server navigate to the FE Server list, select "..." right beside the "Create" button and search for "FE-Server".

User Documentation
This may encompass:


how to use the software and APIs, FAQs, walkthroughs, ...


guided tour (Bootstrap Tour) as user documentation

for SADE portal usage (such as Fontane, BdN, Architrave)
for complex Digital Editions


screencasts


CESSDA's Software Maturity Levels in RDD (CA1.1)

MUST
MUST be SML2, which is defined as follows:

There is external documentation that is accessible and sufficient for an expert user to configure and use the software for the user’s individual needs. Terminology and methodology is not explained.


Actions to Be Taken in RDD

a README(.md) has to be available in the source code repository (cf. General Notions about Documentation)


SHOULD
SHOULD be SML3, which is defined as follows:

There is a user manual that can guide a reasonably skilled user through use and customisation of the software to the user’s individual requirements. Documentation is consistent with current version of the software.


Actions to Be Taken in RDD

a more detailed explanation is available for the user at some place (such as user guide in wikis, etc. pp.)
docs should also be provided in a docs directory in the source code repository
docs are revised regularly during our doc sprints


Version Control
We exclusively use git in RDD. Please see https://git-scm.com/doc for information on how it works.
We recommend to use the Git flow Workflow (also consult the Cheat Sheet).
For using git flow it is safest to protect your master and develop branch on server side to avoid accidental pushes into these branches. All specific branches working on an issue described in a bug tracker may utilize the following naming scheme:


[track]/#[ISSUENUMBER]-[KEYWORD]


e.g. bugfix/#12-flux-capacitor.
A GitHub workflow used in DARIAH-DE and related services is described in the DARIAH-DE Wiki.
You could also use the GitLab flow as a simpler alternative, which can be broken down into 11 Rules.
It is also recommended to automatically close issues via commit message; How this works exactly depends on the Git repository server. Issues can also be referenced across repositories.
We use the following Git servers at the moment in RDD:

Projects (GWDG) --> https://projects.gwdg.de

GitLab (GWDG) --> https://gitlab.gwdg.de

github.org --> https://github.com/subugoe


Which one is suitable for you depends on:

the project you are working on
existing code
whether or not you want to use CI/CD or GitLab Runners
...

We have got an RDD team on GitHub. Feel free to join us!
Consider mirroring of repos for project visibility (e.g. mirror GitLab/Projects code to GitHub?)

Bug Tracking
A bug tracking system is obligatory! Please use the respective bug tracking system of your repo and/or project management solution (please see chapter Version Control)!

Software Tests
We aim to have a test coverage of 100% (except for getter and setter methods). This is understood on a component level, which means that every method should have at least one test. Whether you achieve this by Test Driven Development (TDD) or not is specific to your preferred way to work.
Please keep in mind not only to write a test for each of your functions but also
to consider all possible outcomes. It is e.g. not sufficient to test if a function
creates a file if the written content depends on variables etc.
Examples for writing tests in different programming languages are:

XQuery