Commit e4ff0924 authored by Mathias Goebel's avatar Mathias Goebel 🎠
Browse files

Merge branch 'release/2.9.0'

parents 5081d3dd eed8dad6
# customized config files #
# oxygen project file #
# files generated by ant #
# This post-commit hook creates standard build targets with ant
# This pre-commit hook checks that you havn't left and DONOTCOMMIT tokens in
# your code when you go to commit.
# To use this script copy it to .git/hooks/pre-commit and make it executable.
# This is provided just as an example of how to use a pre-commit hook to
# catch nasties in your code.
# Work out what to diff against, really HEAD will work for any established repository.
if git rev-parse --verify HEAD >/dev/null 2>&1
# Initial commit: diff against an empty tree object
diffstr=`git diff --cached $against | grep -e '^\+.*DONOTCOMMIT.*$'`
if [[ -n "$diffstr" ]] ; then
echo "You have left DONOTCOMMIT in your changes."
echo "It is important that you read and understand them,"
echo "before you proceed and remove them."
exit 1
node {
stage('Preparation') {
checkout scm
sh 'rm -f build/*.xar'
stage('Build') {
sh 'ant'
stage('Publish') {
archiveArtifacts artifacts: 'build/*.xar', onlyIfSuccessful: true
script: "find build/ -name '*.xar' -exec basename {} \\;",
returnStdout: true
sh "curl -X POST -F 'file=@build/${FILENAME}' http://localhost:8181/exist/apps/receiver.xql"
stage('Server') {
build job: 'SADE_Server', wait: false
# About SADE
The Scalable Architecture for Digital Editions (SADE) tries to meet the requirement for an easy to use publication system for electronic resources and Digital Editions. It is an attempt to provide a modular concept for publishing scholarly editions in a digital medium, based on open standards. Furthermore it is a distribution of open source software tools for the easy publication of Digital Editions and digitized texts in general out of the box. SADE is a scalable system that can be adapted by different projects which follow the TEI guidelines or other XML based formats.
This is the modified version for Fontane Notizbücher.
# Fontane-Build
* call `ant sade-dist-build` to build the latest supported eXist db and all SADE packages
* run `./dist-utils/antbuild/build/sade/bin/`
* deploy the xar package from `SADE-Project-develop`
* deployment needs approx. 45 min due to republication of all the data. this is because the TEI documents needs to be stored from in memory, anything else will place whitespaces in the rendered xhtml output.
* there is fastdeploy build target that includes all non TEI-Data + Register and a single notebook (C07). deployment needs about 10 min.
* see http://localhost:8080/exist/apps/SADE/textgrid/index.html
## Tests
- [Markdown rendering](http://localhost:8080/exist/apps/SADE/textgrid/content.html?
- [Dokuwiki (w/o reload)](http://localhost:8080/exist/apps/SADE/textgrid/doku.html)
- [Dokuwiki (w reload)](http://localhost:8080/exist/apps/SADE/textgrid/doku.html?id=gesamtdokumentation_ii._tei-header)
- [Image reload](http://localhost:8080/exist/apps/SADE/textgrid/doku.html?id=gesamtdokumentation_iii.3.21&reload)
- [Search](http://localhost:8080/exist/apps/SADE/textgrid/results.html?q=luther)
- [Literaturverzeichnis](http://localhost:8080/exist/apps/SADE/textgrid/literaturvz.html)
- [Register](http://localhost:8080/exist/apps/SADE/textgrid/register.html)
- [Auth on develop (see cookie)](http://localhost:8080/exist/apps/SADE/textgrid/edition.html?id=/xml/data/16b00.xml&page=)
- [TEI rendering](http://localhost:8080/exist/apps/SADE/textgrid/edition.html?id=/xml/data/16b00.xml&page=1r)
- [Cover](http://localhost:8080/exist/apps/SADE/textgrid/edition.html?id=/xml/data/16b00.xml&page=)
- [TBLE](http://localhost:8080/exist/apps/SADE/textgrid/edition.html?id=/xml/data/16b00.xml&page=60v)
- [single page]()
- [XML-View](http://localhost:8080/exist/apps/SADE/textgrid/xml.html?id=/xml/data/16b00.xml)
- [REST](http://localhost:8080/exist/apps/SADE/textgrid/rest/data/16b00.xml)
Scalable Architecture for Digital Editions
This is the main application for SADE. It serves as XAR package according to the
[EXPath Packaging System]( and is created for the
[eXist database](
## The git repo
This repo brings you the sources of the application which contain a brief
documentation as long as you are not dealing with a fork. Please find the
documentation in the folder `docs` where they are stored in Markdown. Any
default installation of this package serves the documentation as website
integrated in your template and in your instance.
### git hooks
The folder `.hooks` contains git hook scripts to be executed on certain git
events. The usage of these scripts is recommended, as they ensure a smooth
#### pre-commit
We test for `DONOTCOMMIT` comments in the code. As long as a comment like this
is in files, no commit is possible.
#### post-commit
On any commit a new artifact is being created, so you can proceed and test it
## Build
Artifacts go to `build/`.
The default artifact is the xar package.
## Deploy
## Installation
## Known Issues
In the unlikely event of cycling to the next digit in TextGrid URIs you may
encounter issues when you going to use an older URI together with a newer that
starts with the same characters. Example: One will publish the text
`textgrid:foo.0` and over time the URI number increases that much, that
another object that should be stored in the database has the URI
`textgrid:foobar.0`, some processes that tests against using `starts-with()`
will fail or may respond with the wrong object. At the end of 2017 we are
dealing with 5 digits of Latin letters and Arabic numbers (a total of 36
different symbols) starting with "3". So it will take another approx. ten
million items to reach the next digit. So we will talk about this later.
<?xml version="1.0" encoding="UTF-8"?>
<project default="xar" name="SADE">
<xmlproperty file="expath-pkg.xml"/>
<property name="project.version" value="${package(version)}"/>
<property name="" value="SADE"/>
<property name="project.version" value="${package(version)}"/>
<property name="" value="${package(abbrev)}"/>
<property name="build.dir" value="build"/>
<target name="xar">
<mkdir dir="${build.dir}"/>
<zip basedir="." destfile="${build.dir}/${}-${project.version}.xar" excludes="${build.dir}/*"/>
\ No newline at end of file
......@@ -6,4 +6,4 @@
<trigger class="org.exist.extensions.exquery.restxq.impl.RestXqTrigger"/>
\ No newline at end of file
<config xmlns="">
<!-- global parameters -->
<!-- unless you know exaclty what you are doing, leave the project-id as
it is. -->
<param key="project-id">textgrid</param>
<param key="project-title">SADE Referenzinstanz</param>
<param key="textgrid.tgcrud"></param>
<param key="textgrid.webauth"></param>
<param key="textgrid.authZinstance"></param>
<param key="textgrid.public-triplestore"></param>
<param key="textgrid.nonpublic-triplestore"/>
<!-- if you want to distinguish between develop and productive instance,
set sade.develop "true"
use it for example to set up a protected instance -->
<param key="sade.develop">true</param>
<!-- if sade.develop is set to true and you want to look behind the curtain
you probably want to set a password like secret to authenticate -->
<param key="secret">Mellon</param>
<param key="textgrid.digilib"></param>
<param key="textgrid.digilib.defaultSize">1500,</param>
<param key="textgrid.digilib.tile" deprecated="true">dw=100&amp;dh=100</param>
<!-- set if you want to preserve revisions in the database.
if set to "false" all resource names will be trimmed to base-uris and
the database will always store the last published item per base uri -->
<param key="textgrid.preserveRevisions">false</param>
<module key="multilanguage">
<param key="lang.default">de</param>
<param key="lang.alt" description="a semicolon seperated list of alternative languages">en</param>
<!-- parameters for MODULES -->
<module key="navigation">
<!-- Where to find the xml with the navigation template -->
<param key="location">navigation.xml</param>
<module key="wiki">
<param key="confluence.url"></param>
<param key="confluence.lang" description="if the multilanguage plugin is enabled, used languages specified here">
<lang code="de" name="german"/>
<lang code="en" name="english"/>
<lang code="fr" name="french"/>
<module key="multiviewer">
<param key="xslt">
<stylesheet namespace="" location="_assets/TEI-stylsheets/html5/html5.xsl"/>
<stylesheet namespace="" local="true" location="xslt/bol.xsl">
<param name="imgParams" value="?dh=200"/>
<module key="faceted-search">
<param key="viewer-html">content.html?id=/data/</param>
<!-- how much kwic hits to show per document -->
<param key="kwic-hits">3</param>
<param key="kwic-width">20</param>
<param key="thumbnail">false</param>
<param key="hits-per-page">10</param>
<param key="result-xslt">search-hits.xslt</param>
<param key="query-root">
<param key="result-title">
<param key="thumbnail">false</param>
<param key="facets">
<facet key="authors" title="Authors">
<facet key="keywords" title="Schlagwort">
<facet key="dates" title="Zeit">
<facet key="places" title="Publikationsort">
xquery version "3.1";
import module namespace config="" at "modules/config/config.xqm";
import module namespace digilib="" at "../textgrid-connect/digilibProxy.xqm";
import module namespace fontaneTransfo="" at "modules/fontane/transform.xqm";
import module namespace console="";
declare namespace xhtml="";
(:~ This is the URL rewriting engine.
: It routes all requests to the modules and
: evaluates the RESTlike pathes.
: Set up REST pathes in here!
import module namespace config="" at "modules/config.xqm";
declare variable $exist:path external;
declare variable $exist:resource external;
......@@ -13,108 +12,81 @@ declare variable $exist:controller external;
declare variable $exist:prefix external;
declare variable $exist:root external;
let $project := tokenize( $exist:path, "/" )[. != ""][not(ends-with(., "html"))][1]
let $listproject := config:list-projects()[1]
(: forward requests without resources to index.html :)
if ($exist:path eq '') then
<dispatch xmlns="">
<redirect url="{request:get-uri()}/"/>
(: forward requests without project to the first project listed :)
else if (empty($project)) then
<dispatch xmlns="">
<redirect url="{config:list-projects()[1]}/index.html"/>
(: forward requests without resources to index.html :)
else if (ends-with($exist:path, "/") or empty($exist:resource)) then
else if ($exist:path eq "/") then
(: forward root path to index.html :)
<dispatch xmlns="">
<redirect url="index.html"/>
(: forward REST requests to data collection of the project:)
else if (tokenize($exist:path, '/') = "rest") then
<dispatch xmlns="">
<forward url="/../sade-projects/{$project}/data/xml/{substring-after($exist:path, "/rest/")}"/>
(: forward DIGILIB requests to the proxy function :)
else if (tokenize($exist:path, '/') = "digilib") then
let $path := tokenize($exist:path, '/'),
$id := $path[last()],
$if-modified-since := request:get-header("if-modified-since")
digilib:proxy($project, $id, $if-modified-since)
(: REST interface of the interface! :)
else if (tokenize($exist:path, '/') = "get") then
let $reqid := request:get-parameter("id", "/xml/data/16b00.xml"),
$id := $reqid => substring-after("/xml/data/") => substring-before(".xml"),
$pagereq := request:get-parameter("page", "1r"),
$page := if( $pagereq = "" ) then "outer_front_cover" else $pagereq,
$docCollection := $config:projects-dir || $project ||"/data/xml/xhtml/"|| $id ||"/",
$docpath := $docCollection || $page ||".xml",
$doc := doc( $docpath ),
$content :=
switch ($exist:resource)
case "code.html" return
replace(serialize($doc//xhtml:div[@class="teixml"]), "xhtml:", "")
case "trans.html" return
replace(serialize($doc/xhtml:body/xhtml:div/xhtml:div[not(@class="teixml")][not(@class="facs")]), "xhtml:", "")
case "facs.html" return
replace(serialize($doc//xhtml:div[@class="facs"]), "xhtml:", "")
case "toc.html" return
replace(serialize(doc( $docCollection || "toc.xml" )), "xhtml:", "")
case "comm.html" return
replace(serialize($doc//xhtml:div[@class="notes"]), "xhtml:", "")
default return "supported requests are: facs, trans, code"
response:stream($content, "method=text media-type=text/plain")
(: the important stuff :)
else if (ends-with($exist:resource, ".html")) then
let $doc-path := $config:projects-dir || $project || config:get("template") || $exist:resource
(: the html page is run through view.xql to expand templates :)
(: the html page is run through view.xq to expand templates :)
<dispatch xmlns="">
<forward url="{$exist:controller}/modules/view.xql" method="get">
<add-parameter name="doc-path" value="{$doc-path}"/>
<forward url="{$exist:controller}/templates{$exist:path}" method="get"/>
<forward url="{$exist:controller}/modules/view.xq">
<add-parameter name="exist-resource" value="{$exist:resource}" />
<forward url="{$exist:controller}/error-page.html" method="get"/>
<forward url="{$exist:controller}/modules/view.xql"/>
<forward url="{$exist:controller}/templates/error.html" method="get"/>
<forward url="{$exist:controller}/modules/view.xq"/>
(: resource paths starting with $shared are loaded from the shared-resources app :)
else if (contains($exist:path, "/$shared/")) then
(: Resource paths starting with ~assets are loaded from the sade-assets app :)
else if (contains($exist:path, "/~assets/")) then
<dispatch xmlns="">
<forward url="/shared-resources/{substring-after($exist:path, '/$shared/')}">
<forward url="/sade_assets/{substring-after($exist:path, '/~assets/')}">
<set-header name="Cache-Control" value="max-age=3600, must-revalidate"/>
(: check if the resource comes from the template or the data
we try to leave the template manipulation as minimal as
let $path := "/../sade-projects/"|| $project || "/" || config:get("data-dir") || "/" || substring-after($exist:path, $project)
let $path := if( doc-available( $path ) ) then $path else "/../sade-projects/"|| $project || config:get("template") || substring-after($exist:path, $project)
else if(starts-with($exist:resource, "textgrid")) then
let $preserveRevision := config:get("textgrid.preserveRevision") = "true"
let $id := if( contains($exist:resource, ".") and $preserveRevision )
then $exist:resource
else tokenize($exist:resource, "\.")[1]
let $isImage := doc( $config:data-root || "/images.xml" )//image/starts-with(@uri, $id) = true()
<dispatch xmlns="">
<redirect url="modules/textgrid/digilib-proxy.xq?id={$id}"/>
<dispatch xmlns="">
<forward url="{$exist:controller}/templates/content.html" method="get"/>
<forward url="{$exist:controller}/modules/view.xq">
<add-parameter name="id" value="{$id}" />
<add-parameter name="exist-resource" value="content.html" />
<forward url="{$exist:controller}/templates/error.html" method="get"/>
<forward url="{$exist:controller}/modules/view.xq"/>
(: redirect markdown files from documentation to the content.html :)
else if(ends-with($exist:resource, ".md")) then
<dispatch xmlns="">
<redirect url="../content.html?id=docs/{$exist:resource}"/>
else if($exist:resource = "sitemap.xml") then
<urlset xmlns="">
{for $html in xmldb:get-child-resources( $config:app-root || "/templates")
where not( contains($html, "_") )
<loc>{request:get-url() => substring-before("sitemap.xml")}{$html}</loc>
(: everything else is passed through :)
<dispatch xmlns="">
<cache-control cache="yes"/>
<forward url="{$path}">
<set-header name="Cache-Control" value="max-age=3600, must-revalidate"/>
SADE, an acronym for «Scalable Architecture for Digital Editions», is a software
package that includes many components, which are particularly suitable for
developing and displaying a digital edition. It focuses on the presentation of
XML documents that follow the guidelines of the Text Encoding Initiative (TEI).
Since 2012 a larger developer community has constantly been working on this
software, which was originally developed at the Berlin-Brandenburg Academy of
Sciences and Humanities (BBAW) as part of the digitization initiative «The
Electronic Life Of The Academy» (TELOTA). Contributors are among others the
BBAW, the Austrian Academy of Sciences (OeAW; as part of the project CLARIN:
Common Language Resources and Technology Infrastructure), the Max Planck
Institute for the History of Science (MPI-WG), the Cologne Center for
eHumanities (CCEH) and the Göttingen State and University Library (SUB; as
part of the project TextGrid).
The software is based on the XML database eXist and its up to date version
itself is a customization. It was developed for the TextGrid infrastructure.
Over the years a few components became obsolete, so the current version is a
completely rewritten one.
## Features
- faceted search
- preconfigured indexes
- TextGrid clients
- transfer data from the Lab
- fork/download projects
- Lucene string analyzer made for TEI
- easy (as in easy) configuration of charmaps and synonyms
- easy (as in easy) to configure menu bar (frontend)
- wiki parser
- Confluence
- Dokuwiki
- professional template
- prepared for optionally prerendering
### Third party addons
- SemToNotes
- TEI stylesheets (XSLT)
- Markdown parser
By default the eXist database offers a RESTful interface to all of your data,
but with the help of a little XQuery, you can provide on-the-fly transformations
via REST as well.
## Development
You can find our git repositories at
[GWDG's Gitlab](
We are happy to get your bug reports and feature requests!
# Customization
SADE was build to fit the needs of many projects creating digital editions.
For a complete customization, a adaption of some XQuery modules may becomes
necessary. But a few things are made to set up by any user, as long as she is
familiar with the basics of XML. When you want to start preparing your own
instance, look at the file config.xml in the root collection of the application.
There are many things you can set up.
## Top Menu
Edit the top menu (navbar) by entering your preferred structure in the file
<item label="Ein Text" link="view.html?"/>
<submenu label="Links">
<item label="" link=""/>
Mit dieser Konfiguration wird ein minimales Menü aufgebaut, das ein Aufklappbares Menü mit dem Namen "Links" und einem Unterpunkt mit externem Link aufbaut, obendrein einen Link auf der Hauptebene zu "Ein Text", in der Darstellung sieht das dann so aus:
![Menü Screenshot](~assets/docs/sade-dropdown.png)
Sie können einfach zum Tutorialprojekt neue Menüpunkte hinzufügen, diese umbenennen, oder das Menü umstellen, probieren Sie es aus!
[Menü bearbeiten](/exist/apps/eXide/index.html?open=/db/sade-projects/tutorial/navigation.xml)
Schauen Sie das Ergebnis an: [Tutorialprojekt](/exist/apps/SADE/tutorial/)
Sie können natürlich auch eigene Dokumente in das Menü aufnhemen, z.B. publizierte TEI Daten .
# Anpassen des Projekttitels
Auch der Projekttitel lässt sich anpassen, er ist vermerkt in der config.xml als ``<param key="project-title">Mein Projekt</param>``
[Projektkonfiguration bearbeiten](/exist/apps/eXide/index.html?open=/db/sade-projects/tutorial/config.xml).
1. Erstellen Sie einen neuen Link im Menü
2. Ändern Sie den Text von "Text1"
3. Ändern Sie den Projekttitel
4. Verlinken Sie ein selbtpubliziertes TEI Dokument im Menü (hierzu "[Publikation eigener Inhalte](view.html?" lesen)
# Faceted-Search
## Important Notes
Always use a root node that is specified in the xconf.
## Konfiguration des Moduls
Die Konfiguration für das Faceted-Search Modul befindet sich in der config.xml
<module key="faceted-search">
<param key="viewer-html">index.html?id=/xml/data/</param>
[Konfiguration bearbeiten](/exist/apps/eXide/index.html?open=/db/sade-projects/textgrid/config.xml)
## Ändern der Ergebnisdarstellung
Mithilfe der Konfigurations-Option "result-xslt" kann auf ein XSLT Stylesheet verwiesen werden,
welches aus den Metadaten im TEI-Dokument des Einzeltreffers einzelne Suchtreffer darstellt.
<param key="result-xslt">/sade-projects/textgrid/xslt/search-hit.xslt</param>
Eine einfaches XSLT Stylesheet zur Ergebnisanzeige ist vorhanden und kann an eigene TEI-Metadaten angepasst werden.
[XSLT bearbeiten](/exist/apps/eXide/index.html?open=/db/sade-projects/textgrid/xslt/search-hit.xslt). Eine solche
XSLT Transformation für Einzelergebnisse kann im Template definiert werden:
<div data-template="fsearch:result-xslt"/>
## Konfiguration der Facetten
In der Konfiguration des Faceted-Search Moduls können die dargestellten Facetten mithilfe eines XPATH Ausdrucks festgelegt werden.
<param key="facets">
<facet key="persons" title="Person">
[Konfiguration bearbeiten](/exist/apps/eXide/index.html?open=/db/sade-projects/textgrid/config.xml)
# Publizieren aus dem TextGridLab
## Vorbemerkung
Der Grundsätzliche Umgang mit dem TextGridLab sollte bekannt sein, ansonsten ist das [Benutzerhandbuch]( eine gute Resource für weitergehende Informationen.
## Vorbereitung
### Installation des "SADE Publish Tool" im TextGridLab
Um SADE an das TextGridLab anzubinden und komfortables publizieren aus dem Lab zu ermöglichen sollte das "SADE Publish Tool" aus dem Marketplace installiert werden. Dafür im Lab den Menüpunkt "Hilfe->Eclipse-Marketplace" auswählen. Im "Recent" Tab befindet sich der Eintrag "SADE Publish Tool". Nach ein Klick auf "Install" wird dieser ins Lab integriert. Das Lab möchte nun neustarten, das sollte es auch tun.
![Marketplace Screenshot](~assets/docs/marketplace.png)
### Einrichten des "SADE Publish Tool"
Nach dem Neustart aus dem Menu "Fenster->Benutzervorgaben" auswählen. Hier den Eintrag "Sade Publisher" auswählen und eintragen:
> __SADE Url:__ http://localhost:8080/exist/apps/textgrid-connect/publish/tutorial/
> __Authorized User:__ sade
> __Password:__ test
![Preferences Screenshot](~assets/docs/preferences.png)
## Eine erste Publikation
### Öffnen der SADE-Publish Perspektive
Falls noch nicht geschehen, nun bitte im TextGridLab anmelden. Den "Welcome-Screen" schliessen, und dann aus der Tool-Leiste die SADE-Publish-Perspektive asuwählen. Es bietet sich die Folgende Ansicht:
![SADE-Publish Perspective Screenshot](~assets/docs/sade-publish2.png)
Der Knopf "Reset" ist zum Zurücksetzen der Ansicht, der Knopf "Publish" löst den Publikationsvorgang aus, dazu mehr im nächsten Abschnitt.
### Hinzufügen eigener Objekte
Nun können Objekte aus einem bestehenden Projekt publiziert werden. Im Navigator links werden geeignete Objekte ausgewählt, dabei ist auch eine Mehrfachauswahl mithilfe der 'Shift' oder 'Strg' Taste möglich. Über den Kontextmenüintrag "Publish to SADE" im Navigator können die ausgewählten Objekte in die rechte Publikationsauswahl einefügt werden, wie im folgenden Screenshot zu sehen.
![SADE-Publish - Objekte hinzufügen](~assets/docs/sade-publish3.png)