From 81af9805d563be553167a137812ec3b5bf95d321 Mon Sep 17 00:00:00 2001
From: Konstantin Baierer <unixprog@gmail.com>
Date: Fri, 8 Nov 2019 13:55:34 +0100
Subject: [PATCH] repos.json
---
kwalitee/config.yml | 6 ++++++
repos.json | 8 ++++----
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/kwalitee/config.yml b/kwalitee/config.yml
index 5e55822..a7993a7 100644
--- a/kwalitee/config.yml
+++ b/kwalitee/config.yml
@@ -13,8 +13,11 @@ repolist:
- url: https://github.com/OCR-D/ocrd_im6convert
compliant_cli: true
- url: https://github.com/OCR-D/ocrd_keraslm
+ compliant_cli: true
- url: https://github.com/OCR-D/ocrd_kraken
+ compliant_cli: false
- url: https://github.com/OCR-D/ocrd_ocropy
+ compliant_cli: true
- url: https://github.com/OCR-D/ocrd_olena
compliant_cli: true
- url: https://github.com/OCR-D/ocrd_segment
@@ -27,10 +30,13 @@ repolist:
compliant_cli: true
- url: https://github.com/mjenckel/LAYoutERkennung
official: true
+ compliant_cli: false
- url: https://github.com/ocr-d-modul-2-segmentierung/segmentation-runner
official: true
compliant_cli: false
- url: https://github.com/qurator-spk/dinglehopper
+ compliant_cli: true
# - url: https://github.com/qurator-spk/pixelwise_segmentation_SBB
- url: https://github.com/seuretm/ocrd_typegroups_classifier
official: true
+ compliant_cli: true
diff --git a/repos.json b/repos.json
index 2e6b153..e0fea45 100644
--- a/repos.json
+++ b/repos.json
@@ -360,7 +360,7 @@
"url": "https://github.com/OCR-D/ocrd_im6convert"
},
{
- "compliant_cli": false,
+ "compliant_cli": true,
"files": {
"Dockerfile": null,
"README.md": "# ocrd_keraslm\n character-level language modelling using Keras\n\n\n## Introduction\n\nThis is a tool for statistical _language modelling_ (predicting text from context) with recurrent neural networks. It models probabilities not on the word level but the _character level_ so as to allow open vocabulary processing (avoiding morphology, historic orthography and word segmentation problems). It manages a vocabulary of mapped characters, which can be easily extended by training on more text. Above that, unmapped characters are treated with underspecification.\n\nIn addition to character sequences, (meta-data) context variables can be configured as extra input. \n\n### Architecture\n\nThe model consists of:\n\n0. an input layer: characters are represented as indexes from the vocabulary mapping, in windows of a number `length` of characters,\n1. a character embedding layer: window sequences are converted into dense vectors by looking up the indexes in an embedding weight matrix,\n2. a context embedding layer: context variables are converted into dense vectors by looking up the indexes in an embedding weight matrix, \n3. character and context vector sequences are concatenated,\n4. a number `depth` of hidden layers: each with a number `width` of hidden recurrent units of _LSTM cells_ (Long Short-term Memory) connected on top of each other,\n5. an output layer derived from the transposed character embedding matrix (weight tying): hidden activations are projected linearly to vectors of dimensionality equal to the character vocabulary size, then softmax is applied returning a probability for each possible value of the next character, respectively.\n\n![model graph depiction](model-graph.png \"graph with 1 context variable\")\n\nThe model is trained by feeding windows of text in index representation to the input layer, calculating output and comparing it to the same text shifted backward by 1 character, and represented as unit vectors (\"one-hot coding\") as target. The loss is calculated as the (unweighted) cross-entropy between target and output. Backpropagation yields error gradients for each layer, which is used to iteratively update the weights (stochastic gradient descent).\n\nThis is implemented in [Keras](https://keras.io) with [Tensorflow](https://www.tensorflow.org/) as backend. It automatically uses a fast CUDA-optimized LSTM implementation (Nividia GPU and Tensorflow installation with GPU support, see below), both in learning and in prediction phase, if available.\n\n\n### Modes of operation\n\nNotably, this model (by default) runs _statefully_, i.e. by implicitly passing hidden state from one window (batch of samples) to the next. That way, the context available for predictions can be arbitrarily long (above `length`, e.g. the complete document up to that point), or short (below `length`, e.g. at the start of a text). (However, this is a passive perspective above `length`, because errors are never back-propagated any further in time during gradient-descent training.) This is favourable to stateless mode because all characters can be output in parallel, and no partial windows need to be presented during training (which slows down).\n\nBesides stateful mode, the model can also be run _incrementally_, i.e. by explicitly passing hidden state from the caller. That way, multiple alternative hypotheses can be processed together. This is used for generation (sampling from the model) and alternative decoding (finding the best path through a sequence of alternatives).\n\n### Context conditioning\n\nEvery text has meta-data like time, author, text type, genre, production features (e.g. print vs typewriter vs digital born rich text, OCR version), language, structural element (e.g. title vs heading vs paragraph vs footer vs marginalia), font family (e.g. Antiqua vs Fraktura) and font shape (e.g. bold vs letter-spaced vs italic vs normal) etc. \n\nThis information (however noisy) can be very useful to facilitate stochastic modelling, since language has an extreme diversity and complexity. To that end, models can be conditioned on extra inputs here, termed _context variables_. The model learns to represent these high-dimensional discrete values as low-dimensional continuous vectors (embeddings), also entering the recurrent hidden layers (as a form of simple additive adaptation).\n\n### Underspecification\n\nIndex zero is reserved for unmapped characters (unseen contexts). During training, its embedding vector is regularised to occupy a center position of all mapped characters (all other contexts), and the hidden layers get to see it every now and then by random degradation. At runtime, therefore, some unknown character (some unknown context) represented as zero does not disturb follow-up predictions too much.\n\n\n## Installation\n\nRequired Ubuntu packages:\n\n* Python (``python`` or ``python3``)\n* pip (``python-pip`` or ``python3-pip``)\n* virtualenv (``python-virtualenv`` or ``python3-virtualenv``)\n\nCreate and activate a virtualenv as usual.\n\nIf you need a custom version of ``keras`` or ``tensorflow`` (like [GPU support](https://www.tensorflow.org/install/install_sources)), install them via `pip` now.\n\nTo install Python dependencies and this module, then do:\n```shell\nmake deps install\n```\nWhich is the equivalent of:\n```shell\npip install -r requirements.txt\npip install -e .\n```\n\nUseful environment variables are:\n- ``TF_CPP_MIN_LOG_LEVEL`` (set to `1` to suppress most of Tensorflow's messages\n- ``CUDA_VISIBLE_DEVICES`` (set empty to force CPU even in a GPU installation)\n\n\n## Usage\n\nThis packages has two user interfaces:\n\n### command line interface `keraslm-rate`\n\nTo be used with string arguments and plain-text files.\n\n```shell\nUsage: keraslm-rate [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n --help Show this message and exit.\n\nCommands:\n train train a language model\n test get overall perplexity from language model\n apply get individual probabilities from language model\n generate sample characters from language model\n print-charset Print the mapped characters\n prune-charset Delete one character from mapping\n plot-char-embeddings-similarity\n Paint a heat map of character embeddings\n plot-context-embeddings-similarity\n Paint a heat map of context embeddings\n plot-context-embeddings-projection\n Paint a 2-d PCA projection of context embeddings\n```\n\nExamples:\n```shell\nkeraslm-rate train --width 64 --depth 4 --length 256 --model model_dta_64_4_256.h5 dta_komplett_2017-09-01/txt/*.tcf.txt\nkeraslm-rate generate -m model_dta_64_4_256.h5 --number 6 \"f\u00fcr die Wi\u017f\u017fen\"\nkeraslm-rate apply -m model_dta_64_4_256.h5 \"so sch\u00e4dlich ist es Borkickheile zu pflanzen\"\nkeraslm-rate test -m model_dta_64_4_256.h5 dta_komplett_2017-09-01/txt/grimm_*.tcf.txt\n```\n\n### [OCR-D processor](https://github.com/OCR-D/core) interface `ocrd-keraslm-rate`\n\nTo be used with [PageXML](https://www.primaresearch.org/tools/PAGELibraries) documents in an [OCR-D](https://github.com/OCR-D/spec/) annotation workflow. Input could be anything with a textual annotation (`TextEquiv` on the given `textequiv_level`). The LM rater could be used for both quality control (without alternative decoding, using only each first index `TextEquiv`) and part of post-correction (with `alternative_decoding=True`, finding the best path among `TextEquiv` indexes).\n\n```json\n \"tools\": {\n \"ocrd-keraslm-rate\": {\n \"executable\": \"ocrd-keraslm-rate\",\n \"categories\": [\n \"Text recognition and optimization\"\n ],\n \"steps\": [\n \"recognition/text-recognition\"\n ],\n \"description\": \"Rate elements of the text with a character-level LSTM language model in Keras\",\n \"input_file_grp\": [\n \"OCR-D-OCR-TESS\",\n \"OCR-D-OCR-KRAK\",\n \"OCR-D-OCR-OCRO\",\n \"OCR-D-OCR-CALA\",\n \"OCR-D-OCR-ANY\",\n \"OCR-D-COR-CIS\",\n \"OCR-D-COR-ASV\"\n ],\n \"output_file_grp\": [\n \"OCR-D-COR-LM\"\n ],\n \"parameters\": {\n \"model_file\": {\n \"type\": \"string\",\n \"format\": \"uri\",\n \"content-type\": \"application/x-hdf;subtype=bag\",\n \"description\": \"path of h5py weight/config file for model trained with keraslm\",\n \"required\": true,\n \"cacheable\": true\n },\n \"textequiv_level\": {\n \"type\": \"string\",\n \"enum\": [\"region\", \"line\", \"word\", \"glyph\"],\n \"default\": \"glyph\",\n \"description\": \"PAGE XML hierarchy level to evaluate TextEquiv sequences on\"\n },\n \"alternative_decoding\": {\n \"type\": \"boolean\",\n \"description\": \"whether to process all TextEquiv alternatives, finding the best path via beam search, and delete each non-best alternative\",\n \"default\": true\n },\n \"beam_width\": {\n \"type\": \"number\",\n \"format\": \"integer\",\n \"description\": \"maximum number of best partial paths to consider during search with alternative_decoding\",\n \"default\": 100\n }\n }\n }\n }\n```\n\nExamples:\n```shell\nmake deps-test # installs ocrd_tesserocr\nmake test/assets # downloads GT, imports PageXML, builds workspaces\nocrd workspace clone -a test/assets/kant_aufklaerung_1784/mets.xml ws1\ncd ws1\nocrd-tesserocr-segment-region -I OCR-D-IMG -O OCR-D-SEG-BLOCK\nocrd-tesserocr-segment-line -I OCR-D-SEG-BLOCK -O OCR-D-SEG-LINE\nocrd-tesserocr-recognize -I OCR-D-SEG-LINE -O OCR-D-OCR-TESS-WORD -p '{ \"textequiv_level\" : \"word\", \"model\" : \"Fraktur\" }'\nocrd-tesserocr-recognize -I OCR-D-SEG-LINE -O OCR-D-OCR-TESS-GLYPH -p '{ \"textequiv_level\" : \"glyph\", \"model\" : \"deu-frak\" }'\n# get confidences and perplexity:\nocrd-keraslm-rate -I OCR-D-OCR-TESS-WORD -O OCR-D-OCR-LM-WORD -p '{ \"model_file\": \"model_dta_64_4_256.h5\", \"textequiv_level\": \"word\", \"alternative_decoding\": false }'\n# also get best path:\nocrd-keraslm-rate -I OCR-D-OCR-TESS-GLYPH -O OCR-D-OCR-LM-GLYPH -p '{ \"model_file\": \"model_dta_64_4_256.h5\", \"textequiv_level\": \"glyph\", \"alternative_decoding\": true, \"beam_width\": 10 }'\n```\n\n## Testing\n\n```shell\nmake deps-test test\n```\nWhich is the equivalent of:\n```shell\npip install -r requirements_test.txt\ntest -e test/assets || test/prepare_gt.bash test/assets\ntest -f model_dta_test.h5 || keraslm-rate train -m model_dta_test.h5 test/assets/*.txt\nkeraslm-rate test -m model_dta_test.h5 test/assets/*.txt\npython -m pytest test $(PYTEST_ARGS)\n```\n\nSet `PYTEST_ARGS=\"-s --verbose\"` to see log output (`-s`) and individual test results (`--verbose`).\n",
@@ -905,7 +905,7 @@
"url": "https://github.com/OCR-D/ocrd_kraken"
},
{
- "compliant_cli": false,
+ "compliant_cli": true,
"files": {
"Dockerfile": "FROM ocrd/core\nMAINTAINER OCR-D\nENV DEBIAN_FRONTEND noninteractive\nENV PYTHONIOENCODING utf8\nENV LC_ALL C.UTF-8\nENV LANG C.UTF-8\n\nWORKDIR /build-ocrd\nCOPY setup.py .\nCOPY requirements.txt .\nCOPY README.md .\nRUN apt-get update && \\\n apt-get -y install --no-install-recommends \\\n ca-certificates \\\n make \\\n git\nCOPY ocrd_ocropy ./ocrd_ocropy\nRUN pip3 install --upgrade pip\nRUN make deps install\n\nENTRYPOINT [\"/bin/sh\", \"-c\"]\n",
"README.md": "# ocrd_ocropy\n\n[](https://travis-ci.org/OCR-D/ocrd_ocropy)\n\n[](https://hub.docker.com/r/ocrd/ocropy/tags/)\n\n> Wrapper for the ocropy OCR engine\n",
@@ -3615,7 +3615,7 @@
"url": "https://github.com/ocr-d-modul-2-segmentierung/segmentation-runner"
},
{
- "compliant_cli": false,
+ "compliant_cli": true,
"files": {
"Dockerfile": null,
"README.md": "dinglehopper\n============\n\ndinglehopper is an OCR evaluation tool and reads [ALTO](https://github.com/altoxml), [PAGE](https://github.com/PRImA-Research-Lab/PAGE-XML) and text files.\n\n[](https://travis-ci.org/qurator-spk/dinglehopper)\n\nGoals\n-----\n* Useful\n * As a UI tool\n * For an automated evaluation\n * As a library\n* Unicode support\n\nUsage\n-----\n~~~\ndinglehopper some-document.gt.page.xml some-document.ocr.alto.xml\n~~~\nThis generates `report.html` and `report.json`.\n\n\nAs a OCR-D processor:\n~~~\nocrd-dinglehopper -m mets.xml -I OCR-D-GT-PAGE,OCR-D-OCR-TESS -O OCR-D-OCR-TESS-EVAL\n~~~\nThis generates HTML and JSON reports in the `OCR-D-OCR-TESS-EVAL` filegroup.\n\n\n\n",
@@ -3664,7 +3664,7 @@
"url": "https://github.com/qurator-spk/dinglehopper"
},
{
- "compliant_cli": false,
+ "compliant_cli": true,
"files": {
"Dockerfile": null,
"README.md": "# ocrd_typegroups_classifier\n\n> Typegroups classifier for OCR\n\n## Quick setup\n\nIf needed, create a virtual environment for Python 3 (it was tested\nsuccessfully with Python 3.7), activate it, and install ocrd.\n\n```sh\nvirtualenv -p python3 ocrd-venv3\nsource ocrd-venv3/bin/activate\npip3 install ocrd\n```\n\nEnter in the folder containing the tool:\n```cd ocrd_typegroups_classifier/```\n\nInstall the module and its dependencies\n\n```\nmake install\nmake deps\n```\n\nFinally, run the test:\n\n```\nsh test/test.sh\n```\n\n** Important: ** The test makes sure that the system does work. For\nspeed reasons, a very small neural network is used and applied only to\nthe top-left corner of the image, therefore the quality of the results\nwill be of poor quality.\n\n## Models\n\nThe model classifier-1.tgc is based on a ResNet-18, with less neurons\nper layer than the usual model. It was briefly trained on 12 classes:\nAdornment, Antiqua, Bastarda, Book covers and other irrelevant data,\nEmpty Pages, Fraktur, Griechisch, Hebr\u00e4isch, Kursiv, Rotunda, Textura,\nand Woodcuts - Engravings.\n\n## Heatmap Generation ##\nGiven a trained model, it is possible to produce heatmaps corresponding\nto classification results. Syntax:\n\n```\npython3 tools/heatmap.py ocrd_typegroups_classifier/models/classifier.tgc sample.jpg out\n```",
--
GitLab