Dear Gitlab users, due to maintenance reasons, Gitlab will not be available on Thursday 30.09.2021 from 5:00 pm to approximately 5:30 pm.

Commit 89b19c7b authored by Andreas Wagner's avatar Andreas Wagner
Browse files

Fix github putfile method.

parent 864e0126
......@@ -18,6 +18,7 @@ This webservice:
- creates a new [zenodo](https://about.zenodo.org/) deposit with a new DOI, adds the DOI to the TEI file and then uploads it to the deposit
- is capable of uploading the deposit to zenodo and *not publishing it yet* if another user-defined phrase is present; publishes the deposit otherwise
- is capable of looking up an existing deposit (if the TEI file mentions its own zenodo DOI entry) and creating a new version of it. This new version will have a new DOI, so the software deletes the old DOI and adds the new one before uploading the file to zenodo
- uploads and commits the TEI file that now has the new DOI field back to your github repository
<img align="left" style="margin-right:10px;" src="https://upload.wikimedia.org/wikipedia/commons/d/d1/Emblem-notice.svg"/>
Note that, since the XPath library that this service uses only supports basic XPath functions, you cannot really parse or manipulate the values via configuration settings. This means that you have to use some of zenodo's controlled vocabulary in your TEI markup! For instance, the license name or the editor roles in your TEI files are expected to be compatible with zenodo. You could, for example, use the `@n`-attribute of TEI's `&lt;editor&gt;` element to hold the required string like "cc-by" or use zenodo's controlled vocabulary for contributor types to specify the TEI `&lt;editor&gt;`'s `@role`-attribute...
......@@ -28,7 +29,7 @@ There are several ways of obtaining the software. It is not necessary to install
1. The default way of getting the software is downloading an asset on the [releases page](https://gitlab.gwdg.de/rg-mpg-de/tei2zenodo/-/releases). There are precompiled binaries zipped together with a configuration template on that page.
2. The package is maintained as a [Go](https://golang.org/) repository, so if you have Go installed, you can use it to compile and install the software in one single step: with the command `go get -u gitlab.gwdg.de/rg-mpg-de/tei2zenodo`. This will put the executable in the `$GOPATH/bin` directory, so that, on a standard Go installation, it can be found automatically from whatever directory you're in. (I recommend to put the configuration file in a `.t2z` subdirectory of your home directory. *Note: In order to be more standard-conforming, I will soon adopt $XDG_CONFIG_HOME and will have to figure out how this recommendation should be formulated on windows/mac systems.*)
2. The package is maintained as a [Go](https://golang.org/) repository, so if you have Go installed, you can use it to compile and install the software in one single step: with the command `go get -u gitlab.gwdg.de/rg-mpg-de/tei2zenodo`. This will put the executable in the `$GOPATH/bin` directory, so that, on a standard Go installation, it can be found automatically from whatever directory you're in.
3. If you want to compile the source code manually yourself, you need the [Go compiler](https://golang.org/) as well. Then, after retrieving the source code, either by cloning the git repository or by using one of the "download source code" options, you can compile it in its main directory with the command `go build -ldflags "-s -w" -o t2zd cmd/t2zd/main.go`. You can even skip the optimization switches (`-ldflags ...`) or the naming part (`-o t2zd`) and just say `go build cmd/t2zd/main.go` (and use `main` as the command to launch the server), but the longer command is the one I am using most of the time.
......@@ -93,11 +94,11 @@ In the `Git` config section, besides the obvious `host` and `token` keys, there
Since the webservice can receive hooks from no matter where, and then relies on information contained in the hook in order to "follow its nose" and retrieve files that it then uploads to zenodo, you can filter in many ways what is being processed:
If you add a [secret](https://developer.github.com/webhooks/securing/) to your github repository, you can add this string to a configuration setting called `secret`, which is used to check incoming messages against their signatures. You can also specify a repository in the `repo` key, the hooks of which will be processed exclusively. If hooks from other repositories come in, they will be ignored. If you leave the `repo` key empty, on the other hand, they will be processed as well. If you specify something in the `branch` key, it works (a) as a filter in the same way; also, tei2zenodo will retrieve the files to be processed from this branch. If you leave it empty, it defaults to "master". The `hook_user` key is another filter: If you specify a value here, only hooks initiated by this github user will be processed. (In the case of "push" webhooks, the payload's `pusher.name` field is compared to this config value.)
If you add a [secret](https://developer.github.com/webhooks/securing/) to your github repository, you can add this string to a configuration setting called `secret`, which is used to check incoming messages against their signatures. You can also specify a repository in the `repo` key, the hooks of which will be processed exclusively. If hooks from other repositories come in, they will be ignored. If you leave the `repo` key empty, on the other hand, they will be processed as well. If you specify something in the `branch` key, it works (a) as a filter in the same way; also, tei2zenodo will retrieve the files to be processed from this branch. If you leave it empty, it defaults to "master". The `hookUser` key is another filter: If you specify a value here, only hooks initiated by this github user will be processed. (In the case of "push" webhooks, the payload's `pusher.name` field is compared to this config value.)
The processing is triggered by a hook that can comprise several commits, and each of the commits can affect several files. If you specify something in the `commit_phrase` config key, only files of commits the messages of which contain the specified phrase will be processed. Leave empty to process all commits.
The processing is triggered by a hook that can comprise several commits, and each of the commits can affect several files. If you specify something in the `triggerPhrase` config key, only files of commits the messages of which contain the specified phrase will be processed. Leave empty to process all commits.
On the other hand, if you specify something in the `commit_dontpublish_phrase` key, processing happens as normal, but the deposits of all the files affected by commits with messages that contain this phrase will not be finally published. When you log in to zenodo and click on the "Upload" button, you will see your unpublished Uploads, waiting for you to inspect and finally publish them...
On the other hand, if you specify something in the `dontpublishPhrase` key, processing happens as normal, but the deposits of all the files affected by commits with messages that contain this phrase will not be finally published. When you log in to zenodo and click on the "Upload" button, you will see your unpublished Uploads, waiting for you to inspect and finally publish them...
Here is a complete github config section:
......@@ -108,9 +109,9 @@ Here is a complete github config section:
"secret": "ThIs_iS_NoT_ReAlLy_a_sMaRt_sEcReT!!",
"repo": "octocat/hello-world",
"branch": "public",
"hook_user": "foobar",
"commit_phrase": "(push to zenodo)",
"commit_dontpublish_phrase": "test"
"hookUser": "foobar",
"triggerPhrase": "(push to zenodo)",
"dontpublishPhrase": "test"
}
```
......
......@@ -30,6 +30,9 @@ func main() {
}
log.Printf("Starting tei2zenodo daemon")
if Config.Verbose {
log.Printf("conf: %v", Config)
}
// Get routes
router := routing.SetupRoutes(Config)
......
......@@ -19,9 +19,9 @@
"secret": "ThIs_iS_NoT_ReAlLy_a_sMaRt_sEcReT!!",
"repo": "octocat/hello-world",
"branch": "public",
"hook_user": "foobar",
"commit_phrase": "(push to zenodo)",
"commit_dontpublish_phrase": "test"
"hookUser": "foobar",
"triggerPhrase": "(push to zenodo)",
"dontpublishPhrase": "test"
},
"metadata": {
"fields": [
......
......@@ -31,12 +31,13 @@ func Configure(Config *tei2zenodo.Config) error {
"host": "https://sandbox.zenodo.org",
"token": "jLsTkUOMDU2fnGdGivGbB9TnMkPcADhIzEKHqqoVzMRsdYEC0Sqqfz72SPpt"})
viper.SetDefault("Git", map[string]string{"host": "https://api.github.com",
"token": "",
"repo": "octocat/hello-world",
"branch": "master",
"hook_user": "foobar",
"commit_phrase": "",
"commit_dontpublish_phrase": "test"})
"token": "",
"repo": "octocat/hello-world",
"branch": "master",
"hookUser": "foobar",
"commitTriggerPhrase": "",
"commitDontpublishPhrase": "test",
})
viper.SetEnvPrefix("T2Z")
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
......@@ -69,5 +70,7 @@ func Configure(Config *tei2zenodo.Config) error {
panic(fmt.Errorf("Cannot parse config: %+v", err))
}
Config.T2ZCommitMessage = "Zenodo DOI updated"
return nil
}
This diff is collapsed.
......@@ -8,7 +8,6 @@ import (
"fmt"
"io"
"net/http"
"strconv"
"strings"
log "github.com/sirupsen/logrus"
......@@ -27,15 +26,12 @@ import (
func SetupRoutes(conf tei2zenodo.Config) *gin.Engine {
// Switch gin to release mode
// gin.SetMode(gin.ReleaseMode)
gin.SetMode(gin.ReleaseMode)
// Create a gin router with logrus router and stock recovery
router := gin.New()
router.Use(ginerus.Ginerus(), gin.Recovery(), cors.Default())
// Create default gin router with the Logger and Recovery middleware already attached
// router := gin.Default()
// Routes - one for the html/webapp, one group per API version...
router.GET("/tei2zenodo.html", func(c *gin.Context) {
page := c.DefaultQuery("p", "index")
......@@ -68,7 +64,7 @@ func SetupRoutes(conf tei2zenodo.Config) *gin.Engine {
if filename == "" {
f, GFErr := t2zxml.GetFilename(r)
if GFErr != nil {
log.Printf("Problem reading filename: %+v", GFErr)
log.Errorf("Problem reading filename: %+v", GFErr)
AbortMsg(500, tei2zenodo.NewError("errParse", fmt.Sprintf("Error reading filename: %s", GFErr), 500, GFErr), c)
return
}
......@@ -84,24 +80,44 @@ func SetupRoutes(conf tei2zenodo.Config) *gin.Engine {
myDeposit.DoPublish = false
}
// Send file to processing...
PFErr := zenodo.ProcessFile(r, &myDeposit, &conf)
if PFErr != nil {
switch PFErr.Typ {
// Parse TEI file
var md tei2zenodo.ZMetadata
doi, PTErr := t2zxml.ParseTEI(r, &md, &conf)
if PTErr != nil {
switch PTErr.Typ {
case "errNoTEIXML":
{
log.Printf("Problem processing file %s: %+v", myDeposit.Filename, PFErr)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem processing file %s", myDeposit.Filename), 500, PFErr), c)
log.Warnf("Problem with file %s (%s): No TEI file.", myDeposit.Filename, myDeposit.GithubObjSHA)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem with file %s (%s): No TEI file.", myDeposit.Filename, myDeposit.GithubObjSHA), 500, PTErr), c)
return
}
default:
{
log.Printf("Problem processing file %s: %+v", myDeposit.Filename, PFErr)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem processing file %s", myDeposit.Filename), 500, PFErr), c)
log.Errorf("Error parsing TEI file: %v", PTErr)
AbortMsg(500, tei2zenodo.NewError("errParse", fmt.Sprintf("error parsing TEI file %s (%s): %s", myDeposit.Filename, myDeposit.GithubObjSHA, PTErr.Error()), 500, PTErr), c)
return
}
}
}
r.Seek(0, 0)
md.DOI = ""
ZPFErr := zenodo.ProcessFile(r, doi, &md, &myDeposit, &conf)
if ZPFErr != nil {
switch ZPFErr.Typ {
case "errNoTEIXML":
{
log.Errorf("Problem processing file %s: %+v", myDeposit.Filename, ZPFErr)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem processing file %s", myDeposit.Filename), 500, ZPFErr), c)
return
}
default:
{
log.Errorf("Problem processing file %s: %+v", myDeposit.Filename, ZPFErr)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem processing file %s", myDeposit.Filename), 500, ZPFErr), c)
return
}
}
}
log.Printf("====== All done ======")
......@@ -136,32 +152,27 @@ func SetupRoutes(conf tei2zenodo.Config) *gin.Engine {
mac := hmac.New(sha1.New, []byte(conf.Git.Secret))
mac.Write([]byte(file))
expectedMAC := []byte("sha1=" + hex.EncodeToString(mac.Sum(nil)))
log.Printf("Webhook with invalid HMAC signature received, ignoring. Sign.: %s, Message hash: %s", signature, expectedMAC)
log.Warnf("Webhook with invalid HMAC signature received, ignoring. Sign.: %s, Message hash: %s", signature, expectedMAC)
} else {
log.Printf("Webhook with invalid HMAC signature received, ignoring")
log.Warnf("Webhook with invalid HMAC signature received, ignoring")
}
AbortMsg(500, tei2zenodo.NewError("errWebhook", fmt.Sprintf("Webhook with invalid HMAC signature received, ignoring"), 500, nil), c)
AbortMsg(403, tei2zenodo.NewError("errWebhook", fmt.Sprintf("Webhook with invalid HMAC signature received, ignoring"), 403, nil), c)
return
}
if conf.Verbose {
log.Printf(" Webhook HMAC signature verified")
}
log.Debugf(" Webhook HMAC signature verified")
}
// Process Hook to extract files
doPublish, files, PHErr := github.ProcessHook(hookType, r, &conf)
files, PHErr := github.ProcessHook(hookType, r, &conf)
if PHErr != nil {
log.Printf("Error parsing webhook event: %v", PHErr)
log.Errorf("Error parsing webhook event: %v", PHErr)
AbortMsg(500, tei2zenodo.NewError("errWebhook", fmt.Sprintf("problem parsing webhook for %s event", hookType), 500, PHErr), c)
return
}
// log.Printf(" %d files returned", len(files))
log.Printf(" doPublish: %s", strconv.FormatBool(doPublish))
// Send each file to processing
for f := range files {
log.Printf("\n")
log.Printf("Processing file '%s'", f)
log.Printf("--- Process file '%s' ---", f)
var myDeposit tei2zenodo.Deposit
// myDeposit.Filename = strings.Replace(f, "/", "_", -1)
......@@ -172,38 +183,53 @@ func SetupRoutes(conf tei2zenodo.Config) *gin.Engine {
myDeposit.GithubObjSHA = files[f].ObjectSHA
myDeposit.GithubRawURL = files[f].RawURL
// Get file
r, GDLErr := github.DownloadFile(&myDeposit, &conf)
if GDLErr != nil {
log.Printf("Problem downloading file %s (%s): %v ...", myDeposit.Filename, myDeposit.GithubObjSHA, GDLErr)
log.Errorf("Problem downloading file %s (%s): %v ...", myDeposit.Filename, myDeposit.GithubObjSHA, GDLErr)
AbortMsg(500, tei2zenodo.NewError("errNetComm", fmt.Sprintf("error downloading file %s (%s): %s", myDeposit.Filename, myDeposit.GithubObjSHA, GDLErr.Error()), 500, GDLErr), c)
return
}
// log.Tracef("Beginning of submitted file %s: %s ...", filename, r[:100])
ZPFErr := zenodo.ProcessFile(r, &myDeposit, &conf)
if ZPFErr != nil {
switch ZPFErr.Typ {
// Parse TEI file
var md tei2zenodo.ZMetadata
doi, PTErr := t2zxml.ParseTEI(r, &md, &conf)
if PTErr != nil {
switch PTErr.Typ {
case "errNoTEIXML":
{
log.Printf("Problem with file %s (%s): No TEI file.", myDeposit.Filename, myDeposit.GithubObjSHA)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem with file %s (%s): No TEI file.", myDeposit.Filename, myDeposit.GithubObjSHA), 500, ZPFErr), c)
log.Warnf("Problem with file %s (%s): No TEI file.", myDeposit.Filename, myDeposit.GithubObjSHA)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem with file %s (%s): No TEI file.", myDeposit.Filename, myDeposit.GithubObjSHA), 500, PTErr), c)
return
}
default:
{
log.Printf("Problem processing file %s: %v ...", myDeposit.Filename, ZPFErr)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem processing file %s (%s): %s", myDeposit.Filename, myDeposit.GithubObjSHA, ZPFErr.Error()), 500, ZPFErr), c)
log.Errorf("Error parsing TEI file: %v", PTErr)
AbortMsg(500, tei2zenodo.NewError("errParse", fmt.Sprintf("error parsing TEI file %s (%s): %s", myDeposit.Filename, myDeposit.GithubObjSHA, PTErr.Error()), 500, PTErr), c)
return
}
}
}
r.Seek(0, 0)
md.DOI = ""
ZPFErr := zenodo.ProcessFile(r, doi, &md, &myDeposit, &conf)
if ZPFErr != nil {
log.Warnf("Problem processing file %s: %v ...", myDeposit.Filename, ZPFErr)
AbortMsg(500, tei2zenodo.NewError("errZProcessing", fmt.Sprintf("problem processing file %s (%s): %s", myDeposit.Filename, myDeposit.GithubObjSHA, ZPFErr.Error()), 500, ZPFErr), c)
return
}
_, GPFErr := github.PutFile(&myDeposit, &conf, c)
if GPFErr != nil {
log.Printf("Error putting file to github: %v", GPFErr)
log.Errorf("Error putting file to github: %v", GPFErr)
AbortMsg(500, tei2zenodo.NewError("errPutFile", fmt.Sprintf("problem putting file to github: %v", f), 500, GPFErr), c)
return
}
log.Printf(" Successfully processed file %s (DOI %s, github SHA %s).", myDeposit.Filename, myDeposit.DepositDOI, myDeposit.DepositDOI)
/*
// The upload to github requires more than just a token:
// you need to authenticate an app to act in lieu of the user in github...
......
......@@ -21,7 +21,7 @@ func GetFilename(r io.Reader) (string, error) {
// Parse document (in r) wih antchfx/xmlquery...
doc, err := xmlquery.Parse(r)
if err != nil {
log.Printf("Could not parse xml.\n")
log.Errorf("Could not parse xml.")
return "", tei2zenodo.NewError("errParse", "could not parse xml", 500, err)
}
t := xmlquery.FindOne(doc, `/TEI/@xml:id`)
......@@ -39,21 +39,21 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
conf := c.Metadata
var doc *xmlquery.Node
log.Printf(" Parse TEI XML file")
log.Printf("--- Parse TEI file ---")
// Parse document r wih antchfx/xmlquery...
doc, PErr := xmlquery.Parse(r)
if PErr != nil {
log.Printf("Could not parse xml.\n")
log.Errorf("Could not parse xml.")
return "", tei2zenodo.NewError("errParse", "could not parse xml", 500, PErr)
}
if doc == nil || (doc.FirstChild == nil && doc.InnerText() == "") {
log.Printf(" Parse returned no or empty node only.\n")
log.Errorf(" Parse returned no or empty node only.")
return "", tei2zenodo.NewError("errNoTEIXML", "not a TEI XML file", 400, nil)
}
rootElement := xmlquery.FindOne(doc, `/*[1]`)
if !(rootElement.NamespaceURI == "http://www.tei-c.org/ns/1.0" && rootElement.Data == "TEI") {
log.Printf(" This is not a TEI XML file. %+v", rootElement)
log.Errorf(" This is not a TEI XML file. %+v", rootElement)
return "", tei2zenodo.NewError("errNoTEIXML", "not a TEI XML file", 400, nil)
}
......@@ -86,8 +86,8 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
}
for j := range conf.Fields {
if conf.Fields[j].Field == jsonFieldname {
// log.Printf("Found config for %s.", jsonFieldname)
// log.Printf("It's a %s value...", structFieldtype)
// log.Debugf("Found config for %s.", jsonFieldname)
// log.Debugf("It's a %s value...", structFieldtype)
cplx := false
for _, a := range complexTypes {
if a == structFieldtype.String() {
......@@ -110,22 +110,22 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
zJSONFieldname := varType.Field(m).Tag.Get("json")
for m := range confSubfields {
if confSubfields[m].Field == zJSONFieldname {
// log.Printf("Found config for %s.", zJSONFieldname)
// log.Printf("It's a %s value...", zStructFieldtype)
// log.Debugf("Found config for %s.", zJSONFieldname)
// log.Debugf("It's a %s value...", zStructFieldtype)
if confSubfields[m].XPath != "" {
xrpath := confSubfields[m].XPath
var u string
t := xmlquery.FindOne(l, xrpath)
if n := t; t != nil {
u = re.ReplaceAllString(n.InnerText(), " ")
// log.Printf("%s.%s found. Set to '%v' ...\n", structFieldname, zStructFieldname, u)
// log.Debugf("%s.%s found. Set to '%v' ...\n", structFieldname, zStructFieldname, u)
zcf.SetString(u)
}
} else if confSubfields[m].XExpression != "" {
xexpr, err := xpath.Compile(confSubfields[m].XExpression)
if err != nil {
log.Printf("Erroneous XPath expression: %s ...", confSubfields[m].XExpression)
log.Warnf("Erroneous XPath expression: %s ...", confSubfields[m].XExpression)
return "", tei2zenodo.NewError("errBadConfig", fmt.Sprintf("erroneous XPath expression: %s", confSubfields[m].XExpression), 500, err)
}
switch zStructFieldtype.String() {
......@@ -134,7 +134,7 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
// TODO: Check for wrong types (e.g. xepr returning int instead of string)
u = xexpr.Evaluate(xmlquery.CreateXPathNavigator(l)).(string)
if u != "" {
// log.Printf("%s.%s found. Set to '%v' ...\n", structFieldname, zStructFieldname, u)
// log.Debugf("%s.%s found. Set to '%v' ...\n", structFieldname, zStructFieldname, u)
zcf.SetString(u)
}
......@@ -147,15 +147,15 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
newSlice := reflect.Append(f, reflect.ValueOf(n))
zcf.Set(newSlice)
}
// log.Printf("%s.%s found. Set to '%v' ...\n", structFieldname, zStructFieldname, u)
// log.Debugf("%s.%s found. Set to '%v' ...\n", structFieldname, zStructFieldname, u)
default:
log.Printf("Unknown (hardcoded?) metadata type: %s.%s ...", structFieldtype, zStructFieldtype)
log.Errorf("Unknown (hardcoded?) metadata type: %s.%s ...", structFieldtype, zStructFieldtype)
return "", tei2zenodo.NewError("errInternal", fmt.Sprintf("xml: unknown (hardcoded?) metadata type: %s.%s", structFieldtype, zStructFieldtype), 500, nil)
}
} else if confSubfields[m].Field == "name" || confSubfields[m].Field == "type" {
log.Printf("Problem with config: XPath or XExpression missing in %v ...", conf.Fields[j])
log.Errorf("Problem with config: XPath or XExpression missing in %v ...", conf.Fields[j])
return "", tei2zenodo.NewError("errBadConfig", fmt.Sprintf("XPath or XExpression missing in %v ...", conf.Fields[j]), 500, nil)
}
}
......@@ -187,13 +187,13 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
newSlice := reflect.Append(f, reflect.ValueOf(zc.Interface().(tei2zenodo.ZDate)))
f.Set(newSlice)
default:
log.Printf("Problem with type conversion of %s (%s)", structFieldname, varType.Name())
log.Errorf("Problem with type conversion of %s (%s)", structFieldname, varType.Name())
return "", tei2zenodo.NewError("errInternal", fmt.Sprintf("type problem in %s [%s]", structFieldname, varType.Name()), 500, nil)
}
}
} else {
log.Printf("Problem with config: XPath missing in %v ...", conf.Fields[j])
log.Errorf("Problem with config: XPath missing in %v ...", conf.Fields[j])
return "", tei2zenodo.NewError("errBadConfig", fmt.Sprintf("XPath or XExpression missing in %v ...", conf.Fields[j]), 500, nil)
}
......@@ -205,7 +205,7 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
t := xmlquery.FindOne(doc, xpath)
if n := t; t != nil {
u = re.ReplaceAllString(n.InnerText(), " ")
// log.Printf("%s found. Set to '%v' ...\n", structFieldname, u)
// log.Debugf("%s found. Set to '%v' ...\n", structFieldname, u)
f.SetString(u)
}
......@@ -217,16 +217,16 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
newSlice := reflect.Append(f, reflect.ValueOf(v))
f.Set(newSlice)
}
// log.Printf("%s found. Set to '%v' ...\n", structFieldname, u)
// log.Debugf("%s found. Set to '%v' ...\n", structFieldname, u)
default:
log.Printf("Unknown (hardcoded?) metadata type: %s ...", structFieldtype)
log.Errorf("Unknown (hardcoded?) metadata type: %s ...", structFieldtype)
return "", tei2zenodo.NewError("errInternal", fmt.Sprintf("xml: unknown (hardcoded?) metadata type: %s", structFieldtype), 500, nil)
}
} else if conf.Fields[j].XExpression != "" {
xexpr, err := xpath.Compile(conf.Fields[j].XExpression)
if err != nil {
log.Printf("Erroneous XPath expression: %s ...", conf.Fields[j].XExpression)
log.Errorf("Erroneous XPath expression: %s ...", conf.Fields[j].XExpression)
return "", tei2zenodo.NewError("errBadConfig", fmt.Sprintf("erroneous XPath expression: %s", conf.Fields[j].XExpression), 500, err)
}
switch structFieldtype.String() {
......@@ -235,7 +235,7 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
// TODO: Check for wrong types (e.g. xepr returning int instead of string)
u = xexpr.Evaluate(xmlquery.CreateXPathNavigator(doc)).(string)
if u != "" {
// log.Printf("%s found. Set to '%v' ...\n", structFieldname, u)
// log.Debugf("%s found. Set to '%v' ...\n", structFieldname, u)
f.SetString(u)
}
......@@ -248,14 +248,14 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
newSlice := reflect.Append(f, reflect.ValueOf(n))
f.Set(newSlice)
}
// log.Printf("%s found. Set to '%v' ...\n", structFieldname, u)
// log.Debugf("%s found. Set to '%v' ...\n", structFieldname, u)
default:
log.Printf("Unknown (hardcoded?) metadata type: %s ...", structFieldtype)
log.Errorf("Unknown (hardcoded?) metadata type: %s ...", structFieldtype)
return "", tei2zenodo.NewError("errInternal", fmt.Sprintf("xml: unknown (hardcoded?) metadata type: %s", structFieldtype), 500, nil)
}
} else {
log.Printf("Malformed config entry: %v ...", conf.Fields[j])
log.Errorf("Malformed config entry: %v ...", conf.Fields[j])
return "", tei2zenodo.NewError("errBadConfig", fmt.Sprintf("XPath, XExpression or Subfields missing in %v ...", conf.Fields[j]), 500, nil)
}
}
......@@ -263,16 +263,16 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
}
doi := md.DOI
log.Printf(" Success.")
log.Printf(" TEI parsed successfully.")
if c.Verbose {
log.Printf(" Title: %v", md.Title)
log.Debugf(" Title: %v", md.Title)
if len(md.Creators) > 0 {
log.Printf(" Creator 1: %v", md.Creators[0])
log.Debugf(" Creator 1: %v", md.Creators[0])
}
if len(md.Contributors) > 0 {
log.Printf(" Contributor 1: %v", md.Contributors[0])
log.Debugf(" Contributor 1: %v", md.Contributors[0])
}
log.Printf(" DOI: %s", doi)
log.Debugf(" DOI: %s", doi)
}
return doi, nil
......@@ -282,18 +282,16 @@ func ParseTEI(r io.Reader, md *tei2zenodo.ZMetadata, c *tei2zenodo.Config) (stri
// it returns the string serialization of the new document and an error value
func MixinDOI(r io.Reader, doi string, c *tei2zenodo.Config) (string, error) {
log.Printf(" Mix new DOI in TEI document")
// Parse document (in r)...
var doc etree.Document
_, err := doc.ReadFrom(r)
if err != nil {
log.Printf("Could not parse xml.\n")
log.Errorf("Could not parse xml.")
return "", tei2zenodo.NewError("errParse", "could not parse xml", 500, err)
}
pStmt := doc.FindElement(`/TEI/teiHeader/fileDesc/publicationStmt`)
if pStmt == nil {
log.Printf("XML file had no /TEI/teiHeader/fileDesc/publicationStmt element.")
log.Errorf("XML file had no /TEI/teiHeader/fileDesc/publicationStmt element.")
err := fmt.Errorf("XML file had no /TEI/teiHeader/fileDesc/publicationStmt element")
return "", tei2zenodo.NewError("errParse", "XML file had no /TEI/teiHeader/fileDesc/publicationStmt element", 500, err)
}
......@@ -301,7 +299,7 @@ func MixinDOI(r io.Reader, doi string, c *tei2zenodo.Config) (string, error) {
topLevelIdno := pStmt.FindElement(`./idno`)
if topLevelIdno == nil { // publicationStmt does not contain any <idno> element -> add one as last child of pStmt
if c.Verbose {
log.Printf("No idno element present. Create one.")
log.Debugf("No idno element present. Create one.")
}
targetIdno := pStmt.CreateElement("idno")
targetIdno.CreateAttr("type", "DOI")
......@@ -332,10 +330,10 @@ func MixinDOI(r io.Reader, doi string, c *tei2zenodo.Config) (string, error) {
output, err := doc.WriteToString()
if err != nil {
log.Printf("Error serializing xml to string.")
log.Errorf("Error serializing xml to string.")
return "", tei2zenodo.NewError("errSerializing", "problem serializing XML to string", 500, err)
} else if output == "" {
log.Printf("Serializing xml resulted in empty string.")
log.Errorf("Serializing xml resulted in empty string.")
return "", tei2zenodo.NewError("errSerializing", "serializing XML resulted in empty string", 500, err)
}
......
This diff is collapsed.
......@@ -4,15 +4,16 @@ package tei2zenodo
// Config is the struct of the application's general configuration.
type Config struct {
ListenSpec int64
Verbose bool
APIRoot string
FileAPI string
WebhookAPI string
Zenodo ZenodoConfig
Git GitConfig
Metadata MetadataConfig
Log LoggingConfig
ListenSpec int64
Verbose bool
APIRoot string
FileAPI string
WebhookAPI string
Zenodo ZenodoConfig
Git GitConfig
Metadata MetadataConfig
Log LoggingConfig
T2ZCommitMessage string // This is not configurable in fact but stored here to be passed around among modules. Stores the commit message that this service creates and is set in conf.go
}
// LoggingConfig specifies all the parameters needed for logging.
......@@ -35,9 +36,9 @@ type GitConfig struct {
Secret string
Repo string
Branch string
HookUser string `json:"hook_user"`
Phrase string `json:"commit_phrase"`
DontPublishPhrase string `json:"commit_dontpublish_phrase"`
HookUser string
TriggerPhrase string
DontPublishPhrase string
}
// MetadataConfig specifies metadata fields and xpaths to retrieve their values if the need to be different from the defaults.
......@@ -237,7 +238,7 @@ type ZDate struct {
// Error messages
type Error struct {
Typ string // errNoTEIXML, errParse, errWebhook, errZProcessing, errGHProcessing, errSerializing, errInternal, errBadConfig, errNetComm
Typ string // errNoTEIXML, errParse, errWebhook, errZProcessing, errGHProcessing, errSerializing, errInternal, errBadConfig, errNetComm, errBadRequest
Message string
HTTPCode int
ErrorVal error // what may have been raised by previous code
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment