Re-validate tgoaipmh results with openarchives validator

Check results of: http://www.openarchives.org/Register/ValidateSite?log=MW73106Z

• Correct dates (see comments)
• Fix REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&from=junk --> must not be 500!
• Fix REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&until=junk --> must not be 500!
• REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&from=2002-02-05&until=2002-02-06T05:35:00Z --> FAIL Failed to extract error code from the response to request: verb=ListRecords&metadataPrefix=oai_dc&from=2002-02-05&until=2002-02-06T05:35:00Z The request has different granularities for the from and until parameters.
• REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListMetadataFormats&identifier=textgrid%3Ax9t0%2E0 --> FAIL Server failed to respond to the ListMetadataFormats request (HTTP header values: status=500 Internal Server Error, age=0, lifetime=3600, is fresh:=1) FAIL Can't check metadataFormats available for item textgrid:x9t0.0, no response to ListMetadataFormats request. FAIL There are no metadata formats available to use with the GetRecord request. The metadataPrefix oai_dc will be used for later tests even though it seems unsupported.
• REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&until=2010-06-11T02:32:40Z --> FAIL Failed to extract error code from the response to request: verb=ListRecords&metadataPrefix=oai_dc&until=2010-06-11T02:32:40Z The request specified a date one year before the earliestDatestamp given in the Identify response. There should therefore not be any records with datestamps on or before this date and a noRecordsMatch error code should be returned.
• Checking for correct use of resumptionToken (if used) REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc GET FAIL A cursor value was supplied with the resumptionToken but it did not have the correct value zero for the first response. The value was '100'. NOTE Got resumptionToken DnF1ZXJ5VGhlbkZldGNoBQAAAAAABtbaFmk2UHpVbDZXVEc2MEFKQUlRWnoxaHcAAAAAAAbW3BZpNlB6VWw2V1RHNjBBSkFJUVp6MWh3AAAAAAAG1tgWaTZQelVsNldURzYwQUpBSVFaejFodwAAAAAABtbZFmk2UHpVbDZXVEc2MEFKQUlRWnoxaHcAAAAAAAbW2xZpNlB6VWw2V1RHNjBBSkFJUVp6MWh3

added tg-oaipmh label

assigned to @sfunk

created branch 100-re-validate-tgoaipomh-results-with-openarchives-validator to address this issue

changed title from Re-validate tgoaipomh results with openarchives validator to Re-validate tgoaipmh results with openarchives validator

Dates in OAI-PMH

1. OAI-PMH (UTC - see Identify) --> TG & DH

<OAI-PMH>
  <responseDate>2023-01-30T17:42:39Z</responseDate>
</OAI-PMH>

2. OAI-PMH (UTC - see Identify) --> TG & DH

<header>
  <identifier>textgrid:xj2x.0</identifier>
  <datestamp>2012-02-14T21:44:52Z</datestamp>
</header>

3. Datacite Dates (ISO 8601) --> TG & DH

<datacite:dates>
  <datacite:date dateType="Created">2012-02-14T21:44:52Z</datacite:date>
  <datacite:date dateType="Issued">2012-02-14T21:44:52Z</datacite:date>
  <datacite:date dateType="Updated">2012-02-14T21:44:52Z</datacite:date>
</datacite:dates>

4. METS (xsd:dateTime) --> TG

<mets>
  <metsHdr CREATEDATE="2015-11-20T14:46:35.896+01:00" />
</mets>

5. MODS (ISO 8601) --> TG

<mods>
  <recordInfo>
    <recordIdentifier source="TWKM">textgrid:255p8</recordIdentifier>
    <recordCreationDate encoding="iso8601">2015-11-20T13:46:35Z</recordCreationDate>
    <recordChangeDate encoding="iso8601">2021-10-19T11:04:41Z</recordChangeDate>
  </recordInfo>
</mods>

6. DC Dates in dc:date --> TG & DH

All the dates come from dc:date fields, and they could always be free text content (at least according to oai_dc spec), and we shall not use UTC here automatically. I Think it is also conform to OAI-PMH spec.

<oai_dc:dc>
  <dc:date>2012-02-10T23:45:00.507+01:00</dc:date>
</oai_dc:dc>

Examples

https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&metadataPrefix=oai_dc&identifier=textgrid:xj2x.0

https://textgridlab.org/1.0/tgcrud-public/rest/textgrid:xj2x.0/metadata

https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&metadataPrefix=oai_datacite&identifier=textgrid:xj2x.0

https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&metadataPrefix=oai_idiom_mets&identifier=textgrid:255p8.0

Expected date metadata formats

https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&metadataPrefix=oai_idiom_mets&identifier=textgrid:255p8.0

expected date format: <recordCreationDate encoding="iso8601">2015-11-20T14:46:35.896+01:00</recordCreationDate>

https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&metadataPrefix=oai_datacite&identifier=textgrid:xj2x.0

expected date format: <datacite:date dateType="Created">2012-02-14T22:44:52.523+01:00</datacite:date>

https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc

expected date format: <dc:date>2012-02-10T23:45:00.507+01:00</dc:date>

changed the description

marked the checklist item Fix REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&from=junk --> must not be 500! as completed

marked the checklist item Fix REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&until=junk --> must not be 500! as completed

marked the checklist item REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&from=2002-02-05&until=2002-02-06T05:35:00Z --> FAIL Failed to extract error code from the response to request: verb=ListRecords&metadataPrefix=oai_dc&from=2002-02-05&until=2002-02-06T05:35:00Z The request has different granularities for the from and until parameters. as completed

marked the checklist item REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListMetadataFormats&identifier=textgrid%3Ax9t0%2E0 --> FAIL Server failed to respond to the ListMetadataFormats request (HTTP header values: status=500 Internal Server Error, age=0, lifetime=3600, is fresh:=1) as completed

marked the checklist item REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&until=2010-06-11T02:32:40Z --> FAIL Failed to extract error code from the response to request: verb=ListRecords&metadataPrefix=oai_dc&until=2010-06-11T02:32:40Z The request specified a date one year before the earliestDatestamp given in the Identify response. There should therefore not be any records with datestamps on or before this date and a noRecordsMatch error code should be returned. as completed

Add noRecordsMatch error if result is empty for ListIdentifiers and ListRecords response.

changed the description

marked the checklist item Checking for correct use of resumptionToken (if used) as completed

Yeah! See:

OAI Icon
OAI-PMH Data Provider Validation and Registration
Showing validation log
Running validation checks (step 2)

baseURL is https://dev.textgridlab.org/1.0/tgoaipmh/oai
Validation only
Request logged from 134.76.162.165
Checking Identify response
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=Identify GET
PASS Administrator email address is 'textgrid-support@gwdg.de'
PASS Correctly reports OAI-PMH protocol version 2.0
PASS baseURL supplied matches the Identify response
PASS Datestamp granularity is 'seconds'
PASS Extracted earliestDatestamp 2011-06-11T02:32:40Z
Checking ListSets response
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListSets GET
PASS responseDate has correct format: 2023-02-02T17:22:20Z
PASS Extracted 26 set names: { project:TGPR-372fe6dc-57f2-6cd4-01b5-2c4bbefcfd3c project:TGPR-edc6d8fe-374d-0082-9202-62976ac3441d project:TGPR-7e530df5-8487-0f21-5e22-625ff1d961a8 ... }, will use setSpec &set=project:TGPR-372fe6dc-57f2-6cd4-01b5-2c4bbefcfd3c in tests
Checking ListIdentifiers response
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListIdentifiers&metadataPrefix=oai_dc&set=project:TGPR-372fe6dc-57f2-6cd4-01b5-2c4bbefcfd3c GET
PASS responseDate has correct format: 2023-02-02T17:22:21Z
PASS Good ListIdentifiers response, extracted id 'textgrid:x9t0.0' for use in future tests.
Checking ListMetadataFormats response
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListMetadataFormats&identifier=textgrid%3Ax9t0%2E0 GET
PASS responseDate has correct format: 2023-02-02T17:22:22Z
PASS Good ListMetadataFormats response, includes oai_dc
PASS Data provider supports oai_dc metadataPrefix
Checking GetRecord response
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&identifier=textgrid%3Ax9t0%2E0&metadataPrefix=oai_dc GET
PASS responseDate has correct format: 2023-02-02T17:22:23Z
PASS Datestamp in GetRecord response (2012-02-10T22:45:01Z) has the correct form for seconds granularity.
PASS Datestamp in GetRecord response (2012-02-10T22:45:01Z) matched the seconds granularity specified in the Identify response.
PASS Expected setSpec was returned in the response
Checking ListRecords response
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&from=2012-02-10T22:45:01Z&until=2012-02-10T22:45:01Z&metadataPrefix=oai_dc GET
PASS responseDate has correct format: 2023-02-02T17:22:23Z
PASS Response is well formed
PASS ListRecords response correctly included record with identifier textgrid:x9t0.0
Checking exception handling (errors)
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?junk GET
PASS Error response correctly includes error code 'badVerb'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=junk GET
PASS Error response correctly includes error code 'badVerb'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&metadataPrefix=oai_dc GET
PASS Error response correctly includes error code 'badArgument'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&identifier=textgrid:x9t0.0 GET
PASS Error response correctly includes error code 'badArgument'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=GetRecord&identifier=invalid"id&metadataPrefix=oai_dc GET
PASS Error response correctly includes error code 'idDoesNotExist'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListIdentifiers&until=junk GET
PASS Error response correctly includes error code 'badArgument'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListIdentifiers&from=junk GET
PASS Error response correctly includes error code 'badArgument'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListIdentifiers&resumptionToken=junk&until=2000-02-05 GET
PASS Error response correctly includes error code 'badResumptionToken'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&from=junk GET
PASS Error response correctly includes error code 'badArgument'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&resumptionToken=junk GET
PASS Error response correctly includes error code 'badResumptionToken'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&resumptionToken=junk&until=1990-01-10 GET
PASS Error response correctly includes error code 'badArgument'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&until=junk GET
PASS Error response correctly includes error code 'badArgument'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords GET
PASS Error response correctly includes error code 'badArgument'
PASS All 13 error requests properly handled
Checking for version 2.0 specific exceptions
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&from=2002-02-05&until=2002-02-06T05:35:00Z GET
PASS Error response correctly includes error code 'badArgument'
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc&until=2010-06-11T02:32:40Z GET
PASS Error response correctly includes error code 'noRecordsMatch'
Checking that HTTP POST requests are handled correctly
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai POST verb:Identify
PASS POST test 1 for Identify was successful
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai POST identifier:textgrid:x9t0.0 metadataPrefix:oai_dc verb:GetRecord
PASS POST test 2 for GetRecord was successful
Checking for correct use of resumptionToken (if used)
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&metadataPrefix=oai_dc GET
NOTE Got resumptionToken DnF1ZXJ5VGhlbkZldGNoBQAAAAAABxI9Fmk2UHpVbDZXVEc2MEFKQUlRWnoxaHcAAAAAAAcSQRZpNlB6VWw2V1RHNjBBSkFJUVp6MWh3AAAAAAAHEj4WaTZQelVsNldURzYwQUpBSVFaejFodwAAAAAABxI_Fmk2UHpVbDZXVEc2MEFKQUlRWnoxaHcAAAAAAAcSQBZpNlB6VWw2V1RHNjBBSkFJUVp6MWh3
REQUEST https://dev.textgridlab.org/1.0/tgoaipmh/oai?verb=ListRecords&resumptionToken=DnF1ZXJ5VGhlbkZldGNoBQAAAAAABxI9Fmk2UHpVbDZXVEc2MEFKQUlRWnoxaHcAAAAAAAcSQRZpNlB6VWw2V1RHNjBBSkFJUVp6MWh3AAAAAAAHEj4WaTZQelVsNldURzYwQUpBSVFaejFodwAAAAAABxI_Fmk2UHpVbDZXVEc2MEFKQUlRWnoxaHcAAAAAAAcSQBZpNlB6VWw2V1RHNjBBSkFJUVp6MWh3 GET
PASS Resumption tokens appear to work
Summary - success

    Uses https URIs (not specified in protocol)
    Total tests passed: 38
    Total warnings: 0
    Total error count: 0
    Validation status: COMPLIANT

Validation process complete
Thu Feb 2 12:22:37 2023

mentioned in commit e27fa016