Commit 7eeb5371 authored by mhellka's avatar mhellka
Browse files

Documenting snapshot feature

parent 22e0a8ae
......@@ -299,6 +299,12 @@ Transaction management is expensive. Some transaction information must survive e
.Transaction Timeout
Explicit transactions expire after some time of inactivity. They never expire while a HTTP call is still in progress, and will extend their lifetime automatically after each HTTP call. You won't have to worry about that in most cases. If you need a transaction to survive more than a couple of seconds of inactivity (e.g. while waiting for user input), you can specify a higher `timeout` when creating a transaction, or issue cheap HTTP calls from time to time to prevent transactions from dying. Expired transactions are rolled back automatically.
[[snapshots]]
== Snapshots
CDSTAR allows you to create named snapshots from archives. Snapshots are umodifiable point-in-time copies of the archive payload (files and metadata) and can be used to implement versioning, history or publication workflows. Most API endpoints that read from an archives also work for snapshots, simply by replacing the `{archive}` ID with an `{archive}@{snapshot}` reference in the request URL. For example, `GET /v3/ab587f42c257@v1/data.csv` would fetch a file from the `v1` snapshot instead of the current archive state. Any state that is not part of a snapshot (e.g. archive owner or ACL info) is complemented from the archive itself. Details are explained in the endpoint documentations.
[[endpoints]]
= API Endpoints
......
This diff is collapsed.
[[storageProfiles]]
= Storage Profiles
CDSTAR supports and integrates third party LTS (long term storage, e.g. tape libraries) via storage profiles. From the users perspective, a storage profile defines where and how data should be stored. By assigning a storage profile to a CDSTAR archive, the user can control data migration to and from LTS in a coherent, safe and predictable way. The actual data migration happens in the background and is fully managed by CDSTAR.
CDSTAR supports and integrates third party long time storage systems (LTS, e.g. tape libraries) via storage profiles. From the users perspective, a storage profile defines where and how data should be stored. By assigning a storage profile to a CDSTAR archive, the user can control data migration to and from LTS in a coherent, safe and predictable way. The actual data migration happens in the background and is fully managed by CDSTAR.
== Profile mode: HOT vs. COLD
Storage profiles can be configured as either "hot" or "cold", which changes the way CDSTAR handles its local data.
Storage profiles can be either "hot" or "cold", which changes the way CDSTAR handles its local data.
Hot profiles causes CDSTAR to copy the archive content to an external LTS, but keep all data available in CDSTAR as well. While the profile is in effect, only administrative metadata (owner, ACLs, storage profile, ...) can be modified. The actual content (files and metadata) is write-protected to prevent stale LTS copies.
Hot profiles causes CDSTAR to copy the archive content to external storage, but keep all data available in CDSTAR as well. While the profile is in effect, only administrative metadata (owner, ACLs, storage profile, ...) can be modified. The actual content (files and metadata) is write-protected to prevent stale LTS copies.
Cold profiles, on the other hand, allow CDSTAR to re-claim disk space by deleting archive files from disk after a copy was stored in LTS. Meta-data is still kept available, but file content can no longer be accessed through CDSTAR. The profile needs to be changed to `default` or a hot profile to make file content available again.
Cold profiles, on the other hand, allow CDSTAR to re-claim disk space by deleting archive files from disk after a copy was stored externally. Metadata is still kept available, but file content can no longer be accessed through CDSTAR. The profile needs to be changed to `default` or a hot profile to make file content available again.
Hot profiles are meant to increase long term availability or data integrity guarantees by storing important data in a second location. Cold profiles are mostly used to store large amounts of rarely accessed data in a more cost-effective way (e.g. on tape), while keeping meta-data search- and discoverable.
......@@ -44,7 +45,7 @@ LTS handlers are referenced by name, so special care must be taken when removing
Moving data out of the CDSTAR system, especially with *cold* profiles, bears some risks that should be well understood before enabling the LTS feature. Please read this chapter carefully.
After a successful migration to an LTS target, CDSTAR stores the LTS name and a unique location identifier (generated by the LTS) into non-public archive properties. These are used to recover missing files in case of a future profile change. Cold profiles allow CDSTAR to remove local copies of archived files after successfully copying these files to LTS. If the LTS goes away, for whatever reason, then CDSTAR has no way to recover missing files and the archive is stuck in cold state. File content will be unavailable and profile changes will fail.
After a successful migration to an LTS target, CDSTAR stores the LTS name and a unique location identifier (generated by the LTS) into non-public archive properties. These are used to recover missing files in case of a future profile change. Cold profiles allow CDSTAR to remove local copies of archived files after successfully copying these files to LTS. If the LTS goes away, for whatever reason, then CDSTAR has no way to recover missing files and the archive is stuck in cold state. File content will be unavailable and data migration after profile changes will fail.
* Do not remove or rename an LTS target as long as there are archives still referencing it.
* When updating LTS Plugins or changing configuration, ensure that existing location identifiers remain valid.
......@@ -58,7 +59,7 @@ storage (e.g. tape).
The exporter will create a BagIt package in a temporary folder, then rename it
to `[name].bagit` with a unique name. A worker process may check for these
folders and copy or move these to LTS.
folders and copy or move data to LTS.
The importer will create a file named `[name].want` and start the import as
soon as the `[name].bagit` folder can be found. A worker process should check
......
......@@ -5,11 +5,13 @@
{{dt.help|default("_No description_")}}
.Field list for <<{{id}}>>
[cols="0h,0v,1"]
[cols="0h,0v,1a"]
|======
| Field | Type | Description
{% for name, q in dt.fields.items() %}
| {{name}} | {{q.type|default('string')}} | {{q.help|default("_No description_")}}
| {{name}}
| {{q.type|default('string')}}
| {{q.help|default("_No description_")}}
{% endfor -%}
|======
......
......@@ -10,22 +10,24 @@
[[{{routename}}]]
### {{route.title|default(routename)}}
[cols="0,1,0"]
|======
|`*{{route.method}}*`
{set:cellbgcolor:lightgrey}
|`*/v3{{route.path|sub("\\{([^}]+)\\}","[red]#{\\1}#")}}*`
{set:cellbgcolor:!}
|`HTTP/1.1`
|======
{{route.help|default("#_No description_#")}}
++++
{% for path in route.path|list %}
<div style="margin: .5em 0; border: 1px solid lightgrey; line-height: 2em">
<b style="padding: 0 .5em; margin: 1px; background: lightgrey; height: 100%; display: inline-block">{{route.method}}</b>
<code>/v3{{path|sub("\\{([^}]+)\\}","<span class=\"red\">{\\1}</span>")}}</code>
<b style="float: right; margin: 0 .5em">HTTP/1.1</b>
</div>
{% endfor %}
++++
{% if route.beta %}WARNING: This endpoint is marked as *unstable* and is subject to change.{% endif %}
{% if route.consumes %}NOTE: This endpoint consumes: `+{{route.consumes|list|join('+`, `+')}}+`{% endif %}
{% if route.produces %}NOTE: This endpoint produces: `+{{route.produces|list|join('+`, `+')}}+`{% endif %}
{{route.help|default("#_No description_#")}}
{% macro paramRow(name, param) -%}
| {{name}}
| {{param.type|default('string')}}
......
......@@ -8,7 +8,9 @@
4+h|{{group.title|default(groupName)}}
{% for routename, route in group.routes.items() -%}
| <<{{routename}},{{route.title|default(routename)}}>> | `*{{route.method}}*` | `/v3{{route.path}}` | {{route.consumes|list|join('\n')}}
{% for path in [route.path|list|first] -%}
| <<{{routename}},{{route.title|default(routename)}}>> | `*{{route.method}}*` | `+/v3{{path}}+` | {{route.consumes|list|join('\n')}}
{% endfor -%}
{% endfor -%}
{% endfor -%}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment