GLIBRARY 1.0¶
REST API v 1.0¶
gLibrary provides two endopoints depending on the type of operation:
- Data Management (DM): operations that affect the underlying storage backend (Grid DPM servers or Swift Object Storage)
- Metadata Management (MM): operations that affect the underlying Metadata and DB Services (AMGA and PostgreSQL currently)
DM endpoint:
https://glibrary.ct.infn.it/dm/
MM endpoint:
https://glibrary.ct.infn.it:4000/
Internally they have been deployed as two separate services, using two different technology, respectively, Flask, a Python microframework, and Node.js.
Authentication¶
At the moment, APIs can be accessed directly with a X.509 certificates from any HTTPs client (eg: command line scripts with _curl_ or _wget_, Web or Desktop HTTP clients) or indirectly through a Science Gateway.
For X.509 authentication, currently INFN released Robot certificated are allowed, and a given set of users that requested access. If you need to have access to our APIs with a personal X.509 certificate, please contact us at sg-license@ct.infn.it.
To access gLibrary API from web applications or Science Gateway portels, we have implemented server-to-server authentication. We mantein a white list of server IP addresses that are allowed to access glibrary endpoints. To avoid CORS problems, your server should implement a proxy mechanism that blindly redirect API requests to our endpoints. Again, contact us at sg-license@ct.infn.it, to request access for your server.
Data Management¶
File Download¶
Download a file from a Storage Element to the local machine
GET https://glibrary.ct.infn.it/dm/<vo>/<se>/<path:path> HTTP/1.1
Parameters
Parameter | Description |
---|---|
vo | Virtual Organisation (eg: vo.aginfra.eu) |
se | Storage Element (eg: prod-se-03.ct.infn.it) |
path | absolute path of the file to be downloaded |
Response
Short lived URL to download the file over http. Example:
$ curl -L -O https://glibrary.ct.infn.it/dm/vo.aginfra.eu/prod-se-03.ct.infn.it/dpm/ct.infn.it/home/vo.dch-rp.eu/test/IMG_0027.PNG
File Upload¶
Upload a local file to a given Storage Element of a Virtual Organization. The upload of a file requires two steps: the first one prepares the destination storage to receive the upload and returns a short-lived URL to be used by a second API request for the actual upload (second step).
GET https://glibrary.ct.infn.it/dm/put/<vo>/<filename>/<se>/<path:path> HTTP/1.1
PUT http://<storage_host>/<storage_path> HTTP/1.1
Parameters
Parameter | Description |
---|---|
vo | Virtual Organisation (vo.aginfra.eu) |
se | Storage Element where upload the file (eg: prod-se-03.ct.infn.it) |
filename | Name that will be used to store the file on the storage. Can be different by the original name |
path | Absolute path where the file will be located on the storage |
Response
Redirect short-live URL, authorized only for the requesting IP, where the actual file should be uploaded Status 307 (temporary redirect)
Example:
step-1:
$ curl http://glibrary.ct.infn.it/dm/put/vo.aginfra.eu/file-test.txt/prod-se-03.ct.infn.it/dpm/ct.infn.it/home/vo.dch-rp.eu/test/
Output:
{
"redirect": "http://prod-se-03.ct.infn.it/storage/vo.aginfra.eu/2014-04-30/file-test.txt.53441.0?sfn=%2Fdpm%2Fct.infn.it%2Fhome%2Fvo.aginfra.eu%2Ftest%2F%2Ffile-test.txt&dpmtoken=48042a60-005c-4bf1-9eea-58b6a971eb52&token=GgxCE%2FmbfYJv09H0QRFrSInghK0%3D%401398870909%401",
"status": 307
}
Example
step-2:
$ curl -T file-test.txt -X PUT "http://prod-se-03.ct.infn.it/storage/vo.aginfra.eu/2014-04-30/file-test.txt.53441.0?sfn=%2Fdpm%2Fct.infn.it%2Fhome%2Fvo.aginfra.eu%2Ftest%2F%2Frfile-test.txt&dpmtoken=48042a60-005c-4bf1-9eea-58b6a971eb52&token=GgxCE%2FmbfYJv09H0QRFrSInghK0%3D%401398870909%401"
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>201 Created</title>
</head><body>
<h1>Created</h1>
<p>Resource /storage/vo.aginfra.eu/2014-04-30/file-test.txt.53441.0 has been created.</p>
<hr />
<address>Apache/2.2.15 (Scientific Linux) Server at prod-se-03.ct.infn.it Port 80</address>
</body></html>
File Download (Swift Object Storage)¶
GET https://glibrary.ct.infn.it/api/dm/cloud/<host>/<path> HTTP/1.1
Parameters
Parameter | Description |
---|---|
host |
Swift Object-storage front-end (or proxy) |
path |
Object full path, following the Swift format: /v1/<account>/<container>/<object> |
Example:
$ curl https://glibrary.ct.infn.it/api/dm/cloud/stack-server-01.ct.infn.it/v1/AUTH_51b2f4e508144fa5b0c28f02b1618bfd/gridcore/ananas.jpg
Returns:
{
"url": "http://stack-server-01.ct.infn.it:8080/v1/AUTH_51b2f4e508144fa5b0c28f02b1618bfd/gridcore/ananas.jpg?temp_url_sig=c127c c2bda34e4ca45afabe42ed606200daab6b&temp_url_expires=1426760853"
}
The returned URL, that allows the direct download of the requested file from the containing server, has an expiration of 10 seconds.
File Upload (Swift Object Storage)¶
PUT https://glibrary.ct.infn.it/api/dm/cloud/<host>/<path> HTTP/1.1
Parameters
Parameter | Description |
---|---|
host |
Swift Object-storage front-end (or proxy) |
path |
Object full path, following the Swift format: /v1/<account>/<container>/<object> |
Example:
$ curl -X PUT https://glibrary.ct.infn.it/api/dm/cloud/stack-server-01.ct.infn.it/v1/AUTH_51b2f4e508144fa5b0c28f02b1618bfd/gridcore/tracciati/prova.xml
Returns:
{
"url": "http://stack-server-01.ct.infn.it:8080/v1/AUTH_51b2f4e508144fa5b0c28f02b1618bfd/gridcore/tracciati/prova.xml?temp_url_sig=8083f489945585db345b7c0ad015290f8a86b4a0&temp_url_expires=1426761014"
}
Again it returns a temporary URL valid 10 seconds to complete the upload directly to the storage with:
$ curl -X PUT -T prova.xml "http://stack-server-01.ct.infn.it:8080/v1/AUTH_51b2f4e508144fa5b0c28f02b1618bfd/gridcore/tracciati/prova.xml?temp_url_sig=8083f489945585db345b7c0ad015290f8a86b4a0&temp_url_expires=1426761014
File system namespace management¶
These APIs expose a subset of WebDAV functionalities over eInfrastructure Storage Elements. They allow operations such as directory creation (MKCOL
), file metadata retrieval (PROPFIND
), file renaming (MOVE
), file deleting (DELETE
).
PROPFIND https://glibrary.ct.infn.it/dm/dav/<vo>/<se>/<path:path>
DELETE https://glibrary.ct.infn.it/dm/dav/<vo>/<se>/<path:path>
MOVE https://glibrary.ct.infn.it/dm/dav/<vo>/<se>/<path:path>
MKCOL https://glibrary.ct.infn.it/dm/dav/<vo>/<se>/<path:path>
Parameters
Parameter | Description |
---|---|
vo |
Virtual Organisation (vo.aginfra.eu) |
se |
Storage Element where the file is located (eg: prod-se-03.ct.infn.it) |
path |
Absolute path where the file is located on the storage |
Directory Creation¶
Example:
$ curl -X MKCOL http://glibrary.ct.infn.it/dm/dav/vo.aginfra.eu/prod-se-03.ct.infn.it/dpm/ct.infn.it/home/vo.aginfra.eu/test2/
Output:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>201 Created</title>
</head><body>
<h1>Created</h1>
<p>Collection /dpm/ct.infn.it/home/vo.aginfra.eu/test2/ has been created.</p>
<hr />
<address>Apache/2.2.15 (Scientific Linux) Server at prod-se-03.ct.infn.it Port 443</address>
</body></html>
File metadata retrieval¶
Example:
$ curl -X PROPFIND -H "Depth:1" http://glibrary.ct.infn.it/dm/dav/vo.aginfra.eu/prod-se-03.ct.infn.it/dpm/ct.infn.it/home/vo.aginfra.eu/test2/
Output
<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:">
<D:response xmlns:lcgdm="LCGDM:" xmlns:lp3="LCGDM:" xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/dm/dav/vo.ag-infra.eu/prod-se-03.ct.infn.it/dpm/ct.infn.it/home/vo.aginfra.eu/test2/</D:href>
<D:propstat>
<D:prop>
<lcgdm:type>0</lcgdm:type><lp1:resourcetype><D:collection/></lp1:resourcetype>
<lp1:creationdate>2014-04-30T15:25:31Z</lp1:creationdate><lp1:getlastmodified>Wed, 30 Apr 2014 15:25:31 GMT</ lp1:getlastmodified><lp3:lastaccessed>Wed, 30 Apr 2014 15:25:31 GMT</lp3:lastaccessed><lp1:getetag>ca36-536115eb< /lp1:getetag><lp1:getcontentlength>0</lp1:getcontentlength><lp1:displayname>test2</lp1:displayname>< lp1:iscollection>1</lp1:iscollection><lp3:guid></lp3:guid><lp3:mode>040755</lp3:mode><lp3:sumtype></lp3:sumtype>< lp3:sumvalue></lp3:sumvalue><lp3:fileid>51766</lp3:fileid><lp3:status>-</lp3:status><lp3:xattr>{"type": 0}</ lp3:xattr><lp1:owner>5</lp1:owner><lp1:group>102</lp1:group></D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
</D:multistatus>
File deletion¶
$ curl -X DELETE http://glibrary.ct.infn.it/dm/dav/vo.dch-rp.eu/prod-se-03.ct.infn.it/dpm/ct.infn.it/home/vo.aginfra.eu/test/file-test.txt
Repository Management¶
List of the available repositories¶
Returns the list of the available repositories
GET https://glibrary.ct.infn.it:3000/repositories HTTP/1.1
Example
$ curl https://glibrary.ct.infn.it:3000/repositories
Output:
{
"result": [
"/gLibTest",
"/deroberto",
"/gLibIndex",
"/tmp",
"/deroberto2",
"/medrepo",
"/ESArep",
"/EELA",
"/EGEE",
"/testRepo",
"/ChinaRep",
"/templaterepo",
"/myTestRepo",
"/ICCU",
"/aginfra"
"..."
]
}
Repository Creation¶
Description Create a new repository
POST https://glibrary.ct.infn.it:3000/repositories/<repo> HTTP/1.1
Returns:
{
"success": "true"
}
Parameters
Parameter | Description |
---|---|
repo | Repository name |
Example:
$ curl –X POST http://glibrary.ct.infn.it:3000/repositories/agInfra
Retrieve repository information¶
Provides the list of types (model) of a given repository. A type describes the kind of digital objects using a schema (set of attributes).
GET https://glibrary.ct.infn.it:3000/repositories/<repo> HTTP/1.1
Returns an array of all the types available in the given repository. Each object rapresents a supported type, with some properties:
Parameters
Parameter | Description |
---|---|
repo | Repository name |
Response
Property | Description |
---|---|
TypeName |
a label that describes the type (to be shown in the gLibrary browser Interface) |
Path |
the absolute path of the entries in the underlying metadata server (AMGA) |
VisibleAttrs |
the set of attributes visible through the gLibrary browser (both Web and mobile) |
FilterAttrs |
a set of attributes that can be used to filter the entries (digital objects) of the given type |
ColumnWidth |
size of each column (attribute) in the gLibrary browser |
ParentID |
types can be organized in a hierarchical structure (tree), and a type can have a subtype. The root type has id 0 |
Type |
a unique identifier assigned to a given type to refer to it in other API call |
Example:
$ curl http://glibrary.ct.infn.it:3000/repositories/agInfra
Output
{
"results": [
{
"TypeName": "Soil Maps",
"Path": "/agInfra/Entries/SoilMaps",
"VisibleAttrs": "Thumb title creator subject description type format language date",
"FilterAttrs": "creator subject publisher contributor type format language rights",
"ColumnWidth": "80 120 60 60 230 100 100 80 80",
"ParentID": "0",
"id": "1",
"Type": "SoilMaps"
}
]
}
Add a type to a repository¶
Add a new Type to a given repository.
POST https://glibrary.ct.infn.it:3000/<repo> HTTP/1.1
URI Parameters
Parameter | Description |
---|---|
repo | The name of the repository to which we are adding the type to |
Body Parameters
Parameter | Description |
---|---|
__Type |
the unique identifier (string) to be assigned to the type |
__VisibleAttrs |
the set of attributes visible through the gLibrary browser (both Web and mobile) |
__ColumnWidth |
size of each column (attribute) in the gLibrary browser |
__ParentID |
types can be organized in a hierarchical structure (tree), and a type can have a subtype. The root type has id 0 |
{AttributeName}* |
a set of attributes with their data type (allowed data types are varchar, int, float, timestamp, boolean) |
Example:
$ curl -X POST -d "__Type=Documents&__VisibleAttrs='Topic,Meeting,FileFormat,Size,Creator,Version'&__FilterAttr='Topic,FileFormat,Creator&Topic=varchar&Version=int&FileFormat=varchar(3)'&Creator=string" http://glibrary.ct.infn.it:3000/aginfra
Retrieve Type information¶
Returns the information about a given type of a given repository.
GET https://glibrary.ct.infn.it:3000/<repo>/<type> HTTP/1.1
Returns A JSON object with the information of a given type with a list of all its attributes and given data type
Example:
$ curl http://glibrary.ct.infn.it:3000/aginfra/SoilMaps
Output:
{
TypeName: "Soil Maps",
Path: "/aginfra/Entries/SoilMaps",
VisibleAttrs: "Thumb title creator subject description type format language date",
FilterAttrs: "creator subject publisher contributor type format language rights",
ColumnWidth: "80 120 60 60 230 100 100 80 80",
ParentID: "0",
id: "1",
Type: "SoilMaps",
FileName: "varchar(255)",
SubmissionDate: "timestamp",
Description: "varchar",
Keywords: "varchar",
LastModificationDate: "timestamp",
Size: "int",
FileType: "varchar(10)",
Thumb: "int",
ThumbURL: "varchar",
TypeID: "int",
title: "varchar",
creator: "varchar",
subject: "varchar",
description: "varchar",
publisher: "varchar",
contributor: "varchar",
type: "varchar",
format: "varchar",
identifier: "varchar",
source: "varchar",
language: "varchar",
date: "varchar",
relation: "varchar",
coverage: "varchar",
rights: "varchar"
}
List of all the entries of a given type¶
List all the entries and its metadata of a given Type in a repository (default limit to 100)
GET https://glibrary.ct.infn.it:3000/<repo>/<type>/entries HTTP/1.1
Parameters
Parameter | Description |
---|---|
repo | The name of the repository |
type | The name of type |
Example:
$ curl http://glibrary.ct.infn.it:3000/aginfra/SoilMaps/entries
Output:
{
results:
[
{
id: "51",
FileName: "",
SubmissionDate: "2012-11-09 07:02:00",
Description: "",
Keywords: "",
LastModificationDate: "",
Size: "",
FileType: "",
Thumb: "1",
ThumbURL: "",
TypeID: "1",
title: "CNCP 3.0 software",
creator: "Giovanni Trapatoni",
subject: "software|soil management",
description: "CNCP 3.0 database with italian manual. CNCP is the program used for the storing, managing and correlating soil observations.",
publisher: "E doardo A. C. Costantini",
contribu tor: "Giovanni L'Abate",
type: "application",
format: "EXE",
identifier: "http://abp.entecra.it/soilmaps/download/sw-CNCP30.exe",
source: "http://abp.entecra.it/soilmaps/en/downloads.html",
language: "it",
date: "2011-08-03",
relation: "",
coverage: "world",
rights: "All rights reserved"
},
{
id: "53",
FileName: "",
SubmissionDate: "2012-11-09 09:37:00",
Description: "",
Keywords: "",
LastModificationDate: "",
Size: "",
FileType: "",
Thumb: "1",
ThumbURL: "",
TypeID: "1",
title: "Benchmark at Beccanello dome, Sarteano (SI)",
creator: "Edoardo A. C. Costantini",
subject: "soil analysis|soil map|pedology",
description: "Form: Soil profile, Survey: Costanza Calzolari, Reporter: Calzolari",
publisher: "CRA-ABP Research centre for agrobiology and pedology, Florence, Italy",
contributor: "Centro Nazionale di Cartografia Pedologica",
type: "Soil map",
format: "KML",
identifier: "https://maps.google.com/maps/ms?ie=UTF8&hl=it&msa=0&msid=115138938741119011323.000479a7eafdbdff453bf&z=6",
source: "https://maps.google.com/maps/ms?ie=UTF8&hl=it&authuser=0&msa=0&output=kml&msid=215926279991638867427. 00479a7eafdbdff453bf",
language: "en",
date: "2010-09-22",
relation: "",
coverage: "Italy",
rights: "info@soilmaps.it"
}
]
}
Retrieve the metadata of a given entry¶
Retrieve all the metadata (and replica info) the a given entry
GET https://glibrary.ct.infn.it:3000/<repo>/<type>/id HTTP/1.1
Returns The metadata of the given entry and the replicas of the associated digital objects
Parameters
Parameter | Description |
---|---|
repo | The name of the repository |
type | The name of type |
id | The id of the entry to inspect |
Example:
$ curl http://glibrary.ct.infn.it:3000/aginfra/SoilMaps/56
Output:
{
results: {
id: "56",
FileName: "",
SubmissionDate: "2012-11-09 10:03:00",
Description: "",
Keywords: "",
LastModificationDate: "",
Size: "",
FileType: "",
Thumb: "1",
ThumbURL: "",
TypeID: "1",
title: "ITALIAN SOIL INFORMATION SYSTEM 1.1 (ISIS)",
creator: "Costantini E.A.C.|L'Abate G.",
subject: "soil maps|pedology",
description: "The WebGIS and Cloud Computing enabled ISIS service is running for online Italian soil data consultation. ISIS is made up of a hierarchy of geo-databases which include soil regions and aim at correlating the soils of Italy with those of other European countries with respect to soil typological units (STUs), at national level, and soil sub-systems, at regional level",
publisher: "Consiglio per la Ricerca e la sperimentazione in Agricoltura (CRA)-(ABP)|Research centre for agrobiology and pedology, Florence, Italy",
contributor: "INFN, Division of Catania|agINFRA Science Gateway|",
type: "",
format: "CSW",
identifier: "http://aginfra-sg.ct.infn.it/isis",
source: "http://aginfra-sg.ct.infn.it/webgis/cncp/public/",
language: "en",
date: "2012-04-01",
relation: "Barbetti R. Fantappi M., L Abate G., Magini S., Costantini E.A.C. (2010). The ISIS software for soil correlation and typology creation at different geographic scales. In: Book of Extended Abstracts of the 4th Global Workshop on Digital Soil Mapping, CRA, Rome, 6pp",
coverage: "Italy",
rights: "giovanni.labate@entecra.it",
"Replicas": [
{
"url": "https://unipa-se-01.pa.pi2s2.it/dpm/pa.pi2s2.it/home/vo.aginfra.eu/aginfra/maps_example.tif",
"enabled": "1"
},
{
"url": "https://inaf-se-01.ct.pi2s2.it/dpm/ct.pi2s2.it/home/vo.aginfra.eu/aginfra/maps_example.tif",
"enabled": "1"
},
{
"url": "https://unict-dmi-se-01.ct.pi2s2.it/dpm/ct.pi2s2.it/home/vo.aginfra.eu/aginfra/maps_example.tif",
"enabled": "0"
}
]
}
}
Add a new entry¶
Add a new entry with its metadata of a given type
POST https://glibrary.ct.infn.it:3000/<repo>/<type>/ HTTP/1.1
Parameters
Parameter | Description |
---|---|
repo |
The name of the repository |
type |
The if of the type |
Body Parameters
Example:
$ curl -X POST -d “__Replicas=https://prod-se-03.ct.infn.it/dpm/ct.infn.it/home/vo.aginfra.eu/test/maptest.jpg&FileName=maptest.jpg&creator=Bruno&title=Italian%20maps%20example” http://glibrary.ct.infn.it:3000/aginfra/SoilMaps
Delete an entry¶
Delete an entry from a repository of the given type
DELETE https://glibrary.ct.infn.it:3000/<repo>/<type>/id HTTP/1.1
Parameters
Parameter | Description |
---|---|
repo |
The name of the repository |
type |
The name of type |
id |
Id of the entry to be deleted |