Metadata and Media

About metadata records

Record metadata details the various fields of information associated with a research product, journal article, data set, or technical report in OSTI's record collection. Bibliographic reference for the E-Link 2 metadata schema are available online in the API documentation. Interactions with the E-Link 2 API metadata are detailed in the /records endpoint in the online documentation reference for retrieval and searching, as well as creating or updating product information for records to which you have access.

Creating new metadata

POST /records

Single product metadata records in the form of JSON fields may be sent via this API. The returned response code will indicate the state of the submission, with any error information detailed as indicated in the online documentation.

Workflow states and requirements

Unless specified otherwise, metadata API creation and updates are submitted to OSTI, and further automated internal processing will immediately take place and move the record revision to completion state, and thus on its way to OSTI.gov and other output products. However, should one wish to submit a draft or incomplete record, for purposes of obtaining an OSTI ID or DOI for example, one may save using the same operation endpoint and indicate the desired state: POST /records/save.

Saved records will be held indefinitely, and require minimal metadata elements (generally only a title and desired product type). Records in this state do not perform additional processing, and will instead remain in draft state until subsequent metadata updates submit that OSTI ID with any additional required fields. Additional information is provided below for automated internal workflow states and transitions.

Handling validation errors

Should a new record or edit to an existing revision be rejected with a 400 response, the returned JSON should contain references to any missing or invalid metadata fields as defined in the online documentation, including the source field name, reason for rejection, and if applicable any additional useful information. Rejected API submissions do not create new revisions, and should be resubmitted with corrections to proceed.

Updating existing metadata

PUT /records/{id}

Once a metadata product has been created at OSTI successfully, the response will contain the unique OSTI ID value for reference. This value must be used in subsequent edits or revisions of the record, and doing so employs the PUT /records/{id} API endpoint. Submitting complete record JSON is required for new updates, as prior field values will be replaced with each subsequent revision. As with the POST endpoint, response codes indicate the success or failure of each operation.

Revisions

Each revision will be assigned a sequential revision number, as returned in the response JSON from the request. If submitted, this new revision will immediately begin the internal processing, and upon completion will replace any existing prior revision metadata. Note: metadata and media operations are independent; users may update one or the other individually throughout a product's lifecycle.

If desired, the E-Link 2 API history revision endpoint is available to retrieve metadata from prior revisions or perform comparisons on historical revisions of a given OSTI ID.

Retrieving record by OSTI ID

GET /records/{id}

The GET API request will retrieve JSON for any record your account has access to for the most recent revision of that record. The metadata fields are documented here.

Removing records

DELETE /records/{id}

If a record is submitted in error, for example a duplicate of prior submissions, or simply no longer valid or desirable, the DELETE /records/{id} endpoint is available. This operation will remove the existing revision of the record, and upon completion its associated record(s) if any on OSTI.gov or other output products. For auditing purposes, a reason for such deletion is required to document the process. Note that prior revisions, while retained in E-Link history and available via the historical version endpoints, will NOT process or transfer to output products. Once that OSTI ID is removed in such a manner, it may not be further updated through normal API means, and may require OSTI staff support intervention to reverse should that be desired.

Workflow states and internal processing

Records submitted to OSTI (as opposed to saved or draft records) will transition internally via processing to completed state automatically. These workflow states are detailed below, but in general should transition immediately from SO state to R in most all cases.

Post submission errors or issues

Should the record revision indicate an internal state failure, such as SF or SX, the record was rejected for post submission validation or release issues. If possible, submit revisions via update to address these (invalid combinations of distribution limitations or other failures) and processing on the new revision will continue. Should these issues persist, OSTI support staff may be contacted to diagnose the situation.

Media Files and Sets

"Media" is a general term referring to full text files, generally PDF or Word documents, associated with metadata products at OSTI. User interactions with E-Link 2 API commonly take place on individual files. A "media set" is a set of one or more files related to each other. As full text files are associated with a metadata product, automated processing will be performed, resulting in one or more additional files, such as text extractions or OCR-produced PDFs, to become part of the media set.

The media API allows interactions with these media sets and files associated with a particular OSTI ID, retrieving status and additional information about these files that make up the set. Each set will contain a MEDIA_ID value, which along with a REVISION number will uniquely identify the media set. Each file making up the set will have its own unique identifier in the form of MEDIA_FILE_ID value. Users may add, replace, or remove media sets using the appropriate media API endpoints using the MEDIA_ID values as needed. The file based API endpoint interact with individual files by MEDIA_FILE_ID to download a specific file from a set.

Supported file types

At present, E-Link media API supports association of file uploads of PDFs, Word or PowerPoint files (or open-source equivalents such as Apache OpenOffice) for most metadata products. The main exception is for dataset products, which are the only product to require off-site URL links to content.

File names and limitations

File upload size is generally not restricted at present, and it should be noted that any file name may be used by the upload endpoints, as the file name will be stored internally as unique references in the OSTI repository.

Adding media files to a product

POST /media/{id}

New media sets may be created for a metadata product by uploading supported files directly to this API endpoint, using the OSTI_ID value to associate with the metadata. Upon successful upload, the file will be added to a media set, returning the unique MEDIA_ID information for the set itself, the file reference, and any other derived files making up the rest of the set (such as extracted text files). Note that upon successful completion and release of the combined metadata and media files to output products, the other media set files derived from the original file upload are used behind the scenes for indexing and metrics purposes, and generally are not shown or linked from OSTI.gov.

Retrieving media set information

GET /media/{id}

The GET endpoint for media sets will list all media set information, including files making up each one for a given OSTI_ID value. Sets and file listings will include relevant information gathered from each file, such as byte size, mime-type, status, and additional administrative field values.

Retrieving individual file content

GET /media/file/{id}

Users may utilize the MEDIA_FILE_ID of any individual file to which they have access, in order to download the content of that file.

Processing and completion of media

During the automated media processing of uploaded files, a number of supplemental files may be derived from the content. Generally, these files, if any, will be added to the media set of the originally uploaded file. These may include text extracted from the file content, or OCR-generated files if deemed necessary to extract such text. These derived files are generally only used internally at OSTI for indexing or metrics purposes, and not made available externally during post processing.

Each file will include processing states, source information, administrative fields (file size, page counts, PDF version numbers, and checksum values) generated during backend processing.

Replacing media sets with new revisions

PUT /media/{id}/{media-id}

As with metadata, media sets may be revised via replacement, in this case of the originally uploaded file. Note that such revisions may be made entirely independently of metadata revisions as needed. Using the PUT command indicated on the API URL, as well as both the OSTI ID and MEDIA ID value of the set to replace, the newly uploaded file will replace the original file with a new set of files. The MEDIA ID value will be retained, and a new REVISION number will be assigned to the replacement media set. As its processing completes successfully, that media reference will replace the set information on OSTI.gov and other applicable output products.

Previous revisions will be retained in history, but only the most recent completed revision of a given media set will be made available upon successful processing to the output products at OSTI.

Removing media sets

DELETE /media/{id}/{media-id}

As with revisions of metadata, sets may be removed entirely by referencing both the desired OSTI ID and MEDIA ID value in this case. This will be retained for historical purposes, but will no longer be made available via output product or via current E-Link media API queries. Additional media sets, if any, will be unaffected by this operation. Note that a descriptive reason query parameter is required to document the purpose for the removal of this media set for administrative and tracking purposes.

DELETE /media/{id}

If ALL media sets are to be removed from a given OSTI ID, this deletion endpoint may be requested. Note this will disassociate ALL media sets from the indicated metadata. As with the single media set version, a reason is required for documenting and metrics purposes.