Harvesting (OSTI-to-Site) is OSTI’s automated option for electronically submitting 241.1 metadata. Harvesting does not require sites to log into E-Link and is a fully automated process.
To participate in this method for submitting metadata, sites should have a bibliographic database supporting their STI document review/approval/release process and must agree to post full text documents, multimedia, or datasets on their own web server. All STI reported via Harvesting must be unclassified with unlimited access and should be fully accessible on the Web to the public and to public search engines.
Upfront programming is required on both a site’s end and at OSTI to prepare for Harvesting. The coordination, the programming, and the testing period take approximately three months. After that, however, the weekly submission of metadata is automatic and without human intervention. OSTI sends out a weekly, automated query with a data range that is basically asking, “What records do you have in your database that are either new or that have been updated since the last run?” The site’s programmed “script,” which resides at a URL on the site’s external server, then, in turn, queries the review/approval database behind the site’s firewall and generates the reply as an XML output file “on the fly.” This is the file that OSTI receives, parses according to a customized mapping, and deposits into E-Link. All of this typically happens during night or very early morning hours.
A confirmation email is automatically returned to the site from OSTI for each “run” and informs the site as to which records harvested correctly and which ones did not. An error message explains the reason for any record that may not have successfully loaded. Site personnel are responsible for correcting records that did not load and allowing them to be “reharvested” in one of the subsequent, normal runs.