Tokenized Data for FORGE Foundation Models

Yin, Junqi; Dash, Sajal; Wang, Feiyi; Shankar, Mallikarjun (Arjun)

doi:10.13139/OLCF/2004951

Title: Tokenized Data for FORGE Foundation Models

Dataset
Other Related Research

Abstract

This dataset comprises a vast corpus of 257 billion tokens, accompanied by the corresponding vocabulary file employed in the pre-training of FORGE foundation models. The primary data source for this corpus is scientific documents derived from diverse origins, and they have been tokenized using the Hugging Face BPE tokenizer. Further details about this research can be found in the publication titled "FORGE: Pre-Training Open Foundation Models for Science" authored by Junqi Yin, Sajal Dash, Feiyi Wang, and Mallikarjun (Arjun) Shankar, presented at SC'23. The data tokenization pipeline and resulting artifacts use CORE data [Ref: Knoth, P., & Zdrahal, Z. (2012). CORE: three access levels to underpin open access. D-Lib Magazine, 18(11/12)]. For use of these data sets for any purpose, please follow the guidelines provided in https://core.ac.uk/terms .

Authors:

Yin, Junqi; Dash, Sajal; Wang, Feiyi; Shankar, Mallikarjun (Arjun)

ORNL-OLCF

Publication Date:: Wed Oct 18 04:00:00 UTC 2023

DOE Contract Number:: AC05-00OR22725

Research Org.:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: Office of Science (SC)

OSTI Identifier:: 2004951

DOI:: https://doi.org/10.13139/OLCF/2004951

Citation Formats


                    Yin, Junqi, Dash, Sajal, Wang, Feiyi, and Shankar, Mallikarjun. Tokenized Data for FORGE Foundation Models.  United States: N. p., 2023. 
        Web.  doi:10.13139/OLCF/2004951.

Copy to clipboard


                    Yin, Junqi, Dash, Sajal, Wang, Feiyi, & Shankar, Mallikarjun. Tokenized Data for FORGE Foundation Models.  United States.  doi:https://doi.org/10.13139/OLCF/2004951

Copy to clipboard


                    Yin, Junqi, Dash, Sajal, Wang, Feiyi, and Shankar, Mallikarjun. 2023.  
"Tokenized Data for FORGE Foundation Models".  United States.  doi:https://doi.org/10.13139/OLCF/2004951.  https://www.osti.gov/servlets/purl/2004951. Pub date:Wed Oct 18 04:00:00 UTC 2023

Copy to clipboard


                    
@article{osti_2004951,

  title        = {Tokenized Data for FORGE Foundation Models},

  author       = {Yin, Junqi and Dash, Sajal and Wang, Feiyi and Shankar, Mallikarjun},

  abstractNote = {This dataset comprises a vast corpus of 257 billion tokens, accompanied by the corresponding vocabulary file employed in the pre-training of FORGE foundation models. The primary data source for this corpus is scientific documents derived from diverse origins, and they have been tokenized using the Hugging Face BPE tokenizer. Further details about this research can be found in the publication titled "FORGE: Pre-Training Open Foundation Models for Science" authored by Junqi Yin, Sajal Dash, Feiyi Wang, and Mallikarjun (Arjun) Shankar, presented at SC'23. The data tokenization pipeline and resulting artifacts use CORE data [Ref: Knoth, P., & Zdrahal, Z. (2012). CORE: three access levels to underpin open access. D-Lib Magazine, 18(11/12)]. For use of these data sets for any purpose, please follow the guidelines provided in https://core.ac.uk/terms .},

  doi          = {10.13139/OLCF/2004951},

  journal      = {},

  number       = ,

  volume       = ,

  place        = {United States},

  year         = {Wed Oct 18 04:00:00 UTC 2023},

  month        = {Wed Oct 18 04:00:00 UTC 2023}

}

Copy to clipboard

Dataset:

View Dataset

DOI: https://doi.org/10.13139/OLCF/2004951

Save / Share:

Export Metadata

Save to My Library

Similar records in DOE Data Explorer and OSTI.GOV collections:

Fallon FORGE 3D Geologic Model

Dataset Blankenship, Doug

An x,y,z scattered data file for the 3D geologic model of the Fallon FORGE site. Model created in Earthvision by Dynamic Graphic Inc. The model was constructed with a grid spacing of 100 m. Geologic surfaces were extrapolated from the input data using a minimum tension gridding algorithm. The data file is tabular data in a text file, with lithology data associated with X,Y,Z grid points. All the relevant information is in the file header (the spatial reference, the projection etc.) In addition all the fields in the data file are identified in the header.
Fallon FORGE: 3D Geologic Model

Dataset Blankenship, Doug ; Siler, Drew

The 3D geologic model for the Fallon for site was constructed in EarthVision software using methods similar to (Moeck et al., 2009, 2010; Faulds et al., 2010b; Jolie et al., 2012, 2015; Hinz et al., 2013a; Siler and Faulds, 2013; Siler et al., 2016a, b) - References are included in archive. The model contains 48 faults (numbered 1-48), and 4 stratigraphic surfaces from oldest to youngest (1) undivided Mesozoic basement, consisting of Mesozoic metasedimentary, metavolcanic, and plutonic units (Mzu); (2) Miocene volcanic and interbedded sedimentary rocks, consisting primarily of basaltic and basaltic andesite lava flows (Tvs); and (3) late Miocenemore »« less
West Flank Coso FORGE: ArcGIS Data for Geologic Model

Dataset Blankenship, Doug

Archive of ArcGIS data from the West Flank FORGE site located in Coso, California. Archive contains the following eight shapefiles: Polygon of the 3D geologic model (WestFlank3DGeologicModelExtent) Polylines of the traces 3D modeled faults (WestFlank3DModeledFaultTraces) Polylines of the fault traces from Duffield and Bacon, 1980 (WestFlankFaultsfromDuffieldandBacon) Polygon of the West Flank FORGE site (WestFlankFORGEsite) Polylines of the traces of the geologic cross-sections (cross-sections in a separate archive in the GDR) (WestFlankGeologicCrossSections) Polylines of the traces of the seismic reflection profiles through and adjacent to the West Flank site (seismic reflection profiles in a separate archive in the GDR) (WestFlankSiesmicReflectionProfiles) Pointsmore »« less
Utah FORGE: Earth Model Mesh Data for Selected Surfaces

Dataset Podgorney, Robert

This submission contains a number of data files with vertices of meshed/interpolated surfaces used in the Phase 2B earth model. Examples include land surface (based on 10-meter DEM), the granitoid-basin fill contact, several faults, and also interpolated temperature isosurfaces for 175 and 225 degrees C. All data are georeferenced to UTM, zone 12N, NAD 83, NAVD 88.
Utah FORGE: 2024 Discrete Fracture Network Model Data

Dataset Finnila, Aleta ; Jones, Clay

The Utah FORGE 2024 Discrete Fracture Network (DFN) Model dataset provides a set of files representing discrete fracture network modeling for the FORGE site near Milford, Utah. The dataset includes four distinct DFN model file sets, each corresponding to different time frames and modeling approaches in 2024. These models characterize both natural and induced fractures in the geothermal reservoir, which consists of crystalline granitic and metamorphic rock approximately 8,000 feet below the ground surface. The dataset includes a reference DFN model from February 2024 that incorporates planar fractures and well trajectories, as well as upscaled permeability, porosity, compressibility, and storagemore »« less

Similar Records