Towards FAIR Workflows for Federated Experimental Sciences

Conference · Tue Jul 30 04:00:00 EDT 2024

DOI:https://doi.org/10.1109/CAI59869.2024.00256· OSTI ID:2439189

Saranathan, Gayathri ^[1]; Foltin, Martin ^[1]; Tripathy, Aalap ^[1]; Justine, Annmary ^[1]; Ziatdinov, Maxim A. ^[2]; Ghosh, Ayana ^[3]; Roccapriore, Kevin ^[3]; Bhattacharya, Supama ^[1]; Faraboschi, Paolo ^[1]

---

1. Hewlett Packard Enterprise
2. BATTELLE (PACIFIC NW LAB)
3. Oak Ridge National Laboratory

A de-centralized, peer-to-peer AI metadata framework is demonstrated which can enable end-to-end metadata & lineage tracking for distributed Machine Learning pipelines spanning edge, High Performance Computing, and cloud environments. With a specific example of end-to-end microscopy algorithm and datasets, the proposed method shows how to enable reproducibility, audit trail, provenance of metadata artifacts. The emerging needs of automation in experimental sciences, ML-centric workflows, and FAIR metadata management across federated compute environments is addressed.

• Research Organization: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-76RL01830
• OSTI ID: 2439189
• Report Number(s): PNNL-SA-202274
• Country of Publication: United States
• Language: English

Related Subjects

machine learning