Towards FAIR Workflows for Federated Experimental Sciences

Conference · Mon Jul 01 04:00:00 EDT 2024

DOI:https://doi.org/10.1109/CAI59869.2024.00256· OSTI ID:2429798

Saranathan, Gayathri ^[1]; Foltin, Martin ^[1]; Tripathy, Aalap ^[1]; Justine, Annmary ^[2]; Ziatdinov, Maxim ^[3]; Ghosh, Ayana ^[4]; Roccapriore, Kevin ^[4]; Bhattacharya, Suparna ^[1]; Faraboschi, Paolo ^[1]

---

1. Hewlett-Packard
2. Hewlett Packard Enterprise
3. Pacific Northwest National Laboratory (PNNL)
4. ORNL

A de-centralized, peer-to-peer AI metadata framework is demonstrated which can enable end-to-end metadata & lineage tracking for distributed Machine Learning pipelines spanning edge, High Performance Computing, and cloud environments. With a specific example of end-to-end microscopy algorithm and datasets, the proposed method shows how to enable reproducibility, audit trail, provenance of metadata artifacts. The emerging needs of automation in experimental sciences, ML-centric workflows, and FAIR metadata management across federated compute environments is addressed.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 2429798
• Country of Publication: United States
• Language: English