AI Benchmark Democratization and Carpentry

Journal Article · Fri Dec 12 04:00:00 EST 2025 · No journal information

OSTI ID:3008660

von Laszewski, Gregor ^[1]; Brewer, Wesley ^[2]; Thiyagalingam, Jeyan ^[3]; Papay, Juri ^[3]; Foundjem, Armstrong ^[4]; Luszczek, Piotr ^[5]; Emani, Murali ^[6]; Moore, Shirley V. ^[7]; Reddi, Vijay Janapa ^[8]; Sinclair, Matthew D. ^[9]; Lobentanzer, Sebastian ^[10]; Goswami, Sujata ^[11]; Hawks, Benjamin ^[12]; Colombo, Marco ^[13]; Tran, Nhan ^[12]; Kirkpatrick, Christine R. ^[14]; Alsudais, Abdulkareem ^[15]; Barrett, Gregg ^[16]; Li, Tianhao ^[17]; Morehouse, Kirsten ^[18] ; Venkataraman, Shivaram ^[9]; Jain, Rutwik ^[9]; Mathur, Kartik ^[19]; Lu, Victor ^[20]; Singh, Tejinder ^[21]; Mirza, Khojasteh Z. ^[22]; Chen, Kongtao ^[23]; Kunapuli, Sasidhar ^[20]; Farrell, Gavin ^[24]; Umeton, Renato ^[25]; Fox, Geoffrey C. ^[1]

---

1. Virginia U.
2. Oak Ridge
3. Rutherford
4. McGill U.
5. MIT, Lincoln Lab
6. Argonne, PHY
7. Texas U., El Paso
8. Boston U.
9. Wisconsin U., Madison
10. Helmholtz-Zentrum, Berlin
11. LBL, Berkeley
12. Fermilab
13. Illinois U., Urbana
14. UC, San Diego
15. Prince Mohammad U., Al Khobar
16. Johannesburg U.
17. Duke U.
18. Harvard U.
19. Microsoft, Redmond
20. Unlisted
21. Haimson Res., Santa Clara
22. New York U.
23. Google Inc.
24. INFN, Padua
25. Christian Brothers U.

Benchmarks are a cornerstone of modern machine learning, enabling reproducibility, comparison, and scientific progress. However, AI benchmarks are increasingly complex, requiring dynamic, AI-focused workflows. Rapid evolution in model architectures, scale, datasets, and deployment contexts makes evaluation a moving target. Large language models often memorize static benchmarks, causing a gap between benchmark results and real-world performance. Beyond traditional static benchmarks, continuous adaptive benchmarking frameworks are needed to align scientific assessment with deployment risks. This calls for skills and education in AI Benchmark Carpentry. From our experience with MLCommons, educational initiatives, and programs like the DOE's Trillion Parameter Consortium, key barriers include high resource demands, limited access to specialized hardware, lack of benchmark design expertise, and uncertainty in relating results to application domains. Current benchmarks often emphasize peak performance on top-tier hardware, offering limited guidance for diverse, real-world scenarios. Benchmarking must become dynamic, incorporating evolving models, updated data, and heterogeneous platforms while maintaining transparency, reproducibility, and interpretability. Democratization requires both technical innovation and systematic education across levels, building sustained expertise in benchmark design and use. Benchmarks should support application-relevant comparisons, enabling informed, context-sensitive decisions. Dynamic, inclusive benchmarking will ensure evaluation keeps pace with AI evolution and supports responsible, reproducible, and accessible AI deployment. Community efforts can provide a foundation for AI Benchmark Carpentry.

• Research Organization: Helmholtz-Zentrum, Berlin; Boston U.; Virginia U.; Illinois U., Urbana; MIT, Lincoln Lab; Johannesburg U.; Google Inc.; McGill U.; UC, San Diego; Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); Wisconsin U., Madison; Texas U., El Paso; Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Unlisted; Harvard U.; INFN, Padua; Duke U.; Microsoft, Redmond; Rutherford; Argonne National Laboratory (ANL), Argonne, IL (United States); Haimson Res., Santa Clara; New York U.; Christian Brothers U.; Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States); Prince Mohammad U., Al Khobar
• Sponsoring Organization: US Department of Energy
• DOE Contract Number: 89243024CSC000002
• OSTI ID: 3008660
• Report Number(s): FERMILAB-PUB-25-0835-CSAID; oai:inspirehep.net:3093130; arXiv:2512.11588
• Journal Information: No journal information, Journal Name: No journal information
• Country of Publication: United States
• Language: English