Autogenerating a Domain-Specific Question-Answering Data Set from a Thermoelectric Materials Database to Enable High-Performing BERT Models

Sierepeklis, Odysseas; Cole, Jacqueline M.

doi:10.1021/acs.jcim.5c00840

Autogenerating a Domain-Specific Question-Answering Data Set from a Thermoelectric Materials Database to Enable High-Performing BERT Models

Journal Article · Thu Aug 07 00:00:00 EDT 2025 · Journal of Chemical Information and Modeling

DOI:https://doi.org/10.1021/acs.jcim.5c00840· OSTI ID:2586792

Sierepeklis, Odysseas ^[1]; ^[2]

Univ. of Cambridge (United Kingdom). Cavendish Lab.
Univ. of Cambridge (United Kingdom). Cavendish Lab.; Science and Technology Facilities Council (STFC), Oxford (United Kingdom). Rutherford Appleton Lab. (RAL)

We present a method for autogenerating a large domain-specific question-answering (QA) dataset from a thermoelectric materials database. We show that a small language model, BERT, once fine-tuned on this automatically generated dataset of 99,757 QA pairs about thermoelectric materials, affords better performance in the field of thermoelectric materials compared to a BERT model fine-tuned on the generic English-language QA data set, SQuAD-v2. We further show that mixing the two data sets (ours and SQuAD-v2), which have significantly different syntactic and semantic scopes, allows the BERT model to achieve even better performance. The best-performing BERT model fine-tuned on the mixed data set outperforms the models fine-tuned on the other two data sets by scoring an exact match of 67.93% and an F1 score of 72.29% when evaluated on our test data set. This has important implications as it demonstrates the ability to realize high-performing small language models, with modest computational resources, empowered by domain-specific materials data sets which can be generated according to our method.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: Engineering and Physical Sciences Research Council; Science and Technology Facilities Council; USDOE Office of Science (SC), Basic Energy Sciences (BES)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 2586792

Journal Information:: Journal of Chemical Information and Modeling, Journal Name: Journal of Chemical Information and Modeling Journal Issue: 16 Vol. 65; ISSN 1549-9596; ISSN 1549-960X

Publisher:: American Chemical Society (ACS)Copyright Statement

Country of Publication:: United States

Language:: English

References (42)

Temperature dependent solubility of Yb in Yb–CoSb3 skutterudite and its effect on preparation, optimization and lifetime of thermoelectrics Tang, Yinglu; Chen, Sinn-wen; Snyder, G. Jeffrey Journal of Materiomics, Vol. 1, Issue 1 https://doi.org/10.1016/j.jmat.2015.03.008	journal	March 2015
Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science Trewartha, Amalie; Walker, Nicholas; Huo, Haoyan Patterns, Vol. 3, Issue 4 https://doi.org/10.1016/j.patter.2022.100488	journal	April 2022
Synergistically optimizing electrical and thermal transport properties of n -type PbSe Qian, Xin; Xiao, Yu; Chang, Cheng Progress in Natural Science: Materials International, Vol. 28, Issue 3 https://doi.org/10.1016/j.pnsc.2018.05.002	journal	June 2018
ChemDataExtractor 2.0: Autopopulated Ontologies for Materials Science Mavračić, Juraj; Court, Callum J.; Isazawa, Taketomo Journal of Chemical Information and Modeling, Vol. 61, Issue 9 https://doi.org/10.1021/acs.jcim.1c00446	journal	September 2021
BatteryBERT: A Pretrained Language Model for Battery Database Enhancement Huang, Shu; Cole, Jacqueline M. Journal of Chemical Information and Modeling https://doi.org/10.1021/acs.jcim.2c00035	journal	May 2022
OpticalBERT and OpticalTable-SQA: Text- and Table-Based Language Models for the Optical-Materials Domain Zhao, Jiuyang; Huang, Shu; Cole, Jacqueline M. Journal of Chemical Information and Modeling, Vol. 63, Issue 7 https://doi.org/10.1021/acs.jcim.2c01259	journal	March 2023
How Beneficial Is Pretraining on a Narrow Domain-Specific Corpus for Information Extraction about Photocatalytic Water Splitting? Isazawa, Taketomo; Cole, Jacqueline M. Journal of Chemical Information and Modeling, Vol. 64, Issue 8 https://doi.org/10.1021/acs.jcim.4c00063	journal	March 2024
ChemDataExtractor: A Toolkit for Automated Extraction of Chemical Information from the Scientific Literature Swain, Matthew C.; Cole, Jacqueline M. Journal of Chemical Information and Modeling, Vol. 56, Issue 10 https://doi.org/10.1021/acs.jcim.6b00207	journal	October 2016
Structured information extraction from scientific text with large language models Dagdelen, John; Dunn, Alexander; Lee, Sanghoon Nature Communications, Vol. 15, Issue 1 https://doi.org/10.1038/s41467-024-45563-x	journal	February 2024
Extracting accurate materials data from research papers with conversational language models and prompt engineering Polak, Maciej P.; Morgan, Dane Nature Communications, Vol. 15, Issue 1 https://doi.org/10.1038/s41467-024-45914-8	journal	February 2024
MatSciBERT: A materials domain language model for text mining and information extraction Gupta, Tanishq; Zaki, Mohd; Krishnan, N. M. Anoop npj Computational Materials, Vol. 8, Issue 1 https://doi.org/10.1038/s41524-022-00784-w	journal	May 2022
A database of battery materials auto-generated using ChemDataExtractor Huang, Shu; Cole, Jacqueline M. Scientific Data, Vol. 7, Issue 1 https://doi.org/10.1038/s41597-020-00602-2	journal	August 2020
Auto-generated database of semiconductor band gaps using ChemDataExtractor Dong, Qingyang; Cole, Jacqueline M. Scientific Data, Vol. 9, Issue 1 https://doi.org/10.1038/s41597-022-01294-6	journal	May 2022
A database of refractive indices and dielectric constants auto-generated using ChemDataExtractor Zhao, Jiuyang; Cole, Jacqueline M. Scientific Data, Vol. 9, Issue 1 https://doi.org/10.1038/s41597-022-01295-5	journal	May 2022
Auto-generating databases of Yield Strength and Grain Size using ChemDataExtractor Kumar, Pankaj; Kabra, Saurabh; Cole, Jacqueline M. Scientific Data, Vol. 9, Issue 1 https://doi.org/10.1038/s41597-022-01301-w	journal	June 2022
A thermoelectric materials database auto-generated from the scientific literature using ChemDataExtractor Sierepeklis, Odysseas; Cole, Jacqueline M. Scientific Data, Vol. 9, Issue 1 https://doi.org/10.1038/s41597-022-01752-1	journal	October 2022
Automated Construction of a Photocatalysis Dataset for Water-Splitting Applications Isazawa, Taketomo; Cole, Jacqueline M. Scientific Data, Vol. 10, Issue 1 https://doi.org/10.1038/s41597-023-02511-6	journal	September 2023
A database of thermally activated delayed fluorescent molecules auto-generated from scientific literature with ChemDataExtractor Huang, Dingyun; Cole, Jacqueline M. Scientific Data, Vol. 11, Issue 1 https://doi.org/10.1038/s41597-023-02897-3	journal	January 2024
Auto-generated materials database of Curie and Néel temperatures via semi-supervised relationship extraction Court, Callum J.; Cole, Jacqueline M. Scientific Data, Vol. 5, Issue 1 https://doi.org/10.1038/sdata.2018.111	journal	June 2018
14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon Jablonka, Kevin Maik; Ai, Qianxiang; Al-Feghali, Alexander Digital Discovery, Vol. 2, Issue 5 https://doi.org/10.1039/D3DD00113J	journal	January 2023
Machine learning based feature engineering for thermoelectric materials by design Vaitesswar, U. S.; Bash, Daniil; Huang, Tan Digital Discovery, Vol. 3, Issue 1 https://doi.org/10.1039/D3DD00131H	journal	January 2024
MaScQA: investigating materials science knowledge of large language models Zaki, Mohd Digital Discovery, Vol. 3, Issue 2 https://doi.org/10.1039/D3DD00188A	journal	January 2024
Harnessing GPT-3.5 for text parsing in solid-state synthesis – case study of ternary chalcogenides Thway, Maung; Low, Andre K. Y.; Khetan, Samyak Digital Discovery, Vol. 3, Issue 2 https://doi.org/10.1039/D3DD00202K	journal	January 2024
Auto-generating question-answering datasets with domain-specific knowledge for language models in scientific tasks Li, Zongqian; Cole, Jacqueline M. Digital Discovery, Vol. 4, Issue 4 https://doi.org/10.1039/D4DD00307A	journal	January 2025
Machine-learning guided prediction of thermoelectric properties of topological insulator Bi₂Te_3−xSe_x K. E., Vipin; Padhan, Prahallad Journal of Materials Chemistry C, Vol. 12, Issue 20 https://doi.org/10.1039/D4TC01058B	journal	January 2024
An extensive review of tools for manual annotation of documents Neves, Mariana; Ševa, Jurica Briefings in Bioinformatics, Vol. 22, Issue 1 https://doi.org/10.1093/bib/bbz130	journal	December 2019
BioBERT: a pre-trained biomedical language representation model for biomedical text mining Lee, Jinhyuk; Yoon, Wonjin; Kim, Sungdong Bioinformatics https://doi.org/10.1093/bioinformatics/btz682	journal	September 2019
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks Williams, Ronald J.; Zipser, David Neural Computation, Vol. 1, Issue 2 https://doi.org/10.1162/neco.1989.1.2.270	journal	June 1989
Domain Adaptation with BERT-based Domain Classification and Data Selection Ma, Xiaofei; Xu, Peng; Wang, Zhiguo Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019) https://doi.org/10.18653/v1/D19-6109	conference	January 2019
Is Attention Interpretable? Serrano, Sofia; Smith, Noah A. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics https://doi.org/10.18653/v1/P19-1282	conference	January 2019
Training Tips for the Transformer Model Popel, Martin; Bojar, Ondřej The Prague Bulletin of Mathematical Linguistics, Vol. 110, Issue 1 https://doi.org/10.2478/pralin-2018-0002	journal	April 2018
A Photovoltaic Technology Review: History, Fundamentals and Applications Marques Lameirinhas, Ricardo A.; Torres, João Paulo N.; de Melo Cunha, João P. Energies, Vol. 15, Issue 5 https://doi.org/10.3390/en15051823	journal	March 2022
SQuAD: 100,000+ Questions for Machine Comprehension of Text Rajpurkar, Pranav; Zhang, Jian; Lopyrev, Konstantin arXiv https://doi.org/10.48550/arXiv.1606.05250	preprint	January 2016
Mixed Precision Training Micikevicius, Paulius; Narang, Sharan; Alben, Jonah arXiv https://doi.org/10.48550/arXiv.1710.03740	preprint	January 2017
Universal Language Model Fine-tuning for Text Classification Howard, Jeremy; Ruder, Sebastian arXiv https://doi.org/10.48550/arXiv.1801.06146	preprint	January 2018
Know What You Don't Know: Unanswerable Questions for SQuAD Rajpurkar, Pranav; Jia, Robin; Liang, Percy arXiv https://doi.org/10.48550/arXiv.1806.03822	preprint	January 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton arXiv https://doi.org/10.48550/arXiv.1810.04805	preprint	January 2018
Attention is not Explanation Jain, Sarthak; Wallace, Byron C. arXiv https://doi.org/10.48550/arXiv.1902.10186	preprint	January 2019
SciBERT: A Pretrained Language Model for Scientific Text Beltagy, Iz; Lo, Kyle; Cohan, Arman arXiv https://doi.org/10.48550/arXiv.1903.10676	text	January 2019
Entity-Relation Extraction as Multi-Turn Question Answering Li, Xiaoya; Yin, Fan; Sun, Zijun arXiv https://doi.org/10.48550/arXiv.1905.05529	preprint	January 2019
HuggingFace's Transformers: State-of-the-art Natural Language Processing Wolf, Thomas; Debut, Lysandre; Sanh, Victor arXiv https://doi.org/10.48550/arXiv.1910.03771	preprint	January 2019
AIMS-EREA -- A framework for AI-accelerated Innovation of Materials for Sustainability -- for Environmental Remediation and Energy Applications Pratihar, Sudarson Roy; Pai, Deepesh; Nag, Manaswita arXiv https://doi.org/10.48550/arXiv.2311.11060	preprint	January 2023

Similar Records

Language Models for the Prediction of SARS-CoV-2 Inhibitors

Conference · Sat Oct 01 00:00:00 EDT 2022 · International Journal of High Performance Computing Applications · OSTI ID:1892426

Language models for the prediction of SARS-CoV-2 inhibitors

Journal Article · Thu Oct 06 20:00:00 EDT 2022 · International Journal of High Performance Computing Applications · OSTI ID:1891374

Related Subjects

biological databases
extraction
thermodynamic modeling
thermoelectric materials
thermoelectrics

Autogenerating a Domain-Specific Question-Answering Data Set from a Thermoelectric Materials Database to Enable High-Performing BERT Models

Citation Formats

References (42)

Similar Records

Related Subjects