Performance of chatbots in queries concerning fundamental concepts in photochemistry

Journal Article · 03 November 2024 · Photochemistry and Photobiology

DOI: https://doi.org/10.1111/php.14037 · OSTI ID:2476069

Taniguchi, Masahiko  ^[1]; Lindsey, Jonathan S.  ^[1]

---

1. Department of Chemistry North Carolina State University Raleigh North Carolina USA

Abstract The advent of chatbots raises the possibility of a paradigm shift across society including the most technical of fields with regard to access to information, generation of knowledge, and dissemination of education and training. Photochemistry is a scientific endeavor with roots in chemistry and physics and branches that encompass diverse disciplines ranging from astronomy to zoology. Here, five chatbots have each been challenged with 13 photochemically relevant queries. The chatbots included ChatGPT 3.5, ChatGPT 4.0, Copilot, Gemini Advanced, and Meta AI. The queries encompassed fundamental concepts (e.g., “Why is the fluorescence spectrum typically the mirror image of the absorption spectrum?”), practical matters (e.g., “What is the inner filter effect and how to avoid it?”), philosophical matters (“Please create the most important photochemistry questions.”), and specific molecular features (e.g., “Why are azo dyes non‐fluorescent?”). The chatbots were moderately effective in answering queries concerning fundamental concepts in photochemistry but were glaringly deficient in specialized queries for dyes and fluorophores. In some instances, a correct response was embedded in verbose scientific nonsense whereas in others the entire response, while grammatically correct, was utterly meaningless. The unreliable accuracy makes present chatbots poorly suited for unaided educational purposes and highlights the importance of domain experts.

• Sponsoring Organization: USDOE
• Grant/Contract Number: SC0025317
• OSTI ID: 2476069
• Journal Information: Photochemistry and Photobiology, Journal Name: Photochemistry and Photobiology; ISSN 0031-8655
• Publisher: Wiley-BlackwellCopyright Statement
• Country of Publication: United Kingdom
• Language: English