When ChatGPT Meets Vulnerability Management: The Good, the Bad, and the Ugly

Conference · Wed Aug 20 04:00:00 EDT 2025

DOI:https://doi.org/10.1109/ICNC59896.2024.10555953· OSTI ID:2584212

McClanahan, Kylie; Elder, Waeland Sky; Uwibambe, Marie Louise; Liu, Yaling; Heng, Rithyka; Li, Qinghua

Vulnerability management is a very challenging and time-consuming task. For many organizations, security operators need to learn about the properties of vulnerabilities to prioritize and mitigate them. Due to the lack of automated tools for vulnerability assessment, operators usually manually search for and read related information from sources online. Recent advances in large language models, like ChatGPT, open up an opportunity for time savings and may prompt operators to use these models as vulnerability information sources. In this work, we evaluate the ability of ChatGPT and several of its siblings to accurately answer user questions about vulnerability properties as well as to provide information for how to mitigate a vulnerability. We also explore their summarization capabilities when multiple vulnerability advisory documents are provided. We find that the models perform poorly on information retrieval tasks, but they perform quite well on summarization.

• Research Organization: University of Arkansas
• Sponsoring Organization: Department of Energy; National Science Foundation
• DOE Contract Number: CR0000003
• OSTI ID: 2584212
• Country of Publication: United States
• Language: English