Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors

Conference ·
 [1];  [1];  [2];  [3];  [1];  [1];  [1];  [1];  [1];  [1];  [1];  [4];  [5];  [1]
  1. ORNL
  2. Lirio, LLC
  3. SecurityScorecard
  4. Stanford University
  5. N/A
+ Show Author Affiliations

This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial I ntelligence Applications t o Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including ~1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files × 8 tools), a software framework is designed to choreograph the experiment into an automated, time-synced, and reproducible workflow with substantial parallelization. Software with base classes for this framework are provided. A cost-benefit model was configured to integrate the tools’ detection statistics into a comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens for reasoning about the varied statistical results. The results provide insights on state of commercial malware detection.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sponsoring Organization:
USDOE
DOE Contract Number:
AC05-00OR22725
OSTI ID:
2301624
Resource Relation:
Conference: IEEE International Conference on Big Data - Sorrento, , Italy - 12/15/2023 9:00:00 AM-12/18/2023 9:00:00 AM
Country of Publication:
United States
Language:
English

References (8)

Testing SOAR tools in use journal June 2023
The Role of Machine Learning in Cybersecurity journal March 2023
Toward the Detection of Polyglot Files conference August 2022
Quantifiable & comparable evaluations of cyber defensive capabilities: A survey & novel, unified approach journal September 2020
Testing malware detectors journal July 2004
Performance of malware detection tools: A comparison conference May 2014
Static Malware Detection & Subterfuge: Quantifying the Robustness of Machine Learning and Current Anti-Virus conference October 2018
Large-Scale Identification of Malicious Singleton Files conference March 2017

Similar Records

Related Subjects