Beyond the Hype: An Evaluation of Commercially Available Machine-Learning-Based Malware Detectors

Bridges, Robert A.; Oesch, Sean; Iannacone, Michael D.; Huffer, Kelly T.; Jewell, Brian; Nichols, Jeff A.; Weber, Brian; Verma, Miki E.; Scofield, Daniel; Miles, Craig; Plummer, Thomas; Daniell, Mark; Tall, Anne M.; Beaver, Justin M.; Smith, Jared M.

doi:10.1145/3567432

Beyond the Hype: An Evaluation of Commercially Available Machine-Learning-Based Malware Detectors

Journal Article · Wed Feb 15 23:00:00 EST 2023 · Digital Threats: Research and Practice

DOI:https://doi.org/10.1145/3567432· OSTI ID:1965262

^[1]; ^[1]; Iannacone, Michael D. ^[1]; ^[1]; ^[1]; ^[1]; Weber, Brian ^[1]; Verma, Miki E. ^[2]; Scofield, Daniel ^[3]; Miles, Craig ^[3]; Plummer, Thomas ^[4]; Daniell, Mark ^[4]; Tall, Anne M. ^[5]; Beaver, Justin M. ^[6]; Smith, Jared M. ^[7]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Stanford Univ., CA (United States)
Amazon, Inc. (United States)
Lockheed Martin (United States)
MITRE Corporation (United States)
Lirio LLC (United States)
SecurityScorecard (United States)

There is a lack of scientific testing of commercially available malware detectors, especially those that boast accurate classification of never-before-seen (i.e., zero-day) files using machine learning (ML). Consequently, efficacy of malware detectors is opaque, inhibiting end users from making informed decisions and researchers from targeting gaps in current detectors. In this paper, we present a scientific evaluation of four prominent commercial malware detection tools to assist an organization with two primary questions: To what extent do ML-based tools accurately classify previously and never-before-seen files? Is purchasing a network-level malware detector worth the cost? To investigate, we tested each tool against 3,536 total files (2,554 or 72% malicious, 982 or 28% benign) of a variety of file types, including hundreds of malicious zero-days, polyglots, and APT-style files, delivered on multiple protocols. We present statistical results on detection time and accuracy, consider complementary analysis (using multiple tools together), and provide two novel applications of the recent cost-benefit evaluation procedure of Iannacone & Bridges. Although the ML-based tools are more effective at detecting zero-day files and executables, the signature-based tool might still be an overall better option. Both network-based tools provide substantial (simulated) savings when paired with either host tool, yet both show poor detection rates on protocols other than HTTP or SMTP. Our results show that all four tools have near-perfect precision but alarmingly low recall, especially on file types other than executables and office files—37% of malware, including all polyglot files, were undetected. Priorities for researchers and takeaways for end users are given. Code for future use of the cost model is provided.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1965262

Journal Information:: Digital Threats: Research and Practice, Journal Name: Digital Threats: Research and Practice Journal Issue: n/a Vol. n/a; ISSN 2692-1626

Publisher:: Association for Computing Machinery (ACM)Copyright Statement

Country of Publication:: United States

Language:: English

References (30)

Machine Learning Aided Static Malware Analysis: A Survey and Tutorial Shalaginov, Andrii; Banin, Sergii; Dehghantanha, Ali Advances in Information Security https://doi.org/10.1007/978-3-319-73951-9_2	book	January 2018
Instrumenting Competition-Based Exercises to Evaluate Cyber Defender Situation Awareness Reed, Theodore; Nauer, Kevin; Silva, Austin Foundations of Augmented Cognition https://doi.org/10.1007/978-3-642-39454-6_9	book	January 2013
Return on security investment – proving it's worth it Davis, Adrian Network Security, Vol. 2005, Issue 11 https://doi.org/10.1016/S1353-4858(05)70301-9	journal	November 2005
ISRAM: information security risk analysis method Karabacak, Bilge; Sogukpinar, Ibrahim Computers & Security, Vol. 24, Issue 2 https://doi.org/10.1016/j.cose.2004.07.004	journal	March 2005
A concise cost analysis of Internet malware Kondakci, Suleyman Computers & Security, Vol. 28, Issue 7 https://doi.org/10.1016/j.cose.2009.03.007	journal	October 2009
An empirical comparison of botnet detection methods García, S.; Grill, M.; Stiborek, J. Computers & Security, Vol. 45 https://doi.org/10.1016/j.cose.2014.05.011	journal	September 2014
Detection of malicious PDF files and directions for enhancements: A state-of-the art survey Nissim, Nir; Cohen, Aviad; Glezer, Chanan Computers & Security, Vol. 48 https://doi.org/10.1016/j.cose.2014.10.014	journal	February 2015
Quantifiable & comparable evaluations of cyber defensive capabilities: A survey & novel, unified approach Iannacone, Michael D.; Bridges, Robert A. Computers & Security, Vol. 96 https://doi.org/10.1016/j.cose.2020.101907	journal	September 2020
Adversarial attacks against Windows PE malware detection: A survey of the state-of-the-art Ling, Xiang; Wu, Lingfei; Zhang, Jiangyu Computers & Security, Vol. 128 https://doi.org/10.1016/j.cose.2023.103134	journal	May 2023
Performance comparison of intrusion detection systems and application of machine learning to Snort system Shah, Syed Ali Raza; Issac, Biju Future Generation Computer Systems, Vol. 80 https://doi.org/10.1016/j.future.2017.10.016	journal	March 2018
Mobile malware attacks: Review, taxonomy & future directions Qamar, Attia; Karim, Ahmad; Chang, Victor Future Generation Computer Systems, Vol. 97 https://doi.org/10.1016/j.future.2019.03.007	journal	August 2019
The rise of machine learning for detection and classification of malware: Research developments, trends and challenges Gibert, Daniel; Mateu, Carles; Planes, Jordi Journal of Network and Computer Applications, Vol. 153 https://doi.org/10.1016/j.jnca.2019.102526	journal	March 2020
A Comprehensive Review on Malware Detection Approaches Aslan, Omer; Samet, Refik IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2019.2963724	journal	January 2020
LARIAT: Lincoln adaptable real-time information assurance testbed Rossey, L.M.; Cunningham, R.K.; Fried, D.J. 2002 IEEE Aerospace Conference https://doi.org/10.1109/AERO.2002.1036158	conference	January 2002
Investigation of Possibilities to Detect Malware Using Existing Tools Aslan, Omer; Samet, Refik 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA) https://doi.org/10.1109/AICCSA.2017.24	conference	October 2017
Static Malware Detection & Subterfuge: Quantifying the Robustness of Machine Learning and Current Anti-Virus Fleshman, William; Raff, Edward; Zak, Richard 2018 13th International Conference on Malicious and Unwanted Software (MALWARE) https://doi.org/10.1109/MALWARE.2018.8659360	conference	October 2018
National Cyber Range Overview Ferguson, Bernard; Tall, Anne; Olsen, Denise 2014 IEEE Military Communications Conference https://doi.org/10.1109/MILCOM.2014.27	conference	October 2014
How the Cyber Defense Exercise Shaped an Information-Assurance Curriculum Mullins, Barry E.; Lacey, Timothy H.; Mills, Robert F. IEEE Security & Privacy Magazine, Vol. 5, Issue 5 https://doi.org/10.1109/MSP.2007.111	journal	September 2007
Evaluation studies of three intrusion detection systems under various attacks and rule sets Thongkanchorn, Kittikhun; Ngamsuriyaroj, Sudsanguan; Visoottiviseth, Vasaka 2013 IEEE International Conference of IEEE Region 10 (TENCON 2013) https://doi.org/10.1109/TENCON.2013.6718975	conference	October 2013
An Assessment of the Usability of Machine Learning Based Tools for the Security Operations Center Oesch, Sean; Bridges, Robert; Smith, Jared 2020 International Conferences on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics) https://doi.org/10.1109/iThings-GreenCom-CPSCom-SmartData-Cybermatics50389.2020.00111	conference	November 2020
Testing malware detectors Christodorescu, Mihai; Jha, Somesh ACM SIGSOFT Software Engineering Notes, Vol. 29, Issue 4 https://doi.org/10.1145/1013886.1007518	journal	July 2004
Hit 'em where it hurts Doupé, Adam; Egele, Manuel; Caillat, Benjamin Proceedings of the 27th Annual Computer Security Applications Conference https://doi.org/10.1145/2076732.2076740	conference	December 2011
Computing legacy software behavior to understand functionality and security properties Linger, Rick; Pleszkoch, Mark; Prowell, Stacy Proceedings of the Eighth Annual Cyber Security and Information Intelligence Research Workshop https://doi.org/10.1145/2459976.2460025	conference	January 2013
Polyglots Magazinius, Jonas; Rios, Billy K.; Sabelfeld, Andrei Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security - CCS '13 https://doi.org/10.1145/2508859.2516685	conference	January 2013
Large-Scale Identification of Malicious Singleton Files Li, Bo; Roundy, Kevin; Gates, Chris Proceedings of the Seventh ACM on Conference on Data and Application Security and Privacy https://doi.org/10.1145/3029806.3029815	conference	March 2017
A Survey on Malware Detection Using Data Mining Techniques Ye, Yanfang; Li, Tao; Adjeroh, Donald ACM Computing Surveys, Vol. 50, Issue 3 https://doi.org/10.1145/3073559	journal	June 2017
Dynamic Malware Analysis in the Modern Era—A State of the Art Survey Or-Meir, Ori; Nissim, Nir; Elovici, Yuval ACM Computing Surveys, Vol. 52, Issue 5 https://doi.org/10.1145/3329786	journal	September 2020
Security attribute evaluation method Butler, Shawn A. Proceedings of the 24th international conference on Software engineering - ICSE '02 https://doi.org/10.1145/581339.581370	conference	January 2002
A state-of-the-art survey of malware detection approaches using data mining techniques Souri, Alireza; Hosseini, Rahil Human-centric Computing and Information Sciences, Vol. 8, Issue 1 https://doi.org/10.1186/s13673-018-0125-x	journal	January 2018
Extract Me If You Can: Abusing PDF Parsers in Malware Detectors Carmony, Curtis; Zhang, Mu; Hu, Xunchao Proceedings 2016 Network and Distributed System Security Symposium https://doi.org/10.14722/ndss.2016.23483	conference	January 2016

Similar Records

AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors

Conference · Thu Nov 30 23:00:00 EST 2023 · OSTI ID:2301624

Toward the Detection of Polyglot Files

Conference · Mon Aug 08 00:00:00 EDT 2022 · OSTI ID:1885926

Toward the Detection of Polyglot Files

Conference · Mon Aug 01 00:00:00 EDT 2022 · OSTI ID:3002965

Related Subjects

97 MATHEMATICS AND COMPUTING
security and privacy

Beyond the Hype: An Evaluation of Commercially Available Machine-Learning-Based Malware Detectors

Citation Formats

References (30)

Similar Records

Related Subjects