skip to main content


This content will become publicly available on April 5, 2019

Title: Sparse Coding for N-Gram Feature Extraction and Training for File Fragment Classification

File fragment classification is an important step in the task of file carving in digital forensics. In file carving, files must be reconstructed based on their content as a result of their fragmented storage on disk or in memory. Existing methods for classification of file fragments typically use hand-engineered features such as byte histograms or entropy measures. In this paper, we propose an approach using sparse coding that enables automated feature extraction. Sparse coding, or sparse dictionary learning, is an unsupervised learning algorithm, and is capable of extracting features based simply on how well those features can be used to reconstruct the original data. With respect to file fragments, we learn sparse dictionaries for n-grams, continuous sequences of bytes, of different sizes. These dictionaries may then be used to estimate n-gram frequencies for a given file fragment, but for significantly larger n-gram sizes than are typically found in existing methods which suffer from combinatorial explosion. To demonstrate the capability of our sparse coding approach, we used the resulting features to train standard classifiers such as support vector machines (SVMs) over multiple file types. Experimentally, we achieved significantly better classification results with respect to existing methods, especially when the features weremore » used in supplement to existing hand-engineered features.« less
 [1] ;  [1] ;  [1] ;  [1] ;  [1]
  1. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Publication Date:
Report Number(s):
Journal ID: ISSN 1556-6013; 661788
Grant/Contract Number:
Accepted Manuscript
Journal Name:
IEEE Transactions on Information Forensics and Security
Additional Journal Information:
Journal Name: IEEE Transactions on Information Forensics and Security; Journal ID: ISSN 1556-6013
Research Org:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Sponsoring Org:
USDOE National Nuclear Security Administration (NNSA)
Country of Publication:
United States
97 MATHEMATICS AND COMPUTING; File fragment classification; file carving; automated feature extraction; sparse coding; dictionary learning; unsupervised learning; n-gram; support vector machine
OSTI Identifier: