skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Packet Capture Solutions: PcapDB Benchmark for High-Bandwidth Capture, Storage, and Searching

Abstract

PcapDB stands alone when looking at the overall field of competitors, from the cost-effective COTS hardware, to the efficient utilization of disk space that enables a longer packet history. A scalable, 100GbE-enabled system that indexes every packet and indexes flow data without complicated load-balancing requirements. The Transport Layer search and indexing approach led to patent-pending flow indexing technology, providing a specialized database system specifically optimized around providing fast flow searches. While there are a plethora of options in network packet capture, there are very few that are able to effectively manage capture rates of more than 10 Gb/s, distributed capture and querying, and a responsive user interface. By far, the primary competitor in the market place is Endace and DeepSee; in addition to meeting the technical requirements we set out in this document, they provide technical support and a fully 'appliance like' system. In terms of cost, however, our experience has been that the yearly maintenance charges alone outstrip the entire hardware cost of solutions like PcapDB. Investment in cyber security research and development is a large part of what has enabled us to build the base of knowlegable workers needed to defend government resources in the rapidly evolving cybermore » security landscape. We believe projects like Bro, WireCap, and Farm do more than just fill temporary gaps in our capabilities. They give allow us to build the firm foundation needed to tackle the next generation of cyber challenges. PcapDB was built with loftier ambitions than simply solving the packet capture of a single lab site, but instead to provide a robust, scaleable packet capture solution to the DOE complex and beyond.« less

Authors:
 [1];  [1]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1351206
Report Number(s):
LA-UR-17-22359
DOE Contract Number:
AC52-06NA25396
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
96 KNOWLEDGE MANAGEMENT AND PRESERVATION; Packet capture; pcap; network; high speed capture; PcapDB; Bro; Endace; Moloch; FireEye PX; Solera; Bluecoast DeepSee; VAST OpenFPC; Stenographer

Citation Formats

Steinfadt, Shannon Irene, and Ferrell, Paul Steven. Packet Capture Solutions: PcapDB Benchmark for High-Bandwidth Capture, Storage, and Searching. United States: N. p., 2017. Web. doi:10.2172/1351206.
Steinfadt, Shannon Irene, & Ferrell, Paul Steven. Packet Capture Solutions: PcapDB Benchmark for High-Bandwidth Capture, Storage, and Searching. United States. doi:10.2172/1351206.
Steinfadt, Shannon Irene, and Ferrell, Paul Steven. Tue . "Packet Capture Solutions: PcapDB Benchmark for High-Bandwidth Capture, Storage, and Searching". United States. doi:10.2172/1351206. https://www.osti.gov/servlets/purl/1351206.
@article{osti_1351206,
title = {Packet Capture Solutions: PcapDB Benchmark for High-Bandwidth Capture, Storage, and Searching},
author = {Steinfadt, Shannon Irene and Ferrell, Paul Steven},
abstractNote = {PcapDB stands alone when looking at the overall field of competitors, from the cost-effective COTS hardware, to the efficient utilization of disk space that enables a longer packet history. A scalable, 100GbE-enabled system that indexes every packet and indexes flow data without complicated load-balancing requirements. The Transport Layer search and indexing approach led to patent-pending flow indexing technology, providing a specialized database system specifically optimized around providing fast flow searches. While there are a plethora of options in network packet capture, there are very few that are able to effectively manage capture rates of more than 10 Gb/s, distributed capture and querying, and a responsive user interface. By far, the primary competitor in the market place is Endace and DeepSee; in addition to meeting the technical requirements we set out in this document, they provide technical support and a fully 'appliance like' system. In terms of cost, however, our experience has been that the yearly maintenance charges alone outstrip the entire hardware cost of solutions like PcapDB. Investment in cyber security research and development is a large part of what has enabled us to build the base of knowlegable workers needed to defend government resources in the rapidly evolving cyber security landscape. We believe projects like Bro, WireCap, and Farm do more than just fill temporary gaps in our capabilities. They give allow us to build the firm foundation needed to tackle the next generation of cyber challenges. PcapDB was built with loftier ambitions than simply solving the packet capture of a single lab site, but instead to provide a robust, scaleable packet capture solution to the DOE complex and beyond.},
doi = {10.2172/1351206},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Mar 21 00:00:00 EDT 2017},
month = {Tue Mar 21 00:00:00 EDT 2017}
}

Technical Report:

Save / Share:
  • Traditional packet-capture solutions using commodity hardware incur a large amount of overhead as packets are copied multiple times by the operating system. This overhead slows sensor systems to a point where they are unable to keep up with high bandwidth traffic, resulting in dropped packets. Incomplete packet capture files hinder network monitoring and incident response efforts. While costly commercial hardware exists to capture high bandwidth traffic, several software-based approaches exist to improve packet capture performance using commodity hardware.
  • The traditional use of memory and symmetrical set of registers for storage of temporary results of scientific programs requires more execution time, hardware, and instruction-stream bandwidth than necessary. Novel register organizations that can be easily integrated into traditional supercomputer architectures can reduce all of these requirements. Execution speed can be more than doubled by storing temporary results in an asymmetrical set of general-purpose registers or an asymmetrical set of vector registers, instead of in memory and a small register-set. Faster access and a hardware cost one fourth that of traditional vector registers can be had by using a vector registermore » that incorporates a pipelines, random-access-memory chip. If a large enough set of registers is used, the need to store temporary results in memory and then reload them for later use can be eliminated; this saves both instruction-stream bandwidth and execution time. 111 refs., 43 figs., 40 tabs.« less
  • With the increasing number of geographically distributed scientific collaborations and the scale of the data size growth, it has become more challenging for users to achieve the best possible network performance on a shared network. We have developed a forecast model to predict expected bandwidth utilization for high-bandwidth wide area network. The forecast model can improve the efficiency of resource utilization and scheduling data movements on high-bandwidth network to accommodate ever increasing data volume for large-scale scientific data applications. Univariate model is developed with STL and ARIMA on SNMP path utilization data. Compared with traditional approach such as Box-Jenkins methodology,more » our forecast model reduces computation time by 83.2percent. It also shows resilience against abrupt network usage change. The accuracy of the forecast model is within the standard deviation of the monitored measurements.« less
  • Future large scale sciences are anticipated to use massive amount of data in their experiments. DOE's ESnet (Energy Science Network) is developing a 100 Gbps backbone based on this state-of-the-art 100 Gigabit Ethernet standard. ESnet will serve thousands of DOE and non-DOE scientists with its high bandwidth backbone, and connect several national laboratories. Current Ethernet test and debug solutions, such as network traffic capturer/analyzer tools, support up to 10 Gbps speed, and the very few capable of handling 100 Gbps are extremely costly. Such tools are essential in the development of high speed devices and routers, and ultimately the successmore » of 100 Gigabit Ethernet.« less
  • On-board hydrogen/methane storage in fuel cell-powered vehicles is a major component of the national need to achieve energy independence and protect the environment. The main obstacles in hydrogen storage are slow kinetics, poor reversibility and high dehydrogenation temperatures for the chemical hydrides; and very low desorption temperatures/energies for the physisorption materials (MOF’s, porous carbons). Similarly, the current methane storage technologies are mainly based on physisorption in porous materials but the gravimetric and volumetric storage capacities are below the target values. Finally, carbon capture, a critical component of the mitigation of CO2 emissions from industrial plants, also suffers from similar problems.more » The solid-absorbers such as MOFs are either not stable against real flue-gas conditions and/or do not have large enough CO2 capture capacity to be practical and cost effective. In this project, we addressed these challenges using a unique combination of computational, synthetic and experimental methods. The main scope of our research was to achieve fundamental understanding of the chemical and structural interactions governing the storage and release of hydrogen/methane and carbon capture in a wide spectrum of candidate materials. We studied the effect of scaffolding and doping of the candidate materials on their storage and dynamics properties. We reviewed current progress, challenges and prospect in closely related fields of hydrogen/methane storage and carbon capture.[1-5] For example, for physisorption based storage materials, we show that tap-densities or simply pressing MOFs into pellet forms reduce the uptake capacities by half and therefore packing MOFs is one of the most important challenges going forward. For room temperature hydrogen storage application of MOFs, we argue that MOFs are the most promising scaffold materials for Ammonia-Borane (AB) because of their unique interior active metal-centers for AB binding and well defined and ordered pores. Here the main challenge is to find a chemically stable MOF required for regeneration of the AB-spent fuel. Finally, for carbon capture application of MOFs, we investigate the performance of a number of metal–organic frameworks with particular focus on their behavior at the low pressures commonly used in swing adsorption. This comparison clearly shows that it is the process that determines which MOF is optimal rather than there being one best MOF, though MOFs that possess enhanced binding at open metal sites generally perform better than those with high surface area. References: 1. Y. Peng, V. Krungleviciute, J. T. Hupp, O. K. Farha, and T. Yildirim, J. Am. Chem. Soc. 135, 11887 (2013). 2. G. Srinivas, V. Krungleviciute, Z. Guo, and T. Yildirim, Ener. Environ. Sci. 7, 335 (2014). 3. G. Burres, and T. Yildirim, Ener. Environ. Sci. 5, 6453 (2012). 4. G. Srinivas, W. Travis, J. Ford, H. Wu, Z. X. Guo, and T. Yildirim, J. Mat. Chem.1, 4167 (2013). 5. For details, please see http://www.ncnr.nist.gov/staff/taner« less