Using advanced data structures to enable responsive security monitoring

Journal Article · Sun Jan 23 23:00:00 EST 2022 · Cluster Computing

DOI:https://doi.org/10.1007/s10586-021-03463-5· OSTI ID:1883172

Vorobyeva, Janet ^[1]; Delayo, Daniel R. ^[2]; Bender, Michael A. ^[2]; Farach-Colton, Martín ^[3]; Pandey, Prashant ^[4]; Phillips, Cynthia Ann ^[1]; Singh, Shikha ^[5]; Thomas, Eric D. ^[1]; Kroeger, Thomas M. ^[1]

---

1. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
2. Stony Brook Univ., NY (United States)
3. Rutgers Univ., New Brunswick, NJ (United States)
4. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
5. Williams College, Williamstown, MA (United States)

Write-optimized data structures (WODS), offer the potential to keep up with cyberstream event rates and give sub-second query response for key items like IP addresses. These data structures organize logs as the events are observed. To work in a real-world environment and not fill up the disk, WODS must efficiently expire older events. As the basis for our research into organizing security monitoring data, we implemented a tool, called Diventi, to index IP addresses in connection logs using RocksDB (a write-optimized LSM tree). In this work, we extended Diventi to automatically expire data as part of the data structures’ normal operations. We guarantee that Diventi always tracks the N most recent events and tracks no more than N + k events for a parameter k < N, while ensuring the index is opportunistically pruned. To test Diventi at scale in a controlled environment, we used anonymized traces of IP communications collected at SuperComputing 2019. We synthetically extended the 2.4 billion connection events to 100 billion events. We tested Diventi vs. Elasticsearch, a common log indexing tool. In our test environment, Elasticsearch saw an ingestion rate of at best 37,000 events/s while Diventi sustained ingestion rates greater than 171,000 events/s. Our query response times were as much as 100 times faster, typically answering queries in under 80 ms. Furthermore, we saw no noticeable degradation in Diventi from expiration. We have deployed Diventi for many months where it has performed well and supported new security analysis capabilities.

• Research Organization: Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)
• Sponsoring Organization: USDOE National Nuclear Security Administration (NNSA); USDOE Laboratory Directed Research and Development (LDRD) Program; National Science Foundation (NSF)
• Grant/Contract Number: NA0003525
• OSTI ID: 1883172
• Report Number(s): SAND2021-15479J; 702298
• Journal Information: Cluster Computing, Journal Name: Cluster Computing Journal Issue: 4 Vol. 25; ISSN 1386-7857
• Publisher: Springer
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Expiration
Network monitoring
Security monitoring
Write-optimized