Net-Fli: On-the-fly Compression, Archiving and Indexing of Streaming Network Traffic

The ever-increasing number of intrusions in public and commercial networks has created the need for high-speed archival solutions that continuously store streaming network data to enable forensic analysis and auditing. However, "turning back the clock" for post-attack analyses is not a trivial task. The first major challenge is that the solution has to sustain data archiving under extremely high-speed insertion rates. Moreover, the archives created need to be stored in a format that is compressed but still amenable to indexing. The above requirements make general-purpose databases unsuitable for this task, and, thus, dedicated solutions are required. In this paper, we describe a prototype solution that satisfies all requirements for high-speed archival storage, indexing and data querying on network flow information. The superior performance of our approach is attributed to the on-the-fly compression and indexing scheme, which is based on compressed bitmap principles. Typical...

• Type: Journal
• Year: 2010
• Where: PVLDB
• Authors: Francesco Fusco, Marc Ph. Stoecklin, Michail Vlachos
• Keywords: Insertion Rates, Large-scale Data Visualization, PVLDB 2010, Solutions