Sciweavers

Share
OSDI
2008
ACM

Mining Console Logs for Large-Scale System Problem Detection

9 years 3 months ago
Mining Console Logs for Large-Scale System Problem Detection
The console logs generated by an application contain messages that the application developers believed would be useful in debugging or monitoring the application. Despite the ubiquity and large size of these logs, they are rarely exploited in a systematic way for monitoring and debugging because they are not readily machineparsable. In this paper, we propose a novel method for mining this rich source of information. First, we combine log parsing and text mining with source code analysis to extract structure from the console logs. Second, we extract features from the structured information in order to detect anomalous patterns in the logs using Principal Component Analysis (PCA). Finally, we use a decision tree to distill the results of PCA-based anomaly detection to a format readily understandable by domain experts (e.g. system operators) who need not be familiar with the anomaly detection algorithms. As a case study, we distill over one million lines of console logs from the Hadoop f...
Wei Xu, Ling Huang, Armando Fox, David A. Patterso
Added 03 Dec 2009
Updated 03 Dec 2009
Type Conference
Year 2008
Where OSDI
Authors Wei Xu, Ling Huang, Armando Fox, David A. Patterson, Michael I. Jordan
Comments (0)
books