Volume 8, Number 6

With Semantics and Hidden Markov Models to an Adaptive Log File Parser

  Authors

Nadine Kuhnert and Andreas Maier, Friedrich-Alexander University, Germany

  Abstract

We aim to model an adaptive log file parser. As the content of log files often evolves over time, we established a dynamic statistical model which learns and adapts processing and parsing rules. First, we limit the amount of unstructured text by clustering based on semantics of log file lines. Next, we only take the most relevant cluster into account and focus only on those frequent patterns which lead to the desired output table similar to Vaarandi [10]. Furthermore, we transform the found frequent patterns and the output stating the parsed table into a Hidden Markov Model (HMM). We use this HMM as a specific, however, flexible representation of a pattern for log file parsing to maintain high quality output. After training our model on one system type and applying it to a different system with slightly different log file patterns, we achieve an accuracy over 99.99%.

  Keywords

Hidden Markov Models, Parameter Extraction, Parsing, Text Mining, Information Retrieval