Log-based Anomaly Detection of Enterprise Software: An Empirical Study

Abstract

With software increasing in scale and playing an important role in today’s society, application reliability has become a crucial metric. Most applications use logging as a mechanism to diagnose anomalies, but is often employed post-failure. However, since logs provide insight into current software health, they can be leveraged to detect anomalies, which could help with reducing system downtime. Anomaly detection using software execution logs has been explored in several prior studies, using both classical and deep neural network based machine learning models. In recent years, the research has largely focused in using variations of sequence-based deep neural networks (e.g., Long-Short Term Memory and Transformer-based models) for log-based anomaly detection on open-source data. However, they have not been applied in industrial datasets as often. In addition, the studied open-source datasets are typically very large in size with logging statements that do not change over time, which may not be the case with a dataset from an industrial service that is relatively new. In this thesis, we evaluate several state-of-the-art anomaly detection models on an industrial dataset from our research partner, which is much smaller and loosely structured than most open-source datasets. Results show that while all models are capable of detecting anomalies, certain models are better suited for less-structured datasets. We also see that model effectiveness changes when data leaks associated with a random train-test split is removed. A qualitative study of error types identified by the developers on the industrial dataset further shows strengths and weaknesses of the models in detecting different types of anomalies. Finally, we explore the effect of limited training data by gradually increasing the training set size, to evaluate if the model effectiveness does depend on the training set size.