Log-based Impactful Problem Identification using Machine Learning [FSE'18]
Log3C is a general framework that identifies service system problems from system logs. It utilizes both system logs and system KPI metrics to promptly and precisely identify impactful system problems. Log3C consists of four steps: Log parsing, Sequence vectorization, Cascading Clustering and Correlation analysis. This is a joint work by CUHK and Microsoft Research.
The repository contains the source code of Log3C, including data loading, sequence vectorization, cascading clustering, data saving, etc. The core part is the cascading clustering algorithm, which groups a large number of sequence vectors into clusters by iteratively sampling, clustering, matching. For more details, please refer to our paper:
Note: Anaconda (Python 3.5 or above) is highly recommended, all required packages are already installed in Anaconda. You can also install the required packages with the "requirements.txt" by using command:
pip install -r requirements.txt
git clone https://github.com/logpai/Log3C.git
To use the model, open a terminal, change directory to this project code, run the command:
For details, please refer to the code comments.
|Log Seq | Event1 | Event2 | Event3 | Event4 | ... |
|:-------: | :----: |:-------:| :-----:| :-----:| :-----:|
| 1 | 2 | 1 | 0 | 2 | ... |
| 2 | 3 | 2 | 4 | 1 | ... |
| 3 | 2 | 1 | 3 | 3 | ... |
| ... | ... | ... | ... | ... | ... |
Assume that there are N time intervals in total, then we have N such matrixes as well as N KPI values. These K KPI values are stored in one file, which is shown as below.
|Time Interval | KPI | | :----: | :----: | | T0 | 0.05 | | T1 | 0.10 | | T2 | 0.07 | | .. | ... |
Only one KPI file, which contains N KPI values.