# Impact of Log Parsing on Log-based Anomaly Detection

Zanis Ali Khan
University of Luxembourg, Luxembourg
<EMAIL_ADDRESS>
Donghwan Shin
University of Sheffield, United Kingdom
<EMAIL_ADDRESS>
Domenico Bianculli
University of Luxembourg, Luxembourg
<EMAIL_ADDRESS>
Lionel Briand
University of Luxembourg, Luxembourg
University of Ottawa, Canada
<EMAIL_ADDRESS>

###### Abstract

Software systems log massive amounts of data, recording important runtime
information. Such logs are used, for example, for log-based anomaly detection,
which aims to automatically detect abnormal behaviors of the system under
analysis by processing the information recorded in its logs. Many log-based
anomaly detection techniques based on deep-learning models include a pre-
processing step called log parsing. However, understanding the impact of log
parsing on the accuracy of anomaly detection techniques has received
surprisingly little attention so far. Investigating what are the key
properties log parsing techniques should ideally have to help anomaly
detection is therefore warranted.

In this paper, we report on a comprehensive empirical study on the impact of
log parsing on anomaly detection accuracy, using 13 log parsing techniques and
five deep-learning-based anomaly detection techniques on two publicly
available log datasets. Our empirical results show that, despite what is
widely assumed, there is no strong correlation between log parsing accuracy
and anomaly detection accuracy (regardless of the metric used for measuring
log parsing accuracy). Moreover, we experimentally confirm existing
theoretical results showing that it is a property that we refer to as
distinguishability in log parsing results—as opposed to their accuracy—that
plays an essential role in achieving accurate anomaly detection.

Keywords: logs, log parsing, template identification, anomaly detection

##  1 Introduction

Software system execution logs provide valuable information about the runtime
behavior of the system, which is essential for monitoring and troubleshooting.
Among many log analysis approaches, log-based anomaly detection has been
actively studied to automatically detect abnormal behaviors of the system
under analysis by processing the information recorded in logs He et al.
(2021). Recently, anomaly detection techniques based on Deep Learning (DL)
models, such as Long Short-Term Memory (LSTM) Du et al. (2017); Meng et al.
(2019); Zhang et al. (2019) and Convolutional Neural Networks (CNNs) Lu et al.
(2018), have shown promising results.

One common aspect of most anomaly detection techniques is having a pre-
processing step called _log parsing_ (also known as log template
identification). This step is needed because anomaly detection techniques
require structured logs to automatically process them, whereas input logs are
often free-formed or semi-structured, as generated by logging statements
(e.g., printf() and logger.info()) in the source code. Many log parsing
techniques have also been developed to automatically convert unstructured
input logs into structured logs Zhu et al. (2019).

The frequent combination of log parsing and anomaly detection clearly implies
the importance of the former for the latter. Nevertheless, assessing in a
systematic way the impact of log parsing on anomaly detection has received
surprisingly little attention so far. Only recently, Shin et al. (2021)
investigated what ideal log parsing results are in terms of accurate anomaly
detection, but purely from a theoretical standpoint. Le and Zhang (2022)
empirically showed that different log parsing techniques, among other
potential factors, can significantly affect anomaly detection accuracy, but
the accuracy of log parsing results was not adequately measured, and the
correlation between log parsing accuracy and anomaly detection accuracy was
not reported. Although Fu et al. (2023) attempted to address the issue by
measuring log parsing and anomaly detection accuracy, they used only one log
parsing accuracy metric among those available in the literature Khan et al.
(2022); moreover, the log parsing results used to measure the accuracy of the
anomaly detection techniques represented only a small subset (less than 1%) of
all logs used by the anomaly detection techniques considered in the study,
thus limiting the validity of the results.

To systematically investigate the impact of log parsing on anomaly detection
while addressing the issues of the aforementioned studies, this paper reports
on an empirical study, in which we performed a comprehensive evaluation using
13 log parsing techniques and five anomaly detection techniques on two
publicly available log datasets. We considered all three log parsing accuracy
metrics (i.e., grouping accuracy Zhu et al. (2019), parsing accuracy Dai et
al. (2020), and template accuracy Khan et al. (2022)) proposed in the
literature.

Against all assumptions, our results show that there is no strong correlation
between log parsing accuracy and anomaly detection accuracy, regardless of the
metric used for measuring log parsing accuracy. In other words, accurate log
parsing results do not necessarily increase anomaly detection accuracy. To
better understand the phenomenon at play, we investigated another property of
log parsing, _distinguishability_ , a concept proposed by Shin et al. (2021))
that was theoretically shown to relate to anomaly detection accuracy. Our
empirical results confirm that distinguishability in log parsing results is
the property that really matters as far as anomaly detection accuracy is
concerned and that it should be the key target of log parsing.

In summary, the main contributions of this paper are:

  * •

the systematic and comprehensive evaluation of the impact of log parsing on
anomaly detection;

  * •

the investigation of the impact of the distinguishability of log parsing
results on anomaly detection;

  * •

a discussion of the practical implications for the application of log parsing
in the context of anomaly detection.

The rest of the paper is organized as follows. Section 2 provides basic
information used throughout the paper, including the definitions of logs,
messages, and templates, as well as an overview of log parsing and anomaly
detection. Section 3 motivates our study and introduces the research
questions. Section 4 describes the experimental design, including the log
datasets, log parsing techniques, and anomaly detection techniques used in the
experiments. Section 5 presents the experimental results. Section 6 discusses
the practical implications derived from the results. Section 8 concludes the
paper and provides directions for future work.

##  2 Background

In this section, we provide an overview of the main concepts that will be used
throughout the paper. We first introduce the definitions of logs, messages,
and log templates (§ 2.1). We then explain the concept of log parsing (also
known as log template identification) and illustrate different log parsing
accuracy metrics proposed in the literature (§ 2.2). We discuss log-based
anomaly detection and the corresponding accuracy metrics in § 2.3. Finally, we
summarize the recent theoretical results on ideal log parsing for accurate
anomaly detection, introducing the concept of _distinguishability_ for log
parsing results (§ 2.4).

###  2.1 Logs, Messages, and Templates

A _log_ is a sequence of log entries111Note that a log is different from a log
file. In practice, _one_ log file may contain _many_ logs representing the
execution flows of different components/sessions. For example, an HDFS (Hadoop
Distributed File System) log file contains many logs, distinguished by file
block IDs, each representing an independent execution for a specific block.. A
_log entry_ contains various information about the event being logged,
including a timestamp, a logging level (e.g., INFO, DEBUG), and a log message.
A _log message_ can be further decomposed into fixed and variable parts since
it is generated by executing a logging statement that can have both fixed
(hard-coded) strings and program variables in the source code. For example,
the execution of the logging statement “logger.info("Deleting block " + blkID
+ " file " + fileName)” when the program variables blkID and fileName evaluate
to blk-1781 and /hadoop/dfs, respectively, will generate a log entry “11:22:33
INFO Deleting block blk-1718 file /hadoop/dfs” where the log message “Deleting
block blk-1718 file /hadoop/dfs” can be decomposed into the fixed parts (i.e.,
“Deleting block” and “file”) and the variable parts (i.e., “blk-1718” and
“/hadoop/dfs”). A _(log message) template_ masks the various elements of each
variable part with a special character “<*>”; this representation is widely
used in log-based analyses (e.g., log parsing He et al. (2017); Jiang et al.
(2008), anomaly detection Zhang et al. (2019); Du et al. (2017), and log-based
testing Elyasov (2012); Jeong et al. (2020)) when it is important to focus on
the event types captured by a log message.

For instance, the template corresponding to the example log message “Deleting
block blk-1178 file /hadoop/dfs” is “Deleting block <*> file <*>”.

###  2.2 Log Parsing (Log Template Identification)

Although software execution logs contain valuable information about the run-
time behavior of the software system under analysis, they cannot be directly
processed by log-based analysis techniques that require structured input logs
(containing templates) instead of free-formed log messages. Extracting log
templates from log messages is straightforward when the source code with the
corresponding logging statements is available. However, often the source code
is unavailable, for example, due to the usage of 3rd-party, proprietary
components. This leads to the problem of log parsing (log template
identification): How can we identify the log templates of log messages without
accessing the source code?

To address this problem, many automated log-parsing approaches, which take as
input log messages and identify their log templates using different
heuristics, have been proposed in the literature (e.g., AEL Jiang et al.
(2008), Drain He et al. (2017), IPLoM Makanju et al. (2009), LenMa Shima
(2016), LFA Nagappan and Vouk (2010), LogCluster Vaarandi and Pihelgas (2015),
LogMine Hamooni et al. (2016), Logram Dai et al. (2020), LogSig Tang et al.
(2011), MoLFI Messaoudi et al. (2018), SHISO Mizutani (2013), SLCT Vaarandi
(2003), and Spell Du and Li (2016)).

Three different accuracy metrics have been proposed to evaluate the accuracy
of log parsing approaches: Grouping Accuracy (GA) Zhu et al. (2019), Parsing
Accuracy (PA) Dai et al. (2020), and Template Accuracy (TA) Khan et al.
(2022).

Zhu et al. (2019) observed that log parsing can be considered as a clustering
process where log messages with the same template are clustered into the same
group. Based on this idea, they proposed the GA metric to assess if log
messages are correctly grouped. Specifically, GA is defined as the ratio of
log messages correctly parsed by the log parsing approach under evaluation
over the total number of log messages, where a log message is correctly parsed
when its log message group is the same as the ground truth (i.e., a group
generated by oracle templates).

Dai et al. (2020) later proposed PA, to address the issue that GA only
considers message groups, not the equivalence between the templates identified
by the log parsing approach under evaluation and the oracle templates.
Although having correctly grouped messages would be enough in some cases
(e.g., detecting anomalies based on the sequence of template IDs without
considering the content of the templates Du et al. (2017)), correctly
identified templates (i.e., templates identical to the corresponding oracle
ones) matter when the fixed parts of templates are used (e.g., detecting
anomalies based on the semantic information in the templates Zhang et al.
(2019)). To this end, PA replaces the definition of a correctly parsed log
message in GA as follows: a log message is correctly parsed when its
identified template is identical to the oracle template.

Khan et al. (2022) recently proposed the TA metric, since both GA and PA are
defined based on the number of correctly parsed log messages and, therefore,
can be misleading, especially when there are many repeated messages (e.g.,
heartbeat messages). Specifically, they introduced Precision-TA (PTA) and
Recall-TA (RTA), where PTA is defined as the number of correctly identified
templates over the total number of identified templates and RTA is defined as
the number of correctly identified templates over the total number of oracle
templates. Moreover, FTA (short for “F1-measure TA”) is the harmonic mean of
PTA and RTA.

###  2.3 Anomaly Detection

(Log-based) anomaly detection is a technique that aims to identify anomalous
patterns, recorded in input logs, that do not conform to the expected
behaviors of the system under analysis He et al. (2021). It takes as input a
sequence of log templates and determines whether the given sequence represents
a normal behavior of the system or not.

With the recent advances in Deep Learning (DL), many anomaly detection
approaches, which leverage DL models to learn various aspects of log template
sequences of normal and abnormal behaviors and classify them, have been
proposed in the literature; for example, DeepLog Du et al. (2017), LogAnomaly
Meng et al. (2019), and LogRobust Zhang et al. (2019) are based on Long Short-
Term Memory based (LSTM), CNN Lu et al. (2018) is based on Convolutional
Neural Network, and PLELog Yang et al. (2021) is based on Gated recurrent
units (GRUs).

To assess the accuracy of anomaly detection approaches, it is common practice
to use standard metrics from the information retrieval domain, such as
_Precision_ , _Recall_ , and _F1-Score_. These metrics are defined as follows:
$\textit{Precision}=\frac{\textit{TP}}{\textit{TP}+\textit{FP}}$,
$\textit{Recall}=\frac{\textit{TP}}{\textit{TP}+\textit{FN}}$, and
$\textit{F1-score}=\frac{2\times\textit{Precision}\times\textit{Recall}}{\textit{Precision}+\textit{Recall}}$
where TP (True Positive) is the number of abnormal logs correctly identified
by the model, FP (False Positive) is the number of normal logs incorrectly
identified as anomalies by the model, and FN (False Negative) is the number of
abnormal logs incorrectly identified as normal.

###  2.4 Ideal Log Parsing Results for Accurate Anomaly Detection

Given the dependency of anomaly detection on log parsing, Shin et al. (2021)
presented a theoretical analysis on _ideal_ log parsing results for accurate
anomaly detection. The idea behind the analysis is that log parsing can be
regarded as the abstraction of log messages, where some tokens in the messages
are converted to variable parts. Then, if normal and abnormal logs are over-
abstracted by log parsing so that they are indistinguishable from each other,
it is clear that anomaly detection, which takes as input the parsed logs
(i.e., abstracted logs, sequences of templates), cannot distinguish normal
from abnormal logs. Based on this idea, they formally defined the concept of
_distinguishability_ for log parsing results and showed that it is an
essential condition for ideal log parsing results.

Specifically, let $M$ be a set of log messages and $L$ be a set of logs where
a log $l\in L$ is a sequence of log messages
$\langle\mathit{m_{1},m_{2}},\dots,m_{n}\rangle$. Also, let $L_{n}\subseteq L$
be a set of normal logs and $L_{a}\subseteq L$ be a set of abnormal logs such
that $L_{n}\cap L_{a}=\emptyset$ and $L_{n}\cup L_{a}=L$. Given $M$ and a set
of templates (i.e., log parsing results) $T$, an abstraction function
$\tau:M\to T$ that represents a generic log parsing approach can be defined.
Based on $\tau$, an abstraction of a log
$l=\langle\mathit{m_{1},m_{2}},\dots,m_{n}\rangle$ can be defined as
$\tau^{*}(l)=\langle\mathit{\tau(m_{1}),\tau(m_{2})},\dots,\tau(m_{n})\rangle$.
Similarly, an abstraction of a set of logs $L$ can be defined as
$\tau^{**}(L)=\\{\tau^{*}(l)\mid l\in L\\}$. Notice that $\tau^{**}(L)$
represents a log parsing result for a set of logs $L$.

The notion of distinguishability can be defined as follows: $\tau$
_distinguishes_ $L_{n}$ and $L_{a}$ if and only if
$\tau^{**}(L_{n})\cap\tau^{**}(L_{a})=\emptyset$. In other words, a log
parsing approach distinguishes between normal and abnormal logs if and only if
they are still distinguishable after log parsing. When $\tau$ distinguishes
$L_{n}$ and $L_{a}$, $\tau^{**}(L)$ for $L=L_{n}\cup L_{a}$ is called _d-
maintaining_ , meaning that the distinguishability between $L_{n}$ and $L_{a}$
is _maintained_ in the log parsing result.

##  3 Motivation

As discussed in Section 2, log parsing converts unstructured logs into
structured ones, which can then be processed by log-based analysis techniques
like anomaly detection. It is quite natural to speculate that log parsing
results can affect anomaly detection results. Intuitively, the research
literature has assumed that inaccurate log parsing results leads to inaccurate
anomaly detection results. However, this hypothesis has not been fully
investigated in the literature, except for one empirical study Le and Zhang
(2022) and one analytical investigation Shin et al. (2021).

Le and Zhang (2022) recently presented an empirical work investigating several
aspects that can impact Deep Learning (DL)-based anomaly detection approaches,
such as training data selection, data grouping, class distribution, data
noise, and early detection ability. One of their experiments considering data
noise assessed the impact of noise deriving from log parsing results.
Specifically, they used four log parsing techniques (Drain He et al. (2017),
Spell Du and Li (2016), AEL Jiang et al. (2008), and IPLoM Makanju et al.
(2009)) to generate log parsing results for two log datasets (BGL Oliner and
Stearley (2007) and Spirit Oliner and Stearley (2007)). Then, for each log
dataset, they used the different log parsing results as input of five anomaly
detection approaches (DeepLog Du et al. (2017), LogAnomaly Meng et al. (2019),
PLELog Yang et al. (2021), LogRobust Zhang et al. (2019), and CNN Lu et al.
(2018)), and measured the accuracy of the latter. Their experimental results
showed that log parsing approaches highly influence the accuracy of anomaly
detection approaches; for example, the F1-Score of DeepLog on Spirit logs
Oliner and Stearley (2007) decreases from $0.755$ to $0.609$ when Drain is
used instead of IPLoM for log parsing.

Although this is the first clear evidence showing the impact of log parsing
results on anomaly detection accuracy, the scope of the underlying study is
limited. For example, it does not quantitatively assess the accuracy of log
parsing; therefore, it is still unclear if inaccurate log parsing results lead
to inaccurate anomaly detection results. To this end, we define our first
research question as follows: _RQ1 - To which extent does the accuracy of log
parsing affect the accuracy of anomaly detection?_

As summarized in Section 2.4, Shin et al. (2021) recently proposed a
theoretical framework determining the ideal log parsing results for anomaly
detection by introducing the concept of “distinguishability” for log parsing
results. It is argued that, rather than accuracy as previously assumed, what
really matters is the extent to which log parsing results are distinguishable.
However, to the best of our knowledge, there is no empirical work assessing
quantitatively distinguishability in log parsing results and its impact on
anomaly detection accuracy. Therefore, we define our second research question
as follows: _RQ2 - How does the accuracy of anomaly detection vary with
distinguishability of log parsing results?_

Answering the above questions will have a significant impact on both research
and industry in the field of log-based anomaly detection. For example, if the
answer to the first question is that, regardless of the log parsing accuracy
metrics, there is no relationship between log parsing accuracy and anomaly
detection accuracy, then it means that there is no need to use the existing
accuracy metrics to evaluate log parsing results for anomaly detection. This
would completely change the way log parsing tools are evaluated. Similarly, if
the answer to the second question is that the distinguishability of log
parsing results indeed affects anomaly detection, as expected from the recent
theoretical analysis Shin et al. (2021), then this must be the focus of log
parsing evaluations. As a result, our answers will provide essential insights
on better assessing the quality of log parsing techniques for more accurate
anomaly detection.

##  4 Experimental Design

As motivated in Section 3, we will answer the following research questions:

  1. RQ1:

To what extent does the accuracy of log parsing affect the accuracy of anomaly
detection?

  2. RQ2:

What is the relationship between the distinguishability of log parsing results
and anomaly detection accuracy?

###  4.1 Datasets

To answer the research questions, we used publicly available datasets based on
the LogHub benchmark He et al. (2020), which contains a large collection of
log messages from various types of systems including operating systems (Linux,
Windows, and Mac), distributed systems (BGL, Hadoop, HDFS, Thunderbird, and
OpenStack), standalone programs (Proxifier and Zookeeper), and mobile systems
(Android). Among them, we initially selected HDFS, Hadoop, and OpenStack
datasets because (1) they have labels for normal and abnormal logs to be used
for assessing the accuracy of anomaly detection techniques and (2) they have
publicly available source code (of the exact program version used to generate
the logs) from which we could extract correct oracle templates (i.e., ground
truth templates) for each log message. Although the benchmark provides oracle
templates for all log datasets, they are manually generated without accessing
the source code and only for 2K log messages randomly sampled for each
dataset. As discussed by Khan et al. (2022), those manually generated oracle
templates are error-prone; therefore, we used the logging statements in the
source code to extract correct oracle templates.

For HDFS and OpenStack datasets, during the preliminary evaluation, we found
two issues. First, none of the anomaly detection techniques detected any
anomalies in the OpenStack logs due to the very small number of abnormal logs
(4 out of 2068). Therefore, we had to exclude OpenStack. Second, the original
HDFS logs were too large to be processed by the slowest anomaly detection
technique (i.e., LogAnomaly Meng et al. (2019)) when setting a two-day
timeout. Due to the large number of experiments we needed to conduct (i.e.,
all combinations of log parsing and anomaly detection techniques with
additional repeats for distinguishable and indistinguishable log parsing
results, see § 4.4 and § 4.5), we decided to reduce the log dataset sizes,
especially for HDFS. Specifically, we randomly selected $n$ log messages
($n\in\\{10K,20K,\dots,500K\\}$) and checked if the slowest anomaly detection
technique could process the messages and terminate successfully within a
reasonable time limit (2 hours). Through this process, we established that
LogAnomaly (i.e., the slowest log parsing technique) could process up to
$n=300K$ messages within 2 hours. Therefore, we randomly and iteratively
removed logs from the HDFS dataset to build a reduced one until the total
number of remaining messages was less than 300K. Notice that we preserved the
sequence of log messages in each log.

To summarize, we considered two datasets: HDFS (reduced) and Hadoop. Table 1
provides the size information of these datasets, in terms of the number of
oracle templates (O), the number of all logs ($L_{\mathit{all}}$), the number
of normal logs ($L_{\mathit{n}}$), the number of abnormal logs
($L_{\mathit{a}}$), the number of all messages ($M_{\mathit{all}}$), the
number of messages in normal logs ($M_{\mathit{n}}$), and the number of
messages in abnormal logs ($M_{\mathit{a}}$).

Table 1: Log datasets Dataset | O | $L_{all}$ | $L_{n}$ | $L_{a}$ | $M_{all}$ | $M_{n}$ | $M_{a}$
---|---|---|---|---|---|---|---
HDFS (reduced) | 26 | 15295 | 15026 | 269 | 299971 | 292776 | 7195
Hadoop | 175 | 54 | 11 | 43 | 109968 | 14392 | 95576

###  4.2 Log Parsing Techniques

We aimed to use as many log parsing techniques available in the literature as
possible. Since Khan et al. (2022) recently provided a comprehensive
evaluation of 14 log parsing techniques (i.e., AEL Jiang et al. (2008), Drain
He et al. (2017), IPLoM Makanju et al. (2009), LenMa Shima (2016), LFA
Nagappan and Vouk (2010), LKE Fu et al. (2009), LogCluster Vaarandi and
Pihelgas (2015), LogMine Hamooni et al. (2016), Logram Dai et al. (2020),
LogSig Tang et al. (2011), MoLFI Messaoudi et al. (2018), SHISO Mizutani
(2013), SLCT Vaarandi (2003), and Spell Du and Li (2016)), we decided to reuse
their replication package, including all the aforementioned techniques.
However, we had to exclude LKE since our preliminary evaluation results showed
that it could not complete its run for all of our log datasets within the
2-day timeout.

As a result, we considered 13 log parsing techniques in our experiments. For
all the log parsing techniques, we used their default parameters.

###  4.3 Anomaly Detection Techniques

Similar to the case of log parsing techniques, we considered the work of Le
and Zhang (2022), a recent empirical study that evaluated five DL-based
anomaly detection techniques (i.e., DeepLog Du et al. (2017), LogAnomaly Meng
et al. (2019), LogRobust Zhang et al. (2019), PLELog Yang et al. (2021), and
CNN Lu et al. (2018)), and decided to use their replication package, including
all the aforementioned techniques. These techniques are representative of the
state of the art of DL-based anomaly detection techniques.

For all the anomaly detection techniques, we used their default parameters.

###  4.4 Methodology for RQ1

To answer RQ1, for each dataset, we first executed the log parsing techniques
to generate log parsing results and computed their accuracy in terms of GA,
PA, and FTA (see § 2.2). We then executed the anomaly detection techniques on
each of the log parsing results and computed their accuracy in terms of
precision (PR), recall (RE), and F1 score. By doing so, we obtained a tuple of
accuracy values $\langle\mathit{GA,PA,FTA,PR,RE,F1}\rangle$ for each
combination of datasets, log parsing results, and anomaly detection
techniques.

For log parsing, we executed each of the log parsing techniques with a 2-day
timeout. Since MoLFI is non-deterministic, we executed it three times. In
total, we obtained 16 log parsing results (three from the three different
executions of MoLFI and 13 from the remaining log parsing techniques) for each
dataset. For each log parsing result, we computed
$\langle\mathit{GA,PA,FTA}\rangle$ using the oracle templates (and the
messages matching them) for the corresponding datasets.

For anomaly detection, we divided the individual log parsing results into two
disjoint sets, i.e., a training set and a test set, using a split ratio of
80:20. We trained the anomaly detection techniques on each of the training
sets with a 2-day timeout, and used the corresponding test sets to compute
$\langle\mathit{PR,RE,F1}\rangle$. To account for the randomness of anomaly
detection techniques, we repeated the train-and-test process five times and
used the average F1 score.

As a result, we obtained 160 tuples
$\langle\mathit{GA,PA,FTA,PR,RE,F1}\rangle$ from the combinations of two
datasets, 16 log parsing results, and five anomaly detection techniques.

###  4.5 Methodology for RQ2

To answer RQ2, we need distinguishable and indistinguishable log parsing
results to compare in terms of anomaly detection accuracy. Although the log
parsing results generated for RQ1 are available, they are mostly
distinguishable, leading to unbalanced data for RQ2. To systematically assess
the impact of the distinguishability of log parsing results on anomaly
detection accuracy using well-balanced data, we generate pairs of
distinguishable and indistinguishable log parsing results. Specifically, let
$d(R)$ be the distinguishability — expressed as a Boolean value, either _true_
($T$) or _false_ ($F$) — of a log parsing result $R$. For each log parsing
result $R$ generated in the context of RQ1 (i.e., 16 log parsing results for
each of the two datasets), we first created a pair of log parsing results
$\langle R,R^{\prime}\rangle$ by artificially generating $R^{\prime}$ from $R$
such that $d(R^{\prime})=\neg d(R)$ using Algorithms 1 and 2, detailed further
below. By definition, if the log parsing result $R$ is distinguishable then
$R^{\prime}$ will be indistinguishable and vice versa. For the sake of
simplicity, we denote the distinguishable result (be it $R$ or $R^{\prime}$)
as $R_{\mathit{dst}}$ and the indistinguishable one (respectively, either
$R^{\prime}$ or $R$) as $R_{\mathit{ind}}$. We then executed, for all pairs
$\langle R_{\mathit{dst}},R_{\mathit{ind}}\rangle$, all the considered anomaly
detection techniques twice: the first time using $R_{\mathit{dst}}$ as input
and the second time using $R_{\mathit{ind}}$ as input; for each run of each
anomaly detection technique we computed its accuracy in terms of precision,
recall, and F1 score. By doing so, we obtained the anomaly detection accuracy
scores for pairs of distinguishable ($R_{\mathit{dst}}$) and indistinguishable
($R_{\mathit{ind}}$) versions of log parsing results, and then compared them.

For the generation of $R^{\prime}$ from $R$, it is important to minimize the
difference between $R$ and $R^{\prime}$ while achieving $d(R^{\prime})=\neg
d(R)$. This is to ensure that if there is a difference in anomaly detection
scores between $R$ and $R^{\prime}$, it is due to distinguishability and not
to other differences between $R$ and $R^{\prime}$ (e.g., the number of
templates or the size of log parsing results). To do this, we considered the
two cases, when $d(R)=T$ and when $d(R)=F$, separately.

When $d(R)=T$ (i.e., $R$ is $R_{\mathit{dst}}$), it means that templates for
different log messages in $R$ are different enough to distinguish between
normal and abnormal logs in $R$, as explained in Section 2.4. For example, let
us consider two logs $l_{1}=\langle\mathit{m_{1},m_{2}}\rangle$ and
$l_{2}=\langle\mathit{m_{3},m_{4}}\rangle$ where the templates of the four
messages are identified as $\tau(m_{1})=t_{1}$, $\tau(m_{2})=t_{2}$,
$\tau(m_{3})=t_{3}$, and $\tau(m_{4})=t_{2}$, respectively, using a log
parsing technique $\tau$. Figure 1 shows the logs, messages, and templates. In
this case, the log parsing result of $\tau$ for $\\{l_{1},l_{2}\\}$ is
distinguishable, as highlighted in blue in the figure, since
$\tau^{*}(l_{1})=\langle\tau(m_{1}),\tau(m_{2})\rangle=\langle
t_{1},t_{2}\rangle$ and
$\tau^{*}(l_{2})=\langle\mathit{\tau(m_{3}),\tau(m_{4})}\rangle=\langle
t_{3},t_{2}\rangle$ are different (due to $\tau(m_{1})\neq\tau(m_{3})$, i.e.,
$t_{1}\neq t_{3}$). However, if the templates of $m_{1}$ and $m_{3}$ were the
same, then the log parsing result would be indistinguishable. In other words,
as highlighted in red in the figure, we can make the distinguishable log
parsing result of $\tau$ indistinguishable by merging the templates of $m_{1}$
and $m_{3}$ (i.e., by introducing a dummy template $m_{13}$ that corresponds
to both $m_{1}$ and $m_{3}$). Using this idea, to generate
$R^{\prime}=R_{\mathit{ind}}$ from $R=R_{\mathit{dst}}$, we generated the
templates of $R_{\mathit{ind}}$ by iteratively merging the templates of
$R_{\mathit{dst}}$ until $d(R_{\mathit{ind}})=F$. Furthermore, to minimize the
difference between $R_{\mathit{dst}}$ and $R_{\mathit{ind}}$ in terms of the
number of templates (i.e., to minimize the number of templates being merged),
we start with merging the templates with the highest number of matching
messages in the log. This is based on the intuition that the more messages
affected by merging templates, the more likely normal and abnormal logs are to
become indistinguishable.

Algorithm 1 summarizes the above-mentioned idea into the pseudocode for
generating $R_{\mathit{ind}}$ from $R_{\mathit{dst}}$. After initializing
$R_{\mathit{ind}}$ (line 1) as a copy of $R_{\mathit{dst}}$, the algorithm
extracts the set of templates $T$ of $R_{\mathit{dst}}$ (line 1) and sorts the
templates in $T$ in ascending order by the number of matching messages (line
1). The algorithm then iteratively merges the last $n$ templates (starting
from $n=2$ as initialized at line 1) in the sorted templates list $T_{s}$
(i.e., merging the top-$n$ templates that have the highest number of matching
templates) until $R_{\mathit{ind}}$ becomes indistinguishable (lines 1–1).
Notice that the while loop does not continue endlessly since
$R_{\mathit{ind}}$ must be indistinguishable when $n$ becomes $|T_{s}|$ (i.e.,
all templates are merged into one) by definition. The algorithm ends by
returning $R_{\mathit{ind}}$.

1

Input : Distinguishable Log Parsing Result $R_{\mathit{dst}}$

Output : Indistinguishable Log Parsing Result $R_{\mathit{ind}}$

2

3Log Parsing Result (Set of Parsed Logs)
$R_{\mathit{ind}}\leftarrow\textit{copy}(R_{\mathit{dst}})$

4 Set of Templates $T\leftarrow\textit{getTemplates}(R_{\mathit{dst}})$

5 Sorted List of Templates $T_{s}\leftarrow\textit{sortByNumMessages}(T)$

6 Integer $n\leftarrow 2$

7 while  _$d(R_{\mathit{ind}})=\textit{True}$_ do

8 Set of Templates $T_{m}\leftarrow\textit{getLastTemplates}(T_{s},n)$

9 $R_{\mathit{ind}}\leftarrow\textit{mergeTemplates}(T_{m},R_{\mathit{dst}})$

10 $n\leftarrow n+1$

11

12 return $R_{\mathit{ind}}$

13

Algorithm 1 Generating an indistinguishable log parsing result from a distinguishable one Log | Template | Parsed Log (original) |  Parsed Log (after merging $t_{1}$ and $t_{3}$ into $t_{13}$)
---|---|---|---
$l1$ = $\langle m_{1},m_{2}\rangle$ |  $\tau(m_{1})=t_{1}$ , $\tau(m_{2})=t_{2}$ | $\tau^{*}(l_{1})=\langle t_{1},t_{2}\rangle$ | $\tau^{*}(l_{1})=\langle t_{13},t_{2}\rangle$
$l2$ = $\langle m_{3},m_{4}\rangle$ |  $\tau(m_{3})=t_{3}$ , $\tau(m_{4})=t_{2}$ | $\tau^{*}(l_{2})={\langle t_{3},t_{2}\rangle}$ | $\tau^{*}(l_{2})=\langle t_{13},t_{2}\rangle$
Figure 1: An example of making a distinguishable log parsing result
indistinguishable by merging templates

When $d(R)=F$ (i.e., $R$ is $R_{\mathit{ind}}$), although one could do the
dual of merging templates (i.e., splitting templates), it would be very
complex, requiring to answer the following questions: Which templates to
split? How many templates should be generated by splitting one template?
Instead, we adopted another heuristic: we removed the normal (or abnormal)
logs that are indistinguishable from abnormal (or normal) logs. This is based
on our observation that, when $d(R)=F$, only a small number of normal and
abnormal logs are indistinguishable. To minimize the impact of removing logs,
we removed normal logs when the total number of normal logs is larger than
that of abnormal logs (as it is the case for the HDFS dataset); otherwise, we
removed abnormal logs (in the case of the Hadoop dataset).

Algorithm 2 shows how to generate $R_{\mathit{dst}}$ from $R_{\mathit{ind}}$
based on the above idea. It first extracts the set of indistinguishable logs
$L_{\mathit{ind}}$ from $R_{\mathit{ind}}$ (line 2). It then removes either
normal or abnormal logs in $L_{\mathit{ind}}$ from $R_{\mathit{ind}}$ to
generate $R_{\mathit{dst}}$ depending on the total number of normal and
abnormal logs (lines 2–2). Since $R_{\mathit{dst}}$ is the result of removing
indistinguishable (normal or abnormal) logs from $R_{\mathit{ind}}$,
$R_{\mathit{dst}}$ is distinguishable. The algorithm ends by returning
$R_{\mathit{dst}}$.

1

Input : Indistinguishable Log Parsing Result $R_{\mathit{ind}}$

Output : Distinguishable Log Parsing Result $R_{\mathit{dst}}$

2

3Set of Indistinguishable Logs
$L_{\mathit{ind}}\leftarrow\textit{getIndistLogs}(R_{\mathit{ind}})$

4 if
_$\textit{numNormalLogs}(R_{\mathit{ind}})\geq\textit{numAbnormalLogs}(R_{\mathit{ind}})$_
then

5 Set of Parsed Logs $R_{\mathit{dst}}\leftarrow
R_{\mathit{ind}}\setminus\textit{getNormalLogs}(L_{\mathit{ind}})$

6

7else

8 Set of Parsed Logs $R_{\mathit{dst}}\leftarrow
R_{\mathit{ind}}\setminus\textit{getAbnormalLogs}(L_{\mathit{ind}})$

9

10return $R_{\mathit{dst}}$

11

Algorithm 2 Generating a distinguishable log parsing result from an
indistinguishable one

Notice that some of the anomaly detection techniques (i.e., LogRobust Zhang et
al. (2019), PLELog Yang et al. (2021), LogAnomaly Meng et al. (2019)) use the
semantic information of templates, instead of simply using template IDs, by
converting them into semantic vectors Jurafsky and Martin (2019). Therefore,
the notion of “identical” templates for determining the distinguishability of
log parsing results must be revised in terms of the semantic vectors used by
these anomaly detection techniques. To do this, for each log parsing result
$R$, we applied a clustering algorithm to the semantic vectors of all
templates and considered the templates in the same cluster to be identical.
Specifically, we used DBSCAN Backlund et al. (2011) for clustering since it
does not require the number of clusters as an input parameter. For instance,
in the above example of $\tau$ with $m_{1}$ and $m_{3}$, if the semantic
vectors of $\tau(m_{1})$ and $\tau(m_{3})$ belong to the same cluster, then
the templates of $m_{1}$ and $m_{3}$ are considered the same. Then, we
followed the same heuristics described above to generate $R^{\prime}$ from $R$
based on the clustered templates.

##  5 Results

###  5.1 RQ1: Relationship between Log Parsing Accuracy and Anomaly Detection
Accuracy

All log parsing and anomaly detection techniques completed their executions on
both datasets. For all tuples $\langle\mathit{GA,PA,FTA,PR,RE,F1}\rangle$ we
collected, Figure 2 shows the relationship between
$\langle\mathit{GA,PA,FTA}\rangle$ (x-axis) and $F1$ (y-axis) for each dataset
in the form of a scatter plot. To additionally distinguish the results for
different anomaly detection techniques, we used different shapes and colors:
= DeepLog,  = LogAnomaly,  = LogRobust,  = CNN, and  = PLELog.

|
---|---
|
|

(a) HDFS-GA (b) HDFS-PA

(c) HDFS-FTA (d) Hadoop-GA

(e) Hadoop-PA (f) Hadoop-TA

Figure 2: Relationship between TI accuracy and AD accuracy [  = DeepLog,  =
LogAnomaly,  = LogRobust,  = CNN, and  = PLELog ]

Table 2 additionally shows the values of the Spearman’s rank correlation
coefficient $\sigma\langle\mathit{X,Y}\rangle$ between
$X=\langle\mathit{GA,PA,FTA}\rangle$ and $Y=F1$ for each pair of anomaly
detection technique and dataset. The value of
$\sigma\langle\mathit{X,Y}\rangle$, ranging between $-1$ and $+1$, indicates
the degree of the monotonic (not necessarily linear) relationship between $X$
and $Y$; when $\sigma\langle\mathit{X,Y}\rangle\geq+0.7$ (or
$\sigma\langle\mathit{X,Y}\rangle\leq-0.7$), there is a strong positive (or
negative) correlation between $X$ and $Y$ (i.e., $Y$ increases (or decreases)
when $X$ increases) Ali Abd Al-Hameed (2022). Note that, on the Hadoop
dataset, $\sigma\langle\mathit{X,Y}\rangle$ could not be computed for DeepLog,
LogAnomaly, LogRobust, and CNN since the F1 score does not vary with
$\langle\mathit{GA,PA,FTA}\rangle$, indicating no relationship between
$\langle\mathit{GA,PA,FTA}\rangle$ and F1.

Table 2: Spearman correlation coefficients between log parsing accuracy (GA, PA, and FTA) and anomaly detection accuracy (F1 score) | HDFS (reduced) | Hadoop
---|---|---
AD technique | GA | PA | FTA | GA | PA | FTA
DeepLog | $-0.166$ | $0.259$ | $0.198$ | - | - | -
LogAnomaly | $0.265$ | $0.142$ | $0.215$ | - | - | -
LogRobust | $0.360$ | $0.193$ | $0.165$ | - | - | -
CNN | $0.425$ | $0.575$ | $0.508$ | - | - | -
PLELog | $0.155$ | $0.611$ | $0.605$ | $-0.233$ | $-0.017$ | $-0.144$

Overall, both Figure 2 and Table 2 clearly show that there is no strong
correlation between $\langle\mathit{GA,PA,FTA}\rangle$ and $F1$ in all the
cases where
$\langle\mathit{GA,PA,FTA,PR,RE,F1}\rangle$ tuples were successfully
collected. For example, in Figure 2a, LogAnomaly ( ) achieved an F1 score
ranging between 0.2 and 0.5 regardless of the GA score. This means that
increasing log parsing accuracy does not necessarily increase (or decrease)
anomaly detection accuracy. This is counter-intuitive since anomaly detection
uses log parsing results, and having “better” log parsing results is expected
to increase anomaly detection accuracy. However, this happens because even
inaccurate log parsing results can lead to accurate anomaly detection results,
for reasons explained below.

To better understand the reason for the above results, let us consider a
normal log $l_{n}=\langle\mathit{m_{1}^{n},m_{2}^{n}},\dots\rangle$ and an
abnormal log $l_{a}=\langle\mathit{m_{1}^{a},m_{2}^{a}},\dots\rangle$, where
$m_{i}^{x}$ indicates the $i$-th log message in $l_{x}$ for $x\in\\{n,a\\}$.
Using oracle templates, we can group the log messages having the same template
and represent $l_{n}$ and $l_{a}$ as groups; specifically, let
$g_{\mathit{orc}}(l_{x})$ be a sequence of message group indices (i.e., the
$i$-th element of $g_{\mathit{orc}}(l_{x})$ is the message group index of
$m_{i}^{x}$). In this context, let us take two logs from the Hadoop dataset as
a concrete example where
$g_{\mathit{orc}}(l_{n})=\langle\mathit{1,2,3,4,5},\dots\rangle$ and
$g_{\mathit{orc}}(l_{a})=\langle\mathit{6,6,6,6,7},\dots\rangle$. When
templates generated by LogCluster are used to group messages instead of oracle
templates, the sequences of message group indices changes to
$g_{\mathit{LC}}(l_{n})=\langle\mathit{1,2,3,4,4},\dots\rangle$ and
$g_{\mathit{LC}}(l_{a})=\langle\mathit{8,9,10,11,12},\dots\rangle$, clearly
different from $g_{\mathit{orc}}(l_{n})$ and $g_{\mathit{orc}}(l_{a})$,
respectively; in particular, $m_{4}^{n}$ and $m_{5}^{n}$ are incorrectly
grouped together in $g_{\mathit{LC}}(l_{n})$ while $m_{1}^{a}$, $m_{2}^{a}$,
$m_{3}^{a}$, and $m_{4}^{a}$ are incorrectly separated in
$g_{\mathit{LC}}(l_{a})$. The incorrect groupings of LogCluster clearly reduce
GA (as well as PA and TA since incorrect groupings imply incorrect templates).
However, even the incorrect $g_{\mathit{LC}}(l_{n})$ and
$g_{\mathit{LC}}(l_{a})$ are still different enough from each other, so that
an anomaly detection technique can distinguish between normal and abnormal
patterns. This example not only shows why anomaly detection accuracy does not
necessarily depend on log parsing accuracy, but also demonstrates the
importance of distinguishability in log parsing results for anomaly detection;
we will further investigate this aspect in RQ2.

Before we conclude RQ1, one might be curious to know why DeepLog, LogAnomaly,
LogRobust, and CNN result in the same anomaly detection accuracy value on the
Hadoop dataset (as shown in Figures 2d-2f and Table 2). This happens because
(1) the test set of Hadoop contains only 11 logs (1 normal and 10 abnormal
logs, although the number of log messages is in the same order of magnitude as
HDFS; see Table 1 for more details) and (2) the four anomaly detection
techniques classified all the 11 logs in the test set as abnormal. We
speculate that PLELog shows different results from the other anomaly detection
techniques because PLELog uses a special deep learning model (i.e., an
attention-based GRU Cho et al. (2014)). Notice that, in all cases, the results
still corroborate that log parsing accuracy and anomaly detection accuracy do
not have any strong relationship.

The answer to RQ1 is that there is no relationship between log parsing
accuracy and anomaly detection accuracy; increasing log parsing accuracy does
not necessarily increase anomaly detection accuracy, regardless of the metric
(GA, PA, or TA) used for measuring log parsing accuracy.

###  5.2 RQ2: Log Parsing Distinguishability and Anomaly Detection Accuracy

DeepLog (F1) LogAnomaly (F1) LogRobust (F1) CNN (F1) PLELog (F1) Log Parser
$R_{\mathit{dst}}$ $R_{\mathit{ind}}$ $\Delta$ $R_{\mathit{dst}}$
$R_{\mathit{ind}}$ $\Delta$ $R_{\mathit{dst}}$ $R_{\mathit{ind}}$ $\Delta$
$R_{\mathit{dst}}$ $R_{\mathit{ind}}$ $\Delta$ $R_{\mathit{dst}}$
$R_{\mathit{ind}}$ $\Delta$ AEL $0.747$ $0.561$ $0.186$ $0.509$ $0.320$
$0.189$ $0.663$ $0.456$ $0.207$ $0.772$ $0.662$ $0.110$ $0.760$ $0.033$
$0.727$ Drain $0.714$ $0.523$ $0.191$ $0.499$ $0.400$ $0.099$ $0.703$ $0.454$
$0.250$ $0.757$ $0.682$ $0.075$ $0.796$ $0.286$ $0.510$ IPLoM $0.760$ $0.590$
$0.170$ $0.481$ $0.268$ $0.213$ $0.556$ $0.380$ $0.176$ $0.810$ $0.588$
$0.222$ $0.849$ $0.041$ $0.808$ LFA $0.803$ $0.693$ $0.110$ $0.606$ $0.378$
$0.228$ $0.355$ $0.299$ $0.056$ $0.755$ $0.548$ $0.207$ $0.100$ $0.000$
$0.100$ LenMa $0.808$ $0.625$ $0.184$ $0.484$ $0.285$ $0.199$ $0.659$ $0.436$
$0.223$ $0.814$ $0.607$ $0.207$ $0.681$ $0.271$ $0.411$ LogCluster $0.263$
$0.097$ $0.166$ $0.380$ $0.243$ $0.138$ $0.542$ $0.300$ $0.241$ $0.498$
$0.306$ $0.192$ $0.426$ $0.317$ $0.108$ LogMine $0.732$ $0.552$ $0.180$
$0.453$ $0.363$ $0.090$ $0.554$ $0.329$ $0.225$ $0.792$ $0.612$ $0.179$
$0.817$ $0.439$ $0.378$ Logram $0.544$ $0.420$ $0.124$ $0.836$ $0.625$ $0.212$
$0.330$ $0.163$ $0.167$ $0.100$ $0.000$ $0.100$ $0.100$ $0.000$ $0.100$ MoLFI
$0.794$ $0.630$ $0.164$ $0.427$ $0.282$ $0.144$ $0.565$ $0.319$ $0.246$
$0.781$ $0.621$ $0.160$ $0.172$ $0.109$ $0.063$ SHISO $0.778$ $0.629$ $0.149$
$0.544$ $0.238$ $0.306$ $0.679$ $0.446$ $0.233$ $0.796$ $0.589$ $0.207$
$0.839$ $0.341$ $0.498$ SLCT $0.743$ $0.570$ $0.173$ $0.268$ $0.160$ $0.108$
$0.394$ $0.244$ $0.150$ $0.743$ $0.607$ $0.136$ $0.725$ $0.534$ $0.191$ Spell
$0.765$ $0.598$ $0.167$ $0.289$ $0.176$ $0.113$ $0.401$ $0.241$ $0.160$
$0.805$ $0.616$ $0.189$ $0.665$ $0.304$ $0.361$ Average $0.704$ $0.541$
$0.164$ $0.481$ $0.312$ $0.170$ $0.533$ $0.339$ $0.195$ $0.702$ $0.537$
$0.165$ $0.578$ $0.223$ $0.355$

Table 3: Impact of the distinguishability log parsing results on anomaly
detection accuracy for the HDFS (reduced) dataset

DeepLog (F1) LogAnomaly (F1) LogRobust (F1) CNN (F1) PLELog (F1) Log Parser
$R_{\mathit{dst}}$ $R_{\mathit{ind}}$ $\Delta$ $R_{\mathit{dst}}$
$R_{\mathit{ind}}$ $\Delta$ $R_{\mathit{dst}}$ $R_{\mathit{ind}}$ $\Delta$
$R_{\mathit{dst}}$ $R_{\mathit{ind}}$ $\Delta$ $R_{\mathit{dst}}$
$R_{\mathit{ind}}$ $\Delta$ AEL $0.950$ $0.850$ $0.100$ $0.950$ $0.850$
$0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.780$ $0.510$
$0.280$ Drain $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$
$0.100$ $0.950$ $0.850$ $0.100$ $0.900$ $0.000$ $0.000$ IPLoM $0.950$ $0.850$
$0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$
$0.100$ $0.850$ $0.000$ $0.850$ LFA $0.950$ $0.850$ $0.100$ $0.950$ $0.850$
$0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.890$ $0.800$
$0.090$ LenMa $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$
$0.100$ $0.950$ $0.850$ $0.100$ $0.900$ $0.500$ $0.400$ LogCluster $0.950$
$0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$
$0.850$ $0.100$ $0.950$ $0.180$ $0.770$ LogMine $0.950$ $0.850$ $0.100$
$0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$
$0.850$ $0.530$ $0.320$ Logram $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$
$0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.840$ $0.800$ $0.050$ MoLFI
$0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$
$0.950$ $0.850$ $0.100$ $0.900$ $0.740$ $0.160$ SHISO $0.950$ $0.850$ $0.100$
$0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$
$0.900$ $0.000$ $0.900$ SLCT $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$
$0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.840$ $0.000$ $0.000$ Spell
$0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$
$0.950$ $0.850$ $0.100$ $0.700$ $0.190$ $0.510$ Average $0.950$ $0.850$
$0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$ $0.100$ $0.950$ $0.850$
$0.100$ $0.710$ $0.350$ $0.360$

Table 4: Impact of the distinguishability log parsing results on anomaly
detection accuracy for the Hadoop dataset

Table 3 shows the anomaly detection accuracy values (F1 scores) when different
log parsing techniques (rows) and anomaly detection techniques (columns) are
used together on the HDFS (reduced) dataset; under each of the anomaly
detection technique columns, sub-columns $R_{\mathit{dst}}$ and
$R_{\mathit{ind}}$ indicate the F1 scores for distinguishable and
indistinguishable log parsing results, respectively, and $\Delta$ indicates
the difference between $R_{\mathit{dst}}$ and $R_{\mathit{ind}}$. For example,
if we choose AEL for log parsing and DeepLog for anomaly detection, the F1
score decreases from $0.747$ to $0.561$ when $R_{\mathit{ind}}$ is used
instead of $R_{\mathit{dst}}$. The same structure applies to Table 4, which
shows the results on the Hadoop dataset. In Table 4, except for PLELog , the
values for all anomaly detection techniques are identical due to the reasons
explained in the last paragraph of Section 5.1.

In all cases, $\Delta$ is positive, ranging from $0.056$ (LFA-LogRobust on the
HDFS (reduced) dataset) to $0.9$ (SHISO-PLELog on the Hadoop dataset). This
means that the anomaly detection accuracy decreases up to 90 percentage points
(pp) when $R_{\mathit{ind}}$ is used instead of $R_{\mathit{dst}}$. To see if
the differences between $R_{\mathit{dst}}$ and $R_{\mathit{ind}}$ are
significant, we applied the non-parametric Wilcoxon signed rank test Wilcoxon
(1992) for paired samples to the F1 scores of $R_{\mathit{dst}}$ and
$R_{\mathit{ind}}$, for each of the five anomaly detection techniques and the
two datasets. The results show that, for all the anomaly detection techniques
and datasets, the differences between $R_{\mathit{dst}}$ and
$R_{\mathit{ind}}$ are significant ($p\text{-value}<0.005$) in terms of
anomaly detection accuracy.

Considering the definition of distinguishability for log parsing results, it
is intuitive that indistinguishable log parsing results lead to lower anomaly
detection accuracy. It may even be surprising that the accuracy decreases, in
the best case, by only $0.056$. This happens because an indistinguishable log
parsing result may only have a few logs that are indistinguishable in terms of
normal and abnormal behavior. Recall that we did not explicitly control the
number of indistinguishable logs since we aimed to minimize the difference
between distinguishable and indistinguishable versions of each log parsing
result as described in Section 4.5. Nevertheless, the results shown in Tables
3 and 4 are sufficient to confirm the strong impact of distinguishability in
log parsing results on anomaly detection accuracy.

The answer to RQ2 is that the impact of the distinguishability of log parsing
results on anomaly detection accuracy is significant for all anomaly detection
techniques.

###  5.3 Threats to Validity

The oracle templates used in log parsing accuracy are critical to the accuracy
values. For example, as noted by Khan et al. (2022), manually extracting
oracle templates by investigating log messages without accessing the
corresponding source code could result in biased, incorrect oracle templates.
This could be a significant threat to the validity of our results. To mitigate
this, we perused the source code (of the exact version that generated the
logs) for each software system and used the templates directly extracted from
the source code. Although this made us exclude a few log datasets whose source
code was unavailable, it was inevitable to ensure the validity of our results.

Individual log parsing and anomaly detection techniques have distinct hyper-
parameters, which might significantly affect the log parsing and anomaly
detection results. To mitigate this, we used the same hyper-parameter values
proposed by the authors, when available; otherwise, we ran several sanity
checks and used the values that resulted in the same results reported in the
corresponding papers.

Using a specific set of log datasets is a potential threat to external
validity. Though the datasets we have considered have been widely used in
existing literature Le and Zhang (2022); Chen et al. (2021) on log-based
anomaly detection, they may not capture all diverse characteristics of log
data. Further experiments with different datasets are required to improve the
generalizability of our results.

##  6 Findings and Implications

One of the most surprising results from our evaluation is that, using all the
existing log parsing accuracy metrics in the literature, none of them showed a
strong correlation with anomaly detection accuracy. In other words, more
accurate log parsing results are not necessarily better for anomaly detection
accuracy. This implies that log parsing accuracy is not a good indicator of
the quality of log parsing results for anomaly detection purposes. As
explained with an example in Section 5.1, this happens because inaccurate log
parsing results can still be useful for anomaly detection as long as normal
and abnormal logs are distinguishable. At the extreme, a log parsing result
$R_{50}$ with 50% accuracy could be better for anomaly detection than a log
parsing result $R_{100}$ with 100% accuracy if $R_{50}$ distinguishes normal
and abnormal logs while $R_{100}$ does not.

This surprising finding leads to an important practical implication: when used
for anomaly detection purposes, we can no longer choose a log parsing
technique based on accuracy. Instead, as shown in Section 5.2, the
distinguishability of log parsing results should be the main selection
criterion. For example, normal and abnormal logs are often used for training
anomaly detection models; then, candidate log parsing results should be
compared in terms of their capability to distinguish normal and abnormal logs.
If there are multiple techniques that can distinguish between normal and
abnormal logs, then the one with the lowest number of identified templates
would be preferred since reducing the number of templates would increase the
performance of anomaly detection by reducing dimensionality (i.e., the number
of features considered in machine learning models) Shin et al. (2021).

We want to note that our objective here is not to identify the “best” log
parsing and anomaly detection techniques. Nevertheless, through our
experiments, we found that there is no single best technique that
significantly outperforms the others in all cases. To develop better log
parsing techniques targeting anomaly detection, it would be of great help to
consider distinguishability, which was not the case in the development of
existing log parsing techniques.

##  7 Related Work

Table 5: Comparison with related empirical studies Category |  Le and Zhang (2022) |  Fu et al. (2023) |  Our work
---|---|---|---
Objective |  Investigate different factors that might affect anomaly detection accuracy |  Investigate the impact of log parsing techniques on anomaly detection accuracy |  Evaluate the impact of log parsing accuracy and the distinguishability of log parsing results on anomaly detection accuracy
Log parsing accuracy metrics |  N/A |  PA |  PA, GA, and TA
Oracle templates |  N/A |  Manually generated for 2K sample log messages |  Extracted from the corresponding source code
Logs used for measuring log parsing accuracy |  N/A |  Only a small fraction of logs actually used for anomaly detection |  All logs used for anomaly detection
Log parsing techniques |  Drain, Spell, IPLoM and AEL |  Drain, Spell, IPLoM, LFA, Logram, and LenMa |  Drain, Spell, IPLoM, AEL, LFA, Logram, LenMa, LogSig, LogCluster, LogMine, SHISO, MoLFI, and SLCT
Anomaly detection techniques |  DeepLog, LogRobust, LogAnomaly, PLELog, and CNN |  DeepLog, LogRobust, Principal Component Analysis (PCA), LogClustering, Logistic Regression (LR), and Decision Tree (DT) |  DeepLog, LogRobust, LogAnomaly, PLELog, and CNN
Distinguishability Shin et al. (2021) |  Not considered |  Not considered |  Considered

Although individual techniques for log parsing and anomaly detection have been
studied for a long time, systematic studies covering several techniques have
only recently begun to emerge. For example, the most comprehensive evaluation
studies on many log parsing techniques Zhu et al. (2019); Dai et al. (2020);
Khan et al. (2022) were conducted over the last four years. Similarly, the
relationship between log parsing and anomaly detection has received little
attention until very recently. Below, we summarize the recent studies related
to this topic.

Shin et al. (2021) presented the first theoretical study considering the
relationship between log parsing and anomaly detection. As described in
Section 2.4, they established the concept of ideal log parsing results for
anomaly detection. We adopted their theoretical foundation, especially the
notion of distinguishability in log parsing results, and empirically showed
that distinguishability is indeed essential for anomaly detection. To the best
of our knowledge, our work is the first empirical study showing the importance
of log parsing distinguishability for anomaly detection.

As explained in Section 3, Le and Zhang (2022) presented an empirical study on
factors that could affect anomaly detection accuracy. Although a part of their
study investigated the impact of log parsing on anomaly detection accuracy,
they investigated four log parsing techniques but did not assess log parsing
accuracy. As a result, they only showed that using different log parsing
techniques leads to different anomaly detection accuracy scores. In our study,
on the other hand, we explicitly measured log parsing accuracy, collected 160
pairs of log parsing accuracy and anomaly detection accuracy values using
different combinations of log parsing and anomaly detection techniques, and
showed that there is no strong correlation between log parsing accuracy and
anomaly detection accuracy based on Spearman’s correlation.

During the writing of this paper, Fu et al. (2023) also presented an empirical
study on the impact of log parsing on anomaly detection performance. Although
their motivation and research questions are close to ours, there are several
key differences. First, for measuring log parsing accuracy, they used the
manually generated, error-prone oracle templates Khan et al. (2022) provided
with the 2K log messages randomly sampled by Zhu et al. (2019). In other
words, only a very small fraction of the logs used for anomaly detection was
used to measure log parsing accuracy in their study. In our study, however,
the same logs used for anomaly detection are used to measure log parsing
accuracy, and the oracle templates are directly extracted from the
corresponding source code. Second, they considered only one log parsing
accuracy metric (GA), whereas we considered all three log parsing metrics (GA,
PA, and TA) since different metrics assess complementary aspects of log
parsing Khan et al. (2022). Third, log parsing distinguishability, which is an
essential factor that substantially affects anomaly detection accuracy (as
shown in our RQ2), is only considered in our study. Finally, they only
considered two deep-learning-based anomaly detection techniques (DeepLog and
LogRobust), and focused also on more traditional machine learning approaches
(such as Principal Component Analysis, clustering, logistic regression, and
decision trees). Such differences allow us to report new findings and provide
concrete recommendations, as summarized in Section 6.

Table 5 summarizes the key differences between the previous empirical studies
(i.e., Le and Zhang (2022), Fu et al. (2023)) and our work.

##  8 Conclusion and Future Work

In this paper, we reported on a comprehensive empirical study on the impact of
log parsing on anomaly detection accuracy, using 13 log parsing techniques and
five DL-based anomaly detection techniques on two publicly available log
datasets. Analyzing log parsing results for anomaly detection, surprisingly,
we did not find any significant relationship between log parsing accuracy and
anomaly detection accuracy, regardless of metric used for the former
(including GA, PA, and FTA). This implies that, as opposed to common research
practice to date, we can no longer choose a log parsing technique based on its
accuracy when used for anomaly detection. Instead, we experimentally confirmed
existing theoretical results showing that the distinguishability of log
parsing results plays an essential role in achieving accurate anomaly
detection. It is therefore highly recommended that in the future we rely on
such distinguishability to evaluate, compare, and select log parsing
techniques.

As part of future work, we plan to extend our study with more publicly
available datasets and anomaly detection techniques to increase the
generalizability of our results. We aim to include state-of-the-art few-shot
anomaly detection techniques Huang et al. (2022); Pang et al. (2021), which
require only a limited amount of training data and could be more effective in
practice. We also plan to provide a more granular analysis of the
distinguishability for log parsing results by defining a new metric that
assesses the degree of distinguishability.

## References

  * Ali Abd Al-Hameed [2022] Khawla Ali Abd Al-Hameed.  Spearman’s correlation coefficient in statistical analysis.  _International Journal of Nonlinear Analysis and Applications_ , 13(1):3249–3255, 2022.
  * Backlund et al. [2011] Henrik Backlund, Anders Hedblom, and Niklas Neijman.  A density-based spatial clustering of application with noise.  _Data Mining TNM033_ , pages 11–30, 2011.
  * Chen et al. [2021] Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael R Lyu.  Experience report: deep learning-based system log analysis for anomaly detection.  _arXiv preprint arXiv:2107.05908_ , 2021.
  * Cho et al. [2014] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder–decoder approaches.  In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_ , pages 103–111, Doha, Qatar, October 2014. Association for Computational Linguistics.  doi: 10.3115/v1/W14-4012.  URL https://aclanthology.org/W14-4012.
  * Dai et al. [2020] H. Dai, H. Li, C. S. Chen, W. Shang, and T. Chen.  Logram: Efficient log parsing using n-gram dictionaries.  _IEEE Transactions on Software Engineering (TSE)_ , 48:879 – 892, 2020.  doi: 10.1109/TSE.2020.3007554.
  * Du and Li [2016] Min Du and Feifei Li.  Spell: Streaming parsing of system event logs.  In _2016 IEEE 16th International Conference on Data Mining (ICDM)_ , pages 859–864, Los Alamitos, CA, USA, 2016. IEEE, IEEE.  doi: 10.1109/CNSM.2015.7367331.
  * Du et al. [2017] Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar.  Deeplog: Anomaly detection and diagnosis from system logs through deep learning.  In _2017 ACM Conference on Computer and Communications Security (SIGSAC)_ , CCS ’17, page 1285–1298, New York, NY, USA, 2017. Association for Computing Machinery.  ISBN 9781450349468.  doi: 10.1145/3133956.3134015.
  * Elyasov [2012] Alexander Elyasov.  Log-based testing.  In _2012 34th International Conference on Software Engineering (ICSE)_ , pages 1591–1594, Los Alamitos, CA, USA, 2012. IEEE, IEEE.
  * Fu et al. [2009] Qiang Fu, Jian-Guang Lou, Yi Wang, and Jiang Li.  Execution anomaly detection in distributed systems through unstructured log analysis.  In _2009 IEEE international conference on data mining (ICDM)_ , pages 149–158, Los Alamitos, CA, USA, 2009. IEEE, IEEE.  doi: 10.1109/ICDM.2009.60.
  * Fu et al. [2023] Ying Fu, Meng Yan, Zhou Xu, Xin Xia, Xiaohong Zhang, and Dan Yang.  An empirical study of the impact of log parsers on the performance of log-based anomaly detection.  _Empirical Software Engineering_ , 28(1):1–39, 2023.
  * Hamooni et al. [2016] Hossein Hamooni, Biplob Debnath, Jianwu Xu, Hui Zhang, Guofei Jiang, and Abdullah Mueen.  Logmine: Fast pattern recognition for log analytics.  In _25th ACM International on Conference on Information and Knowledge Management (CIKM)_ , pages 1573–1582, New York, NY, USA, 2016. Association for Computing Machinery.  doi: 10.1145/2983323.2983358.
  * He et al. [2017] Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R Lyu.  Drain: An online log parsing approach with fixed depth tree.  In _2017 IEEE International Conference on Web Services (ICWS)_ , pages 33–40, Los Alamitos, CA, USA, 2017. IEEE, IEEE.  doi: 10.1109/ICWS.2017.13.
  * He et al. [2020] Shilin He, Jieming Zhu, Pinjia He, and Michael R. Lyu.  Loghub: A large collection of system log datasets towards automated log analytics, 2020.  URL https://arxiv.org/pdf/2008.06448.pdf.
  * He et al. [2021] Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R. Lyu.  A survey on automated log analysis for reliability engineering.  _ACM Comput. Surv._ , 54(6), jul 2021.  ISSN 0360-0300.  doi: 10.1145/3460345.  URL https://doi.org/10.1145/3460345.
  * Huang et al. [2022] Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang.  Registration based few-shot anomaly detection.  In _European Conference on Computer Vision_ , pages 303–319, New York, NY, USA, 2022. Springer, Springer.
  * Jeong et al. [2020] Sooyong Jeong, Ajay Kumar Jha, Youngsul Shin, and Woo Jin Lee.  A log-based testing approach for detecting faults caused by incorrect assumptions about the environment.  _IEICE Transactions on Information and Systems_ , 103(1):170–173, 2020.
  * Jiang et al. [2008] Zhen Ming Jiang, Ahmed E Hassan, Parminder Flora, and Gilbert Hamann.  Abstracting execution logs to execution events for enterprise applications.  In _2008 The Eighth International Conference on Quality Software (QSIC)_ , pages 181–186, Los Alamitos, CA, USA, 2008. IEEE, IEEE.  doi: 10.1109/QSIC.2008.50.
  * Jurafsky and Martin [2019] Daniel Jurafsky and James H Martin.  Vector semantics and embeddings.  _Speech and language processing_ , pages 1–31, 2019.
  * Khan et al. [2022] Zanis Ali Khan, Donghwan Shin, Domenico Bianculli, and Lionel Briand.  Guidelines for assessing the accuracy of log message template identification techniques.  In _Proceedings of the 44th International Conference on Software Engineering (ICSE’22)_ , page 1095–1106, New York, NY, United States, 2022\. ACM, ACM.
  * Le and Zhang [2022] Van-Hoang Le and Hongyu Zhang.  Log-based anomaly detection with deep learning: how far are we?  In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_ , pages 1356–1367, New York, NY, USA, 2022. IEEE, ACM.
  * Lu et al. [2018] Siyang Lu, Xiang Wei, Yandong Li, and Liqiang Wang.  Detecting anomaly in big data system logs using convolutional neural network.  In _2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech)_ , pages 151–158, Los Alamitos, CA, USA, 2018. IEEE, IEEE.
  * Makanju et al. [2009] Adetokunbo AO Makanju, A Nur Zincir-Heywood, and Evangelos E Milios.  Clustering event logs using iterative partitioning.  In _15th ACM SIGKDD international conference on Knowledge discovery and data mining (SIGKDD)_ , pages 1255–1264, New York, NY, USA, 2009\. Association for Computing Machinery.  doi: 10.1145/1557019.1557154.
  * Meng et al. [2019] Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et al.  Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs.  In _IJCAI_ , volume 19, pages 4739–4745, New York, NY, USA, 2019\. ACM.
  * Messaoudi et al. [2018] Salma Messaoudi, Annibale Panichella, Domenico Bianculli, Lionel Briand, and Raimondas Sasnauskas.  A search-based approach for accurate identification of log message formats.  In _2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC)_ , pages 167–16710, New York, NY, USA, 2018. ACM, Association for Computing Machinery.  doi: 10.1145/3196321.3196340.
  * Mizutani [2013] Masayoshi Mizutani.  Incremental mining of system log format.  In _2013 IEEE International Conference on Services Computing (SCC)_ , pages 595–602, Los Alamitos, CA, USA, 2013. IEEE, IEEE.  doi: 10.1109/SCC.2013.73.
  * Nagappan and Vouk [2010] Meiyappan Nagappan and Mladen A Vouk.  Abstracting log lines to log event types for mining software system logs.  In _2010 7th IEEE Working Conference on Mining Software Repositories (MSR)_ , pages 114–117, Los Alamitos, CA, USA, 2010. IEEE, IEEE.  doi: 10.1109/MSR.2010.5463281.
  * Oliner and Stearley [2007] Adam Oliner and Jon Stearley.  What supercomputers say: A study of five system logs.  In _37th annual IEEE/IFIP international conference on dependable systems and networks (DSN’07)_ , pages 575–584, Edinburgh, UK, 2007. IEEE, IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07).
  * Pang et al. [2021] Guansong Pang, Choubo Ding, Chunhua Shen, and Anton van den Hengel.  Explainable deep few-shot anomaly detection with deviation networks.  _arXiv preprint arXiv:2108.00462_ , abs/2108.00462, 2021.
  * Shima [2016] Keiichi Shima.  Length matters: Clustering system log messages using length of words, 2016\.  URL https://arxiv.org/abs/1611.03213.
  * Shin et al. [2021] Donghwan Shin, Zanis Ali Khan, Domenico Bianculli, and Lionel Briand.  A theoretical framework for understanding the relationship between log parsing and anomaly detection.  In _International Conference on Runtime Verification_ , pages 277–287, Cham, 2021. Springer, Springer.
  * Tang et al. [2011] Liang Tang, Tao Li, and Chang-Shing Perng.  Logsig: Generating system events from raw textual logs.  In _20th ACM international conference on Information and knowledge management (CIKM)_ , pages 785–794, New York, NY, USA, 2011. ACM.  doi: 10.1145/2063576.2063690.
  * Vaarandi and Pihelgas [2015] R. Vaarandi and M. Pihelgas.  Logcluster - a data clustering and pattern mining algorithm for event logs.  In _2015 11th International Conference on Network and Service Management (CNSM)_ , pages 1–7, Los Alamitos, CA, USA, 2015. IEEE.  doi: 10.1109/CNSM.2015.7367331.
  * Vaarandi [2003] Risto Vaarandi.  A data clustering algorithm for mining patterns from event logs.  In _3rd IEEE Workshop on IP Operations & Management (IPOM)_, pages 119–126, Los Alamitos, CA, USA, 2003. IEEE, IEEE.  doi: 10.1109/IPOM.2003.1251233.
  * Wilcoxon [1992] Frank Wilcoxon.  Individual comparisons by ranking methods.  In _Breakthroughs in statistics_ , pages 196–202. Springer, New York, NY, USA, 1992.
  * Yang et al. [2021] Lin Yang, Junjie Chen, Zan Wang, Weijing Wang, Jiajun Jiang, Xuyuan Dong, and Wenbin Zhang.  Semi-supervised log-based anomaly detection via probabilistic label estimation.  In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ , pages 1448–1460, Madrid, Spain, 2021. IEEE, IEEE.
  * Zhang et al. [2019] Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, et al.  Robust log-based anomaly detection on unstable log data.  In _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , pages 807–817, Tallin, Estonia, 2019. The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE).
  * Zhu et al. [2019] Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu.  Tools and benchmarks for automated log parsing.  In _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_ , pages 121–130, Los Alamitos, CA, USA, 2019. IEEE, IEEE.  doi: 10.1109/ICSE-SEIP.2019.00021.