Comparing Trace Similarity Metrics Across Logs and Evaluation Measures
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Standard
Comparing Trace Similarity Metrics Across Logs and Evaluation Measures. / Back, Christoffer Olling; Simonsen, Jakob Grue.
Advanced Information Systems Engineering: 35th International Conference, CAiSE 2023, Zaragoza, Spain, June 12–16, 2023, Proceedings. ed. / Marta Indulska; Iris Reinhartz-Berger; Carlos Cetina; Oscar Pastor. Springer, 2023. p. 226-242 (Lecture Notes in Computer Science, Vol. 13901)).Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - Comparing Trace Similarity Metrics Across Logs and Evaluation Measures
AU - Back, Christoffer Olling
AU - Simonsen, Jakob Grue
N1 - Supported by Innovation Fund Denmark as part of DIREC initiative
PY - 2023
Y1 - 2023
N2 - Trace similarity is a prerequisite for several process mining tasks, e.g. identifying process variants and anomalies. Many similarity metrics have been presented in the literature, but the similarity metric itself is seldom subject to controlled evaluation. Instead, they are usually demonstrated in conjunction with downstream tasks, e.g. process model discovery, and often evaluated qualitatively or with limited comparison. In this paper, we isolate similarity metrics from downstream tasks and compare them wrt. evaluation measures adapted from metric learning and clustering literature. We present a comparison of 18 similarity metrics across 4 evaluation measures and 12 event logs. Friedman and Nemenyi tests for statistical significance show that certain similarity metrics consistently outperform on some evaluation measures, but their mean rank varies across evaluation measures. One similarity metric based on a weighted eventually-follows relation does stand out as consistently outperforming, and the simplest n-gram similarity metrics also perform well. Our results demonstrate that choice of evaluation measures will determine the contours of the metric that are revealed. This study may be harnessed as a baseline for benchmarking future work on trace similarity, and describes tools for quantitative evaluation that we hope will inspire empirical rigor in future work.
AB - Trace similarity is a prerequisite for several process mining tasks, e.g. identifying process variants and anomalies. Many similarity metrics have been presented in the literature, but the similarity metric itself is seldom subject to controlled evaluation. Instead, they are usually demonstrated in conjunction with downstream tasks, e.g. process model discovery, and often evaluated qualitatively or with limited comparison. In this paper, we isolate similarity metrics from downstream tasks and compare them wrt. evaluation measures adapted from metric learning and clustering literature. We present a comparison of 18 similarity metrics across 4 evaluation measures and 12 event logs. Friedman and Nemenyi tests for statistical significance show that certain similarity metrics consistently outperform on some evaluation measures, but their mean rank varies across evaluation measures. One similarity metric based on a weighted eventually-follows relation does stand out as consistently outperforming, and the simplest n-gram similarity metrics also perform well. Our results demonstrate that choice of evaluation measures will determine the contours of the metric that are revealed. This study may be harnessed as a baseline for benchmarking future work on trace similarity, and describes tools for quantitative evaluation that we hope will inspire empirical rigor in future work.
KW - Empirical Evaluation
KW - Process Mining
KW - Similarity Metric
U2 - 10.1007/978-3-031-34560-9_14
DO - 10.1007/978-3-031-34560-9_14
M3 - Article in proceedings
AN - SCOPUS:85164023434
SN - 978-3-031-34559-3
T3 - Lecture Notes in Computer Science
SP - 226
EP - 242
BT - Advanced Information Systems Engineering
A2 - Indulska, Marta
A2 - Reinhartz-Berger, Iris
A2 - Cetina, Carlos
A2 - Pastor, Oscar
PB - Springer
T2 - 35th International Conference on Advanced Information Systems Engineering, CAiSE 2023
Y2 - 12 June 2023 through 16 June 2023
ER -
ID: 387382847