- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PulseLog, a supervised method for detecting anomalies in system logs. It uses log messages and runtime information to learn how system behavior changes over time. The model combines long-term patterns and short-term changes to identify unusual behavior. The authors test PulseLog on five public datasets and compare it with 15 baselines. It achieves the best or tied F1 scores, although the gains are small on most datasets. The paper also tests limited training data, cross-system transfer, and robustness to LLM-generated noise. The results are promising, but it remains unclear how well the method handles real-world log corruption.
Critical strengths
-
The paper addresses an important problem in log anomaly detection. It aims to handle noisy log messages while capturing changes in system behavior over time.
-
The method combines long-term patterns, short-term changes, and runtime context. This provides a useful way to model different types of information in system logs.
-
The experiments cover five public datasets and 15 baselines. The authors also test noise robustness, limited training data, and cross-system transfer. These additional experiments help assess the method under different settings.
Critical weaknesses
-
Log anomaly detection has been extensively studied, with existing methods already achieving strong results on standard benchmarks. The proposed method mainly combines established techniques, and its improvements over existing approaches are limited. It remains unclear what new challenge the method addresses or why the added complexity is necessary.
-
The noise formulation is not well justified. The authors introduce several types of synthetic corruption and predefined noise rates, but do not explain how these choices relate to noise observed in real systems. It is also unclear whether the proposed method models noise explicitly or simply learns features that are less sensitive to the generated perturbations.
-
The improvements over strong baselines are limited on most datasets. For example, the F1 gains are only 0.02 percentage points on HDFS v3 and 0.08 on BGL. These small differences make it difficult to justify the added model complexity. Several selected benchmarks are already close to saturation, with existing methods reaching nearly 99% or even 100% F1. These settings provide limited evidence that the proposed method can handle more difficult or realistic anomaly detection problems.
-
The paper claims to detect deviations in system-state transitions. However, the model is trained using standard binary classification loss, without directly validating the learned transitions. It remains unclear whether the model actually captures meaningful behavioral changes or simply learns discriminative features.
Questions for authors
-
What is the main methodological difference between PulseLog and existing log anomaly detection methods? Given the small performance gains on most benchmarks, what specific limitation does the proposed architecture address?
-
How were the synthetic noise types and corruption rates selected? Is there evidence that they reflect noise observed in real production systems, beyond format and label consistency checks?
-
How do the authors verify that the learned state transitions represent meaningful system behavior rather than features learned for binary classification?
-
Since several benchmarks are already close to saturation, what evidence supports the method's advantages in more challenging anomaly detection settings?
Thank you for your constructive feedback. We added production logging examples and noise tests on HDFS v1 and Thunderbird (Sec. 4.4), along with behavioral state and transition probing analysis (Sec. 4.5).
The distinction lies in the behavioral representation and how it evolves. PulseLog constructs persistent and transient evidence separately, then uses runtime context to condition state retention, evidence injection, and the local residual update. The same semantic event can therefore lead to different state changes under different runtime conditions. This addresses the entanglement of stable operation, benign observation changes, and short-lived fault evidence in a shared event representation. Sections 3.2–3.4 specify this observation-to-trajectory dependency. We use established neural operators, but organize them around conditional state evolution rather than treating runtime parameters only as appended features. The added structure is intended to preserve useful transient evidence while reducing sensitivity to unreliable local observations. To evaluate this benefit, we have extended the noise robustness experiments to HDFS v1 and Thunderbird. The results show clear performance advantages under noise conditions encountered in practice, supporting the architectural design beyond small gains on already saturated benchmarks.
We have expanded Sec. 4.4 and Appendix B.4 to document the production motivation and corruption protocol. LogRobust (FSE 2019) describes logging-statement evolution and processing noise, including evaluation on Microsoft’s online-service logs. AWS ALB documentation illustrates status-code pairs such as 200 200 for successful requests and 502 - for failures, as well as a target-processing time of -1 when a target fails to respond before timeout. Slack’s May 2020 incident report describes increased HTTP 503 errors triggering alerts. LogRobust motivates testing robustness to observation instability, while the AWS and Slack examples show that runtime fields can carry genuine fault evidence that must be preserved. Accordingly, Appendix B.4 specifies constrained perturbations to templates, tokens, parameters, and values, excluding edits that alter fault semantics. PulseLog learns context-conditioned behavioral representations from imperfect observations; it does not estimate an explicit probabilistic corruption process. Our robustness experiments evaluate whether this mechanism tolerates observation noise while preserving fault-relevant evidence.
The learned states are designed to summarize behavioral history by retaining persistent operational context and transient evidence, with state evolution conditioned on runtime context. To make this interpretation testable, we added a frozen-representation diagnostic protocol in Sec. 4.5/Fig. 2. It probes execution states from frozen hidden states and transitions from adjacent hidden-state pairs, using current-event features, a random encoder, and pooled history as baselines. An adjacent-event control receives the same two events to assess the transition information retained by the representation. The detection and robustness results demonstrate that this context-conditioned state design enables effective anomaly detection under imperfect observations. The learned states retain substantially richer behavioral information than the baseline representations, supporting the effectiveness of our context-conditioned state design.
Sec. 4.4/Table 6 now evaluates corruption on BGL, HDFS v1, and Thunderbird. At 5% noise, PulseLog exceeds the strongest baseline by 2.50 and 2.15 F1 points on HDFS v1 and Thunderbird, respectively. Table 4 separately varies training size from 5K to 20K windows with a fixed test set. PulseLog is best or tied best across budgets, reaching 97.74% F1 on Hadoop with 5K windows. Together with the ablations, these results demonstrate the architecture’s effectiveness under noise and limited training data.