- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper targets metric-level anomaly diagnosis in cloud operations multivariate time series (locating affected metrics and time intervals and explaining anomalous behavior) and proposes an agent framework, DIAGCODER. Its core innovation lies in: using executable Python code as the action space, with a policy writing, revising, and executing analysis programs in a stateful sandbox, and iterating through execution feedback to converge to metric-level anomaly interval reports with "verifiable computational evidence." For training, it adopts end-to-end RL (GRPO), reducing reliance on annotations. Based on Qwen3.5-27B and trained only on the industrial dataset VMPD, it achieves the highest average F1 (0.430) on the three benchmarks VMPD/PSM/MSL, surpassing 12 detection baselines and 4 closed-source frontier LLMs (under the same framework).
Critical strengths
- The "code-as-action" design is novel: it breaks through the expressive limitations of fixed toolboxes and can construct context-specific custom analyses such as reference segments and derived series (the case in Figure 1 is persuasive).
- The end-to-end RL training is reasonable: it jointly optimizes analysis and decision-making with point-level F1 as the reward, requires no expert trajectories/preference annotations, and is engineering-scalable.
Critical weaknesses
- The evaluation metric is single: only point-level F1 (no interval-level metrics/AUC, etc.), lacking more comprehensive measures for "interval localization."
- The comparison with frontier LLMs is incomplete: inference cost, latency, and token consumption are not reported, which weakens the persuasiveness of "27B outperforms closed-source models."
- Stability needs further clarification: no statistical significance tests are provided, and the number of repetitions/variance is not reported.
Questions for authors
- Can you supplement interval-level metrics (e.g., VUS-PR, Affiliation-F, point-adjusted F1) to comprehensively evaluate localization quality?
- In the comparison with frontier LLMs, what are the respective token costs, inference latencies, and per-window costs? How cost-effective are they?
- Can you report variance across multiple runs or significance tests? Is the gap between F1 0.576 on VMPD and 0.523 for GPT-5.5 robust?
Follow-Up on Our Rebuttal
Dear Reviewer,
Thank you again for your time and thoughtful feedback. We know this is a busy period. As the discussion draws to a close, we would appreciate it if you could take a look at our responses and the revised manuscript.
Please let us know if any of your concerns remain or if there is anything else we can clarify. We would be happy to discuss any further questions with you.
Best regards,
The Authors