acceptodds
Under review as a conference paper at ICLR 2027

DiagCoder: Training Language Models to Diagnose Multivariate Time-Series Anomalies via Executable Code

Abstract

Diagnosing anomalies in multivariate cloud telemetry requires identifying affected metrics and time intervals and explaining their abnormal behavior. Existing approaches often produce entity-level anomaly scores, make one-shot language-model judgments, or rely on predefined diagnostic operators, limiting their ability to investigate context-dependent anomalies. We propose DiagCoder, an agentic framework that uses executable code as its action space for multivariate time-series diagnosis. Guided by diagnostic perspectives rather than a fixed workflow, DiagCoder iteratively constructs and executes analyses tailored to temporal and cross-metric context, producing metric-level anomaly intervals with inspectable computational evidence. To learn effective diagnostic strategies, DiagCoder jointly optimizes analysis construction and diagnostic decision-making through end-to-end reinforcement learning, without requiring expert reasoning trajectories. Built on Qwen3.5-27B and trained only on industrial cloud telemetry, DiagCoder achieves the best overall performance across the industrial dataset and public benchmarks, outperforming existing anomaly-detection approaches and leading closed-source large language models within the same framework.

Primary area:
learning on time series and dynamical systems
Keywords:
Multivariate Time Series, Anomaly Detection, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QtLQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets metric-level anomaly diagnosis in cloud operations multivariate time series (locating affected metrics and time intervals and explaining anomalous behavior) and proposes an agent framework, DIAGCODER. Its core innovation lies in: using executable Python code as the action space, with a policy writing, revising, and executing analysis programs in a stateful sandbox, and iterating through execution feedback to converge to metric-level anomaly interval reports with "verifiable computational evidence." For training, it adopts end-to-end RL (GRPO), reducing reliance on annotations. Based on Qwen3.5-27B and trained only on the industrial dataset VMPD, it achieves the highest average F1 (0.430) on the three benchmarks VMPD/PSM/MSL, surpassing 12 detection baselines and 4 closed-source frontier LLMs (under the same framework).

Critical strengths
  1. The "code-as-action" design is novel: it breaks through the expressive limitations of fixed toolboxes and can construct context-specific custom analyses such as reference segments and derived series (the case in Figure 1 is persuasive).
  2. The end-to-end RL training is reasonable: it jointly optimizes analysis and decision-making with point-level F1 as the reward, requires no expert trajectories/preference annotations, and is engineering-scalable.
Critical weaknesses
  1. The evaluation metric is single: only point-level F1 (no interval-level metrics/AUC, etc.), lacking more comprehensive measures for "interval localization."
  2. The comparison with frontier LLMs is incomplete: inference cost, latency, and token consumption are not reported, which weakens the persuasiveness of "27B outperforms closed-source models."
  3. Stability needs further clarification: no statistical significance tests are provided, and the number of repetitions/variance is not reported.
Questions for authors
  1. Can you supplement interval-level metrics (e.g., VUS-PR, Affiliation-F, point-adjusted F1) to comprehensively evaluate localization quality?
  2. In the comparison with frontier LLMs, what are the respective token costs, inference latencies, and per-window costs? How cost-effective are they?
  3. Can you report variance across multiple runs or significance tests? Is the gap between F1 0.576 on VMPD and 0.523 for GPT-5.5 robust?
Reviewer_BXwwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiagCoder is an LLM agent for diagnosing anomalies in multivariate time series. Its actions are Python programs that run in a stateful sandbox, and a natural-language "skill" suggests structural, periodic, local and cross-metric perspectives instead of a fixed workflow. The agent writes and revises analyses and outputs metric-level anomaly intervals with short descriptions.

Critical strengths
  1. The paper has a clear controlled study of action space and RL: The tool-based and code-based agents share the backbone, guidance, budget, reward and output format and differ only in the action interface. They are compared before and after RL (Table 3) and across five backbones (Appendix C), which isolates a design question that matters beyond anomaly detection.

  2. Useful behavioral analysis: Splitting performance into candidate coverage, retention and rejection (Table 4) shows what RL changes in the agent. True-candidate retention goes from 0.766 to 0.876 and false-candidate rejection from 0.707 to 0.786, which says more than F1 alone.

Critical weaknesses
  1. The headline advantage comes from the proprietary in-distribution dataset: DiagCoder is trained only on VMPD, and its lead in average F1 is driven by VMPD (0.576 vs. 0.448 for LLMAD). On the two public, out-of-distribution datasets the picture is different. Its PSM F1 (0.390) is below TimeRadar (0.511), Time-RCD (0.468), AnomaMind (0.408), LLM-TSAD (0.402), Qwen3.7-Plus in the same framework (0.430), and its own pre-RL version (0.402). Averaged over PSM and MSL it gets about 0.357, below TimeRadar (about 0.410) and Time-RCD (about 0.371). So RL helps in domain but does not reliably transfer, and on PSM it hurts. VMPD is also small (115 test windows), unreleased, and the only dataset with metric-level labels. The claims of state-of-the-art results across benchmarks and of out-of-distribution transfer are not supported as written.
  2. The diagnosis and "inspectable evidence" are never evaluated: The title, abstract and introduction motivate the method by explanations and cross-metric interpretation, but the reward and every reported metric only score interval localization (point F1), as the limitations section admits. We do not know whether the reports are correct, whether they agree with the code that was actually run, or whether RL on F1 makes the explanations worse, since nothing rewards them. Without even a small expert or LLM-judge study of report quality, the paper evaluates an anomaly localizer rather than a diagnosis system, and as a localizer its public-benchmark results are mixed. Additional concerns: The cost per window is not reported. A multi-turn 27B agent with a sandbox container is far more expensive than a detector that scores a window in milliseconds, and this decides whether it is practical for telemetry from hundreds of VMs. PSM and MSL also have known labelling problems [2], and because labels there are OR-aggregated across metrics, they cannot test metric-level localization. Point-wise F1 also ignores event structure, and range-aware metrics such as VUS [5] or the affiliation metrics [4] would make the comparison with detectors fairer. Finally, the training recipe builds on recent work on RL with code tools for LLMs, such as ReTool [1] and ToRL [3], which is not discussed.

References:

[1] ReTool: Reinforcement Learning for Strategic Tool Use in LLMs.

[2] Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress.

[3] ToRL: Scaling Tool-Integrated RL.

[4] Local Evaluation of Time Series Anomaly Detection Algorithms.

[5] Volume Under the Surface: A New Accuracy Evaluation Measure for Time-Series Anomaly Detection.

Questions for authors
  1. Could the authors evaluate on a public benchmark with metric-level labels, e.g. SMD with its interpretation labels or ASD from InterFusion? This would test the metric-level claim outside VMPD. (Clarification; this could change my rating.)

  2. Why does RL lower PSM F1 (0.402 to 0.390, with recall dropping from 0.519 to 0.419)? Is the policy fitting VMPD's anomaly rate (3.29%) rather than PSM's (11.07%)?

  3. Could the authors assess report quality on a sample of VMPD windows, checking both whether the descriptions are correct and whether they match the executed computations, before and after RL?

  4. What are the average number of turns, tokens and wall-clock time per window for DiagCoder, compared with a deep detector and with GPT-5.5 in the same framework?

  5. How does the F1 reward handle windows with no labelled anomalies, where F1 is undefined? Could that choice push the policy toward over- or under-reporting?

1 comment
AuthorsOpenReview ↗

Follow-Up on Our Rebuttal

Dear Reviewer,

 

Thank you again for your time and thoughtful feedback. We know this is a busy period. As the discussion draws to a close, we would appreciate it if you could take a look at our responses and the revised manuscript.

Please let us know if any of your concerns remain or if there is anything else we can clarify. We would be happy to discuss any further questions with you.

 

Best regards,
The Authors

Reviewer_uUknOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors present DiagCoder, a fine-tuned LLM designed for anomaly detection in multivariate time series that leverages executable code. The approach starts with Qwen and uses RL to improve its ability to write custom code (in this case Python) to flag anomalies on a given input, producing anomaly intervals for each metric along with a written report explaining them. The authors demonstrate its effectiveness compared to existing anomaly detection methods (deep learning detectors, time series foundation models, and other LLM-based approaches) as well as to "out of the box" LLMs that write the code within the same framework.

Critical strengths

The general idea seems natural and common now in deep learning. You take an off the shelf neural network and fine tune it for a very specific task, in this case writing code to evaluate anomalies. The authors demonstrate that there is room to improve a general purpose model when focusing on this specific task, at least on data similar to what it was trained on.

Critical weaknesses

The scalability of this approach isn't discussed. If the goal is to flag anomalies almost continuously (say on a daily or hourly basis across a large system), then the cost of this approach seems prohibitive. Now the paper positions itself as "what to do after you find the anomaly", and there is existing work that does something similar, where an alert or lightweight statistical detector flags the anomaly and the expensive model is saved for the flagged cases (e.g., Chen et al., 2024; Chandrayan et al., 2025; Chen, Zaharia & Zou, 2023), or where the LLM is used offline to build detectors that run without LLM calls (Gu et al., 2025). However, the evaluation of the method is entirely based on F1 scores, so there is a mismatch here. Other papers use human ratings or "LLMs as judges" to assess the quality of the generated explanations (e.g., Ahmed et al., 2023; Lan et al., 2025; Zheng et al., 2023), but this isn't done at all in this paper, not even a check of whether the reports are consistent with the code outputs. So the empirical work doesn't really match the stated aims of the paper.

I also find the empirical gains less convincing than the headline suggests. The model is trained on a private dataset, and that is where nearly all of its advantage comes from. On the two public datasets (PSM and MSL), DiagCoder averages 0.357 F1, behind existing methods such as TimeRadar (0.410) and even an untrained Gemini in the same framework (0.364), and RL actually lowers its score on PSM. Three benchmarks, one of them private, is also thin compared to most anomaly detection papers, and the only public datasets used lack metric-level labels, so the paper's metric-level focus is only tested on the private data.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Follow-Up on Our Rebuttal

Dear Reviewer,

 

Thank you again for your time and thoughtful feedback. We know this is a busy period. As the discussion draws to a close, we would appreciate it if you could take a look at our responses and the revised manuscript.

Please let us know if any of your concerns remain or if there is anything else we can clarify. We would be happy to discuss any further questions with you.

 

Best regards,
The Authors

AuthorsOpenReview ↗

A Gentle Follow-Up to Reviewer uUkn

Dear Reviewer uUkn,

 

With a few hours remaining in the discussion period, we wanted to follow up briefly. If you have a chance to revisit our response, we would appreciate hearing whether the revisions address your concerns about the diagnostic reports and the evidence on public datasets. Even a brief note about any remaining question would give us a chance to clarify it before the discussion closes.

Thank you again for your thoughtful review and for the time you have given our paper.

 

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.