- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper uses an LLM agent as a detector developer. For each class of related series, the agent explores labeled training windows in a sandbox and writes a standalone Python detector, revising it over several submissions based on validation metrics and labeled validation examples. The best candidate is then run on held-out series from the same class with no LLM calls. On TSB-AD, detectors written by all seven developer models beat the strongest established method in VUS-PR. The paper also proposes C-DAPO, a variant of DAPO that gives each submission extra credit for later improvement, and uses it to train Qwen3.5-35B-A3B as a developer.
Critical strengths
The strongest contribution is the developer paradigm. The LLM is used only to build the detector. The final Python program is inspectable and runs without LLM calls. Complete held-out series remain outside development, which matters because the agent receives repeated validation feedback. The ablations show that diagnostic feedback helps and that unrestricted validation access leads to overfitting. The study also tests seven developer models and releases the code, costs, and checkpoints.
Critical weaknesses
-
The agent uses labels at every stage. It sees training labels and labeled validation examples. Thresholds are set with validation labels, and the final program is chosen with label-based scores. The paper does not show that the baselines get the same labels or tuning. So the comparison does not show how much the LLM-written code itself helps. In this setup, each new class also needs labeled anomalies and a new agent run.
-
The paper does not compare with Argos, the most similar earlier method. Like AgentAD, Argos uses labeled data and LLM feedback to build detectors that run without an LLM. So it is unclear whether AgentAD does better than Argos under the same conditions. The paper also does not compare with recent LLM- or VLM-based detectors, such as LLMAD and VLM4TS, which it discusses.
-
Each detector is built and tested on series from the same class, which share the same dataset, domain, and channel layout. So the results show that detectors work on similar series, not on new datasets. TSB-AD is also public, so the developer models may already know it.
-
Each class uses one data split and one agent run. Some gains over the best baseline are small. The RL comparison uses only seven classes, with no repeated seeds and no confidence intervals. The Segment Holdout results are not a matched comparison, but they are bolded with the main results.
-
C-DAPO uses about 84% more tokens and 48% more submissions than DAPO, even though both have the same limits. The paper does not show how much of the gain comes from the method and how much from this extra compute. The claim that it matches larger models is also limited. It beats DeepSeek-V4.1-Flash, is about equal to Qwen3.8-27B, and is worse than both GLM models.
Questions for authors
- How do the baselines perform with per-class model selection and tuning on the same data?
- How does AgentAD compare with Argos, and with recent LLM- or VLM-based detectors?
- Do the detectors work on classes or datasets they were not built for?
- How much do the results change across data splits and repeated runs?
- How does C-DAPO compare with DAPO at the same token budget?
You were right about Segment Holdout. The baselines were scored on the final portion specified by the benchmark, while AgentAD was scored on its own internal-test windows. We recomputed Table 1 with the same train, validation and test windows for every method. All seven language-model developers still exceed the strongest established method on mean VUS-PR. Revisions are marked in blue.
W1/Q1. Supervision and tuning. “Uses labels at every stage” conflates what the evaluator knows with what the agent sees. Labels mark individual timestamps. The agent sees training labels and selected labeled validation examples. Under Series Holdout it gets only aggregate internal-test scores from development recordings. Entirely separate recordings are held out. Under Segment Holdout the internal-test windows are the final holdout, so even their aggregate scores stay hidden during development. The delivered detector receives signals and scores each timestamp. Only the evaluator uses holdout labels.
A class groups recordings from the same dataset and domain with the same channel layout. Most neural baselines are trained or fine-tuned on normal segments identified from that class’s development labels. DADA and Time-RCD use released checkpoints. TSPulse is reported both zero-shot and fine-tuned. Classical methods fit each series without labels. All methods calibrate thresholds on validation data. Threshold choice cannot improve VUS-PR (Section 2.3, Table R6). AgentAD can also revise code after inspecting failures. We used published baseline settings and did no additional per-class search over methods or hyperparameters, so Table 1 cannot answer what that search would achieve.
Each new class requires labeled development data and one run. Among developers with public API prices, the highest average cost was under two dollars per class. The detector then runs on a CPU without model calls (Section 4.5).
W3/Q3. Generalization and pretraining exposure. Series Holdout tests a detector on new recordings from its class. A program built around ECG beat intervals has little use on server traffic. The developer can examine traffic data and write a suitable program. We tested that ability on seven classes from five datasets and five domains, excluded throughout supervised fine-tuning and RL. C-DAPO improved on DAPO on average (Section 5.3). Transfer of a frozen detector across datasets remains untested.
Even knowing that a TSB-AD recording contains an anomaly leaves its start and end unknown. In Series Holdout the developer never sees the holdout recording. It gets a short domain description and inspects development arrays through Python under standardized filenames. We supply neither dataset nor original file names. We saw no indication of recalled labels in the trajectories we inspected. We cannot audit pretraining corpora, but a public benchmark alone is no evidence that a developer recovered held-out annotations.
W2/Q2. Argos and LLM/VLM baselines. Argos also dispenses with the LLM after development. Its main evaluation combines binary rules with a trained base detector and uses point-adjusted F1. AgentAD writes a complete program that returns continuous scores. Their published results cannot settle which system wins under common conditions. We would be willing to add a camera-ready comparison with matched data access, budgets and evaluation. LLMAD calls a model for anomaly decisions and VLM4TS for interval refinement on new inputs. A comparison would need to count those ongoing inference costs. Section 2.2 explains our focus on standalone detectors.
W4/Q4. Robustness. One split and one run per developer per class cannot estimate seed or split variation. All seven developers beat the strongest baseline on mean Series Holdout VUS-PR. Comparing the best developer and best baseline within each class gives 26 wins in 33 classes (Appendix B.5). Small margins may vary. We will add repeated splits and runs in the camera-ready version, including for the seven-class RL comparison.
W5/Q5. Compute and model size. C-DAPO used more tokens under the same limits. Table R3 reconstructs the retained detector and cumulative cost after each attempt from saved runs, carrying both forward for runs that stopped early. At equal attempt count, DAPO led on holdout VUS-PR at the fifth and C-DAPO at the seventh as it continued revising. At similar mean token use, C-DAPO after five attempts was slightly ahead of DAPO’s final result. An identical token cutoff remains untested.
On Series Holdout VUS-PR the Qwen3.5-35B-A3B developer trained with C-DAPO beats DeepSeek-V4.1-Flash, is close to Qwen3.8-27B, and trails both GLM models (Table 3). We narrowed the claim accordingly.