- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FAROS is an evaluation framework for coding agents that construct executable time-series anomaly detectors from labeled support data. It records conversation histories and workspaces during ongoing agent search, then invokes a separate Finalizer to synthesize detectors from selected checkpoints. Repeated searches and repeated synthesis from each checkpoint support a decomposition of performance variability into between-search-state and within-state synthesis components.
Critical strengths
- The studied problem is important.
- The paper is easy to follow.
Critical weaknesses
- Code is not provided.
- The motivation is not clear. What will code agent do in such anomaly detection setting? And why the running process of code agent is important for this task, which motivates this paper. And why different amount of searches should be considered?
- Lacks of deeply explained fail case. Failure analysis is important.
- No novelty method is proposed.
Questions for authors
Please see the weakesses above.
2 comments
Reviewer Feedback
Thanks the authors for the response. But the code is still unavailable, I can not verify the proposed framework. I decide to maintain my score.
We thank the reviewer for the constructive comments and suggestions, which have helped us improve the manuscript. We appreciate the reviewer's recognition that the problem is important and that the paper is easy to follow.
We will release the code for the FAROS framework before publication. Until then, the appendices describe what is needed to reproduce the experiments, namely the datasets and support construction (Appendix A), the prompts and recorded-state semantics (Appendix B), the metrics (Appendix C), the variance estimators (Appendix D), and the baseline configurations (Appendix E).
The coding agent builds a detector program from a few labeled series. As described in Sec. 3.2, it reads the support values and labels, analyzes them by executing Python code, and implements and tests candidate detectors with a provided scorer on the support set. It repeats this cycle with no fixed number of iterations.
The running process matters because the search is open-ended. Given the same task, runs differ in their number of interactions, generated output, and sequence of candidate detectors (Sec. 1). Evaluating only the final detector therefore compares runs after different amounts of search and along different construction paths. This hides two things that the process reveals, namely how detector quality changes as the search proceeds, and how much runs differ from each other at the same stage of construction. FAROS records states during the search so that both can be measured at matched points.
Different amounts of search must be considered because the conclusions depend on them. At a common 30k output-token target with K=16, low reasoning uses 176.8 Explorer responses on average and xhigh only 36.7, yet low reaches lower AP (0.777 vs. 0.924, Appendix G). Along search, the AP gap between xhigh and medium reasoning grows from 0.018 at 5k to 0.095 at 30k (K=1), and xhigh at 10k already exceeds low at 30k (K=16, Sec. 5.2). A single endpoint shows only one point of these trajectories, so it cannot tell how much a setting searched from how well it searched.
We added a failure analysis of detector synthesis (new Appendix I) and point to the existing analysis of search failures (Appendix F). In 1.8% of the recorded states, detectors synthesized from the same state differ in AP by at least 0.5, and these states account for 66.7% of the synthesis variance. In these states, another synthesis from the same state is accurate, so the failure arises when the state is converted into a program. The low-scoring programs contain small implementation errors. For example, one evaluates a polynomial baseline at the edge of its window instead of the centre (AP 0.993, 0.038, and 0.993 for three syntheses), and another reverses a convolution kernel so that a slope score changes sign (0.938, 0.938, and 0.047). In 73% of these states the faulty program also scores poorly on the support data, so a support check would detect it. Failures can also arise in search. With the Qwen Explorer, AP declines at larger label budgets, and support AP declines with it, so the decline is already present in what the search found (Appendix F).
FAROS is an evaluation framework, and its novelty lies in how it measures open-ended coding-agent search rather than in a new detector. It compares searches at matched progress and separates variability across independently reached states from variability in converting a state into a program. This yields findings that endpoint evaluation cannot provide. At 30k, search-state variability accounts for 91–98% of AP variance at K=1 (Figure 5), and reasoning strength and search length are not interchangeable on synthetic data (Sec. 5.2). Evaluation methodology is an established contribution type (e.g., Agarwal et al., "Deep Reinforcement Learning at the Edge of the Statistical Precipice," NeurIPS 2021; Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", ICLR 2024).
We hope these additions address the reviewer's concerns and would be grateful if the reviewer could reconsider the score.