acceptodds
Under review as a conference paper at ICLR 2027

FAROS: Characterizing Coding-Agent Search for Executable Time-Series Anomaly Detectors

Abstract

Coding agents can construct executable anomaly detectors from labeled time-series data. However, their open-ended search can vary substantially across runs in both length and resulting detector quality, making systematic evaluation difficult. We introduce FAROS (Framework for Agentic Run Observation and Synthesis), an evaluation framework that records search states during ongoing coding-agent search and applies a detector synthesis procedure to obtain detector readouts from selected states. This enables comparison of detector quality at selected points during search. It also separates variability across independent search states from variability in repeated synthesis from a fixed state. Using FAROS, we study detector construction from few labeled anomaly events and how reasoning setting and search effort affect detector quality and repeatability. Our experiments yield three findings. First, coding agents can construct detectors competitive with specialized pretrained time-series anomaly detectors from few labeled anomaly events. Second, at larger common output-token targets, stronger reasoning achieves higher mean detection performance, while longer lower-effort search does not recover the same performance. Third, we observe contrasting changes in search-state and synthesis variability across reasoning settings and the two tested label budgets as search proceeds. At the largest output target, search-state variability remains the larger component across reasoning settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
time-series anomaly detection, LLM agents, coding agents, agentic search, executable detector synthesis, few-shot learning, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_LvYmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FAROS is an evaluation framework for coding agents that construct executable time-series anomaly detectors from labeled support data. It records conversation histories and workspaces during ongoing agent search, then invokes a separate Finalizer to synthesize detectors from selected checkpoints. Repeated searches and repeated synthesis from each checkpoint support a decomposition of performance variability into between-search-state and within-state synthesis components.

Critical strengths
  1. The studied problem is important.
  2. The paper is easy to follow.
Critical weaknesses
  1. Code is not provided.
  2. The motivation is not clear. What will code agent do in such anomaly detection setting? And why the running process of code agent is important for this task, which motivates this paper. And why different amount of searches should be considered?
  3. Lacks of deeply explained fail case. Failure analysis is important.
  4. No novelty method is proposed.
Questions for authors

Please see the weakesses above.

2 comments

We thank the reviewer for the constructive comments and suggestions, which have helped us improve the manuscript. We appreciate the reviewer's recognition that the problem is important and that the paper is easy to follow.

  1. Code is not provided.

We will release the code for the FAROS framework before publication. Until then, the appendices describe what is needed to reproduce the experiments, namely the datasets and support construction (Appendix A), the prompts and recorded-state semantics (Appendix B), the metrics (Appendix C), the variance estimators (Appendix D), and the baseline configurations (Appendix E).

  1. The motivation is not clear. What will code agent do in such anomaly detection setting? And why the running process of code agent is important for this task, which motivates this paper. And why different amount of searches should be considered?

The coding agent builds a detector program from a few labeled series. As described in Sec. 3.2, it reads the support values and labels, analyzes them by executing Python code, and implements and tests candidate detectors with a provided scorer on the support set. It repeats this cycle with no fixed number of iterations.

The running process matters because the search is open-ended. Given the same task, runs differ in their number of interactions, generated output, and sequence of candidate detectors (Sec. 1). Evaluating only the final detector therefore compares runs after different amounts of search and along different construction paths. This hides two things that the process reveals, namely how detector quality changes as the search proceeds, and how much runs differ from each other at the same stage of construction. FAROS records states during the search so that both can be measured at matched points.

Different amounts of search must be considered because the conclusions depend on them. At a common 30k output-token target with K=16, low reasoning uses 176.8 Explorer responses on average and xhigh only 36.7, yet low reaches lower AP (0.777 vs. 0.924, Appendix G). Along search, the AP gap between xhigh and medium reasoning grows from 0.018 at 5k to 0.095 at 30k (K=1), and xhigh at 10k already exceeds low at 30k (K=16, Sec. 5.2). A single endpoint shows only one point of these trajectories, so it cannot tell how much a setting searched from how well it searched.

  1. Lacks of deeply explained fail case. Failure analysis is important.

We added a failure analysis of detector synthesis (new Appendix I) and point to the existing analysis of search failures (Appendix F). In 1.8% of the recorded states, detectors synthesized from the same state differ in AP by at least 0.5, and these states account for 66.7% of the synthesis variance. In these states, another synthesis from the same state is accurate, so the failure arises when the state is converted into a program. The low-scoring programs contain small implementation errors. For example, one evaluates a polynomial baseline at the edge of its window instead of the centre (AP 0.993, 0.038, and 0.993 for three syntheses), and another reverses a convolution kernel so that a slope score changes sign (0.938, 0.938, and 0.047). In 73% of these states the faulty program also scores poorly on the support data, so a support check would detect it. Failures can also arise in search. With the Qwen Explorer, AP declines at larger label budgets, and support AP declines with it, so the decline is already present in what the search found (Appendix F).

  1. No novelty method is proposed.

FAROS is an evaluation framework, and its novelty lies in how it measures open-ended coding-agent search rather than in a new detector. It compares searches at matched progress and separates variability across independently reached states from variability in converting a state into a program. This yields findings that endpoint evaluation cannot provide. At 30k, search-state variability accounts for 91–98% of AP variance at K=1 (Figure 5), and reasoning strength and search length are not interchangeable on synthetic data (Sec. 5.2). Evaluation methodology is an established contribution type (e.g., Agarwal et al., "Deep Reinforcement Learning at the Edge of the Statistical Precipice," NeurIPS 2021; Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", ICLR 2024).

We hope these additions address the reviewer's concerns and would be grateful if the reviewer could reconsider the score.

Reviewer_LvYmOpenReview ↗

Reviewer Feedback

Thanks the authors for the response. But the code is still unavailable, I can not verify the proposed framework. I decide to maintain my score.

Reviewer_h74JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FAROS, a framework for evaluating coding-agent search in time-series anomaly detection. FAROS records intermediate search states and synthesizes detectors from them, enabling comparisons at matched search progress and separating variability arising from independent searches and detector synthesis. Experiments investigate few-label detector construction, reasoning effort, and search repeatability.

Critical strengths

The paper addresses an interesting problem in evaluating coding-agent search beyond final outcomes.

The experiments include diverse baselines, reasoning settings, and repeated runs.

The variance decomposition provides useful insights into the reliability of agent-generated detectors.

Critical weaknesses
  • Several core ideas, including intermediate progress evaluation [1], snapshot-based branching [2], and repeated completion sampling [3], have been explored in prior work. AgentLens [4] also investigates process-level evaluation of coding-agent trajectories. FAROS does not sufficiently establish its methodological novelty beyond combining these existing techniques.

  • The paper does not sufficiently establish that Finalizer-generated detector performance faithfully reflects the actual quality of intermediate Explorer states. Since the Finalizer uses a fixed high-reasoning model, it may independently develop an effective detector even when the Explorer has made little meaningful progress, thereby overestimating the quality of that state. Conversely, an Explorer may have discovered useful detection strategies that the Finalizer fails to utilize, leading to underestimation. The endpoint sanity check does not rule out these possibilities, and the lack of ablations across Finalizer reasoning strengths or model backbones leaves the reliability of the intermediate-state measurements unclear.

  • The variance estimates rely on only 3–5 Explorer runs and three synthesis replicas per state. Without confidence intervals or uncertainty analysis for the variance components, it is difficult to assess the reliability of the reported trends.

  • FAROS requires additional high-reasoning synthesis calls, but their computational overhead is not comprehensively evaluated. The paper also lacks sufficient justification for using three synthesis replicas and does not establish whether simpler, less expensive evaluation strategies could provide comparable insights.

[1] Ma et al. (2024). AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. Advances in Neural Information Processing Systems (NeurIPS 2024), Datasets and Benchmarks Track. https://arxiv.org/abs/2401.13178

[2] Nakash et al. (2026). Efficient Agent Evaluation via Diversity-Guided User Simulation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), Industry Track, pp. 1627–1648. https://arxiv.org/abs/2604.21480

[3] Wang et al. (2024). Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision. Findings of EMNLP 2024, pp. 7309–7319. https://arxiv.org/abs/2402.02658

[4] Sahoo et al. (2026). AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation. arXiv:2605.12925. https://arxiv.org/abs/2605.12925

Questions for authors

see weaknesses

1 comment

We thank the reviewer for the constructive comments and suggestions, which have helped us improve the manuscript.

Several core ideas, including intermediate progress evaluation [1], snapshot-based branching [2], and repeated completion sampling [3], have been explored in prior work. AgentLens [4] also investigates process-level evaluation of coding-agent trajectories. FAROS does not sufficiently establish its methodological novelty beyond combining these existing techniques.

FAROS shares individual primitives with these works, and its contribution lies in what they are combined to measure. None of the previous works converts an intermediate state of an ongoing search into an executable artifact evaluated on held-out data, aligns independent searches at a common progress measure, or separates variability across independently reached states from variability in converting a state into an artifact. The last requires several independent searches and several syntheses per state, which endpoint evaluation cannot provide (Figure 5). AgentBoard [1] and MiPS [3] were not cited in the original version, so we added them to the related work and described how they differ from FAROS. DIVERT [2] and AgentLens [4] are already discussed there.

The paper does not sufficiently establish that Finalizer-generated detector performance faithfully reflects the actual quality of intermediate Explorer states. Since the Finalizer uses a fixed high-reasoning model, it may independently develop an effective detector even when the Explorer has made little meaningful progress, thereby overestimating the quality of that state. Conversely, an Explorer may have discovered useful detection strategies that the Finalizer fails to utilize, leading to underestimation. The endpoint sanity check does not rule out these possibilities, and the lack of ablations across Finalizer reasoning strengths or model backbones leaves the reliability of the intermediate-state measurements unclear.

New Table 3 (Sec. 5.2) tests both failure modes by reading out the same recorded Explorer states with four Finalizers, Luna with medium, high, and xhigh reasoning and Qwen3.8-27B. For overestimation, states recorded before any Explorer response contain only the task instructions, and from them all four Finalizers reach only 0.221–0.303 AP, against 0.761–0.790 (K=1) and 0.914–0.937 (K=16) at 30k. The Finalizer thus cannot build strong detectors without the Explorer's progress. If limited Finalizer capability caused underestimation, weaker Finalizers would read out lower AP, but AP differs by at most 0.003 at 30k across the three Luna reasoning settings.

The variance estimates rely on only 3–5 Explorer runs and three synthesis replicas per state. Without confidence intervals or uncertainty analysis for the variance components, it is difficult to assess the reliability of the reported trends.

We repeated the variance analysis with ten syntheses per state at all six output targets and added nested bootstrap intervals (revised Figure 5, Appendix D). At 30k, search-state variability accounts for 91–98% of AP variance at K=1 and 71–80% at K=16, and it is larger than synthesis variability in at least 97% of bootstrap replicates. Along search, two trends hold beyond uncertainty. With one labeled event, synthesis variance decreases under all three reasoning settings, and with sixteen labeled events, search-state variance decreases under high and xhigh reasoning.

FAROS requires additional high-reasoning synthesis calls, but their computational overhead is not comprehensively evaluated. The paper also lacks sufficient justification for using three synthesis replicas and does not establish whether simpler, less expensive evaluation strategies could provide comparable insights.

Each synthesis is a single model response with at most three attempts (Secs. 3.3 and 4.2), whereas reaching the 30k target takes 30.7–209.2 Explorer responses on average (Appendix G). Three syntheses per state therefore add few calls relative to the search they evaluate. The estimator in Appendix D needs at least two syntheses per state to separate the two components, and three already support the main conclusion. With three syntheses, search-state variability accounts for 87.8–98.9% of AP variance at K=1 and 60.1–85.1% at K=16 at 30k (Appendix H), and ten syntheses give the same conclusion (90.8–98.2% and 70.5–80.2%, Sec. 5.2). Ten syntheses are needed mainly to resolve changes along search with confidence intervals. As a cheaper alternative, a medium-reasoning Finalizer reads out within 0.003 AP of the xhigh Finalizer at 30k (Table 3).

We hope these additions address the reviewer's concerns and would be grateful if the reviewer could reconsider the score.

Reviewer_oerGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FAROS evaluates coding agents as they construct time-series anomaly detectors. By repeatedly generating detectors from saved intermediate states, it separates variability across searches from variability during final code synthesis.

Critical strengths
  • Studying intermediate states provides useful insights beyond final detector performance.

  • The experiments include multiple runs, label budgets, real-world data, and sensitivity checks for invalid outputs.

Critical weaknesses
  • All experiments use the same high-reasoning Finalizer. It remains unclear how much the findings depend on this choice and how faithfully it reflects the Explorer’s progress.

  • The variance trends need stronger uncertainty estimates. Each state has only three syntheses, and a few states account for most synthesis variance.

  • The reasoning vs search-length findings are mainly established on synthetic data with one Explorer model. Their broader applicability remains uncertain.

Questions for authors
  1. How sensitive are the findings to the Finalizer model and its reasoning setting?

  2. Do the variance trends remain stable with more synthesis repetitions?

  3. Do similar search trajectories emerge on real-world data and with another Explorer model?

1 comment

We thank the reviewer for the constructive comments and suggestions, which have helped us improve the manuscript. We appreciate the reviewer's recognition that studying intermediate states provides useful insights beyond final detector performance.

All experiments use the same high-reasoning Finalizer. It remains unclear how much the findings depend on this choice and how faithfully it reflects the Explorer's progress. Q1. How sensitive are the findings to the Finalizer model and its reasoning setting?

The readout reflects the recorded state and changes little with the Finalizer. New Table 3 (Sec. 5.2) reads out the same recorded Explorer states with four Finalizers, Luna with medium, high, and xhigh reasoning and Qwen3.8-27B. States recorded before any Explorer response contain only the task instructions, and from them all four Finalizers reach only 0.221–0.303 AP, against 0.761–0.790 (K=1) and 0.914–0.937 (K=16) at 30k. The Finalizer therefore does not build strong detectors without the Explorer's progress. Across the three Luna reasoning settings, AP differs by at most 0.003 at 30k and 0.026 at 5k. The Qwen Finalizer, which cannot see Luna's encrypted reasoning, reads out 0.020–0.029 lower AP at 30k and preserves the gain from 5k to 30k.

The variance trends need stronger uncertainty estimates. Each state has only three syntheses, and a few states account for most synthesis variance. Q2. Do the variance trends remain stable with more synthesis repetitions?

The level of the decomposition is stable. We repeated the analysis with ten syntheses per state at all six output targets and added nested bootstrap intervals (revised Figure 5, Appendix D). At 30k, search-state variability accounts for 91–98% of AP variance at K=1 and 71–80% at K=16, and it is larger than synthesis variability in at least 97% of bootstrap replicates.. Along search, two trends hold beyond uncertainty. With one labeled event, synthesis variance decreases under all three reasoning settings. With sixteen labeled events, search-state variance decreases under high and xhigh reasoning. Regarding concentration, new Appendix I shows that 1.8% of states account for 66.7% of synthesis variance. In these states, another synthesis from the same state is accurate, and the low-scoring programs contain small implementation errors such as a reversed convolution kernel (AP 0.938, 0.938, and 0.047 for three syntheses).

The reasoning vs search-length findings are mainly established on synthetic data with one Explorer model. Their broader applicability remains uncertain. Q3. Do similar search trajectories emerge on real-world data and with another Explorer model?

Quality improves along search on real data and with another Explorer, but the advantage of stronger reasoning holds only on synthetic data. On TSB-AD-U (Table 2b), we ran Luna searches with medium, high, and xhigh reasoning on all 27 cases. AP increases from 5k to 30k under every setting (by 0.024–0.047), whereas the three settings differ by at most 0.010 at 30k. For the Qwen3.8-27B Explorer, we compared searches with thinking enabled and disabled at matched output (Table 2a). AP increases along search in both cases (from 0.553 to 0.662 with thinking and from 0.505 to 0.588 without, K=1). With one labeled event, disabling thinking lowers AP at every target, by 0.048–0.084. With sixteen labeled events, it lowers AP only at 5k.

We hope these additions address the reviewer's concerns and would be grateful if the reviewer could reconsider the score.

Reviewer_StdDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FAROS studies how coding agents construct time series anomaly detectors from a small amount of tag data. It records the intermediate search states, aligns them according to the search progress, and uses a fixed Finalizer to synthesize multiple detectors from each state. This allows comparisons across search phases and separates search state variability from synthetic variability. The experiment examined the detection quality, reasoning settings and the evolution of these variance components.

Critical strengths

S1. The motivation is reasonable. Endpoint evaluation may ignore the differences in the search process, and repeated composition provides a practical way to control variability.

S2. The overall writing of the paper is fluent and not difficult to read.

S3. The results are verified by experiments based on synthetic data.

Critical weaknesses

W1. The contribution in terms of results feels relatively limited. The results in Table 1 do not provide a good proof of the effectiveness of the method, especially when compared to ordinary agents, which are also much worse than tsfm. Then I felt that there were no practical and interesting examples or visualizations of anomaly detection to illustrate the insights generated by the method, which were not impressive enough. Most experiments were conducted on synthetic datasets, except for one actual dataset, which weakened the generalizability and value of the research.

W2. The ablation experiment is not sufficient. The method used components such as state recording, progress alignment, repeated program synthesis, and standard nested variance estimator, but did not perform well in ablation. The value of the search process was not well reflected

W3.I feel that there are relatively few innovative points in the methods, and the text and images read with a feeling of excessive defensive writing generated by AI, without highlighting its highlights

Questions for authors

See weaknesses.

1 comment

We thank the reviewer for the constructive comments and suggestions, which have helped us improve the manuscript.

W1. The contribution in terms of results feels relatively limited. The results in Table 1 do not provide a good proof of the effectiveness of the method, especially when compared to ordinary agents, which are also much worse than tsfm. Then I felt that there were no practical and interesting examples or visualizations of anomaly detection to illustrate the insights generated by the method, which were not impressive enough. Most experiments were conducted on synthetic datasets, except for one actual dataset, which weakened the generalizability and value of the research.

Table 1 is not meant to show that FAROS outperforms the compared methods. FAROS is an evaluation framework, not a detection method, and Table 1 asks whether coding agents can construct competitive detectors from a few labeled events at all (RQ1). They can. On the synthetic dataset, the standard agent reaches 0.792–0.944 AP against 0.846–0.943 for fine-tuned Time-RCD and is the best method at K=4 and K=8. On TSB-AD-U, FAROS (0.551) and the standard agent (0.544) exceed every baseline, including Time-RCD (0.511).

The paper contains concrete examples of what the agent builds and where it fails. Appendix J gives the complete code of detectors built from a single labeled event. Each uses a principle suited to its anomaly type, for example a linear predictor fitted in both time directions for frequency anomalies (AP 0.883) and a lag-difference score for flat-trend anomalies (AP 0.909). New Appendix I shows the opposite case. Large disagreements between detectors synthesized from the same state come from small bugs, for example a reversed convolution kernel that flips a slope score (AP 0.938, 0.938, and 0.047).

We also added real-world results along search. On TSB-AD-U (Table 2b), AP increases from 5k to 30k output tokens under every reasoning setting (by 0.024–0.047). We further added results with a second Explorer, Qwen3.8-27B. We found that disabling thinking lowers AP at every target with one labeled event, by 0.048–0.084 (Table 2a).

W2. The ablation experiment is not sufficient. The method used components such as state recording, progress alignment, repeated program synthesis, and standard nested variance estimator, but did not perform well in ablation. The value of the search process was not well reflected

We read this comment as asking what observing the search process adds. Because FAROS is a measurement method, removing one of its components does not yield a weaker detector but leaves a quantity unobserved. We therefore show what each component reveals.

Recording and reading out intermediate states shows how detection quality develops during a search. States recorded before any search yield only 0.22–0.30 AP, against 0.76–0.79 (K=1) and 0.91–0.94 (K=16) after 30k output tokens, for four different Finalizers (Table 3). Aligning searches at matched output shows that reasoning strength and search length are not interchangeable. The AP gap between reasoning setting of xhigh and medium grows from 0.018 at 5k to 0.095 at 30k (K=1), and xhigh at 10k exceeds low at 30k (0.823 vs. 0.777, K=16). A comparison at a single endpoint observes only one point of these trajectories. Repeated synthesis with the nested estimator shows where run-to-run variability arises. With ten syntheses per state (Figure 5, Appendix D), search-state variability accounts for 91–98% (K=1) and 71–80% (K=16) of AP variance at 30k. Along search, synthesis variance decreases with one labeled event under every setting, while differences across searches remain. With sixteen labeled events, search-state variance under high reasoning falls from 0.026 at 5k to 0.007 at 30k, so independent searches converge. Additional search thus stabilizes different stages of construction depending on the label budget.

W3.I feel that there are relatively few innovative points in the methods, and the text and images read with a feeling of excessive defensive writing generated by AI, without highlighting its highlights

The novelty of FAROS lies in proposing a new evaluation framework. It compares open-ended searches at matched progress and separates search-state from synthesis variability, which yields the practical findings above that endpoint evaluation cannot provide. Evaluation methodology is an established contribution type (e.g., Agarwal et al., “Deep Reinforcement Learning at the Edge of the Statistical Precipice", NeurIPS 2021; Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, ICLR 2024).

We hope these additions address the reviewer's concerns and would be grateful if the reviewer could reconsider the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.