acceptodds
Under review as a conference paper at ICLR 2027

Beyond Isolated Tasks: Benchmarking and Automating Full-Cycle Time Series Analysis

Abstract

Real-world time series analysis is a complex, multi-stage process that typically requires human experts to explore heterogeneous records, model temporal patterns, and support downstream decisions. Yet, existing time series models and benchmarks largely focus on carefully curated, isolated tasks, leaving dependencies between stages largely unexplored. We argue that reliable downstream performance requires both accurate upstream results and the ability to effectively leverage them. To address the scarcity of data for evaluating the cross-stage dependencies in full-cycle time series analysis (FC-TSA), we collect real hydrologic records across 15 gauged catchments and introduce the first benchmark for evaluating Full-Cycle Analysis Capabilities for Time Series: FacTS-Bench. It follows the complete real-world hydrologic analysis process, containing over 1.5K scored instances across four core capabilities: (1) Environment Exploration prepares analysis-ready series from unfamiliar workspaces for subsequent stages; (2) Situation Perception identifies events, assesses anomalies, and establishes current conditions; (3) Forecast Orchestration builds forecasts to inform; (4) Warning Decisions produces evidence-backed warning and dispatch plans. Crucially, FacTS-Bench traces how intermediate results affect downstream outputs, capturing cross-stage dependencies overlooked by existing task-specific benchmarks. To further understand cross-stage dependencies, we present AutoTSer, an autonomous analyst for FC-TSA. It coordinates a Planner, Worker, and Auditor through two mechanisms: (1) Evidence-Grounded Collaboration checks intermediate results against source data, while (2) Boundary-Constrained Progression checks whether those results meet the requirements of the next analytical stage. AutoTSer further learns from alternative operations and their downstream effects, using offline reinforcement learning to train a compact controller. This enables (3) Lightweight Self-Evolution without fine-tuning the underlying language models. Full-cycle evaluations demonstrate improved downstream decision quality through more reliable use of intermediate results. Further analysis shows that agent self-evolution helps retain beneficial operations and avoid harmful ones.

Primary area:
learning on time series and dynamical systems
Keywords:
Multimodal Time Series, Time Series Models, Time Series

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ux2BOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a benchmark for hydrologic analysis across the complete lifecycle rather than just time series forecasting. It contains 4 capabilities - environment exploration, situation perception, forecast orchestration and warning decisions. They also propose a cross-capability analyzer AutoTSER.

Critical strengths

The benchmark covers multiple tasks within time series analysis rather than isolated tasks which makes it useful for the hydrologic analysis task.

Critical weaknesses

The paper has a very narrow domain and this is misrepresented in the problem formulation and the title. IT is not a general benchmark to evaluate full-cycle analysis capabilities but is a very specific benchmark for hydrologic domain. This severely limits the contributions of the paper - would have been better if this was clear in the title and abstract. Further, many sentences oversell the contributions (example, one of the contributions listed is “New Models” whereas they are proposing and RL approach for evaluation).

Questions for authors

None

11 comments
AuthorsOpenReview ↗

Response to Reviewer ux2B [Part 1]

We would like to sincerely thank Reviewer ux2B for the review and comments.

W1: The paper has a very narrow domain and this is misrepresented in the problem formulation and the title. It is not a general benchmark to evaluate full-cycle analysis capabilities but is a very specific benchmark for hydrologic domain. This severely limits the contributions of the paper - would have been better if this was clear in the title and abstract.

We sincerely appreciate your feedback on the scope and presentation of our work. We have revised the title to “Beyond Isolated Tasks: Benchmarking and Automating Full-Cycle Time Series Analysis in Hydrology.” The abstract now explicitly introduces FacTS-Bench as a benchmark for full-cycle time series analysis in the hydrologic domain. We make this scope consistent throughout the paper and add a Limitations section. Below, we explain the choice of hydrology, the value of the operational records, the reusable construction procedure, and the additional evaluation of AutoTSer on TimeSage-MT and TSAIA.

(1) Hydrology Connects Observations to Decisions We choose hydrology because flood operations connect all stages required for full-cycle evaluation. Rainfall and river measurements establish current conditions. Forecasts describe their evolution. Operating rules and engineering states constrain warning and dispatch actions. Historical dispatch records document decisions made by operational authorities. These aligned records allow us to evaluate intermediate analyses and their downstream use within the same real event.

(2) Operational Data and a Reusable Construction Procedure The operational records are a substantive data contribution. They connect time series to actual gate, pump, and reservoir operations under documented constraints. Experts use these records and the conditions at each forecast origin to establish necessary, acceptable, and unsupported reference actions. This turns operational experience into an executable benchmark for assessing evidence-backed decisions. Our five-stage procedure in Section 4.2 and Appendix B‾\underline{\text{Section 4.2 and Appendix B}} covers data collection, processing and annotation, task construction, reference construction, and quality control. It specifies how to assemble source workspaces, intermediate facts, observed forecast targets, and reviewed decision references. Teams with time series and operational decision records can follow this procedure to construct their own full-cycle benchmarks, supplying their domain-specific variables, operating rules, and action references.

AuthorsOpenReview ↗

Response to Reviewer ux2B [Part 2]

W1 (continued):The paper has a very narrow domain and this is misrepresented in the problem formulation and the title. It is not a general benchmark to evaluate full-cycle analysis capabilities but is a very specific benchmark for hydrologic domain. This severely limits the contributions of the paper - would have been better if this was clear in the title and abstract.

(3) Revised Scope and Added Limitations The revised title and abstract explicitly identify hydrology as the benchmark domain. The revisions below clarify the data contribution, benchmark-construction procedure, and evaluation scope.`

LocationSummary of Revisions
TitleBeyond Isolated Tasks: Benchmarking and Automating Full-Cycle Time Series Analysis in Hydrology
Abstract: benchmark introductionWe introduce FacTS-Bench, a benchmark for evaluating full-cycle time series analysis in the hydrologic domain. Built from observations and operational records across 15 gauged catchments, it contains 1,590 scored instances spanning four capabilities and eight subtasks. Its evaluation follows intermediate results through successive stages to forecasts and decisions.
Introduction: motivation for the benchmarkFull-cycle evaluation requires observations, forecasts, and decision references for the same evolving event. We choose hydrology because flood operations connect these elements in a documented analytical workflow. Rainfall and river measurements establish current conditions, forecasts describe their evolution, and operating rules constrain warning and dispatch actions. Historical dispatch records link these conditions to decisions made by operational authorities. This allows us to evaluate intermediate analyses and trace their effects on downstream decisions.
Contribution 1: replace “New Setting and Testbed”New Setting and Full-Cycle Benchmark. We formulate full-cycle time series analysis and introduce FacTS-Bench as an executable benchmark in hydrology. It connects eight subtasks through explicit intermediate outputs and evaluates their downstream use. We provide a reproducible construction procedure that converts observations, operating rules, and expert-reviewed decision records into linked tasks and scoring references.
Section 3: opening paragraph before the formal definitionWe formulate full-cycle time series analysis as a sequence of stages in which later stages use intermediate results produced by earlier ones. Each task specifies a request, source records, available resources, and constraints on information use and actions. FacTS-Bench instantiates this setting in hydrology through environment exploration, situation perception, forecast orchestration, and warning decisions.
Limitations: new section before ConclusionScaling full-cycle time series analysis benchmarks is constrained by access to historical decision records and the expert effort needed to construct action references. The controller selects from predefined operations, so its gains depend on both trace coverage and the quality of available alternatives. Future work will expand full-cycle datasets to include more events and application domains, and use additional paired traces to improve evidence checks and plan revisions.
AuthorsOpenReview ↗

Response to Reviewer ux2B [Part 3]

W1 (continued):The paper has a very narrow domain and this is misrepresented in the problem formulation and the title. It is not a general benchmark to evaluate full-cycle analysis capabilities but is a very specific benchmark for hydrologic domain. This severely limits the contributions of the paper - would have been better if this was clear in the title and abstract.

(4) Scope of FacTS-Bench (1,590 instances, 8 subtasks, 15 catchments)

Within the hydrologic domain, the benchmark is broad in problem types and data sources:

CapabilitySubtasksInstances
C1 Environment ExplorationWorkspace Grounding258
C2 Situation PerceptionFlood Event Segmentation; Observation Anomaly Adjudication; Preforecast Data and State Analysis375
C3 Forecast OrchestrationCandidate Forecast Orchestration; Forecast Model Development524
C4 Warning DecisionsWarning and Dispatch Planning; Decision Dependency Verification433
Total8 subtasks1,590
  • Eight problem types: retrieval, segmentation, classification, state description, forecast combination, model building, planning and verification.
  • Two countries: seven regulated catchments in China and eight CAMELS-LUX catchments in Luxembourg, with 129 stations and different rainfall regimes, terrain and degrees of regulation.
  • Complete chains: 8 real flood events from five years, each run 3 times per framework (72 chains), where every stage consumes the agent's own earlier output.

Full-Cycle Results (8 events × 3 runs, 24 chains per framework)

CompleteC1 Fidelity ↑C2 Acc. ↑C3 nMAE ↓C4 F1 ↑Tokens (M) ↓
Claude Code12/240.6780.6409.4500.4093.790
OpenCode18/240.7990.7354.5220.6654.065
AutoTSer24/240.8080.9024.7030.7950.265

Completed Runs per Event (of 3)

E1E2E3E4E5E6E7E8Total
Claude Code1101223212/24
OpenCode3111333318/24
AutoTSer3333333324/24

Same-Event Comparison (matched completed runs)

AutoTSer vs.Common EventsC4 F1 ↑ (AutoTSer / other)Events with Higher F1Events with Lower Forecast nMAE
OpenCode80.797 / 0.6688 of 85 of 8
Claude Code70.790 / 0.5216 of 74 of 7

Findings:

(1) AutoTSer completes all three runs of all eight events (24/24), against 18/24 and 12/24.

(2) Its decisions are better on the same events: decision F1 is higher in 8 of 8 events against OpenCode and in 6 of 7 against Claude Code.

(3) It uses about 7% of the tokens of the other two agents (0.27M vs. 3.8–4.1M per chain).

Reviewer_kEDUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that real-world time-series problems are more nuanced than traditionally assumed. Standard approaches focus on tasks such as forecasting or classification, whereas practical applications often require a deeper understanding of the data.

The authors introduce three main contributions:

a) A new benchmark setting based on hydrological records with expert annotations; b) AUTOTSER, a multi-agent pipeline with a trained controller that guides execution; c) An analysis of cross-stage dependencies.

AUTOTSER is evaluated on FACTS-BENCH and compared with general-purpose LLMs and time-series agents. The paper also includes detailed performance breakdowns and ablation studies.

Critical strengths

The paper makes a compelling observation that real-world time-series problems often extend beyond isolated tasks, and that multi-step analysis of messy data can introduce and propagate errors.

The fact-grounding procedure implemented in AUTOTSER directly addresses this challenge, and the experimental results support its effectiveness.

Critical weaknesses

The main weakness is the lack of diversity in the types of time-series problems evaluated. Although the benchmark is a meaningful contribution, it covers only four flood scenarios, a very limited range of cases. More importantly, all data come from the hydrology domain. This narrow scope substantially limits the evidence for the method’s broader real-world applicability.

Regarding novelty, the combined use of specialized planning, implementation, and verification agents has already been explored in the literature. The main novel component appears to be the controller, yet its effectiveness is insufficiently evaluated.

The evidence presented also raises a substantive concern: a non-negative development gain does not necessarily demonstrate that the controller has learned beneficial strategies. It may simply have learned to avoid unnecessary or harmful development operations. A simpler decision-making mechanism could potentially achieve similar performance.

Questions for authors

Could the authors provide additional experiments, or at least case studies, involving time-series problems from other real-world domains?

In Table 2, why does the C3 Orchestration nMAE remain nearly identical across all backbones, while the other metrics vary substantially?

In Table 4, the results vary considerably across levels. Given the very small sample size, this raises concerns about the statistical significance of the findings. Could the authors provide the raw results alongside an appropriate statistical significance analysis?

14 comments
AuthorsOpenReview ↗

Response to Reviewer kEDU [Part 1]

We would like to sincerely thank Reviewer kEDU for the insightful review and comments.

W1: The main weakness is the lack of diversity in the types of time-series problems evaluated. Although the benchmark is a meaningful contribution, it covers only four flood scenarios, a very limited range of cases. More importantly, all data come from the hydrology domain. This narrow scope substantially limits the evidence for the method’s broader real-world applicability.

Q1: Could the authors provide additional experiments, or at least case studies, involving time-series problems from other real-world domains?

We sincerely appreciate this suggestion. The four flood scenarios in the original Table 3 are complete event-level chains evaluated from raw records through operational decisions. FacTS-Bench as a whole contains 1,590 task instances across 8 time series subtasks, 15 catchments, and 129 stations. To fully address your concern, we evaluated AutoTSer on two public benchmarks from other real-world domains, with the controller off and without any hydrologic skill:

  • TimeSage-MT [1] (80 conversations across four levels): AutoTSer achieves 72.09 mean analytical quality, compared with 68.51 for Claude Code and 65.11 for OpenCode. It ranks first at every level and in seven of the eight application domains.
  • TSAIA [2] (multi-step tasks, 4 domains, 335 tasks): AutoTSer reaches 74.0% success, against 70.1% for the official agent.

We also add two case studies.

(1) Why the Hydrologic Domain

Hydrology connects time series analysis to operational decisions with observable outcomes. A flood event provides multi-source measurements, subsequent observations, operating rules, and historical dispatch records. These records support evaluation throughout a connected workflow:

  • Data preparation: rainfall, water levels, and hydraulic-structure states must be extracted and aligned across source files.
  • Situation analysis: the observations and operational context support event segmentation, anomaly identification, and state assessment.
  • Forecasting: post-origin observations provide the reference trajectories.
  • Decisions: operating rules and historical dispatch records support expert-reviewed warning and dispatch references.
  • Real cross-stage dependencies: a wrong rainfall parse or a wrong state changes the forecast and then the dispatch plan.

The same event links these stages, allowing us to evaluate both intermediate results and the propagation of upstream errors into forecasts and decisions. The TimeSage-MT and TSAIA experiments then evaluate the agent on time series problems from other application domains. Moreover, the construction procedure of FacTS-Bench (Section 4.2 and Appendix B) only needs these kinds of record, so groups that hold operating records in other domains can apply it to build full-cycle samples of their own.

(2) FacTS-Bench: 1,590 Instances, 8 Problem Types, 15 Catchments

The four flood scenarios are only the complete-chain study. The benchmark itself covers:

CapabilitySubtasksInstances
C1 Environment ExplorationWorkspace Grounding258
C2 Situation PerceptionFlood Event Segmentation; Observation Anomaly Adjudication; Preforecast Data and State Analysis375
C3 Forecast OrchestrationCandidate Forecast Orchestration; Forecast Model Development524
C4 Warning DecisionsWarning and Dispatch Planning; Decision Dependency Verification433
Total8 subtasks1,590

The eight subtasks are different problem types: retrieval, segmentation, classification, state description, forecast combination, model building, planning and verification. The data come from 15 catchments and 129 stations in two countries, with different rainfall regimes, terrain and degrees of regulation.

The complete-chain study has also been extended from 4 to 8 flood events with 3 runs per framework (72 chains):

CompleteC1 Fidelity ↑C2 Acc. ↑C3 nMAE ↓C4 F1 ↑Tokens (M) ↓
Claude Code12/240.6780.6409.4500.4093.790
OpenCode18/240.7990.7354.5220.6654.065
AutoTSer24/240.8080.9024.7030.7950.265

The other reviewers also note the coverage of connected tasks:

Reviewer PjTV: "The benchmark connects source preparation, situation analysis, forecasting, and decisions. This exposes errors that isolated forecasting tasks cannot measure."

Reviewer ux2B: "The benchmark covers multiple tasks within time series analysis rather than isolated tasks which makes it useful for the hydrologic analysis task."

The experiments in other domains are in Parts 3 and 4.

[1] TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning, 2026.

[2] When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference, 2025.

AuthorsOpenReview ↗

Response to Reviewer kEDU [Part 2]

Q1 (continued): Could the authors provide additional experiments, or at least case studies, involving time-series problems from other real-world domains?

(3) TimeSage-MT: 80 Conversations in Eight Domains

  • Benchmark: 20 conversations at each of four levels, with 450 user turns per framework. Levels 3–4 cover energy, transportation, telecom, manufacturing, retail, climate, finance, and healthcare.
  • Systems: AutoTSer, Claude Code, OpenCode, and AION use the same backbone. AutoTSer runs with the controller off and without hydrologic skills.
  • Metrics: analytical quality (AQ, 0–100), averaged first within conversations; Level-4 final-turn decision quality (DQ, 0–100); and nine capability scores from the official evaluation code.
  • Statistics: conversation-paired 95% bootstrap intervals and two-sided paired tests with Holm adjustment.

All results have been added to Table 4 and Appendix H of Revised Paper‾\underline{\text{Table 4 and Appendix H of Revised Paper}}. For your convenience, we summarize them below.

Analytical Quality and Decision Quality (0–100; Higher Is Better). Best scores are bold; second-best scores are underlined.

MethodL1 AQL2 AQL3 AQL4 AQMean AQL4 DQ
Conversations202020208020
AutoTSer78.2475.4668.6166.0672.0961.83
Claude Code75.61‾\underline{75.61}74.19‾\underline{74.19}64.23‾\underline{64.23}60.02‾\underline{60.02}68.51‾\underline{68.51}58.50‾\underline{58.50}
OpenCode71.8071.1359.1158.3965.1150.50
AION28.3727.5317.5013.0021.6012.58

Paired Differences in Analytical Quality [95% CI] (conversation as the unit)

AutoTSer minusOverall (80)L3 (20)L4 (20)
Claude Code+3.58 [0.24, 6.50]+4.37 [0.58, 7.85]+6.03 [1.42, 10.89]
OpenCode+6.98 [3.93, 9.91]+9.50 [4.76, 13.83]+7.66 [3.73, 11.43]
AION+50.49 [46.53, 54.39]+51.10 [46.43, 56.01]+53.06 [47.43, 58.52]

Analytical Quality by Application Domain (Levels 3–4)

MethodClimateEnergyFinanceHealthcareManufacturingRetailTelecomTransportation
Conversations68228464
AutoTSer68.1066.3065.6479.7266.3763.5865.1271.89
Claude Code64.45‾\underline{64.45}60.09‾\underline{60.09}59.98‾\underline{59.98}66.93‾\underline{66.93}63.56‾\underline{63.56}63.67‾\underline{63.67}57.35‾\underline{57.35}64.15
OpenCode58.0758.2559.5261.0154.3863.8255.1068.42‾\underline{68.42}
AION22.4418.479.177.897.2318.6512.3521.74

Outcome Scores across Nine Capabilities (official evaluation code; 0–1)

MethodReadSelectConfigureRememberChainAdaptDecideCalibrateGround
Applicable conversations798080606010404040
AutoTSer0.2380.4690.205‾\underline{0.205}0.812‾\underline{0.812}0.976‾\underline{0.976}0.0000.8020.476‾\underline{0.476}0.588‾\underline{0.588}
Claude Code0.213‾\underline{0.213}0.3820.2100.8720.9930.0000.774‾\underline{0.774}0.4780.624
OpenCode0.1660.416‾\underline{0.416}0.205‾\underline{0.205}0.7910.9680.0000.7180.4530.511
AION0.0760.0790.1960.7400.8820.0000.3710.2750.374

Direct Prompting (40-conversation panel shared by five methods)

L1L2L3L4AVG
Direct Prompt37.8828.1218.3120.5226.21
AION30.5822.9016.4314.3521.07
OpenCode70.9674.8959.3559.4766.17
Claude Code77.2476.8263.3360.7969.55
AutoTSer79.1779.1769.6066.7273.67

Findings:

(1) AutoTSer achieves the highest AQ at every level and the highest L4 decision quality. Mean AQ is 72.09, and L4 DQ is 61.83.

(2) The overall improvement is supported by paired comparisons. The AQ difference is +3.58 [0.24, 6.50] against Claude Code and +6.98 [3.93, 9.91] against OpenCode. The Holm-adjusted p-values are 0.0246 and 0.000040, respectively.

(3) The gains are larger on domain-grounded reasoning. At L3 and L4, AutoTSer improves over Claude Code by 4.37 and 6.03 points, and over OpenCode by 9.50 and 7.66 points. It has the highest AQ in seven of the eight application domains.

(4) The capability scores locate the advantage in data reading, method selection, and decisions. AutoTSer leads on Read (0.238), Select (0.469), and Decide (0.802). The complete nine-capability table reports all dimensions.

Overall, the expanded evaluation shows stronger analytical quality and decision performance across multi-turn tasks outside hydrology.

AuthorsOpenReview ↗

Response to Reviewer kEDU [Part 3]

Q1 (continued): Could the authors provide additional experiments, or at least case studies, involving time-series problems from other real-world domains?

(4) TSAIA: Multi-Step Analysis and Decisions

  • Benchmark: multi-step tasks from energy, climate, finance and healthcare, with deterministic success criteria: 195 analysis tasks and 140 decision questions.
  • Systems: the official CodeAct agent of the benchmark and AutoTSer, on the same backbone. AutoTSer uses the same configuration as on TimeSage-MT.
  • Execution budget: both methods start with at most three code executions per task. We report the initial and supplemented results.

Success Rate (%)

MethodAnalysis TasksDecision QuestionsOverall
Num of Tasks195140335
CodeAct (official agent),73.8‾\underline{73.8}65.0‾\underline{65.0}70.1‾\underline{70.1}
AutoTSer,74.473.674.0

Findings:

(1) AutoTSer has the higher success rate: 74.0% against 70.1% for the official agent.

(2) The gain is largest on decision questions: 73.6% against 65.0% (+8.6 points, 95% CI [1.4, 15.7]).

(3) No task-specific design is used. AutoTSer runs with the controller off and without any skill written for this benchmark.

(5) Case Studies

Case 1: healthcare (TimeSage-MT, ECG with sleep-apnea labels). One turn asks for a sliding-window KNN model with a chronological 70/10/20 split. AutoTSer reports that the validation split contains no apnea sample, and identifies that the validation score does not assess apnea detection. Claude Code states that apnea rates are comparable across splits, although its own table shows 0% in training and validation and 93.8% in test. OpenCode reports R² = 1.00 without comment. Analytical quality on this conversation: 85.5 for AutoTSer, 69.3 for Claude Code, 57.9 for OpenCode.

Case 2: manufacturing (TimeSage-MT, Level 4, sensor data with pass/fail labels). One turn asks for 96-step and 672-step rolling statistics. AutoTSer reports that the 672-step window has no valid value because of six missing readings, and bases the drift statement on the 96-step window. Claude Code fills the window with partial data, so its early "672-step" values are expanding-window statistics. OpenCode reports only the final window. In the final decision turn, Claude Code presents a majority-class KNN score as validation, and OpenCode reads a non-significant sensor (p = 0.84) as separating the classes. Analytical quality: 80.0 vs. 61.7 and 69.4; decision quality: 66.7 vs. 41.7 and 50.0.

These cases show how AutoTSer uses computed evidence to assess class coverage, window validity, and statistical support before drawing conclusions. Case analyses and answer excerpts are provided in Appendix H.3 of Revised Paper‾\underline{\text{Appendix H.3 of Revised Paper}}.

Overall, AutoTSer achieves higher overall analytical quality on TimeSage-MT and a higher supplemented success rate on TSAIA. These results extend the evaluation from hydrologic workflows to multi-turn analysis and decision tasks in eight other application domains.

Reviewer_PjTVOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FACTS-BENCH for connected hydrologic analysis and AUTOTSER for checking evidence, preserving handoff constraints, and selecting optional operations. The benchmark covers 15 catchments and 1,590 scored subtask instances, while the complete-chain comparison uses four flood cases.

Critical strengths
  1. A useful evaluation setting. The benchmark connects source preparation, situation analysis, forecasting, and decisions. This exposes errors that isolated forecasting tasks cannot measure.

  2. A sensible distinction between checks. Evidence checking asks whether a result is correct, while boundary checking asks whether it can be used in the next stage. Keeping these questions separate is useful for analysing failures in a connected workflow.

  3. Controlled analysis of error propagation. The upstream-correction experiments trace changes through the selected models to final actions. They show that correcting an input can affect decisions without reducing forecast error.

Critical weaknesses
  1. Limited complete-chain evidence. The full-cycle comparison covers eight flood events, each repeated three times. This remains a limited basis for broad claims about reliable full-cycle automation. Repeated runs measure execution variability but do not increase scenario diversity.

Although the benchmark overall covers 15 catchments, the diversity represented in the complete-chain evaluation is unclear. Appendix G.1 states that each workspace contains 49 raw station files, including 32 rainfall-gauge series, but does not clearly establish whether the eight events cover distinct catchments or station configurations. The selection criteria are specified for the four additional events but are less clear for the initial four. Furthermore, the main full-cycle comparisons do not report confidence intervals, making the reliability and generalizability of the results difficult to assess.

  1. The controller is not fully specified. The paper does not provide enough numerical detail on the utility weights, selection thresholds, and training traces to reproduce the controller from the text alone. The offline cost of collecting its alternative execution branches also needs to be reported.

  2. Reference quality needs quantification. The benchmark relies partly on expert judgments about anomalies and operational decisions. The paper describes expert checking but does not report annotator agreement or explain how disputed judgments were resolved.

  3. Decision-reference validity requires further justification. Warning and dispatch actions are evaluated against expert-defined reference sets constructed from historical operational records and rules. However, historical decisions are not necessarily optimal, and multiple operational strategies may be defensible under the same conditions.

The distinction between necessary and acceptable actions is useful, but recall is calculated only against the necessary set. This is appropriate if those actions are genuinely required across all valid strategies. If alternative action combinations can satisfy the same operational objective, however, they need to be adequately represented in the evaluation. Otherwise, Action F1 could penalize defensible alternatives and may not reliably reflect operational decision quality.

  1. Cross-stage dependency evaluation remains narrow. The upstream-correction experiments provide useful diagnostic evidence, but they primarily examine how correcting rainfall-data processing affects downstream predictions and decisions. The analysis uses only eight events, with one paired continuation per event, and therefore provides limited evidence about other types of cross-stage dependencies and error propagation.

The observed decision effects are also mixed, including cases where corrections reduce Action F1. These decreases do not necessarily invalidate the corrections, but they require explanation. The current experiment does not establish whether the framework consistently handles different dependency types or translates improved upstream processing into better operational decisions.

  1. Self-evolution lacks dedicated full-cycle validation. The self-evolution mechanism is mainly evaluated through isolated subtasks, without a dedicated full-cycle ablation demonstrating its contribution to end-to-end performance. Although it improves selective forecasting-model adoption, improvements on other tasks are limited, and isolated warning-planning F1 remains unchanged. Its effectiveness for the paper’s central full-cycle objective is therefore insufficiently demonstrated.

  2. The source of the decision improvements remains unclear. AutoTSer achieves higher full-cycle Action F1 than OpenCode, at 0.795 versus 0.665. However, forecasting accuracy is comparable, and the decision advantage persists on jointly completed runs. These observations suggest that forecasting accuracy and completion may not fully explain the gap.

With shared hydrology skills and reference inputs, OpenCode matches AutoTSer’s isolated planning F1 of 0.297. Upstream situation analysis or domain-specific guidance may therefore contribute to the full-cycle advantage, but the baseline skill configuration in the full-cycle experiments is unclear. Furthermore, the boundary-check ablation does not show improved Action F1 on jointly completed runs, with reported scores of 0.791 versus 0.818.

Controlled experiments using identical upstream outputs and matched domain guidance are needed to distinguish the contributions of situation analysis, planning, orchestration, and boundary checking. Without these controls, the paper cannot convincingly attribute its decision improvements specifically to more reliable cross-stage information use.

Questions for authors
  1. More independent complete chains. Can the authors provide additional independent chains and repeated runs, reporting completion together with common-case forecast comparisons? This would strengthen the main claim beyond the four current cases.

  2. A complete controller recipe. Can the authors provide the numerical utility, gating rules, trace counts, and train-test separation used for controller learning? These details are needed to assess the reported benefit of self-evolution.

  3. Reliability of expert references. How many independent annotators contributed to the decision references, and what was their agreement before reconciliation? Please explain how disagreements were resolved, since these references determine the final decision scores.

8 comments
AuthorsOpenReview ↗

Response to Reviewer PjTV [Part 1]

We would like to sincerely thank Reviewer PjTV for the insightful review and comments.

W1: Limited complete-chain evidence. The full-cycle comparison covers only four flood cases, each run once. This is too small a basis for broad claims about reliable full-cycle automation.

Q1: More independent complete chains. Can the authors provide additional independent chains and repeated runs, reporting completion together with common-case forecast comparisons? This would strengthen the main claim beyond the four current cases.

We sincerely appreciate this concrete suggestion. To fully address your concern, we doubled the independent chains from 4 to 8 flood events and ran every chain 3 times per framework (72 chains). AutoTSer completes 24/24 chains, against 18/24 for OpenCode and 12/24 for Claude Code. On common events, its decision F1 is higher in 8 of 8 events against OpenCode and in 6 of 7 against Claude Code.

(1) Eight Events, Three Runs Each (72 chains)

  • Events: eight historical floods from five years (2014–2018). Each chain starts from 49 raw station files, including 32 rainfall-gauge series, with a 7-day history and a 24-hour forecast window. The eight windows do not overlap.
  • Why the full-cycle chains are few: each chain needs one real flood with its complete raw records, the observed outcome and an expert-reviewed dispatch reference built from real operations. Such events are rare and valuable.
  • New events: the four new events were selected by a rule fixed before any run (first 08:00 origin with stage ≥ 8.5 m, 24-h rainfall ≥ 10 mm and 6-h rise ≥ 0.05 m). Their reference actions were frozen before the first run.
  • Runs: three runs per event and framework, each with 100 model calls and 30 minutes.
  • Scoring: runs are averaged within an event, then events are averaged. A missing C1, C2 or C4 output scores zero.

All results have been added to Table 3 and Appendix G.1 of Revised Paper‾\underline{\text{Table 3 and Appendix G.1 of Revised Paper}}. For your convenience, we summarize them below.

Full-Cycle Results (24 chains per framework)

CompleteC1 Fidelity ↑C2 Acc. ↑C3 nMAE ↓C4 F1 ↑Tokens (M) ↓
Claude Code12/240.6780.6409.4500.4093.790‾\underline{3.790}
OpenCode18/24‾\underline{18/24}0.799‾\underline{0.799}0.735‾\underline{0.735}4.5220.665‾\underline{0.665}4.065
AutoTSer24/240.8080.9024.703‾\underline{4.703}0.7950.265

Completed Runs per Event (of 3)

E1E2E3E4E5E6E7E8Total
Claude Code1‾\underline{1}1‾\underline{1}01‾\underline{1}2‾\underline{2}2‾\underline{2}32‾\underline{2}12/24
OpenCode31‾\underline{1}1‾\underline{1}1‾\underline{1}333318/24‾\underline{18/24}
AutoTSer3333333324/24

Findings:

(1) AutoTSer completes all three runs of all eight events, with the highest state accuracy (0.902) and decision F1 (0.795), at about 7% of the tokens of the other two agents.

(2) The other agents fail at the forecast-to-decision handoff. Of their 15 workflow-incomplete runs, 14 first fail at the forecast stage. In 11 of them a numerical forecast and a plan exist, but the forecast evidence cited by the plan cannot be matched to the computed candidates.

The full per-event table is in Part 2, and the common-case forecast comparison in Part 3.

AuthorsOpenReview ↗

Response to Reviewer PjTV [Part 2]

Q1 (continued): Can the authors provide additional independent chains and repeated runs, reporting completion ...

(2) Full Per-Event Results

Each cell is the mean over the three runs of that event. Complete is the number of completed runs.

Best values are bold; second-best values are underlined‾\underline{\text{underlined}}. Brown Mean\color{brown}{\text{Mean}} labels identify each framework's average. Event rows are compared within each event; Mean rows are compared across frameworks.

MethodEventCompleteC1 Fidelity ↑C2 Acc. ↑C3 nMAE ↓C4 F1 ↑Tokens (M) ↓
Claude CodeE11/30.8130.75848.8810.8115.036
E21/30.7990.78823.0610.5333.645
E30/30.6390.7580.2340.4373.991
E41/30.5900.5760.5470.6364.633
E52/30.7990.6970.9780.4183.599
E62/30.4930.4850.8990.0003.240
E73/30.6530.6060.9380.1892.682
E82/30.6390.4550.0620.2473.310
Mean\color{brown}{\text{Mean}}12/240.6780.6409.4500.4093.790‾\underline{3.790}
OpenCodeE13/30.6460.75827.7470.8962.633
E21/30.5910.6362.9660.3003.585
E31/30.9860.9390.2610.8834.025
E41/30.9570.8180.6590.8326.667
E53/30.7990.6972.5110.5313.346
E63/30.6390.6061.0120.5144.217
E73/30.9790.7880.9430.5193.777
E83/30.7960.6360.0790.8424.268
Mean\color{brown}{\text{Mean}}18/24‾\underline{18/24}0.799‾\underline{0.799}0.735‾\underline{0.735}4.5220.665‾\underline{0.665}4.065
AutoTSerE13/30.8910.97031.4661.0000.300
E23/30.7690.9392.1950.4940.328
E33/30.7251.0000.2000.8440.264
E43/30.8050.9090.6071.0000.280
E53/30.8280.9090.9290.7760.277
E63/30.8400.8181.1110.6990.221
E73/30.8280.7581.0570.5710.232
E83/30.7780.9090.0560.9740.216
Mean\color{brown}{\text{Mean}}24/240.8080.9024.703‾\underline{4.703}0.7950.265

Findings:

(1) AutoTSer has 3/3 completed runs in every event; OpenCode in 5 events and Claude Code in 1.

(2) AutoTSer has the highest decision F1 in 6 of the 8 events and is within 0.04 of the highest in the other two. It reaches 1.000 in E1 and E4.

(3) Token use is stable across events (0.22–0.33M per chain), against 2.6–6.7M for the other agents. (3) Common-Case Forecast Comparison

Each comparison uses the same events and the same run indices for both frameworks. nMAE divides the error by the standard deviation of the 169 observations before the forecast origin. For E1, the pre-origin standard deviation is approximately 0.05 m, which magnifies normalized forecast errors. We therefore also report paired event-level comparisons.

AutoTSer vs. OpenCode (8 common events, 18 matched runs each)

C3 nMAE ↓Events with Lower nMAEC4 F1 ↑Events with Higher F1
OpenCode4.5123‾\underline{3} of 80.668‾\underline{0.668}0‾\underline{0} of 8
AutoTSer4.651‾\underline{4.651}5 of 80.7978 of 8

AutoTSer vs. Claude Code (7 common events, 12 matched runs each)

C3 nMAE ↓Events with Lower nMAEC4 F1 ↑Events with Higher F1
Claude Code19.271‾\underline{19.271}2‾\underline{2} of 70.521‾\underline{0.521}1‾\underline{1} of 7
AutoTSer5.2854 of 70.7906 of 7

Coverage of the 24 Runs per Framework

RunsCompleteForecasts ProducedIncompleteInterface or Local Faults
Claude Code24121893
OpenCode24182360
AutoTSer24242400

Findings:

(1) Decisions are better on the same events. AutoTSer has the higher decision F1 in 8 of 8 events against OpenCode (0.797 vs. 0.668) and in 6 of 7 against Claude Code (0.790 vs. 0.521).

(2) Forecasts are more accurate on most common events: 5 of 8 against OpenCode and 4 of 7 against Claude Code (5.285 vs. 19.271).

(3) Forecast accuracy and decision quality are different abilities. With a similar forecast error to OpenCode (4.65 vs. 4.51), AutoTSer gives better decisions in all eight events. This is the cross-stage effect the benchmark is built to measure.

Overall, with 8 independent events and 72 chains, AutoTSer completes every chain and gives better decisions on the same events. The extended study confirms the full-cycle conclusions of the submission.

AuthorsOpenReview ↗

Response to Reviewer PjTV [Part 3]

W2: The controller is not fully specified. The paper does not provide enough numerical detail on the utility weights, selection thresholds, and training traces to reproduce the controller from the text alone. The offline cost of collecting its alternative execution branches also needs to be reported.

Q2: A complete controller recipe. Can the authors provide the numerical utility, gating rules, trace counts, and train-test separation used for controller learning? These details are needed to assess the reported benefit of self-evolution.

Many thanks for pointing this out. All requested details are now provided: the numerical utility, the gating rules, the trace counts, the train–test separation and the offline cost. They have been added to Appendix D.4 of Revised Paper‾\underline{\text{Appendix D.4 of Revised Paper}}. For your convenience, we summarize them below.

(1) Numerical Utility

The controller scores predefined operations from the current analytical state, retaining the reference action if none passes. Development features include pre-origin validation results, portfolio composition, and differences between developed and reference forecasts.

DecisionSubtasksDefaultOptional OperationUtility
Extra reviewGrounding, Statecurrent outputextra source or fact reviewquality gain − 0.001 per extra model call
Skip reviewEvent, Anomaly, Plan, VerifyAuditor reviewskip reviewquality(draft) − quality(reviewed) + 0.001
Keep draftPlan, Verifyreviewed outputretain draftquality(draft) − quality(reviewed)
Proposal choiceOrchestrationsubmitted forecastsaved pre-origin proposalnMAE(default) − nMAE(proposal)
AdoptionDevelopmentOrchestration forecastblend 25/50/75/100% developed weightnMAE(Orchestration) − nMAE(blend)

(2) Gating Rules

  • Estimator: fitted Q-iteration with 256 extremely randomized trees per decision; minimum leaf size 2, feature fraction 0.8, seed 37.
  • Statistics: mean and 10th-percentile gain across trees, the fraction predicting negative gain, and support distance.
  • Standard gate: positive mean and 10th-percentile utility gains, harm probability ≤ 0.25 and support distance ≤ 0.75. Review routing also requires positive mean and 10th-percentile quality gains (tolerance 10⁻⁹).
  • Support distance: nearest-state distance for the same action, computed as the root mean square of component differences. Numeric differences use training-set standard deviations, are clipped at three standard deviations, and scaled to [0,1]; categorical mismatches are 0/1.
  • Adoption gate: three inner folds choose between retaining the default and gates (min expected gain, min lower gain, max harm probability): (0,0,0.25), (0.01,−0.01,0.25), (0.01,−0.05,0.50). Selection maximizes Gain with at most 5% harmed cases. Four of five outer folds select (0.01,−0.01,0.25). The support limit is the 95th percentile of inner-fold distances (0.29–0.32).

(3) Trace Counts and Train–Test Separation

Evaluation is held out for every subtask. Development has 261 valid Gain records from 262 assigned cases, including 251 numerical pairs. Five outer station-held-out folds evaluate the controller; three inner folds select gates.

SubtaskPaired Pool (Train per Fold)Held-Out CasesSeparation
Grounding7 (6)7leave one workspace out
Event2717chronological
Anomaly4424chronological
State29 (19–20)303 held-out-site folds
Orchestration18973chronological
Development251 (197–204)251 numerical; 261 with Gain5 outer station folds × 3 inner folds
Plan510fit on 2015, test on 2016
Verify3072fit on 2015, test on 2016

(4) Offline Cost of Collecting Branches

Trace SourceSubtasksRecordsModel CallsTokens (M)
Optional review branchesGrounding, State37630.91
Developed portfolio and Orchestration referenceDevelopment2623904.83
Draft and reviewed outputsEvent, Anomaly, Plan, Verify21866413.07
Forecasts and saved proposalsOrchestration2625637.24

Development uses 390 model calls, 4.826M tokens and 753 forecast-model training attempts. Calibration, fitting and evaluation take 122.4 s on one CPU thread. The last two trace sources are saved during the main benchmark runs.

(5) Benefit under This Protocol

Policy on the Same Saved OutcomesGain ↑Adoptions (of 251)Improved / Harmed
Keep the Orchestration forecast0.00000/0
Always adopt the developed portfolio (controller off)−1.56825194/157
Same 33 adoptions, shuffled allocation (mean)−0.06033—
Controller+0.0173325/8

In summary, held-out Gain is +0.0167: 25 of 33 adoptions improve forecasts. Matched-count random allocation yields −0.060 (10,000 shuffles, p=0.0024).

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to All Reviewers & Summary of Revisions

We sincerely thank all reviewers for their comments and suggestions, which have helped us strengthen the paper.

This paper introduces FacTS-Bench, a benchmark for full-cycle time series analysis in hydrology, comprising 1,590 scored instances across eight subtasks and 15 catchments. It connects source preparation, situation analysis, forecasting and operational decisions using observations and expert-reviewed operational records. We also introduce AutoTSer, which combines evidence checks, boundary checks and learned execution control. The benchmark and framework support controlled analysis of how upstream errors affect forecasts and decisions.

Reviewers recognized the value of connected evaluation: “This exposes errors that isolated forecasting tasks cannot measure.” (PjTV) and “The benchmark covers multiple tasks within time series analysis rather than isolated tasks which makes it useful for the hydrologic analysis task.” (ux2B). They also highlighted the framework's effectiveness: “The fact-grounding procedure implemented in AUTOTSER directly addresses this challenge, and the experimental results support its effectiveness.” (kEDU).

We have expanded the experiments and clarified the methodology to address the reviewers' questions. Below is a summary of the major revisions:

  • More complete chains and repeated runs (PjTV; kEDU): We expand from four events evaluated once to eight events evaluated three times per framework, totaling 72 runs. AutoTSer completes 24/24 chains, compared with 18/24 for OpenCode and 12/24 for Claude Code. On common events, AutoTSer achieves higher decision F1 in 8/8 events against OpenCode and 6/7 against Claude Code. Updated → Table 3 and Appendix G.1‾\underline{\text{Table 3 and Appendix G.1}}

  • Evaluation on external time series benchmarks (kEDU; ux2B): We expand TimeSage-MT from 20 to 80 conversations and add 335 matched TSAIA tasks. Without the controller or hydrologic skills, AutoTSer ranks first at every TimeSage level, with significant overall AQ gains over all three baselines after Holm correction. TSAIA supplemented success is 74.0%, versus 70.1% for CodeAct under the same execution protocol. We add domain-level results, nine-capability scores, confidence intervals and paired tests. Updated → Table 4 and Appendix H‾\underline{\text{Table 4 and Appendix H}}

  • Controller recipe and comparisons with simple policies (PjTV; kEDU): We specify the utility, gates, trace counts, train/test separation and offline costs. Station-held-out evaluation gives +0.0167 Development Gain. Randomly allocating the same 33 adoptions with the same blend proportions gives −0.0596 mean Gain (p = 0.0024). Comparisons with forecast retention, fixed mixtures and a shallow decision tree show the value of selecting operations according to the execution state. Newly added → Section 6.3 and Appendix D.4‾\underline{\text{Section 6.3 and Appendix D.4}}

  • Framework comparisons and backbone-dependent orchestration (kEDU; ux2B): We add shared-skills and controller-off comparisons. We also compare fixed forecast assembly and original Worker selection across seven backbones, explaining the nearly invariant C3 nMAE in Table 2 and reporting both forecast error and peak timing. Newly added → Tables 1–2 and Appendix G.2‾\underline{\text{Tables 1–2 and Appendix G.2}}

  • Expert references and annotation agreement (PjTV): We document the participation of eight experts with doctoral-level hydrologic knowledge, agreement before reconciliation, and the resolution process for flood-event and anomaly references. We explain how historical dispatch records, operating rules and expert consultation establish necessary, acceptable and unsupported decision references. Newly added → Section 4 and Appendix B.2‾\underline{\text{Section 4 and Appendix B.2}}

  • Extended ablation and error-propagation analysis (PjTV; kEDU): Boundary-check ablations and upstream-correction experiments now cover eight events. Upstream correction leaves final forecasts unchanged in six events while action F1 changes in five. We also expand trace-based failure analysis from 12 to 20 erroneous runs per framework, updating the error distributions and case analysis. Updated → Figures 4–5 and Appendix G.2–G.4‾\underline{\text{Figures 4–5 and Appendix G.2–G.4}}

  • Benchmark scope, construction procedure and framework contribution (kEDU; ux2B): We explain why hydrology supports full-cycle evaluation and how teams with operational time series and decision records can reuse the construction procedure. We replace “New Models” with “Agent Framework”, clarify the controller's role in agent execution, and add a Limitations section. Updated → Abstract, Introduction, Sections 3–5 and Limitations‾\underline{\text{Abstract, Introduction, Sections 3–5 and Limitations}}

We thank the reviewers again for their valuable suggestions. We would be happy to answer further questions and look forward to your feedback.

Best regards,

The Authors

AuthorsOpenReview ↗

Request of Reviewer's attention and feedback

Dear Reviewer,

As a gentle reminder, it has been more than two days since we submitted our rebuttal, and we were wondering whether our response has addressed your concerns. We look forward to your feedback and are prepared to address any further concerns you may have.

Thank you again for your constructive feedback and continued support, which have inspired further improvements to our paper.

Best regards,

The Authors

AuthorsOpenReview ↗

Request of Reviewer's attention and feedback

Dear Reviewers,

It has been more than three days since we posted our rebuttal and revised manuscript, and the discussion period is drawing to a close. We would greatly appreciate your feedback on our responses.

We have addressed the review points individually and expanded the evaluation to 72 full-cycle runs, 80 TimeSage-MT conversations, and 335 TSAIA tasks, alongside additional methodological clarifications and analyses. Revisions are highlighted in brown.

We hope you will revisit your assessment in light of these explanations and results. We are happy to discuss any remaining questions.

Thank you again for your time and suggestions. We look forward to hearing from you.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.