- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark for hydrologic analysis across the complete lifecycle rather than just time series forecasting. It contains 4 capabilities - environment exploration, situation perception, forecast orchestration and warning decisions. They also propose a cross-capability analyzer AutoTSER.
Critical strengths
The benchmark covers multiple tasks within time series analysis rather than isolated tasks which makes it useful for the hydrologic analysis task.
Critical weaknesses
The paper has a very narrow domain and this is misrepresented in the problem formulation and the title. IT is not a general benchmark to evaluate full-cycle analysis capabilities but is a very specific benchmark for hydrologic domain. This severely limits the contributions of the paper - would have been better if this was clear in the title and abstract. Further, many sentences oversell the contributions (example, one of the contributions listed is “New Models” whereas they are proposing and RL approach for evaluation).
Questions for authors
None
11 comments
Response to Reviewer ux2B [Part 2]
W1 (continued):The paper has a very narrow domain and this is misrepresented in the problem formulation and the title. It is not a general benchmark to evaluate full-cycle analysis capabilities but is a very specific benchmark for hydrologic domain. This severely limits the contributions of the paper - would have been better if this was clear in the title and abstract.
(3) Revised Scope and Added Limitations The revised title and abstract explicitly identify hydrology as the benchmark domain. The revisions below clarify the data contribution, benchmark-construction procedure, and evaluation scope.`
| Location | Summary of Revisions |
|---|---|
| Title | Beyond Isolated Tasks: Benchmarking and Automating Full-Cycle Time Series Analysis in Hydrology |
| Abstract: benchmark introduction | We introduce FacTS-Bench, a benchmark for evaluating full-cycle time series analysis in the hydrologic domain. Built from observations and operational records across 15 gauged catchments, it contains 1,590 scored instances spanning four capabilities and eight subtasks. Its evaluation follows intermediate results through successive stages to forecasts and decisions. |
| Introduction: motivation for the benchmark | Full-cycle evaluation requires observations, forecasts, and decision references for the same evolving event. We choose hydrology because flood operations connect these elements in a documented analytical workflow. Rainfall and river measurements establish current conditions, forecasts describe their evolution, and operating rules constrain warning and dispatch actions. Historical dispatch records link these conditions to decisions made by operational authorities. This allows us to evaluate intermediate analyses and trace their effects on downstream decisions. |
| Contribution 1: replace “New Setting and Testbed” | New Setting and Full-Cycle Benchmark. We formulate full-cycle time series analysis and introduce FacTS-Bench as an executable benchmark in hydrology. It connects eight subtasks through explicit intermediate outputs and evaluates their downstream use. We provide a reproducible construction procedure that converts observations, operating rules, and expert-reviewed decision records into linked tasks and scoring references. |
| Section 3: opening paragraph before the formal definition | We formulate full-cycle time series analysis as a sequence of stages in which later stages use intermediate results produced by earlier ones. Each task specifies a request, source records, available resources, and constraints on information use and actions. FacTS-Bench instantiates this setting in hydrology through environment exploration, situation perception, forecast orchestration, and warning decisions. |
| Limitations: new section before Conclusion | Scaling full-cycle time series analysis benchmarks is constrained by access to historical decision records and the expert effort needed to construct action references. The controller selects from predefined operations, so its gains depend on both trace coverage and the quality of available alternatives. Future work will expand full-cycle datasets to include more events and application domains, and use additional paired traces to improve evidence checks and plan revisions. |
Response to Reviewer ux2B [Part 3]
W1 (continued):The paper has a very narrow domain and this is misrepresented in the problem formulation and the title. It is not a general benchmark to evaluate full-cycle analysis capabilities but is a very specific benchmark for hydrologic domain. This severely limits the contributions of the paper - would have been better if this was clear in the title and abstract.
(4) Scope of FacTS-Bench (1,590 instances, 8 subtasks, 15 catchments)
Within the hydrologic domain, the benchmark is broad in problem types and data sources:
| Capability | Subtasks | Instances |
|---|---|---|
| C1 Environment Exploration | Workspace Grounding | 258 |
| C2 Situation Perception | Flood Event Segmentation; Observation Anomaly Adjudication; Preforecast Data and State Analysis | 375 |
| C3 Forecast Orchestration | Candidate Forecast Orchestration; Forecast Model Development | 524 |
| C4 Warning Decisions | Warning and Dispatch Planning; Decision Dependency Verification | 433 |
| Total | 8 subtasks | 1,590 |
- Eight problem types: retrieval, segmentation, classification, state description, forecast combination, model building, planning and verification.
- Two countries: seven regulated catchments in China and eight CAMELS-LUX catchments in Luxembourg, with 129 stations and different rainfall regimes, terrain and degrees of regulation.
- Complete chains: 8 real flood events from five years, each run 3 times per framework (72 chains), where every stage consumes the agent's own earlier output.
Full-Cycle Results (8 events × 3 runs, 24 chains per framework)
| Complete | C1 Fidelity ↑ | C2 Acc. ↑ | C3 nMAE ↓ | C4 F1 ↑ | Tokens (M) ↓ | |
|---|---|---|---|---|---|---|
| Claude Code | 12/24 | 0.678 | 0.640 | 9.450 | 0.409 | 3.790 |
| OpenCode | 18/24 | 0.799 | 0.735 | 4.522 | 0.665 | 4.065 |
| AutoTSer | 24/24 | 0.808 | 0.902 | 4.703 | 0.795 | 0.265 |
Completed Runs per Event (of 3)
| E1 | E2 | E3 | E4 | E5 | E6 | E7 | E8 | Total | |
|---|---|---|---|---|---|---|---|---|---|
| Claude Code | 1 | 1 | 0 | 1 | 2 | 2 | 3 | 2 | 12/24 |
| OpenCode | 3 | 1 | 1 | 1 | 3 | 3 | 3 | 3 | 18/24 |
| AutoTSer | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 24/24 |
Same-Event Comparison (matched completed runs)
| AutoTSer vs. | Common Events | C4 F1 ↑ (AutoTSer / other) | Events with Higher F1 | Events with Lower Forecast nMAE |
|---|---|---|---|---|
| OpenCode | 8 | 0.797 / 0.668 | 8 of 8 | 5 of 8 |
| Claude Code | 7 | 0.790 / 0.521 | 6 of 7 | 4 of 7 |
Findings:
(1) AutoTSer completes all three runs of all eight events (24/24), against 18/24 and 12/24.
(2) Its decisions are better on the same events: decision F1 is higher in 8 of 8 events against OpenCode and in 6 of 7 against Claude Code.
(3) It uses about 7% of the tokens of the other two agents (0.27M vs. 3.8–4.1M per chain).
Response to Reviewer ux2B [Part 1]
We would like to sincerely thank Reviewer ux2B for the review and comments.
We sincerely appreciate your feedback on the scope and presentation of our work. We have revised the title to “Beyond Isolated Tasks: Benchmarking and Automating Full-Cycle Time Series Analysis in Hydrology.” The abstract now explicitly introduces FacTS-Bench as a benchmark for full-cycle time series analysis in the hydrologic domain. We make this scope consistent throughout the paper and add a Limitations section. Below, we explain the choice of hydrology, the value of the operational records, the reusable construction procedure, and the additional evaluation of AutoTSer on TimeSage-MT and TSAIA.
(1) Hydrology Connects Observations to Decisions We choose hydrology because flood operations connect all stages required for full-cycle evaluation. Rainfall and river measurements establish current conditions. Forecasts describe their evolution. Operating rules and engineering states constrain warning and dispatch actions. Historical dispatch records document decisions made by operational authorities. These aligned records allow us to evaluate intermediate analyses and their downstream use within the same real event.
(2) Operational Data and a Reusable Construction Procedure The operational records are a substantive data contribution. They connect time series to actual gate, pump, and reservoir operations under documented constraints. Experts use these records and the conditions at each forecast origin to establish necessary, acceptable, and unsupported reference actions. This turns operational experience into an executable benchmark for assessing evidence-backed decisions. Our five-stage procedure in Section 4.2 and Appendix B covers data collection, processing and annotation, task construction, reference construction, and quality control. It specifies how to assemble source workspaces, intermediate facts, observed forecast targets, and reviewed decision references. Teams with time series and operational decision records can follow this procedure to construct their own full-cycle benchmarks, supplying their domain-specific variables, operating rules, and action references.