acceptodds
Under review as a conference paper at ICLR 2027

TS-AgentBench: Disentangling Feature Necessity from Robustness in Agentic Time-Series Forecasting

Abstract

Agent benchmarks typically summarize performance with a single leaderboard score, which is useful for ranking but insufficient for assessing reliability. We introduce TS-AgentBench, a benchmark suite combining 20 real-world Kaggle competitions with a structural causal synthetic environment. Both support standard leaderboard evaluation, and their rankings strongly agree (Spearman ), showing that the synthetic cohort provides a scalable, leakage-immune complement to real-world tasks. Beyond ranking, TS-AgentBench introduces a three-condition diagnostic protocol – clean input, corrupted feature, and complete feature absence – to evaluate whether agents rely on useful information or are affected by irrelevant signals. We find that agents are more often harmed than helped by additional covariates, revealing reliability issues hidden by aggregate scores. These effects are difficult to detect within individual tasks because repeated runs can produce substantially different solution strategies, reflecting instability in agent decision-making rather than benchmark noise. TS-AgentBench therefore provides both a realistic leaderboard and a diagnostic framework for understanding agent reliability beyond performance alone.

Primary area:
datasets and benchmarks
Keywords:
benchmark, agentic time series, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_obRQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TS-AgentBench combines twenty real forecasting tasks with a synthetic causal environment. Its central idea is to compare clean, corrupted, and absent-feature inputs: a flat corruption curve may mean that an agent is robust, but it may also mean that the agent ignores the feature. The paper also measures variability across agent-generated pipelines. It finds preliminary agreement between real and synthetic rankings, substantial execution instability, and negative point estimates for some historical sensor features. However, no individual synthetic necessity test or pooled real-data ablation excludes zero.

The real and synthetic cohorts play different roles. The real tasks provide varied forecasting workflows; the simulator provides known variable roles and controlled perturbations. The matched ranking comparison uses seven agent–model configurations that complete a common subset of tasks. The necessity experiments are narrower: one scaffold, three models and the warehouse profile. The protocol reports an unnormalized sensitivity curve when the availability contrast is not resolved, avoiding an unstable relative score. The real-data experiment removes auxiliary information at a coarser level and is reported as inconclusive rather than a confirmed replication.

Critical strengths
  1. I find the benchmark setup useful. It evaluates the whole forecasting pipeline, including parsing, joining tables, and deciding which features to use. These are real sources of failure for agents. Adding an absent-feature condition is also a reasonable way to ask what a flat robustness curve actually means.

    I appreciate the decision to include raw relational files and practical data problems rather than evaluate forecasts only after preprocessing has already been solved. Parsing dates, joining tables and recognizing which inputs exist at the forecast origin are legitimate capabilities to test. The synthetic generator also makes controlled interventions possible, and the explicit real/synthetic distributional comparison helps readers judge where those interventions are representative. The suite could be a useful development tool even if the initial agent results remain uncertain.

  2. The repeated-run analysis is a useful contribution in its own right. An agent can produce quite different pipelines on the same task, so a single successful run tells us little about its reliability. The paper is also clear that the real-data experiment does not confirm the synthetic result.

    The paper also handles several negative results responsibly. It does not report normalized robustness ratios when the denominator is unresolved, and it checks whether trimming extreme rollouts changes the direction or significance of the availability effects. The deterministic forecasts provide useful non-LLM controls. These choices make the paper more informative than a leaderboard alone. I encourage the authors to preserve this candor while narrowing the claims to what the diagnostic design actually establishes.

Critical weaknesses
  1. My main question is whether the experiments identify the distinction that motivates the benchmark. The Agentic Benchmark Checklist (ABC; Zhu et al., 2025) emphasizes checking that task performance reflects the capability being claimed. Here, none of the nine synthetic cells passes the necessity gate. The consistently negative estimates concern sensors that stop at the forecast origin, while the clean lookahead-bait features are pure noise. Removing a file can also change the schema, context and pipeline the agent chooses. The paper does not yet show that its test distinguishes robust use of useful information from ignoring it. A controlled example with an informative feature available during forecasting, together with a matched nuisance-input control, would address this concern directly.

    This is a construct-validation problem, not simply a request for a positive result. A well-designed diagnostic should behave as expected on cases where the underlying mechanism is known: a pipeline that uses a genuinely predictive available input, a pipeline that ignores it, and a pipeline whose behavior changes only because the input layout changes. The current causal-parent sensors are unavailable across the prediction window, while the clean bait variables contain no new information. Those choices can reveal poor engineering decisions, but they leave the central separation of useful dependence from robust use under-demonstrated. A small positive-control experiment would be more directly informative than increasing the number of uncalibrated cells.

    The code-level analysis is also deliberately limited to three runs. That is a useful illustration, not a population measure of engagement. I would therefore avoid interpreting a statistically nonzero availability effect as evidence that the model used the feature in its fitted predictor, or a nonsignificant effect as evidence that the feature was unnecessary. The paper already acknowledges this distinction; the interpretation of the headline findings should follow it consistently.

  2. ABC also emphasizes explaining evaluation failures and checking what trivial outputs receive from the grader. The paper already reports repeated runs, confidence intervals, deterministic forecasts and trimming checks, which are useful. However, the matched leaderboard excludes tasks on which a configuration fails, while the synthetic grader sets missing predictions to zero. I cannot tell how those different policies affect the reported rankings, or what happens with blank and malformed submissions. Appendix G also pools rollout medians across unequally replicated seeds, and the six-cell sign test uses dependent model/operator cells. Complete attempt counts and a failure-inclusive analysis would help establish what population and reliability claim these results support.

    The complete-case restriction in the matched panel is understandable for comparing like-for-like tasks, but it conditions the comparison on successful execution by all included configurations. It should be accompanied by a view that includes execution reliability, because inability to finish the pipeline is itself relevant to an agent benchmark. Likewise, replacing missing predictions with zero may define a numerical forecast score, but it does not describe the same event as an agent failing to produce a valid submission. Reporting these outcomes separately would make both the engineering and forecasting interpretations clearer.

    I would also distinguish stochastic agents from stochastic grading. Low ICC and rare catastrophic pipelines are important observations about the agents, but they do not by themselves establish that the same saved prediction artifact receives the same score in a clean rerun. The Docker isolation and hidden labels are appropriate safeguards; explicit reset and artifact-rescoring checks would complete the description of the measurement process. These concerns should be addressed proportionately, using the existing evaluator and logs rather than requiring a completely new benchmark.

Overall Assessment

Presentation: The setup and its limitations are generally clear.

The distinction between availability effects and actual code-level feature use is especially important and should remain visible in the abstract and conclusions. Appendix F’s small code-trace example helps explain why these are not interchangeable.

Informativeness: The pipeline-instability results are useful, but the main empirical separation remains inconclusive.

Soundness: The target of the uncertainty analysis and the handling of failures need clarification.

I currently lean toward rejection because the central necessity-versus-robustness measurement has not been convincingly validated, and failure-conditioned comparisons make the reliability interpretation uncertain. The benchmark’s task coverage, deterministic controls and repeated-run analysis are useful. A targeted validation of what the availability contrast measures, together with transparent failure handling, could change my view.

This is a worthwhile benchmark direction, and the effort to expose pipeline instability deserves encouragement. My recommendation does not rest on the absence of significant individual effects alone. It rests on whether the claimed diagnostic has been calibrated and whether the inference reflects the complete set of attempted evaluations. A focused revision around those issues could make the benchmark substantially more useful without needing to replace its entire task collection.

Questions for authors
  1. Can the clean/corrupted/absent protocol distinguish a reference pipeline that demonstrably uses an informative feature available during forecasting from one that ignores it? Does the distinction survive a matched change in file size/schema that supplies no additional information?

  2. What exactly is d, how are seeds weighted and paired in the bootstrap, and how do the conclusions change when all attempted tasks and failures are accounted for? How does the grader handle blank, missing and malformed submissions, and what scores do those controls receive relative to the existing naive forecasts?

  3. Can the same saved prediction artifact be rescored in a fresh environment with identical results, and how are table joins, target alignment and residual files checked? Which failures are attributable to the agent versus the grader or runtime?

  4. What is the population to which the main claims generalize, given that the necessity experiments use one scaffold/profile and the ranking agreement is based on seven configurations? Which reported conclusions remain when execution failures are included and dependent real datasets are handled together?

Reviewer_JJPHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TS-AgentBench, a benchmark for evaluating LLM agents that construct time-series forecasting pipelines. It combines 20 Kaggle-derived tasks with a structural causal simulator supporting controlled perturbations of data values, schemas, and timestamps. The main methodological proposal is a clean/corrupted/absent evaluation protocol, with a hierarchical bootstrap test determining whether the clean–absent performance difference is distinguishable from zero before reporting a normalized robustness score. The experiments examine agreement between real and synthetic rankings, the effects of auxiliary covariates, and variability across repeated agent executions. The results show substantial execution-to-execution variability and consistently negative point estimates for historical sensor availability in the drift and missingness settings, although none of the individual necessity tests reaches statistical significance.

Critical strengths
  1. The paper identifies a concrete ambiguity in robustness evaluation: an agent can appear insensitive to corrupted inputs simply because those inputs contribute little to its predictions. Adding an absent condition provides a useful reference for interpreting corruption curves. The decision to withhold the normalized score when its denominator is not reliably separated from zero is also sensible, particularly given the substantial variability observed in agent execution (Section 4.1).

  2. The benchmark evaluates a substantial portion of the forecasting workflow. Agents must inspect raw relational files, parse timestamps, join tables, select features, and produce executable forecasting pipelines. Combining these competition-style tasks with a programmable simulator makes it possible to study specific data-handling problems while retaining the practical demands of code generation and model fitting. The eight perturbation operators cover several distinct sources of failure that aggregate forecasting scores would not identify on their own (Section 2; Appendices B and C).

Critical weaknesses
  1. My main concern is that the diagnostic experiments lack a positive control in which auxiliary covariates are both informative and available during forecasting. In the drift and missingness settings, sensor observations stop at the forecast origin, while the prediction horizon extends for 300 steps. In the clean lookahead setting, the added columns are pure noise. These choices make it difficult to establish whether the protocol can identify successful use of useful information and robustness to its corruption. Moreover, the clean–absent difference measures the effect of changing the agent’s inputs and potentially its entire solution strategy; it does not isolate dependence on a feature in the fitted model. A controlled comparison crossing predictive relevance with forecast-time availability would make the intended distinction more convincing.

  2. The evidence for the claim that additional covariates systematically harm agents remains suggestive. None of the nine synthetic cells passes the necessity gate, and none of the three pooled real-data effects excludes zero. Consequently, the proposed Relative Score is not exercised in any statistically resolved setting. The strongest directional result is the sign test over six negative drift/missingness cells, but these cells share models and operators, and the paper acknowledges that they are not fully independent. The reported null results are worth documenting; they do, however, limit the broader claim in the abstract. An analysis that accounts for dependence, together with evidence on how effect precision changes with the evaluation budget, would help establish what conclusions this protocol can support in practice.

  3. The diagnostic findings have substantially narrower coverage than the benchmark’s leaderboards. All necessity experiments use Smolagents, three models, and the warehouse profile, excluding the Claude Code configurations that lead the real-data evaluation. This omission matters because Section 3.1 already shows that the relative performance of scaffolds depends on the underlying model. The real–synthetic rank correlation is a useful preliminary check, but it involves only seven configurations and is computed on tasks with valid runs for every included configuration. Together with the documented gap in tail behavior and temporal smoothness, this leaves the transfer of the diagnostic findings uncertain. Replication on the leading configurations and rankings that account for execution failures would provide a stronger connection between the controlled analysis and real-world performance.

Questions for authors

See Weaknesses

Reviewer_aH5rOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TS-AgentBench, a benchmark for evaluating LLM agents on time-series forecasting. It consists of 20 real-world Kaggle forecasting competitions and a synthetic cohort generated from a structural causal model, in which the causal role of each input file is known and individual features can be corrupted by pathology operators. Rankings on the two cohorts agree across seven agent–model configurations (Spearman ρ = 0.786). The paper further proposes a clean/corrupted/absent protocol with a bootstrap “necessity gate” to separate whether an agent depends on a covariate from how robust it is to its corruption. Experiments show that no cell passes the gate, that agents tend to perform worse when causally relevant covariates are present, and that repeated runs on the same task vary substantially.

Critical strengths

None

Critical weaknesses
  1. The proposed diagnostic is never validated. The paper’s central methodological contribution, the necessity gate and Relative Score, does not pass in any of the nine (operator, model) cells, so the Relative Score is never computed. The paper also includes no positive control, i.e., a covariate that is both informative and available at forecast time, on which the gate is shown to detect genuine necessity.

  2. The main empirical finding is largely an artifact of the task design. The conclusion that agents perform worse when covariates are present comes from drift and missingness cells whose sensor covariates are provided only up to the forecast origin. They are therefore unavailable over the forecast horizon, and pipelines that rely on them have no inputs at prediction time. The result reflects this design choice rather than a general property of agents.

Questions for authors
  1. Can the authors add a setting where a covariate is both causally relevant and available over the forecast window, and show that the necessity gate passes there?
  2. Since the sensor covariates end at the forecast origin, they cannot be used as contemporaneous inputs over the forecast horizon. How much of the negative effect in the drift and missingness cells is explained by this design choice, rather than by agents being misled by irrelevant information?
  3. No single cell is significant, and the real-data test shows no significant effect. How many seeds and rollouts would be needed for the gate to resolve the observed effect sizes, given the low ICC? Is the current design sufficiently powered to support the paper’s conclusions?
  4. All necessity experiments use only Smolagents and the warehouse profile, while the two Claude Code configurations lead the real leaderboard. Do the negative effect and the instability findings hold for other scaffolds and profiles?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.