- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TS-AgentBench combines twenty real forecasting tasks with a synthetic causal environment. Its central idea is to compare clean, corrupted, and absent-feature inputs: a flat corruption curve may mean that an agent is robust, but it may also mean that the agent ignores the feature. The paper also measures variability across agent-generated pipelines. It finds preliminary agreement between real and synthetic rankings, substantial execution instability, and negative point estimates for some historical sensor features. However, no individual synthetic necessity test or pooled real-data ablation excludes zero.
The real and synthetic cohorts play different roles. The real tasks provide varied forecasting workflows; the simulator provides known variable roles and controlled perturbations. The matched ranking comparison uses seven agent–model configurations that complete a common subset of tasks. The necessity experiments are narrower: one scaffold, three models and the warehouse profile. The protocol reports an unnormalized sensitivity curve when the availability contrast is not resolved, avoiding an unstable relative score. The real-data experiment removes auxiliary information at a coarser level and is reported as inconclusive rather than a confirmed replication.
Critical strengths
-
I find the benchmark setup useful. It evaluates the whole forecasting pipeline, including parsing, joining tables, and deciding which features to use. These are real sources of failure for agents. Adding an absent-feature condition is also a reasonable way to ask what a flat robustness curve actually means.
I appreciate the decision to include raw relational files and practical data problems rather than evaluate forecasts only after preprocessing has already been solved. Parsing dates, joining tables and recognizing which inputs exist at the forecast origin are legitimate capabilities to test. The synthetic generator also makes controlled interventions possible, and the explicit real/synthetic distributional comparison helps readers judge where those interventions are representative. The suite could be a useful development tool even if the initial agent results remain uncertain.
-
The repeated-run analysis is a useful contribution in its own right. An agent can produce quite different pipelines on the same task, so a single successful run tells us little about its reliability. The paper is also clear that the real-data experiment does not confirm the synthetic result.
The paper also handles several negative results responsibly. It does not report normalized robustness ratios when the denominator is unresolved, and it checks whether trimming extreme rollouts changes the direction or significance of the availability effects. The deterministic forecasts provide useful non-LLM controls. These choices make the paper more informative than a leaderboard alone. I encourage the authors to preserve this candor while narrowing the claims to what the diagnostic design actually establishes.
Critical weaknesses
-
My main question is whether the experiments identify the distinction that motivates the benchmark. The Agentic Benchmark Checklist (ABC; Zhu et al., 2025) emphasizes checking that task performance reflects the capability being claimed. Here, none of the nine synthetic cells passes the necessity gate. The consistently negative estimates concern sensors that stop at the forecast origin, while the clean lookahead-bait features are pure noise. Removing a file can also change the schema, context and pipeline the agent chooses. The paper does not yet show that its test distinguishes robust use of useful information from ignoring it. A controlled example with an informative feature available during forecasting, together with a matched nuisance-input control, would address this concern directly.
This is a construct-validation problem, not simply a request for a positive result. A well-designed diagnostic should behave as expected on cases where the underlying mechanism is known: a pipeline that uses a genuinely predictive available input, a pipeline that ignores it, and a pipeline whose behavior changes only because the input layout changes. The current causal-parent sensors are unavailable across the prediction window, while the clean bait variables contain no new information. Those choices can reveal poor engineering decisions, but they leave the central separation of useful dependence from robust use under-demonstrated. A small positive-control experiment would be more directly informative than increasing the number of uncalibrated cells.
The code-level analysis is also deliberately limited to three runs. That is a useful illustration, not a population measure of engagement. I would therefore avoid interpreting a statistically nonzero availability effect as evidence that the model used the feature in its fitted predictor, or a nonsignificant effect as evidence that the feature was unnecessary. The paper already acknowledges this distinction; the interpretation of the headline findings should follow it consistently.
-
ABC also emphasizes explaining evaluation failures and checking what trivial outputs receive from the grader. The paper already reports repeated runs, confidence intervals, deterministic forecasts and trimming checks, which are useful. However, the matched leaderboard excludes tasks on which a configuration fails, while the synthetic grader sets missing predictions to zero. I cannot tell how those different policies affect the reported rankings, or what happens with blank and malformed submissions. Appendix G also pools rollout medians across unequally replicated seeds, and the six-cell sign test uses dependent model/operator cells. Complete attempt counts and a failure-inclusive analysis would help establish what population and reliability claim these results support.
The complete-case restriction in the matched panel is understandable for comparing like-for-like tasks, but it conditions the comparison on successful execution by all included configurations. It should be accompanied by a view that includes execution reliability, because inability to finish the pipeline is itself relevant to an agent benchmark. Likewise, replacing missing predictions with zero may define a numerical forecast score, but it does not describe the same event as an agent failing to produce a valid submission. Reporting these outcomes separately would make both the engineering and forecasting interpretations clearer.
I would also distinguish stochastic agents from stochastic grading. Low ICC and rare catastrophic pipelines are important observations about the agents, but they do not by themselves establish that the same saved prediction artifact receives the same score in a clean rerun. The Docker isolation and hidden labels are appropriate safeguards; explicit reset and artifact-rescoring checks would complete the description of the measurement process. These concerns should be addressed proportionately, using the existing evaluator and logs rather than requiring a completely new benchmark.
Overall Assessment
Presentation: The setup and its limitations are generally clear.
The distinction between availability effects and actual code-level feature use is especially important and should remain visible in the abstract and conclusions. Appendix F’s small code-trace example helps explain why these are not interchangeable.
Informativeness: The pipeline-instability results are useful, but the main empirical separation remains inconclusive.
Soundness: The target of the uncertainty analysis and the handling of failures need clarification.
I currently lean toward rejection because the central necessity-versus-robustness measurement has not been convincingly validated, and failure-conditioned comparisons make the reliability interpretation uncertain. The benchmark’s task coverage, deterministic controls and repeated-run analysis are useful. A targeted validation of what the availability contrast measures, together with transparent failure handling, could change my view.
This is a worthwhile benchmark direction, and the effort to expose pipeline instability deserves encouragement. My recommendation does not rest on the absence of significant individual effects alone. It rests on whether the claimed diagnostic has been calibrated and whether the inference reflects the complete set of attempted evaluations. A focused revision around those issues could make the benchmark substantially more useful without needing to replace its entire task collection.
Questions for authors
-
Can the clean/corrupted/absent protocol distinguish a reference pipeline that demonstrably uses an informative feature available during forecasting from one that ignores it? Does the distinction survive a matched change in file size/schema that supplies no additional information?
-
What exactly is d, how are seeds weighted and paired in the bootstrap, and how do the conclusions change when all attempted tasks and failures are accounted for? How does the grader handle blank, missing and malformed submissions, and what scores do those controls receive relative to the existing naive forecasts?
-
Can the same saved prediction artifact be rescored in a fresh environment with identical results, and how are table joins, target alignment and residual files checked? Which failures are attributable to the agent versus the grader or runtime?
-
What is the population to which the main claims generalize, given that the necessity experiments use one scaffold/profile and the ranking agreement is based on seven configurations? Which reported conclusions remain when execution failures are included and dependent real datasets are handled together?