TS-AgentBench: Disentangling Feature Necessity from Robustness in Agentic Time-Series Forecasting

Agent benchmarks typically summarize performance with a single leaderboard score, which is useful for ranking but insufficient for assessing reliability. We introduce TS-AgentBench, a benchmark suite combining 20 real-world Kaggle competitions with a structural causal synthetic environment. Both support standard leaderboard evaluation, and their rankings strongly agree (Spearman $\rho =0.786$), showing that the synthetic cohort provides a scalable, leakage-immune complement to real-world tasks. Beyond ranking, TS-AgentBench introduces a three-condition diagnostic protocol – clean input, corrupted feature, and complete feature absence – to evaluate whether agents rely on useful information or are affected by irrelevant signals. We find that agents are more often harmed than helped by additional covariates, revealing reliability issues hidden by aggregate scores. These effects are difficult to detect within individual tasks because repeated runs can produce substantially different solution strategies, reflecting instability in agent decision-making rather than benchmark noise. TS-AgentBench therefore provides both a realistic leaderboard and a diagnostic framework for understanding agent reliability beyond performance alone.