acceptodds
Withdrawn from ICLR 2027

STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) deployed in real-world agentic applications must be capable of replanning and adapting when mid-task disruptions invalidate their prior decisions. Existing dynamic benchmarks primarily measure whether LLMs can detect temporal changes in a timely manner, leaving the complementary challenge of adaptive replanning under spatio-temporal dynamics largely unexplored. We introduce STT-Arena (Spatio-Temporal Tool-Use Arena), a benchmark of 227 high-quality interactive tasks spanning nine spatio-temporal conflict types and four solvability levels. Each task is grounded in a realistic, executable environment equipped with injected spatio-temporal triggers that can abruptly invalidate an ongoing plan, forcing the model to detect the state shift and construct a revised execution strategy. Extensive evaluation of frontier LLMs reveals that even the SOTA proprietary models, including Claude-4.6-Opus, achieves less than 40% overall accuracies, highlighting the fundamental difficulty of spatio-temporal dynamic reasoning. Systematic analysis of failure trajectories uncovers three recurring error modes of existing models: Stale-State Execution, Misdiagnosis of Dynamic Triggers, and Missing Post-Adaptation Verification. Guided by these findings, we propose an iterative trajectory refinement technique that eliminates these failure patterns from training data, and combine it with online RL to produce STT-Agent-4B which outperforms frontier LLMs on STT-Arena.

Primary area:
datasets and benchmarks
Keywords:
Spatio-Temporal Dynamics, Tool-Use Benchmark, Adaptive Replanning, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sa9uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces STT-Arena, a benchmark of 227 interactive tool-use tasks in which spatio-temporal triggers change environment state or tool availability mid-execution, so the agent must detect the change and replan. Tasks cover nine conflict subtypes and four levels (easy, medium, hard, impossible). Environments, blueprints, user profiles, checklists, and check functions are synthesized with Qwen-3.5-397B through a three-stage pipeline with dual-agent verification and human review. Across 20 LLMs, the best model (Claude-4.6-Opus) reaches 35.39% Pass@1. The paper identifies three failure modes (stale-state execution, misdiagnosis of dynamic triggers, missing post-adaptation verification), proposes iterative trajectory refinement, and trains STT-Agent-4B with SFT and RL, which reaches 27.17%.

Critical strengths
  1. The targeted capability — abandoning and rebuilding a plan after an abrupt mid-task state change, including recognizing infeasibility — matters for deployed agents and is not the focus of most tool-use benchmarks. The evaluation covers a broad set of closed and open models with three runs each.
Critical weaknesses
  1. Benchmark validity is not established. All environments, triggers, user profiles, checklists, and check functions are LLM-generated, and the human review reports no agreement, rejection, or error statistics. The examples shown raise concrete doubts. The check function in Fig. 17 merely tests that a shipment with the given goods ID exists, which does not by itself verify that the plan adapted to the trigger; because the initial configuration is not shown, it is also unclear whether this check could pass without the agent's actions. The example task in the appendix (pp. 22–24) is internally inconsistent: the deadline is 08:00 in the user goal but 10:00 in the checklist, and the clinic closes at 12:00 in the mutation but at 10:00 in the user profile. How Pass@1 is derived from the per-check reward R_fea is not defined. The conflict taxonomy in the Stage-1 filtering prompt (Fig. 23: state drift, validity expiration, …, moving-window resource) differs from the one used to report results (Table 11: window expiry, priority reorder, …, handoff failure), and the taxonomy inserted into the Stage-2 generation prompts (Figs. 31–33) is never shown. Every headline number depends on these checks and labels, so this is the main reason for my rating.

  2. The modeling claims are overstated and possibly confounded. The abstract says STT-Agent-4B "outperforms frontier LLMs," but at 27.17% it trails eight evaluated models, including Claude-4.6-Opus (35.39%) and GPT-5.4 (34.51%). Training data come from the same pipeline, generator, and templates as the benchmark, and the paper does not state that test environments are disjoint from training ones. About 7.2 of the 16.6-point gain over Qwen-3-4B comes from the 30 Impossible tasks (10.0 → 64.4), which is consistent with learning to declare infeasibility; the false-infeasibility rate on solvable tasks is not reported. Reporting errors add to the concern: "xx%" left in the Table 3 caption; "Appendix ??" for the human baseline; Qwen-3.5-35B-A3B's Easy/Medium/Hard results (including standard deviations) duplicating GPT-5.4-mini's, while its Overall (12.48) does not match its own weighted average (≈14.4; similarly, Kimi-K2.5 reports 19.18 against a weighted average of ≈20.6); and a claim that inter-model gaps narrow monotonically with k that Fig. 6 does not show.

Questions for authors
  1. Are the Stage-1 static environments and tool code used to generate SFT/RL data disjoint from those behind the 227 test tasks? If not, please report results on held-out environments.
  2. How is Pass@1 computed for solvable tasks (must all checks pass?), and what fraction of check functions are already satisfied by the initial state or by a trajectory with no state-changing calls?
  3. How often does STT-Agent declare infeasibility on solvable tasks, compared with the base model?
  4. Which conflict taxonomy was inserted into the Stage-2 generation prompts, and how does it map to the Table 11 categories used to report results?
Reviewer_uaGjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an executable benchmark for evaluating tool-using agents under changing temporal and spatial constraints. Agents must adapt their plans to complete a task or recognize when it has become infeasible. The authors construct and validate the environments, evaluate multiple models under static and dynamic conditions, and analyze common failures. They also use trajectory refinement, supervised fine-tuning, and reinforcement learning to train STT-Agent.

Critical strengths
  1. The benchmark addresses a practical gap in tool-use evaluation: agents need to respond when changes invalidate an ongoing plan. The executable environments and controlled temporal and spatial disruptions provide a useful way to study this ability.

  2. The authors support the benchmark with execution checks, dual-agent verification, human review, and controlled comparisons. The training results are also encouraging, with fewer tool calls after refinement and better performance on solvable tasks from the full training pipeline.

Critical weaknesses
  1. The main text does not clearly introduce STT-Agent or explain its training procedure, which is a presentation issue and really confusing. STT-Agent is trained on data generated using the same pipeline as STT-Arena. Without a clear training/test split, it is hard to tell whether the gains reflect generalization or familiarity with the task setup.

  2. Low overall success rates do not necessarily indicate poor replanning, since models also struggle with static tasks. A clearer separation of basic execution failures and failures after a disruption would strengthen this conclusion. The static/dynamic comparison is useful, but more behavioral analysis would help distinguish basic execution failures from failures to adapt after a conflict.

Questions for authors
  1. How is STT-Agent trained, and how are training and evaluation tasks separated in terms of source queries, environments, and conflict templates?

  2. Among trajectories that encounter a disruption, how often do agents recover and complete the task?

Reviewer_McgmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

STT-Arena is a benchmark paper, of 227 complex tool use tasks. The authors generate complexity in the form of introducing spatio-temporal conflicts in the tasks to invalidate the plan, to test if the system can detect and navigate around the conflicts. The authors have benchmarked across a large set of open and closed models and also produces a small STT Agent which is SFT+RL trained with iterative trajectory refinement performing close to SOTA.

Critical strengths
  1. The research question and the premise is sound. The taxonomy is sensible and is the experimental setup can be largely replicated.
  2. I like that that authors have shared all the prompts and copious amount of details on the experiments in the appendix. This is crucial for any dataset/benchmark paper and the shared artifacts are very comprehensive and will lead again to successful replication and consistency of results.
Critical weaknesses
  1. The low absolute number performance of number is a bit difficult to digest. If the human eval sits at 56% (the appendix describing how it is done is apparently broken) then surely the other models are expected to do worse. After reading the methodology, the tasks themselves do not, in theory, seem very complex, they are all an execution of a finite state machine or a linear chain of events probably run through a ReACT agent. Recent studies and real world deployments have demonstrated strongly that the performance of the models in solving a complex task depends very tightly with the harness used to solve the task. We dont know enough about how the system is architected so these low numbers and the reasons for why these numbers are low could also be attributed to evaluation or harness even.

  2. STT-Agent is distilled from qwen 397B which in this case is a weak teachers since it's own performance is capped at 29%. So the training on 2K trajectories is largely redundant and only leads to a 2pp improvement even after refinement. Speaking of the refinement, the results in the avg. number of calls is not surprising. This also leads to a train-test leakage question, are these completely different from the original set of 227 datasets? if yes, then what is stopping the authors at only 227? if no, then this is a gap is the experiments. Please quantify overlap. Further details on the training require exposition, for example message block is never defined, in an agentic setting block could mean turns, could mean tool results, could mean reasoning blocks etc. so reorder/delete/modify blocks could mean the LLM edits the SFT training transcript, or deletes a blind retry or reorder a diagnosis etc, we dont know. Further, the refinement essentially deletes the redundant steps from the SFT trajectories, so the model is trained on a shorter transcripts and to perform tasks in a shorter trajectory. The call-count reduction (32.70 to15.30, Table 2) could just be mechanical.

  3. The overall metric is a micro average of the 4 types of hardness. So it is a simple total passed/227 an obfusvated plain mean. This has several consequences, a) a high metric can be achieved by doing extremely well just on the easy tasks b) it down weights the impossible cases which is where the STT Agent's refinement gain is largest. c) it also is blending fractional scores with binary impossible score which is strange. A macro average would have probably been better with some separate reporting of the impossible score.

  4. The scale is too low for this benchmark to be useful. Only 227 data points does not have any specific justification of why this cannot be scaled up, it is totally LLM generated and human verified so only 227 data points across 9 categories (4 levels) leads to about ~20-40 datapoints per categories that is not enough to characterize behavior of these models. Authors should quantify and justify this.

  5. Section 4 states several observations already known and unsurprising. Test time scaling leads to better results (mostly) atleast to multi step tasks have been knows. User simulator is essentially privileged knowledge, and has also known to lead to better solution, theres an entire field on this topic, reasoning helps but design matters is largely an artifact of mostly true with the Qwen anomaly appearing to stem from its output being parsed in a non-standard way rather than from any deeper effect.

  6. The static vs dynamic comparison in Figure 5 cannot be interpreted correctly by readers. Removing the triggers and reconstructing the checklist changes the actual task and the rubric at the same time. A de-triggered dynamic task is a straight forward execution task, not the same task minus the dynamics. The impossible tasks in this scenario may even become solvable tasks once triggers are removed altering the mix. The composition and the variance of each condition are not disclosed in the experiment, and there are no error bars on 4 models. A change in model ranking just shows that execution and planning are different skills across models and its a stretch to conclude that the static score is overfitting to fixed patterns.

  7. The harness is simplistic, no parallel tool calls, no conditionals, not even code gen/CodeAct paradigm, just constrained one tool call at a time. Its unlikely showing the entire spectrum of capabilities of these models to solve complex problems. It does not also show how the tool call errors and feedback are provided, how the outputs are formatted or parsed etc. to rule out the ranking sensitivity to the harness implementations. There are no ablations provided on this either.

Questions for authors

See weaknesses.

Comments on the paper

1 comment

Dear Reviewers,

I hope this message finds you well.

I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.

Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.

If you have any questions or need further assistance, please do not hesitate to reach out.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.