- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces STT-Arena, a benchmark of 227 interactive tool-use tasks in which spatio-temporal triggers change environment state or tool availability mid-execution, so the agent must detect the change and replan. Tasks cover nine conflict subtypes and four levels (easy, medium, hard, impossible). Environments, blueprints, user profiles, checklists, and check functions are synthesized with Qwen-3.5-397B through a three-stage pipeline with dual-agent verification and human review. Across 20 LLMs, the best model (Claude-4.6-Opus) reaches 35.39% Pass@1. The paper identifies three failure modes (stale-state execution, misdiagnosis of dynamic triggers, missing post-adaptation verification), proposes iterative trajectory refinement, and trains STT-Agent-4B with SFT and RL, which reaches 27.17%.
Critical strengths
- The targeted capability — abandoning and rebuilding a plan after an abrupt mid-task state change, including recognizing infeasibility — matters for deployed agents and is not the focus of most tool-use benchmarks. The evaluation covers a broad set of closed and open models with three runs each.
Critical weaknesses
-
Benchmark validity is not established. All environments, triggers, user profiles, checklists, and check functions are LLM-generated, and the human review reports no agreement, rejection, or error statistics. The examples shown raise concrete doubts. The check function in Fig. 17 merely tests that a shipment with the given goods ID exists, which does not by itself verify that the plan adapted to the trigger; because the initial configuration is not shown, it is also unclear whether this check could pass without the agent's actions. The example task in the appendix (pp. 22–24) is internally inconsistent: the deadline is 08:00 in the user goal but 10:00 in the checklist, and the clinic closes at 12:00 in the mutation but at 10:00 in the user profile. How Pass@1 is derived from the per-check reward R_fea is not defined. The conflict taxonomy in the Stage-1 filtering prompt (Fig. 23: state drift, validity expiration, …, moving-window resource) differs from the one used to report results (Table 11: window expiry, priority reorder, …, handoff failure), and the taxonomy inserted into the Stage-2 generation prompts (Figs. 31–33) is never shown. Every headline number depends on these checks and labels, so this is the main reason for my rating.
-
The modeling claims are overstated and possibly confounded. The abstract says STT-Agent-4B "outperforms frontier LLMs," but at 27.17% it trails eight evaluated models, including Claude-4.6-Opus (35.39%) and GPT-5.4 (34.51%). Training data come from the same pipeline, generator, and templates as the benchmark, and the paper does not state that test environments are disjoint from training ones. About 7.2 of the 16.6-point gain over Qwen-3-4B comes from the 30 Impossible tasks (10.0 → 64.4), which is consistent with learning to declare infeasibility; the false-infeasibility rate on solvable tasks is not reported. Reporting errors add to the concern: "xx%" left in the Table 3 caption; "Appendix ??" for the human baseline; Qwen-3.5-35B-A3B's Easy/Medium/Hard results (including standard deviations) duplicating GPT-5.4-mini's, while its Overall (12.48) does not match its own weighted average (≈14.4; similarly, Kimi-K2.5 reports 19.18 against a weighted average of ≈20.6); and a claim that inter-model gaps narrow monotonically with k that Fig. 6 does not show.
Questions for authors
- Are the Stage-1 static environments and tool code used to generate SFT/RL data disjoint from those behind the 227 test tasks? If not, please report results on held-out environments.
- How is Pass@1 computed for solvable tasks (must all checks pass?), and what fraction of check functions are already satisfied by the initial state or by a trajectory with no state-changing calls?
- How often does STT-Agent declare infeasibility on solvable tasks, compared with the base model?
- Which conflict taxonomy was inserted into the Stage-2 generation prompts, and how does it map to the Table 11 categories used to report results?
Dear Reviewers,
I hope this message finds you well.
I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.
Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.
If you have any questions or need further assistance, please do not hesitate to reach out.
Best regards,
AC