- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces AgentEscapeBench, an escape-room-style benchmark that test whether LLM agents can infer and execute new tool-use process under long-range dependencies. Each task is an automatically generated DAG of tools and items. The benchmark has 270 instances across 5 difficulty levels. Experiments with 16 LLMs and human participants show that success drops sharply with difficulty. Trajectory analysis attributes model failures mainly to poor state tracking and failure to propagate intermediate results.
Critical strengths
-
The motivation is clear, and proposed benchmark reflects the motivation effectively.
-
The evaluation scope is broad, with a wide range of models and human baseline under same interface. The prompt and example trajectory is provided transparently.
Critical weaknesses
- A lot of texts are not aligned with the content of the table. (ex. "Claude-Opus-4.6 ... and leads at all levels above 5.", but underperform in Diff 10 and 15 compared to Gemini, "All three metrics degrade monotonically...", but it is incorrect regarding to Table 1, etc.)
Questions for authors
- Section 3.3 reports that the required interaction horizon is the strongest predictor of success, while critical-path length adds no signal once horizon is included. Then how the authors argue that difficulty comes mainly from long-range dependencies, not the number of required steps?
2 comments
Dear Reviewer UbTE,
Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.
Regards from the authors.
We thank the reviewer for the positive assessment of the motivation, evaluation scope, and transparency of the benchmark. We appreciate the careful reading and the opportunity to improve the consistency of the presentation and clarify the interpretation of task difficulty.
1. Consistency between the text and Table 1
Thank you for identifying these inconsistencies. We will revise the corresponding statements to match the table precisely.
First, the statement that Claude-Opus-4.6 “leads at all levels above 5” is too broad. At difficulty 10, Claude-Opus-4.6, Gemini-3.1-Pro-Preview, and GPT-5.4 share the highest model SR of 83.3%, while Gemini has higher Sub. and Disc. than Claude. At difficulty 15, Gemini has a higher SR than Claude. We will replace the original statement with:
Second, “all three metrics degrade monotonically with difficulty for every model” overstates the observed trend. Table 1 contains both ties and increases between adjacent tiers; for example, Claude's discovery rate increases from difficulty 10 to difficulty 15. We will replace this statement with:
We will also systematically check ranking statements, trend descriptions, and numerical comparisons throughout the main text and captions against the corresponding tables, ensuring that each claim specifies the relevant metric and difficulty range.
2. Interaction horizon and long-range dependencies
Thank you for asking us to clarify this relationship. Interaction horizon describes the amount of interaction required by a workflow, while dependency structure describes the information and ordering constraints that must be maintained throughout that interaction. These are related aspects of the task.
In the generated workflows, outputs obtained earlier are used as inputs to later operations, and agents must track which prerequisites have been resolved as new tools and items become available. Consequently, extending the workflow increases the duration over which the agent must preserve and correctly use task state.
Critical-path length captures the longest chain in the DAG, but it does not fully characterize this coordination burden. For example, a chain can have a concurrent working set of one regardless of its length, whereas interleaved branches can require retaining several outputs simultaneously for later use. Appendix D therefore reports complementary structural properties, including graph width and concurrent working set. The latter increases from 1.47 at difficulty 5 to 5.00 at difficulty 25, reflecting an increase in the number of intermediate results that must remain available during execution.
The finding in Section 3.3 establishes that interaction horizon is the strongest predictor among the structural variables examined, and that critical-path length provides no additional predictive signal once horizon is included. This result concerns the predictive contribution of critical-path depth; it does not imply that the interactions are independent or that information from earlier steps is irrelevant.
Our trajectory diagnostics provide the complementary behavioral perspective. Premature invocation measures execution before designated prerequisites have been resolved, while clue adherence measures whether downstream arguments trace back to the appropriate upstream outputs. These measurements characterize how agents handle dependency state and intermediate results during extended interactions.
We will clarify the main conclusion as follows:
This formulation aligns the interpretation with the structural analysis while preserving the benchmark's central motivation: evaluating whether agents can sustain reliable, dependency-aware tool use over extended interactions.