acceptodds
Under review as a conference paper at ICLR 2027

AgentEscapeBench: Evaluating Tool-Grounded Dependency Reasoning in LLM Agents

Abstract

As LLM-based agents increasingly rely on external tools, it is important to evaluate their ability to sustain tool-grounded reasoning beyond familiar workflows and short-range interactions. We introduce AgentEscapeBench, an escape-room-style benchmark that tests whether agents can infer, execute, and revise novel tool-use procedures under explicit long-range dependency constraints. Each task defines a directed acyclic dependency graph over tools and items, requiring agents to invoke real external functions, track hidden state revealed incrementally, propagate intermediate results, and submit a deterministically verifiable final answer. AgentEscapeBench includes 270 instances across five difficulty tiers and supports fully automated evaluation. Experiments with sixteen LLM agents and human participants show that performance drops sharply as the required interaction horizon and interleaved dependency complexity increase: humans decline from 98.3% success at difficulty-5 to 80.0% at difficulty-25, while the best model drops from 90.0% to 60.0%. Trajectory analysis attributes model failures mainly to breakdowns in long-range state tracking, clue adherence, and intermediate-result propagation. These findings suggest that current agents can often handle local tool use but still struggle with deep contextual dependencies. We hope AgentEscapeBench can serve as a diagnostic testbed for measuring tool-grounded dependency reasoning and informing future training efforts toward more robust state tracking, intermediate-result propagation, and interaction planning.

Primary area:
generative models
Keywords:
LLM agents, benchmark, tool-use reasoning, function calling, multi-step planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UbTEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AgentEscapeBench, an escape-room-style benchmark that test whether LLM agents can infer and execute new tool-use process under long-range dependencies. Each task is an automatically generated DAG of tools and items. The benchmark has 270 instances across 5 difficulty levels. Experiments with 16 LLMs and human participants show that success drops sharply with difficulty. Trajectory analysis attributes model failures mainly to poor state tracking and failure to propagate intermediate results.

Critical strengths
  • The motivation is clear, and proposed benchmark reflects the motivation effectively.

  • The evaluation scope is broad, with a wide range of models and human baseline under same interface. The prompt and example trajectory is provided transparently.

Critical weaknesses
  • A lot of texts are not aligned with the content of the table. (ex. "Claude-Opus-4.6 ... and leads at all levels above 5.", but underperform in Diff 10 and 15 compared to Gemini, "All three metrics degrade monotonically...", but it is incorrect regarding to Table 1, etc.)
Questions for authors
  • Section 3.3 reports that the required interaction horizon is the strongest predictor of success, while critical-path length adds no signal once horizon is included. Then how the authors argue that difficulty comes mainly from long-range dependencies, not the number of required steps?
2 comments

We thank the reviewer for the positive assessment of the motivation, evaluation scope, and transparency of the benchmark. We appreciate the careful reading and the opportunity to improve the consistency of the presentation and clarify the interpretation of task difficulty.

1. Consistency between the text and Table 1

Thank you for identifying these inconsistencies. We will revise the corresponding statements to match the table precisely.

First, the statement that Claude-Opus-4.6 “leads at all levels above 5” is too broad. At difficulty 10, Claude-Opus-4.6, Gemini-3.1-Pro-Preview, and GPT-5.4 share the highest model SR of 83.3%, while Gemini has higher Sub. and Disc. than Claude. At difficulty 15, Gemini has a higher SR than Claude. We will replace the original statement with:

Claude-Opus-4.6 achieves the highest model success rate at difficulties 20 and 25, while the leading model at lower difficulty levels depends on the tier and metric.

Second, “all three metrics degrade monotonically with difficulty for every model” overstates the observed trend. Table 1 contains both ties and increases between adjacent tiers; for example, Claude's discovery rate increases from difficulty 10 to difficulty 15. We will replace this statement with:

Performance generally declines as difficulty increases, with metric-specific fluctuations and ties across adjacent tiers.

We will also systematically check ranking statements, trend descriptions, and numerical comparisons throughout the main text and captions against the corresponding tables, ensuring that each claim specifies the relevant metric and difficulty range.

2. Interaction horizon and long-range dependencies

Thank you for asking us to clarify this relationship. Interaction horizon describes the amount of interaction required by a workflow, while dependency structure describes the information and ordering constraints that must be maintained throughout that interaction. These are related aspects of the task.

In the generated workflows, outputs obtained earlier are used as inputs to later operations, and agents must track which prerequisites have been resolved as new tools and items become available. Consequently, extending the workflow increases the duration over which the agent must preserve and correctly use task state.

Critical-path length captures the longest chain in the DAG, but it does not fully characterize this coordination burden. For example, a chain can have a concurrent working set of one regardless of its length, whereas interleaved branches can require retaining several outputs simultaneously for later use. Appendix D therefore reports complementary structural properties, including graph width and concurrent working set. The latter increases from 1.47 at difficulty 5 to 5.00 at difficulty 25, reflecting an increase in the number of intermediate results that must remain available during execution.

The finding in Section 3.3 establishes that interaction horizon is the strongest predictor among the structural variables examined, and that critical-path length provides no additional predictive signal once horizon is included. This result concerns the predictive contribution of critical-path depth; it does not imply that the interactions are independent or that information from earlier steps is irrelevant.

Our trajectory diagnostics provide the complementary behavioral perspective. Premature invocation measures execution before designated prerequisites have been resolved, while clue adherence measures whether downstream arguments trace back to the appropriate upstream outputs. These measurements characterize how agents handle dependency state and intermediate results during extended interactions.

We will clarify the main conclusion as follows:

Agent reliability decreases as tasks require longer sequences of tool interactions with interleaved dependencies. Trajectory analysis identifies difficulties in maintaining execution state, respecting dependency readiness, and propagating intermediate results across these interactions.

This formulation aligns the interpretation with the structural analysis while preserving the benchmark's central motivation: evaluating whether agents can sustain reliable, dependency-aware tool use over extended interactions.

Dear Reviewer UbTE,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_j74uOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper claims it proposes an “escape room style” benchmark to assess the agent's capability to discover a correct route of tools in a predesigned environment. They expect this benchmark can potentially present agent’s failures from long-range state tracking, clue adherence and intermediate-result propagation. Thus provide insights for future model training direction.

The benchmark is built upon a predefined DAG skeleton with typed input and output ports, where each node is instantiated from a library of 32 tool, 2 terminal, 16 item, and 4 container templates. It includes 270 cases across 5 difficulty tiers defined by the number of nodes in the DAG. Through experiments on 16 models and a human baseline, the authors conclude that increasing horizon complexity and dependency leads to sharp performance degradation, with failures attributed to three specific breakdowns rather than individual tool-calling ability.

Critical strengths
  • The author propose a complete, machine-verifiable benchmark artifact with a controllable generator
  • The evaluations are conducted on a wider breadth breakdowns rather than single leaderboard number
  • The escape-room framing is an effective presentational choice. The underlying task is not new, but casting it as a room with container, drops and narrative clues makes the dependency structure concrete
Critical weaknesses
  • The performance degradation can not be attributed to deep contextual dependencies

The only controlled difficulty variable is the number of nodes. However, several other factors increase simultaneously, including available parameters (5→26), hidden nodes (2→7), step budget (35→200), and degenerate nodes (1→23 instances per tier). Any process with a fixed per-step failure probability would naturally show declining success rates as the number of steps increases. For example, assuming a per-step success probability of p = 0.98, pⁿ yields success rates of 90/82/74/67/60% for 5/10/15/20/25 nodes, closely matching Claude-Opus-4.6's 90/83/80/71/60%. This suggests that the headline performance degradation can be largely explained by accumulated local errors, without requiring any deep contextual dependency failures.

  • The claim that "reasoning models do not dominate" is not sufficiently supported

The paper uses DeepSeek-Reasoner underperforming DeepSeek-Chat to argue that the bottleneck lies in tool grounding rather than reasoning depth. In order to support such general conclusion, the author should at lease make comparison across multiple model families with reasoning setting or different tuned model variants.

  • The proposed failure diagnostics lack actionable insights for model improvement

The paper attributes agent failures to state tracking, clue adherence, and intermediate-result propagation, but the proposed metrics do not clearly distinguish these capabilities. These errors require different training strategies, yet the paper provides no controlled experiments to separate them. More importantly, the authors do not demonstrate how these diagnostics can guide training or improve agent performance.

Questions for authors
  1. Could the observed degradation be explained by the accumulation of local tool-calling errors over longer interaction horizons, rather than a specific deficiency in long-range dependency reasoning?
  2. How does AgentEscapeBench distinguish knowing what to do from knowing when to do it?
  3. How can the authors determine whether the observed failures require improvements to the underlying model, rather than simple changes to the agent scaffold, such as explicit state tracking, intermediate-result storage, or dependency-aware planning?
4 comments

We thank the reviewer for recognizing the executable benchmark artifact, controllable generation process, and breadth of the trajectory analysis. We clarify below how the diagnostics characterize long-horizon tool-use behavior, how they inform targeted training-data construction, and how we will make the scope of our empirical claims more precise.

1. Accumulated local errors and dependency coordination

We address this concern by examining how errors arise and are corrected during execution. Our analysis combines final success rates with trajectory-level measurements of dependency readiness, feedback-driven correction, and intermediate-result propagation. These measurements characterize the behaviors observed during successful and unsuccessful interactions.

The expression pnp^n models a sequence of nn independent, equally reliable requirements that must all succeed. Interpreting pp as first-attempt tool-call accuracy does not describe our interactive protocol: a node can require multiple interactions, failed calls provide feedback, and agents can retry and recover. In an additional audit of the saved difficulty-5–20 trajectories, 1,410 of 5,403 attempted model–source-tool pairs were not resolved on their first execution; 1,289 of these 1,410 pairs (91.42%) were subsequently resolved. This analysis considers initial nonterminal source tools. Thus, an initial local error is frequently recoverable, and multiplying first-attempt accuracies would not account for the observed correction process.

We also examined parameter use under conditions that remove uncertainty about the designated source. We restricted the analysis to instances without the out-of-range base arguments identified in our tool-semantics audit, and required that (i) the input's exact upstream annotation had been returned to the agent, (ii) the source's latest observed output was correct, and (iii) this output matched the target's designated input. Counting only the first eligible use of each target input port in each trajectory, 51 of 1,932 uses supplied a different value and did not produce the correct target output. This identifies concrete propagation failures even after the relevant reference and value have become available.

For example, in one Claude-Opus-4.6 trajectory, the agent first received hex_cipher [upstream: text_xor_hex.hex], later obtained the correct hexadecimal output from text_xor_hex, and then supplied the node-name string text_xor_hex instead of that output to the decryptor. The decryptor returned an empty output. This is a directly observable failure to bind an available intermediate result to a downstream argument.

These analyses identify specific execution and correction patterns beyond the aggregate success curve. In particular, preserving dependency state, using available source values, and incorporating feedback are concrete targets for improving reliability across extended tool interactions. We will emphasize this connection between observed behavior and model-improvement objectives in the revised discussion.

2. Distinguishing what to do from when to do it

We interpret this distinction operationally: “what to do” concerns selecting and parameterizing an operation using the available task information, while “when to do it” concerns whether the operation is ready to execute under the current dependency state. Our existing metrics provide complementary views of these decisions.

Behavioral dimensionExisting metricWhat it measures
Grounding downstream arguments in the task's information sourcesClue adherence rateWhether submitted arguments trace back to outputs of the designated upstream nodes
Identifying and refining source-tool inputsSource-node convergence speedAttempts needed to resolve source inputs using initial information and execution feedback
Respecting the execution order implied by dependenciesPremature invocation rateThe fraction of non-source invocations made before all designated data-flow predecessors have been resolved

Clue adherence directly addresses the parameter-provenance aspect of “what to do” and is complemented by source-node convergence and the local tool-call error analysis in Section 4.2. Premature invocation addresses “when to do it” relative to the designated dependencies. For example, identifying a decryptor and its required input types does not by itself ensure that the agent correctly tracks whether the required ciphertext and key have already been obtained.

Action parameterization and execution timing are coupled in a dependency-driven task, so we treat these metrics as complementary behavioral measurements rather than independent abilities. A premature call can also use an incorrect argument, affecting both measurements. To examine value use separately from source availability, the additional audit reported above conditions on an already exposed reference and a correct observed source output; it identifies 51 unsuccessful binding deviations among 1,932 first eligible uses. We will clarify these relationships and distinguish successful alternative computations when interpreting deviations from the designated workflow.

3. Actionable diagnostics for training stronger tool-use capabilities

Our intended use of the diagnostics is to identify recurring behavioral failure patterns and translate them into concrete targets for training-data construction and model improvement. In particular, state tracking and management, clue utilization, and intermediate-result propagation are capabilities that models need to sustain throughout long-horizon tool interactions. We will make this training-oriented motivation more explicit.

The identified patterns suggest the following targeted data and supervision:

Observed failure patternTargeted training dataCorresponding evaluation
Poor state tracking and premature downstream invocationInterleaved tool-use trajectories that maintain which nodes are visible, resolved, or pending, and choose actions whose required inputs are availablePremature invocation rate, sub-problem resolution, and final success
Weak clue utilization or incorrect intermediate-result propagationTrajectories that associate clues and outputs with their sources, retain those associations across intervening actions, and correctly bind them to downstream inputs amid competing candidate valuesClue adherence and binding accuracy conditioned on source availability
Repeated incorrect attempts despite informative feedbackRecovery trajectories that revise the affected arguments using parameter-level feedback and avoid repeating disproven choicesSource-node convergence speed and eventual resolution

High-quality trajectories exhibiting these behaviors can be incorporated into mid-training or post-training. For example, extended interaction traces can provide continued-training data, while post-training can use supervised action sequences or preference comparisons between successful continuations and verified failures under the same observation history. The typed dependencies and deterministic execution provide a basis for checking these examples and identifying where a continuation violates an available reference, fails to incorporate feedback, or invokes an unresolved dependency without producing a correct result.

The training objective is to internalize reliable state updating, clue utilization, and dependency coordination in the model's action policy, with the goal of improving its transferable capability across unfamiliar tool compositions and long-horizon tasks. This connects each diagnosis to a specific form of data and supervision, as well as a measurable evaluation target. Generalization can then be assessed on held-out compositions under a fixed evaluation scaffold and budget.

Explicit state stores and dependency-aware planners can also improve performance and are complementary to this model-level training direction. We believe that scaffolding can help models better exercise their existing capabilities, but improvements from scaffolding alone may remain limited when the model lacks the underlying abilities for state tracking, clue utilization, and dependency coordination. This motivates training models to interpret evolving observations and make reliable decisions throughout extended interactions. Our diagnostics are useful for evaluating improvements from both training and scaffold design. We will expand the discussion to explain these concrete training implications and their connection to the observed failure patterns.

4. Scope of the reasoning-model comparison

Thank you for pointing out that this observation should be stated more precisely. We will revise the heading and discussion of “reasoning models do not dominate” to describe the results of the evaluated model variants and configurations, taking their tier-specific differences into account. We will also remove speculative explanations about the effects of extended reasoning and avoid extrapolating this comparison to reasoning models in general.

Dear Reviewer j74u,

Thank you again for your thoughtful feedback. We would appreciate it if you could revisit our response during the discussion period. We have added trajectory analyses and clarified the model-improvement implications of our diagnostics.

  • Error recovery and result propagation. Our audit shows that 91.42% of initial source-tool failures were subsequently resolved. We also identified concrete downstream value-binding failures even when the exact source reference and correct output were already available.
  • “What to do” and “when to do it.” We explain how clue adherence, source-node convergence, and premature invocation provide complementary measurements of argument selection and execution readiness.
  • Actionable model improvement. We connect the observed failure patterns to specific training-data targets for mid-training or post-training, including state tracking, clue utilization, intermediate-result propagation, and feedback-driven correction. We also discuss how these directions complement scaffold improvements.

We will additionally scope the reasoning-model comparison to the evaluated variants and configurations and remove speculative explanations.

We would be grateful for your feedback on whether these clarifications address your concerns and welcome any remaining questions. We hope you will consider the additional evidence and clarified scope when updating your assessment.

Best regards, The Authors

Reviewer_xFiTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AgentEscapeBench is a synthetic benchmark for evaluating tool-grounded dependency reasoning in LLM agents. Each instance is generated from a directed acyclic graph of computational tools and items, with narrative clues, incrementally revealed entities, and a deterministically evaluated final answer. The benchmark contains 270 instances across five node-count tiers. The authors evaluate sixteen models and human participants, and analyze intermediate progress, discovery, premature tool invocation, parameter provenance, and interaction efficiency. Results show decreasing success as the required interaction horizon and dependency complexity increase.

Critical strengths
  1. The proposed setting addresses a useful evaluation question: whether agents can coordinate unfamiliar tool compositions, preserve intermediate results, and execute dependencies over extended interactions. A procedurally generated environment with accessible ground-truth structure could support more precise diagnosis than aggregate success rates alone.

  2. The benchmark provides executable interactions, deterministic answer evaluation, partial-progress metrics, and trajectory-level diagnostics. The structural analysis also usefully distinguishes node count, interaction horizon, critical-path length, and concurrent working set, rather than treating graph size as equivalent to dependency depth.

Critical weaknesses
  1. The provided execution trace raises a concrete concern about the semantic validity of the tools and generated tasks. In Figure 5, num_base_convert is documented as converting between bases 2–36. However, Figure 7 invokes it with from_base="22096807214944" and to_base="48331203657939". The call succeeds and returns the original input unchanged, despite both bases being outside the documented domain. The agent explicitly notices this inconsistency before making the call. This conflicts with the claim that conventional tool semantics prevent undocumented behavior from confounding evaluation, and suggests that deterministic forward execution alone does not establish semantic validity. If invalid compositions are accepted through permissive fallback behavior, failures may reflect inconsistent tool contracts rather than dependency reasoning. The benchmark needs an audit of domain constraints and tool behavior, including the frequency and consequences of such cases.

  2. The benchmark does not establish that success requires the dependency inference and long-range propagation claimed by the paper. Appendix A.6 describes implicit parameter provenance and inference from narrative clues, but Appendix E, explicitly presented as the context visible to the model, includes exact annotations such as "[upstream: calc_gcd.gcd]" and "[upstream: calc_bigmul.product]". These annotations directly disclose individual dependency edges. If they are available during evaluation, an important part of the task becomes following explicit references rather than inferring latent dependencies. Moreover, graph edges need not correspond to semantically necessary computation: in Figure 7, the final modular exponentiation has base 1 and returns 1, making its result independent of the positive exponent and the particular modulus shown. Since evaluation accepts the final answer and permits repeated submissions, low-entropy outputs may allow shortcuts around the intended graph. Without annotation-removal controls, shortcut baselines, and checks that designated dependencies actually affect the target, node count and structural complexity cannot reliably establish the reasoning burden being measured.

Questions for authors
  1. Why does num_base_convert accept bases outside its documented 2–36 range and return its input unchanged in Figure 7? Is this the behavior used in the reported evaluation? Please quantify how many instances contain out-of-domain arguments, silent fallback behavior, or other violations of documented tool semantics, and report results after correcting or excluding them.

  2. Are the "[upstream: node.output]" and "[value: ...]" annotations in Appendix E actually visible to evaluated agents, or were they added for presentation? If visible, how does performance change when exact provenance annotations are removed while narrative clues are retained? A baseline that extracts these references and maintains an explicit dependency table would help establish the remaining reasoning difficulty.

  3. How frequently can the final answer be obtained without executing the designated dependency graph, for example through constant-valued computations, irrelevant operands, or a small set of common output guesses? Please evaluate a limited-budget answer-guessing baseline and audit whether the claimed minimum interaction horizon remains necessary under such shortcuts.

  4. Please clarify the human-validation protocol. Sections 2–3 state that every instance was successfully solved by at least one human participant, but Section 3.1 also states that each instance was attempted exactly once, while Table 1 reports human success below 100%. Were separate validation and evaluation cohorts used? Please specify the participant counts, attempts, budgets, and any instance filtering between these stages.

The answers to these questions could materially change my assessment because they determine whether the benchmark implements its stated task and measures the intended capability.

5 comments

1. Out-of-range bases and tool semantics

Thank you for pointing out that the tool description does not fully specify the behavior shown in Figure 7. The out-of-range branch is a deliberate non-rejecting fallback: when either base is outside 2–36, the tool returns the input unchanged instead of rejecting the call. This design preserves data flow and keeps the composed task executable.

Task construction and evaluation use the same deterministic implementation, and the returned value is explicitly observable to the agent. Although the incomplete description may cause uncertainty about the local conversion behavior, it does not block the agent from obtaining an output and continuing the tool-based reasoning needed to solve the task. The agent can use the returned value in subsequent calls, preserve intermediate state, and continue coordinating the remaining tool interactions. The saved trajectories include successful executions involving this fallback.

This behavior preserves executability under the implemented interface, but was insufficiently documented in the tool description. We will update the description to explicitly state the supported range and the fallback behavior.

We audited all 270 released instances and identified 80 instances containing out-of-range designated base arguments, involving 117 converter nodes. We excluded these instances and recomputed all three Table 1 metrics—SR, Sub., and Disc.—for all 16 models and every available difficulty level. The remaining task counts at difficulties 5, 10, 15, 20, and 25 are 59, 51, 37, 35, and 8, respectively.

Each cell below reports SR / Sub. / Disc. (%). Sub. is averaged over all evaluated instances, including unsuccessful ones. ΔSR is the success-rate difference between difficulty 5 and the hardest available tier, in percentage points; “—” indicates an unmeasured tier.

ModelDifficulty 5Difficulty 10Difficulty 15Difficulty 20Difficulty 25ΔSR
Claude-Opus-4.689.8 / 91.9 / 99.184.3 / 87.3 / 93.286.5 / 88.6 / 97.577.1 / 84.4 / 96.2100.0 / 88.0 / 100.0-10.2
Gemini-3.1-Pro-Preview93.2 / 94.2 / 100.084.3 / 88.0 / 94.091.9 / 90.3 / 97.260.0 / 82.0 / 88.712.5 / 62.5 / 75.880.7
GPT-5.496.6 / 91.9 / 99.182.4 / 82.9 / 90.875.7 / 79.8 / 89.348.6 / 68.9 / 77.150.0 / 63.5 / 62.446.6
Kimi-K2.594.9 / 91.9 / 100.082.4 / 84.3 / 90.075.7 / 80.0 / 84.237.1 / 69.0 / 78.616.7 / 57.3 / 70.378.2
DeepSeek-Chat86.4 / 89.2 / 98.274.5 / 83.9 / 90.254.1 / 76.2 / 85.525.7 / 64.9 / 77.4—60.7
DeepSeek-Reasoner78.0 / 89.5 / 99.162.7 / 79.4 / 88.754.1 / 74.8 / 83.628.6 / 68.4 / 84.9—49.4
Gemini-3-Flash-Preview88.1 / 93.2 / 100.064.7 / 82.5 / 89.448.6 / 73.3 / 83.614.3 / 60.7 / 77.9—73.8
Doubao-Seed-2.0-Pro86.4 / 89.2 / 98.564.7 / 75.3 / 84.124.3 / 59.5 / 74.314.7 / 54.7 / 66.1—71.7
GPT-555.9 / 66.4 / 78.941.2 / 62.5 / 65.737.8 / 60.4 / 69.620.0 / 23.1 / 25.3—35.9
MiniMax-M271.2 / 88.1 / 98.251.0 / 74.3 / 84.127.0 / 59.5 / 76.55.7 / 45.0 / 61.4—65.5
Kimi-K250.8 / 54.6 / 62.133.3 / 43.1 / 49.718.9 / 41.3 / 54.78.6 / 34.4 / 48.9—42.3
Qwen3-235B-A22B89.8 / 88.8 / 99.156.9 / 74.5 / 86.110.8 / 46.1 / 63.3——79.0
Qwen3-Next-80B-A3B78.0 / 83.7 / 96.433.3 / 61.6 / 76.68.1 / 43.1 / 61.6——69.9
Qwen3-32B61.0 / 75.3 / 89.119.6 / 48.2 / 54.02.7 / 32.1 / 48.0——58.3
Qwen3-14B50.8 / 65.1 / 84.413.7 / 42.9 / 58.10.0 / 20.7 / 45.3——50.8
Qwen3-8B28.8 / 43.1 / 73.00.0 / 24.5 / 52.20.0 / 18.4 / 47.2——28.8

The table uses available saved trajectories matched to the released instances. The retained sample sizes are as stated above.

After exclusion, all 11 models evaluated at both difficulties 5 and 20 still have lower success at difficulty 20, and all 16 models have lower success at difficulty 15 than at difficulty 5. These results show that the main observed difficulty-related performance decline persists after removing the identified instances.

2. Provenance annotations and the remaining reasoning requirements

The [upstream: ...] and [value: ...] annotations are visible to evaluated agents where applicable. They were deliberately introduced to disambiguate selected inputs when multiple available values share the same type, particularly large integers. Without such disambiguation, identifying the appropriate input can reduce to trying multiple numerical candidates. Our design aims to prevent arbitrary enumeration from becoming the dominant challenge, while retaining the need to reason about tool composition and coordinate execution.

These annotations do not expose the complete dependency structure. Our audit of the 270 released instances shows that only 853 of 3,165 data-flow edges (26.95%) are eligible for exact upstream annotations. There are also 122 instances with no exact upstream annotations at all. Source-node inputs still require inference, and 1,164 of the benchmark's 3,750 nodes are initially hidden behind item/drop interactions. Agents must discover these nodes and coordinate their use as the environment unfolds.

Moreover, identifying an input's source does not compute its value or determine the complete interaction strategy. Agents must obtain the required upstream outputs, preserve intermediate values, and supply them to subsequent tools. Terminal tools additionally support different input combinations with different processing behavior, requiring agents to select and execute the appropriate interactions.

To directly examine whether difficulty persists without exact upstream references, we recomputed performance on the instances that contain none. For the same 11 models evaluated through difficulty 20, the mean success rates are:

Difficulty5101520
Instances with no exact upstream annotations54332410
Mean SR across the same 11 models (%)83.5066.3953.7931.11

All 11 models individually have lower success at difficulty 20 than at difficulty 5 on this subset. Across all 16 models, success at difficulty 15 is also lower than at difficulty 5. This demonstrates that the observed difficulty is not confined to instances carrying exact upstream references.

We additionally measured first-attempt resolution of initial source tools, which receive neither upstream nor initial-value annotations. We evaluated nonterminal source tools, crediting correct parameter feedback, matching inputs, or matching outputs as success. Across the saved difficulty-5–20 trajectories, 1,410/5,403 attempted model–source-tool pairs (26.10%) were not resolved on their first execution. This rate remains 25.03% (927/3,704) after excluding the base-domain instances identified above. These results quantify the remaining challenge of identifying and using source inputs, which provenance annotations do not supply.

Together, these analyses identify substantial work beyond reading references: inferring source inputs, resolving unannotated dependencies, discovering new tools, selecting appropriate tool interactions, and maintaining intermediate values throughout execution. We will clarify the annotation policy and include these quantitative diagnostics in the revised paper.

3. Answer shortcuts, limited-budget guessing, and execution evidence

Thank you for highlighting the need to evaluate answer shortcuts directly. We conducted two limited-budget answer-guessing experiments, recomputed model performance after excluding instances covered by the stronger guessing strategy, and audited the saved trajectories for intermediate progress and answer provenance.

Limited-budget answer-guessing controls. Both strategies use the unchanged submission interface, perform no node investigation or computational tool calls, and stop upon success or after exhausting a budget of K submissions.

  • Cross-fitted frequency baseline: We partition the 270 instances into five folds, stratified by difficulty. For each held-out fold, candidate answers are ranked by their frequency in the other four folds. Thus, each evaluated instance is excluded from the data used to construct its candidate sequence.
  • Oracle frequency control: We rank answers using their frequencies across all 270 instances and submit the same ranked sequence for every instance. This deliberately grants the guessing strategy access to the benchmark-wide answer distribution. Under the exact-match submission rule, selecting the K most frequent answers maximizes coverage among all task-independent fixed lists of K candidates.

Frequency ties are resolved deterministically. The results are:

Submission budget KCross-fitted frequency baselineOracle frequency control
12/270 (0.74%)7/270 (2.59%)
311/270 (4.07%)18/270 (6.67%)
515/270 (5.56%)24/270 (8.89%)
1023/270 (8.52%)37/270 (13.70%)

Even with access to the complete answer-frequency distribution, the optimal task-independent list covers 37/270 instances (13.70%) within ten submissions. Reporting multiple budgets also makes the result independent of a single arbitrarily chosen submission limit.

Performance after excluding guessable instances. We then removed all 37 instances covered by the oracle Top-10 list and recomputed performance from the saved model trajectories. On the remaining instances, all 11 models evaluated at both difficulties 5 and 20 still have lower success at difficulty 20. Across these same models, mean success decreases from 82.16% at difficulty 5 to 32.28% at difficulty 20. All 16 models also retain lower success at difficulty 15 than at difficulty 5. These model-level findings remain unchanged when we additionally exclude every answer tied at the Top-10 cutoff, removing 45 instances in total. Thus, the main difficulty-related performance decline persists after removing the instances covered by this strong guessing control.

Trajectory-level execution evidence. Sub-problem resolution (Sub.) provides complementary evidence of intermediate progress. Across 3,539 saved trajectories at difficulties 5–20, the 1,837 successful trajectories have a mean Sub. of 89.49% and a median of 93.33%. None succeeds with zero resolved nodes.

Furthermore, 1,814/1,837 successful trajectories (98.75%) obtain the accepted answer from the final node before submitting it. We inspected the remaining 23 trajectories individually; they include answer reuse and alternative computation. Requiring this final-node output evidence changes pooled success over the same trajectories from 51.91% to 51.26%, a decrease of 0.65 percentage points.

Together, these analyses quantify common-answer guessing, demonstrate that the difficulty trend persists after excluding the covered instances, and show substantial intermediate execution in successful model trajectories. We will include these controls and clarify that the reported interaction horizon characterizes execution of the designated dependency structure. The guessing experiments separately quantify opportunities to bypass that workflow through answer-only submission.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1 / 2)

We thank the AC and reviewers for their careful assessment of AgentEscapeBench. During the rebuttal, we conducted additional answer-guessing experiments, benchmark audits, and analyses of saved trajectories. We summarize the contribution, new evidence, and planned revisions below; detailed protocols and results appear in our reviewer-specific responses.

1. Contribution and recognized strengths

AgentEscapeBench evaluates whether agents can discover and execute unfamiliar tool workflows while maintaining state and propagating intermediate results across extended interactions. It provides 270 executable instances across five difficulty levels, deterministic evaluation, and trajectory-level diagnostics, with experiments covering 16 LLMs and a human baseline.

The reviewers recognized the executable and controllable benchmark design and breadth of the analysis (j74u), as well as the clear motivation, broad model and human evaluation, and transparency of the prompts and trajectories (UbTE). Our additional analyses strengthen the empirical basis for this contribution.

2. The difficulty-related performance decline persists across three targeted robustness checks

To address xFiT’s concerns about tool semantics, provenance annotations, and answer shortcuts, we performed three separate analyses:

  • Tool semantics. We audited all 270 instances, identified 80 containing out-of-range designated base arguments, and excluded them. We then recomputed success rate, sub-problem resolution, and discovery rate for all 16 models at every available difficulty level. The difficulty-related decline persists. We also clarified the implemented deterministic fallback and will explicitly document its behavior in the tool description.

  • Provenance annotations. Exact upstream annotations cover only part of the dependency structure: 122 instances contain no such annotations. On this subset, mean success across the same 11 models evaluated through difficulty 20 decreases from 83.50% at difficulty 5 to 31.11% at difficulty 20. Thus, the observed difficulty also occurs in workflows without exact upstream references.

  • Limited-budget answer guessing. We evaluated a cross-fitted frequency baseline and an oracle frequency control through the unchanged submission interface. With ten submissions, they solve 8.52% and 13.70% of instances, respectively. The oracle control uses the full answer distribution and maximizes coverage among task-independent fixed lists of ten candidates. After excluding all 37 instances it solves, mean success across the same 11 models still decreases from 82.16% to 32.28% between difficulties 5 and 20.

In each of these three separate analyses, all 11 models evaluated at both difficulties 5 and 20 retain lower success at difficulty 20.

Trajectory evidence further characterizes successful execution: among 1,837 successful trajectories at difficulties 5–20, mean sub-problem resolution is 89.49%, and 98.75% obtain the accepted answer from the final node before submission. These results document substantial intermediate execution alongside the answer-guessing controls.

AuthorsOpenReview ↗

General Response (Part 2 / 2)

3. The diagnostics identify concrete targets for improving long-horizon tool use

In response to j74u and UbTE, we clarify the relationship between structural predictors and observed execution behavior. Interaction horizon is the strongest structural predictor examined. We will state the central finding precisely: agent reliability decreases over longer sequences of tool interactions with interleaved dependencies, with trajectory analysis identifying difficulties in maintaining execution state, respecting dependency readiness, and propagating intermediate results.

Additional audits provide direct behavioral evidence. Among initial nonterminal source tools unresolved on their first execution, 91.42% were subsequently resolved, showing that first-attempt accuracy alone does not characterize the interactive correction process. Separately, we identified 51 unsuccessful value-binding deviations among 1,932 eligible uses, even after the exact source reference and correct source output were available.

These observations motivate specific training-data targets: maintaining evolving task state, binding intermediate outputs to downstream inputs, and correcting actions using feedback. We will explain how verified trajectories exhibiting these behaviors could support mid-training or post-training of the underlying model capabilities, with the diagnostics measuring subsequent improvement. These proposed training directions complement state stores and dependency-aware scaffolds.

4. We will clarify the human protocol and align the presentation with the evidence

For xFiT, we distinguish the 10-participant solvability-validation cohort from the five additional participants in the baseline evaluation. Every instance had a demonstrated human solution during validation; the baseline separately measured one attempt per instance. Its 270-attempt design prioritized full benchmark coverage within the available time and annotation budget; having all five participants attempt every instance would require 1,350 attempts.

For j74u, we will restrict the reasoning-model comparison to the evaluated variants and configurations and remove speculative explanations. For UbTE, we will correct the ranking and monotonicity statements and systematically check textual comparisons against the tables. These revisions will make the scope and interpretation of the findings explicit and consistent throughout the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.