acceptodds
Under review as a conference paper at ICLR 2027

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Abstract

Large language models (LLMs) enable agents to solve long-horizon tasks by generating a plan and then executing it in an environment. However, successful planning requires two distinct capabilities: selecting an appropriate plan for the task and executing it faithfully. Existing planner–executor systems can fail at either stage: the agent may deviate from a structured plan during execution, or it may faithfully execute a plan that is poorly matched to the task or environment. Final task success alone cannot distinguish between these two sources of failure. We therefore ask: can LLM agents be trusted to execute the plans they commit to, and do different tasks and environments benefit from different planning modes? To study this, we introduce a diagnostic framework for the *Plan Declaration–Execution Gap* and propose *Planning-as-Routing*, in which the LLM declares one of four planning modes: *{Predefined, Sequential, Hierarchical, or Search}*. A deterministic router sends the task to the corresponding pattern-specific executor. Across four benchmarks and three LLMs, we find three consistent patterns. First, generic Plan+ReAct often fails to preserve the declared planning structure, especially for longer plans: across three benchmarks, only \(22\)–\(45%\) of trajectories preserve it, whereas pattern-specific executors enforce the intended structure. Second, planning-mode effectiveness varies across environments and models: Search performs best on ALFWorld, Hierarchical on SWE-bench, and the strongest pattern can vary across models within the same benchmark. Third, the largest gains come from plan execution: pattern-specific executors raise task success from \(0.48\) to \(0.92\) on ALFWorld and from \(0.36\) to \(0.44\) on SWE-bench Verified over Plan+ReAct. However, current LLMs do not reliably select the strongest planning mode for a task, while few-shot examples can improve planning-mode selection for some benchmark–model combinations. Overall, reliable agent planning requires both selecting an effective planning mode and executing it with a matching executor: routing substantially closes the execution gap, while selecting the right mode for each task remains open.

Primary area:
applications to robotics, autonomy, planning
Keywords:
agentic planning, Planning with Language Models, llm-based agents, plan execution, plan declaration, planning patterns

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_GcEMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the Plan Declaration–Execution Gap in LLM agents—the discrepancy between declaring a structured plan and faithfully executing it. The authors categorize agent planning into four modes inspired by cognitive science: Predefined (plan-once), Sequential (step-by-step replanning), Hierarchical (orchestrator–worker tree decomposition), and Search (parallel candidate rollouts selected by a rubric judge). They compare three setups across four benchmarks (ALFWorld, Mind2Web, WebArena, SWE-bench Verified) and three backbone LLMs (Qwen3.6-35B-A3B, DeepSeek-V4-Flash, Gemma-4-26B): (1) Flat ReAct, (2) Plan+ReAct (plan passed in context), and (3) Planning-as-Routing, where the LLM declares a planning mode and a deterministic router dispatches the task to a corresponding pattern-specific executor. Through extensive experiments and control baselines, the authors find that:

  1. Generic Plan+ReAct fails to preserve declared plan structures on 55–78% of trajectories, especially as plan length increases or when hierarchical/search plans are declared.
  2. Pattern-specific executors substantially improve task success over Plan+ReAct (e.g., from 0.480 to 0.918 on ALFWorld, and 0.360 to 0.442 on SWE-bench Verified), though the single best executor varies by model and benchmark.
  3. Crucially, the paper's own controls in Appendices J and L reveal two major negative findings: (a) once multi-rollout stochasticity is controlled via 3×3\times retries of the single strongest fixed pattern (pass@3(p⋆)\text{pass@3}(p^\star)), the per-task oracle routing headroom (Hretry@3H_{\text{retry}@3}) nearly vanishes across benchmarks (−0.025-0.025 to +0.027+0.027), and (b) zero-shot LLM mode declarations perform no better than a task-blind random permutation null (p≥0.094p \ge 0.094). Few-shot demonstrations improve top-1 routing performance primarily by shifting global mode preferences rather than achieving reliable per-task discrimination.
Critical strengths
  1. Rather than overclaiming the superiority of Planning-as-Routing or the Search executor, the authors include: (i) a distribution-matched permutation null (Table 18) proving that zero-shot LLM routing declarations have no statistically significant task-specific discriminative power over a task-blind prior, (ii) a retry-matched baseline (Table 17) showing that repeating the single best fixed pattern 3 times (pass@3(p⋆)\text{pass@3}(p^\star)) matches or exceeds the 4-pattern per-task oracle (DSRmax⁡\text{DSR}_{\max}), and (iii) a 3×3\times Flat ReAct + Judge control (Table 16) to isolate the multi-rollout advantage of Search. This level of diagnostic honesty is rare in agent papers and provides genuinely useful insights for agent evaluation.
  2. Evaluating four distinct agent architectures across four major benchmarks (ALFWorld, Mind2Web, WebArena, SWE-bench Verified) and three models across multiple seeds is a substantial empirical undertaking. The finding that intrinsic LLM-judged plan quality has almost zero correlation with downstream task success (∣r∣≤0.181|r| \le 0.181, Table 15), whereas enforcing execution control flow yields large gains on long-horizon tasks (Figure 6), is an important takeaway for practitioners designing agentic scaffolds.
Critical weaknesses
  1. The paper frames Planning-as-Routing as a solution to match individual tasks to appropriate planning modes. However, Table 3 shows that Routing@1 consistently underperforms simply picking the single best fixed executor (Best Fixed S⋆S^\star) on 10 out of 11 benchmark–model pairs (e.g., 0.721 vs. 0.918 on ALFWorld, 0.460 vs. 0.580 on WebArena). More importantly, Table 17 shows that even with an Oracle router, the headroom over simply retrying the best fixed pattern (Hretry@3H_{\text{retry}@3}) is effectively zero (−0.025-0.025 to +0.027+0.027). Furthermore, Table 18 demonstrates that the LLM router cannot discriminate at the task level (Δ(π)≈0\Delta(\pi) \approx 0). Even the few-shot gains (Table 4) are acknowledged in Appendix L (line 1683) to "improve raw top-1 performance, but rarely task-specific discrimination"—meaning few-shot examples primarily teach the router to favor the globally dominant executor. Thus, the empirical evidence does not support per-task routing; it shows that practitioners should select the best fixed scaffold for a given domain/model and allocate budget to retries.
  2. In RQ1, the authors claim that generic Plan+ReAct fails to preserve declared structures (particularly Search and Hierarchical), whereas pattern-specific executors enforce them "by construction." This comparison is confounded in three ways:
  • First, asking a single-thread, non-backtracking Plan+ReAct loop to execute a Search plan (which requires independent environment resets and competing rollouts) is impossible without an environment-reset tool. In Appendix E.2 (page 21), the "Search" trace under Plan+ReAct simply concatenates Path 1 and Path 2 sequentially in the same episode. Penalizing Plan+ReAct for not doing multi-episode search when it lacks the harness mechanism to reset the environment measures a harness limitation rather than a model failure.
  • Second, the pattern-specific executors explicitly instruct the worker LLM to ignore the declared plan step if needed (e.g., Appendix O.2.1, line 1995: "If the subtask does not match what is needed, IGNORE it and do what the task needs"), yet they are credited with 100% structural fidelity "by construction" simply because the outer Python loop dispatches subtask prompts in order, while Plan+ReAct is scored by an action-matching verifier.
  • Third, the rule-based verifier used on Plan+ReAct has low agreement with human annotators (Cohen's κ=0.31\kappa = 0.31 and 0.340.34, Section 5 line 315) and human–human agreement is similarly low (κ=0.35\kappa = 0.35, and κ=0.19\kappa = 0.19 on Qwen3.6), raising concerns about the reliability of the exact structural maintenance percentages in Table 1.
Questions for authors
  1. Given that Hretry@3H_{\text{retry}@3} is near zero across almost all settings (Table 17)—meaning 3 retries of the best fixed pattern matches or beats the 4-pattern Oracle—and per-task declarations show no advantage over the task-blind permutation null (Table 18), why is per-task Planning-as-Routing preferable to simply selecting the best domain-level scaffold and using retries? Would the paper be stronger if reframed around this empirical finding rather than pitching task-level routing as the core solution?
  2. In Appendix O.2.1 (line 1995) and O.2.3 (line 2140), the Predefined and Hierarchical executors are instructed: "If the subtask/atomic action does not match what is needed, IGNORE it and do what the task needs." If a pattern-specific leaf worker ignores the declared subtask and executes different actions, why is it considered to preserve structure "by construction," whereas Plan+ReAct doing the same is penalized by the rule-based action matcher? What are the structural maintenance scores if the exact same rule-based action verifier is applied to the action streams of the pattern-specific executors?
  3. Why was test execution (pytest) blocked in SWE-bench Verified, and how might this restriction bias the comparison against feedback-driven modes like Sequential and Flat ReAct? Additionally, why does Example 7 in the SWE-bench few-shot declaration prompt (Appendix O.4, lines 2678–2688) instruct the agent to create and execute unit tests when test execution is blocked in the environment?
10 comments

We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.

W1/Q1. why is per-task Planning-as-Routing preferable to simply selecting the best domain-level scaffold and using retries? Would the paper be stronger if reframed around this empirical finding rather than pitching task-level routing as the core solution?

We thank the reviewer for this valuable suggestion. We now reframed the paper around this finding. The revised manuscript states it directly:

  • Abstract (lines 39–40): "pattern-specific execution substantially closes the execution gap, while selecting the right mode for each task remains open."

  • Introduction (lines 133–134): current declarations "trail the best fixed planning pattern on all benchmark–model pairs except Gemma-4-26B on WebArena, and show no reliable task-specific advantage over task-blind mode selection."

  • Section 5, RQ3 (lines 432–434): "the gains come from the execution architecture rather than automatic task-level mode selection."

  • Discussion: "For current models, however, task-level planning-mode selection does not consistently outperform selecting the strongest fixed pattern for each benchmark–model pair and retrying it."

  • A fixed scaffold with retries itself requires Planning-as-Routing.

    • A domain-level scaffold presupposes knowing the best pattern, which is never known in advance: it changes across benchmarks and models (Search for DeepSeek-V4 on ALFWorld, Hierarchical on SWE-bench, Predefined for Gemma-4 on Mind2Web; Table 2) and can only be identified by dispatching every pattern to its matching executor, which is exactly the forced-dispatch measurement
    • Planning-as-Routing provides. The retries must also run on the pattern-specific executor: a generic loop cannot execute Search at all and preserves Hierarchical plans in only 0.8–11.4% of ALFWorld episodes (Table 15). The strong fixed-pattern-with-retries result is therefore a result of Planning-as-Routing, not an alternative to it.
  • Where per-task routing adds value beyond a fixed scaffold. Three conditions, each measured in the paper:

    • On Gemma-4/WebArena, Routing@1 reaches 0.432 vs. 0.378 for the strongest fixed pattern, with significant task-specific matching (Δ = +0.051, Holm p = 0.042; Table 24); about 94% of its advantage comes from matching modes to tasks.
    • Better plan-selectors pay off immediately. Few-shot demonstrations already raise Routing@1 by up to +0.16 by steering models toward stronger modes; because every declaration runs on its matching executor, each improvement in selection converts directly into task success.
    • Executors differ substantially in cost (Appendix M, Figures 9–10); retries multiply the cost of the strongest pattern, while the forced-dispatch matrix allows cost-aware selection policies to be evaluated without re-running agents.
  • Role of Planning-as-Routing.

    • Its contribution is the separation itself: by executing each declared mode with its matching executor, it measures execution failures and selection failures independently. This separation is what reveals that execution, not selection, drives current gains.
    • As mode selection improves, the same framework translates better declarations directly into pattern-specific execution, evaluated against retry-matched fixed baselines (Table 22).

W2.1.. Does penalizing Plan+ReAct for Search-plan failures conflate harness limitations with model failures?

We thank the reviewer for this observation, and we agree.

  • Plan+ReAct runs within a single episode without a reset mechanism, so it cannot implement a Search plan that requires independent rollouts.
  • The Search example in Appendix E.2 illustrates this limitation: Plan+ReAct explores alternatives within the same episode rather than executing independent candidate trajectories.
  • We therefore distinguish executor-level capability limitations from model-level deviations. This is precisely our central claim: declaring a planning pattern is not sufficient without the matching execution architecture, and Search is the extreme case, where the required control structure cannot be expressed in a generic loop at all.
  • Importantly, our evidence of the declaration--execution gap does not depend on Search.
    • Search accounts for at most about 4% of Plan+ReAct declarations in every pair reported in Appendix G, Table 15 (e.g., 5 of 134 ALFWorld tasks per seed for DeepSeek-V4, 3 of 1,341 on Mind2Web, and none for Qwen3.6 and Gemma-4 on ALFWorld), so excluding it changes overall maintenance in Table 1 by at most 1.2 percentage points.
    • Predefined, Sequential and Hierarchical plans are fully executable within a single episode, so their violations are genuine deviations: Hierarchical declarations under Plan+ReAct maintain their structure in only 11.4%, 4.6% and 0.8% of ALFWorld episodes for DeepSeek-V4, Qwen3.6 and Gemma-4.

W2.2/Q2 What are the structural maintenance scores if the exact same rule-based action verifier is applied to the action streams of the pattern-specific executors?

We thank the reviewer for this question, which helps us to distinguish two levels of structural fidelity more clearly.

  • Dispatch-level fidelity. In pattern-specific execution, the controller maintains the prescribed planning organization. For Predefined, the declared plan remains fixed and subtasks are dispatched in order. For Hierarchical, the controller preserves the parent–child organization and dispatches the corresponding leaf subtasks. Importantly, a worker executing alternative actions does not overwrite or reorder the controller's declared plan. Thus, Structural preservation “by construction” therefore refers to the order and organization of dispatched plan units, not necessarily to the actions performed while executing them.

  • Action-level fidelity. We agree that action-level maintenance is not guaranteed. the agent may perform actions that differ from its assigned subtask while the controller maintains the declared structure. To evaluate this consistently, we applied the identical rule-based verifier used for Plan+ReAct (same step cleaning, action-matching rules and in-order criterion) to the actual action streams of the pattern-specific executors.

Appendix E.4 Table 14 Episode-level structure maintenance

BenchmarkModelPredefinedSequentialHierarchicalPlan+ReAct
ALFWorldDeepSeek-V444.0%62.0%49.6%23.1%
ALFWorldQwen3.647.8%69.2%56.2%20.5%
ALFWorldGemma-457.5%61.1%57.0%19.6%
Mind2WebDeepSeek-V463.6%63.2%84.3%45.8%
Mind2WebQwen3.642.6%51.8%67.5%36.1%
Mind2WebGemma-444.2%52.7%75.6%28.4%
WebArenaDeepSeek-V482.6%88.6%97.6%67.5%
WebArenaQwen3.673.7%85.9%89.2%42.0%
WebArenaGemma-484.2%91.2%97.7%61.4%
  • Under the same verifier, all three pattern-specific executors maintain the declared structure more often than Plan+ReAct, in every benchmark–model pair.

    • On ALFWorld, maintenance rises from 19.6–23.1% under Plan+ReAct to 44.0–69.2%. On Mind2Web, Predefined and Sequential improve on it by 6.5–24.3 percentage points; on WebArena, by 15.1–43.9 points.
    • The execution gap reported in RQ1 therefore holds when all executors are measured with the same instrument. The improvement is largest where generic execution fails most: for Gemma-4 on ALFWorld, maintenance rises from 19.6% to 57.5% under Predefined, which is also where Predefined’s success gain is largest (+0.362).
  • Why does action-level maintenance remain below 100%? Two mechanisms contribute.

    • First, the agent may perform actions that do not correspond to the currently assigned plan unit, without modifying the declared plan itself. This is deliberate: plans are written before the environment is observed, so some declared steps turn out to be infeasible.
    • Second, the execution budget may run out before all declared subtasks are executed. Because the verifier requires every scoreable plan unit to be matched in order, both deviations and incomplete execution lower the reported rate.
  • Importantly, an unmatched action is not an incorrect action, and an incomplete plan does not mean its executed steps were wrong; we therefore report dispatch-level fidelity, action-level maintenance, plan completion and task success as separate properties (Appendix E.4, Tables 13–14; refer common responses to CQ2).

W2.3 the rule-based verifier used on Plan+ReAct has low agreement with human annotators

  • We have provided a detailed discussion in our responses to CQ1.1 and CQ1.2. Kindly refer to those responses for more comprehensive information.

Q3.1 Why was pytest blocked in SWE-bench Verified, and could this disadvantage feedback-driven executors?

Why test execution was blocked.

  • Our SWE-bench Verified setup does not use per-instance Docker environments. Since instances require different dependency versions, running tests could require runtime package installation, introducing additional overhead and environment inconsistencies. We therefore disabled pip, pytest, and conda for all executors, including Flat ReAct and Plan+ReAct. Final patches are evaluated separately using the SWE-bench Verified evaluation harness, which determines patch success (PSR). We will clarify this distinction in the manuscript. Appendix D now states this distinction between the agent runtime and the evaluation environment.

Potential bias.

  • The restriction applies identically to all executors, and the results show no systematic penalty for feedback-driven modes: with DeepSeek-V4, Flat ReAct outperforms Plan+ReAct (0.384 vs. 0.360) and Sequential outperforms Predefined (0.395 vs. 0.372). Our SWE-bench results characterize planning patterns under static code inspection, and the revised manuscript states this setting explicitly.

Q3.2 why does Example 7 in the SWE-bench few-shot declaration prompt (Appendix O.4, lines 2678–2688) instruct the agent to create and execute unit tests when test execution is blocked in the environment?

We thank reviewer for this question.

  • Example 7. The few-shot prompt only selects the planning mode: the router outputs a mode ranking, and the demonstration plans are never executed or passed to the executors. Example 7 reproduces verbatim the plan that the Hierarchical planner produced for a held-out task, to illustrate that decomposition style.
  • Once a mode is declared, the matching executor generates its own plan for the new task, in the same runtime where test execution is blocked. Appendix O.4 now states this.
Reviewer_bjZsOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gap between the planning approach declared by an LLM agent and its actual execution, distinguishing between the selection of a planning mode and the execution of that plan. The authors propose "Planning-as-Routing," a framework that maps a declared planning mode to its corresponding executor. Experiments demonstrate that specialized executors can improve task performance; however, the model’s ability to select an appropriate planning mode for a given task remains limited.

Critical strengths
  • The research addresses a valuable question. By separately examining planning choices and the consistency of their execution, the study sheds light on agent behaviors that are often obscured when looking solely at final task success rates, offering a clear perspective on why planning fails.

  • The methodology is sound, and the experiments are sufficiently comprehensive. The framework is intuitive; the evaluation spans various models and real-world agent tasks, incorporating both retry-based comparisons and baselines for task-agnostic action selection. Furthermore, the authors’ transparent reporting of negative findings, such as the limited benefits of routing, enhances the credibility of their conclusions.

Critical weaknesses
  • The sources of performance gains have not been fully disentangled. Although various control experiments have been conducted, the use of specialized executors simultaneously alters task decomposition, context organization, and search mechanisms; consequently, it is difficult to attribute the improvements solely to a more faithful execution of the plan.

  • The advantages of adaptive routing remain unclear. Current mode-selection strategies often underperform relative to an optimal fixed-mode baseline, and control experiments involving retries further weaken the evidence for the complementarity of these modes. While the paper acknowledges this limitation, the practical utility and specific value of such routing mechanisms require more precise definition.

Questions for authors
  1. Building upon existing baselines, how can we further disentangle the performance gains attributable to planning constraints themselves from those resulting from other design aspects of the executor?

  2. Do the improvements derived from few-shot examples reflect better task-to-pattern matching, or do they primarily shift the overall pattern preference? Which types of tasks are most likely to benefit from adaptive routing?

I will consider increasing the score for this work, depending on the response provided in the rebuttal.

4 comments

We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.

W1/Q1.1 Building upon existing baselines, how can we further disentangle the performance gains attributable to planning constraints themselves from those resulting from other design aspects of the executor ?

We thank the reviewer for this question. We answer it with evidence from the existing baselines and with a new controlled ablation that isolates the planning constraint directly (Appendix J.4, Table 23).

Our existing baselines already provide four pieces of evidence.

  • First, the gains are largest where generic execution frequently breaks the plan.
    • Comparing Predefined with Plan+ReAct minimizes confounding from additional mechanisms such as hierarchical decomposition, replanning and multi-candidate search.
    • Across 12 benchmark-model pairs, Predefined’s improvement is strongly negatively correlated with the share of Plan+ReAct trajectories that preserve their declared structure (Spearman =-0.76, p=0.006).
    • This relationship suggests that structural enforcement is an important contributor to the observed improvements, particularly when generic execution frequently deviates from the declared plan.
  • Second, the improvements do not come from additional environment interactions.
    • On Mind2Web with Gemma-4-26B, Predefined raises task success from 0.043 to 0.147 while reducing environment interactions from 6.76 to 2.34 per task.
    • On ALFWorld with DeepSeek-V4, it raises success from 0.480 to 0.556 at essentially the same interaction count (12.53 vs. 12.55).
  • Third, extra attempts and trajectory selection do not explain Search’s advantage (Appendix J.1, Table 21).
    • On ALFWorld with DeepSeek-V4, Search reaches 0.918, compared with 0.526 for 3×Flat ReAct+Judge, which uses the same judge. It also exceeds Flat ReAct pass@3 (0.600), a hindsight oracle that counts a task as successful whenever at least one of the three attempts succeeds.
    • This suggests that Search's advantage also comes from generating stronger candidate trajectories, rather than simply selecting among additional generic attempts.
  • Fourth, our process-level metrics further distinguish planning quality, structural enforcement, and execution completion.
    • Beyond task success, we separately evaluate plan quality, plan adherence, full-plan completion, and plan-order faithfulness (Section 3.3, Tables 1–2, and Appendix I).
    • On ALFWorld with DeepSeek-V4, Sequential completes all its declared plan units (100% adherence), whereas Hierarchical completes 81.2% of its executable leaf nodes, despite achieving higher task success (0.840 versus 0.633).
    • Moreover, judged plan quality is only weakly correlated with task success ((|r|\leq 0.181)).

These results demonstrate that generating a high-quality plan, preserving its execution structure, completing its steps, and achieving task success are distinct properties. Together, these metrics provide complementary measurements for analyzing whether improvements arise from planning constraints or other executor mechanisms.

W1/Q1.2 How to separate the effects further, building on these baselines.?

  • Controlled 2×2 ablation. To isolate the constraint itself, we varied two factors independently on ALFWorld with Qwen3.6-35B (134 tasks):
    • executor architecture (generic ReAct vs. Predefined) and
    • structural enforcement (disabled vs. enabled).
    • All four conditions use the same generated plans and the same execution budget, so absolute values differ from Table 2.
    • CIs are paired over the 134 tasks (Table 23).
ExecutorEnforcement offEnforcement onEnforcement effect [95% CI]
Generic ReAct0.4930.590+0.097 [+0.008, +0.187]
Predefined0.4400.619+0.179 [+0.082, +0.276]
Architecture effect [95% CI]−0.052 [−0.142, +0.037]+0.030 [−0.045, +0.105]Interaction: +0.082 [−0.037, +0.209]
  • Structural enforcement significantly improves success under both architectures: +0.097 for generic ReAct and +0.179 for Predefined; both CIs exclude zero.
  • Architecture alone has no significant effect once enforcement is held fixed (−0.052 and +0.030), and the interaction is not significant.

The planning constraint is therefore a direct, measurable source of the gains from pattern-specific execution, and the additional mechanisms (replanning, decomposition, candidate generation) contribute on top of it, as the executor-level comparisons and the Search decomposition show (see response to Reviewer bh33 W3). Section 5 (RQ3, lines 440–442) reports this ablation.

W2/Q2.1 Do the improvements derived from few-shot examples reflect better task-to-pattern matching, or do they primarily shift the overall pattern preference?

We thank the reviewer for these insightful questions. We extended the task-blind permutation analysis to few-shot declarations and compared zero-shot and few-shot policies on identical tasks (Appendix L, Tables 25–27), decomposing each paired gain into a shift in overall mode preference (Δnull) and a change in task-specific matching (ΔΔ).

  • Few-shot gains primarily shift the overall pattern preference, with one clear exception.
    • ALFWorld: +0.026 of DeepSeek-V4's +0.056 gain and +0.030 of Qwen3.6's +0.050 gain come from preference shifts; the task-specific components (+0.029, +0.020) are not significant after Holm correction.
    • Mind2Web and SWE-bench: both components are negligible. On Mind2Web, only about 9% of tasks have different outcomes across planning modes, which bounds what any selector can gain; Qwen3.6 shows a small significant task-specific component in step success (+0.008 SSR, Holm p = 0.013).
    • WebArena/DeepSeek-V4 is the exception: +0.103 of its +0.122 gain (84%) is task-specific matching (Holm p = 0.002; Table 25). Here, demonstrations teach the model which mode to use for which task.

Table 26; Holm correction across the rows of this table. WebArena and SWE-bench pairs are reported in Table 25

Benchmark / ModelGain [95% CI]ΔnullΔΔHolm p (ΔΔ)
ALFWorld TSR / DeepSeek-V4+0.056 [+0.013, +0.098]+0.026+0.0290.260
ALFWorld TSR / Qwen3.6-35B+0.050 [+0.013, +0.085]+0.030+0.0200.470
Mind2Web TSR / DeepSeek-V4−0.003 [−0.009, +0.002]−0.001−0.0021.000
Mind2Web TSR / Qwen3.6-35B+0.001 [−0.004, +0.006]+0.003−0.0021.000
Mind2Web SSR / DeepSeek-V4−0.002 [−0.008, +0.003]−0.003+0.0001.000
Mind2Web SSR / Qwen3.6-35B+0.018 [+0.012, +0.024]+0.010+0.0080.013

W2/Q2.2 Which task categories benefit most from adaptive routing?

We additionally analyzed ALFWorld task categories and Mind2Web generalization splits, comparing few-shot routing against zero-shot routing, task-blind preferences, and the strongest fixed planning pattern.

  • Routing helps where no single pattern dominates and pattern successes are complementary across tasks:
    • Flat pattern profiles. Gemma-4-26B on WebArena, whose four patterns lie within 0.09 of each other, is the one pair where zero-shot declarations beat the best fixed pattern (0.432 vs. 0.378), with significant task-specific matching (Δ = +0.051, Holm p = 0.042; Table 24).
    • Categories with complementary patterns.
      • On ALFWorld, Pick Two & Place shows the clearest complementarity: for DeepSeek-V4, the forced-pattern oracle reaches 0.914 vs. 0.829 for the strongest fixed pattern (Hierarchical), and few-shot routing (0.783) is statistically indistinguishable from it. Clean & Place shows a positive task-specific component (+0.073) that does not survive Holm correction.
      • On Mind2Web, Qwen3.6-35B has a significant task-specific improvement in the Cross-Domain split (+0.010 SSR, Holm p = 0.008).
    • Large few-shot gains do not imply adaptive routing.
      • Heat & Place gains the most from few-shot prompting (0.427 → 0.672 for DeepSeek-V4), but 82% of that gain (+0.201) is a preference shift toward Search (6% → 48% of declarations), and fixed Search remains stronger (0.819).

Table 27. Few-shot routing by ALFWorld task category

Task categoryModelZero-shot → Few-shotΔnull / ΔΔFew-shot − Best Fixed [95% CI]
Heat & PlaceDeepSeek-V40.427 → 0.672+0.201 / +0.044−0.147 [−0.231, −0.065]
Heat & PlaceQwen3.6-35B0.450 → 0.480+0.050 / −0.020−0.193 [−0.312, −0.074]
Pick Two & PlaceDeepSeek-V40.742 → 0.783+0.010 / +0.032−0.046 [−0.156, +0.044]
Pick Two & PlaceQwen3.6-35B0.569 → 0.664+0.058 / +0.038−0.145 [−0.268, −0.037]
Clean & PlaceDeepSeek-V40.751 → 0.876+0.052 / +0.073−0.079 [−0.136, −0.033]
Clean & PlaceQwen3.6-35B0.683 → 0.776+0.056 / +0.038−0.012 [−0.084, +0.060]

Overall, no category yet shows a statistically established advantage of few-shot routing over the strongest fixed pattern. Adaptive routing is most valuable for task distributions with flat pattern profiles and complementary pattern successes, and the task-blind decomposition is the tool that identifies them. Section 5 (RQ4) and Appendix L.1 (Tables 26–27) report these analyses.

Thanks! I've increased my score. GOOD LUCK!

We sincerely thank the reviewer for the positive feedback and for raising their rating, and we greatly appreciate their time and consideration.

Reviewer_m6nhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study explores whether language model agents actually follow the plans they say they will and whether their performance improves when their execution matches the planning approach they choose. It looks at a standard ReAct agent against agents that use a pre-defined sequential, hierarchical or search-based execution. Across four benchmarks and three models they find that the standard ReAct agent frequently strays from the plan it has declared and that performance can be improved by matching the executor to the planning pattern. They also find that the models don't consistently pick the best planning pattern for a given task.

Critical strengths

The paper decouples plan selection from plan execution. It's comparing Plan+ReAct to pattern-specific executors to see if a plan performs better when the executor actually follows the plan. The results across ALFWorld, Mind2Web, WebArena and SWE-bench Verified using three models and three random seeds provide a good breadth of coverage.

The authors go beyond just looking at whether the agent has completed the task. They look at whether the actions taken are in accordance with the stated steps, whether the plan is appropriate for the task and whether the task in fact succeeds. The forced pattern analysis is useful in differentiating between a poor executor and a poor choice of planning mode. The paper also reports on model calls and token usage and includes a repeated ReAct control for the multi-candidate Search executor.

The analysis points to a significant weakness in the proposed routing approach. Model declarations don't reliably pick the best planning pattern and the improvement from using few-shot examples is highly inconsistent across the different benchmarks. That's a far more complete picture than just reporting the benefit of the best fixed executor.

Critical weaknesses

The central measure of plan adherence used in the paper has very poor agreement with human judgement. The rule based verifier has κ = 0.31 and 0.34 with the human annotators respectively. Human-human agreement is also low at κ = 0.35. The reported plan preservation rates are therefore very difficult to interpret. The paper needs to validate the measure on a larger sample, report an adjudicated error analysis and demonstrate that its main conclusions are robust under alternative matching rules.

Structural adherence doesn't establish correct execution. The verifier simply checks that the declared plan steps are in order (allowing for any number of extra actions in between). It makes no assessment of whether those actions are correct or whether they in fact modify the environment. Routed executors are not subject to the same verifier as they have their dispatch logs enforcing the order of actions by design. The paper should state its primary finding as being improved preservation of plan structure and report action correctness and task success as separate metrics.

The abstract could in fact tell the difference between fixed pattern performance and routing performance. On ALFWorld with DeepSeek-V4 we have the best fixed pattern at 0.918 task success, Routing@1 at 0.721 and Plan+ReAct at 0.480. The 0.918 is clearly the value of having a very good pattern specific executor. It does not indicate that the model is in any way reliably routing tasks to that executor. The same gap is evident on WebArena. The abstract and the conclusions should report these results separately.

The performance comparison really does need a clearer accounting of execution budget. Search comes in at about 3.35 candidates per task and hierarchical execution uses more model calls than the simpler patterns. The paper does provide cost analysis and a repeated ReAct control for Search so that at least that element is covered. Can the authors also provide a consistent comparison of success against total model calls, total generated tokens and total environment interactions for each of the patterns? That would tell us whether the improvement is due to the structure of the execution, the extra attempts or both.

The suite is quite broad actually but the results don't really support any one general ranking of the planning modes. The best mode is very much benchmark and model dependent and the metrics vary from task to task. The paper should therefore be content to make statements at the benchmark/model level and not suggest that one mode is in general the best. It should also report the uncertainty on the few shot gains which range from almost zero to 0.156 depending on the setting.

Questions for authors

Apart from my points enumerated under "Weaknesses", I have few additional questions:

  1. The verifier is key to the paper's assertion that generic Plan+ReAct frequently strays from its stated plan. Can you validate this on a larger, independently sampled set of trajectories? Can you report confidence intervals and have the annotators resolve the disagreements?

  2. Can you report for each executor how often its actions are valid and result in the desired state changes? Please separate that out from whether the executor maintains the declared plan order and whether the task ultimately succeeds.

  3. For DeepSeek-V4 the abstract reports ALFWorld success at 0.92 and SWE-bench success at 0.44. Table 3 shows these are the best fixed pattern results. The model selected Routing@1 results are 0.721 and 0.415. Can you please report both sets of results clearly in the abstract so that the reader can see the benefit of the best executor versus the benefit of the automatic mode selection?

  4. The executors use vastly different amounts of computation (particularly Search and Hierarchical). Can you compare their task success at matched inference budgets, or provide success versus cost curves using model calls, generated tokens and environment interactions?

  5. Few-shot mode selection yields gains very close to zero in a number of cases (some of the results are in fact slightly negative). Can you report the paired confidence intervals across the same evaluation tasks and seeds and tell us which of the gains are actually different from zero?

4 comments

We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.

W1/Q1. Can you report confidence intervals and have the annotators resolve the disagreements? Can you validate this on a larger, independently sampled set of trajectories?

Thank you for this interesting question.

  • We have provided a detailed discussion in our responses to CQ1.1 and CQ1.2. Kindly refer to those responses for more comprehensive information.

W2/Q2. Can you report for each executor how often its actions are valid and result in the desired state changes?

  • We have provided a detailed discussion in our responses to CQ2. Kindly refer to those responses for more comprehensive information.

W3/Q3. Can you distinguish best-fixed executor performance from automatic Routing@1 performance in the abstract?

  • We have revised the abstract to clearly distinguish the performance of the best fixed executors from that of automatic planning-mode selection (Routing@1).
  • The revised sentence reads: “Third, the largest gains come from pattern-specific execution: for DeepSeek-V4, the best fixed executors improve task success from 0.48 to 0.92 on ALFWorld and from 0.36 to 0.44 on SWE-bench Verified over Plan+ReAct, whereas automatic planning-mode selection (Routing@1) achieves (0.721) and (0.415), respectively.”

This clarification has been incorporated into the revised manuscript.

W4/Q4. Can you compare hieararchial/search task success at matched inference budgets, or provide success versus cost curves using model calls, generated tokens and environment interactions?

We thank the reviewer for this suggestion. Our current manuscript already includes an inference-cost analysis in Appendix M (Figures 7 and 8), reporting total generated tokens, thinking-token usage, and LLM calls per task across the evaluated execution patterns.

  • We now report task success jointly with LLM calls, generated tokens and environment interactions for every executor, benchmark and model (Appendix M, Figures 9-10; success is annotated above each bar). Together with the rollout-matched control (Table 20) and the per-candidate Search decomposition (Appendix J.1, Table 21), these give direct matched-cost comparisons.

Three findings emerge:

  • Structured execution can improve success at comparable or lower cost.
    • On ALFWorld with DeepSeek-V4, Predefined uses approximately the same LLM calls (19) and environment interactions (12.6 vs. 12.5) as Plan+ReAct, but fewer generated tokens (2.1k vs. 3.6k), while improving task success from 0.480 to 0.557. On Mind2Web with Gemma-4, Predefined achieves higher success than Plan+ReAct (0.147 vs. 0.043), using fewer LLM calls (11 vs. 15) and environment interactions (2.3 vs. 6.8).
  • Hierarchical performs favorably against repeated ReAct execution.
    • Compared with the 3×Flat ReAct+Judge control (Table 20), Hierarchical achieves higher success on ALFWorld with DeepSeek-V4 (0.840 vs. 0.526) and Qwen3.6 (0.714 vs. 0.664), WebArena with DeepSeek-V4 (0.537 vs. 0.452), and SWE-bench with DeepSeek-V4 (0.442 vs. 0.410). Its observed LLM-call counts are also lower than the approximate three-rollout ReAct call costs in these cases, although total inference budgets are not strictly matched.
  • Higher computational cost does not consistently improve success.
    • On ALFWorld, Search outperforms Hierarchical with DeepSeek-V4 (0.918 vs. 0.840) but underperforms with Gemma-4 (0.495 vs. 0.601), despite using more LLM calls.
    • Similarly, on SWE-bench with DeepSeek-V4, Hierarchical achieves higher success than Search (0.442 vs. 0.400) with fewer calls (approximately 73 vs. 83).
    • On Mind2Web with Gemma-4, Predefined outperforms Search (0.147 vs. 0.086) while requiring fewer calls (11 vs. 27) and generated tokens (8.4k vs. 19k).

Overall, Search incurs the highest LLM-call cost and Hierarchical the highest token cost, and Appendix M Figures 9-10 now make these costs explicit. The structural gains are not explained by additional computation. The results also motivate cost-aware executor selection, which the forced-dispatch matrix allows to be evaluated without re-running agents.

W5/Q5. Can you report the paired confidence intervals across the same evaluation tasks and seeds and tell us which of the gains are actually different from zero?

We thank the reviewer for this question. We now report 95% paired bootstrap CIs for every few-shot gain (Appendix L, Table 25).

  • Both conditions are evaluated on the identical task set (demonstration tasks removed from both), and each of 10,000 bootstrap draws resamples tasks with replacement and applies the same draw to both conditions and every seed. P-values are Holm-corrected across the nine benchmark–model pairs.
  • We also report the change in task-specific matching: the change in excess success over the frequency-matched task-blind null, which separates genuine task-to-mode matching from a shift in overall mode preference. (Table 4 reports zero-shot results on all tasks; Appendix L Table 25 is the paired comparison on identical tasks.)
BenchmarkModelGain [95% CI]Holm pMatching change [95% CI]Holm p
ALFWorldDeepSeek-V4+0.056 [+0.007, +0.106]0.184+0.029 [−0.009, +0.068]0.448
ALFWorldQwen3.6+0.050 [+0.014, +0.088]0.047+0.020 [−0.009, +0.051]0.648
Mind2WebDeepSeek-V4−0.003 [−0.009, +0.003]1.000−0.002 [−0.008, +0.003]1.000
Mind2WebQwen3.6+0.001 [−0.004, +0.007]1.000−0.002 [−0.006, +0.002]1.000
Mind2WebGemma-4+0.000 [−0.009, +0.009]1.000−0.002 [−0.010, +0.006]1.000
SWE-benchDeepSeek-V4−0.010 [−0.023, +0.004]0.890−0.006 [−0.017, +0.006]1.000
SWE-benchQwen3.6+0.013 [+0.001, +0.025]0.245+0.008 [−0.003, +0.019]0.588
WebArenaDeepSeek-V4+0.037 [-0.017, +0.091]0.954+0.083 [+0.035, +0.133]0.005
WebArenaQwen3.6+0.006 [−0.036, +0.048]1.000+0.019 [−0.019, +0.057]0.995
  • Which gains differ from zero:

    • Significant after correction: 1 of 9 pairs. ALFWorld/Qwen3.6 (+0.050, p = 0.047).
    • Nominal only: 2 pairs. ALFWorld/DeepSeek-V4 and SWE-bench/Qwen3.6 have CIs that exclude zero, but they do not survive correction.
    • Indistinguishable from zero: the remaining 6. None of the negative gains is significant. On Mind2Web all intervals lie within ±0.009, so the effect there is reliably close to zero rather than merely underpowered.
  • What drives the significant gains.

    • The two significant gains have different causes. For WebArena/DeepSeek-V4, most of the gain (+0.083 p=0.005) comes from better task-specific matching. It is the only pair where few-shot prompting significantly improves which mode is chosen for which task.
    • For ALFWorld/Qwen3.6, the change in task-specific matching is not significant (+0.020, with a CI that includes zero), so the gain mainly reflects a shift toward stronger planning patterns overall rather than better per-task selection.

Dear Reviewer m6nh,

As we approach the end of the discussion phase, we kindly invite you to review our responses and revised manuscript. We have carefully addressed your concerns and incorporated additional analyses, controls, and clarifications.

If our responses have satisfactorily addressed your concerns, we would be grateful if you would consider updating your evaluation. We would also be happy to address any remaining questions.

Thank you for your time and constructive feedback.

Regards,
The Authors

AuthorsOpenReview ↗

Final outreach for feedback or comments

Dear Reviewer m6nh,

As the author-reviewer discussion period comes to a close, we would like to ask whether you have any further feedback on our responses or the revised manuscript. We have addressed each of your concerns, including human validation across two benchmarks, action validity, inference-cost reporting, and paired confidence intervals for the few-shot results.

If our responses have satisfactorily addressed your concerns, we would be grateful if you could consider updating your evaluation. We would also be happy to address any remaining questions.

Thank you again for your time and constructive feedback.

Regards,
The Authors

Reviewer_bh33OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the Plan Declaration-Execution Gap in LLM agents, separating failures to select an appropriate plan from failures to execute a declared plan faithfully. The authors compare three conditions: Flat ReAct, Plan + ReAct, where a declared plan is passed as context to a generic ReAct executor, and Planning-as-Routing, where the LLM declares one of four planning modes (i.e. predefined, sequential, hierarchical or search), and a deterministic router dispatches the task to a matching pattern-specific executor. Forced-dispatch matrices, in which every pattern is executed on every task, are used to compute the best fixed pattern, a per-task oracle and the routing headroom.

Experiments use three open-weight LLM backbones (DeepSeek-V4-Flash, Qwen3.6-35B-A3B and Gemma-4-26B) on ALFWorld, Mind2Web, SWE-bench Verified and WebArena. A rule-based verifier finds that only 22-45% of Plan + ReAct trajectories preserve the declared structure on ALFWorld, Mind2Web and SWE-bench, with fidelity dropping sharply for longer plans. Pattern-specific executors substantially outperform Plan + ReAct, but LLM mode declarations generally trail the best fixed pattern and carry no reliable task-specific information beyond each model's overall mode preferences.

Critical strengths
  • Separating plan selection from plan execution is a useful diagnostic lens. The finding that a plan given only as context to a generic ReAct executor is often not followed, and almost never for plans of eight or more steps on ALFWorld (Table 1), while dedicated executors yield large gains on ALFWorld and WebArena, is informative for anyone designing planner-executor agents.
  • The evaluation is extensive and self-critical. Full forced-dispatch matrices over four benchmarks allow oracle ceilings to be computed directly.
    • The authors add controls that temper their own claims, with many interesting findings being presented in Appedix: a retry-matched control for the oracle, a rollout-matched control for Search and a task-blind permutation null for mode declarations.
Critical weaknesses
  • The results provide little support for Planning-as-Routing. Routing@1 trails the best fixed pattern in 10 of the 11 benchmark-model pairs in Table 3, and three retries of the best fixed pattern close most of the oracle headroom.
    • Few-shot declarations remain below the best fixed pattern in most configurations (e.g. 0.812 vs 0.918 on ALFWorld).
    • On Mind2Web, Routing@1 is even below Flat ReAct in five of six cells, which contradicts the claim that routing "substantially closes the execution gap" (as stated on line 37).
    • The gains in the abstract compare the Plan + ReAct setting with a best fixed pattern selected in hindsight on the evaluation tasks, not with the proposed routing.
  • Plan-following (RQ1) is hard to measure reliably. Human-human agreement on whether a trajectory preserves the declared structure is only κ=0.35\kappa = 0.35 on 100 ALFWorld trajectories, and verifier-human agreement is κ=0.31/0.34\kappa = 0.31/0.34. No validation is reported for the other benchmarks.
    • The comparison is also asymmetric. Routed executors are not verified at the action level, as their fidelity holds "by construction" (on lines 207-209), yet their worker prompts state that the subtask is "a HINT, not a rigid script" and should be ignored if it does not fit (e.g. in the executor prompts for ALFWorld and SWE-bench). At the same time, Plan + ReAct trajectories declared as sequential are scored against their initial order.
  • Search conflates planning structure with extra rollouts and judging. The rollout-matched control (3× Flat ReAct + judge) is useful, but it only matches Flat ReAct.
    • I believe it remains unclear whether Search's advantage comes from candidate-plan generation or from best-of-nn selection over a stronger executor.
Questions for authors
  • Line 131 states that declarations "trail the best fixed planning pattern on every benchmark-model pair", but Table 3 reports Routing@1 = 0.432 vs 0.378 for Gemma-4-26B on WebArena. Could you run the task-blind null for the Gemma-4-26B pairs?
  • Could you report environment interactions and per-subtask step budgets per executor? These are mentioned on line 256 but not reported. I believe they matter for interpreting the gains of the Hierarchical and Search executors over Plan + ReAct.
  • Why is Gemma-4-26B missing from SWE-bench in Tables 1 and 3, and from three of the four benchmarks in Table 4?
  • Table 3 states that Routing@1 uses thinking disabled, but several of its values match the "On" column of Table 4 (e.g. Mind2Web TSR for DeepSeek-V4: 0.051). Which setting is reported?
  • As far as I understand, Mind2Web is typically evaluated offline on cached page snapshots. How are Search's candidate plans executed independently in this setting?
  • Search is attributed to LATS (Zhou et al., 2023; on line 202), but it is implemented as independent rollouts plus a rubric judge, with no tree search or value backpropagation. Could you clarify this difference?
9 comments

We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.

W1. The results provide little support for Planning-as-Routing.

We thank the reviewer for this observation. We now revised the paper explicitly separates two contributions: the gains come from pattern-specific execution, while task-level mode selection does not yet outperform the strongest fixed pattern. This is stated in the Abstract (L39–40), Introduction (L133–134), Section 5 (RQ3, L432–434) and Discussion: Routing@1 trails the strongest fixed pattern in every benchmark–model pair except Gemma-4-26B on WebArena, including with few-shot prompting.

Routing versus generic execution. Plan+ReAct and Routing@1 use the same declarations but different executors. Routing@1 outperforms Plan+ReAct in 9 of 12 comparisons (e.g., 0.721 vs. 0.480 on ALFWorld; Table 3), demonstrating the benefit of matching planning modes to their executors.

  • The strongest fixed pattern is selected retrospectively and varies across benchmarks and models (Table 2). We acknowledge that repeated fixed-pattern execution closes much of the oracle gap. Routing@1 exceeds Best Fixed only for Gemma-4/WebArena (0.432 vs. 0.378; Table 24).
  • Few-shot demonstrations improve Routing@1 by up to +0.16 (0.656 to 0.812), primarily through shifts toward stronger planning-mode preferences.
  • Pattern-specific execution improves structural maintenance (42.6–84.3% vs. 28.4–45.8%; Table 14). Task success on Mind2Web is dominated by per-step element grounding, so all methods lie within 0.006 TSR for DeepSeek-V4 and Qwen3.6.
  • These results support pattern-specific execution while identifying reliable task-specific routing as an open challenge.

W2.1 Human–human and verifier–human agreement are low on ALFWorld, with no validation reported for other benchmarks.

We thank the reviewer for highlighting this concern. We have strengthened the validation and report it in Section 5 (RQ1) and Appendix E.3.

  • As detailed in our response to common CQ1.1 (ALFWorld) regarding verifier validation, we conducted additional analysis on 100 plan-length-stratified ALFWorld trajectories, including 95% confidence intervals and joint adjudication of all 27 annotator disagreements.
  • The population-weighted adjudicated human violation rate is 63% [52%, 73%], rising with plan length from 50% (short) to 68% (medium) and 94% (long plans), independently confirming that generic Plan+ReAct frequently deviates from its declared plan.
  • Before adjudication, the annotators agreed on 73% [64%, 81%] of trajectories (κ = 0.35 [0.22, 0.48]). The verifier's agreement with each annotator (κ = 0.31 and 0.34) matches human–human agreement. The annotators applied the same strict criterion as the verifier (every scoreable step matched in declared order), which is a demanding judgment for free-form trajectories.
  • Against adjudicated labels, the verifier reaches 89% recall and 35% specificity. Specifically, the verifier detects nearly all human-identified violations and over-flags borderline cases, mainly in short plans.
  • We therefore distinguish the central finding, which is supported by human judgments, from the exact verifier-based percentages in Table 1, which should be interpreted cautiously.
  • Also, every executor is scored with the same verifier and the same rules (see W2.2).
  • Extended analysis on Mind2Web (common CQ1.2)

W2.2 The comparison is also asymmetric. Plan+ReAct trajectories declared as sequential are scored against their initial order.

As detailed in our full response to Reviewer GcEM Q2, we distinguish two levels of structural fidelity and measure both (Section 3, lines 211–216; Appendix E.4, Tables 13–14).

  • Dispatch-level fidelity: Pattern-specific controllers preserve the declared execution organization, but their workers may perform actions different from assigned subtasks. Thus, preservation "by construction" refers only to the controller's dispatch structure, not action-level adherence.
  • Action-level fidelity: Because executor prompts let the agent adapt actions to the observed state, action-level fidelity is measured, not assumed. Therefore, we applied the same rule-based verifier to the actual action trajectories of pattern-specific executors and Plan+ReAct. Under this common rule-based verifier, every executor maintains the declared structure more often than Plan+ReAct in every benchmark–model pair. On ALFWorld with DeepSeek-V4: Plan+ReAct 23.1%, Predefined 44.0%, Hierarchical 49.6% (complete leaf level), Sequential 62.0%.
  • For Sequential, replanning can legitimately revise the remaining steps. We therefore distinguish adherence to the initial declared plan from adherence to an updated plan; deviations from the initial plan should not automatically be interpreted as execution failures.

Full results for all executors, benchmarks and models are in our response to Reviewer GcEM (Q2).

W3. Search conflates planning structure with extra rollouts and judging.

We thank the reviewer for this concern. We now decompose Search per candidate on ALFWorld, WebArena and Mind2Web (Appendix J.1, Table 21). Each candidate is executed from a reset environment, so every candidate has its own outcome; across all runs, the judge-selected candidate reproduces the logged task score on every task (0 mismatches).

For each model we report:

  • Single candidate: the mean success of individual candidates, without any selection. The success of the first generated candidate alone is shown in parentheses.
  • Judge-selected: the success of the trajectory chosen by the rubric judge, i.e. the reported Search result.
  • Oracle@3: the success rate when at least one of the first three candidates succeeds, which is the ceiling of the candidate pool.
  • Judge accuracy: how often the judge picks a successful candidate on tasks where some candidates succeed and others fail.

We compare these with the corresponding Flat ReAct values from Table 20. All values are means±SD over seeds 42, 13 and 7.

Appendix J.1 Table21:

BenchmarkModelFlat ReAct: single/3×+Judge/pass@3Search: single (first)Search: judgeSearch: oracle@3Judge acc.
ALFWorldDeepSeek-V40.440/0.526/0.6000.803±0.013 (0.828)0.918±0.0330.935±0.0280.928
ALFWorldQwen3.60.535/0.664/0.7580.667±0.023 (0.672)0.796±0.0040.866±0.0070.850
ALFWorldGemma-40.112/0.172/0.2160.278±0.047 (0.239)0.495±0.0710.515±0.0640.946
WebArenaDeepSeek-V40.455/0.452/0.5940.580±0.003 (0.545)0.580±0.0060.670±0.0100.542
WebArenaQwen3.60.332/0.386/0.5480.566±0.025 (0.527)0.583±0.0290.659±0.0250.626
WebArenaGemma-40.318/0.349/0.4190.295±0.028 (0.267)0.304±0.0360.358±0.0560.559
Mind2WebDeepSeek-V40.058/0.059/0.0770.054±0.003 (0.054)0.057±0.0070.080±0.0060.522
Mind2WebQwen3.60.042/0.044/0.0690.045±0.003 (0.048)0.052±0.0010.086±0.0020.512
Mind2WebGemma-40.051/0.053/0.0570.140±0.007 (0.136)0.086±0.0080.274±0.0020.221
  • Search's advantage comes from candidate-plan generation, not from best-of-N selection.
    • Without any selection, across ALFWorld and WebArena, single Search candidates outperform Flat ReAct for DeepSeek-V4 and Qwen3.6 (e.g., 0.803 vs. 0.440 and 0.580 vs. 0.455 for DeepSeek-V4), while Search’s oracle@3 also exceeds ReAct’s pass@3, indicating that stronger candidate trajectories, not just selection, contribute substantially to Search’s gains.
  • Selection also contributes, and its share depends on the model.
    • Measured from a single Flat ReAct rollout, candidate generation accounts for about 76% of Search’s gain for DeepSeek-V4, about 51% for Qwen3.6 and about 43% for Gemma-4, whose individual candidates are weaker. For Qwen3.6, judge selection adds exactly as much to Search as it does to Flat ReAct (+0.129), so Search’s advantage over 3×Flat ReAct+Judge comes entirely from better candidates.
  • The judge is effective but not the bottleneck. On ALFWorld, accuracy reaches 85–95%, approaching the oracle (0.918 vs. 0.935 for DeepSeek-V4). On WebArena (54–63%) and Mind2Web (22–52%), the results highlight opportunities to improve candidate selection in challenging settings.

Overall, Search combines candidate-plan generation with rubric-based selection. Both contribute, and generation is the larger factor for the stronger backbones.

Q1.1. Line 131 states that declarations "trail the best fixed planning pattern on every benchmark-model pair", but Table 3 reports Routing@1 = 0.432 vs 0.378 for Gemma-4-26B on WebArena.

We have corrected the statement in the Introduction (Line 133-134 in revised pdf).

Q1.2 Could you run the task-blind null for the Gemma-4-26B pairs?

Gemma-4-26B on WebArena is the only benchmark–model pair where zero-shot declarations exceed the best fixed pattern (Routing@1 = 0.432 vs. S = 0.378).

  • Following the reviewer's suggestion, we extended the task-blind permutation analysis to Gemma-4-26B on WebArena (Appendix J.3, Table 24).
  • The results show that Routing@1 achieves (0.432±0.014), compared with (0.381±0.017) for the task-blind null, yielding (\Delta = +0.051±0.003). This improvement is statistically significant (one-sided (p=0.004), two-sided (p=0.006), Holm-corrected (p=0.042)). The null itself matches the best fixed pattern (0.381 vs. 0.378), so about 94% of the advantage over S* comes from task-specific matching rather than from the overall mode mix. These findings provide evidence that Gemma-4-26B on WebArena exhibits task-specific planning-mode discrimination beyond its overall mode preferences, unlike the other evaluated benchmark–model pairs.
  • The pattern profile explains the exception: on WebArena, Gemma-4-26B's four patterns lie within 0.09 of each other (0.290–0.378, Table 2), so no single pattern dominates and per-task selection has room to help.

Table 24 and the corresponding statements in Sections 1 and 5 are updated in the pdf.

Q2. Could you report environment interactions and per-subtask step budgets per executor?

  • We clarify that all executors use the same configured execution parameters (max_iter=8, step_budget=3, recursion_limit=48), with Search allowing up to four candidates and Hierarchical allowing a maximum decomposition depth of three and branching factor of four. The subtask step budget is fixed, but the number of subtasks or candidate trajectories varies across execution architectures. Consequently, the same configuration does not imply identical aggregate resource consumption.

  • Appendix D Table 10 (already present) reports the realized number of Search candidates (3.35 on average), Table 11 the environment interactions per task, and Table 12 the realized hierarchical decomposition.

Table 11: Environment interactions per task across execution strategies.

BenchmarkModelSequentialPredefinedHierarchicalSearchPlan+ReActFlat ReAct
ALFWorldDeepSeek-V414.40±0.1312.55±0.1117.78±0.1520.73±0.3912.53±0.0413.41±0.13
Qwen3.614.39±0.1111.36±0.3115.56±0.3418.30±0.2211.05±0.4711.11±0.28
Gemma-413.83±0.3212.12±0.1019.03±0.7721.35±0.9812.88±0.6614.00±0.31
Mind2WebDeepSeek-V46.94±0.006.83±0.016.95±0.006.89±0.006.88±0.006.88±0.00
Qwen3.67.24±0.086.90±0.077.13±0.277.01±0.026.82±0.026.78±0.03
Gemma-42.84±0.072.34±0.114.21±0.014.55±0.156.76±0.006.75±0.02
WebArenaDeepSeek-V417.17±0.3819.88±4.5228.01±2.0028.23±2.1415.69±0.0515.93±0.65
Qwen3.626.58±2.1418.86±0.0635.64±9.4826.45±1.3716.72±0.4016.63±0.01
Gemma-423.86±0.2917.23±0.2227.31±0.0630.20±0.2715.25±0.3015.04±0.37
SWE-bench†DeepSeek-V425.94±6.4130.58±0.3058.91±3.7276.28±0.5625.43±0.6928.65±0.57
Qwen3.643.24±4.3960.58±0.2991.43±2.9696.54±0.2123.73±0.3728.76±0.46
Gemma-458.85±0.5036.32± 0.2773.03±36.9774.07± 0.3126.70±0.626.4± 0.40

What the environment interaction counts show.

  • Stronger performance does not always require more environment interactions. On ALFWorld, Predefined uses approximately the same number of tool calls as Plan+ReAct (12.55 versus 12.53 for DeepSeek-V4, with ratios of 0.94–1.03 across models). Nevertheless, its task success is higher in the evaluated settings, with the largest difference of +0.362 TSR for Gemma-4. Similarly, on Mind2Web, Gemma-4 Predefined uses fewer interactions than Plan+ReAct (2.34 versus 6.76) while achieving higher task success (0.147 versus 0.043).
  • Search retains an advantage over repeated generic execution in some environments. On ALFWorld with DeepSeek-V4, Search achieves 0.918 TSR using 20.73 tool calls per task, compared with 0.526 TSR for 3×Flat ReAct+Judge (Table 20). On WebArena, Search achieves 0.580 versus 0.452. Search's interaction counts include all candidate trajectories. These comparisons suggest that additional independent ReAct rollouts alone do not fully account for Search's performance advantage in these settings.
  • More environmen interactions do not necessarily produce higher success. On SWE-bench with DeepSeek-V4, Search uses more tool calls than Hierarchical (76.28 versus 58.91), but achieves lower patch success (0.400 versus 0.442). Thus, the relationship between interaction cost and task performance depends on the execution architecture and benchmark.
  • Mind2Web isolates structure directly. Interaction counts there are essentially fixed by the step-wise offline protocol. Gemma-4's Predefined executor uses fewer calls than Plan+ReAct (2.3 vs. 6.8) yet triples its TSR (0.147 vs. 0.043).

Table 12: Hierarchical decomposition depth and executable leaf subtasks. Depth is determined dynamically through [ATOMIC] and [DECOMPOSE] decisions, with max_depth=3 and max_branch=4.

BenchmarkModelnd=0d=1d=2d=3Mean depthMean leavesDepth-3 share
Mind2WebDeepSeek-V41,3410197595632.414.8142%
Qwen3.61,3416318464582.315.2134%
Gemma-41,341104089322.697.3170%
WebArenaDeepSeek-V418602151692.908.1891%
Qwen3.618600351512.819.6181%
Gemma-41860011852.9910.1999.5%
SWE-benchDeepSeek-V4500013164712.9213.9094%
Qwen3.6500024903862.729.3077%
Gemma-4500031783912.7212.7978.2%
  • The hierarchical-depth analysis shows that WebArena and SWE-bench frequently reach the maximum decomposition depth, with larger numbers of executable leaf subtasks than Mind2Web. This helps characterize the additional execution and coordination demands of Hierarchical planning. Overall, these results clarify how environment interactions vary across executors despite shared configuration limits. Further, increased interaction counts alone do not consistently explain performance differences, while acknowledging that the experiments do not strictly match total inference computation across architectures.

  • Tables 11–12 are updated in Appendix D in revised pdf.

Comments on the paper

4 comments

We are grateful to all reviewers for their strong positive feedback, time and their constructive suggestions, which will further strengthen the impact of our work.

CQ1.1 Can you report confidence intervals and have the annotators resolve the disagreements?

We thank the reviewer for this suggestion. We now provide a plan-length-stratified human validation of the rule-based verifier, including 95% confidence intervals and adjudicated annotations (Section 5, RQ1; Appendix E.3). From 984 ALFWorld Plan+ReAct trajectories population, we sampled 100 trajectories for human annotation, stratified by declared plan length: 40 short-plan trajectories (≤3 steps), 44 medium-plan trajectories (4–6 steps), and all 16 long-plan trajectories (≥7 steps). To account for the different sampling fractions, we use inverse-probability weights (7.375, 15.295, and 1.0, respectively) when estimating population-level structural-violation rates, and 95% CIs are [weighted bootstrap over trajectories, 10,000 draws].

Structural-violation rates [95% CI]:

RaterViolation rate [95% CI]
Annotator A163% [53%, 73%]
Annotator A269% [58%, 78%]
Rule-based verifier80% [70%, 89%]
Adjudicated human labels63% [52%, 73%]
  • Before adjudication, the annotators achieved 73% raw agreement [64%, 81%] (Cohen's κ = 0.35 [0.22, 0.48]). All 27 disagreements were jointly reviewed and resolved, producing consensus labels for all 100 trajectories.

Violation rates by plan length (adjudicated):

Plan lengthViolation rate [95% CI]
Short (≤3 steps)50% [35%, 65%]
Medium (4–6 steps)68% [55%, 82%]
Long (≥7 steps)94% [81%, 100%]
All (population-weighted)63% [52%, 73%]
  • Note: CIs are bootstrap percentile intervals; they are asymmetric near 0% and 100%.

  • Verifier validation against adjudicated labels. On the 98 trajectories scoreable by the verifier, population-weighted accuracy is 69% [59%, 78%], Cohen's κ is 0.26 [0.04, 0.47], precision is 69% [59%, 80%], recall is 89% [79%, 97%], and specificity is 35% [18%, 53%].

Three findings:

  • The central claim holds independently of the verifier. Adjudicated human judgments identify structural violations in 63% [52%, 73%] of Plan+ReAct trajectories. Frequent deviation from the declared plan is therefore not an artifact of automated matching.
  • Violations increase with plan length. The adjudicated rate rises from 50% for short plans to 68% for medium and 94% for long plans, consistent with the plan-length pattern in Table 1.
  • The verifier is more sensitive than specific. Its high recall (89%) captures most human-identified violations, but its low specificity (35%) means it also flags some trajectories that humans judge to maintain their plans, especially short plans (specificity 15%). The verifier therefore overstates violations in absolute terms: Table 1 implies about 77% violations on ALFWorld, compared with 63% under adjudicated human judgments.

Overall, these results support our conclusion that generic Plan+ReAct frequently deviates from its declared plan, while making the verifier's limitations explicit.

CQ1.2 Can you validate this on a larger, independently sampled set of trajectories?

We thank the reviewer for this suggestion. We additionally annotated 200 independently sampled Mind2Web Plan+ReAct trajectories (3929) using two human annotators (using the same protocol (Appendix E.3)). Before adjudication, the annotators achieve 78.5% raw agreement (95% CI [72%, 84%]) and Cohen's κ=0.58\kappa=0.58 (95% CI [0.47, 0.68]). All 43 disagreements at the action level by matching each declared plan unit to its corresponding executed action, or marking it unmatched, yielding adjudicated labels for all 200 trajectories.

Mind2Web: Structural-violation rates [95% CI]:

RaterViolation rate [95% CI]
Annotator A164% [57%, 71%]
Annotator A248% [41%, 55%]
Rule-based verifier59% [52%, 66%]
Adjudicated human labels64% [56%, 71%]

Mind2Web: Violation rates by plan length (adjudicated)

Plan lengthViolation rate [95% CI]
Short (≤3 steps)58% [46%, 69%]
Medium (4–6 steps)67% [57%, 76%]
Long (≥7 steps)92% [82%, 100%]
All (population-weighted)64% [56%, 71%]
  • The adjudicated structural-violation rate on Mind2Web is 64% (95% CI [56%, 71%]), closely matches ALFWorld (63% [52%, 73%]), and on both benchmarks violations rise sharply with plan length (to 92–94% for long plans).
  • The verifier over-flags on ALFWorld (80% vs. 63%) and slightly under-flags on Mind2Web (59% vs. 64%), and our conclusions based on the adjudicated human rates.
  • Together, 300 adjudicated trajectories on two benchmarks establish that generic Plan+ReAct frequently fails to maintain its declared planning structure.

CQ2. Can you report for each executor how often its actions are valid and result in the desired state changes?

We now report action execution separately from plan maintenance for every executor, computed from the same runs (Appendix E.4, Table 13); task success is reported separately in Tables 2-3.

  • Action Valid is the fraction of actions the environment accepts (ALFWorld: admissible commands; Mind2Web: well-formed operations on a listed element).
  • Effective is the fraction of valid actions that change the environment state on ALFWorld, and the fraction of actions matching the reference step on Mind2Web. Maintained is action-level structure maintenance under the Table 1 verifier.

Table 13. ALFWorld (seed 42)

ExecutorValid (DeepSeek)Effective (DeepSeek)Maintained (DeepSeek)Valid (Qwen)Effective (Qwen)Maintained (Qwen)Valid (Gemma)Effective (Gemma)Maintained (Gemma)
Plan+ReAct0.8680.91623.1%0.8680.92020.5%0.7930.87519.6%
Predefined0.9120.93544.0%0.8560.94247.8%0.8460.91257.5%
Sequential0.9290.91162.0%0.8730.92169.2%0.8850.85961.1%
Hierarchical0.8790.86749.6%0.8400.92756.2%0.8200.84857.0%
Search0.8600.899N/A0.8400.913N/A0.7960.901N/A

Table 13. Mind2Web (seed 42)

ExecutorValid (DeepSeek)Effective (DeepSeek)Maintained (DeepSeek)Valid (Qwen)Effective (Qwen)Maintained (Qwen)Valid (Gemma)Effective (Gemma)Maintained (Gemma)
Plan+ReAct1.0000.41045.8%0.9960.33836.1%1.0000.34428.4%
Predefined1.0000.42863.6%0.9960.33442.6%1.0000.44344.2%
Sequential1.0000.41663.2%0.9970.33451.8%0.9990.41352.7%
Hierarchical1.0000.39584.3%0.9230.30367.5%0.9970.36875.6%
Search1.0000.444N/A0.9980.385N/A1.0000.431N/A
  • Note: Plan maintenance is not reported for Search because its multi-candidate planning structure does not define a single fixed action sequence for evaluation using the ordered-step verifier. Action validity and effectiveness are evaluated normally.

  • Action execution varies little across executors; plan maintenance varies strongly.

    • On ALFWorld, valid-action rates lie within 0.79–0.93 and effective rates within 0.85–0.94, and no executor exceeds Plan+ReAct by more than 10 points on either measure. Plan maintenance, by contrast, rises from 20–23% under Plan+ReAct to 44–69% under the pattern-specific executors. With Qwen3.6, Plan+ReAct and Sequential are nearly identical in validity (0.868 vs. 0.873) and effectiveness (0.920 vs. 0.921), yet Sequential maintains the declared structure in 69.2% of episodes versus 20.5%.
    • On Mind2Web, actions are almost always valid because the agent selects from a provided element list, and effective rates stay within a few points of Plan+ReAct (Gemma-4 Predefined is the exception, 0.443 vs. 0.344).
  • The three properties are distinct.

    • Higher plan maintenance does not imply more effective actions: on Mind2Web with DeepSeek-V4, Hierarchical has the highest maintenance (84.3%) but lower effectiveness (0.395) than Search (0.444).
    • Because task success differs substantially across pattern-specific executors (Tables 2–3) while action-level execution does not, the executors improve success by organizing similar actions around the declared plan, not by executing individual actions more reliably.

We thank the reviewers for their constructive feedback. We summarize the main additions below.

Overall, Planning-as-Routing's main findings hold under the additional controls and validations (human adjudication on two benchmarks, a common action-level verifier, controlled ablation, matched-rollout and per-candidate comparisons, cost analysis, and paired CIs with Holm correction): generic Plan+ReAct frequently deviates from its declared plan, pattern-specific execution improves task success, and reliable task-level mode selection remains an open challenge. These additional analyses further strengthen our original conclusions and clarify where Planning-as-Routing provides benefits.

  1. Human validation of the declaration–execution gap (m6nh, bh33, GcEM; Sec. 5 RQ1, App. E.3). We adjudicated 100 plan-length-stratified ALFWorld Plan+ReAct trajectories and additionally annotated 200 independently sampled Mind2Web trajectories with two human annotators, resolving all disagreements. The adjudicated violation rates are 63% [52%, 73%] on ALFWorld and 64% [56%, 71%] on Mind2Web. Violations increase with plan length on both benchmarks, reaching 92–94% for long plans. The verifier overestimates violations on ALFWorld (80% vs. 63%) and underestimates them on Mind2Web (59% vs. 64%), so it shows no consistent directional bias, and the declaration–execution gap remains supported by adjudicated human labels.

  2. New baselines and controls.

    • Structural fidelity (bh33, GcEM; Sec. 3, App. E.4, Table 14). We distinguish dispatch-level fidelity from action-level structural maintenance and apply the same rule-based verifier to Plan+ReAct and the three ordered-plan executors. All three maintain their declared structure more often than Plan+ReAct across every evaluated benchmark–model pair. For example, on ALFWorld with DeepSeek-V4, maintenance is 23.1% for Plan+ReAct, compared with 44.0% for Predefined, 49.6% for Hierarchical, and 62.0% for Sequential.

    • Action validity and effectiveness (m6nh; App. E.4, Table 13). We report action validity and effectiveness separately from structural maintenance. For example, on ALFWorld with Qwen3.6, Plan+ReAct achieves 0.868 action validity and 0.920 effectiveness, yet maintains its declared structure in only 20.5% of trajectories. This demonstrates that successful individual actions do not necessarily imply faithful plan execution.

    • Planning constraint ablation (bjZs; Sec. 5 RQ3, App. J.4, Table 23). A 2×2 design varies architecture and structural enforcement independently. Enforcement significantly improves success under both architectures: +0.097 [+0.008, +0.187] for generic ReAct and +0.179 [+0.082, +0.276] for Predefined. Architecture alone has no significant effect.

    • Search candidate decomposition (bh33; App. J.1, Tables 20–21). Without selection, individual Search candidates already outperform single Flat ReAct rollouts: 0.803 vs. 0.440 on ALFWorld and 0.580 vs. 0.455 on WebArena with DeepSeek-V4. On ALFWorld, stronger candidate trajectories account for much of Search's improvement, with judge selection providing additional gains. Search also outperforms 3×Flat ReAct+Judge on ALFWorld (0.918 vs. 0.526).

    • Cost analysis (m6nh; App. M, Figures 9–10). We report LLM calls, generated tokens, and environment interactions alongside task success. At comparable interaction cost, Predefined improves ALFWorld success from 0.480 to 0.556 with DeepSeek-V4. Hierarchical also outperforms 3×Flat ReAct+Judge with fewer LLM calls on ALFWorld, WebArena and SWE-bench (DeepSeek-V4).

    • Executor statistics (bh33; App. D, Tables 10–12). We report environment interactions per task, and decomposition depth under shared step budgets.

    • Few-shot statistics (m6nh, bjZs; Sec. 5 RQ4, App. L, Tables 25–27). We report paired bootstrap CIs on identical tasks and seeds, with Holm-adjusted p-values. Two of nine pairs show a significant gain. WebArena with DeepSeek-V4 is the clear case of task-specific matching: +0.103 of its +0.122 gain (84%), Holm p = 0.002. Elsewhere, gains primarily reflect shifts in overall planning-mode preferences.

  3. Reframing and clarifications (bh33, m6nh, GcEM, bjZs).

    • The Abstract, Introduction, Sec. 5 and Discussion now separate execution gains from selection gains.
    • Documented the Mind2Web offline protocol and the SWE-bench runtime/evaluation split (App. D);
    • Removed the attribution of Search to LATS;
    • Corrected the Table 3 Routing@1 entries and the Gemma-4-26B WebArena task-blind null (Table 24); and
    • Added missing Gemma-4-26B results and clarification of the SWE-bench few-shot demonstrations (App. O.4).

We believe these additional validations and controlled analyses further strengthen our original conclusions while clarifying the benefits of pattern-specific execution and the remaining challenges of reliable task-level mode selection.

The author-reviewer discussion period will end soon. Please review and acknowledge the authors’ responses. If your concerns have been addressed, please reconsider your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.