- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the Plan Declaration–Execution Gap in LLM agents—the discrepancy between declaring a structured plan and faithfully executing it. The authors categorize agent planning into four modes inspired by cognitive science: Predefined (plan-once), Sequential (step-by-step replanning), Hierarchical (orchestrator–worker tree decomposition), and Search (parallel candidate rollouts selected by a rubric judge). They compare three setups across four benchmarks (ALFWorld, Mind2Web, WebArena, SWE-bench Verified) and three backbone LLMs (Qwen3.6-35B-A3B, DeepSeek-V4-Flash, Gemma-4-26B): (1) Flat ReAct, (2) Plan+ReAct (plan passed in context), and (3) Planning-as-Routing, where the LLM declares a planning mode and a deterministic router dispatches the task to a corresponding pattern-specific executor. Through extensive experiments and control baselines, the authors find that:
- Generic Plan+ReAct fails to preserve declared plan structures on 55–78% of trajectories, especially as plan length increases or when hierarchical/search plans are declared.
- Pattern-specific executors substantially improve task success over Plan+ReAct (e.g., from 0.480 to 0.918 on ALFWorld, and 0.360 to 0.442 on SWE-bench Verified), though the single best executor varies by model and benchmark.
- Crucially, the paper's own controls in Appendices J and L reveal two major negative findings: (a) once multi-rollout stochasticity is controlled via retries of the single strongest fixed pattern (), the per-task oracle routing headroom () nearly vanishes across benchmarks ( to ), and (b) zero-shot LLM mode declarations perform no better than a task-blind random permutation null (). Few-shot demonstrations improve top-1 routing performance primarily by shifting global mode preferences rather than achieving reliable per-task discrimination.
Critical strengths
- Rather than overclaiming the superiority of Planning-as-Routing or the Search executor, the authors include: (i) a distribution-matched permutation null (Table 18) proving that zero-shot LLM routing declarations have no statistically significant task-specific discriminative power over a task-blind prior, (ii) a retry-matched baseline (Table 17) showing that repeating the single best fixed pattern 3 times () matches or exceeds the 4-pattern per-task oracle (), and (iii) a Flat ReAct + Judge control (Table 16) to isolate the multi-rollout advantage of Search. This level of diagnostic honesty is rare in agent papers and provides genuinely useful insights for agent evaluation.
- Evaluating four distinct agent architectures across four major benchmarks (ALFWorld, Mind2Web, WebArena, SWE-bench Verified) and three models across multiple seeds is a substantial empirical undertaking. The finding that intrinsic LLM-judged plan quality has almost zero correlation with downstream task success (, Table 15), whereas enforcing execution control flow yields large gains on long-horizon tasks (Figure 6), is an important takeaway for practitioners designing agentic scaffolds.
Critical weaknesses
- The paper frames Planning-as-Routing as a solution to match individual tasks to appropriate planning modes. However, Table 3 shows that
Routing@1consistently underperforms simply picking the single best fixed executor (Best Fixed) on 10 out of 11 benchmark–model pairs (e.g., 0.721 vs. 0.918 on ALFWorld, 0.460 vs. 0.580 on WebArena). More importantly, Table 17 shows that even with an Oracle router, the headroom over simply retrying the best fixed pattern () is effectively zero ( to ). Furthermore, Table 18 demonstrates that the LLM router cannot discriminate at the task level (). Even the few-shot gains (Table 4) are acknowledged in Appendix L (line 1683) to "improve raw top-1 performance, but rarely task-specific discrimination"—meaning few-shot examples primarily teach the router to favor the globally dominant executor. Thus, the empirical evidence does not support per-task routing; it shows that practitioners should select the best fixed scaffold for a given domain/model and allocate budget to retries. - In RQ1, the authors claim that generic
Plan+ReActfails to preserve declared structures (particularly Search and Hierarchical), whereas pattern-specific executors enforce them "by construction." This comparison is confounded in three ways:
- First, asking a single-thread, non-backtracking
Plan+ReActloop to execute a Search plan (which requires independent environment resets and competing rollouts) is impossible without an environment-reset tool. In Appendix E.2 (page 21), the "Search" trace underPlan+ReActsimply concatenates Path 1 and Path 2 sequentially in the same episode. PenalizingPlan+ReActfor not doing multi-episode search when it lacks the harness mechanism to reset the environment measures a harness limitation rather than a model failure. - Second, the pattern-specific executors explicitly instruct the worker LLM to ignore the declared plan step if needed (e.g., Appendix O.2.1, line 1995: "If the subtask does not match what is needed, IGNORE it and do what the task needs"), yet they are credited with 100% structural fidelity "by construction" simply because the outer Python loop dispatches subtask prompts in order, while
Plan+ReActis scored by an action-matching verifier. - Third, the rule-based verifier used on
Plan+ReActhas low agreement with human annotators (Cohen's and , Section 5 line 315) and human–human agreement is similarly low (, and on Qwen3.6), raising concerns about the reliability of the exact structural maintenance percentages in Table 1.
Questions for authors
- Given that is near zero across almost all settings (Table 17)—meaning 3 retries of the best fixed pattern matches or beats the 4-pattern Oracle—and per-task declarations show no advantage over the task-blind permutation null (Table 18), why is per-task Planning-as-Routing preferable to simply selecting the best domain-level scaffold and using retries? Would the paper be stronger if reframed around this empirical finding rather than pitching task-level routing as the core solution?
- In Appendix O.2.1 (line 1995) and O.2.3 (line 2140), the Predefined and Hierarchical executors are instructed: "If the subtask/atomic action does not match what is needed, IGNORE it and do what the task needs." If a pattern-specific leaf worker ignores the declared subtask and executes different actions, why is it considered to preserve structure "by construction," whereas
Plan+ReActdoing the same is penalized by the rule-based action matcher? What are the structural maintenance scores if the exact same rule-based action verifier is applied to the action streams of the pattern-specific executors? - Why was test execution (
pytest) blocked in SWE-bench Verified, and how might this restriction bias the comparison against feedback-driven modes likeSequentialandFlat ReAct? Additionally, why does Example 7 in the SWE-bench few-shot declaration prompt (Appendix O.4, lines 2678–2688) instruct the agent to create and execute unit tests when test execution is blocked in the environment?
10 comments
W2.2/Q2 What are the structural maintenance scores if the exact same rule-based action verifier is applied to the action streams of the pattern-specific executors?
We thank the reviewer for this question, which helps us to distinguish two levels of structural fidelity more clearly.
-
Dispatch-level fidelity. In pattern-specific execution, the controller maintains the prescribed planning organization. For Predefined, the declared plan remains fixed and subtasks are dispatched in order. For Hierarchical, the controller preserves the parent–child organization and dispatches the corresponding leaf subtasks. Importantly, a worker executing alternative actions does not overwrite or reorder the controller's declared plan. Thus, Structural preservation “by construction” therefore refers to the order and organization of dispatched plan units, not necessarily to the actions performed while executing them.
-
Action-level fidelity. We agree that action-level maintenance is not guaranteed. the agent may perform actions that differ from its assigned subtask while the controller maintains the declared structure. To evaluate this consistently, we applied the identical rule-based verifier used for Plan+ReAct (same step cleaning, action-matching rules and in-order criterion) to the actual action streams of the pattern-specific executors.
Appendix E.4 Table 14 Episode-level structure maintenance
| Benchmark | Model | Predefined | Sequential | Hierarchical | Plan+ReAct |
|---|---|---|---|---|---|
| ALFWorld | DeepSeek-V4 | 44.0% | 62.0% | 49.6% | 23.1% |
| ALFWorld | Qwen3.6 | 47.8% | 69.2% | 56.2% | 20.5% |
| ALFWorld | Gemma-4 | 57.5% | 61.1% | 57.0% | 19.6% |
| Mind2Web | DeepSeek-V4 | 63.6% | 63.2% | 84.3% | 45.8% |
| Mind2Web | Qwen3.6 | 42.6% | 51.8% | 67.5% | 36.1% |
| Mind2Web | Gemma-4 | 44.2% | 52.7% | 75.6% | 28.4% |
| WebArena | DeepSeek-V4 | 82.6% | 88.6% | 97.6% | 67.5% |
| WebArena | Qwen3.6 | 73.7% | 85.9% | 89.2% | 42.0% |
| WebArena | Gemma-4 | 84.2% | 91.2% | 97.7% | 61.4% |
-
Under the same verifier, all three pattern-specific executors maintain the declared structure more often than Plan+ReAct, in every benchmark–model pair.
- On ALFWorld, maintenance rises from 19.6–23.1% under Plan+ReAct to 44.0–69.2%. On Mind2Web, Predefined and Sequential improve on it by 6.5–24.3 percentage points; on WebArena, by 15.1–43.9 points.
- The execution gap reported in RQ1 therefore holds when all executors are measured with the same instrument. The improvement is largest where generic execution fails most: for Gemma-4 on ALFWorld, maintenance rises from 19.6% to 57.5% under Predefined, which is also where Predefined’s success gain is largest (+0.362).
-
Why does action-level maintenance remain below 100%? Two mechanisms contribute.
- First, the agent may perform actions that do not correspond to the currently assigned plan unit, without modifying the declared plan itself. This is deliberate: plans are written before the environment is observed, so some declared steps turn out to be infeasible.
- Second, the execution budget may run out before all declared subtasks are executed. Because the verifier requires every scoreable plan unit to be matched in order, both deviations and incomplete execution lower the reported rate.
-
Importantly, an unmatched action is not an incorrect action, and an incomplete plan does not mean its executed steps were wrong; we therefore report dispatch-level fidelity, action-level maintenance, plan completion and task success as separate properties (Appendix E.4, Tables 13–14; refer common responses to CQ2).
W2.3 the rule-based verifier used on Plan+ReAct has low agreement with human annotators
- We have provided a detailed discussion in our responses to CQ1.1 and CQ1.2. Kindly refer to those responses for more comprehensive information.
Q3.1 Why was pytest blocked in SWE-bench Verified, and could this disadvantage feedback-driven executors?
Why test execution was blocked.
- Our SWE-bench Verified setup does not use per-instance Docker environments. Since instances require different dependency versions, running tests could require runtime package installation, introducing additional overhead and environment inconsistencies. We therefore disabled
pip,pytest, andcondafor all executors, including Flat ReAct and Plan+ReAct. Final patches are evaluated separately using the SWE-bench Verified evaluation harness, which determines patch success (PSR). We will clarify this distinction in the manuscript. Appendix D now states this distinction between the agent runtime and the evaluation environment.
Potential bias.
- The restriction applies identically to all executors, and the results show no systematic penalty for feedback-driven modes: with DeepSeek-V4, Flat ReAct outperforms Plan+ReAct (0.384 vs. 0.360) and Sequential outperforms Predefined (0.395 vs. 0.372). Our SWE-bench results characterize planning patterns under static code inspection, and the revised manuscript states this setting explicitly.
Q3.2 why does Example 7 in the SWE-bench few-shot declaration prompt (Appendix O.4, lines 2678–2688) instruct the agent to create and execute unit tests when test execution is blocked in the environment?
We thank reviewer for this question.
- Example 7. The few-shot prompt only selects the planning mode: the router outputs a mode ranking, and the demonstration plans are never executed or passed to the executors. Example 7 reproduces verbatim the plan that the Hierarchical planner produced for a held-out task, to illustrate that decomposition style.
- Once a mode is declared, the matching executor generates its own plan for the new task, in the same runtime where test execution is blocked. Appendix O.4 now states this.
We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.
W1/Q1. why is per-task Planning-as-Routing preferable to simply selecting the best domain-level scaffold and using retries? Would the paper be stronger if reframed around this empirical finding rather than pitching task-level routing as the core solution?
We thank the reviewer for this valuable suggestion. We now reframed the paper around this finding. The revised manuscript states it directly:
Abstract (lines 39–40): "pattern-specific execution substantially closes the execution gap, while selecting the right mode for each task remains open."
Introduction (lines 133–134): current declarations "trail the best fixed planning pattern on all benchmark–model pairs except Gemma-4-26B on WebArena, and show no reliable task-specific advantage over task-blind mode selection."
Section 5, RQ3 (lines 432–434): "the gains come from the execution architecture rather than automatic task-level mode selection."
Discussion: "For current models, however, task-level planning-mode selection does not consistently outperform selecting the strongest fixed pattern for each benchmark–model pair and retrying it."
A fixed scaffold with retries itself requires Planning-as-Routing.
Where per-task routing adds value beyond a fixed scaffold. Three conditions, each measured in the paper:
Role of Planning-as-Routing.
W2.1.. Does penalizing Plan+ReAct for Search-plan failures conflate harness limitations with model failures?
We thank the reviewer for this observation, and we agree.