acceptodds
Under review as a conference paper at ICLR 2027

PACE: POLICY-NATIVE ADAPTIVE DECISION TIMING FOR LONG-HORIZON REASONING

Abstract

Long-horizon reasoning requires deciding not only what actions to take, but how many to execute open-loop before replanning. This number—the execution depth—balances replanning cost against compounding execution errors. Most current systems either fix execution depth as a hand-tuned scalar or adjust it at inference time using heuristic rules decoupled from the policy; both can be suboptimal. We treat execution depth as a learnable, history-conditioned variable of the policy itself and propose PACE, a model-native vision–language policy that jointly predicts what to execute and for how long under a hard decision budget. We evaluate PACE on four long-horizon environments spanning full and partial observability and visual and textual modalities: Sliding Puzzle, Sokoban, ALFWorld, and ScienceWorld. Across all seven evaluation settings, PACE improves success rates by 3.1–15.6 percentage points while using fewer decisions on average, achieving empirical success–decision Pareto dominance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation-Model Agents, Long-horizon Reasoning, Adaptive replanning, Action chunking, Partial observability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z8noOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents a problem of determining execution depth: “how many actions should an agent execute open-loop before it replans?” The authors argue that this should be learned rather than use of fixed value. So the problem can be seen as a decision process with a hard budget K on the number of policy calls. PACE adds two haeds to Qwen VLM: a depth head that picks h and an action decoder that classifies over a fixed per-environment command list, outputs 8 actions and executes only the first h. On sliding Puzzle, Sokoban, ALFWorld and ScienceWorld, PACE beats the best fixed depth while using slightly fewer decisions.

Since many methods have relied on fixed execution horizon, the problem that this paper focuses on seems worth addressing. However, I’m not persuaded that the use of adaptive depth brings advantage because a hand-picked K is too small to finish most tasks, and there is no intermediate feedback even though the simulators produce it for free. Furthermore, the main comparison does not isolate what it claims to isolate. Together with fairly limited novelty relative to prior work on execution horizons, this leads me to lean towards rejection.

Critical strengths
  1. The problem statement is easy to understand.
  2. The evaluation has performed in diverse environment with diverse settings, providing multifaced evidence of the effect of the proposed method.
Critical weaknesses
  1. The idea of learning how long to commit to an action, jointly with the action itself, is not new. In FiGAR (Sharma et al., 2017), the policy is factorized into an action component and a repetition component. This is very close to the PACE design. The main difference is that PACE executes a sequence of different actions open-loop, whereas FiGAR repeats a single action. Dynamic Execution Horizon Prediction for Chunk-based Robot Policies (Zhao et al., 2026) also already report that learned or adaptive horizons beat fixed ones and follow the task phase. What remains is applying the idea to discrete VLM agents, with a closed action vocabulary and a hard budget. That reads to me more as an engineering combination than a new insight.’
  2. I suspect that the experimental setup is constructed favoring adaptive depth. The authors set K below the length of the reference solutions. For example, in ScienceWorld, the references take 6-22, 23-32, and 35-153 actions for short, medium, and long groups, with decision budgets K = 6, 9, and 18, respectively. With h = 1, the agent can execute at most K actions, which is fewer than most tasks require. It therefore scores at or near 0%, no matter how good its individual decisions are. Under such a budget every policy is forced to execute long chunks, so it seems that a policy using long chunks where they are safe and short ones where they are risky has more advantage over single fixed depth setting. A comparison with close-loop execution (h=1) under a larger budge seems required.
  3. In the experiment, the agent never uses the feedback that the simulators already return after every action (invalid actions do not stop a chunk). Compared to the simple rule like stopping the chunk as soon as the environment reports and invalid action, what advantages can be brought by the use of PACE?
Questions for authors
  1. What do the authors see as the main novel idea of PACE beyond the mentioned previous works?
  2. Can you clarify when adaptive depth helps and when it does not? For example, how does the gain change with reversibility, with observability, or with different K?
  3. The paper models cost as a fixed limit on the number of decisions: once the agent has made K decisions, the episode fails. However, the cost of calling a model is usually time or money in practical. It would help to report the actual time or token cost of PACE.
  4. All fixed-depth baselines reuse PACE’s checkpoint, which was trained to mix depths, and simply force a single depth at test time. Could you train a fixed-depth policy with RL under the same budget and compute, at least for the best h, and compare it with PACE? Without this comparison, it is hard to tell whether the gain comes from adaptive depth or from the RL training itself.
5 comments
AuthorsOpenReview ↗

Response to Reviewer z8no (1/2)

We thank the reviewer for the careful review, and for finding the problem statement easy to understand and the evaluation across diverse environments and settings to provide multifaceted evidence. All three weaknesses are addressed with new experiments or new formal results; points shared with other reviewers are covered in detail in the General Response (G1–G5).

W1 / Q1 (novelty relative to FiGAR and DEHP). We agree that learning how long to commit, and that adaptive horizons can beat fixed ones, is known; the original submission cites this lineage (§2, Remark 3), and PACE does not claim it. What PACE adds is what changes when each decision has a cost, as when every decision is a call to a large model:

  • Timing matters even for an optimal policy (new Prop. 2). Without a decision cost, deciding at every step is optimal; with a cost, an adaptive depth can strictly beat every fixed depth from 1 to 8.
  • A frozen policy is not enough (new Prop. 3). Under a decision cost, actions later in a chunk run without feedback, so they must also be right. A frozen policy can fail under any depth policy, while adapting the actions succeeds. This motivates adapting the actions too; our ablations support learning both inside the policy, rather than timing on top of a frozen one as in DEHP (Zhao et al., 2026).

The experiments support both points and show that the differences from FiGAR (Sharma et al., 2017) and DEHP matter empirically:

  • Action repetition, as in FiGAR, does not fit these domains: our Repeat-last baseline solves 0.0% on Sliding Puzzle and ScienceWorld.
  • The DEHP-style variant (frozen policy, only the depth head trained) helps in the text settings but falls below the SFT-best fixed depth on both puzzles (−3.1 / −17.2 pp).
  • PACE beats every variant that learns only timing or only actions in all seven settings (21/21).

Details: General Response G1.

W2 (the budget favors adaptive depth; closed-loop h = 1 under a larger budget). We ran exactly this comparison. With relaxed budgets (K = 50; 80 / 150 for ScienceWorld medium / long), step-wise control (h = 1) still loses to PACE in all seven settings, by 5.2 to 65.6 pp (4.9% vs. 70.5% on ScienceWorld long), and it needs more decisions per success in all seven. On the puzzles, we also trained dedicated h = 1 policies with GRPO at K = 50; PACE beats them as well (+11.0 / +7.8 pp). At these budgets, successful episodes finish well before the limit (a median of 3–10 decisions for PACE), yet PACE still leads and still mixes depths (75–100% of episodes use at least two). The advantage of PACE thus persists at the larger budgets we tested. The tight main budgets are deliberate: we study how to spend a small number of decisions, each a full forward pass of a 7B VLM. Details: General Responses G3 (Table R4) and G5.

W3 (stopping a chunk at the first invalid action). We evaluated this rule on ALFWorld for PACE and for three policies without a learned depth: fixed-depth policies RL-trained at h = 7 and h = 8, and an action-only policy (see General Response G5, Table R10). The rule replaces the learned decision of when to stop with a fixed trigger: stop and replan at the first rejected action.

  • It does not replace learned timing. With the rule, every policy without a learned depth stays below PACE (at most 43.1 vs. 54.3 on seen; 29.1 vs. 48.7 on unseen).
  • It is not free under a budget. On PACE, the rule lowers success (48.3 / 43.6), mostly because the truncations use up decisions: budget exhaustion accounts for about 71% of the net drop on seen and all of it on unseen.
  • An invalid action is often cheap, and the rest of the chunk can still be useful. A rejected command leaves the state unchanged and uses no decision. In PACE's ScienceWorld episodes, 56% of the actions after the first invalid one are still valid.
  • The costly errors do not trigger the rule. They are valid actions executed too early, such as a miscounted push into a deadlock in Sokoban, or an answer submitted before its evidence is observed in ScienceWorld (App. D.11).

Q4 (a fixed-depth policy RL-trained under the same budget and compute). We trained fixed-depth policies with RL at their own depth, from the same SFT checkpoint and with the same budget, RL cycles, hyperparameters and checkpoint selection. We did this for the SFT-best depth and for the RL-best depth of Table 2, including depths outside H.

  • PACE beats them in all seven settings: against the SFT-best depth by +4.7 to +18.0 pp, significantly in four (McNemar p < 0.05). It also uses fewer decisions per success in all seven.
  • On both puzzles, RL at a fixed depth does not even improve over SFT (Sliding Puzzle 45.3 vs. 48.4; Sokoban 31.2 vs. 34.4). The gain therefore does not come from RL training itself.
  • A decomposition into variants that learn only timing or only actions points the same way: PACE beats all of them (21/21).

Details: General Response G2, Tables R2–R3.

AuthorsOpenReview ↗

Response to Reviewer z8no (2/2)

Q2 (when does adaptive depth help, and how does the gain change with reversibility, observability and K?). We answer this in detail in General Response G5. In short:

  • Theory. For a learned policy, choosing the depth by state can help at any K (Prop. 1); under a budget, it can help even at the optimum (Prop. 2). PACE improves over the same-checkpoint best fixed depth in all seven settings: +8.6 to +15.6 pp in six and +3.1 pp on Sokoban (+8.5 to +13.1 and +1.6 vs. the strongest baseline in Table 2); the gain is positive in all seven, and against the same-checkpoint best fixed depth it is significant in 2 by McNemar's test and in 3 by the bootstrap CI.
  • K. Evaluated at larger budgets than used in training, PACE leads the best fixed depth by 4.7–15.0 pp in all seven settings (Table 3). At the relaxed budgets, successful episodes finish well before the limit, yet PACE still beats step-wise control in all seven settings and still mixes depths (G3).
  • Observability. Not every new observation is worth a decision (Flaspohler et al., 2020). At the relaxed budgets, step-wise control, which observes after every action, still has lower success than PACE in both partially observable domains (G5).
  • Reversibility. The gain is smallest on Sokoban, where deadlocks cannot be undone and cause 68–79% of failures; there, learned timing mainly saves cost (25.5% fewer environment steps per episode). ScienceWorld is also irreversible, yet its gains are much larger than on Sokoban, so reversibility alone does not set the gain (G5).
  • What the selected depths show (new appendix table). At the main budgets, PACE uses at least two depths in 58–98% of episodes, and two independently trained backbones most often select the same depth in six of seven settings, so the learned depth depends on the task.

We did not vary observability or reversibility within a domain, so comparisons across settings are correlational, not causal.

Q3 (actual time or token cost). We count decisions because each decision is one full forward pass of the 7B VLM backbone over the current context, which dominates compute, memory traffic and latency.

  • The extra cost per decision is negligible. The depth head is one linear layer, and the action decoder is a two-layer module over a closed action vocabulary.
  • PACE generates no free text. Each decision outputs one depth class and eight action classes, so the output length is constant, with no reasoning tokens. The cost is set by the number of decisions and the context length.
  • Decision count measures backbone calls and serves as a proxy for inference compute. Executing more actions in a chunk needs no further model call.
  • Token cost: PACE uses fewer input tokens per success in all seven settings, measured in the text settings and bounded in the puzzles. We counted the input tokens of every decision in the evaluation logs, including failed episodes.
    • Text settings (ALFWorld, ScienceWorld). We count the prompt plus the text observation. Tokens per decision differ by less than 9% between PACE and the best fixed depth, so the gap comes mainly from the number of decisions: the best fixed depth needs 1.16–1.38× more input tokens per success (e.g., 10,072 vs. 7,545 on ALFWorld seen).
    • Puzzles. The input is a fixed prompt plus one image of the current board, with no earlier frames. The board size varies across instances but not within an episode, and the logs do not store image tokens, so we bound the ratio: whether image tokens are constant, proportional to the board area, or a mix of the two, the best fixed depth needs 1.39× more input tokens per success on Sliding Puzzle and 1.08–1.10× more on Sokoban.
  • Counting model calls is common practice. Recent agent benchmarks also cap the number of model calls rather than environment steps, with several actions allowed per call (e.g., CubeBench (Gao et al., 2025) caps the model calls per run, each of which may execute several moves).

Table 14 reports decisions per success, including the cost of failed episodes. Our formal results also do not depend on counting decisions: they hold when each decision has a cost, such as the time, money or tokens of one model call, as long as one decision costs less than half the value of success (new per-decision-cost Remark, App. B).

Summary of changes. In response to this review, we added the closed-loop comparison at relaxed budgets, the invalid-action stop rule for three policies without a learned depth, an analysis of when chunking helps (G5), fixed-depth policies RL-trained at their own depth, statistics of the selected depths, the measured token cost, and new formal results (Props. 2–3 and two Remarks, App. B). We also revised the abstract, contributions and related work to state the novelty more precisely. We hope these results address the reviewer's concerns, and we would be grateful if the reviewer would consider them in the final assessment.

Dear Reviewer z8no,

With the discussion period nearing its end, we would like to follow up to see if our response has resolved your questions. We would be glad to provide any further clarification if needed.

Kind regards,
The Authors

Reviewer_nv1LOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how many actions an agent should execute open-loop before it re-observes and replans (the execution depth), under a hard budget K on the number of decisions. PACE attaches a linear depth head over H = {1, 2, 4, 8} and a small autoregressive decoder over a closed action vocabulary to a Qwen2.5-VL-7B backbone. The decoder always produces eight actions, and the depth head chooses how many of them to execute. Training is SFT on sub-chunks of reference trajectories, followed by GRPO under the budget. Proposition 1 states that if some reachable history has a depth with higher continuation success, switching to that depth there raises overall success. On Sliding Puzzle, Sokoban, five ALFWorld task families and twelve ScienceWorld task types, PACE beats the best fixed depth by 3.1–15.6 points while using fewer decisions on average. All fixed depths are evaluated on PACE's own RL checkpoint.

Critical strengths

S1: The same-checkpoint protocol is careful and covers four environments and seven settings. All fixed depths 1–8 are tested, and the best one, like the entropy threshold, is chosen in hindsight on the evaluation set, which favors the baselines. The gain is positive in every setting and persists at larger budgets without retraining (Table 3).

S2: The problem is posed cleanly, and the factorized head (Table 1) avoids free-text parsing at a cost of only ∣H∣+8∣Ae∣|H| + 8|A_e| logits. The failure analyses (App. D.9) and the state-dependent depth patterns (Fig. 5) make the learned behavior interpretable. The paper also releases a puzzle environment with exact-solver diagnostics which make the path verifiable.

Critical weaknesses

W1: The main claim, that learned depth beats fixed depth, is not isolated. All fixed depths reuse PACE's checkpoint, RL-trained with its own depth head; no policy is RL-trained at a fixed depth. RL did not help fixed depths on the puzzles (best fixed 48.4 → 45.3 and 34.4 → 32.8, Table 2). I computed that on Sokoban PACE beats SFT-only best fixed by 1 of 64 episodes.

W2: The success–decision Pareto dominance (C3) only restates the success gain. In the puzzles every failure uses the full budget K, so mean decisions are yDsucc+(1−y)Ky D_{succ} + (1-y) K; higher success alone lowers them. From Tables 2 and 14, I recomputed decisions within successful episodes: PACE needs more than best fixed on Sliding Puzzle (9.2 vs 8.1) and Sokoban (3.9 vs 3.7).

W3: The evidence is thin, and the theory does not cover the reported comparisons.

  • Each setting is one run without confidence intervals. From the reported success rates I infer only 55–138 test episodes per setting. The Sokoban gain is 2 episodes, and I computed that even the most favorable pairing gives McNemar p=0.5p = 0.5. Joint training also loses to the frozen-decoder variant in all five text settings, by 2.6–8.2 points (Table 4).
  • I checked Proposition 1 and Corollary 1. The proofs are correct, but together they amount to the standard one-step policy-improvement identity (Remark 3). Their premise is a reachable history where another depth has higher continuation success. The paper never tests it, although the puzzles' exact solvers would allow estimating it.
  • Assumption 1 fixes one shared action kernel, so the guarantee covers only the same-checkpoint comparison questioned in W1, not fixed-depth policies trained on their own. The guarantee also stops at H = {1, 2, 4, 8} (Remark 1). Yet in three settings the best fixed depth in Table 2 lies outside H: Sliding Puzzle (h = 3), ScienceWorld short (h = 6) and ScienceWorld medium (h = 5/7).
Questions for authors
  1. Can you add fixed-depth policies RL-trained at their own depth (for example the SFT-best depth and depth 8) and a step-wise GRPO agent, with the same compute and budget? This is the comparison that would isolate adaptive timing.
  2. Can you report decisions per successful episode, results over several training seeds, and paired per-instance tests against the best fixed depth?
  3. The frozen-decoder variant beats PACE in every text setting (Table 4). Which variant do you propose, and how were checkpoints and the per-domain hyperparameters in Table 9 selected?
  4. TempoRL (Biedenkapp et al., https://arxiv.org/abs/2106.05262), which is not cited, already learns an action together with how long to commit to it, and Zhao et al. (2026, https://arxiv.org/abs/2606.11408) train an RL horizon head over a chunk policy. Beyond applying this idea to a VLM under a decision budget, what does PACE add?
6 comments
AuthorsOpenReview ↗

Response to Reviewer nv1L (1/3)

We thank the reviewer for a very careful review, including the recomputations, and for noting the careful same-checkpoint protocol, the cleanly posed problem, the factorized head that avoids free-text parsing at a cost of only |H| + 8|A| logits, the interpretable failure analyses and depth patterns, and the released puzzle environment with exact-solver diagnostics. Shared points are detailed in the General Response (G1–G5).

W1 (the main claim is not isolated: no policy is RL-trained at a fixed depth). We added exactly this comparison. In every setting, we RL-trained fixed-depth policies at their own depth, with the same SFT start, budget, RL cycles, hyperparameters and checkpoint selection as PACE, for the SFT-best depth (h = 8 on Sokoban and ALFWorld, as the reviewer suggested) and the RL-best depth of Table 2, including depths outside H (see General Response G2, Table R2).

  • PACE beats them in all seven settings, by +4.7 to +18.0 pp against the SFT-best depth (significant in four, McNemar p < 0.05), and uses fewer decisions per success than both runs in all seven settings.
  • Our original protocol did not weaken the baselines. Fixed depths trained on their own never exceed the same-checkpoint fixed depths of Table 2.
  • On both puzzles, RL at a fixed depth does not improve over SFT (Sliding Puzzle 43.8 and 45.3 vs. 48.4; Sokoban 31.2 vs. 34.4). As the reviewer observed, RL helps only when the depth is learned.
  • A decomposition points the same way. PACE beats every variant that learns only timing or only actions in all 21 comparisons, with fewer decisions per success (General Response G2, Table R3).

On Sokoban, we agree that the gain over the SFT-only best fixed depth is one episode out of 64 (see W3).

W2 (Pareto dominance only restates the success gain). We agree that on the puzzles, where every failure uses the full budget, mean decisions partly reflect success, and we reproduce the reviewer's numbers (Sliding Puzzle 9.18 vs. 8.07; Sokoban 3.87 vs. 3.71). However, this comparison averages over different sets of instances: each method's own solved episodes. We therefore split it.

Table R8. Split by which method solves each instance.

SettingFixed hOwn solved episodes: PACE / fixedInstances only PACE solves: PACE (n)Both solved (n)Both solved, decisions: PACE / fixed, Δ [95% CI]Both solved, steps: PACE / fixed, Δ [95% CI]
Sliding Puzzle39.18 / 8.0710.38 (13)268.58 / 8.04, +0.54 [−0.27, +1.31]24.6 / 23.0, +1.6 [−0.7, +4.2]
Sokoban83.87 / 3.714.20 (10)133.62 / 3.31, +0.31 [−0.46, +1.00]17.5 / 21.9, −4.5 [−8.9, −0.8]
ALFWorld seen44.32 / 4.944.50 (20)434.23 / 4.79, −0.56 [−1.28, +0.12]16.0 / 17.4, −1.5 [−4.0, +0.7]
ALFWorld seen74.32 / 4.234.10 (20)434.42 / 4.12, +0.30 [−0.56, +1.16]17.2 / 24.6, −7.4 [−12.2, −2.7]
ALFWorld unseen74.32 / 4.514.39 (23)344.26 / 4.29, −0.03 [−0.91, +0.85]23.0 / 26.4, −3.4 [−9.2, +2.3]
ALFWorld unseen84.32 / 4.384.59 (22)354.14 / 4.20, −0.06 [−0.77, +0.66]23.8 / 28.8, −5.0 [−9.8, −0.4]
ScienceWorld short63.55 / 3.884.27 (11)203.15 / 3.75, −0.60 [−0.80, −0.40]19.6 / 20.4, −0.8 [−1.9, +0.3]
ScienceWorld medium55.21 / 5.884.73 (15)235.52 / 5.78, −0.26 [−1.22, +0.70]36.7 / 27.0, +9.6 [+3.7, +16.1]
ScienceWorld medium75.21 / 5.125.40 (10)285.14 / 5.25, −0.11 [−1.00, +0.79]34.1 / 33.3, +0.8 [−5.9, +7.6]
ScienceWorld long87.91 / 6.749.70 (10)257.20 / 6.60, +0.60 [−0.68, +1.80]41.5 / 48.7, −7.2 [−18.5, +2.0]

Means per episode; Δ = PACE − fixed, paired bootstrap CI; bold: CI excludes zero in PACE's favor. Tied best depths both listed.

  • PACE also solves harder instances. On the instances that only PACE solves, PACE needs more decisions than on the instances both solve (e.g., Sliding Puzzle: 10.38 vs. 8.58), in 8 of 10 rows. These harder instances raise PACE's average over its own solved episodes.
  • On the same instances, decisions are similar. On instances both methods solve, we detect no significant difference in decision counts in nine of ten comparisons, and PACE needs fewer on ScienceWorld short.
  • On the same instances, PACE often needs fewer environment steps, significantly on Sokoban and ALFWorld (three comparisons). The one exception is ScienceWorld medium against h = 5 (+9.6 steps), where PACE mostly uses h = 8.

So the comparison over each method's own solved episodes is confounded by instance difficulty. PACE solves more instances, including harder ones, with no significant difference in decision counts in nine of ten comparisons on the instances both solve, and its cost per success is lower (Table 14). We revised C3 accordingly and added this table to the appendix.

AuthorsOpenReview ↗

Response to Reviewer nv1L (2/3)

W3 (thin evidence).

  • Single runs and no confidence intervals. We add per-instance paired tests for all settings and two more training seeds for ALFWorld and Sliding Puzzle. The gain is positive in all seven settings, significant in two by McNemar, and the bootstrap CI excludes zero in three (see General Response G4, Table R5). All nine per-seed gains are positive (minimum +7.8 pp), and the seed in Table 2 gives the smallest gain on Sliding Puzzle and ALFWorld unseen (see General Response G4, Table R6). A second, independently trained backbone (gemma-3-12b-it) shows a positive gain in all seven settings, significant in four, with fewer decisions per success throughout (see General Response G4, Table R7).
  • Sokoban. We agree. As §4.2 already notes, the success gain on Sokoban is modest, and our new paired test shows that it is not significant (p = 0.81). We revised the abstract and contributions to say that PACE improves success in six of seven settings with no significant difference on Sokoban. On Sokoban, PACE is more efficient: 25.5% fewer environment steps per episode, fewer steps and decisions per success (83.6 vs. 123.2; 14.6 vs. 16.0; Table 14), and fewer steps on instances both solve (17.5 vs. 21.9; 95% CI of the difference [−8.9, −0.8]; Table R8).
  • Joint training loses to the frozen-decoder variant in the text settings. We propose PACE, with full joint training; it beats the best fixed depth in every setting, and all main results use it. We rename the frozen-decoder variant to PACE (frozen AD), since its action loss still updates the backbone LoRA, which changes the actions indirectly through the representation that the frozen decoder reads. It goes further in the text domains (+2.6 to +8.2 pp) but is not enough on the puzzles (−14.1 / −15.6 pp below PACE). See General Response G2, Table R3 and "Which variant we propose".

W3 (the theory does not cover the reported comparisons).

  • Prop. 1 is standard. We agree that Prop. 1 and Cor. 1 specialize one-step policy improvement (Remark 3), and the revision presents them as such.
  • Depths outside H. We agree. Applied to a depth outside H, Prop. 1 yields a policy that still uses that depth elsewhere, which is not a policy over H, so for PACE the guarantee holds only within H (revised Remark 1). That PACE beats depths 3, 5, 6 and 7, including fixed-depth policies trained on their own (W1), is an empirical result beyond this guarantee. The new Prop. 2 shows that such a separation is possible in principle: on a constructed task, an adaptive depth over H strictly beats every fixed depth from 1 to 8.
  • Assumption 1 (one shared action kernel). We agree that Prop. 1 covers only the same-checkpoint comparison. This is on purpose: fixing the actions isolates the effect of timing. Fixed-depth policies trained on their own are covered in two other ways. Empirically, PACE beats them in all seven settings (W1). In theory, Prop. 2 holds against fixed-depth policies with any action mechanism, including ones trained at their own depth, though only on a constructed task.
  • The premise is not tested. We did not directly estimate the continuation-value condition in Prop. 1. The same-checkpoint results provide empirical evidence for beneficial timing choices, but do not verify this sufficient condition. Prop. 2 separately establishes that a strict separation is possible on a constructed task.
  • What the budget adds (new, App. B). Prop. 2: without a decision cost, deciding at every step is optimal; with a budget, an adaptive depth over H can succeed with probability 1 while every fixed depth from 1 to 8 fails with high probability. Prop. 3: under a budget, a frozen action kernel can fail under every depth policy, even one that sees the sampled chunk, while adapting the actions succeeds. Both results also hold with a soft cost per decision, as long as one decision costs less than half the value of success (new Remark). These are existence results, not guarantees for the learned network (Remark 2).
AuthorsOpenReview ↗

Response to Reviewer nv1L (3/3)

Q1 (fixed-depth policies RL-trained at their own depth, and a step-wise GRPO agent). The fixed-depth policies are in W1 (see General Response G2, Table R2). A step-wise agent is a policy RL-trained at h = 1. Under the main budgets, it can execute at most K actions, so it fails on every instance whose solution needs more than K actions, however it is trained; the h = 1 evaluation of our checkpoint reaches 21.6% / 15.4% on ALFWorld seen / unseen and at most 1.6% elsewhere. We therefore trained dedicated h = 1 policies with GRPO at a relaxed budget (K = 50) on the puzzles; due to limited compute and time during the rebuttal, we could not train them for the text domains.

Table R9. Dedicated step-wise policies at K = 50.

Setting (K = 50)Dedicated h = 1 (GRPO)PACE checkpoint, evaluated at h = 1PACE
Sliding Puzzle57.8 / 60.956.2 / 63.168.8 / 36.0
Sokoban39.1 / 97.240.6 / 92.746.9 / 62.2

Each cell: success rate (%) / decisions per success.

  • PACE beats the dedicated step-wise policies (+11.0 / +7.8 pp), which need 1.7× and 1.6× more decisions per success.
  • Training at h = 1 adds little over evaluating our checkpoint at h = 1 (within 2 pp), which supports using the latter in the text domains. There, with relaxed budgets (K = 50; 80 / 150 for ScienceWorld medium / long), step-wise control also loses to PACE in all five settings, including 4.9% vs. 70.5% on ScienceWorld long (see General Response G3, Table R4).

Q2 (decisions per successful episode, training seeds, paired tests). Decisions per successful episode, split into instances solved by both methods and instances solved by only one, are in Table R8 (W2). Results over three training seeds and paired per-instance tests are in W3 (see General Response G4, Tables R5–R6).

Q3 (which variant, and how hyperparameters and checkpoints were selected). We propose PACE, with full joint training (W3; see General Response G2, Table R3 and "Which variant we propose"). Checkpoints and hyperparameters were selected on a validation set that is disjoint from both the training and the test instances (for ALFWorld, held out from the training split). Checkpoints are chosen by validation success, with ties resolved toward the earlier checkpoint; independently trained fixed-depth baselines follow the same rule at their own depth. One hyperparameter configuration is chosen per domain, and the shared hyperparameters are identical for PACE and all baselines. This selection is based on PACE's own validation success, so it may favor PACE; on the other hand, the best fixed depth of every baseline is chosen in hindsight on the test set, which favors the baselines. Across three training seeds, PACE leads in every seed. The decision budgets were set from the lengths of the reference trajectories (App. D.4).

Q4 (novelty beyond TempoRL and DEHP). We agree that learning how long to commit is known; we added TempoRL (Biedenkapp et al., 2021), which, like FiGAR (Sharma et al., 2017), learns how long to repeat a single action without a decision cost (see G1), and the submission already cites DEHP (Zhao et al., 2026). PACE does not claim this principle. What PACE adds is what changes when each decision has a cost:

  • Timing matters even for an optimal policy (new Prop. 2), while without a decision cost, deciding at every step is optimal.
  • A frozen policy is not enough (new Prop. 3): actions later in a chunk run without feedback, so they must also be right. This motivates adapting the actions too; our ablations support learning both inside the policy, rather than timing on top of a frozen one as in DEHP. The DEHP-style variant helps in the text settings but falls below the SFT-best fixed depth on both puzzles (−3.1 / −17.2 pp), and PACE beats every variant that learns only one side (21/21).
  • A recipe that makes this joint learning work on large multimodal backbones: a factorized action space with only |H| + 8|A| logits, which the reviewer noted, trained with GRPO through LoRA on 7B and 12B backbones.

Details are in General Response G1.

Summary of changes. In response to this review, we added fixed-depth policies RL-trained at their own depth, a decomposition into variants that learn only one side, the paired comparison on instances solved by both methods, per-instance paired tests, two more training seeds, dedicated step-wise policies, and new formal results (Props. 2–3 and two Remarks, App. B). We also revised the abstract and contributions (including Sokoban) and the description of Prop. 1. We hope these results address the reviewer's concerns, and we would be grateful if the reviewer would consider them in the final assessment.

Reviewer_LxcaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper investigates the problem of LLM-based in multi-step long-horizon settings. Specifically, a plan-then-act loop where the agent makes sequential actions interactively within an environment.
  • To make the agents more efficient at inference-time, action chunking is often used such that the policy generates a sequence of hh actions (the 'execution depth') to be take consecutively from the current state. Classically, hh is estimated and fit for a particular dataset and can be a sub-optimal choice.
  • The paper proposes dynamic selection of execution depth. This is implemented by extending an LLM backbone with two additional predictor heads: action and depth. The model (LoRA adapters) is trained individually per task with SFT and GRPO.
  • The approach is evaluated on a range of datasets (ALFWorld, ScienceWorld, etc.). Results are promising: dynamic selection of depth outperforms standard RL training and the fixing the best depth per task.
Critical strengths
  1. Useful bottleneck identified: Indeed, many prior works (e.g., robotics) have shown action chunking is beneficial. This is typically statically set, which generally works in certain situations involving periodic perception and sensor control. However, in certain cases (notably agentic), a statically determined depth (length of action chunk sequence) can be sub-optimal e.g., initial exploration might warrant lower depth. The authors exploit and prototype this useful insight.
  2. Promising results: The results fro their approach is promising e.g., showing consistent gains across multiple tasks.
Critical weaknesses

Major Concerns

1. Single backbone: Qwen2.5-VL-7B / lower baseline success rates compared to prior works

  • While the results are promising, they are in reference to a single (somewhat dated) backbone model: Qwen2.5-VL-7B. Two sub-concerns here:
  • (a) Unclear if proposed approach would show similar benefits with other backbone models
  • (b) The base performance itself of Qwen2.5-VL-7B seems rather low. For instance, the paper reports success rates with GRPO of 38-45pp on ALFWorld (Table 2). In contrast, prior work (e.g., "Group-in-Group Policy Optimization for LLM Agent Training", Feng et al., NeurIPS 2025) reports 77.6pp. It is unclear why this discrepancy arises (because of the very large admissible set of actions here? / Table 6)

Minor Concerns

2. Rationale: Independent sampling of depth and actions

  • An important technical decision of the proposed approach is independently sampling the depth and sequence of actions.
  • This appears somewhat counter-intuitive, since sampling depth conditioned on the predicted action chunk appears to provide a richer signal (e.g., an invalid move can be pruned)

3. Task-specific chunking; generalizability unclear

  • While I believe the dynamically determining the execution depth is useful, the authors evaluate in a task-specific setting i.e., the models are trained in a task-specific manner for task-specific vocabulary of actions (L234 "each puzzle is trained separately")
  • As a result, it remains unclear whether the learnt capability (dynamic execution depth) generalizes across tasks. I believe this somewhat weakens the impact of the results.

4. Action-chunking in partially-observable environment: justification

  • Generally, the idea of action chunking is perfectly reasonable in a fully-observable environments (e.g., Sokoban), where the policy can account and preplan the actions.
  • However, the paper also studies POMDPs where relevance of action-chunking is somewhat unclear. For instance in ALFWorld, where an agent needs to find an item by opening various doors, drawers and cabinets. In this specific case, the set of admissible actions are dependent on the agent's location and environment's state.
Questions for authors

Clarification requests:

  1. Why is there a significant gap between the baseline success rates (Prompting/GRPO) between the Qwen2.5-7B model in this paper as opposed to prior works?
  2. What is the rationale for sampling depth independent of selected action chunks?

Suggestions for future work:

  1. To evaluate whether learnt "execution depth" can generalize across tasks
  2. Providing analysis on when/whether action chunking is reasonable for POMDPs
4 comments
AuthorsOpenReview ↗

Response to Reviewer Lxca (1/2)

We thank the reviewer for the constructive review and the concrete questions, and for recognizing that a statically set execution depth is a useful bottleneck to address, notably in agentic settings, and that our results show consistent gains across multiple tasks. Points shared with other reviewers are detailed in the General Response (G1–G5).

Major 1a (single backbone). We repeated the main comparison with gemma-3-12b-it, a different model family, using the same pipeline, budgets and protocol (see General Response G4, Table R7). PACE beats the best fixed depth of the same checkpoint, selected in hindsight over h = 1, …, 8, in all seven settings (+6.3 to +20.0 pp), significantly in four (McNemar p < 0.05), and uses fewer decisions per success in all seven (e.g., 11.7 vs. 15.4 on ALFWorld seen). It also beats the best fixed depth of the SFT checkpoint in all seven settings. The direction of the gain matches Qwen2.5-VL-7B in every setting.

Major 1b / Q1 (gap to prior ALFWorld results). Our relaxed-budget experiment shows that the tight decision budget is an important constraint in our setup, but it does not isolate the causes of the gap across papers. The 77.6% is the GRPO baseline that GiGPO (Feng et al., 2025) reports with Qwen2.5-7B-Instruct (77.6 ± 5.2). The two protocols differ substantially:

GiGPOOurs (main results)
Model calls per episodeup to 50, one action per callat most K = 9 decisions
Feedbackafter every actiononly between chunks (open-loop chunks)
Action choiceadmissible actions listed in the prompt at every stepclosed vocabulary of 5,698 commands; only the first action of a chunk is masked to admissible commands (about 32 per state)
BackboneQwen2.5-7B-InstructQwen2.5-VL-7B

To test the effect of the budget, we relaxed it to K = 50 with the same checkpoint (see General Response G3, Table R4):

Table R11. ALFWorld with the main and the relaxed budget.

SplitK = 9: PACEK = 9: h = 1K = 50: PACEK = 50: h = 1
seen54.3 / 11.921.6 / 39.175.9 / 25.370.7 / 36.2
unseen48.7 / 13.815.4 / 57.264.1 / 36.555.6 / 58.2

Each cell: success rate (%) / decisions per success. "h = 1" is the PACE checkpoint evaluated with one action per decision.

  • Relaxing the budget substantially improves success. With the same checkpoint and vocabulary, K = 50 raises PACE to 75.9 / 64.1 and step-wise control to 70.7 / 55.6. Step-wise control executes one action per decision, so at most 50 environment steps, as in the 50-step protocols; it is comparable to 71.9 / 54.7 (Qwen3-4B) on seen / unseen in SPACE (Yang et al., 2026), and below 77.6 ± 5.2 in GiGPO (one overall split) and 78.9 / 75.0 (Llama-3.1-8B-Instruct) in SPACE. PACE at K = 50 executes several actions per decision, so it is not matched to these protocols in environment steps.
  • Action interface. The first action of each chunk is chosen among the admissible commands of the current state (31.6 on average on ALFWorld seen); later actions in a chunk are chosen from the full 5,698-command vocabulary (Table 6) without this restriction. We do not isolate the effect of the vocabulary or the action interface.
  • Other differences to GiGPO. As App. C.4.5 notes, GiGPO adds step-level credit assignment, which is complementary to decision timing, and trains with more compute: 150 iterations with 16 × 8 trajectories each, about 19,200 trajectories. We train for 50–100 cycles under our compute budget (Table 9).

These numbers are not directly comparable: splits, task families and backbones differ, and we exclude Pick Two, the hardest task family, which favors our numbers. We claim only that the tight budget is an important constraint in our setup.

AuthorsOpenReview ↗

Response to Reviewer Lxca (2/2)

Minor 2 / Q2 (why sample depth and actions independently?). This is a deliberate design (§3, Table 1).

  • A standard design. Predicting several outputs with separate heads on a shared representation is common, e.g., class and box heads in object detection (Ren et al., 2015), action branching in RL (Tavakoli et al., 2018), and multi-token prediction in language models (Gloeckle et al., 2024). Here it keeps the output small (|H| + 8|A| logits from one forward pass) and the two log-probabilities separate for GRPO.
  • It loses nothing at the optimum (new Remark, App. B): the budget-K problem has a deterministic optimal policy (Lemma 1), and any deterministic joint choice is a product of two point masses. It assumes full-history policies; PACE's puzzle inputs show only the current board, so for PACE it is a guide, not a guarantee.
  • The two heads still adapt to each other. They read the same representation, and the RL loss includes only the executed actions, so the selected depth decides which decoder positions receive gradient.

We also tested the reviewer's suggestion: the depth head additionally reads the decoded chunk (a → h), trained from the same SFT checkpoint for the same number of RL cycles.

Table R12. Depth conditioned on the decoded chunk (a → h).

SettingBest fixed depth (PACE checkpoint)a → hPACE
ALFWorld seen45.7 / 14.947.4 / 13.954.3 / 11.9
ALFWorld unseen38.5 / 18.840.2 / 18.348.7 / 13.8
Sliding Puzzle45.3 / 26.256.2 / 21.460.9 / 18.8

Each cell: success rate (%) / decisions per success.

This chunk-conditioned variant does not improve over PACE in any of the three settings, and it needs more decisions per success. As for pruning invalid moves, a version with more information does not help either: stopping a chunk at the first invalid action uses the environment's own feedback, which is more than the chunk alone reveals, yet on ALFWorld it lowers PACE's success, mostly because the extra decisions use up the budget (G5, Table R10).

Minor 3 (does the learned depth generalize across tasks?). In our understanding, generalizing across tasks or domains typically relies on joint training over many diverse tasks and can hardly be achieved by changing the model or training method alone, so it is beyond the scope of this paper. We train one policy per domain because the action vocabularies differ across domains. Within this scope, the learned depth does generalize in three ways:

  • To unseen environments. On ALFWorld unseen, the gain over the best fixed depth is at least as large as on seen in all four comparisons (Table R13).
  • Across task types. One PACE policy covers five task families in ALFWorld, and twelve task types in three horizon groups with different budgets in ScienceWorld.
  • To budgets not used in training. Without retraining, PACE leads the best fixed depth by 4.7–15.0 pp at larger budgets in all seven settings (Table 3).

Table R13. Gain of PACE over the best fixed depth (pp) on ALFWorld seen and unseen.

ComparisonSeenUnseen
Main results (Table 2)+8.6+10.3
Three training seeds (G4, Table R6)+10.3 ± 3.7+12.2 ± 1.8
gemma-3-12b-it (G4, Table R7)+9.5+13.7 (p = 0.007)
RL-trained fixed depth (G2, Table R2)+9.5+10.3

We thank the reviewer for this valuable point and leave transfer across tasks and domains to future work (App. F).

Minor 4 (why chunk in partially observable environments?). Following the reviewer's suggestion, we added an analysis of when chunking helps (G5):

  • Not every new observation is worth a decision (Flaspohler et al., 2020). At the relaxed budgets, step-wise control, which observes after every action, still has lower success than PACE on ALFWorld (70.7 vs. 75.9 seen; 55.6 vs. 64.1 unseen) and on ScienceWorld.
  • Location-dependent admissible actions. Only the first action of a chunk is masked to the admissible commands (about 32 per state on ALFWorld), so later actions can be invalid. A rejected command leaves the state unchanged and uses no decision, and stopping at the first invalid action lowers PACE's success under the budget (G5, Table R10).
  • The costly errors are valid actions executed too early, such as an answer submitted before its evidence is observed (App. D.11). This is why the actions must be adapted too, not only the depth.

Summary of changes. In response to this review, we added a second backbone (gemma-3-12b-it), relaxed-budget results on the gap to GiGPO, a protocol comparison with GiGPO, a new Remark on independent sampling with the a → h experiment, the seen / unseen comparison, and an analysis of when chunking helps under partial observability (G5). We hope these results address the reviewer's concerns, and we would be grateful if the reviewer would consider them in the final assessment.

Dear Reviewer Lxca,

With the discussion period nearing its end, we would like to follow up to see if our response has resolved your questions. We would be glad to provide any further clarification if needed.

Kind regards,
The Authors

Dear Reviewer Lxca,

In case it is helpful before the discussion closes, here is a short summary of how our responses address each point:

  • Second backbone (Major 1a). With gemma-3-12b-it, PACE has higher success than the best fixed depth of the same checkpoint in all seven settings (+6.3 to +20.0 pp), significant in four, with fewer decisions per success in all seven (Table R7).
  • Gap to GiGPO (Major 1b / Q1). With the same checkpoint, relaxing the budget to K = 50 raises PACE from 54.3 / 48.7 to 75.9 / 64.1 on ALFWorld seen / unseen (Table R11), so the tight decision budget is an important constraint in our setup. The reply also compares the two protocols and lists the remaining differences.
  • Independent sampling (Minor 2 / Q2). A new Remark shows that it loses nothing at the optimum for full-history policies. We also tested the reviewer's suggestion, a depth head that also reads the decoded action chunk (a → h): it does not improve over PACE in the three settings tested (Table R12).
  • Generalization (Minor 3). Transfer across tasks and domains typically relies on joint training over many diverse tasks and is beyond the scope of this paper; we thank the reviewer for this valuable point and added it to future work (App. F). Within a domain, the learned depth generalizes in three ways:
    • From seen to unseen environments (ALFWorld). The gain is at least as large on unseen as on seen in all four comparisons (e.g., +10.3 vs. +8.6 pp in the main results; Table R13).
    • Across horizons (ScienceWorld). One PACE policy covers short, medium and long tasks and leads the best fixed depth in each (+10.1, +10.9 and +13.1 pp; Table 2), although the best fixed depth differs across them (h = 6, 5/7 and 8).
    • Across budgets. Without retraining, PACE leads the best fixed depth by 4.7–15.0 pp at larger budgets in all seven settings (Table 3).
  • Partial observability (Minor 4). We added an analysis of when chunking helps under partial observability (General Response G5), and a controlled study that varies observability within a domain is in future work (App. F).

Thank you again for your time.

Comments on the paper

11 comments
AuthorsOpenReview ↗

General Response (1/11): Summary of new experiments and results

We thank all reviewers for their careful and constructive reviews. We are encouraged that the reviewers find the problem worth addressing and clearly posed (z8no, nv1L), agree that a statically set execution depth is a useful bottleneck to address (Lxca), and see the evaluation across four environments and seven settings as diverse and careful, with consistent gains (z8no, nv1L, Lxca).

We added new experiments addressing the reviewers' main concerns, as well as new formal results (App. B) on when timing and action adaptation matter under a decision budget. New and substantially revised content in the PDF that responds to the reviews is marked in blue. Table, figure and section numbers refer to the original submission, except App. D.11 (failure cases; D.8 originally). Tables added in this response are numbered R1–R13; in the revised PDF they are Tables 22 (R2), 3/24 (R3), 5/23 (R4, R9, R11), 20 (R5), 21 (R6), 6/25 (R7), 18 (R8), 26 (R10) and 27 (R12).

Table R1. Newly added experiments and theory.

AdditionAddressesFinding
Fixed-depth policies RL-trained at their own depth (incl. depths outside H)nv1L W1/Q1, z8no Q4PACE outperforms them in 7/7 settings; on both puzzles, RL at a fixed depth does not improve over SFT (Table R2)
Decomposition: timing-only, two action-only variants, PACE (frozen AD)nv1L W1/Q3, z8no W1PACE outperforms every variant that adapts only one side (21/21): neither RL on the actions alone nor learning the depth alone reproduces the gain (Table R3)
Relaxed budgets vs. step-wise control (K = 50; 80 / 150 for ScienceWorld medium / long), incl. dedicated h = 1 policiesz8no W2/Q2, Lxca Major 1b, Minor 4, nv1L Q1PACE outperforms step-wise control in 7/7 settings (70.5% vs. 4.9% on ScienceWorld long) with fewer decisions per success. At these budgets, successful episodes finish well before the limit (median 3–10 decisions for PACE), yet PACE still mixes depths (75–100% of episodes use at least two). On ALFWorld, K = 50 raises PACE to 75.9 / 64.1 and step-wise control (≤ 50 environment steps) to 70.7 / 55.6: the tight budget is an important constraint in our setup, though this does not isolate the gap to prior results (Feng et al., 2025; Yang et al., 2026; Tables R4, R9, R11)
Two additional training seeds (ALFWorld, Sliding Puzzle)nv1L W3/Q2All nine per-seed gains are positive (minimum +7.8 pp; Table R6)
Per-instance paired testsnv1L W3/Q2PACE has higher success in all 7 settings; against the same-checkpoint best fixed depth, significant in 2 by McNemar’s test and in 3 by the paired bootstrap 95% CI (Table R5)
Second backbone (gemma-3-12b-it)Lxca Major 1aPACE has higher success in all 7 settings, significant in 4 by McNemar’s test, and needs fewer decisions per success in all 7 (Table R7)
Paired comparison on instances solved by both methodsnv1L W2PACE also solves harder instances, which raises its average over solved episodes; on instances both solve, no significant difference in decision counts in nine of ten comparisons and PACE uses fewer environment steps in most settings (Table R8)
Invalid-action stop rule on ALFWorld, for fixed-depth policies RL-trained at h = 7 and h = 8, an action-only policy, and PACEz8no W3, Lxca Q2With the rule, every policy without a learned depth stays below PACE; on PACE, the rule lowers success, mostly by using up the budget (Table R10)
What happens after an invalid action (PACE, ScienceWorld)z8no W3, Lxca Minor 4, Q256% of the later actions in the chunk are still valid, and 36% of these chunks still gain score (G5)
Sokoban: deadlocks and detoursz8no Q2Deadlocks cause 68–79% of failures; at matched environment steps, PACE reaches step-wise success with one fifth of the decisions (G5)
Admissible commands per state (ALFWorld)Lxca Major 1bAbout 32, against a 5,698-command vocabulary; only the first action of a chunk is masked to them (G3)
Depth conditioned on the decoded action chunkLxca Q2Does not improve over PACE (3/3 settings; Table R12)
Does the learned depth generalize?Lxca Minor 3To unseen ALFWorld environments, yes: the gain over the best fixed depth is at least as large on unseen as on seen in all 4 comparisons (Table R13); across domains, future work (App. F)
Formal results: Props. 2–3 and a Remark (App. B)z8no W1/Q2, nv1L W3/Q4, Lxca Q2Under a budget, adaptive depth over H can strictly outperform every fixed depth in {1, …, 8}, including depths outside H (Prop. 2), and a frozen action kernel can fail under every depth policy (Prop. 3); independent sampling of depth and actions is lossless at the optimum for full-history policies (Remark)

Below we address the five common issues raised by several reviewers (G1–G5). Reviewer-specific points are answered in each review thread.

AuthorsOpenReview ↗

General Response (2/11): G1. Novelty and relation to prior work (1/3)

G1. Novelty and relation to prior work (Reviewer z8no W1/Q1, Reviewer nv1L Q4)

In one sentence. When each decision has a cost, timing matters even for an optimal policy (Prop. 2), and adapting timing alone can be insufficient when a frozen policy makes open-loop errors (Prop. 3); our experiments support learning both together.

What is already known. The principle that state-dependent commitment can outperform a fixed one is long established, and the original submission already acknowledges this lineage: §2 cites option interruption (Sutton et al., 1999), option-critic and its deliberation cost (Harb et al., 2018), adaptive chunking with learned values (AQC; Gireesh et al., 2026), and the RL-trained horizon head over a frozen policy of DEHP (Zhao et al., 2026, a concurrent arXiv preprint); Remark 3 states that Prop. 1 specializes option interruption "without claiming a new policy-improvement principle"; and our Repeat-last baseline follows FiGAR's learned action repetition (Sharma et al., 2017). We added TempoRL (Biedenkapp et al., 2021; §2). FiGAR and TempoRL learn how long to repeat a single action, without a cost per decision. PACE executes a sequence of different actions under a decision cost, so the later actions in a chunk must also be learned (Prop. 3); in our domains, repeating one action is rarely useful (Repeat-last: 0.0% on Sliding Puzzle and ScienceWorld, Table 2). PACE does not claim this principle. The deliberation cost also charges for decisions, but there each option still reads a new observation at every step; in PACE, a decision is a model call, and the actions within a chunk run without feedback.

Why a decision cost matters.

  • Decisions are the expensive step. In agents built on large models, each decision is a model call that costs time, compute or money. Executing a step is much cheaper. Executing several actions per call is already common practice. As Reviewer z8no notes, "many methods have relied on fixed execution horizon".
  • Chunking creates a new question. With one action per call, a limit of 50 steps is a limit of 50 calls. Once a call can execute several actions, the policy must decide how many to execute per call.
  • It changes the classic answer. Without a decision cost, deciding at every step is optimal (new Prop. 2(i), App. B). With a cost, it is not: choosing how many actions to execute per call can beat every fixed depth (Prop. 2(ii); the per-decision-cost Remark for a cost per call). Timing then becomes a decision the policy must learn.

When this applies. Our results need one condition: each decision has a cost, and executing more actions in the same chunk does not add to that cost. In the paper, the cost is a hard budget K, the maximum number of decisions per episode. A soft cost charged per decision, such as the time, money or tokens of each model call, gives the same two results, as long as one decision costs less than half the value of success (new per-decision-cost Remark, App. B).

AuthorsOpenReview ↗

General Response (3/11): G1. Novelty and relation to prior work (2/3)

(G1 continued.)

Novelty 1 (problem and theory): under a decision cost, timing matters, and adapting it alone can be insufficient.

Step 1: why timing matters.

  • For an optimal policy (Prop. 2): without a decision cost, deciding at every step (h = 1) is optimal, and choosing the depth gives no benefit. With a cost, choosing the depth can beat every fixed depth from 1 to 8, including depths outside H. So for an optimal policy, the decision cost is the only reason timing matters.
  • For a learned, non-optimal policy, choosing the depth can help even without a cost. By the argument of Prop. 2(i), such gains come from the policy not being optimal, as with any learned policy. DEHP is such a case: its objective has no cost per decision, and it motivates adaptive horizons by a trade-off in which "longer execution horizons improve smoothness and efficiency but reduce reactivity".
  • In our setting, both apply. With the same checkpoint, relaxing the budget to K = 50 (80 for ScienceWorld medium) raises h = 1 from 0–21.6% to 14.5–70.7%, so the main budget limits step-wise control. At these budgets, successful episodes finish well before the limit (a median of 3–10 decisions for PACE), yet h = 1 still loses to PACE in all seven settings (4.9% vs. 70.5% on ScienceWorld long at K = 150; Table R4). This is consistent with the learned policy not being optimal, the case covered by Prop. 1 at any K.

Step 2: a decision cost makes a frozen policy insufficient. Under a decision cost, the policy must run long chunks without feedback, so the later actions in each chunk must also be right. Choosing the depth well cannot fix wrong actions. New Prop. 3: a frozen policy can fail under any depth policy, even one that sees the sampled chunk first, while adapting the actions succeeds. In the construction of Prop. 3, where the first action is always correct, the frozen policy acting one step at a time would suffice without a decision cost; the decision cost is what makes it insufficient.

Step 3: in our experiments, learning both inside the policy works best. In PACE, the depth head reads the policy's own representation, which is trained together with the actions. The experiments support this design:

  • PACE beats every variant that learns only one side (timing only, or actions only in two variants) in all seven settings (21/21; Table R3).
  • The DEHP-style variant (frozen policy, only the depth head trained) is even worse than the SFT-best fixed depth on both puzzles (−3.1 / −17.2 pp; Table R3).
  • Training the shared representation as well raises success by +6.0 to +14.8 pp in the five text settings; on the puzzles, the action decoder must also be trained, which adds +14.1 / +15.6 pp.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.