acceptodds
Under review as a conference paper at ICLR 2027

Learning Chunk Values from Partial Execution in Offline-to-Online Reinforcement Learning

Abstract

Action chunking, in which a policy generates sequences of actions rather than individual actions, can improve the sample efficiency of reinforcement learning through temporally coherent exploration and multi-step value learning. Adaptive action chunking selects how many actions to execute based on the current state, allowing the agent to replan using new observations. Though only part of a full chunk is executed, the policy must learn to generate a complete action chunk. Learning the value of that chunk from partial execution can provide feedback for improving the entire sequence. In this paper, we propose Partial-Execution Q-Chunking (PEQC), an offline-to-online reinforcement learning method that learns the values of complete action chunks from partial execution. PEQC retains generated chunks and constructs additional learning targets for the full-chunk value used in policy optimization by combining observed rewards with the estimated value of the original unexecuted actions. On OGBench, PEQC improves average evaluation success during online fine-tuning on several manipulation and navigation tasks relative to the same agent without this additional supervision. The observed gains suggest that full-chunk values can be effectively learned from partial execution while retaining the flexibility to replan before completing the generated action sequence.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, action chunking, offline-to-online reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fbw8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Partial-Execution Q-Chunking (PEQC), which learns a Q-function over prefixes of action chunks and adaptively selects the chunk size based on Q-values at test time. The Q-function is trained using Bellman backups with full action chunks, and the policy is extracted using QC-FQL. The method is evaluated on OGBench and outperforms prior methods on 2 out of 6 tasks.

Critical strengths

S1. The method is simple and easy to understand.

  • The method extends existing action-chunking approaches by learning a Q function over action-chunk prefixes and extracting the policy using QC-FQL (+ PE backups during online phase)
  • The execution length is adaptively selected using a Boltzmann policy over the Q values, a simple mechanism for dynamically adjusting the action-chunk size.
Critical weaknesses

W1. The experimental results are not sufficiently strong.

  • While PEQC outperforms prior methods on AntSoccer and cube-triple, its performance on the remaining tasks is generally comparable to or worse than existing methods, such as FQL, QC-FQL, and QC (I classified ACSAC / ACH as concurrent works, so I'm not mentioning those methods in this comparison).
  • Although limited empirical improvements are not necessarily a reason for rejection on their own, I think the technical novelty (using prefix-based Q-values for adaptive action-chunk execution) of the proposed method is not enough to compensate the weak experimental results
  • Given the limited empirical improvements, I am not convinced that the paper shows a sufficient advancement over prior work.

W2. The proposed value learning objective lacks a clear mathematical interpretation.

  • The value function is estimated by averaging Q-values over full action chunks. However, the Q-function is trained to evaluate partially executed action chunks through a different backup procedure, so I'm not sure the resulting Q function will be a Q function of a interepretable policy.
  • That is, the proposed objective appears to be a heuristic design choice rather than a Bellman-consistent objective, and the paper does not provide a sufficiently clear mathematical motivation for it.
  • The ablation studies provide some empirical support for this design choice, but given the limited improvements discussed in W1, I do not find the current justification sufficiently convincing.
Questions for authors

Q1. Could the authors clarify the mathematical interpretation of the value learning objective?

  • What policy does the learned Q function correspond to? e.g., can the proposed Bellman backup be derived as the policy-evaluation operator?
  • If the objective does not correspond to either the full-chunk policy or the adaptive execution policy, what motivates this particular combination of value estimation and Bellman backups?

Q2. Could the authors provide offline-to-online learning curves rather than only reporting final performance?

  • Since PEQC is designed for offline-to-online reinforcement learning, it would be helpful to report performance throughout online fine-tuning, as done in prior work such as QC.
  • In particular, these results would clarify whether the proposed method improves initial offline performance, online sample efficiency, or primarily final performance.

Q3. Could the authors provide evidence that adaptive action-chunk execution is actually being utilized?

  • The paper motivates PEQC through the benefits of adaptively selecting action-chunk execution lengths. However, it is unclear whether the learned execution policy actually selects different chunk lengths across states or simply converges to a Markovian policy or a fixed full-chunk policy.
  • It would be helpful to visualize the distribution of selected execution lengths, potentially across different tasks and stages of training.
3 comments

We would like to express our sincere gratitude for the detailed questions about the value objective and empirical contribution. We respond to each point below. Revisions are marked in blue.

W1. Empirical contribution.

We extended the state-value comparison to five tasks and three seeds per family (Table 21). With the weighted-prefix state-value target, PEQC reaches 79.20% final success on AntMaze and 83.07% on HumanoidMaze. Table 1 reports FQL means of 74% and 56% and QC means of 47% and 32%, respectively, under their published protocols. See Appendix K for the detailed discussion.

W2, Q1. Interpretation of the value objective.

Section 3.1 and Appendix B.1 clarify the policy evaluated by the default full-chunk objective. Let πH\pi_H generate a chunk using the one-step policy μω\mu_\omega, execute all HH actions, and repeat from the resulting state. With the actor fixed, its exact sequence values satisfy

Q^{\pi_H}(s_t,\boldsymbol{a}_{t:t+k}) &=\mathbb{E}\\!\left[ R_t^k+\gamma^k V^{\pi_H}(s_{t+k}) \mid s_t,\boldsymbol{a}_{t:t+k} \right],\\\\ V^{\pi_H}(s_t) &=\mathbb{E}_z Q^{\pi_H}(s_t,\mu_\omega(s_t,z)), \end{aligned}$$ where $R_t^k$ is the discounted reward accumulated over the first $k$ steps. Each prefix value evaluates its prescribed actions followed by the same full-chunk policy. Appendix B.1 derives the corresponding policy-evaluation operator. The actor increases predicted full-chunk value while remaining close to the behavior flow policy, giving an approximate, regularized greedy update in action-chunk space. State-value regression supplies the updated continuation estimate to the shorter-prefix Bellman targets. For a prescribed chunk and execution length $k$, the partial-execution (PE) backup uses the return decomposition $$Q^{\pi_H}(s_t,\boldsymbol{a}^{\pi}_{t:t+H}) = \mathbb{E}\\!\left[ R_t^k+\gamma^k Q^{\pi_H}(s_{t+k},\boldsymbol{a}^{\pi}_{t+k:t+H}) \right].$$ The suffix value evaluates the remaining $H-k$ original actions before continuing with $\pi_H$. Thus, the full-chunk backup and the PE backup evaluate the same original chunk under this reference policy. Adaptive execution uses these reference values to choose when to replan. With exact prefix values, greedy length selection has expected discounted return no lower than full-chunk execution by the same actor, since $H$ remains an available choice (Appendix B.1). **Q2. Offline-to-online learning curves.** Following the reviewer's suggestion, Figures 2 and 5 show one million offline updates followed by one million online environment steps. Both variants start from the same pretrained model, and PE is introduced only during the online phase. Curves average three seeds per task, with Figure 2 also averaging the five tasks in each family. PE achieves a higher mean normalized area under the online success curve (AUC) on Cube Triple, AntMaze, AntSoccer, and Puzzle. **Q3. Adaptive execution.** Figure 7 shows the selected execution-length distributions under greedy evaluation. On AntMaze, HumanoidMaze, and AntSoccer, each evaluated policy selects all five candidate lengths. The shortest prefix is most frequent, while full chunks account for about one fifth of decisions. Figure 6 reports the lengths used during data collection.

Thank you for your response.

Empirical contribution.

Thank you for running the experiments. However, I think using weighted prefixes should not be considered as "tunable hyperparameter", as it is one of the core component (value target). Specifically, I think it is an unfair comparison towards other baseline, as this comparison is comparing "maximum of two algorithm" compare to a single algorithm.

Interpretation of the value objective.

Thank you for giving the technical insight. The main problem I had in mind (and the authors restates it in the rebuttal) was "With exact prefix values, greedy length selection has expected discounted return no lower than full-chunk execution by the same actor, since remains an available choice". This literally means that we do not have to do adaptive execution, we can just train a markovian policy out of the objective.

Adaptive execution

Thank you for running the experiment. I think this works as an empirical evidence on adaptive execution, but still misses an explanation on why the proposed objective encourages (or leads to) adaptive execution, and why it is better than learning a markovian policy.

Therefore, I maintain my score.

Thank you for clarifying the remaining concerns.

Comparison of value targets.

We agree that the two value targets should be treated as distinct objective variants rather than as tunable hyperparameters. To clarify, our main benchmark comparison in Table 1 consistently uses the full-chunk target across all environments. The weighted-prefix results in Table 21 are presented as an ablation of the value objective, and we interpret them separately from the main benchmark comparison. Thus, the current main comparison does not select the better-performing variant for each task. However, we acknowledge that our method does not achieve the best performance on all six environments in Table 1, although neither does any other baseline. As discussed in the limitations and future work section, further extending the evaluation to visual observations and multi-task learning would help assess the applicability of partial-execution supervision beyond the current experimental setting.

Value learning and adaptive execution.

PEQC separates the horizon over which actions are jointly optimized from the length executed before replanning. The actor learns a complete action chunk, while the prefix critic evaluates how long to execute it.

The exact values in Appendix B.1 describe the learned full-chunk policy πH\pi_H. For this policy, Equation (6) gives

&Q^{\pi_H}(s_t,\boldsymbol{a}^{\pi}_{t:t+H}) -Q^{\pi_H}(s_t,\boldsymbol{a}^{\pi}_{t:t+k}) \\\\ &\quad=\gamma^k\mathbb{E}\\!\left[ Q^{\pi_H}(s_{t+k},\boldsymbol{a}^{\pi}_{t+k:t+H}) -V^{\pi_H}(s_{t+k}) \right]. \end{aligned}$$ The right-hand side compares completing the original suffix with generating a new chunk from the reached state. A positive difference favors completing the original chunk, while a negative difference favors replanning after $k$ actions. Their ordering can depend on the state and generated actions even when the values are exact. Thus, exact evaluation does not make $k=1$ universally preferable. Each prefix head learns its corresponding return, and greedy length selection uses these differences to determine when to replan. Training a Markovian actor from the single-action head would replace the full-chunk policy-improvement objective. Our motivation for retaining chunk optimization is to jointly improve action sequences under the learned behavior prior, using the temporal structure of offline data and multi-step value learning of Q-chunking [1]. Prefix values provide flexibility in executing these sequences. The PE backup supports this combination by providing supervision for the original full-chunk value even when execution ends after a shorter prefix. [1] Li et al., Reinforcement Learning with Action Chunking, 2025. Once again, we are grateful for your insightful feedback throughout the discussion, which has significantly helped us improve our paper.
Reviewer_Hu1XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PEQC keeps the original action chunk after replanning and adds a TD target based on the observed rewards and the estimated value of its unexecuted suffix. The point is to train the full-chunk value used by the actor, even when only part of that chunk is executed.

Critical strengths

The paired experiments give useful evidence that the backup helps learning on Cube Triple and AntMaze. Online AUC improves by about 11 and 13 percentage points, respectively, and both gains remain significant after correction for multiple comparisons. On Cube Triple, the baseline eventually catches up; reporting the learning curves rather than just final success matters here.

Critical weaknesses
  • Much of the improvement is already available without evaluating the unused suffix. In Table 9, an extra executed-prefix backup raises AntMaze AUC from 43.3 to 53.6, versus 56.5 for PEQC. The corresponding gap on Cube Triple is also only about three points. This leaves a fairly small gain for the paper’s main new idea.

  • The collection rule changes the picture considerably. Under greedy collection, the AntMaze AUC gain shrinks to 1.3 points, and the AntSoccer gain reverses. HumanoidMaze is worse with the backup under all three rules. I would put these results closer to the main comparison, rather than leave them in Appendix I.

Questions for authors

Please see the negatives above.

In addition, the behavior constraint applies to the full chunk at its starting state. How well does it control the suffix queries at the later state, especially when execution stops very early?

4 comments

We sincerely appreciate the reviewer's careful assessment and recognition of the online learning improvements. We address the concerns below, with the corresponding manuscript revisions marked in blue. AUC denotes the normalized area under the online success curve.

Contribution of the original suffix.

We increased the number of training seeds from three to five and compared Prefix-VV with PE on all five tasks in each environment. Unlike the executed-prefix backup in Table 9, both targets supervise the same original full-chunk prediction, using either the state value at st+ks_{t+k} (Prefix-VV) or the value of the original suffix (PE). The variants share the offline checkpoint, auxiliary weight, and training budget for each task and seed.

The table below reports the Prefix-VV and Original (PE) results, including AUC and final success rate (Final). Both metrics are percentages, reported as mean ± sample standard deviation across five seeds after averaging the five tasks within each seed.

EnvironmentMetricPrefix-VVOriginal (PE)
Cube TripleAUC76.70 ± 6.8283.98 ± 2.59
Final94.32 ± 8.2299.36 ± 0.61
AntMazeAUC49.01 ± 4.7154.33 ± 3.75
Final71.84 ± 6.3377.28 ± 7.14
HumanoidMazeAUC53.00 ± 4.7456.32 ± 3.49
Final57.76 ± 7.7159.36 ± 6.12

PE achieves higher mean online AUC and final success than Prefix-VV in all three environments. Paired tests with Holm correction support its AUC advantage over Prefix-VV on Cube Triple and HumanoidMaze.

Dependence on data collection.

Following the reviewer's suggestion, Section 4.1 now refers readers to the collection comparisons in Appendix I alongside the main learning results. Table 17 reports both learners under softmax, greedy, and mixture collection.

We also conducted additional training with greedy collection on Cube Triple. PE increases mean AUC from 79.15 to 84.45, showing a learning advantage under this collector as well. Under uniform collection on AntMaze, PE increases mean AUC from 44.49 to 50.88 (Table 18).

I appreciate the feedback from the authors, which have resolved most of my points.

We appreciate the reviewer’s kind acknowledgment that our response has resolved the concerns. We sincerely thank the reviewer for the positive assessment.

We would like to follow up on your question about how the behavior constraint applies when chunk execution stops early.

The behavior constraint applies to the entire chunk at its starting state, not at the state reached after partial execution. Even if execution stops after the first action, the critic evaluates the remaining actions from the original chunk at the reached state.

We assess the accuracy the PE target using Monte Carlo rollouts that execute the remaining actions from the reached state and then follow the full-chunk policy. Across the sampled partial executions in Table 13 (Appendix G.2), the PE target has lower mean squared error than the state-value target on Cube Triple and HumanoidMaze, with similar error on AntMaze.

Once again, we are grateful for your thoughtful feedback throughout the discussion.

Reviewer_abgcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PEQC, an extension of QC-FQL for offline-to-online RL with adaptive action chunking. A causal Transformer critic predicts values for different chunk prefixes to decide how many actions to execute. The main contribution is a partial-execution backup that updates the full-chunk Q-value using the observed prefix return and the estimated value of the unexecuted suffix. Experiments on 30 state-based OGBench tasks show statistically significant online AUC improvements on Cube Triple (+10.96) and AntMaze (+13.16), but no significant gains on the other task families and a decline in HumanoidMaze final success.

Critical strengths
  1. The paper addresses a real mismatch between evaluating full action chunks and executing only prefixes, with a clear explanation of the proposed backup.
  2. The experimental reporting is particularly thorough, with paired evaluations, corrected significance tests, confidence intervals, and openly reported negative results.
  3. The ablations are detailed and useful, covering alternative backup targets, execution strategies, data collection methods, and wall-clock performance.
  4. The method is easy to integrate as an auxiliary loss without additional environment interaction, although training takes about 1.3–1.4× longer.
Critical weaknesses
  1. It's unclear how much the original-suffix bootstrap, the paper's distinctive idea, actually helps. In Table 9, a simpler executed-prefix backup trained on the same chunk records recovers most of the gain (AntMaze AUC 53.6 vs. 56.5 for PEQC and 43.3 without a backup; Cube Triple 80.7 vs. 83.9 vs. 76.4). The remaining gaps are within the seed SDs and are not tested, and in one of three earlier comparisons a fresh-suffix variant beats PEQC (HumanoidMaze Task 1).
  2. The gain depends heavily on the data collector (Table 15). Softmax collection, used in the main comparison, is the no-backup learner's worst collector on AntMaze and AntSoccer. From Table 15, when each learner uses its best collector, the AntMaze AUC gain drops from +13.2 to +5.0 and the final-success gain from +20.8 to +1.6 points. The main text does not discuss this.
  3. The statistical support is thinner than the aggregate results in Fig. 3 suggest. After Holm correction, only 2 of 12 comparisons are significant, both on AUC, and no final-success gain survives (Table 5). One of them, Cube Triple, uses the earlier development-phase runs (App. D). HumanoidMaze final success drops by 9.87 points, and after normalizing by wall-clock time, only AntMaze keeps an advantage (Table 17).
  4. The adaptive collector produces almost the same marginal length distribution as uniform sampling (about 80% partial chunks and mean k/H≈0.59k/H ≈ 0.59 vs. 80% / 0.60, Table 14). Greedy and mixture collectors are compared (App. I), but training with uniform-random or fixed-H collection is not, and β is fixed at 1.
  5. The backup is closely related to intra-option learning and to the all-persistence target of Sabbioni et al. (2023), which the paper itself discusses. The difference it states, applying the idea to policy-generated continuous sequences, is small. The paper also does not analyze the bias of evaluating a suffix generated for sts_t at the reached state st+ks_{t+k}, and it does not measure the target's error directly.
  6. The evaluation is limited to state-based OGBench with three seeds. λ=1/H\lambda = 1/H, HH and β\beta are not studied, and the 1/31/3 scaling of the actor update in the navigation families is not explained. The earlier H=10H = 10 Puzzle runs are excluded without results or a reason, and prior adaptive-chunking methods such as ACH are only compared through published numbers under different protocols.
Questions for authors
  1. Can you supervise the same original full chunk with three different continuation targets? They are (i) the prefix-only target Rk+γkV(st+k)R^k + γ^k V(s_{t+k}), (ii) a fresh-suffix value at st+ks_{t+k}, and (iii) the original-suffix value used by PEQC. Please run this at least on the three families in Table 9 (Cube Triple, AntMaze, HumanoidMaze), with paired tests and more seeds. This is close to the controlled comparison suggested in Appendix C. A clear advantage for the original suffix would raise my score.
  2. Table 15 already has runs for all three collectors on AntMaze, HumanoidMaze and AntSoccer. Using these runs, what is the PE effect, with paired tests, when both learners use greedy collection, and when each learner uses its best collector? With greedy collection for both, the AUC differences are only +1.3 / −0.7 / −3.4. Does the AntMaze gain survive a paired test in either setting? Can you extend this to the cube and puzzle families?
  3. Appendix D says the Cube Triple learning curves retain "the earlier experiment". The Table 6 numbers for Tasks 2 and 4 also match the development-phase runs in Appendix G (e.g., 58.67 vs. 83.47 AUC on Task 2). Can you confirm that the significant Cube Triple AUC result comes from these runs, and report the paired test for the later Cube Triple experiment in Table 9 (76.40 vs. 83.93)?
  4. The PE target evaluates a suffix generated for sts_t at the reached state st+ks_{t+k}. Can you measure the error of the PE target directly, e.g., with Monte-Carlo rollouts of the original suffix from the reached state, and compare it with the error of the executed-prefix target? The Table 11 setup seems to make this feasible.
  5. Both auxiliary backups lower HumanoidMaze final success (58.9 and 59.5 vs. 69.3, Table 9), and the PE backup has a negative mean actor-update effect there (Table 11). Can you diagnose this decline? For example, track value error over training, or check whether a smaller or annealed λ\lambda removes it.
  6. Can you train with uniform-random collection, with and without the PE backup, and add fixed-H collection as a reference without adaptive execution? (Under fixed-H collection the backup has no partial records.) Does the value-based softmax collector do anything beyond matching the uniform marginal length distribution in Table 14?
  7. How do the results change with λ\lambda (including annealing), HH, the softmax temperature β\beta, and the 1/3 scaling of the actor update used in the navigation families (Appendix D)? Why were the H = 10 Puzzle runs excluded, and what were their results?
5 comments

We greatly appreciate the reviewer's detailed suggestions for evaluating the partial-execution (PE) backup.

Revisions are marked in blue. AUC denotes the normalized area under the online success curve.

Q1. Contribution of the original suffix.

Following the reviewer's suggestion, we increased the number of training seeds from three to five and compared the three continuation targets on all five tasks in each family. All targets supervise the same original full-chunk prediction, using the state value at st+ks_{t+k} (Prefix-VV), a fresh suffix, or the original suffix. The variants share the offline checkpoint, auxiliary weight, and training budget for each task and seed.

We reproduce Table 10 below, including AUC and final success rate (Final). Both metrics are percentages, reported as mean ± sample standard deviation across five seeds after averaging the five tasks within each seed.

EnvironmentMetricPrefix-VVFreshOriginal (PE)
Cube TripleAUC76.70 ± 6.8281.01 ± 2.8983.98 ± 2.59
Final94.32 ± 8.2299.36 ± 0.7399.36 ± 0.61
AntMazeAUC49.01 ± 4.7149.86 ± 8.2354.33 ± 3.75
Final71.84 ± 6.3369.60 ± 16.8577.28 ± 7.14
HumanoidMazeAUC53.00 ± 4.7453.60 ± 4.7956.32 ± 3.49
Final57.76 ± 7.7159.36 ± 8.0259.36 ± 6.12

Original achieves the highest mean AUC in all three families and the highest mean final success on AntMaze. Paired tests with Holm correction across the six AUC comparisons support its advantage over Prefix-VV on Cube Triple and HumanoidMaze.

Q2. Collection rules.

We extended the collection study to Cube Triple with greedy collection. Across five tasks and three seeds, PE increases mean online AUC from 79.15 to 84.45 (Table 18), with the improvement supported by a paired test.

Q3. Cube Triple experiments.

The main Cube Triple comparison uses the runs that also appear in the Task 2 and Task 4 analyses. Table 9 uses separate online training runs, with AUC increasing from 76.40 to 83.93. The paired randomization test supports this gain of 7.53 points, with a paired bootstrap 95% confidence interval of [5.25,9.84][5.25, 9.84].

Q4. Direct evaluation of the backup target.

We define the Monte Carlo (MC) reference Y^\widehat{Y} as the sum of the observed prefix rewards and the MC returns obtained by executing the original suffix from the reached state, followed by the full-chunk policy. Using the same models and reached states, the mean squared error between the PE target and Y^\widehat{Y} is lower than that between the state-value target and Y^\widehat{Y} on Cube Triple and HumanoidMaze (Table 13).

Q5. HumanoidMaze performance and additional analysis.

Table 22 tracks success rate and full-chunk value error over six checkpoints. After the success rate peaks at 400K steps, it decreases from 66.53% to 59.47%, while full-chunk mean squared error decreases from 357.84 to 187.79. Thus, the decline is not accompanied by increasing error on these value evaluations. Alternatively, Table 21 examines the effect of the state-value target. With weighted-prefix values, PE raises mean AUC from 75.99 to 78.43, while final success is similar at 83.73% without PE and 83.07% with PE. Monte Carlo evaluation further shows higher mean rollout return and lower suffix prediction error with PE in this setting (Table 23). See Appendix K.2 for the detailed discussion.

Q6. Uniform and fixed-HH collection.

Within the rebuttal period, we focused these additional training comparisons on AntMaze, using all five tasks and three seeds. PEQC achieves 74.53% final success with softmax collection, compared with 69.20% with uniform and 65.87% with fixed-HH collection (Tables 17 and 18). All three use greedy adaptive execution during evaluation. Table 18 also reports uniform collection without PE.

Q7. Sensitivity and Puzzle horizon.

During the rebuttal period, we focused the additional sensitivity experiments on evaluating the effect of PE weights (λ\lambda) and horizons. Table 24 reports the effect of λ\lambda ranging from zero to one. All evaluated positive weights improve the mean AUC on AntMaze when using the weighted-prefix target.

We use H=5H=5 for the main Puzzle comparison following the QC reference. Table 25 now includes both horizons on all five tasks. Both learners perform better at H=10H=10, with similar means. PE improves the H=5H=5 results, bringing them closer to the H=10H=10 results. The configurations use different pretrained models and PE weights. The H=10H=10 PE result on Task 1 uses two seeds due to time constraints. Other Puzzle results use three. We will complete the third seed and update the table in the final version.

Thank you for the detailed response and the additional experiments. The authors resolved my question about the Cube Triple runs. The new comparisons of continuation targets, collection rules, λ\lambda, and the Puzzle horizon also clarify how the PE backup behaves across settings. I will maintain my original score.

We thank the reviewer for confirming that our responses resolved your questions. In the original review, it is noted that a clear advantage for the original suffix would raise the rating. In the suggested controlled comparison (Table 10; five seeds, all five tasks per family), all three variants supervise the same original full-chunk value and differ only in what follows the executed prefix: the state value at the reached state (Prefix-V, your (i)), the value of a newly generated suffix (Fresh, your (ii)), or the value of the unexecuted original suffix (Original, PEQC, your (iii)). Only Original makes the target consistent with the predicted full-chunk value. Consistent with this, mean AUC increases in the order Prefix-V < Fresh < Original in all three families, and Original also has the lowest variance across seeds in every family.

We would appreciate it if the reviewer could let us know which concerns are remaining, so that we can address them in the final version. Once again, we sincerely appreciate your insightful comments on this point.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful assessments and constructive feedback.

The central contribution of our method (PEQC) is to learn the value of an originally generated action chunk from partial execution, combining observed prefix rewards with the estimated value of the original suffix. We would like to highlight the following key point to our revision.

Clarifying the proposed value learning and adaptive execution (Section 3.1, Appendices B.1, G.2 and H)

PEQC separates the horizon over which actions are jointly optimized from the length executed before replanning. The actor learns a complete action chunk, while the prefix critic evaluates how long to execute it.

The exact values in Appendix B.1 describe the learned full-chunk policy πH\pi_H. For this policy, Equation (6) gives

&Q^{\pi_H}(s_t,\boldsymbol{a}^{\pi}_{t:t+H}) -Q^{\pi_H}(s_t,\boldsymbol{a}^{\pi}_{t:t+k}) \\\\ &\quad=\gamma^k\mathbb{E}\\!\left[ Q^{\pi_H}(s_{t+k},\boldsymbol{a}^{\pi}_{t+k:t+H}) -V^{\pi_H}(s_{t+k}) \right]. \end{aligned}$$ The right-hand side compares completing the original suffix with generating a new chunk from the reached state. A positive difference favors completing the original chunk, while a negative difference favors replanning after $k$ actions. Their ordering can depend on the state and generated actions even when the values are exact. Thus, exact evaluation does not make $k=1$ universally preferable. Each prefix head learns its corresponding return, and greedy length selection uses these differences to determine when to replan. This provides an intuitive explanation of how our proposed method can lead to adaptive execution. We also report the execution-length distributions during both data collection and evaluation. Figure 7 shows the distribution of selected execution lengths under greedy evaluation. Each evaluated policy selects among all five candidate lengths. The shortest prefix is selected most frequently, while full chunks account for approximately one-fifth of the decisions. Thus, the learned execution rule uses both short and long prefixes during evaluation. Figure 6 also reports the execution lengths used during data collection, showing similar trends. In addition, we addressed issues on the collection comparisons (Appendix I, Table 18), effect of the original suffix (Table 10), learning progress (Figures 2, 5), and other technical comments (Appendix K) and responded to each reviewer. We hope these additions, together with our detailed responses to the individual reviews, help clarify the contribution and its supporting evidence. Once again, we thank the reviewers and Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.