- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Partial-Execution Q-Chunking (PEQC), which learns a Q-function over prefixes of action chunks and adaptively selects the chunk size based on Q-values at test time. The Q-function is trained using Bellman backups with full action chunks, and the policy is extracted using QC-FQL. The method is evaluated on OGBench and outperforms prior methods on 2 out of 6 tasks.
Critical strengths
S1. The method is simple and easy to understand.
- The method extends existing action-chunking approaches by learning a Q function over action-chunk prefixes and extracting the policy using QC-FQL (+ PE backups during online phase)
- The execution length is adaptively selected using a Boltzmann policy over the Q values, a simple mechanism for dynamically adjusting the action-chunk size.
Critical weaknesses
W1. The experimental results are not sufficiently strong.
- While PEQC outperforms prior methods on AntSoccer and cube-triple, its performance on the remaining tasks is generally comparable to or worse than existing methods, such as FQL, QC-FQL, and QC (I classified ACSAC / ACH as concurrent works, so I'm not mentioning those methods in this comparison).
- Although limited empirical improvements are not necessarily a reason for rejection on their own, I think the technical novelty (using prefix-based Q-values for adaptive action-chunk execution) of the proposed method is not enough to compensate the weak experimental results
- Given the limited empirical improvements, I am not convinced that the paper shows a sufficient advancement over prior work.
W2. The proposed value learning objective lacks a clear mathematical interpretation.
- The value function is estimated by averaging Q-values over full action chunks. However, the Q-function is trained to evaluate partially executed action chunks through a different backup procedure, so I'm not sure the resulting Q function will be a Q function of a interepretable policy.
- That is, the proposed objective appears to be a heuristic design choice rather than a Bellman-consistent objective, and the paper does not provide a sufficiently clear mathematical motivation for it.
- The ablation studies provide some empirical support for this design choice, but given the limited improvements discussed in W1, I do not find the current justification sufficiently convincing.
Questions for authors
Q1. Could the authors clarify the mathematical interpretation of the value learning objective?
- What policy does the learned Q function correspond to? e.g., can the proposed Bellman backup be derived as the policy-evaluation operator?
- If the objective does not correspond to either the full-chunk policy or the adaptive execution policy, what motivates this particular combination of value estimation and Bellman backups?
Q2. Could the authors provide offline-to-online learning curves rather than only reporting final performance?
- Since PEQC is designed for offline-to-online reinforcement learning, it would be helpful to report performance throughout online fine-tuning, as done in prior work such as QC.
- In particular, these results would clarify whether the proposed method improves initial offline performance, online sample efficiency, or primarily final performance.
Q3. Could the authors provide evidence that adaptive action-chunk execution is actually being utilized?
- The paper motivates PEQC through the benefits of adaptively selecting action-chunk execution lengths. However, it is unclear whether the learned execution policy actually selects different chunk lengths across states or simply converges to a Markovian policy or a fixed full-chunk policy.
- It would be helpful to visualize the distribution of selected execution lengths, potentially across different tasks and stages of training.
3 comments
Thank you for your response.
Empirical contribution.
Thank you for running the experiments. However, I think using weighted prefixes should not be considered as "tunable hyperparameter", as it is one of the core component (value target). Specifically, I think it is an unfair comparison towards other baseline, as this comparison is comparing "maximum of two algorithm" compare to a single algorithm.
Interpretation of the value objective.
Thank you for giving the technical insight. The main problem I had in mind (and the authors restates it in the rebuttal) was "With exact prefix values, greedy length selection has expected discounted return no lower than full-chunk execution by the same actor, since remains an available choice". This literally means that we do not have to do adaptive execution, we can just train a markovian policy out of the objective.
Adaptive execution
Thank you for running the experiment. I think this works as an empirical evidence on adaptive execution, but still misses an explanation on why the proposed objective encourages (or leads to) adaptive execution, and why it is better than learning a markovian policy.
Therefore, I maintain my score.
Thank you for clarifying the remaining concerns.
Comparison of value targets.
We agree that the two value targets should be treated as distinct objective variants rather than as tunable hyperparameters. To clarify, our main benchmark comparison in Table 1 consistently uses the full-chunk target across all environments. The weighted-prefix results in Table 21 are presented as an ablation of the value objective, and we interpret them separately from the main benchmark comparison. Thus, the current main comparison does not select the better-performing variant for each task. However, we acknowledge that our method does not achieve the best performance on all six environments in Table 1, although neither does any other baseline. As discussed in the limitations and future work section, further extending the evaluation to visual observations and multi-task learning would help assess the applicability of partial-execution supervision beyond the current experimental setting.
Value learning and adaptive execution.
PEQC separates the horizon over which actions are jointly optimized from the length executed before replanning. The actor learns a complete action chunk, while the prefix critic evaluates how long to execute it.
The exact values in Appendix B.1 describe the learned full-chunk policy . For this policy, Equation (6) gives
&Q^{\pi_H}(s_t,\boldsymbol{a}^{\pi}_{t:t+H}) -Q^{\pi_H}(s_t,\boldsymbol{a}^{\pi}_{t:t+k}) \\\\ &\quad=\gamma^k\mathbb{E}\\!\left[ Q^{\pi_H}(s_{t+k},\boldsymbol{a}^{\pi}_{t+k:t+H}) -V^{\pi_H}(s_{t+k}) \right]. \end{aligned}$$ The right-hand side compares completing the original suffix with generating a new chunk from the reached state. A positive difference favors completing the original chunk, while a negative difference favors replanning after $k$ actions. Their ordering can depend on the state and generated actions even when the values are exact. Thus, exact evaluation does not make $k=1$ universally preferable. Each prefix head learns its corresponding return, and greedy length selection uses these differences to determine when to replan. Training a Markovian actor from the single-action head would replace the full-chunk policy-improvement objective. Our motivation for retaining chunk optimization is to jointly improve action sequences under the learned behavior prior, using the temporal structure of offline data and multi-step value learning of Q-chunking [1]. Prefix values provide flexibility in executing these sequences. The PE backup supports this combination by providing supervision for the original full-chunk value even when execution ends after a shorter prefix. [1] Li et al., Reinforcement Learning with Action Chunking, 2025. Once again, we are grateful for your insightful feedback throughout the discussion, which has significantly helped us improve our paper.
We would like to express our sincere gratitude for the detailed questions about the value objective and empirical contribution. We respond to each point below. Revisions are marked in blue.
W1. Empirical contribution.
We extended the state-value comparison to five tasks and three seeds per family (Table 21). With the weighted-prefix state-value target, PEQC reaches 79.20% final success on AntMaze and 83.07% on HumanoidMaze. Table 1 reports FQL means of 74% and 56% and QC means of 47% and 32%, respectively, under their published protocols. See Appendix K for the detailed discussion.
W2, Q1. Interpretation of the value objective.
Section 3.1 and Appendix B.1 clarify the policy evaluated by the default full-chunk objective. Let πH generate a chunk using the one-step policy μω, execute all H actions, and repeat from the resulting state. With the actor fixed, its exact sequence values satisfy
Q^{\pi_H}(s_t,\boldsymbol{a}_{t:t+k}) &=\mathbb{E}\\!\left[ R_t^k+\gamma^k V^{\pi_H}(s_{t+k}) \mid s_t,\boldsymbol{a}_{t:t+k} \right],\\\\ V^{\pi_H}(s_t) &=\mathbb{E}_z Q^{\pi_H}(s_t,\mu_\omega(s_t,z)), \end{aligned}$$ where $R_t^k$ is the discounted reward accumulated over the first $k$ steps. Each prefix value evaluates its prescribed actions followed by the same full-chunk policy. Appendix B.1 derives the corresponding policy-evaluation operator. The actor increases predicted full-chunk value while remaining close to the behavior flow policy, giving an approximate, regularized greedy update in action-chunk space. State-value regression supplies the updated continuation estimate to the shorter-prefix Bellman targets. For a prescribed chunk and execution length $k$, the partial-execution (PE) backup uses the return decomposition $$Q^{\pi_H}(s_t,\boldsymbol{a}^{\pi}_{t:t+H}) = \mathbb{E}\\!\left[ R_t^k+\gamma^k Q^{\pi_H}(s_{t+k},\boldsymbol{a}^{\pi}_{t+k:t+H}) \right].$$ The suffix value evaluates the remaining $H-k$ original actions before continuing with $\pi_H$. Thus, the full-chunk backup and the PE backup evaluate the same original chunk under this reference policy. Adaptive execution uses these reference values to choose when to replan. With exact prefix values, greedy length selection has expected discounted return no lower than full-chunk execution by the same actor, since $H$ remains an available choice (Appendix B.1). **Q2. Offline-to-online learning curves.** Following the reviewer's suggestion, Figures 2 and 5 show one million offline updates followed by one million online environment steps. Both variants start from the same pretrained model, and PE is introduced only during the online phase. Curves average three seeds per task, with Figure 2 also averaging the five tasks in each family. PE achieves a higher mean normalized area under the online success curve (AUC) on Cube Triple, AntMaze, AntSoccer, and Puzzle. **Q3. Adaptive execution.** Figure 7 shows the selected execution-length distributions under greedy evaluation. On AntMaze, HumanoidMaze, and AntSoccer, each evaluated policy selects all five candidate lengths. The shortest prefix is most frequent, while full chunks account for about one fifth of decisions. Figure 6 reports the lengths used during data collection.