- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Implicit Action Repetition (IAR), a policy optimization method that leverages sequence-conditioned critics by evaluating actions under hypothetical repeated-action prefixes. The proposed method enables parallel prediction of continuation actions using a causal Transformer policy, while retaining autoregressive sampling during environment interaction. Experiments on Meta-World ML1 and Box Pushing demonstrate improved performance over existing baselines.
Critical strengths
- The paper demonstrates impressive empirical results across all 50 Meta-World ML1 tasks and Box Pushing, outperforming several strong baselines. The broad experimental evaluation is a clear strength of this work.
- The proposed method is carefully designed, combining repeated-action conditioning with parallel actor optimization using a causal Transformer. The overall framework is technically sound.
Critical weaknesses
- While the motivation for leveraging sequence critics in actor optimization is clear, I would appreciate further clarification of why repeated-action prefixes are particularly beneficial. Although the paper explains their computational advantages, it provides limited insight into why they should yield better policy updates than other conditioning strategies. For example, [1] motivate action repetition by addressing the effective control frequency and [2] show that safely extended actions can mitigate policy-gradient instability. In contrast, it remains unclear what analogous learning benefit is expected from hypothetical repeated-action conditioning in IAR. Overall, the paper places more emphasis on technical implementation than on understanding the underlying mechanism, and a deeper analysis would strengthen the contribution.
- The main novelty appears to lie in using detached repeated-action prefixes for actor optimization and enabling parallel evaluation of continuation actions, while other key components, such as sequence-conditioned critics (T-SAC [3]), best-of-N action selection (Q-chunking [4]), and distributional value learning, build on prior work. Although the overall empirical results are strong, the ablation comparing IAR with recursive policy updates shows relatively modest improvements on six tasks (58% vs. 51%), with substantially overlapping uncertainty bands. I believe stronger evidence isolating the benefits of the proposed actor-update mechanism would help establish the significance of the contribution.
[1] Alberto Maria Metelli et al., Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning, in ICML, 2020.
[2] Seohong Park et al., Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods, in NeurIPS, 2021.
[3] Dong Tian et al., Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns, in ICLR, 2026.
[4] Qiyang Li et al., Reinforcement Learning with Action Chunking, in NeurIPS, 2025.
Questions for authors
- Could the authors further clarify why repeated-action prefixes are specifically used for actor optimization?
- Could the authors provide a more controlled comparison with alternative prefix construction strategies, such as replay-buffer action prefixes, while keeping the actor objective and other components fixed? This would help isolate the benefits of repeated-action conditioning.
- I wonder whether there might be environments where repeated-action conditioning could be detrimental to actor optimization. Could the authors discuss what types of environments might pose challenges for the proposed approach?
- I noticed that some figures are placed alongside the text. In particular, Figures 2, 4, and 6 seem to somewhat affect readability. I wonder whether presenting these as standalone figures might improve the overall readability of the paper.