acceptodds
Under review as a conference paper at ICLR 2027

IAR: Implicit Action Repetition for Policy Optimization with Sequence Critics

Abstract

In actor–critic learning, policy updates depend on both the learned value function and the action contexts in which it is queried. We study action repetition as a mechanism for constructing these contexts during policy optimization. We introduce implicit-action repetition (IAR), an actor-update mechanism for continuous control with sequence-conditioned critics. For each replay state, IAR samples an initial action and constructs prefixes that repeat it for different durations. The policy then samples a continuation action conditioned on each prefix, and the actor is optimized using the resulting sequence values. Repeated prefixes and critic parameters are held fixed during differentiation, allowing the actor to learn under hypothetical sustained-action contexts without requiring those repetitions to be executed in the environment. Once the initial action is sampled, all prefixes are available, allowing a causal Transformer to evaluate the continuation distributions in parallel. This removes sequential feedback among continuation predictions during actor updates, while environment interaction and critic bootstrapping retain autoregressive sampling. Across all 50 manipulation tasks under the Meta-World ML1 protocol, the resulting method achieves an aggregate success rate of approximately , compared with approximately for T-SAC. These results support repeated-action conditioning as an effective mechanism for policy optimization with sequence critics.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Optimization, Implicit Action Repetition, Sequence-Conditioned Critics, Transformer Policies, Continuous Control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7bTLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Implicit Action Repetition (IAR), a policy optimization method that leverages sequence-conditioned critics by evaluating actions under hypothetical repeated-action prefixes. The proposed method enables parallel prediction of continuation actions using a causal Transformer policy, while retaining autoregressive sampling during environment interaction. Experiments on Meta-World ML1 and Box Pushing demonstrate improved performance over existing baselines.

Critical strengths
  1. The paper demonstrates impressive empirical results across all 50 Meta-World ML1 tasks and Box Pushing, outperforming several strong baselines. The broad experimental evaluation is a clear strength of this work.
  2. The proposed method is carefully designed, combining repeated-action conditioning with parallel actor optimization using a causal Transformer. The overall framework is technically sound.
Critical weaknesses
  1. While the motivation for leveraging sequence critics in actor optimization is clear, I would appreciate further clarification of why repeated-action prefixes are particularly beneficial. Although the paper explains their computational advantages, it provides limited insight into why they should yield better policy updates than other conditioning strategies. For example, [1] motivate action repetition by addressing the effective control frequency and [2] show that safely extended actions can mitigate policy-gradient instability. In contrast, it remains unclear what analogous learning benefit is expected from hypothetical repeated-action conditioning in IAR. Overall, the paper places more emphasis on technical implementation than on understanding the underlying mechanism, and a deeper analysis would strengthen the contribution.
  2. The main novelty appears to lie in using detached repeated-action prefixes for actor optimization and enabling parallel evaluation of continuation actions, while other key components, such as sequence-conditioned critics (T-SAC [3]), best-of-N action selection (Q-chunking [4]), and distributional value learning, build on prior work. Although the overall empirical results are strong, the ablation comparing IAR with recursive policy updates shows relatively modest improvements on six tasks (58% vs. 51%), with substantially overlapping uncertainty bands. I believe stronger evidence isolating the benefits of the proposed actor-update mechanism would help establish the significance of the contribution.

[1] Alberto Maria Metelli et al., Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning, in ICML, 2020.

[2] Seohong Park et al., Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods, in NeurIPS, 2021.

[3] Dong Tian et al., Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns, in ICLR, 2026.

[4] Qiyang Li et al., Reinforcement Learning with Action Chunking, in NeurIPS, 2025.

Questions for authors
  1. Could the authors further clarify why repeated-action prefixes are specifically used for actor optimization?
  2. Could the authors provide a more controlled comparison with alternative prefix construction strategies, such as replay-buffer action prefixes, while keeping the actor objective and other components fixed? This would help isolate the benefits of repeated-action conditioning.
  3. I wonder whether there might be environments where repeated-action conditioning could be detrimental to actor optimization. Could the authors discuss what types of environments might pose challenges for the proposed approach?
  4. I noticed that some figures are placed alongside the text. In particular, Figures 2, 4, and 6 seem to somewhat affect readability. I wonder whether presenting these as standalone figures might improve the overall readability of the paper.
Reviewer_s1B8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I am putting my entire review here, as a sort of protest against the review form, which just makes it harder to be clear, in my honest opinion.

Paper 5573: IAR: Implicit action repetition for policy optimization with sequence critics

This paper introduces a new idea---implicit action repetition, or IAR---for actor-critic RL algorithms that learn "sequence values", such as the T-SAC algorithm. The idea has to do with action sequences that are repetitions of an initial action. The paper forms this idea into a full RL method by combining it with several other ideas (Best-of-N selection and distributional critics). Then the final method is compared with T-SAC and other methods on 50 Meta-World ML1 tasks and found to perform the best.

Unfortunately, I found that I could not understand the substance of any part of the paper. I could not understand the main idea of IAR, its predecessor T-SAC, or what the experiments were. Undoubtedly some of this lack of understanding is my fault, but not all of it. Below I identify several areas where the presentation in the paper is lacking and arguably does not make sense and thus cannot be understood. It is also true that there is a lot to explain, and thus it is not easy to explain all of it clearly. But this is to understand the problem with the paper, not to excuse it. The bottom line is that I was able to learn nothing from the paper except the claims. What is the main idea? What is the actual algorithm? What is the problem being solved? Since I cannot answer these questions, what am I to do? Accept or reject? I do have sympathy for the authors; their task is hard, and I could be better informed. But it is not good for a field to have accepted papers that reviewers and other readers cannot understand. They will be built on, and held up as positive examples. I think we are already suffering from this sort of thing having been done too much in the past. There may be a contribution in this paper, but I certainly can't see it, and so I must recommend rejection.


Some detailing of the communication problems with the paper:

The problem may be that the paper presumes a lot as being standard practice, and thus does not say many of the things that are essential to understand the setting and the problem, or possibly it is just being a bit loose with definitions. In the preliminaries section, it says let D denote a replay buffer, with no further explanation. I am uncertain what to make of this. I know what a replay buffer is, it is a buffer of past transitions that is populated as the agent gains experience. Of course, this conventional notion of replay buffer changes over time; it is a D_t, not a D. Am I misunderstanding, or am I correct and the authors are just being informal and leaving off the _t? This problem continues with the definition of the loss (2), which includes D with no _t. Is the loss a function of time? (Also, it seems to be a function of \phi, but this is not shown.) Then there is (3), defining the "sampled TD target" y. I am really reaching to understand what this means. Is there a different y for each t, and again this is being omitted? Or is there a different y for each transition in D? Is y a random variable? It would seem so, as it depends on a', which is random. This might be resolved, and have a purpose, if we were then told how this y was used in the algorithm, but we are not.

There is also a major problem in the next part of presentation, under "Sequence-Conditioned Value Functions". The first sentence in defines an action sequence in a bizarre way, violating proper mathematical conventions. It says something like Let A_t:t+l-1 = (a_t, ..., a_t+l-1) denote an action sequence. The question is, what does this mean? Why are there ts in this definition? If one really meant an arbitrary sequence of l actions, then there should be no t. Equation (4) refers to time-indexed rewards (with respect to A_t:t+l-1), so, no, the authors do not mean an arbitrary sequence. They must be referring to some sequence of events that have an independent definition---like the actual actions, states, and rewards that happened. That is, perhaps A_t:t+l-1 is supposed to be a subsequence of an actual sequence. Is that is what is meant? If so, it would be greatly clarifying if this was stated, and if a_t always meant the tth action in that real sequence. But how then was this actual sequence generated? And how are we to make sense of (4), which has s_t as an input? Is s_t an actual input to an actual function? Is s_t a free or dummy variable, as in an actual function of its inputs, or is specifying s_t as way of informally grounding t as it appears in the rest of the equation? Either way, it is terribly sloppy, multiple violations of mathematical convention, and cannot be clearly understood. This problem persists and makes the next few pages un-understandable. (for example, what does s_t ~ D mean?) And even the algorithm given deep in the appendix does not make anything clear.

Critical strengths

I don't find this part of the review form helpful.

Critical weaknesses

see review above.

Questions for authors

see review above.

Reviewer_Gt7BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose a policy optimization method, namely implicit-action repetition (IAR), based on a Sequence Critic. IAR repeats the initial action to construct hypothetical action prefixes of different lengths when updating actors. New subsequent actions are sampled and optimized using value estimates from the Sequence Critic for each prefix. Transformer can compute Actor updates across multiple temporal horizons in parallel, without actually executing these repeated actions in the environment. The empirical results show the effectiveness of the proposed approach on Meta-World tasks and Box Pushing.

Critical strengths

IAR moves action repetition from environment exploration to the Actor optimization stage, using a Sequence Critic to provide learning signals across different temporal horizons. IAR avoids the recursive dependency of generating conditional actions sequentially during Actor updates, enabling parallel computation with a Transformer while preserving autoregressive environment interaction and Critic bootstrapping, which is novel.

The experiment demonstrates strong overall performance across all 50 Meta-World tasks and substantial improvements on Box Pushing. Authors also conduct several ablations examining IAR, Best-of-N, update horizon, and Critic design.

Critical weaknesses

The paper draws motivation from prior work on action repetition in reinforcement learning. However, IAR does not actually execute these repeated actions in the environment; instead, it uses them as hypothetical contexts for Actor updates. Therefore, the conventional motivation for action repetition does not directly explain why repeating the initial action should provide more effective learning signals for policy optimization.

Fixed repeated prefixes enable parallel computation, but other predefined action prefixes could potentially offer the same advantage. The paper lacks comparisons with alternatives such as prefixes from real trajectories or other fixed-action prefixes, leaving the distinctive value of repeating the initial action insufficiently established.

The Sequence Critic is trained on action sequences collected from real replay trajectories, whereas IAR constructs hypothetical prefixes by repeatedly copying the same initial action during Actor updates. For continuous-action policies, such exactly repeated action sequences may be uncommon in the replay distribution, and the distribution shift could become more pronounced as the repetition length increases.

Consequently, it is unclear whether the Critic's value estimates and the resulting Actor optimization gradients remain reliable under these hypothetical prefixes. The paper does not sufficiently analyze the relationship between repetition length, distribution shift, and Critic estimation accuracy.

The paper already compares IAR with Recursive Policy Update. In Figure 8(a), IAR achieves an aggregate final success rate of approximately 58%, compared with approximately 51% for the recursive baseline. However, their uncertainty intervals overlap substantially, and the relative performance varies across tasks.

In contrast, Figure 8(c) shows that removing Best-of-N reduces the aggregate success rate from approximately 58% to 32%, indicating that other components also make substantial contributions. Furthermore, the main ablations cover only six Meta-World tasks, leaving it unclear whether the benefits of IAR itself consistently generalize across all 50 tasks.

Questions for authors
  1. Why is repeating the initial action chosen as the context for Actor optimization? Could the authors compare it with real replay-trajectory prefixes, random fixed prefixes, or perturbed action prefixes to demonstrate the distinctive advantages of this design?

  2. How different are the hypothetical repeated-action prefixes from the real replay-data distribution? As the repetition length increases, do the Critic's value estimates and the resulting Actor optimization gradients remain reliable?

  3. Could the authors compare IAR with Recursive Policy Update on a larger set of Meta-World tasks and further investigate why their relative performance varies across tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.