acceptodds
Desk rejected at ICLR 2027

ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using standard off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement Learning, Action Chunking, Online Policy Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ohm9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ExRL, an adaptive action-chunk length selection mechanism for RL. Their idea is to simply treat the choice of different action-chunk lengths as a discrete action, and let an RL algorithm learn to find the optimal action-chunk length. Specifically, they train a chunk length-conditioned critic and take the argmax (or softmax) to decide the length to execute at test time. Combined with appropriate data-driven RL techniques (CQL, BC constraining, etc.), they show that the resulting algorithm (ExRL) works in both sim and real environments across diverse settings.

Critical strengths
  • The paper is generally well-written. I liked the "Remark" section, which helps clarify the motivation behind the method.
  • Empirical results are quite thorough. I especially appreciate the real-world results, which show that their method can be applied outside simulated environments as well. They also compare against quite a large number of previous baselines across different categories.
Critical weaknesses
  • One major weakness is novelty. As the authors also mention in the related work section, there are quite a large number of previous methods in the literature that share the same fundamental idea (i.e., treating the action-chunk length as an action), including ACSAC, ACH, Zhao et al. (2026), etc. While the details of these methods are not exactly the same, I don't think this method is significantly different from the other papers to justify a separate ICLR publication. For example, the authors argue that ExRL is different from Zhao et al. (2026) in that "it [Zhao et al. (2026)] relies on PPO for optimization", but as the authors themselves mention in the paper ("ExRL is agnostic to the choice of RL algorithm"), the choice of a base algorithm doesn't seem to be essential to the algorithm (i.e., the idea in Zhao et al. (2026) can directly be applied to any other algorithm, like Q-learning or SAC).
  • Another concern is in the objective itself. ExRL (and other similar works) picks the action length based on the learned Q function. In theory, the optimal action chunk length is 1 because it's always better to be maximally adaptive (if we ignore practical issues like inference delays). I'm not sure what prevents ExRL from collapsing to this degenerate solution at the end. Of course, it might just happen not to collapse in practice (which I assume is the case from the qualitative results in the paper), which is fine, but I was wondering if there's a more principled reason to believe why we should choose action chunk lengths in this way.
Questions for authors

Please see the weakness section.

Reviewer_yLDGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ExRL adapts pretrained action-chunked robot policies by learning how many actions to execute before replanning. It uses duration-aware Q-learning or discrete SAC, with optional online behavior cloning and demonstration augmentation. Experiments across several policy architectures and two real-world tasks show improved sample efficiency and task performance.

Critical strengths

First, execution-length control provides a simple, policy-agnostic adaptation interface. Improvements across different architectures and real-world tasks demonstrate its practical value without requiring changes to the base action generator.

In addition, the ablations provide useful evidence for the mechanism. In particular,conditioning on predicted chunks and the shuffled-length control support the importance of matching execution lengths to specific states and predictions.

Critical weaknesses
  1. The contribution beyond DEHP needs stronger validation. DEHP already learns execution lengths conditioned on observations and predicted chunks while freezing the base policy. ExRL’s off-policy formulation and extensions are potentially valuable, but Appendix B compares against a PPO adapter on only one task, with insufficient detail to establish fidelity to DEHP.

  2. The baseline comparisons do not fully establish the claimed sample-efficiency advantage. Several baselines are modified to optimize complete long chunks, while comparisons using their preferred shorter execution lengths or original per-step formulations are limited. The optimal fixed-k baseline evaluates the frozen policy and does not resolve this concern. Frequent replanning also introduces model-query costs that are not quantified.

Questions for authors
  1. Does the PPO baseline reproduce DEHP’s duration-aware advantage estimation and critic design? What tuning was performed, and is the advantage consistent beyond Can?

  2. Do the gains persist when DSRL and residual RL baselines use suitably tuned execution lengths or retain their original per-step formulations?

  3. How do base-policy query counts and wall-clock training times compare at matched performance levels?

  4. In Table 3, it specifies TD-target clipping to [-1000, 0], whereas parts organization uses positive stage rewards. Was this clipping configuration changed for that task?

Reviewer_4Rh6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ExRL, which adapts pretrained robot policies online by learning how many actions from each predicted chunk to execute before replanning. The core version freezes the base policy and learns execution lengths using duration-aware Q-learning or discrete SAC, while reusing online and offline experience. Online behavior cloning is an optional extension. The method is evaluated across multiple base policies, simulation tasks, and two real-world robot tasks.

Critical strengths
  1. A practically relevant problem. Reducing online adaptation of long action sequences to a discrete execution-length decision provides a lightweight adaptation mechanism that does not require modifying the base policy.

  2. Compelling results in several settings. The experiments cover DP, ACT, and several generalist policies, with substantial gains on some tasks. In the real-world insertion experiment, ExRL succeeded in 22 of 25 evaluation trials, compared with 10 of 25 for the best tested fixed-length baseline.

  3. Informative controlled analyses. Figure 7 examines the roles of duration-aware backups and action-chunk inputs. In Appendix D.2, shuffling the assignment of execution lengths to state-chunk pairs while preserving the same multiset of lengths substantially reduces performance, supporting the value of conditional assignment itself.

Critical weaknesses
  1. Limited methodological novelty. DEHP [1], acknowledged by the authors as concurrent work, also freezes the base policy and learns execution lengths through RL. ACSAC [2] and ACH [3] use value-based adaptive length selection; the VLA experiments in AQC [4] also include a fixed policy with a critic trained to select prefixes. Closely related approaches therefore already exist for both execution-length control and value-based prefix selection. The methodological increment lies primarily in off-policy adaptation and experience reuse for pretrained long-chunk policies, with limited additional algorithmic insight.

  2. Missing direct comparisons with existing value-based methods for adaptive execution length. The experiments do not directly compare against value-based adaptive length methods such as ACSAC, ACH, and AQC. The PPO comparison in Appendix B does not establish what the proposed implementation offers over existing value-based alternatives in terms of sample efficiency, task success, or applicability. Given the modest methodological increment, this comparison gap leaves the added value of the proposed implementation insufficiently supported.

  3. Limited cross-task validation for generalist policies. Although the standard simulation experiments cover multiple tasks, all four generalist policies are tested on the same single pick-and-place task in LIBERO. The multi-model results support the adaptability of the interface, but evidence of consistent gains across different task structures remains limited. I regard this as a secondary concern about empirical coverage.

Questions for authors

Could the authors compare ExRL with one representative value-based approach to adaptive execution length, using the same frozen base policy, observations, candidate lengths, available data, and online interaction budget, to establish its specific advantages in sample efficiency, task success, or applicability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.