- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ExRL, an adaptive action-chunk length selection mechanism for RL. Their idea is to simply treat the choice of different action-chunk lengths as a discrete action, and let an RL algorithm learn to find the optimal action-chunk length. Specifically, they train a chunk length-conditioned critic and take the argmax (or softmax) to decide the length to execute at test time. Combined with appropriate data-driven RL techniques (CQL, BC constraining, etc.), they show that the resulting algorithm (ExRL) works in both sim and real environments across diverse settings.
Critical strengths
- The paper is generally well-written. I liked the "Remark" section, which helps clarify the motivation behind the method.
- Empirical results are quite thorough. I especially appreciate the real-world results, which show that their method can be applied outside simulated environments as well. They also compare against quite a large number of previous baselines across different categories.
Critical weaknesses
- One major weakness is novelty. As the authors also mention in the related work section, there are quite a large number of previous methods in the literature that share the same fundamental idea (i.e., treating the action-chunk length as an action), including ACSAC, ACH, Zhao et al. (2026), etc. While the details of these methods are not exactly the same, I don't think this method is significantly different from the other papers to justify a separate ICLR publication. For example, the authors argue that ExRL is different from Zhao et al. (2026) in that "it [Zhao et al. (2026)] relies on PPO for optimization", but as the authors themselves mention in the paper ("ExRL is agnostic to the choice of RL algorithm"), the choice of a base algorithm doesn't seem to be essential to the algorithm (i.e., the idea in Zhao et al. (2026) can directly be applied to any other algorithm, like Q-learning or SAC).
- Another concern is in the objective itself. ExRL (and other similar works) picks the action length based on the learned Q function. In theory, the optimal action chunk length is 1 because it's always better to be maximally adaptive (if we ignore practical issues like inference delays). I'm not sure what prevents ExRL from collapsing to this degenerate solution at the end. Of course, it might just happen not to collapse in practice (which I assume is the case from the qualitative results in the paper), which is fine, but I was wondering if there's a more principled reason to believe why we should choose action chunk lengths in this way.
Questions for authors
Please see the weakness section.