- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes ACSAC, an offline-to-online RL method which adaptively decides a chunk size with a Transformer-based critic that predicts Q-values for prefixes of action chunks. The policy is extracted via BC rejection sampling. ACSAC consistently outperforms prior offline-to-online RL methods in OGBench.
Critical strengths
S1. Method is straightforward and simple.
- The method trains a Transformer critic that learns a Q function for the prefixs of the action chunk (similar to [1])
- Then it extracts the policy with rejection sampling from the BC policy.
S2. Strong theoretical and experimental results.
- The thoeretical results shows that the proposed bellman operator is a -contraction in a deterministic MDP (Section E), demonstrating theoretical motivation on what this algorithm should converge, and how it is anticipated to work.
- ACSAC consistently outperforms prior offline and offline-to-online methods.
- Also, qualitative results (Section 5.3) and ablation studies (Section 5.4) support that adaptive action chunking is important and ACSAC's objective is encouraging adaptive action chunking.
[1] Ge Li et al., TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning, ICLR 2025
Critical weaknesses
None
Questions for authors
Q1. Why markovian policy is not the optimal policy?
- I think that it is due to the BC constraints from the markovian policy, but it feels counterintuitive to me that even after online adaptation, the non-markovian policy can be better (as non-markovian policy is always suboptimal due to reactivity)
- Would it be possible to provide explanation or intuition on why non-markovian policy can be better although the MDP is deterministic and fully-observed?
- This will help the readers who felt the same "conunterintuitive feeling" from reading the paper.
2 comments
Thank you for the clarification, I'll maintain the positive score.
Response to Reviewer aG3W
We sincerely thank the reviewer for the positive assessment and the insightful question.
[Q1] Markovian optimality and BC constraints after online adaptation
A: A deterministic and fully-observed MDP has an optimal Markovian policy. As the reviewer suggests, the BC constraints matter here. A Markovian policy fitted to individual actions can miss a pattern in the demonstrations that a policy fitted to action chunks can learn. This advantage can remain even after online adaptation because training continues with the same behavior constraint as online transitions are added to the dataset. Moreover, ACSAC retains reactivity, executing a longer prefix when commitment helps and replanning sooner when reactivity matters.
BC constraints. Even in a fully-observed MDP, a demonstrator’s next action can depend on preceding actions. The OGBench play datasets used for manipulation were collected by open-loop, non-Markovian expert policies[1], producing action sequences that “cannot be well captured by a Markovian behavior constraint”[2]. Fitting full-length action chunks with a flow BC policy lets ACSAC capture these non-Markovian behavior patterns.
Intuition. In the 1D line-world of Lazzati et al.[3], reaching either endpoint succeeds. While an optimal Markovian policy moves toward the nearer endpoint, the demonstrator picks left or right at the start of an episode and keeps moving in that direction. At the same position, successful demonstrations therefore take opposite actions. Sampling from that per-state action distribution at every step can produce a random walk, whereas an action-chunking policy keeps one direction within each chunk and reaches an endpoint sooner on average[3].
Even after online adaptation. Single-step, multi-step, and action-chunking methods all continue training as online transitions are added to the dataset under the same behavior constraints used offline. Because the combined dataset still contains the multi-modal offline demonstrations, a single-step behavior policy continues to fit opposite actions at the same state, whereas ACSAC’s flow BC policy proposes coherent action chunks while the critic selects prefixes across N candidates. Executing coherent action sequences also helps fixed-chunk policies and ACSAC reach sparse rewards that single-step sampling can miss. Finally, a multi-step TD target alone leaves single-step actor execution unchanged, whereas a chunked critic conditions on the executed action sequence to back up multi-step rewards without the off-policy bias of n-step returns[2]; on OGBench, all Q-chunking methods outperform their 1-step TD and n-step counterparts[2].
We hope this explanation helps readers who share the same counterintuitive feeling, and we are happy to provide any further clarification.
References
[1] Seohong Park et al., OGBench: Benchmarking Offline Goal-Conditioned RL, ICLR 2025.
[2] Qiyang Li et al., Reinforcement Learning with Action Chunking, NeurIPS 2025.
[3] Filippo Lazzati et al., Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?, arXiv 2026.