- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents RAM-A, which is a diffusion-based model's RL technic which bounds the grad in the RL stage, to keep the model's pre-training abilities and multi-modality, while staying competitive among SOTA RL methods.
Critical strengths
- The advantage of RAM-A on multi-modality and pre-training ability keeping is clearly shown and well presented.
- Ablation study shows how the RAM-A helps to prevent major shifts, and therefore keeping the multi-modality and pre-training skills.
Critical weaknesses
- The motivation is questionable. To my understanding, post-training is designed for turning the pre-trained model into a specific application scope. It is unclear the motivation for keeping pre-training skills on other tasks in this stage, other than doing a weighted pre-training, or a weighted SFT.
- This paper is majorly talking about RL technics. In this sense, I think real robot experiments, though already clearly stated in the limitations, might still be important.
- The proposed method's novelty is questionable to me. I would be interested in its advantages over grad clipping.
Questions for authors
Please see the weaknesses.
2 comments
We apologize for posting our response so shortly before the rebuttal deadline. We just realized that our response was not submitted correctly. We regret that this leaves little time for further discussion.
We thank the reviewer for the review and for recognizing the retention and multimodality results.
Regarding the questions:
1. Why preserve pre-trained capabilities during post-training?
Robots operate in dynamic environments, where learning through interaction is valuable. We believe this learning should also help build broader capabilities over time and not just specialize it for one task. A robot learning better insertion may still need to pick and stack. We study whether RL can improve the rewarded skill while preserving existing abilities and alternative successful behaviors. Weighted SFT could also help, but requires revisiting demonstrations. Our setting uses only interactions and rewards from the current task.
2. Real robot experiments
We see our primary contribution as algorithmic. RAM-A introduces a bounded advantage-regression update for on-policy optimization of flow policies. The central properties that we investigate are: optimization stability, task performance, and retention during fine-tuning. Simulation allows us to study these properties through reproducible comparisons and targeted ablations. We do not consider hardware validation necessary to show these algorithmic claims.
3. Advantages over gradient clipping
Gradient clipping limits the size of each update, but repeated updates can still move the policy far from its reference. RAM-A instead bounds the regression target around a reference prediction. This limits the influence of extreme advantage estimates before gradients are combined. Furthermore, these mechanisms are complementary and can be used together. In fact, we do have gradient clipping applied in all our experiments. But gradient clipping alone will not stabilize the algorithm as the new ablation results in the revised experiments section shows.
I keep my score since I don't see author's rebuttals.