- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose an alternative to sparse-reward RL post-training of VLAs, using inverse RL to recover a dense reward from expert demonstrations. The authors claim that reinforcement learning often suffers from initial performance drops and the unlearning of already learned behaviors. They propose optimizing an averaged action, in which the actions predicted by the VLA are averaged with the action predictions of a small corrective policy. In addition the corrective policy is initialized from a BC checkpoint, thereby mitigating initial performance drops. The CSIL recipe is followed, in which the log-policy ratio is used as the reward. The authors incorporate recent improvements from the literature, including normalization and regularization techniques, into CSIL and call the improved version CSIL++. Experiments show that CSIL++ works better than CSIL on an image-based task. In addition, CSIL works better than the selected baselines for VLA fine-tuning across six different tasks.
Critical strengths
- The idea of using CSIL for VLA post-training is clear and well explained.
- Overall, the experiments cover a proper number of tasks and are extensive. The results look promising and show that CSIL++ works well for fine-tuning VLAs.
- The experiments on demonstration efficiency and the comparison with BC using additional data are really nice and show how RL performs better.
- The paper clearly presents how averaged and residual actions differ in terms of their reward, which is quite interesting.
Critical weaknesses
- “The authors claim that reinforcement learning often suffers from initial performance drops and the unlearning of already learned behaviors.” How do you know that the performance drop is caused by the unlearning of initial behaviors? In DSRL-style RL, only a small policy is trained while the VLA is kept frozen, so there is no unlearning. Moreover, such a performance drop can be mitigated by a gated start, warm-up steps, or bounded residuals. Please clearly discuss and demonstrate the reason for such a performance drop, analyzing the critic as well, rather than only how the policy differs from the BC starting point.
- CSIL++ seems to be an updated version of CSIL with better normalization and regularization. It is clear that employing such recent findings from the literature will improve prior methods. Therefore, the novelty of this component is questionable, and it is hard to accept it as a key contribution of the paper.
- The improvement over CSIL appears limited, since CSIL++ is compared with CSIL on only two tasks, and on one of them both methods perform equally well. It would be great if this comparison could be extended to two more tasks.
- In Table 1, the lowest upper quartile is bolded, which is not common practice. Please bold the average values instead. In addition, ResFit performs better than CSIL++ on Coffee and Mug Cleanup and equally well on Hammer Cleanup. Please bold these numbers properly, otherwise, the table is misleading.
- The strong performance of ResFit in Table 1 is not properly discussed. Please clearly show how ResFit’s performance is unstable and compare it with that of CSIL++.
- Please compare the method with DSRL with a gated start. In addition, a comparison with filtered SFT would be nice, since success/failure detection is already available.
- CSIL++ uses its own encoder, but this choice is not ablated. Although it is not critical, an ablation would be valuable to see what happens when VLA features are used instead.
Questions for authors
Questions are listed in Weaknesses part, but here are them again:
- Please clearly discuss and demonstrate the reason for a performance drop in RL post-training, analyzing the critic as well, rather than only how the policy differs from the BC starting point.
- Please extend the comparison between CSIL++ and CSIL and include at least two more tasks.
- Please bold the numbers correctly on Table 1, it is misleading in the current shape.
- The strong performance of ResFit in Table 1 should be properly discussed, and clearly show how ResFit’s performance is unstable and compare it with that of CSIL++.
- Please compare the method with DSRL with a gated start. In addition, a comparison with filtered SFT would be nice, since success/failure detection is already available.
- Can you ablate the choice of using a separate encoder for CSIL++?
2 comments
Response to Rebuttal
Thanks to the authors for the response and additional experiments. The extended CSIL vs. CSIL++ comparison, the corrected bolding, the ResFit discussion, and the DSRL comparison are appreciated. However, they also narrow the empirical claim. With correct bolding, ResFit has the lowest regret on Coffee and Mug Cleanup and is within 0.01 on Hammer Cleanup. DSRL matches CSIL++ in final success rate. The advantage of CSIL++ is therefore mostly on sample efficiency and avoiding the initial drop, not on the final performance and text should reflect this. Therefore this makes my comment on performance drop (W1) more important and it remains untested. By gated start, I mean keeping the learned policy’s contribution off, or annealed in, until the critic has been trained on base-policy data. If DSRL or ResFit with such a start or XQC+OD with KL term no longer drops initially, then the main claim of the paper is considerably weakened. For filtered SFT, your reading was correct and it is a common approach for VLA post-training, would be great to see a comparison with it. Regarding the encoder (W7), I accept the cost argument, but “better and faster” is unsupported without an ablation, so please soften it. Given these, although my main concerns are far from being addressed, I will keep my score.
Rebuttal
We thank the reviewer for their extensive review!
Since the reviewer has not clarified their comments in the past seven days, we provide a partial rebuttal here for the aspects we can address at this stage.
We recommend the reviewer read the original CSIL paper by Watson et al. 2023 to appreciate this phenomenon. If the BC performance is strong but performance drops after a few RL updates, the actor’s objective must cause the policy to unlearn the initial strong behaviour. This is due to function approximation errors in the critic. CSIL fixes this with a dense coherent reward, critic pretraining and KL regularization against the BC policy in the actor loss. We would appreciate it if the reviewer could clarify the source of their confusion with this phenomena.
The DSRL paper’s results show initial performance drops across environments so we would disagree that the method does not unlearn. Our experimental results using their code reproduce this unlearning as well.
The development of CSIL++ is crucial to this project, after observing that CSIL does not scale to harder tasks such as Square and beyond. The original CSIL paper only benchmarks on image-based Lift and Can in robomimic. Pi-0.5 already solves Lift and Can with BC, so for IRL finetuning to be useful and meaningful we need a method that scales to harder tasks that pi-0.5 struggles to solve with BC. We believe the capability that CSIL++ unlocks makes it valuable as a central contribution, as the paper would not be publishable without it due to poor experimental results.
We have new results showing the better performance of CSIL++ compared to CSIL across all six mimicgen tasks we consider.
Fixed. However we would argue this is a perfectly valid method selection criteria, as methods with high performance variance can have competitive medians even if several seeds fail to solve the task, such as ResFit, so using the upper quartile captures this lack of consistency.
We have added this to the appendix in Section C.3.
The pi-0.5 encoder is much larger than our CNN. Averaging over 1000 inference calls, we measure on our hardware
Steps/s: Resnet encoder: ~1800 VLA embeddings: ~45
FLOPs per step: Resnet encoder: 0.365 GFLOP VLA embeddings: 2560 GFLOP
Therefore using the pi-0.5 encoder would slow our experiments down by an impractical degree, and this is with freezing the representations for the RL policy.