- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how supervised pretraining affects robot policies before and after finetuning. Using repeated rollouts from the same initial condition, it finds that longer training can improve pass@1 while reducing pass@k for larger k and hurting later finetuning. The authors propose a checkpoint selection rule based on validation action-error quantiles and an augmentation method that generates observation-action pairs. Experiments use diffusion policies on MimicGen and a VLA on RoboCasa, with posttraining settings including residual RL and supervised finetuning. RoboCasa uses an action-only augmentation variant.
Critical strengths
-
The comparison between checkpoints with similar pass@1 is useful. Their different finetuning outcomes show why average success alone can be misleading when choosing a pretrained policy. Seeing this in both MimicGen and RoboCasa makes the result more convincing.
-
The split-data experiments and frozen-encoder controls support the overfitting explanation. Replacing the late checkpoint’s low-noise denoising steps with earlier weights also recovers much of the lost coverage. These experiments help explain where the effect comes from.
Critical weaknesses
-
My main concern is the lack of experiments on real robots. All evaluation is in MimicGen and RoboCasa simulation. The paper motivates offline checkpoint selection partly by the cost of real robot rollouts, but does not test whether the selector or augmentation helps on hardware. It is also unclear whether the same coverage loss appears under real sensing noise and contact dynamics.
-
The RL results need a separate test of the learned policies: the reported curves use success over the latest 100 training episodes. For the selector, I would like a direct comparison with mean validation action MSE and low-noise denoising loss at matched finetuning budgets. Matching a similarly defined pass@k oracle does not show how much downstream performance the selection rule gains.
Questions for authors
- Are any real robot results available that show the same coverage–reliability tradeoff or improvements from the proposed selector and augmentation?
- When evaluated on shared held-out initial conditions, how does the selected checkpoint compare with checkpoints chosen by mean validation action MSE or low-noise denoising loss after the same finetuning budget?
4 comments
RL train–eval gap and checkpoint selection by low-noise denoising loss
The RL results need a separate test of the learned policies: the reported curves use success over the latest 100 training episodes.
We thank the reviewer for raising this insightful question. We have added evaluation curves for held-out initial configurations to https://28544-rebuttal.github.io/#rl-gap and appendix C.2 of the updated manuscript. Every 50,000 environment steps, we evaluate the policy on the same 500 held-out initial configurations. When viewed side by side, the training and held-out evaluation curves show no noticeable performance gap.
For the selector, I would like a direct comparison with mean validation action MSE and low-noise denoising loss at matched finetuning budgets. Matching a similarly defined pass@k oracle does not show how much downstream performance the selection rule gains. When evaluated on shared held-out initial conditions, how does the selected checkpoint compare with checkpoints chosen by mean validation action MSE or low-noise denoising loss after the same finetuning budget?
We thank the reviewer for raising this insightful question! We have added comparison results and discussion to https://28544-rebuttal.github.io/#rl-gap and appendix C.2 of the updated manuscript***.***
Mean validation action MSE plateaus late in training, with its minimum within 0.3% of the final checkpoint's value, so selecting by it effectively gives the final checkpoint. We apologize for not clarifying this in the original submission. Please refer to Fig. 14(a), where best-of-k action MSE with (k=1) is exactly the mean validation action MSE. Selecting by this metric therefore gives the final-checkpoint baseline already included in our comparisons.
As for low-noise denoising score-matching loss (i.e., ε-prediction loss), first, it does not directly measure action MSE after the full sampling chain. Prior work likewise reports that validation denoising score matching loss or flow-matching loss is not significantly correlated with real-world task performance [1]. In contrast, our proposed coverage–reliability balance criterion uses sampled-action MSE, which correlates more strongly with task performance.
Furthermore, our Fig. 12 experiment only serves as a diagnostic experiment showing that the last two denoising calls account for most of the closed-loop pass@k tradeoff on one specific task, namely Threading D2. This does not establish that the same cutoff can be applied universally to other tasks. Moreover, selecting the noise-level cutoff is nontrivial. In Fig. 12(c), we distinguish the effects of different denoising bands through large-scale closed-loop evaluation. By contrast, our coverage–reliability balance criterion is fully offline and explicitly accounts for both favorable-tail sample quality (coverage) and typical and high-error samples (reliability) with no hand-tuned hyperparameters or thresholds.
We use the final two DDIM steps (same as Fig. 12(c)) as the cutoff for all five MimicGen tasks. For each task, we averaged the held-out ε-prediction MSE uniformly over all training diffusion timesteps at or below this cutoff and selected the checkpoint with the lowest average loss. This criterion selects epoch 100 for all five tasks. On Hammer Cleanup and Mug Cleanup, this happens to be the checkpoint with the highest pass@10, but on the remaining three tasks, epoch 100 is too early for either pass@1 or large-k pass@k to have reached its peak. Therefore, a fixed cutoff that works on one task might not identify a good checkpoint on the others.
Thank you again for your time and thoughtful feedback! Please let us know if further clarification or additional experimental results could help improve the score. We truly appreciate your input!
Reference:
- Allshire, Arthur, et al. "Scalable Behavior Cloning with Open Data, Training, and Evaluation." arXiv preprint arXiv:2606.27375 (2026).
Thank you for the additional experiments. The held-out RL evaluations address my concern about relying on training success, and the new comparison clarifies how the selector performs against the low-noise denoising baseline. The Push-T results also show that checkpoint selection and JointAug can improve rollout performance on hardware. Based on these additions, I am raising my rating to Weak acceptance.
Thank You for Your Feedback!
Thank you so much for taking the time to read our rebuttal and for raising your score. Your insightful feedback has really helped strengthen our submission!
Real robot results
We thank the reviewer for the insightful comment and acknowledge the lack of real-robot experiments in the original submission. To test whether our simulation findings carry over to a real robot, we added experiments on a 6-DoF Push-T task using only a wrist-mounted camera (https://28544-rebuttal.github.io/#real-robot, Sec. 6 of the revised manuscript). We train a diffusion policy on 680 demonstration trajectories and repeat our offline analyses on 60 held-out trajectories. The best-of-k validation action MSE and error quantiles exhibit the same patterns observed in simulation, consistent with the coverage–reliability tradeoff.
We evaluate three checkpoints on the same 20 fixed initial configurations, with five rollouts per checkpoint per configuration (300 rollouts in total). Fig. 7(e) in the updated manuscript reports the following results.
The offline-selected baseline checkpoint outperforms the final baseline checkpoint by 10 percentage points on pass@1 and 20 percentage points on pass@5. JointAug further improves pass@5 to 1.00, with comparable pass@1 to the offline-selected baseline. Both checkpoints also exhibit more behavioral diversity and better recovery capability than the final baseline checkpoint, as illustrated in the videos on our website.
To examine whether checkpoint selection also benefits downstream world-model learning, we train two action-conditioned world models on the same demonstration data, augmented with 200 rollout trajectories from either the offline-selected or final baseline checkpoint. All rollouts are included regardless of success. We evaluate both models on the same 100 held-out trajectories collected from intermediate policy checkpoints. Table 3 reports the following results.
Chunk 1 consists of four frames predicted from two ground-truth context frames, and chunk 2 consists of the next four frames predicted autoregressively. The world model trained with offline-selected checkpoint rollouts achieves higher PSNR and lower LPIPS in both chunks, suggesting that preserving policy coverage can also benefit downstream world-model learning.