- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes RoboDrop, a data-filtering framework for improving VLA model post-training by filtering unreliable robot demonstrations. The method estimates sample quality through local gradient compatibility with clean validation samples matched by task semantics and visual context. During a one-epoch warm-up, RoboDrop computes sample-level compatibility scores along the training trajectory, aggregates them at the episode level, and filters low-quality demonstrations using fixed-ratio or automatic selection. Experiments on LIBERO, Robomimic-MH, and real-robot tasks demonstrate improved discrimination of unreliable demonstrations and downstream policy performance.
Critical strengths
- The paper addresses data quality in VLA post-training, an important challenge for adapting pretrained models to new robotic tasks.
- The proposed local gradient compatibility provides an intuitive, model-aware approach to identifying unreliable demonstrations.
- The method is evaluated on synthetic corruptions, naturally suboptimal demonstrations, and real-robot tasks, with comparisons against multiple baselines.
Critical weaknesses
- In practical settings, how to define or sample reference episodes? The authors mention that they sample demonstrations produced by proficient operators. It's not clear what's the criteria of sampling or just random sampling. Besides, the assumption of proficient operators produces high-quality demonstrations while less-skilled operators produces flawed demonstrations is not rigorous. Less-skilled operators may produce valuable failure recovery trajectories.
- The method relies on matching candidate samples to clean validation demonstrations, which may limit the model to fit to the reference distribution in a stronger way. In particular, failure recovery often involves states rarely observed in successful demonstrations. Could RoboDrop incorrectly filter valuable recovery behaviors simply because they are out of distribution relative to the reference set?
- How does the method distinguish operation validity and style? Low gradient compatibility does not necessarily indicate incorrect supervision, as different operators may adopt distinct but equally valid action strategies. The current evaluation largely relies on operator proficiency as a proxy for data quality. It remains unclear whether RoboDrop can distinguish genuinely incorrect actions from valid demonstrations with different execution styles.
- Although the paper emphasizes fine-grained supervision auditing, sample-level scores are ultimately averaged for episode-level filtering. Moreover, the experiments primarily evaluate episode-level corruption or quality labels, without directly assessing the detection of localized errors within otherwise valid trajectories.
Questions for authors
- Has the BIC-based filtering strategy been evaluated on entirely clean datasets (0% contamination)? Can it reliably avoid removing valid demonstrations when no filtering is necessary?
- In Robomimic-MH, RoboDrop uses π0.5 for data scoring but X-VLA for downstream post-training. How does this compare with using the same backbone for both scoring and training?
Thank you for the detailed responses and the additional experiments. Regarding Q1, the additional experiments provide useful evidence that visually similar contexts with different valid actions can still exhibit compatible gradients. The comparison between clean pairs, random pairs, and clean–corrupted pairs supports the authors' argument that gradient compatibility is more informative than action similarity alone. Although the degree of separation varies across tasks, I find the additional analysis helpful in clarifying the behavior of RoboDrop in such scenarios.
Regarding Q2, the Right-only reference experiment is particularly useful, as it demonstrates that RoboDrop does not simply discard all valid action modes absent from the reference set. The retention of 45/54 proficient demonstrations from the two unrepresented modes is encouraging. However, the retention rate for the Left mode (10/17) remains noticeably lower than that for the Front mode (35/37), suggesting that the method may still be sensitive to substantial reference bias. Nevertheless, the similar downstream rollout success across the three configurations provides additional evidence of robustness. I encourage the authors to acknowledge this remaining sensitivity as a limitation in the final version.
Regarding Q3, the additional experiments effectively address my concern. The localized-corruption results show that worst-window aggregation can improve sensitivity to localized errors, while the results on naturally collected demonstrations support full-trajectory mean as a more robust default when corruption patterns are unknown. I think the results are reasonable and consider this concern largely addressed.
Overall, I appreciate the additional analyses and clarifications. The authors have addressed most of my concerns, and I lean toward Weak Accept.