- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies data augmentation for robot imitation learning and asks when and why augmented trajectories help behavior cloning. It develops a theoretical analysis of training on mixed real and augmented data, introducing trajectory coverage efficiency to measure the non-redundant information contributed by augmented trajectories and augmentation error to measure their deviation from expert-consistent behavior. These quantities lead to an effective sample size and a generalization bound with a statistical term and an augmentation-bias term, predicting diminishing returns and eventual saturation as more augmented data are added. From this analysis, the authors propose NYC, a trajectory-wise, geometry-consistent, diffusion-based augmentation system that combines geometry-aware view transformation, trajectory-wise consistency, diffusion-based visual repair, and inverse-kinematics action relabeling. Controlled Push-T experiments validate the predicted effects of augmentation quantity, coverage, and error. On five bimanual RLBench tasks, NYC improves over a real-only baseline and often outperforms prior augmentation methods, though gains vary by task.
Critical strengths
-
This paper provides a useful conceptual framework by explicitly separating the value of augmentation into information gain and augmentation bias. The resulting Novel Yet Clean principle is intuitive but meaningful: augmentation should provide sufficiently new trajectory information while introducing as little bias as possible.
-
The NYC framework is well motivated and technically reasonable: explicit geometry-aware warping enables larger trajectory-level changes, trajectory-wise consistency reduces temporal distortion, and diffusion-based repair addresses visual artifacts. The approach shows consistent gains over a real-only baseline and strong results on several evaluated tasks.
Critical weaknesses
-
There is a gap between theory and practice. The empirical proxies, such as ResNet-50 trajectory features and spectral distance, are reasonable but not tightly connected to the generalization bound. The assumptions, including Lipschitz expert and augmentation operators, β-mixing, deterministic dynamics, and finite horizon, are strong, and the bound is primarily a scaling result rather than a directly predictive tool. This makes the theoretical contribution more suggestive than definitive.
-
The experiments are not comprehensive enough. First, all experiments are in simulation, with no real-robot validation. Second, the abstract claims state-of-the-art performance on four of five bimanual tasks, yet Table 4 shows NYC underperforming baselines on Co-Lift Tray and Bi-Straighten Rope. This overstatement, together with the small number of seeds and limited baselines, weakens the empirical case. Third, the ablation study of each component in NYC is missing. In addition, NYC requires depth and camera intrinsics and trains a separate diffusion repair model for each task. This is a considerable computational overhead for a method motivated partly by the difficulty of scaling robot data collection.
Questions for authors
1.How sensitive is gamma_cover to the choice of trajectory encoder (ResNet-50)? An encoder-sensitivity analysis would clarify whether TCE is truly measuring useful trajectory coverage rather than generic appearance diversity.
2.How does each component in NYC contribute to downstream improvements?
3.What is the computational cost of training the diffusion repair model and generating augmented trajectories?
4.Why does NYC underperform D-CODA on Co-Lift Tray and More-real on Bi-Straighten Rope?
5.Can NYC work without depth, with a single camera, or across embodiments? What would be required for real-robot deployment?
6.Does the theory provide a practical rule for choosing the optimal augmentation amount, beyond observing saturation empirically?
2 comments
Response to Reviewer dmjf [2/2]
5. Real-world deployment costs
Deployment in real environments requires only an RGB+Depth camera. RGB+Depth cameras are widely used as standard equipment for embodied data collection, such as Intel RealSense5 and the Orbbec Astra series. Depth cameras are also easy to obtain. Our method is independent of the robot and only requires its camera parameters. It can therefore be readily extended to the real world.
6. Optimal amount of augmented data
We have tried to use our current theory to calculate the optimal amount of augmented data. The augmentation saturation point can indeed be predicted by solving a fixed-point equation within our theoretical framework. However, real-world robot training depends on many factors, making it difficult and expensive to accurately predict when the success rate will saturate. We therefore did not include this as a potential application. Our theory can mainly be used to measure augmentation efficiency and augmentation error, for example to compare methods or improve modules, and to predict and explain how performance changes with the amount of augmented data.
Responses to Weaknesses
1. Gap between theory and practice
To connect theory and practice, we validate the proposed theory through the experiments in Section 5.1. We also propose using spectral distances, feature extractors, and related methods to assess augmentation efficiency and augmentation bias, and successfully apply them to evaluate our new data augmentation system. In addition, the experiments in the appendix (Figure 13) demonstrate how the augmentation saturation theorem is reflected in practice. As noted in the conclusion, we acknowledge that our current theory relies on some strong assumptions. We thank the reviewer for pointing this out and have further refined the theory in the revised manuscript.
2. Experimental completeness
We do acknowledge the lack of real-robot experiments as a limitation of our work and will consider transferring our work to real robots in future research. However, we position our main contribution as theoretical analysis supported by extensive experiments in simulation. We also explain the feasibility of deployment in real environments in our response to Question 5. In our response to Question 4, we clarify that the claim of state-of-the-art performance on four of five tasks refers to comparisons with other data augmentation methods. On Bi-Straighten Rope, NYC outperforms all augmentation baselines, while More-real uses additional real data. We also discuss why NYC does not achieve the best performance on Co-Lift Tray. About ablation study, we add an ablation experiment on the diffusion repair module in our response to Question 2. About computational cost, we explain that the requirements of our algorithm are reasonable in our response to Question 3. Moreover, real-robot data collection involves equipment costs and labor costs, and faces data scarcity. We therefore consider efficient data augmentation using only generative models a feasible and reasonable solution.
We thank the reviewer again for the valuable feedback. We hope that our responses address your concerns, and we would be happy to provide any further clarification.
Response to Reviewer dmjf [1/2]
Firstly, We thank the reviewer for the insightful review and helpful suggestions. We appreciate the valuable feedback that helps us improve the work. We address the points under Questions and Weaknesses separately below.
Responses to Questions
1. Sensitivity of γcover to the encoder
We have added experiments with different encoders to demonstrate that γcover is insensitive to the specific encoder type and that our method's advantage is consistent across encoders. The results (NYC / D-CODA) are as follows:
2. Module contributions and ablation experiments
In our system, geometric shifts combined with diffusion repair reduce error and improve augmentation efficiency. Trajectory-level augmentation reduces augmentation error. These improvements are demonstrated in Table 1, Table 2, and Figure 5. Because our modules are highly interdependent, ablation experiments on individual modules may not show clear effects. This is also why our theoretical analysis introduces trajectory coverage efficiency and action spectrum analysis: they directly assess the augmentation system's efficiency and error, rather than reflecting them only indirectly through success rates.
We have added an ablation experiment that removes the diffusion repair module and retains only trajectory-level augmentation and geometric shifts. The success rate drops substantially, showing the necessity of diffusion repair for reducing augmentation error.
3. Diffusion model training and sampling costs
NYC has a clear computational cost advantage over baselines such as D-CODA and DMD. For one task setting, training takes approximately 4 hours on 8 A100 GPUs, and sampling 100 new trajectories, each with 100 new images, takes 2 hours. D-CODA and DMD require 2–3 times the computational cost and also require retraining when robot arm positions change. Moreover, our training objective is much simpler—repairing images rather than understanding injected position conditions—so convergence is faster, which can further reduce costs.
4. Why our method did not perform best on two tasks
We thank the reviewer for carefully examining our experimental results. Below, we discuss why our method did not achieve the best results on two tasks.
Bi-Straighten Rope: More-real adds 100 real trajectories and is not a baseline method for comparison. It is normal for data augmentation methods not to outperform More-real. Our analysis suggests that augmentation performs poorly on Bi-Straighten Rope because this task may be highly sensitive, limiting the benefit of training with augmented data. Even so, our method outperforms all other augmentation methods because of its low augmentation error.
Co-Lift Tray: This task is difficult: both arms must lift a tray together while keeping a ball from falling. Training on real data yields a success rate below 10%, with substantial randomness, and performance also depends on ACT training. Even if a new dataset provides more information, it may make ACT harder to converge. Success rates may therefore not accurately reflect augmentation quality. Indeed, More-real performs worse than Real-only despite adding real data, possibly because ACT struggles to converge, especially as new information increases. However, our advantage on this task remains consistent when measured by trajectory coverage efficiency and action spectrum analysis.
Overall, our method achieves the best results among all data augmentation methods on 4 out of 5 tasks, which demonstrates NYC's advantage.