acceptodds
Under review as a conference paper at ICLR 2027

Novel Yet Clean: Towards Theory-Guided Data Augmentation for Robot Imitation Learning

Abstract

Data augmentation is a standard remedy for the demonstration-data bottleneck in robot imitation learning, yet existing methods are largely designed empirically and lack a unified theory of when and why augmentation helps. We develop a behavior-cloning analysis that jointly models real and augmented trajectories and reveals a fundamental principle for effective augmentation: augmented data should provide sufficiently new information while introducing as little augmentation error as possible—*Novel Yet Clean*. Guided by this principle, we propose **NYC**, a trajectory-wise, geometry-consistent, diffusion-based augmentation system that increases trajectory coverage through geometry-aware view synthesis while reducing augmentation bias through trajectory-wise consistency and diffusion-based visual repair. Extensive experiments on Push-T and five bimanual RLBench tasks validate both the theoretical predictions and the effectiveness of the resulting system. NYC improves task success rates by +29% +180% over a real-only baseline and achieves state-of-the-art performance on four of the five bimanual tasks, demonstrating that our theoretical principles can effectively translate into practical system design and downstream performance gains.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Imitation Learning, Data Augmentation, Diffusion Models, Behavior Cloning, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dmjfOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies data augmentation for robot imitation learning and asks when and why augmented trajectories help behavior cloning. It develops a theoretical analysis of training on mixed real and augmented data, introducing trajectory coverage efficiency to measure the non-redundant information contributed by augmented trajectories and augmentation error to measure their deviation from expert-consistent behavior. These quantities lead to an effective sample size and a generalization bound with a statistical term and an augmentation-bias term, predicting diminishing returns and eventual saturation as more augmented data are added. From this analysis, the authors propose NYC, a trajectory-wise, geometry-consistent, diffusion-based augmentation system that combines geometry-aware view transformation, trajectory-wise consistency, diffusion-based visual repair, and inverse-kinematics action relabeling. Controlled Push-T experiments validate the predicted effects of augmentation quantity, coverage, and error. On five bimanual RLBench tasks, NYC improves over a real-only baseline and often outperforms prior augmentation methods, though gains vary by task.

Critical strengths
  • This paper provides a useful conceptual framework by explicitly separating the value of augmentation into information gain and augmentation bias. The resulting Novel Yet Clean principle is intuitive but meaningful: augmentation should provide sufficiently new trajectory information while introducing as little bias as possible.

  • The NYC framework is well motivated and technically reasonable: explicit geometry-aware warping enables larger trajectory-level changes, trajectory-wise consistency reduces temporal distortion, and diffusion-based repair addresses visual artifacts. The approach shows consistent gains over a real-only baseline and strong results on several evaluated tasks.

Critical weaknesses
  • There is a gap between theory and practice. The empirical proxies, such as ResNet-50 trajectory features and spectral distance, are reasonable but not tightly connected to the generalization bound. The assumptions, including Lipschitz expert and augmentation operators, β-mixing, deterministic dynamics, and finite horizon, are strong, and the bound is primarily a scaling result rather than a directly predictive tool. This makes the theoretical contribution more suggestive than definitive.

  • The experiments are not comprehensive enough. First, all experiments are in simulation, with no real-robot validation. Second, the abstract claims state-of-the-art performance on four of five bimanual tasks, yet Table 4 shows NYC underperforming baselines on Co-Lift Tray and Bi-Straighten Rope. This overstatement, together with the small number of seeds and limited baselines, weakens the empirical case. Third, the ablation study of each component in NYC is missing. In addition, NYC requires depth and camera intrinsics and trains a separate diffusion repair model for each task. This is a considerable computational overhead for a method motivated partly by the difficulty of scaling robot data collection.

Questions for authors

1.How sensitive is gamma_cover to the choice of trajectory encoder (ResNet-50)? An encoder-sensitivity analysis would clarify whether TCE is truly measuring useful trajectory coverage rather than generic appearance diversity.

2.How does each component in NYC contribute to downstream improvements?

3.What is the computational cost of training the diffusion repair model and generating augmented trajectories?

4.Why does NYC underperform D-CODA on Co-Lift Tray and More-real on Bi-Straighten Rope?

5.Can NYC work without depth, with a single camera, or across embodiments? What would be required for real-robot deployment?

6.Does the theory provide a practical rule for choosing the optimal augmentation amount, beyond observing saturation empirically?

2 comments
AuthorsOpenReview ↗

Response to Reviewer dmjf [1/2]

Firstly, We thank the reviewer for the insightful review and helpful suggestions. We appreciate the valuable feedback that helps us improve the work. We address the points under Questions and Weaknesses separately below.

Responses to Questions

1. Sensitivity of γcover\gamma_{\mathrm{cover}} to the encoder

We have added experiments with different encoders to demonstrate that γcover\gamma_{\mathrm{cover}} is insensitive to the specific encoder type and that our method's advantage is consistent across encoders. The results (NYC / D-CODA) are as follows:

EncoderCo-Lift BallCo-Push BoxCo-Lift TrayDual Push ButtonsBi-Straighten RopeMean ratio
ResNet-500.0469 / 0.01490.0502 / 0.01300.0458 / 0.00860.0507 / 0.01420.0499 / 0.01493.85×
ResNet-340.0497 / 0.01720.0517 / 0.01290.0398 / 0.01110.0646 / 0.02490.0582 / 0.02043.19×
DenseNet-1210.0555 / 0.01820.0545 / 0.01420.0476 / 0.01230.0634 / 0.01560.0627 / 0.01773.67×
EfficientNet-B00.1190 / 0.04350.0997 / 0.01840.0889 / 0.02190.1491 / 0.04500.1537 / 0.04213.84×

2. Module contributions and ablation experiments

In our system, geometric shifts combined with diffusion repair reduce error and improve augmentation efficiency. Trajectory-level augmentation reduces augmentation error. These improvements are demonstrated in Table 1, Table 2, and Figure 5. Because our modules are highly interdependent, ablation experiments on individual modules may not show clear effects. This is also why our theoretical analysis introduces trajectory coverage efficiency and action spectrum analysis: they directly assess the augmentation system's efficiency and error, rather than reflecting them only indirectly through success rates.

We have added an ablation experiment that removes the diffusion repair module and retains only trajectory-level augmentation and geometric shifts. The success rate drops substantially, showing the necessity of diffusion repair for reducing augmentation error.

TaskNYC (Mean ± Std)Without diffusion repair (Mean ± Std)Mean decrease
Co-Lift Ball56.0% ± 3.3%18.7% ± 10.5%37.3 %
Co-Push Box77.3% ± 1.9%52.0% ± 15.0%25.3 %
Dual Push Buttons49.3% ± 3.8%48.0% ± 3.3%1.33 %

3. Diffusion model training and sampling costs

NYC has a clear computational cost advantage over baselines such as D-CODA and DMD. For one task setting, training takes approximately 4 hours on 8 A100 GPUs, and sampling 100 new trajectories, each with 100 new images, takes 2 hours. D-CODA and DMD require 2–3 times the computational cost and also require retraining when robot arm positions change. Moreover, our training objective is much simpler—repairing images rather than understanding injected position conditions—so convergence is faster, which can further reduce costs.

4. Why our method did not perform best on two tasks

We thank the reviewer for carefully examining our experimental results. Below, we discuss why our method did not achieve the best results on two tasks.

Bi-Straighten Rope: More-real adds 100 real trajectories and is not a baseline method for comparison. It is normal for data augmentation methods not to outperform More-real. Our analysis suggests that augmentation performs poorly on Bi-Straighten Rope because this task may be highly sensitive, limiting the benefit of training with augmented data. Even so, our method outperforms all other augmentation methods because of its low augmentation error.

Co-Lift Tray: This task is difficult: both arms must lift a tray together while keeping a ball from falling. Training on real data yields a success rate below 10%, with substantial randomness, and performance also depends on ACT training. Even if a new dataset provides more information, it may make ACT harder to converge. Success rates may therefore not accurately reflect augmentation quality. Indeed, More-real performs worse than Real-only despite adding real data, possibly because ACT struggles to converge, especially as new information increases. However, our advantage on this task remains consistent when measured by trajectory coverage efficiency and action spectrum analysis.

Overall, our method achieves the best results among all data augmentation methods on 4 out of 5 tasks, which demonstrates NYC's advantage.

AuthorsOpenReview ↗

Response to Reviewer dmjf [2/2]

5. Real-world deployment costs

Deployment in real environments requires only an RGB+Depth camera. RGB+Depth cameras are widely used as standard equipment for embodied data collection, such as Intel RealSense5 and the Orbbec Astra series. Depth cameras are also easy to obtain. Our method is independent of the robot and only requires its camera parameters. It can therefore be readily extended to the real world.

6. Optimal amount of augmented data

We have tried to use our current theory to calculate the optimal amount of augmented data. The augmentation saturation point can indeed be predicted by solving a fixed-point equation within our theoretical framework. However, real-world robot training depends on many factors, making it difficult and expensive to accurately predict when the success rate will saturate. We therefore did not include this as a potential application. Our theory can mainly be used to measure augmentation efficiency and augmentation error, for example to compare methods or improve modules, and to predict and explain how performance changes with the amount of augmented data.

Responses to Weaknesses

1. Gap between theory and practice

To connect theory and practice, we validate the proposed theory through the experiments in Section 5.1. We also propose using spectral distances, feature extractors, and related methods to assess augmentation efficiency and augmentation bias, and successfully apply them to evaluate our new data augmentation system. In addition, the experiments in the appendix (Figure 13) demonstrate how the augmentation saturation theorem is reflected in practice. As noted in the conclusion, we acknowledge that our current theory relies on some strong assumptions. We thank the reviewer for pointing this out and have further refined the theory in the revised manuscript.

2. Experimental completeness

We do acknowledge the lack of real-robot experiments as a limitation of our work and will consider transferring our work to real robots in future research. However, we position our main contribution as theoretical analysis supported by extensive experiments in simulation. We also explain the feasibility of deployment in real environments in our response to Question 5. In our response to Question 4, we clarify that the claim of state-of-the-art performance on four of five tasks refers to comparisons with other data augmentation methods. On Bi-Straighten Rope, NYC outperforms all augmentation baselines, while More-real uses additional real data. We also discuss why NYC does not achieve the best performance on Co-Lift Tray. About ablation study, we add an ablation experiment on the diffusion repair module in our response to Question 2. About computational cost, we explain that the requirements of our algorithm are reasonable in our response to Question 3. Moreover, real-robot data collection involves equipment costs and labor costs, and faces data scarcity. We therefore consider efficient data augmentation using only generative models a feasible and reasonable solution.

We thank the reviewer again for the valuable feedback. We hope that our responses address your concerns, and we would be happy to provide any further clarification.

Reviewer_r7CTOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper researches a generalization theory for imitation learning data augmentation, called NOVEL YET CLEAN (NYC), means the augmented data should provide sufficiently new information while introducing as little augmentation error as possible. It provides theoretical proof to support their claim in Eq 6 and 7 and concluded that increasing trajectory coverage efficiency and reducing augmentation error are two effective ways to guide data augmentation. Experiments on RLBench and PerAct2 shows the proposed theory is effective.

Critical strengths
  1. The paper is researching on a generalization theory for imitation learning data augmentation, which is a very impacting motivation.
  2. The theory of NYC is novel and well-proven by theory.
  3. Effective-sample-size generalization bound concludes trajectory coverage efficiency must be increased.
  4. Saturation of augmentation gains suggests that augmentation error is also important
  5. Experiments on RLBench and PerAct2 shows the proposed theory is effective and largely outperforming policy with other augmentation methods.
Critical weaknesses
  1. To what scale of augmented data, the proposed theory works? It will be good if have this trend reported
  2. Data augmentation for imitation learning my includes a lot other modalities other than vision and action, e.g. tactile, acoustic,etc. Does the theory still hold in these cases?
  3. Lack of real-world deploy experiment will weaken the proposed theory.
Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer r7CT

We thank the reviewer for the thoughtful review. Our responses to the questions are below.

1. Effectiveness across different amounts of augmented data

Our theory is general and applies across different ratios of augmented to original data. When the augmented dataset is small relative to the original dataset, we predict that performance improves as novelty increases and augmentation error decreases. This is verified in our experiments (Section 5.1). When the augmented dataset is large relative to the original dataset, performance converges to an upper bound determined by augmentation error, as given in Theorem 2. The supplementary experiments in the appendix (Fig. 13) further show that the changes in success rate with the relative amount of augmented data agree with our theoretical predictions.

2. Different modalities

Our theoretical definition is not restricted to vision. We use visual features in these tasks because vision is the model's most important direct input in this setting, as in most robotic settings. However, our theoretical definition is not restricted to vision. Features such as tactile distributions, action sequences, and multimodal distributions can also be used as inputs.

3. Real-world deployment

We also recognize this issue and acknowledge the lack of real-robot experiments as a limitation of our work. However, we position our contribution as theoretical analysis supported by extensive experiments in simulation. Real-robot tests involve many factors and may not be suitable for validating the theory. We do will consider transferring our work to real robots in future research.

Please let us know your thoughts, and we are more than happy to answer any further questions.

Reviewer_vfHyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates data augmentation for robotic imitation learning and proposes the “novel and clean” principle: augmented data should provide new information while minimizing errors. Through a generalization error analysis, the authors characterize the trade-off between effective sample size and augmentation bias. They develop NYC, a system that combines whole-trajectory transformations, geometric reprojection, diffusion-based inpainting, and action relabeling to generate demonstrations. Experiments on Push-T and five simulated bimanual tasks show that the method generally outperforms baselines trained only on the original data.

Critical strengths

The main comparisons in the bimanual experiments keep the data quantity and policy training settings consistent. All three augmentation methods start from the same 100 demonstrations, add 100 generated trajectories, and use the same ACT policy and number of training steps. This reduces the influence of differences in the original data budget or policy architecture on the performance comparison.

The division of responsibilities between geometric transformation and diffusion-based inpainting is clear. NYC first uses depth and camera parameters to establish the target viewpoint, then employs a diffusion model to fill holes and correct projection artifacts. This explicitly constrains the spatial transformation through geometry, reducing the burden on the generative model to handle both viewpoint changes and visual details.

The training pairs for the diffusion-based inpainting model are constructed in a clever way. The authors apply a viewpoint transformation to an original image and then reverse the transformation, using the resulting image with reprojection defects as the degraded input and the original image as the supervision target. This allows the inpainting model to be trained directly on existing demonstrations without collecting additional paired images from new viewpoints.

Critical weaknesses

Trajectory coverage efficiency may assign high novelty scores to exact duplicates. Equation (4) on page 4 measures aggregate similarity across all pairs of augmented and original trajectories. Consider a simple example: the original dataset contains nn trajectories with mutually orthogonal normalized features, and the augmented dataset is an exact copy of the original dataset. Substituting these into the equation gives a coverage efficiency of 1−1/n1 - 1/\sqrt{n}, which equals 0.9 when n=100n = 100. Thus, data that provide no new information can still receive a high coverage score. This suggests that the metric does not reliably distinguish novel trajectories from duplicates of existing ones.

The generalization analysis does not adequately account for the difference between the training and target distributions. Equation (2) defines the target risk over the original expert's state distribution, whereas the training data come from a mixture of original and augmented data. When bounding the statistical term, Appendix A.2 directly applies a standard generalization bound without explicitly proving that augmentation preserves the target distribution or introducing a distribution-shift term. Even if the augmented action labels are entirely correct, concentrating many augmented samples in different regions of the state space may alter the training objective. This effect cannot be explained solely by action-label errors.

The population expectation of augmentation error is directly used to bound error on a finite training set. In Equation (56) of the appendix, εaug2\varepsilon_{\mathrm{aug}}^2 is an expectation, but Equation (57) directly uses it to upper-bound the empirical loss on the augmented training set. A finite-sample average is not necessarily bounded above by its population expectation. Establishing the high-probability result claimed by the theorem generally requires an additional concentration inequality and a corresponding error term. The proof does not justify this step, which warrants particular care when augmented samples are correlated.

Visual differences do not necessarily constitute new information that is useful for the task. Section 4.4 extracts trajectory features using a pretrained ResNet-50 and measures coverage through feature similarity. This metric may be sensitive to changes in backgrounds, textures, or occlusions, as well as generation artifacts, without accurately capturing task-critical factors such as contact locations or the relative poses of the two arms. The paper neither examines whether coverage scores remain stable across different feature extractors nor establishes that they reliably predict task success rates.

Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer vfHy [1/2]

We sincerely thank the reviewer for the suggestions on our theory. For Questions 1, 2, and 3, we have carefully revised the theoretical definitions, proofs, and presentation of the conclusions, adding the necessary conditions and corrections. We have uploaded a revised version with these theoretical improvements. The changes are mainly in the theory section and Appendix A, and all changes are marked in blue. We thank the reviewer for helping us improve the theory. We have also noted some limitations of our modeling in the conclusion.

1. Definition of γcover\gamma_{\mathrm{cover}}

The counterexample for γcover\gamma_{\mathrm{cover}} does reveal an issue with its original definition. However, none of our subsequent theoretical derivations depend on its specific expression. We have therefore redefined γcover\gamma_{\mathrm{cover}} in the revised version (Definition 2) and retained the original expression only as one possible way to compute it (Eq. (5)). There could be many ways to compute γcover\gamma_{\mathrm{cover}}. Also, under the current computation method, different trajectories cannot have mutually orthogonal normalized features in practical settings, so the effect would not be as large as 0.9 (See Figure 12). In our experiments, we use this computation only to compare relative values of γcover\gamma_{\mathrm{cover}} and thus compare how much new information different methods introduce, as shown in Table 1.

We have added experiments using alternative ways to compute γcover\gamma_{\mathrm{cover}}, including nearest-neighbor distance and projection residuals. The results (NYC / D-CODA) are as follows:

MetricCo-Lift BallCo-Push BoxCo-Lift TrayDual Push ButtonsBi-Straighten Rope
Projection residual γproj\gamma_{\mathrm{proj}}0.0308 / 0.00120.0372 / 0.00100.0265 / 0.00100.0356 / 0.00090.0341 / 0.0009
Nearest-neighbor distance0.1334 / 0.02580.1422 / 0.02390.1332 / 0.02710.1403 / 0.02250.1391 / 0.0225
RBF-MMD²/20.3314 / 0.02120.4777 / 0.02320.5098 / 0.04910.4064 / 0.01740.3981 / 0.0151
Optimal transport W1W_10.1376 / 0.02580.1480 / 0.02390.1417 / 0.02710.1447 / 0.02250.1429 / 0.0225
Kernel subspace residual0.8470 / 0.06640.9432 / 0.07040.9899 / 0.13730.9036 / 0.05560.8728 / 0.0499

These results show that alternative computations of γcover\gamma_{\mathrm{cover}} are feasible, that comparing algorithms using its relative values is reliable, and that NYC has an advantage.

2. Neglecting distribution shift

The reviewer noted that augmentation bias may also be affected by the distribution shift introduced by augmented data. This is a helpful correction to a theoretical detail. In the revised version, we have added an assumption on this bias and absorbed the distribution error into the statistical term (Eq. (46), Appendix A). We have also carefully revised the proof (Eqs. (40)–(54), Appendix A). We added this assumption because, in most robotic imitation learning settings, differences between trajectories arise from randomly distributed robot arm and target positions. The augmented data are generated by applying zero-mean Gaussian perturbations to the robot arm positions in the original data. Datasets large enough for policy convergence are also generally large, so the distribution shift introduced by this augmentation is limited.

3. Population expectation of augmentation error

To address this issue, we have revised the definition of augmentation error to express it over a finite training set (Eq. (3)), so that the conclusion holds rigorously. We have also updated the proof accordingly (Eq. (67), Appendix A). The new definition does not affect our theoretical conclusions. We introduce augmentation bias to explain that, as the amount of augmented data increases, the loss gradually converges to a lower bound determined by augmentation bias.

AuthorsOpenReview ↗

Response to Reviewer vfHy [2/2]

4. Feature extractors

We use visual features to compute trajectory coverage efficiency in these tasks because vision is the model's most important direct input in this setting, as in most robotic settings. However, our theoretical definition is not restricted to vision. Features such as action sequences, tactile distributions, and dual-arm position distributions can also be used as inputs.

We have therefore added experiments with different feature extractors to demonstrate that γcover\gamma_{\mathrm{cover}} is insensitive to the specific encoder type and that our method's advantage is consistent across encoders. The results (NYC / D-CODA) are as follows:

EncoderCo-Lift BallCo-Push BoxCo-Lift TrayDual Push ButtonsBi-Straighten RopeMean ratio
ResNet-500.0469 / 0.01490.0502 / 0.01300.0458 / 0.00860.0507 / 0.01420.0499 / 0.01493.85×
ResNet-340.0497 / 0.01720.0517 / 0.01290.0398 / 0.01110.0646 / 0.02490.0582 / 0.02043.19×
DenseNet-1210.0555 / 0.01820.0545 / 0.01420.0476 / 0.01230.0634 / 0.01560.0627 / 0.01773.67×
EfficientNet-B00.1190 / 0.04350.0997 / 0.01840.0889 / 0.02190.1491 / 0.04500.1537 / 0.04213.84×

We hope that our responses can fully address your concerns and look forward to your feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful evaluations and insightful suggestions. We have provided point-by-point responses to each reviewer and summarize the main theoretical revisions and additional experiments below for the reviewers and the Area Chair.

  1. Refined theoretical definitions and proofs. Following Reviewer vfHy’s suggestions, we have carefully revised the theoretical section, including the definitions, proofs, and presentation of the conclusions, adding the necessary conditions and corrections. Specifically, we have revised the definition of γcover\gamma_{\mathrm{cover}} to distinguish its theoretical definition from specific computational choices, introduced an explicit assumption to control distribution shift, and reformulated augmentation error using an empirical definition over the finite training set, updating the proofs accordingly. These revisions are primarily in the theory section and Appendix A and are highlighted in blue in the uploaded manuscript.

  2. Additional coverage-metric and encoder experiments. We have added comparisons using five alternative metrics and four different encoders. NYC’s advantage over D-CODA remains consistent across the tested metrics and encoders. Across the four encoders, the mean coverage-efficiency ratios over tasks range from 3.19× to 3.85×. These results support the robustness of the comparative findings and the feasibility of using the proposed approach to evaluate augmentation systems.

  3. Additional ablation of the diffusion repair module. Removing this module decreases success rates on all three tested tasks, with drops of 37.3 and 25.3 percentage points on Co-Lift Ball and Co-Push Box, respectively. These results further illustrate the module’s contribution.

We have also discussed the connections between theory and experiments, the computational costs of our method, differences in performance across tasks, and the costs of real-world deployment, among other issues.

We appreciate the reviewers’ suggestions, which have helped us improve the paper. We look forward to your feedback and would be happy to provide any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.