- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether conclusions about robot demonstration data remain stable under changes in the downstream learning procedure. The authors propose a paired evaluation framework that crosses two source-data conditions with two learning operators, and measures both the within-operator data effect and the corresponding interaction across operators. The main evidence comes from Meta-World, where the measured clean-versus-perturbed data advantage changes substantially under different GMM adaptation rules, with additional deterministic-policy and PyBullet experiments used as controls to test the robustness and scope of this phenomenon.
Critical strengths
- The experimental design is careful. The paired four-cell setup reports both absolute performance and interaction effects, which avoids misleading conclusions from data gaps alone.
- The paper includes useful negative controls. In particular, the deterministic Meta-World setting shows almost no interaction, while PyBullet shows a clear loss-dependent effect, making the empirical conclusions more credible.
Critical weaknesses
- The main evidence is too narrow for the broad claim. The strongest result comes from small state-based policies, synthetic label corruption, and a GMM adaptation setting where fixing the variance also improves overall performance by a large margin. This makes it difficult to tell whether the paper identifies a general issue in robot-data evaluation or mainly a specific optimization pathology.
- The theory is only weakly connected to the main empirical result. The analysis concerns a one-step smooth objective, while the experiments evaluate closed-loop success after 250 adaptation updates, and the proposed first-update diagnostic does not reliably predict the final interaction.
Questions for authors
- Can the authors demonstrate the same operator-dependent data comparison with a substantially different and more realistic policy/data setting, rather than synthetic label corruption with small MLP policies?
- What practical role is the local analysis intended to play if the first-update diagnostic does not predict the final long-horizon interaction? Is it only qualitative intuition, or can it be made predictive?
3 comments
Thank you for the additional experiments and clarifications. The Diffusion Policy results are a useful addition and partially address my concern about the generality of the original findings.
However, the new results lack task-wise breakdowns and variability estimates, making the robustness of the reported interaction difficult to assess.
More importantly, the theoretical contribution remains unclear. While I appreciate the clarification that the analysis is not intended to predict long-horizon outcomes, it is still unclear what substantive insight it provides beyond qualitative intuition, given its limited connection to the empirical findings.
Overall, my original concerns are only partially addressed, and my assessment remains unchanged.
Response to Reviewer 48sU
We thank the reviewer for the thoughtful follow-up and for acknowledging that the additional Diffusion Policy results strengthen the generality of our findings. We would like to clarify two points regarding the empirical evidence and the intended theoretical contribution.
1. Empirical robustness and scope of the contribution.
We agree that task-wise breakdowns and variability estimates would provide a more complete characterization of the additional Diffusion Policy experiment. We therefore regard this experiment as complementary evidence rather than a standalone claim of universal generalization.
Importantly, the paper's central finding is already supported by multiple controlled comparisons with explicit uncertainty estimates. The primary GMM experiment reports an interaction of −5.99 pp (95% CI [−8.94, −3.26]), while the independent PyBullet experiment reports +12.19 pp (95% CI [+6.64, +18.59]), with positive task-level interactions for both Reach and Push. The Diffusion Policy results extend these observations to a substantially different policy architecture, practical demonstration-selection strategies, and full fine-tuning versus LoRA, without relying on synthetic label corruption or GMM variance estimation.
Our claim is not that every adaptation change must produce a significant interaction, but that data comparisons can depend on the learning operator and should therefore be evaluated accordingly. The paired four-cell protocol provides a concrete way to measure this dependence.
2. Substantive role of the theoretical analysis.
We appreciate the opportunity to clarify that this is primarily an empirical and methodological contribution, rather than a theory-driven prediction framework for long-horizon robot learning.
Nevertheless, Equation (2) establishes a precise result beyond qualitative intuition: the first-order data–operator interaction depends on the difference in validation-aligned operator changes across the two source policies, rather than on the average improvement produced by an operator. It also provides a second-order remainder bound and a sufficient condition for determining the sign of the local interaction. This explains why an operator can substantially improve both policies without necessarily changing their measured data contrast, and why the four absolute outcomes must be considered alongside the interaction. The main contribution therefore stands independently of long-horizon theoretical predictability: we identify and quantify an evaluation ambiguity, introduce a controlled protocol for measuring it, and demonstrate its practical relevance across different learning settings.
We hope this clarification helps distinguish the paper's intended contribution from a stronger theoretical claim that we do not make, and we respectfully invite the reviewer to reconsider the assessment in light of the combined evidence.
Response to Reviewer 48sU
We thank the reviewer for the constructive comments. We address both concerns with an additional Diffusion Policy experiment and a clarification of the theoretical contribution.
1. Generalization Beyond GMM and Synthetic Label Noise
We have extended our evaluation to Diffusion Policy on the RoboMimic Lift, Can, and Square manipulation tasks, using demonstration selection based on task coverage versus random sampling.
Unlike our original experiments, this setting compares two practical adaptation strategies—full fine-tuning and LoRA—without relying on synthetic label corruption or GMM variance estimation. Both adaptation strategies use matched initial weights, target demonstrations, training budgets, and evaluation scenarios.
The results are:
The data contrast decreases by 5.3 percentage points under LoRA, despite both adaptation strategies benefiting from coverage-selected demonstrations. This finding is particularly informative: operator-dependent data comparisons arise even when both data conditions are meaningful demonstration-selection strategies and neither adaptation procedure suffers from the GMM variance-estimation issue. Together with our original Meta-World and PyBullet experiments, this additional result broadens the empirical evidence for learning-operator-dependent data effects.
2. Role of the Local Theoretical Analysis
The purpose of our theoretical analysis is to characterize when data–operator interactions arise, rather than to predict long-horizon success from a single gradient step.
Equation (2) establishes a first-order cancellation condition: an operator change can substantially improve both source policies while producing little interaction if their validation-aligned improvements are similar. Conversely, different responses at the two source checkpoints can change the measured data contrast. This distinction is supported by our experiments. The GMM setting exhibits a substantial interaction, whereas the deterministic matched-group control yields a near-zero interaction (−0.27 pp). The additional Diffusion Policy experiment further illustrates that different adaptation operators can change the magnitude of a data advantage without reversing its sign. Thus, the theory provides a mechanistic explanation for why adaptation improvements and data-comparison changes need not coincide, while the paired four-cell protocol measures the resulting long-horizon interaction directly.
Summary
The additional Diffusion Policy results demonstrate that adaptation-sensitive data comparisons extend beyond the original GMM setting and synthetic label corruption. Combined with our controlled interventions and theoretical analysis, these findings strengthen the central conclusion: the measured advantage of demonstration data depends not only on the data itself, but also on the adaptation procedure used to exploit it.