acceptodds
Under review as a conference paper at ICLR 2027

Adaptation Changes Measured Data Comparisons

Abstract

Choosing robot demonstration data requires comparing the policies learned from them, often after adaptation to a target task. A success difference measured with one adapter does not reveal whether that comparison persists with another: the adapter may change both overall performance and the effect attributed to the source data. We address this ambiguity with a paired four-cell evaluation that crosses two source-data conditions with two learning operators, holds the remaining experimental choices fixed, and reports both absolute success and the change in the data contrast. In Meta-World, fixing Gaussian-mixture adaptation scales changes the perturbed-minus-clean effect by percentage points (95% interval ), while improving clean-data success by points. Coupled channel interventions distinguish reductions in the data contrast accompanied by degraded versus improved performance. A separate PyBullet direct-training comparison reduces the noisy-data penalty by points () under Huber rather than MSE loss. A deterministic-policy control using the same matched acquisition groups yields an interaction of points (). These results demonstrate adaptation-sensitive data comparisons and provide a protocol for checking a measured data advantage against the learning procedure intended to use it.

Primary area:
datasets and benchmarks
Keywords:
robot experiences

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_48sUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether conclusions about robot demonstration data remain stable under changes in the downstream learning procedure. The authors propose a paired 2×22\times2 evaluation framework that crosses two source-data conditions with two learning operators, and measures both the within-operator data effect and the corresponding interaction across operators. The main evidence comes from Meta-World, where the measured clean-versus-perturbed data advantage changes substantially under different GMM adaptation rules, with additional deterministic-policy and PyBullet experiments used as controls to test the robustness and scope of this phenomenon.

Critical strengths
  1. The experimental design is careful. The paired four-cell setup reports both absolute performance and interaction effects, which avoids misleading conclusions from data gaps alone.
  2. The paper includes useful negative controls. In particular, the deterministic Meta-World setting shows almost no interaction, while PyBullet shows a clear loss-dependent effect, making the empirical conclusions more credible.
Critical weaknesses
  1. The main evidence is too narrow for the broad claim. The strongest result comes from small state-based policies, synthetic label corruption, and a GMM adaptation setting where fixing the variance also improves overall performance by a large margin. This makes it difficult to tell whether the paper identifies a general issue in robot-data evaluation or mainly a specific optimization pathology.
  2. The theory is only weakly connected to the main empirical result. The analysis concerns a one-step smooth objective, while the experiments evaluate closed-loop success after 250 adaptation updates, and the proposed first-update diagnostic does not reliably predict the final interaction.
Questions for authors
  1. Can the authors demonstrate the same operator-dependent data comparison with a substantially different and more realistic policy/data setting, rather than synthetic label corruption with small MLP policies?
  2. What practical role is the local analysis intended to play if the first-update diagnostic does not predict the final long-horizon interaction? Is it only qualitative intuition, or can it be made predictive?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 48sU

We thank the reviewer for the constructive comments. We address both concerns with an additional Diffusion Policy experiment and a clarification of the theoretical contribution.

1. Generalization Beyond GMM and Synthetic Label Noise

We have extended our evaluation to Diffusion Policy on the RoboMimic Lift, Can, and Square manipulation tasks, using demonstration selection based on task coverage versus random sampling.

Unlike our original experiments, this setting compares two practical adaptation strategies—full fine-tuning and LoRA—without relying on synthetic label corruption or GMM variance estimation. Both adaptation strategies use matched initial weights, target demonstrations, training budgets, and evaluation scenarios.

The results are:

AdaptationRandomCoverage-selectedData contrast
Full fine-tuning68.2%75.4%+7.2 pp
LoRA70.1%72.0%+1.9 pp

The data contrast decreases by 5.3 percentage points under LoRA, despite both adaptation strategies benefiting from coverage-selected demonstrations. This finding is particularly informative: operator-dependent data comparisons arise even when both data conditions are meaningful demonstration-selection strategies and neither adaptation procedure suffers from the GMM variance-estimation issue. Together with our original Meta-World and PyBullet experiments, this additional result broadens the empirical evidence for learning-operator-dependent data effects.

2. Role of the Local Theoretical Analysis

The purpose of our theoretical analysis is to characterize when data–operator interactions arise, rather than to predict long-horizon success from a single gradient step.

Equation (2) establishes a first-order cancellation condition: an operator change can substantially improve both source policies while producing little interaction if their validation-aligned improvements are similar. Conversely, different responses at the two source checkpoints can change the measured data contrast. This distinction is supported by our experiments. The GMM setting exhibits a substantial interaction, whereas the deterministic matched-group control yields a near-zero interaction (−0.27 pp). The additional Diffusion Policy experiment further illustrates that different adaptation operators can change the magnitude of a data advantage without reversing its sign. Thus, the theory provides a mechanistic explanation for why adaptation improvements and data-comparison changes need not coincide, while the paired four-cell protocol measures the resulting long-horizon interaction directly.

Summary

The additional Diffusion Policy results demonstrate that adaptation-sensitive data comparisons extend beyond the original GMM setting and synthetic label corruption. Combined with our controlled interventions and theoretical analysis, these findings strengthen the central conclusion: the measured advantage of demonstration data depends not only on the data itself, but also on the adaptation procedure used to exploit it.

Thank you for the additional experiments and clarifications. The Diffusion Policy results are a useful addition and partially address my concern about the generality of the original findings.

However, the new results lack task-wise breakdowns and variability estimates, making the robustness of the reported interaction difficult to assess.

More importantly, the theoretical contribution remains unclear. While I appreciate the clarification that the analysis is not intended to predict long-horizon outcomes, it is still unclear what substantive insight it provides beyond qualitative intuition, given its limited connection to the empirical findings.

Overall, my original concerns are only partially addressed, and my assessment remains unchanged.

AuthorsOpenReview ↗

Response to Reviewer 48sU

We thank the reviewer for the thoughtful follow-up and for acknowledging that the additional Diffusion Policy results strengthen the generality of our findings. We would like to clarify two points regarding the empirical evidence and the intended theoretical contribution.

1. Empirical robustness and scope of the contribution.

We agree that task-wise breakdowns and variability estimates would provide a more complete characterization of the additional Diffusion Policy experiment. We therefore regard this experiment as complementary evidence rather than a standalone claim of universal generalization.

Importantly, the paper's central finding is already supported by multiple controlled comparisons with explicit uncertainty estimates. The primary GMM experiment reports an interaction of −5.99 pp (95% CI [−8.94, −3.26]), while the independent PyBullet experiment reports +12.19 pp (95% CI [+6.64, +18.59]), with positive task-level interactions for both Reach and Push. The Diffusion Policy results extend these observations to a substantially different policy architecture, practical demonstration-selection strategies, and full fine-tuning versus LoRA, without relying on synthetic label corruption or GMM variance estimation.

Our claim is not that every adaptation change must produce a significant interaction, but that data comparisons can depend on the learning operator and should therefore be evaluated accordingly. The paired four-cell protocol provides a concrete way to measure this dependence.

2. Substantive role of the theoretical analysis.

We appreciate the opportunity to clarify that this is primarily an empirical and methodological contribution, rather than a theory-driven prediction framework for long-horizon robot learning.

Nevertheless, Equation (2) establishes a precise result beyond qualitative intuition: the first-order data–operator interaction depends on the difference in validation-aligned operator changes across the two source policies, rather than on the average improvement produced by an operator. It also provides a second-order remainder bound and a sufficient condition for determining the sign of the local interaction. This explains why an operator can substantially improve both policies without necessarily changing their measured data contrast, and why the four absolute outcomes must be considered alongside the interaction. The main contribution therefore stands independently of long-horizon theoretical predictability: we identify and quantify an evaluation ambiguity, introduce a controlled protocol for measuring it, and demonstrate its practical relevance across different learning settings.

We hope this clarification helps distinguish the paper's intended contribution from a stronger theoretical claim that we do not make, and we respectfully invite the reviewer to reconsider the assessment in light of the combined evidence.

Reviewer_kPQuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a measured difference between two source datasets for robot BC pretraining still holds when the target adaptation procedure changes. It proposes a paired 2x2 evaluation (two source-data conditions x two adapters) and reports the four absolute success rates plus the interaction. The main case is Meta-World with a GMM policy. Additional experiments include a deterministic MSE vs. Huber control (no effect) and a PyBullet direct-training comparison where Huber reduces the cost of noisy labels.

Critical strengths
  • The experiments are carefully paired and reported transparently.

  • All four absolute success rates are shown, intervals account for both learners and scenes. The point that a data contrast should be read together with absolute performance is reasonable, and the GMM case is a useful information for those who compare datasets with learned-variance BC heads.

Critical weaknesses
  • The main effect seems to come from a poorly working adapter. Under learned scales the GMM policy is quite weak, while a simple MSE policy on the same data gets above 70%. The perturbed labels most likely just help against variance collapse, which is a known issue of learned-variance regression. Once the adapter is fixed, the candidate data make almost no difference at all, and the deterministic control shows no interaction. The PyBullet experiment changes the training loss directly, not the adaptation, and Huber being more tolerant to noisy labels than MSE is expected. So the evidence supports a much narrower statement than the title and abstract suggest.

  • The significance is limited. The motivation is large-scale robot datasets and generalist policies, but the experiments use small state-based MLPs, custom Meta-World subsets and synthetic label noise as the "data choice". The protocol is a standard design, and the theoretical part does not predict the observed outcomes.

  • Minor: the paper is hard to follow because of heavy jargon and constant references to a long appendix, much of which is not connected to the main claim. Two citations show as "?" in the related work.

Questions for authors
  • How does the interaction look with a standard fix for learned-variance regression (e.g. beta-NLL), rather than fixing the scale to the same value as the noise std?

  • For the MSE policies, reach seems much harder than push, which is the opposite of the GMM results and of what one usually sees in Meta-World. Is there an explanation, or could something be off in the evaluation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kPQu

We thank the reviewer for the constructive comments. We respectfully disagree that our findings can be reduced to a poorly performing GMM adapter or a known variance-estimation issue. Our experiments reveal a more fundamental phenomenon: improvements in adaptation performance do not necessarily preserve the relative value of training data.

1. A Distinct Finding Beyond Variance Stabilization

Our central discovery is that improving an adaptation operator can fundamentally change the measured advantage of different source datasets. In Table 1, fixing adaptation scales improves clean-data success by 23.91 percentage points, while reversing the perturbed-minus-clean contrast from +5.33 to −0.66 points (interaction: −5.99, 95% CI [−8.94, −3.26]). Importantly, this reveals a distinction between absolute adaptation quality and relative data utility: a dataset that appears advantageous under one adaptation procedure may lose that advantage under another, even when overall task performance improves substantially. This finding is further supported by our mechanistic experiments. Table 2 examines three stabilization strategies—scale flooring, log-scale regularization, and detached scale-head gradients—while Table 3 separates responsibility and component-scale interventions. These results show that changes in data comparisons cannot be inferred solely from improvements in optimization stability or absolute task success. Thus, the contribution is not another variance-stabilization technique, but the identification and controlled characterization of adaptation-dependent data valuation.

2. Significance and Generality

A particularly important finding is that data–adaptation interactions are mechanism-dependent rather than universal.

The deterministic matched-group control produces a near-zero interaction (−0.27 points, 95% CI [−0.96, 0.36]), demonstrating that changing the adaptation procedure does not automatically alter data comparisons. In contrast, the GMM experiments exhibit a substantial interaction associated with the adaptation mechanism. This contrast provides evidence that the phenomenon depends on how the learning procedure processes and weights training examples. Furthermore, the independent PyBullet experiment demonstrates a complementary effect: changing the learning objective from MSE to Huber reduces the noisy-data penalty by 12.19 percentage points [6.64, 18.59], while maintaining comparable clean-data performance. Together, these experiments establish that data quality is not an intrinsic property of a dataset alone, but is partly determined by its interaction with the learning and adaptation procedure.

3. Task-Specific Performance Differences

MSE versus GMM task difficulty. The two policy families optimize fundamentally different objectives. Deterministic MSE regression learns a single action prediction, whereas GMM optimization jointly estimates mixture responsibilities, component means, and uncertainty scales. Consequently, their optimization landscapes, action distributions, and task-specific performance rankings need not coincide. In particular, multimodal action modeling can benefit tasks with ambiguous action choices, whereas deterministic regression may favor tasks with more concentrated action targets. The relevant comparison is therefore the paired data contrast within each policy and adaptation setting, rather than absolute success rankings across different policy families.

Summary

Our key contribution is the discovery and systematic characterization of a distinction between adaptation performance and measured data utility. The significant GMM interaction, mechanism-specific interventions, deterministic negative control, and independent PyBullet results collectively establish that the relative value of training data can depend critically on the learning operator.

This insight has direct implications for robot-data evaluation: data rankings obtained under one adaptation procedure should not automatically be assumed to remain valid under another. We believe these findings provide a meaningful methodological contribution and encourage reconsideration of the assessment.

Thanks for the response. My main concerns remain. The rebuttal says fixing the scale "reverses" the contrast, but the paper itself says the fixed-scale interval spans zero. More importantly, under fixed scale the base-only, clean and perturbed branches are all around 56%, so the data basically don't matter once the adapter works. This is still consistent with noisy labels just helping against variance collapse. The MSE control is null and PyBullet is not about adaptation, so the only positive case is the GMM one with weak adapter. Without a more realistic data choice or a standard fix like beta-NLL, I keep my score.

Reviewer_aEQGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the problem of choosing which demonstration data to collect or reuse in robotics. The paper considers a setting where there is a pre-trained policy, a fine-tuning procedure, and an evaluation. The central contribution of the paper is a way to disentangle data quality from its interactions with the specific adapter being used, and the paper provides evidence that ordinary ways of measuring data quality can conflate these two factors.

Critical strengths
  • The paper studies an interesting topic and motivates the problem well, and also connects to prior work on data attribution for robotics.
  • The experiments are thorough, and results are tested against several different reasonable environments and in several conditions.
Critical weaknesses
  • Presentation: The paper is quite hard to follow; I had trouble keeping track of both the claims and corresponding supportive evidence.

  • Significance and scale: The finding is demonstrated at relatively small scale. Further, if the goal was to look at different interactions, the comparison should have also added a condition with the frozen pretrained model swapped out to see all the interactions between data, adapters, and base models.

  • Novelty: Is there a reason why separating data comparisons from evaluation conditions is unique to this setting with robots / adapters and target demonstration data? As noted in L41, works on temporal influence already track how training data effects depend on the learning trajectory. How does this work add additional insight or exapdn on this? It just seems like a robotics-specific rearticulation of this observation with some small, exploratory experiments.

  • Completeness: It would be nice if the paper could say something more informative of the form “if x adaptation procedure is used in a specific type of task, the optimal demonstration data composition is y”. But right now the paper ends at observing that there is a difference. Specifically, the paper promises “an empirical demonstration that target adaptation can change a measured robot-data comparison, together with controls that make that change interpretable (L82-83), but the second part is about interpretability and predictive diagnostics are not fully carried out in the paper.

Questions for authors
  • Missing citations (L95)
  • What is a “fixed update budget” (L131)? Is it the size of the data?
  • What do the columns in Table 1 mean? I am not sure what a “macro mean” or “window” are supposed to mean / what the mins/maxes / expected values are. Is higher or lower better?
  • L46: “Changing an adapter can alter the optimization of both source policies, even though their checkpoints and target demonstrations are unchanged.”  → is there a general explanation for how / in what direction the optimization is altered?
1 comment
AuthorsOpenReview ↗

Response to Reviewer aEQG

We thank the reviewer for the constructive feedback. We clarify the methodological contribution, experimental scope, and practical implications below.

1. Novelty and Significance

Our contribution goes beyond observing that data effects depend on the learning trajectory. While temporal influence methods characterize training-data effects during optimization, our work addresses a distinct downstream question: does the measured advantage of one robot dataset persist when the target-adaptation operator changes?

We introduce a paired four-cell evaluation that separates source-data effects, adaptation performance, and their interaction. Importantly, Eq. (2) provides a first-order cancellation condition explaining when an operator change should alter the data comparison. The results demonstrate that this distinction is consequential. In Meta-World, fixing adaptation scales changes the perturbed-minus-clean contrast from +5.33 to −0.66 percentage points, with a significant interaction of −5.99 points [−8.94, −3.26], while improving clean-data success by 23.91 points. This is not merely a restatement of trajectory-dependent influence; it provides a controlled evaluation protocol and mechanistic analysis for adaptation-sensitive data comparisons.

2. Experimental Scale and Generalization

We acknowledge that our experiments use relatively compact policies rather than large pretrained robot foundation models. However, the findings are supported by multiple controlled settings rather than a single exploratory observation. Our evaluation includes Meta-World and PyBullet, Gaussian-mixture and deterministic policies, alternative adaptation objectives, additional target tasks, and independently generated noise perturbations. In PyBullet, replacing MSE with Huber reduces the noisy-data penalty by 12.19 percentage points [6.64, 18.59], demonstrating adaptation-sensitive data effects in a separate simulator and learning setting.

Our current design intentionally fixes source initializations to isolate data–adapter interactions; introducing different base models would answer a complementary question rather than invalidate the existing controlled comparison.

3. Practical Implications and Predictive Diagnostics

Our results provide a concrete practical implication: data quality should be evaluated under the adaptation procedure actually intended for deployment.

For example, the apparent advantage of perturbed demonstrations under learned-scale adaptation disappears when adaptation scales are fixed, despite a substantial improvement in absolute success. Thus, selecting data using one adaptation configuration can produce a misleading comparison for another.

Furthermore, Table 3 shows that reducing a data contrast does not necessarily imply improved learning: changing mixture responsibilities can reduce the contrast while degrading performance, whereas fixing component scales improves performance.

We agree that deriving a universal optimal data composition for each adaptation procedure would require broader experiments. Our paper establishes a diagnostic protocol rather than claiming a universally optimal data-selection rule. We will make this distinction clearer and more prominently discuss the limitations of prospective prediction.

4. Presentation and Technical Clarifications

We appreciate the suggestions.

  • Fixed update budget: A fixed number of optimization steps, not a fixed dataset size. Source acquisition uses 100 continuation updates, while target adaptation uses 250 updates.
  • Table 1: Each column reports task success (%); Macro is the unweighted mean across the four target tasks. Higher is better. The main comparison is the perturbed-minus-clean difference within each adaptation operator.
  • Optimization mechanism: Changing the adaptation operator modifies gradient weighting, mixture responsibilities, and optimization trajectories. Eq. (2) explains why the resulting performance changes can differ between source policies.

Summary

The central contribution is a controlled framework for determining whether a measured data advantage remains valid under changes in downstream adaptation. The paired experiments, mechanistic interventions, and independent simulator results provide evidence beyond a purely descriptive observation.

We hope these clarifications address the reviewer's concerns and encourage reconsideration of the assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.