- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides diagnostics for understanding how training-data variation affects robot policies. Using empirical NTKs, factor-sensitivity ratios, signal-to-noise ratios, effective rank, and CKA, it characterizes policies as memorizing, adapting to relevant variation, or ignoring irrelevant variation. The diagnostics are conducted across a toy visual task, ManiSkill RL, ACT imitation learning, VLA policies, and a small hardware lighting experiment.
Critical strengths
- The paper looks beyond task success rates and asks what mechanisms might explain why data variation improves or fails to improve robustness. The use of the NTK has a clear local interpretation in terms of how an update on one probe affects another.
Critical weaknesses
- Memorize, adapt, and ignore are operational interpretations of kernel structure and ratios, but the paper does not establish that these are unique or causal learning mechanisms. The measures can depend on the chosen probes, factor parameterization, checkpoint, and representation. Strong correlations do not by themselves validate the mechanistic interpretation.
- The practical predictive claim is supported by a limited set of tasks and factors. The VLA model-selection example covers three models under lighting perturbations, while the hardware study has one task and ten trials per condition. It is unclear how reliably the metrics generalize to new tasks, nuisance factors, or probe distributions.
Questions for authors
- How stable are the proposed regime labels and FSR/SNR trends across training seeds, probe sets, and alternative factor parameterizations?
- Can the authors validate the diagnostic on additional tasks or factors where probe construction is not closely matched to the evaluation distribution?
- What is the real-world trial protocol and uncertainty for the hardware lighting result, and do the diagnostics predict performance on conditions not used to construct probes?
3 comments
Q2 (probe construction different from evaluation distribution) Our probes are defined by the factor of interest alone, and in three of the paper's experiments they are drawn from neither the training nor the evaluation distribution; we will make this explicit in the revision.
- Toy CNN (Fig. 1). Policies are trained on discrete subsets of a 27-color palette, while OOD evaluation uses continuous color randomization together with randomized object scale, background and surface texture. The SNR probes are 5 fixed palette colors at 3 states, i.e. a different color set from training and a different distribution from evaluation. Across all 80 runs (8 values of N_color x 10 subsets), Spearman correlation between SNR and OOD success is 0.92.
- Hardware ACT (Fig. 6, Table 3). Probes are synthetic brightness/contrast/gamma edits of validation frames. Evaluation is physical lamp changes on the real robot. The SNR gap (2.5 vs 5.5) tracks the measured success gap (1/10 vs 8/10 bright, 1/10 vs 6/10 dark).
- VLA model ranking (Fig. 3). Probes are RGB-multiplier edits of LIBERO frames, while the validation target is LIBERO-plus's Light suite, whose perturbations are applied inside the simulator by the benchmark authors.
Q3 (hardware protocol and uncertainty) Probe construction. Probes are built from training-episode frames by applying synthetic lighting changes (brightness, contrast, and gamma; Fig. 6b) with standard image processing. Probe construction therefore uses only held-out validation data and does not involve the test lighting.
Real-world evaluation. We evaluate an ACT cube pick-and-place policy under three lighting conditions: Base (matching the training distribution), Bright (OOD), and Dark (OOD). The OOD conditions are produced by changing the physical lights in the lab (Fig. 7). Each condition is evaluated over 10 trials, comparing a baseline policy against a policy trained with Lightness DR. These real lighting conditions are unseen in both the training data and the probe construction, so the diagnostic is tested on conditions it was not built from. The training SNR rises from 2.5 to 5.5 with Lightness DR (Fig. 6a), consistent with the real-world gains: OOD success improves from 1/10 to 8/10 (Bright) and from 1/10 to 6/10 (Dark), and in-distribution success improves from 6/10 to 10/10 (Table 3).
Uncertainty. With only 10 trials per condition, the uncertainty in the success rates is substantial. We therefore view this study as a consistency check on one comparison between two policies, not as statistical validation of the diagnostic’s predictive power. We will state this explicitly in the limitations section.
Thank you for the response. I maintain my score.
We thank the reviewer for the insightful feedback.
W1 (mechanistic interpretation / causality) We agree with the reviewer that “memorize / adapt / ignore” are operational descriptions of kernel structure, not proven unique or causal mechanisms, and that the term “mechanism” can overstate this. Our goal is a diagnostic: a way to tell, without rollouts, how a policy treats a factor of variation. For this purpose, a reliable association between the kernel statistics and policy behavior is what matters, and we do not claim that the regimes cause robustness.
Our revised introduction defines each regime in terms of what the kernel shows: “memorize” means different values of a factor form separate clusters, “adapt” means they share a common structure, and “ignore” means the representation is relatively insensitive to that factor. We will state in the limitations that the statistics depend on the probe design, factor parameterization, checkpoint, and representation.
We also note two points that support the diagnostic use. First, the training data variation is controlled (e.g., the N_color sweep, confounded vs. independent variation, and the step-matched control in Fig. 4b), so changes in the statistics follow from a deliberate change in the data. Second, Appendix A.2 shows that eNTK similarity predicts how a gradient step on one input changes the output on another (ρ = 0.97 ± 0.06), whereas activation similarity does not (best layer ρ = 0.22 ± 0.28). This gives the kernel a direct, local interpretation in terms of training transfer.
W2 and Q1 (scope and stability of the predictive claims) We thank the reviewer for this fair comment. We agree that the evidence for the predictive use of these metrics is limited. In particular, the VLA model-selection example covers only three models, and the hardware study covers one task with ten trials per condition. We view these as supporting evidence rather than conclusive validation.
Our goal is to introduce the “memorize, adapt, ignore” framework as an initial step and to draw attention to an underexplored connection between representation-kernel metrics and the learning mechanisms of robot policies. Establishing how reliably the metrics generalize across all tasks, nuisance factors, and probe distributions would require a more extensive study than this paper provides.
To address stability (Q1), we report the following:
Seeds. Every ACT result (Tables 1 and 2b) is a 3-seed mean, and the SNR orderings hold in every seed; we will add the standard deviations. The toy sweep uses 10 independent color subsets per N_color, each with its own seed, and the SNR trend and its Spearman correlation with OOD success (ρ = 0.92 over 80 runs) are computed across all of them. The RL, RoboTwin, and hardware policies are single runs, and we now state this explicitly.
Probe sets. Fig. 3b already reports mean ± std over 5 disjoint probe groups. On a second ACT probe grid and on separately rendered bimanual probes, the metric magnitudes shift but every ordering is preserved.
Factor parameterization. Our framework does not fix how a factor is parameterized. A probe set is defined only by the factor of interest, with all other inputs held fixed, and it can be drawn from the training distribution, from the OOD evaluation distribution, or from outside both. For example, the hardware probes are synthetic lightness perturbations of training frames, while the evaluation uses real lighting changes. A formal, structured treatment of how to specify and choose probe parameterizations is beyond the scope of this paper, and we leave it to future work.
Beyond this, the experiments span a wide range of settings: a toy illustration, RL with CNN policies, imitation learning with transformer visuomotor policies, VLA applications (model selection and shortcut detection), and hardware validation. Across these, the kernel metrics consistently distinguish different learning behaviors. We believe this breadth supports our core claim that kernel metrics are a useful diagnostic for robot learning, even though each individual setting has limited coverage.
Following the reviewer’s feedback, we have rewritten the introduction to clarify the motivation and scope. We have also added these concerns about limited coverage to the limitations section.