acceptodds
Under review as a conference paper at ICLR 2027

Recovering Clean Representation Similarity with Independent Replicas

Abstract

Sampling debiasing does not necessarily remove the effect of observation noise on centered kernel alignment (CKA). Even when noise leaves the cross-model HSIC numerator unbiased, it can inflate the two self factors and distort both similarity scores and layer rankings. We introduce Independent-Replica CKA (IR-CKA), which pairs Gram matrices from two conditionally independent observations inside each minibatch factor and averages the factors before normalization. Under the stated mean-preserving noise assumptions, all three factors are unbiased for their clean finite-pool counterparts. Comparisons with feature averaging and -replica cross-Grams identify the corrections retained when independence is used elsewhere in the construction. We establish concentration under random partitions of a fixed input pool and mean-square error for PSD-stabilized IR with ridge , under moment and positive self-factor margin conditions. Experiments with frozen ViT and ResNet representations show that IR and its variants recover clean similarity matrices and bidirectional layer relationships more accurately than shared-replica, feature-averaging, and cross-Gram baselines under moderate feature dropout. Under stronger corruption, PSD stabilization restores complete estimates and improves several retrieval measures, although accurate recovery is not uniform across layers or metrics. These results demonstrate the practical value of placing replica independence within all three HSIC factors for recovering clean representation similarity from noisy observations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Deep Representation, Representation Similarity, Centered Kernel Alignment, Unbiased HSIC estimator

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2mv1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Independent-Replica CKA (IR-CKA) to recover clean neural representation similarity from repeated noisy observations. It shows that sampling-debiased CKA can remain biased under observation noise, particularly through inflated self-normalization factors. IR-CKA addresses this by pairing conditionally independent replicas within all three minibatch HSIC₁ factors. The paper characterizes the residual biases of alternative estimators, proves unbiasedness of the IR factors, and establishes concentration and error guarantees for a PSD-stabilized variant. Experiments on frozen ViT and ResNet representations demonstrate improved recovery of clean CKA matrices and bidirectional layer relationships under independent feature dropout, particularly at moderate noise levels.

Critical strengths
  1. The paper clearly distinguishes sampling bias from observation-noise bias and explains why seemingly reasonable alternatives, including feature averaging and cross-Gram constructions, retain residual bias. The proposed replica pairing eliminates these biases in all three HSIC₁ factors under the stated assumptions. The supplementary proofs provide a coherent theoretical foundation, including random minibatch partitioning and normalized estimation error. This makes the central methodological claim mathematically convincing.

  2. The experiments use matched observations and partitions, including baselines with the same number of independent replicas. Improvements are evaluated through similarity-matrix recovery, bidirectional layer retrieval, and factor-level diagnostics, providing evidence beyond a single aggregate metric. The explicit reporting of failures under strong corruption further strengthens the evaluation by distinguishing numerical stabilization from accurate similarity recovery.

Critical weaknesses
  1. The main recovery experiments apply synthetic, mean-preserving feature dropout to representations extracted from frozen deterministic networks, where clean features are already available. Although this controlled setting appropriately validates the theory, the paper does not demonstrate a realistic application in which clean representations are inaccessible but independent, approximately mean-preserving noisy observations are available.

  2. Independent measurements have already been used to remove noise-induced bias in representational geometry and correlation estimation. While the paper provides a careful analysis of replica placement within minibatch HSIC₁ and establishes estimator-specific guarantees, the broader conceptual advance over existing crossvalidated and noise-corrected approaches remains somewhat incremental. The paper could more clearly establish why this particular construction represents a substantive methodological contribution beyond adapting established independent-measurement correction principles to a specific CKA estimator.

Questions for authors
  1. Can the authors identify a concrete representation-analysis setting where clean features are genuinely unavailable but independent, approximately mean-preserving noisy replicas can be obtained? How well do the assumptions underlying IR-CKA reflect the observation noise encountered in that setting?

  2. What are the essential methodological differences between IR-CKA and existing crossvalidated representation-similarity or noise-corrected correlation estimators (particularly Diedrichsen et al. 2021 and Pospisil and Bair 2022)? Which properties or guarantees are genuinely specific to the proposed construction rather than consequences of established independent-measurement correction principles?

  3. At p=0.7p=0.7, all 304 projected factor matrices yield IR-PSD scores with absolute values above 0.999. How sensitive are the reported strong-noise recovery and ranking results to the ridge magnitude? Could a larger or scale-aware ridge meaningfully reduce this saturation while preserving clean-score recovery accuracy?

Diedrichsen, J., Berlot, E., Mur, M., Schütt, H. H., Shahbazi, M., & Kriegeskorte, N. (2021). Comparing representational geometries using whitened unbiased-distance-matrix similarity. Neurons, Behavior, Data Analysis, and Theory, 5(3), 1–31. https://doi.org/10.51628/001c.27664

Pospisil, D. A., & Bair, W. (2022). Accounting for bias in the estimation of r² between two sets of noisy neural responses. Journal of Neuroscience, 42(50), 9343–9355. https://doi.org/10.1523/JNEUROSCI.0198-22.2022

Reviewer_LbyMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies linear CKA between two networks when features are observed with noise. It proposes IR-CKA, which pairs Gram matrices from two independent noisy replicas inside each minibatch HSIC1HSIC_1 factor. It derives factor means and concentration bounds. IR-CKA is used with some ViT-ResNet pairs all pretrained on ImageNet and evaluated only on CIFAR-10 under synthetic feature dropout.

Critical strengths

Noise-induced bias in representation similarity is a legitimate problem.

Critical weaknesses
  1. The paper is really hard to read, and once you dig the idea out, it fits in one sentence: pair two independent noisy copies inside each HSIC1HSIC_1 term. That’s Spearman’s correction for attenuation, already adapted for analyze learned representations (Walther et al., 2016; Diedrichsen et al., 2021; Pospisil and Bair, 2022), so the novelty is thin.

  2. The only setting that is used to verify the assumptions is synthetic dropout applied after feature extraction, and the authors themselves exclude augmentation. There’s no real case where independent replicas come up naturally, and no link to where CKA actually matters today, such as the convergence debate (Huh et al., 2024; Groger et al., ICML 2026).

  3. All IR results come from one pair of supervised ImageNet backbones on CIFAR-10, compared only against variants the authors built. There’s no comparison with existing corrected CKA estimators (Murphy et al., 2024; Chun et al., 2025), and the method breaks at strong noise, with about a quarter of the scores undefined.

Questions for authors
  1. Where do two independent, mean-preserving noisy copies of the same features show up in practice? Can you show one real case where the corrected CKA is used in practice and changes a conclusion?

  2. Does IR stay defined and accurate for encoders with different objectives and pretraining data (e.g., DINOv2, CLIP image vs text), and on in-distribution data like ImageNet val?

Reviewer_LRLhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IR-CKA that uses two independent noisy observations to recover clean representation similarity by removing noise-induced bias from all three HSIC factors.

Critical strengths

IR-CKA shows a nice correction using two independent replicas.

Critical weaknesses

The theoretical guarantees rely on the positive self-factor margin assumption. This assumption could be difficult to satisfy for weak-signal representations. The paper does not empirically verify this assumption across the evaluated layers. In addition, no discussion about how the guarantees behave when the margin becomes small.

Questions for authors

Could the authors evaluate IR-CKA on LLM representations to investigate its generalization beyond vision models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.