- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Independent-Replica CKA (IR-CKA) to recover clean neural representation similarity from repeated noisy observations. It shows that sampling-debiased CKA can remain biased under observation noise, particularly through inflated self-normalization factors. IR-CKA addresses this by pairing conditionally independent replicas within all three minibatch HSIC₁ factors. The paper characterizes the residual biases of alternative estimators, proves unbiasedness of the IR factors, and establishes concentration and error guarantees for a PSD-stabilized variant. Experiments on frozen ViT and ResNet representations demonstrate improved recovery of clean CKA matrices and bidirectional layer relationships under independent feature dropout, particularly at moderate noise levels.
Critical strengths
-
The paper clearly distinguishes sampling bias from observation-noise bias and explains why seemingly reasonable alternatives, including feature averaging and cross-Gram constructions, retain residual bias. The proposed replica pairing eliminates these biases in all three HSIC₁ factors under the stated assumptions. The supplementary proofs provide a coherent theoretical foundation, including random minibatch partitioning and normalized estimation error. This makes the central methodological claim mathematically convincing.
-
The experiments use matched observations and partitions, including baselines with the same number of independent replicas. Improvements are evaluated through similarity-matrix recovery, bidirectional layer retrieval, and factor-level diagnostics, providing evidence beyond a single aggregate metric. The explicit reporting of failures under strong corruption further strengthens the evaluation by distinguishing numerical stabilization from accurate similarity recovery.
Critical weaknesses
-
The main recovery experiments apply synthetic, mean-preserving feature dropout to representations extracted from frozen deterministic networks, where clean features are already available. Although this controlled setting appropriately validates the theory, the paper does not demonstrate a realistic application in which clean representations are inaccessible but independent, approximately mean-preserving noisy observations are available.
-
Independent measurements have already been used to remove noise-induced bias in representational geometry and correlation estimation. While the paper provides a careful analysis of replica placement within minibatch HSIC₁ and establishes estimator-specific guarantees, the broader conceptual advance over existing crossvalidated and noise-corrected approaches remains somewhat incremental. The paper could more clearly establish why this particular construction represents a substantive methodological contribution beyond adapting established independent-measurement correction principles to a specific CKA estimator.
Questions for authors
-
Can the authors identify a concrete representation-analysis setting where clean features are genuinely unavailable but independent, approximately mean-preserving noisy replicas can be obtained? How well do the assumptions underlying IR-CKA reflect the observation noise encountered in that setting?
-
What are the essential methodological differences between IR-CKA and existing crossvalidated representation-similarity or noise-corrected correlation estimators (particularly Diedrichsen et al. 2021 and Pospisil and Bair 2022)? Which properties or guarantees are genuinely specific to the proposed construction rather than consequences of established independent-measurement correction principles?
-
At , all 304 projected factor matrices yield IR-PSD scores with absolute values above 0.999. How sensitive are the reported strong-noise recovery and ranking results to the ridge magnitude? Could a larger or scale-aware ridge meaningfully reduce this saturation while preserving clean-score recovery accuracy?
Diedrichsen, J., Berlot, E., Mur, M., Schütt, H. H., Shahbazi, M., & Kriegeskorte, N. (2021). Comparing representational geometries using whitened unbiased-distance-matrix similarity. Neurons, Behavior, Data Analysis, and Theory, 5(3), 1–31. https://doi.org/10.51628/001c.27664
Pospisil, D. A., & Bair, W. (2022). Accounting for bias in the estimation of r² between two sets of noisy neural responses. Journal of Neuroscience, 42(50), 9343–9355. https://doi.org/10.1523/JNEUROSCI.0198-22.2022