Linear CKA Conflates Class-Mean and Within-Class Agreement: An Exact Covariance Decomposition on Labeled Benchmarks

Linear centered kernel alignment (CKA) is a widely used measure of similarity between neural network representations, and it is commonly evaluated on labeled benchmarks. Its first step, global centering, removes the dataset-wide mean but not the offset of each class mean from it. A high CKA score can come from two networks placing the same classes in the same regions of feature space, even when they arrange the images inside each class in unrelated ways. Splitting each representation into orthogonal between-class and within-class components decomposes the cross-covariance matrix in the CKA numerator, and the two self-covariance matrices in its denominator, exactly into a between-class and a within-class term, whose squared Frobenius norms combine with a signed interference term. A constructive example makes ordinary CKA arbitrarily close to 1 while within-class CKA remains exactly 0, so a high score alone does not establish within-class agreement. Across four backbones and four labeled datasets, the final-depth gap between ordinary and within-class CKA is positive in all 24 (model pair, dataset) combinations, ranging from 0.022 to 0.273, with a positive interference term throughout. After projecting both representations to a common 64 dimensions, the gap exceeds every value of a 100-permutation label-shuffling null, at the floor empirical $p=1/101$ and Benjamini-Hochberg adjusted $q=0.0099$, while a random balanced partition of matching granularity reduces it to near zero. The overlap between the class subspace and each backbone's dominant principal-component subspace exceeds a random-subspace null by factors of six to five hundred. On ImageNet-1K, replacing ordinary CKA with within-class CKA moves the global maximum of the cross-depth heatmap for four of six model pairs, so class-mean structure can change the layer correspondence such a heatmap reports.