acceptodds
Under review as a conference paper at ICLR 2027

When Do Unseen Data Concentrate at Class-Specific Points? Kernel Data Piling of High-Dimensional Small-Sample-Size Data

Abstract

We consider classification problems in which observations are represented by high-dimensional feature vectors while only a small, fixed number of labeled training samples are available from each class. For this purpose, we work under a high-dimension-low-sample-size (HDLSS) asymptotic regime, where the feature dimension tends to infinity while the training sample sizes remain fixed. We study when independent test observations, after being mapped by a kernel classifier fitted to these labeled samples, concentrate at class-specific locations. We identify conditions under which this class-specific test concentration occurs and characterize when the resulting class locations are distinct for several commonly used kernels. We further show that components with very large within-class variation can prevent test-score concentration throughout the empirical span generated by the training sample. To address this failure, we propose a kernel correction that restores separated class-specific concentration under suitable conditions. Simulation studies provide empirical support for the theoretical results developed throughout the paper.

Primary area:
learning theory
Keywords:
Few-shot learning, Small-sample learning, Kernel method, High dimension low sample size, Data piling, Spiked covariance, Reproducing kernel Hilbert space

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2wotOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when unseen test observations concentrate at class-specific locations in high-dimensional, low-sample-size classification. It introduces KMDP, extending classical data piling to reproducing kernel Hilbert spaces, and establishes theoretical conditions for test-data concentration and class separation. The paper also identifies a critical-spike regime where test piling fails and proposes a feature correction to restore it. Simulation results support the main theoretical findings.

Critical strengths

1.The paper addresses an interesting theoretical question: whether classwise concentration observed in training data can extend to independent test observations. 2.Extending maximal data piling to nonlinear kernel spaces is a meaningful contribution, particularly the characterization of kernel-dependent class separation. 3.The theoretical analysis is carefully developed. The impossibility result for critical spikes in the Gaussian kernel span and the subsequent recovery guarantee are particularly interesting. 4.The paper is generally well organized, and the simulations provide consistent support for the theoretical results.

Critical weaknesses

1.The connection to practical few-shot learning remains somewhat limited. Although the paper motivates its analysis through frozen representations and few-shot classification, the experiments rely entirely on synthetic data. It is unclear how closely real pretrained representations satisfy the theoretical assumptions. 2.The feature correction relies on relatively strong assumptions. In particular, it requires consistent estimation of the dominant spike subspace, potentially using additional unlabeled data. Its practical effectiveness may depend on how well this subspace can be identified without removing discriminative information. 3.The scope of some theoretical results is relatively specific. The critical-spike impossibility result is established for the empirical Gaussian kernel span. A clearer discussion of its implications for other kernels would help readers understand the generality of the findings.

Questions for authors
  1. Could the authors provide more intuition for why the Gaussian kernel can separate test piles based solely on residual variance differences, while the linear and polynomial kernels cannot?
  2. Could the authors include a small experiment using real pretrained features, or at least discuss how the HDLSS and spiked-covariance assumptions might be assessed in practical few-shot settings?
  3. How should the correction rank be selected in practice? Appendix B shows that rank overestimation can remove useful class-discriminative information, particularly under transductive correction. Could the authors provide some practical guidance?
  4. To what extent might the critical-spike obstruction extend beyond Gaussian kernels? Even a brief discussion of the expected behavior for other kernel families would be helpful.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 2wot

We thank the reviewer for the favorable evaluation of our work. We address Weakness 1 and Question 2 in General Responses 2 and 3, which present real frozen-feature experiments and examine the model assumptions and test-score geometry.

We agree with the concern in Weakness 2 that spike-subspace estimation and preservation of discriminative information pose important practical challenges. General Response 4 empirically examines these limitations and addresses Question 3 concerning correction-rank selection and sensitivity.

We address Weakness 3 and Question 4 in General Response 5, which extends the critical-spike obstruction to a broader family of kernels.

We respond to Question 1 individually below.

Question 1

Could the authors provide more intuition for why the Gaussian kernel can separate test piles based solely on residual variance differences, while the linear and polynomial kernels cannot?

Consider two classes with the same mean but different residual variances. Under the weak-spike conditions of Theorem 1, each class forms a cloud with a different radius. The classes therefore differ in their spread, even though their centers coincide.

The linear kernel compares observations through their inner product. In high dimensions, independent residual vectors become nearly orthogonal [1], so their contribution to the normalized inner product vanishes. The limiting inner product therefore reflects the common mean but not the different cloud sizes. Since residual variance information vanishes from the inner product in the high-dimensional limit, applying a polynomial transformation cannot recover it. Consequently, the linear and polynomial kernels fail to separate test piles when classes differ only in residual variance.

The Gaussian kernel instead compares distances. Even when two residual vectors are nearly orthogonal, the distance between them still depends on their lengths. Thus, the difference between classes forming clouds of different sizes remains visible in the Gaussian kernel similarities. Importantly, the exponential dependence on squared distance converts these differences into distinct test-pile locations, as established by the separation criterion in Theorem 1. This mechanism also extends to other strictly convex radial kernel profiles, such as rational quadratic and Laplacian kernels, under analogous regularity conditions. The same intuition applies after consistent spike correction under Theorem 3.

[1] Hall et al. Geometric Representation of High Dimension, Low Sample Size Data. Journal of the Royal Statistical Society: Series B, 67(3):427–444, 2005.

Reviewer_CGVTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Kernel Maximal Data Piling (KMDP) within the HDLSS (High-Dimension, Low-Sample-Size) / few-shot learning framework. The method imposes a constraint within a Reproducing Kernel Hilbert Space (RKHS) such that the training scores for each class are identical, while simultaneously maximizing inter-class separation. This yields a unique maximizing subspace and a closed-form expression for the coefficients. The authors demonstrate that, under a "weak spike" condition, independent test scores also exhibit "piling" (concentration at specific values), and they provide a kernel-based criterion for inter-class separation. Under a "critical spike" condition, the empirical span associated with a Gaussian kernel possesses a non-vanishing lower bound on its variance; however, "piling" can be restored by estimating and subsequently removing the spike subspace. The approach is validated primarily through simulated few-shot experiments involving three to five classes, demonstrating the phenomena of training-set piling, the presence (or absence) of piling in the test set, and the restoration of piling following the proposed correction.

Critical strengths
  1. The work extends the multiclass MDP framework to a Reproducing Kernel Hilbert Space (RKHS), establishing the uniqueness and a closed-form representation for the KMDP subspace; this provides a clear theoretical framework that unifies linear kernels with general kernels.

  2. It explicitly distinguishes between "weak" and "critical" spike regimes and offers distinct criteria, tailored to Gaussian versus linear/polynomial kernels, for assessing the separation of "piling" effects in test data, demonstrating significant insight.

  3. The theorems and simulation results concerning "spike-subspace correction" effectively substantiate the intuition that one should remove dominant variations before constructing a kernel; the experimental design aligns well with the theoretical conclusions.

Critical weaknesses
  1. The theoretical contribution is relatively incremental compared to existing MDP/DDP frameworks: the linear kernel case is already closely related to DDP, while KMDP primarily introduces kernelization and multi-class extensions. Furthermore, the obstruction identified for the critical spike is restricted to the empirical span under a Gaussian assumption, limiting the scope of the findings.

  2. The assumptions are quite strong and difficult to satisfy in practical few-shot learning scenarios: these include a fixed number of training samples, a spiked covariance structure, diffuse residuals, restricted kernel forms, and, in the Gaussian case, a requirement for positive factor density. Additionally, the proposed correction necessitates prior knowledge of the rank rr or a consistent estimate of the projection operator, casting doubt on the feasibility of the approach when using actual "frozen" features.

  3. Insufficient validation: the evaluation relies primarily on simulations, lacking benchmarks based on real-world few-shot tasks. Theoretically, the analysis focuses on asymptotic behavior as p→∞p \to \infty (the "piling" phenomenon) without providing finite-sample error bounds or guarantees for classification accuracy. Moreover, the practical impact of the non-overlap between training and testing "piles" on downstream classification performance remains inadequately addressed.

Questions for authors
  1. Regarding Theorem 3: Is it necessary to know the true spike rank rr a priori? If the rank is estimated incorrectly, how robust is the correction method in terms of mitigating "piling" and maintaining class separation?

  2. What is the relationship between the paper’s criteria for "piling" and class separation, and the actual classification error rate? Is it possible to provide guarantees on classifier performance based on these separation criteria in the finite-sample regime?

  3. In real-world frozen-feature few-shot learning tasks (beyond simulations), do the assumptions, namely the existence of a "spiked" model, a "diffuse" residual, and a known rank, still hold? If not, how does the method’s performance degrade?

  4. Regarding the transductive correction in Theorem 4: Since the projector is estimated using the same test batch upon which it is subsequently evaluated, even if labels are not explicitly used for fitting, is there a risk of circularity in the evaluation, or is the result sensitive to the specific composition of the batch?

  5. In Corollary 1, the limit n→∞n \to \infty is taken after first taking the limit p→∞p \to \infty (for a fixed nn). What practical implications does this have for few-shot learning scenarios, where both nn and pp are finite?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CGVT (1/2)

We thank the reviewer for the careful assessment and constructive comments. We address Weakness 3 and Question 2 in General Responses 1 and 2, which discuss classification implications and experiments. We address Question 1 in General Response 4, which discusses rank selection for correction, and Question 3 in General Response 3, which examines spiked-model assumptions and test piling in real frozen features. We respond to the remaining weaknesses and questions individually below.

Weakness 1

The theoretical contribution is relatively incremental compared to existing MDP/DDP frameworks: the linear kernel case is already closely related to DDP, while KMDP primarily introduces kernelization and multi-class extensions. Furthermore, the obstruction identified for the critical spike is restricted to the empirical span under a Gaussian assumption, limiting the scope of the findings.

Since the linear kernel is a special case of our framework, its connection to existing DDP theory is natural. However, extending the framework to general kernels reveals new aspects of data piling, as discussed below.

In kernel learning, the choice of kernel determines the function space and covariance structure of learned functions, making it important to select a kernel suited to the data and task. Our results provide kernel-dependent criteria for the concentration and separation of unseen test scores, showing how different kernels can exploit different characteristics of the class distributions, including mean and variance differences. These findings offer a theoretical basis for kernel selection and extend beyond existing linear DDP theory, which characterizes class separation only under mean differences.

The obstruction theorem further distinguishes our results from linear DDP. Under suitable conditions, linear DDP can achieve test piling within the span of the training observations even in the presence of strong spikes. In contrast, our results show that for certain nonlinear kernels, no linear combination of kernel features of train data can eliminate the persistent test-score variation. We also propose a spike-subspace correction to address this additional difficulty arising in the extended kernel setting. We agree that restricting the obstruction theorem to the Gaussian kernel limits its generality. In addressing the reviewers' comments, we found that the result extends to a much broader family of radial kernels, including widely used Gaussian, rational-quadratic, Laplace, and Matérn kernels. We present this extension and its underlying intuition in General Response 5.

Weakness 2

The assumptions are quite strong and difficult to satisfy in practical few-shot learning scenarios: these include a fixed number of training samples, a spiked covariance structure, diffuse residuals, restricted kernel forms, and, in the Gaussian case, a requirement for positive factor density. Additionally, the proposed correction necessitates prior knowledge of the rank or a consistent estimate of the projection operator, casting doubt on the feasibility of the approach when using actual "frozen" features.

We agree that the theoretical assumptions present important challenges for practical applications. We have partially addressed these concerns through additional experiments. General Response 3 examines the covariance structure and test-score geometry of real frozen features, providing empirical evidence while acknowledging that finite-dimensional experiments cannot verify all asymptotic assumptions. General Response 4 addresses the sensitivity of correction-rank to the actual frozen feature, and a data-driven selection rule.

AuthorsOpenReview ↗

Response to Reviewer CGVT (2/2)

Question 4

Regarding the transductive correction in Theorem 4: Since the projector is estimated using the same test batch upon which it is subsequently evaluated, even if labels are not explicitly used for fitting, is there a risk of circularity in the evaluation, or is the result sensitive to the specific composition of the batch?

Using the same unlabeled test batch for fitting and evaluation is standard in transductive learning and does not, by itself, constitute circular reasoning. In this setting, test features are available during fitting, while their labels remain unknown. Theorem 4 goes beyond this general principle by proving that the evaluated test scores concentrate around population-defined limiting centers, rather than centers fitted to the same batch. The proof explicitly accounts for the dependence through projector consistency and a deterministic perturbation bound that does not require independence between the projector and the evaluated observations.

However, this guarantee does depend on batch composition. As discussed in Appendix A, class proportions and between-class variation can change the leading eigenspace of the unlabeled mixture covariance. If the batch selection prevents consistent recovery of the target spike subspace, the theorem's guarantee no longer applies.

Sample splitting provides an alternative that avoids data reuse. When sufficiently many observations are available, one can estimate the spike projector using one subset and evaluate the piling of test score on an independent held-out subset. This ensures independence between the estimated projector and the evaluation observations, allowing a direct assessment of test piling on unseen data. However, splitting reduces the sample size available for projector estimation and does not itself guarantee projector consistency.

Question 5

In Corollary 1, the limit n→∞n \to \infty is taken after first taking the limit p→∞p \to \infty (for a fixed nn). What practical implications does this have for few-shot learning scenarios, where both pp and nn are finite?

The HDLSS limit p→∞p\to\infty with fixed nn provides an asymptotic approximation to the many-feature, few-labeled-sample regime encountered in few-shot learning or biological data analysis. Sequential asymptotics of this kind have also been used in HDLSS theory; for example, Jung et al. [1] take the dimension to infinity and then the sample size to establish consistency of a principal-component-count estimator. This approach allows us to avoid the more involved random matrix analysis associated with jointly increasing pp and nn.

In our case, the purpose of the subsequent n→∞n\to\infty limit is not to study a setting with infinitely many labeled observations, but to characterize how the training and test pile locations change as nn increases within the high-dimensional approximation. Corollary 1 shows that test pile separation is nondecreasing and the training–test displacement vanishes as more labeled observations become available. Thus, its practical implication concerns how additional train data affect the score geometry in the high-dimensional data.

Reference

[1] Jung et al. On the Number of Principal Components in High Dimensions. Biometrika, 105(2):389–402, 2018.

Reviewer_CGVTOpenReview ↗

review

Thank you for your rebuttal. Some of my concerns have been addressed, but others remain unresolved. In particular, regarding the relationship between the assumptions/theorems presented in this paper and their practical application. You cited reference [1]; while I acknowledge that the assumptions made in [1] are reasonable within that “context”, they are not appropriate for the methodology proposed in this paper, especially given the focus on small-sample tasks. It is like studying a low-dimensional problem while assuming infinite dimensionality; just as phenomena like the "curse of dimensionality" do not manifest in low-dimensional spaces, the assumptions used here do not hold in a small-sample setting. Furthermore, many critical issues inherent to small-sample scenarios may be overlooked under these assumptions, a situation that differs significantly from the context of [1].

We thank the reviewer for the additional comments. We would like to clarify two points that may not have been sufficiently clear in our previous response.

First, our reference to Jung et al. (2018) was intended solely as an example of the use of sequential limits in HDLSS asymptotic theory. We did not suggest that the assumptions or methodological setting of that paper are directly applicable to ours. The relevance of the citation is the asymptotic approach itself, rather than the specific problem being studied.

Second, the HDLSS framework, in which d→∞d\to\infty while nn remains fixed, is specifically designed to study high-dimensional problems with small sample sizes. Thus, the small-sample nature of few-shot learning is explicitly retained in the first stage of our analysis. The subsequent limit as n→∞n\to\infty in Corollary 1 serves to characterize how the limiting score geometry evolves as the number of labeled observations increases.

The practical implication is that, in high-dimensional few-shot learning, increasing the number of labeled observations is expected to improve the separation of test score piles while reducing the discrepancy between training and test score locations. Corollary 1 provides theoretical support for this behavior within the HDLSS approximation.

We agree that asymptotic results do not necessarily provide exact guarantees for finite dd and nn, and that some finite-dimensional effects may not be fully captured. However, HDLSS asymptotics are motivated precisely by high-dimensional, small-sample problems, and sequential limits are a mathematical tool for studying particular properties of this regime. Our theoretical results aim to provide insight into the geometric behavior of the proposed method, rather than uniform finite-sample guarantees.

We hope these clarifications address the reviewer's concerns.

Reviewer_uDHHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies kernel score maps in high-dimensional settings with few labeled samples. The feature dimension increases while the number of labeled training samples per class remains fixed. The authors examine whether test scores from the same class concentrate and whether different classes concentrate at distinct locations. They introduce kernel maximal data piling and derive conditions for concentration and separation under several common kernels. At the critical spike scale, within-class variation can prevent concentration. To address this issue, they estimate the high-variance directions and remove the corresponding components before constructing the kernel. They prove that this correction restores within-class test-score concentration when these directions are consistently estimated and the stated conditions hold. The class-specific locations are also distinct if sufficient class differences remain after projection. Simulations support the theoretical predictions and examine sensitivity to sample sizes, correction rank, and kernel bandwidth.

Critical strengths
  1. The paper offers a valuable perspective on how training geometry extends to unseen samples. It distinguishes within-class concentration from between-class separation and clarifies how the kernel affects these properties.

  2. The paper shows that adjusting coefficients within the normalized empirical Gaussian span cannot eliminate persistent test-score variation under the stated conditions. It also establishes when the proposed correction restores concentration. Simulations support these theoretical predictions.

Critical weaknesses

The paper does not fully specify how the score map is converted into class predictions, particularly when training and test centers differ. Without an explicit decision rule and classification-error comparisons before and after correction, the practical classification benefits remain unclear.

Questions for authors

Clarification requests:

1.Is the term “kernel classifier” used accurately in the abstract? The paper appears to focus on a score map and its geometric properties rather than a fully specified classification rule.

2.Some experimental quantities need clearer definitions. How are the analytically computed distances in Figure 2(b) and the projector error in Figure 5(a) calculated?

3.Can the authors provide classification-error results for the existing simulations to demonstrate whether the improved score geometry benefits classification?

Suggestion for future work:

Future work could explore data-driven strategies for selecting which directions to remove and which to retain.

1 comment
AuthorsOpenReview ↗

Response to Reviewer uDHH

We thank the reviewer for the positive assessment of our theoretical contributions and the constructive suggestions. We address the concerns raised in Critical Weaknesses and Question 3 in General Responses 1 and 2, where we clarify the classification implications of our theoretical results, provide explicit decision rules and a finite-sample error bound, and present additional classification experiments. We respond to Questions 1 and 2 individually below.

Question 1

Is the term “kernel classifier” used accurately in the abstract? The paper appears to focus on a score map and its geometric properties rather than a fully specified classification rule.

Thank you for highlighting this ambiguity. Our primary focus is on characterizing when a fitted score map induces class-specific concentration and separation of independent test scores, rather than on proposing a classifier with superior predictive performance. We agree that “kernel score map” more precisely describes the object studied and will replace “kernel classifier” with this term in the abstract.

Question 2

Some experimental quantities need clearer definitions. How are the analytically computed distances in Figure 2(b) and the projector error in Figure 5(a) calculated?

Figure 2(b). We report the minimum pairwise Euclidean distance between the finite-dimensional conditional test-score means, min⁡k<ℓ∥mk−mℓ∥2\min_{k<\ell}\lVert m_k-m_\ell\rVert_2, where mk:=E[g(Xk,0)∣Fp]m_k:=\mathbb{E}[g(X_{k,0})\mid\mathcal{F}_p], as defined in Section 3. The kernel expectations are computed in closed form under the Gaussian data-generating model, using Gaussian quadratic-form integration for the Gaussian kernel and Gaussian moment formulas for the linear and polynomial kernels.

Figure 5(a). We report the operator-norm distance ∥P^−P∥op\lVert\hat P-P\rVert_{\mathrm{op}} between the estimated projector P^=U^U^⊤\hat P=\hat U\hat U^\top, obtained from the leading r=2r=2 eigenvectors of the sample-centered unlabeled covariance matrix, and the population spike-subspace projector P=UU⊤P=UU^\top. This distance is computed as (1−σmin⁡2(U⊤U^))12(1-\sigma_{\min}^2(U^\top\hat U))^{\frac12}, where σmin⁡(U⊤U^)\sigma_{\min}(U^\top\hat U) denotes the smallest singular value of U⊤U^U^\top\hat U.

We apologize for the lack of clarity and will revise the experimental descriptions to make these definitions and calculations explicit.

Suggestion for future work

Future work could explore data-driven strategies for selecting which directions to remove and which to retain.

We agree that developing data-driven strategies for identifying which directions to remove and which to retain is an important direction for future work. In General Response 4, we present a simple support-informed rank-selection strategy and evaluate its effectiveness and limitations. Developing more reliable methods for distinguishing nuisance directions from class-discriminative directions remains essential for the practical application of our theoretical results. We appreciate the reviewer's valuable suggestion.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response 1: Scope and classification

We thank all reviewers for their careful and constructive reviews. Since several comments raise overlapping questions, we have organized our responses to these shared concerns into General Responses.

This response addresses questions related to classification. We clarify the scope of our contribution and provide explicit classification rules and a finite-sample error bound connecting test-score geometry to classification performance.

Scope of the contribution.

Our primary objective is to characterize when a kernel score map fitted from a small, fixed number of labeled observations induces within-class concentration and between-class separation for unseen test data, and when this geometry is obstructed or restored. These properties can support accurate classification, but our main contribution is the theoretical characterization of this phenomenon rather than a state-of-the-art classification method. We clarify this connection through explicit decision rules, an error bound, and classification experiments on simulated and real data.

This perspective is conceptually related to deep representations that support simple final-layer classifiers with few labeled examples. Establishing a formal connection between such representations and our score maps is a direction for future work.

Classification rules.

To obtain class predictions from the KMDP score map, we consider four classification rules. The first assigns each observation to the nearest training-score centroid (Train NC), while the second uses estimated test-score centers (Test NC) to account for the training–test displacement characterized in Theorem 1. The test centers are estimated from training kernel similarities without using test labels. Alternatively, we fit logistic regression (Logistic) or a linear SVM (SVM) to the labeled KMDP training scores and use the fitted classifiers to predict test labels.

Finite-sample error bound for NC.

For either NC rule, let ckc_k denote its fitted class center. For Train NC, ck=gˉkc_k=\bar g_k converges to the training-pile limit tktrt_k^{\mathrm{tr}} in (10) of Theorem 1, whereas for Corrected Test NC, ck=t^kc_k=\hat t_k converges to the test-pile limit tkt_k in (11). As defined in Section 3, let mkm_k and CkC_k be the conditional mean and covariance of class-kk test score, given the fitting information Fp\mathcal F_p. For distinct centers, define the minimum signed distance from mkm_k to the NC decision boundaries: $ h_k(c)=\min_{\ell\ne k} \frac{\lVert m_k-c_\ell\rVert_2^2-\lVert m_k-c_k\rVert_2^2}{2\lVert c_\ell-c_k\rVert_2}.$

Whenever hk(c)>0h_k(c)>0, we obtain $\Pr\!\left\{ \hat y_{\mathrm{NC}}(X_{k,0})\ne k \mid\mathcal F_p \right\} \le \min\!\left\{ 1,\frac{\operatorname{tr}(C_k)}{h_k(c)^2} \right\}$ by applying conditional Markov's inequality to its squared deviation. Thus, misclassification requires the test score to deviate from mkm_k by at least hk(c)h_k(c). It holds for finite dimensions and training sizes whenever the conditional second moments exist.

By Theorems 1 and 3, tr⁡(Ck)→P0\operatorname{tr}(C_k)\xrightarrow{P}0 and mkm_k converges to its limiting test center tkt_k. For Test NC, convergence of the classwise training kernel averages and fitted KMDP coefficients yields consistent center estimates. Thus, pairwise separation of the limiting test centers gives hk(c)→P12min⁡ℓ≠k∥tk−tℓ∥2>0.h_k(c)\xrightarrow{P}\frac12\min_{\ell\ne k}\lVert t_k-t_\ell\rVert_2>0.

For Train NC, the fitted centers converge to the training-pile limits tktrt_k^{\mathrm{tr}}. From (10)–(11), we obtain ∥tk−tℓtr∥22−∥tk−tktr∥22=Bkk+Bℓℓ−2Bkℓ+ηℓnℓ−ηknk.\lVert t_k-t_\ell^{\mathrm{tr}}\rVert_2^2-\lVert t_k-t_k^{\mathrm{tr}}\rVert_2^2=B_{kk}+B_{\ell\ell}-2B_{k\ell}+\frac{\eta_\ell}{n_\ell}-\frac{\eta_k}{n_k}.

Hence, Train NC also has strictly positive limiting margins if Bkk+Bℓℓ−2Bkℓ>∣ηk/nk−ηℓ/nℓ∣B_{kk}+B_{\ell\ell}-2B_{k\ell}>|\eta_k/n_k-\eta_\ell/n_\ell| for every class pair. In particular, when ηk/nk\eta_k/n_k is common across classes, this reduces to the test-pile separation condition in Theorem 1. The same conclusions apply under Theorem 3 with the corresponding corrected quantities.

Therefore, whenever the respective margin conditions hold, hk(c)h_k(c) converges to a positive value while tr⁡(Ck)→P0\operatorname{tr}(C_k)\xrightarrow{P}0. The finite-sample error bound then implies that the conditional classification error converges to zero for both NC rules. Thus, training–test displacement does not necessarily impair classification, and center calibration need not improve every finite-sample result.

AuthorsOpenReview ↗

General response 2: Classification experiments

In this response, we provide classification-error comparisons on both the existing simulation settings and real-world frozen-feature few-shot learning tasks. We compare four KMDP-based classification rules with three downstream classifier heads adopted in few-shot learning methods: the nearest-prototype classifier after centering and normalization (NP head) used in SimpleShot [1]; the logistic-regression head (Logistic head) used in RFS [2]; and the cosine-similarity classifier head (Cosine head) used in Baseline++ [3]

To assess the downstream classification implications of KMDP score geometry, we compare classification procedures using the same input features, without reproducing the original feature-learning pipelines.

Simulations.

We use the manuscript's simulation settings with p=4096p=4096, three classes, five labeled examples per class, and 200 repetitions. Feature correction removes two principal components estimated from M=320M=320 independent unlabeled observations, identically for every method. In the table below, corr. refers to this feature correction. Here, Mean denotes the distinct class means setting, whereas Variance denotes the different residual variances setting.

MethodMeanMean (corr.)VarianceVariance (corr.)
NP head27.35 (0.77)0.00 (0.00)66.68 (0.07)63.41 (0.09)
Logistic head13.93 (0.68)0.00 (0.00)66.78 (0.07)66.67 (0.06)
Cosine head11.06 (0.66)0.00 (0.00)66.64 (0.06)66.70 (0.06)
Logistic10.11 (0.38)0.00 (0.00)65.94 (0.08)66.51 (0.01)
SVM9.94 (0.38)0.00 (0.00)65.95 (0.08)66.63 (0.00)
Train NC10.30 (0.38)0.00 (0.00)66.13 (0.07)66.65 (0.00)
Test NC18.24 (0.65)0.00 (0.00)66.95 (0.10)19.00 (0.26)

Classification error (%) by method and simulation setting, reported as mean (SE); lower is better. "corr." denotes feature correction.

In the mean-difference setting, KMDP with Train NC, Logistic, or SVM performs well even before feature correction. Although test-score centers lie closer together than training piles, estimating those centers does not improve classification here. After the common feature correction, all seven methods have zero observed error in this setting. In contrast, in the variance-difference setting, all seven methods perform near chance before feature correction, with errors ranging from 65.94% to 66.95%. Even after feature correction, the other six methods remain close to chance, whereas KMDP with Test NC substantially reduces its error from 66.95% to 19.00%, outperforming corrected Train NC (66.65%). These results illustrate that feature correction and test-center calibration can play complementary roles in classification when classes differ only in residual variance.

Real frozen feature experiment.

We evaluate object recognition (CIFAR-FS and Caltech-101), texture recognition (DTD), cat/dog breed recognition (Oxford-IIIT Pets), and flower recognition (Flowers102), using the same frozen DINOv2 ViT-B/14 768-dimensional CLS features [4]. For each dataset, all heads are refit on the same last 200 original episodes. Each paired episode contains five classes, five training images, and 15 test images per class.

MethodCIFAR-FSDTDOxford-IIIT PetsFlowers102Caltech-101
NP head4.12 (0.27)12.07 (0.46)1.90 (0.25)0.07 (0.02)0.41 (0.10)
Logistic head3.43 (0.24)10.67 (0.45)1.80 (0.23)0.06 (0.02)0.29 (0.08)
Cosine head3.27 (0.25)10.93 (0.44)1.89 (0.26)0.07 (0.02)0.21 (0.08)
Logistic3.45 (0.25)10.48 (0.43)1.83 (0.25)0.07 (0.02)0.34 (0.08)
SVM3.40 (0.24)10.45 (0.44)1.83 (0.25)0.07 (0.02)0.27 (0.07)
Train NC4.79 (0.28)11.93 (0.45)1.74 (0.23)0.07 (0.02)0.51 (0.09)
Test NC6.63 (0.34)16.41 (0.55)2.42 (0.33)0.17 (0.10)0.83 (0.12)

Classification error (%) by method and dataset, reported as mean (SE); lower is better.

Logistic and SVM rules achieve errors comparable to the established heads on these shared frozen features. Test NC does not improve over Train NC on these datasets, further illustrating that center calibration is not uniformly advantageous.  We will incorporate these contents related to classification into the revised manuscript, while keeping the main contribution focused on the conditions for classwise concentration and separation of unseen test scores.

References

[1] Wang et al. SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning. arXiv:1911.04623, 2019.

[2] Tian et al. Rethinking Few-Shot Image Classification: A Good Embedding Is All You Need? ECCV 2020.

[3] Chen et al. A Closer Look at Few-Shot Classification. ICLR 2019.

[4] Oquab et al. DINOv2: Learning Robust Visual Features without Supervision. TMLR 2024.

AuthorsOpenReview ↗

General Response 3: Model diagnostics and test-score geometry in real frozen features

This response addresses the applicability of our assumptions and the occurrence of test piling in real frozen features. We provide two complementary analyses: covariance diagnostics of the real frozen features and direct measurements of test-score concentration and separation.

Scope of the model assumption.

The spiked covariance model in (4) allows non-Gaussian factors, class-dependent spike strengths, and correlated, anisotropic residual features. Diffuse residuals may also contain weaker spikes. Thus, the model is more general than Gaussian data with isotropic noise, although it retains assumptions on the common spike structure, latent independence, and moments.

Covariance diagnostics.

We analyzed 89,805 deduplicated images across 387 classes from CIFAR-FS, DTD, Oxford-IIIT Pets, Flowers102, and Caltech101, using frozen DINOv2 ViT-B/14 CLS features (p=768p=768). For the within-class covariance Σk\Sigma_k, we examined three quantities: the largest sample eigenvalue divided by dimension, λ^1,k/p\hat\lambda_{1,k}/p; the average within-class variance, vk=tr⁡(Σk)/pv_k=\operatorname{tr}(\Sigma_k)/p; and the squared-eigenvalue energy, Δk=tr⁡(Σk2)/p2\Delta_k=\operatorname{tr}(\Sigma_k^2)/p^2. Under the weak-spike conditions in (4) and (6), vk→σk2>0v_k\to\sigma_k^2>0 and Δk→0\Delta_k\to0. We estimated vkv_k from the sample covariance and Δk\Delta_k using squared inner products of independent, disjoint within-class differences.

Datasetλ^1/p\hat\lambda_1/pv^k\hat v_kΔ^k\hat\Delta_k
CIFAR-FS0.21 (0.01)1.90 (0.02)0.09 (0.00)
DTD0.31 (0.02)2.02 (0.05)0.17 (0.01)
Oxford-IIIT Pets0.18 (0.01)1.30 (0.06)0.07 (0.01)
Flowers1020.12 (0.01)0.74 (0.02)0.03 (0.00)
Caltech1010.21 (0.01)1.23 (0.04)0.07 (0.01)

Classwise covariance diagnostics, reported as means across classes (SE).

The average variance per feature remains non-negligible, ranging from 0.74 to 2.02. Meanwhile, Δk\Delta_k is comparatively small relative to vk2v_k^2. These results suggest that within-class variation is distributed across multiple directions rather than concentrated almost entirely in one direction, supporting the nondegenerate and diffuse residual-variance conditions in (6).

However, finite-dimensional covariance diagnostics cannot determine the spike-growth exponent, distinguish weak from critical spikes, or verify the common spike subspace and latent independence. In particular, the full covariance Σk\Sigma_k does not separately identify the latent residual covariance RkR_k.

Test-score concentration and separation.

We additionally evaluated 200 predefined five-way, five-shot tasks per dataset. For each task, KMDP was fitted using five labeled images per class, and all remaining unique images from the selected classes were used as test observations. Train and test sets were disjoint within each task. We used the Gaussian kernel with bandwidth τ=5\tau=5, without feature correction or a spike-rank input. The evaluation was performed in the full four-dimensional normalized KMDP score space. We measured the maximum pairwise empirical test CDNV, as defined in Section 5.1, and the minimum distance between empirical test-class centers.

DatasetMaximum test CDNVMinimum test-center distance
CIFAR-FS0.33 (0.11)0.24 (0.00)
DTD0.54 (0.13)0.19 (0.00)
Oxford-IIIT Pets0.07 (0.01)0.40 (0.01)
Flowers1020.01 (0.00)0.62 (0.01)
Caltech1010.12 (0.08)0.47 (0.01)

Maximum pairwise test CDNV and minimum test-center distance, reported as mean (SE) across tasks. Lower CDNV and larger distances indicate stronger concentration and separation.

Flowers102 exhibits the clearest relative concentration, with a mean maximum CDNV of 0.01 and a mean minimum test-center distance of 0.62, followed by Oxford-IIIT Pets (0.07; 0.40) and Caltech101(0.12; 0.47). The other datasets show greater variation across tasks, so these averages do not imply uniformly tight piling in every task.

Interpretation.

The covariance diagnostics provide finite-dimensional information about the feature geometry relevant to our assumptions, while the test-score measurements directly examine the concentration and separation studied in Theorem 1. Although these experiments cannot verify every asymptotic assumption, the small within-class score variation relative to between-class separation, particularly for Flowers102, Oxford-IIIT Pets, and Caltech101, provides empirical evidence of an approximate finite-dimensional analogue of test piling in real frozen features.

These results demonstrate that the test-score geometry characterized by our theory can be observed beyond idealized simulations, without requiring feature correction or prior knowledge of the spike rank in these experiments. They do not establish that correction is unnecessary in general or resolve rank selection when correction is required.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.