- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies when unseen test observations concentrate at class-specific locations in high-dimensional, low-sample-size classification. It introduces KMDP, extending classical data piling to reproducing kernel Hilbert spaces, and establishes theoretical conditions for test-data concentration and class separation. The paper also identifies a critical-spike regime where test piling fails and proposes a feature correction to restore it. Simulation results support the main theoretical findings.
Critical strengths
1.The paper addresses an interesting theoretical question: whether classwise concentration observed in training data can extend to independent test observations. 2.Extending maximal data piling to nonlinear kernel spaces is a meaningful contribution, particularly the characterization of kernel-dependent class separation. 3.The theoretical analysis is carefully developed. The impossibility result for critical spikes in the Gaussian kernel span and the subsequent recovery guarantee are particularly interesting. 4.The paper is generally well organized, and the simulations provide consistent support for the theoretical results.
Critical weaknesses
1.The connection to practical few-shot learning remains somewhat limited. Although the paper motivates its analysis through frozen representations and few-shot classification, the experiments rely entirely on synthetic data. It is unclear how closely real pretrained representations satisfy the theoretical assumptions. 2.The feature correction relies on relatively strong assumptions. In particular, it requires consistent estimation of the dominant spike subspace, potentially using additional unlabeled data. Its practical effectiveness may depend on how well this subspace can be identified without removing discriminative information. 3.The scope of some theoretical results is relatively specific. The critical-spike impossibility result is established for the empirical Gaussian kernel span. A clearer discussion of its implications for other kernels would help readers understand the generality of the findings.
Questions for authors
- Could the authors provide more intuition for why the Gaussian kernel can separate test piles based solely on residual variance differences, while the linear and polynomial kernels cannot?
- Could the authors include a small experiment using real pretrained features, or at least discuss how the HDLSS and spiked-covariance assumptions might be assessed in practical few-shot settings?
- How should the correction rank be selected in practice? Appendix B shows that rank overestimation can remove useful class-discriminative information, particularly under transductive correction. Could the authors provide some practical guidance?
- To what extent might the critical-spike obstruction extend beyond Gaussian kernels? Even a brief discussion of the expected behavior for other kernel families would be helpful.
Response to Reviewer 2wot
We thank the reviewer for the favorable evaluation of our work. We address Weakness 1 and Question 2 in General Responses 2 and 3, which present real frozen-feature experiments and examine the model assumptions and test-score geometry.
We agree with the concern in Weakness 2 that spike-subspace estimation and preservation of discriminative information pose important practical challenges. General Response 4 empirically examines these limitations and addresses Question 3 concerning correction-rank selection and sensitivity.
We address Weakness 3 and Question 4 in General Response 5, which extends the critical-spike obstruction to a broader family of kernels.
We respond to Question 1 individually below.
Question 1
Consider two classes with the same mean but different residual variances. Under the weak-spike conditions of Theorem 1, each class forms a cloud with a different radius. The classes therefore differ in their spread, even though their centers coincide.
The linear kernel compares observations through their inner product. In high dimensions, independent residual vectors become nearly orthogonal [1], so their contribution to the normalized inner product vanishes. The limiting inner product therefore reflects the common mean but not the different cloud sizes. Since residual variance information vanishes from the inner product in the high-dimensional limit, applying a polynomial transformation cannot recover it. Consequently, the linear and polynomial kernels fail to separate test piles when classes differ only in residual variance.
The Gaussian kernel instead compares distances. Even when two residual vectors are nearly orthogonal, the distance between them still depends on their lengths. Thus, the difference between classes forming clouds of different sizes remains visible in the Gaussian kernel similarities. Importantly, the exponential dependence on squared distance converts these differences into distinct test-pile locations, as established by the separation criterion in Theorem 1. This mechanism also extends to other strictly convex radial kernel profiles, such as rational quadratic and Laplacian kernels, under analogous regularity conditions. The same intuition applies after consistent spike correction under Theorem 3.
[1] Hall et al. Geometric Representation of High Dimension, Low Sample Size Data. Journal of the Royal Statistical Society: Series B, 67(3):427–444, 2005.