- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors study whether a candidate representation or foundation model will improve prediction beyond existing features before downstream training. The approach residualizes the representation against existing features , defines an “orthogonality gap” from the alignment of the residualized representation with the residual target signal, and states
It further proposes a practical score using residualization, a Marchenko–Pastur spectral threshold, and a reliability estimate. Synthetic and frozen-representation experiments span text, gene expression, proteins, layer selection, and transferability ranking.
The practical question is important and the residualization intuition is useful. However, the central theorem is a Frisch–Waugh–Lovell-style identity for additive linear projection, while the paper repeatedly presents it as an exact ceiling for predictors “of any capacity,” including nonlinear models and universal RKHSs. In addition, the population quantity called training-free depends on the unknown target signal, and the manuscript does not establish that its practical spectral estimator identifies or bounds that quantity.
Critical strengths
- The motivation itself is sound and worth exploring: when will an expensive representation add information beyond an existing feature set?
- The authors report negative results, including a failed model-ranking claim, a tie with LogME, and cases where the estimated gain is conservative.
- The empirical scope is broad, using open-weight representations from text, genomics, and protein modeling.
- The attempted applications to layer choice and experimental-design ranking are practically motivated.
Critical weaknesses
-
The proof supports incremental population least squares after adding to an additive linear span. It does not bound predictors that use nonlinear transformations or interactions of and . An immediate scalar counter example takes no baseline features, , and . The linear covariance and claimed projection gap are zero, but the nonlinear predictor has .
-
The definitions of and contain the unobserved target signal . They are useful oracle quantities, but they cannot generally be computed from the distribution of representation features and a scalar reliability estimate. For example, take no baseline features, , and reliability :
- Task A has , giving an attainable gain of one.
- Task B has an independent , giving an attainable gain of zero.
The feature distribution and reliability are identical in both tasks. No feature-only statistic can distinguish their target alignment. A Marchenko–Pastur threshold can identify spectral structure, but spectral mass is not evidence that the structure aligns with .
-
The introduction motivates deciding whether to fine-tune a foundation model, but the experiments use frozen embeddings with downstream predictors. The appendix itself lists end-to-end fine-tuning as future work. Fine-tuning changes the representation and hence its residual geometry, so a score computed on the initial frozen is not automatically a ceiling on fine-tuned performance.
-
The theory assumes , orthogonal noise, and a known or estimable . The paper mentions replicates or subsampling, but does not give a justified estimator for each experiment, especially single-label text classification, DepMap responses, and protein assays. Subsampling observations does not in general identify measurement reliability. The assumptions, exact formulas, data splits, and uncertainty for every task are needed.
-
Synthetic experiments designed around linear/additive assumptions cannot establish a universal ceiling. Even a single meaningful nonlinear violation would refute an exact claim. The reported 97% MLP coverage is evidence against a literal universal upper bound unless the uncovered cases are fully explained by uncertainty defined in advance.
Questions for authors
- What exact predictor class defines both sides of Theorem 1? How does the theorem handle the interaction counterexample above?
- Does the implemented gap estimator use target labels? If yes, which labels, splits, and cross-fitting operations are used? If no, how is target alignment identifiable?
- What formal result connects the Marchenko–Pastur estimator to the oracle , and under what distributional and dimensional assumptions?
- How are and the equality condition defined when is vector-valued?
- What exact reliability estimator is used in every real-data task, including classification?
- Why should a score computed from frozen embeddings bound or predict the benefit of changing those embeddings by fine-tuning?
- How do results compare with simple fixed-layer, k-NN, and recent regression-transferability baselines?
- Can the authors provide complete data counts, split rules, preprocessing, regularization, hyperparameters, confidence-interval construction, and design-selection protocol?