acceptodds
Under review as a conference paper at ICLR 2027

The Orthogonality Gap: Predicting, Before Training, When a Representation (or Foundation Model) Will Help

Abstract

Teams routinely fine-tune a foundation model onto a task only to find it beats no linear baseline—after paying the compute. We show that outcome is decidable in advance, from geometry, for free. Given inputs X already in hand and a candidate representation f, only the part of f orthogonal to span(X) can improve an out-of-sample prediction; a re-encoding of X—which a foundation model computed from X often is—has no such part. We formalize this as the Orthogonality Gap and prove an exact decomposition of the attainable OOS gain (Frisch–Waugh–Lovell composed with an attenuation bound), ΔR²_OOS = g⋆·r²·ρ² ≤ g⋆·r², separating a task-intrinsic ceiling—orthogonal signal g⋆ times target reliability r²—from a representation's alignment ρ² with that signal, with equality characterized. The bound is computed from features alone, with no downstream training, and predicts whether a representation will help and by how much before a single fine-tuning run; empirically it is never violated by a linear downstream and holds in 97% of MLP cases. On four real open-weight foundation models across four domains the forecast is calibrated in both directions: a sentence encoder's positive gain to within 0.01; a frozen single-cell transformer's near-zero gain over linear PCs of its own input; and a protein language model's positive gain on designed-protein stability and enzyme engineering (direction a priori, magnitude conservatively under-predicted)—where acting on the forecast selects designs +0.200 more stable (p=0.002). Applied inside a transformer, the Gap selects the best-transferring layer training-free across four models at 0.004 mean accuracy regret versus probe-every-layer. Against LogME, the strongest training-free transferability score, it is statistically tied on real LLM embeddings while adding a proven ceiling, a reliability term, an incremental form, and a faster variant. We report every negative—a refuted phase transition, a retracted ranking claim.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
transfer learning, transferability estimation, foundation models, representation learning, out-of-sample generalization, Frisch–Waugh–Lovell, random matrix theory, training-free model selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dYrqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study whether a candidate representation or foundation model will improve prediction beyond existing features before downstream training. The approach residualizes the representation ff against existing features XX, defines an “orthogonality gap” from the alignment of the residualized representation with the residual target signal, and states

ΔR2=g(f;X)r2=g⋆(X)r2ρ2≤g⋆(X)r2.\Delta R^2 = g(f;X)r^2 = g^\star(X)r^2\rho^2 \leq g^\star(X)r^2.

It further proposes a practical score using residualization, a Marchenko–Pastur spectral threshold, and a reliability estimate. Synthetic and frozen-representation experiments span text, gene expression, proteins, layer selection, and transferability ranking.

The practical question is important and the residualization intuition is useful. However, the central theorem is a Frisch–Waugh–Lovell-style identity for additive linear projection, while the paper repeatedly presents it as an exact ceiling for predictors “of any capacity,” including nonlinear models and universal RKHSs. In addition, the population quantity called training-free depends on the unknown target signal, and the manuscript does not establish that its practical spectral estimator identifies or bounds that quantity.

Critical strengths
  1. The motivation itself is sound and worth exploring: when will an expensive representation add information beyond an existing feature set?
  2. The authors report negative results, including a failed model-ranking claim, a tie with LogME, and cases where the estimated gain is conservative.
  3. The empirical scope is broad, using open-weight representations from text, genomics, and protein modeling.
  4. The attempted applications to layer choice and experimental-design ranking are practically motivated.
Critical weaknesses
  1. The proof supports incremental population least squares after adding ff to an additive linear span. It does not bound predictors that use nonlinear transformations or interactions of XX and ff. An immediate scalar counter example takes no baseline features, f=Z∼N(0,1)f=Z\sim N(0,1), and y=Z2−1y=Z^2-1. The linear covariance and claimed projection gap are zero, but the nonlinear predictor f2−1f^2-1 has R2=1R^2=1.

  2. The definitions of gg and g⋆g^\star contain the unobserved target signal ss. They are useful oracle quantities, but they cannot generally be computed from the distribution of representation features and a scalar reliability estimate. For example, take no baseline features, f∼N(0,1)f\sim N(0,1), and reliability r2=1r^2=1:

  • Task A has y=fy=f, giving an attainable gain of one.
  • Task B has an independent y∼N(0,1)y\sim N(0,1), giving an attainable gain of zero.

The feature distribution and reliability are identical in both tasks. No feature-only statistic can distinguish their target alignment. A Marchenko–Pastur threshold can identify spectral structure, but spectral mass is not evidence that the structure aligns with yy.

  1. The introduction motivates deciding whether to fine-tune a foundation model, but the experiments use frozen embeddings with downstream predictors. The appendix itself lists end-to-end fine-tuning as future work. Fine-tuning changes the representation and hence its residual geometry, so a score computed on the initial frozen ff is not automatically a ceiling on fine-tuned performance.

  2. The theory assumes y=s+ϵy=s+\epsilon, orthogonal noise, and a known or estimable r2=∥s∥2/∥y∥2r^2=\|s\|^2/\|y\|^2. The paper mentions replicates or subsampling, but does not give a justified estimator for each experiment, especially single-label text classification, DepMap responses, and protein assays. Subsampling observations does not in general identify measurement reliability. The assumptions, exact formulas, data splits, and uncertainty for every task are needed.

  3. Synthetic experiments designed around linear/additive assumptions cannot establish a universal ceiling. Even a single meaningful nonlinear violation would refute an exact claim. The reported 97% MLP coverage is evidence against a literal universal upper bound unless the uncovered cases are fully explained by uncertainty defined in advance.

Questions for authors
  1. What exact predictor class defines both sides of Theorem 1? How does the theorem handle the y=Xfy=Xf interaction counterexample above?
  2. Does the implemented gap estimator use target labels? If yes, which labels, splits, and cross-fitting operations are used? If no, how is target alignment identifiable?
  3. What formal result connects the Marchenko–Pastur estimator to the oracle gg, and under what distributional and dimensional assumptions?
  4. How are ρ\rho and the equality condition defined when ff is vector-valued?
  5. What exact reliability estimator is used in every real-data task, including classification?
  6. Why should a score computed from frozen embeddings bound or predict the benefit of changing those embeddings by fine-tuning?
  7. How do results compare with simple fixed-layer, k-NN, and recent regression-transferability baselines?
  8. Can the authors provide complete data counts, split rules, preprocessing, regularization, hyperparameters, confidence-interval construction, and design-selection protocol?
Reviewer_NJRNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a measure to predict on top of a given representation, whether another new representation will help and by how much before actually fine-tuning on the new representation. The paper shows that only the part of the new representation that is orthogonal to the given representation adds to enable a better prediction, the improvements also depends on the noise level of the target labels, which are intuitive and nice to learn.

Critical strengths

-The problem is well motivated, it might be practically convenient to be able to predict if a fine-tuning will help before the actual fine-tuning happens. -The paper evaluates on four foundation models across four domains, including a sentence embedding model for the text domain, and three models for the general bio domain spanning over single cell, protein and enzyme domains.

Critical weaknesses

-The setting or the question asked may not reflect how people use fine-tuning these days, for example for the text domain, I don't think people still would like to use TF-IDF to represent a sentence, more often, people directly apply sentence embeddings for all kinds of prediction tasks and might care more about the direct result of a fine-tuned model instead of how much does fine-tuning add to existing weak representations.

-There's no algorithm given for calculating the proposed metric, the protocol is one line (L156), but I still feel unclear how to compute g(f_i, x), will it be possible to add an algorithm or pseudo code? how costly? what's the time complexity?

-I found the paper hard to read, acronyms were not spelled out or explained, e.g., since FWL is such an important prior foundation work, I expect it to be briefly explained so the paper is self-contained. Some sentences are not even complete sentences, e.g., L413-416. Some sentences are very dense with numbers with little explanation, e.g., L134-137, L150-152. Table 1 does not look very useful and the numbers there do not immediately make sense.

Questions for authors

I'm not sure if you can claim your approach achieves near oracle tranformer layer selection as you only compared to a probing approach, the ranking results of the probing approach can be suboptimal, right?

Reviewer_LdV4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how to forecast the performance of a foundation model in a training-free manner and from its underlying geometry. In particular, they propose the Orthogonality Gap, claimed to be computed from features alone and without downstream training, to predict how effective a representation is. Empirically, the authors include experiments that apply this gap to foundation models like Geneformer and ESM-2 and confirm some predictive power of this metric.

Critical strengths
  • The topic is empirically meaningful: The problem in this paper has practical implications: evaluating quality of representation/foundation models through repeated fine-tuning can waste compute, and being able to forecast the performance before training is highly relevant for efficient evaluation.
Critical weaknesses

Inaccurate Theory and Empirical Support: I find contents of this paper unorganized and highly misleading. In particular, the following technical concerns stood out to me:

  • Theorem 1 is both overstated and incorrect. It claims that the derived ceiling is tight and “unattainable by any predictor of any capacity”. However, the proof only considers a linear function class, and there can be many counterexamples where a nonlinear predictor is the best and beats any linear predictor. The authors should make efforts to improve the rigor and separate empirical evidence from theory. This gap in theory largely invalidates the sections that follow.
  • The Orthogonality Gap is not clearly formulated and hugely under-explained. It is claimed to be “computed from features alone”, but from section “the instrument, as a procedure” on page 2, it requires measuring g(f;X)=∥∥Pf⊥s⊥∥∥2/∥∥s∥∥2g(f; X) = \|\|P_{f_{\perp}}s_{\perp}\|\|^2/\|\|s\|\|^2, where ss is the unknown task signal and Pf⊥P_{f_{\perp}} is the projection onto span(f⊥)span(f_{\perp}). I do not think these quantities can be measured from features only. Many central quantities like the Marchenko-Pastur noise edge (line 90) are not explained and there is no unifying way to compute the gap throughout this paper.

Lack of Motivation and Absence of Context: The authors did not include related citations to contextualize their work, leading to many unsupported claims. Just to name a few, in the first paragraph, the claim “on many tabular and omics tasks, larger and pretrained encoders do not beat a linear map” is not supported by any cited works. Relevant papers on topics like how representation affects post-training are entirely missing, so it is difficult to gauge the context of this paper and motivation. Please cite central works built upon and include a related work section.

Overall, given the insufficiency in both theoretical and empirical results and the lack of motivation, I suggest that the paper should be substantially rewritten and submitted to a future venue.

Questions for authors

Please refer to the Weakness section. In addition, there appear to be various experiment results, and some reports are unclear. Can you specify what “four real open-weight foundation models across four domains” in the abstract exactly refer to?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.