acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

Abstract

Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require source or target data, which rules them out before a target domain exists. Those that use the weights alone apply one statistic to every architecture, and do not separate robust models from fragile ones. We show the answer is encoded in the spectral structure of pretrained weights. Two forces shape that structure. Architecture determines how information is stored in weight matrices. Pretraining strategy determines what is rewarded. Together they set a spectral geometry that governs OOD robustness. We prove that the OOD accuracy gap is bounded by how tightly the source representations concentrate. A statistic computed from the pretrained weights alone serves as a proxy for that concentration. The direction of that proxy reverses between architecture families. We operationalize it: the direction is stable within one (architecture × strategy) combination, the finest grouping we test, which we call a cell. Pooled over 116 models spanning 7 modalities, a single statistic ranks OOD robustness weakly, because cells of opposite direction cancel. Within a cell, the statistic selected for it orders 92% of model pairs by OOD robustness in-sample. The selection does not leak the target: for each model family outside the matrix we logged the cell, metric and sign before running its OOD evaluation, and the predicted direction held in every case: EEG, genomic and protein. Acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention. The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Out-of-Distribution Generalization, Foundation Models, Spectral Geometry, Data-Free Diagnostics, Pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BbDtOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether spectral statistics of pretrained weights can predict how much performance drops under distribution shift. The authors select different statistics for different architecture and pretraining groups, and evaluate the relationships across 116 models. They also test spectral interventions and predictions on additional model families.

Critical strengths
  1. I like the idea of assessing a model from its released weights alone. If the rankings hold for new models, perhaps this could help decide which checkpoints are worth spending evaluation time on. Accounting for differences in architecture and pretraining also seems reasonable.

  2. The range of model families is pretty wide. I also appreciate that the authors try interventions and prospective predictions, since correlations alone wouldn't have been sufficient. These experiments make the paper interesting, although I have concerns about how some results are interpreted.

Critical weaknesses
  1. I wasn't sure why several models were excluded for breaking the proposed trend. Appendix B gives this reason for the smaller ESM2 models and GENA-LM-large, among others. Wouldn't retaining these models be important for testing the relationship? From my understanding, the metric and sign are also selected using the same outcomes, and the leave-one-out calculation only recomputes the correlation after removing each model. I don't think this tests prediction on the omitted model, so I'm not yet convinced by the headline ranking results.

  2. I had trouble reconciling some of the broader claims with the appendix. In Appendix R, target accuracy improves by 0.008 while source accuracy drops by 0.308. Is the smaller gap mainly coming from worse source performance? The two measured repair cases also increase feature stable rank, which seems opposite to the proposed explanation. I think the claims about spectral geometry governing robustness need to be more cautious.

Questions for authors
  1. Could the authors show whether the correlations hold with the excluded models included, and evaluate metric/sign selection on held-out model families? This would illuminate how much of the reported ranking performance is likely to carry over to new models.

  2. Could the authors separate source and target accuracy changes for the main interventions?

  3. Proposition 4 uses participation ratio, but PR-mean in the experiments denotes mean stable rank. How does the proposition apply to the quantity actually measured?

Reviewer_6VvmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This study investigates a question of substantial practical relevance: whether the out-of-distribution (OOD) robustness of a foundation model can be inferred solely from its pretrained parameters, prior to the availability of any target-domain data. The authors note a key limitation of weight-space diagnostics: using one statistic across architectures cancels opposing signals between architecture families, weakening predictive power. To resolve this, the paper conditions on the joint (architecture × pretraining strategy) cell, arguing both jointly shape the spectral geometry of weight matrices and must be considered together. Within each cell, the authors demonstrate that a cell-specific spectral statistic orders 92% of model pairs by OOD robustness in-sample, and validate this framework prospectively on model families outside the training matrix, including EEG, genomic, and protein modalities. The theoretical contribution is a bound showing that the OOD accuracy gap is governed by how tightly source representations concentrate, with the pretrained weight statistic serving as a proxy for this concentration. Beyond diagnosis, the authors demonstrate that acting directly on spectral concentration narrows the OOD gap by 24% while retaining 87.5% of in-distribution accuracy.

Critical strengths
  1. The paper addresses an intriguing and practically relevant problem: predicting a foundation model’s out-of-distribution generalization performance in the absence of any available target-domain data. This diagnostic perspective, which does not require empirical data, is practically valuable.

  2. The manuscript introduces a framework for data-free prediction of out-of-distribution (OOD) robustness, supported by both theoretical justification and demonstrated practical applicability across multiple modalities.

  3. The paper makes a practical contribution: addressing spectral concentration reduces the OOD gap by 24% while preserving 87.5% in-distribution accuracy, a trade-off rarely seen in OOD robustness work. Metric-guided layer selection beats random targeting in 9 of 10 models, showing the proposed spectral diagnostic both predicts and effectively guides targeted improvements.

Critical weaknesses
  1. While the authors claim that each metric is theoretically motivated, the assignment of different statistics to different cells lacks sufficient justification.

  2. The small sample size for per-cell evaluations raises concerns about the statistical reliability of the reported results, making it questionable whether the observed correlations carry sufficient significance to support the paper’s broader claims.

  3. The introduction suffers from poor clarity and limited readability. A substantial portion of it is occupied by figures, yet the accompanying explanations are insufficient to guide the reader through the visual content.

(minor) The content within Figure 1 and Figure 2 is rendered at an excessively small scale, making it difficult to read and interpret the details presented.

(minor) The manuscript employs numerous abbreviations that are not adequately defined or explained, thereby reducing readability and potentially impeding comprehension for readers who are unfamiliar with the relevant terminology.

Questions for authors
  1. In Table 3, the authors presented mean| ρ\rho | rather than ρ\rho. Is this because of the negative values in line 307? Can the authors explain what negative values mean, was this expected?

  2. When using the SVD bottleneck, can the authors compare the results on various rank r?

  3. Can the authors provide details on the EEG, genomic, and protein data?

Reviewer_9Go2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether a foundation model's out-of-distribution robustness can be read from its released weights before any target data exists. It groups 116 public checkpoints into 15 cells by architecture and pretraining strategy, computes a set of spectral statistics of the weight matrices with WeightWatcher, and reports that within each cell one statistic, with a cell-specific sign, orders models by their OOD accuracy drop. Theorem 1 bounds the OOD gap by the concentration of source features under given assumptions, and they motivate candidate weight statistics per architecture. It reports that a feature-rank bottleneck narrows the OOD gap by 24 % at 87.5 % ID retention and that metric-guided layer repair beats random targeting. FInally it reports prospective tests on EEG, protein and genomic families whose cell, metric and sign were logged before evaluation.

Critical strengths

The model set is large, public and listed, so every weight statistic can be recomputed by anyone, and fifteen cells across six or seven domains is broad. Theorem 1 seems correct as a conditional statement, and the paper says how conditional it is. In the two protein cells and BiDi-MLM NLP the selected statistic also far exceeds log parameter count in Table 8, which is the best evidence in the paper that a spectral statistic carries information that size does not.

Critical weaknesses

The metric, its sign and part of the sample were chosen on the same outcomes they are scored on, and selection alone accounts for most of the reported strength

The intervention headline is in-distribution loss, with OOD accuracy at chance. The gap is ID minus OOD accuracy, so any intervention that lowers ID accuracy lowers the gap with OOD accuracy unchanged

The 92 % pairwise accuracy is incompatible with Table 1

The conditioning hierarchy of Table 3 is largely what best-of-ten selection in shrinking groups produces under noise

The prospective tests do not show that "The selection does not leak the target"

Model size is not separated from the selected statistic.

The Discussion says "a model can be assessed from its weights alone, with no target data, no forward pass and no labeled examples" (p. 9), while the protocol says "If direction is unknown (new domain): calibrate with ≥ 4 models in the same cell; determine direction empirically." (p. 36). The diagnostic is not data-free at setup.

The theory does not predict the metric or its sign, and the quantity it bounds is never measured across the models

ViT×DINO and Mamba (n = 5) are kept because "both marginal cells clear the per-cell validity threshold" (p. 26), while HyenaDNA (n = 5) "does not meet the p ≤ 0.05 threshold for primary cell inclusion" (p. 33), against the statement that "no post-hoc selection of cells was performed" (p. 26)

Questions for authors
  1. Can you release the per-model table (metric values, ID accuracy, OOD accuracy, drop, probe seeds) during the discussion period?
  2. On that table, what are the per-cell permutation p-values of the full pipeline, and what does a nested leave-one-out selection give, with log parameter count treated the same way?
  3. What are the per-model OOD accuracies, not the gaps, before and after the bottleneck at r = 64 and after targeted repair?
  4. How were the pairwise ranking accuracy 0.924 and the median 0.958 computed, and with what tie rule and model set?
  5. Which prediction was logged for AMPLIFY, and what does the EEG test give with LUNA-base and CBraMod included?
  6. Within each cell, what is the partial Spearman correlation of the selected statistic with OOD drop controlling for log parameter count?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.