- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether spectral statistics of pretrained weights can predict how much performance drops under distribution shift. The authors select different statistics for different architecture and pretraining groups, and evaluate the relationships across 116 models. They also test spectral interventions and predictions on additional model families.
Critical strengths
-
I like the idea of assessing a model from its released weights alone. If the rankings hold for new models, perhaps this could help decide which checkpoints are worth spending evaluation time on. Accounting for differences in architecture and pretraining also seems reasonable.
-
The range of model families is pretty wide. I also appreciate that the authors try interventions and prospective predictions, since correlations alone wouldn't have been sufficient. These experiments make the paper interesting, although I have concerns about how some results are interpreted.
Critical weaknesses
-
I wasn't sure why several models were excluded for breaking the proposed trend. Appendix B gives this reason for the smaller ESM2 models and GENA-LM-large, among others. Wouldn't retaining these models be important for testing the relationship? From my understanding, the metric and sign are also selected using the same outcomes, and the leave-one-out calculation only recomputes the correlation after removing each model. I don't think this tests prediction on the omitted model, so I'm not yet convinced by the headline ranking results.
-
I had trouble reconciling some of the broader claims with the appendix. In Appendix R, target accuracy improves by 0.008 while source accuracy drops by 0.308. Is the smaller gap mainly coming from worse source performance? The two measured repair cases also increase feature stable rank, which seems opposite to the proposed explanation. I think the claims about spectral geometry governing robustness need to be more cautious.
Questions for authors
-
Could the authors show whether the correlations hold with the excluded models included, and evaluate metric/sign selection on held-out model families? This would illuminate how much of the reported ranking performance is likely to carry over to new models.
-
Could the authors separate source and target accuracy changes for the main interventions?
-
Proposition 4 uses participation ratio, but PR-mean in the experiments denotes mean stable rank. How does the proposition apply to the quantity actually measured?