acceptodds
Under review as a conference paper at ICLR 2027

Representation Risk in Pretrained Image Encoders

Abstract

Applied researchers increasingly convert images into features with pretrained encoders, then use those features in a downstream prediction model. The encoder is often treated as an implementation detail. We show that it can instead be a consequential source of model uncertainty. We call this uncertainty representation risk: plausible pretrained encoders map the same images into different feature spaces and can yield sharply different out-of-sample conclusions. We compare modern and legacy frozen encoders across house prices, racehorse workout performance, breast-cancer histology, chest radiographs, facial age, and rice disease. With the same images, split, and linear-head protocol, replacing ResNet50 with DINOv2 raises test from 0.395 to 0.637 for houses and from 0.043 to 0.137 for horses. No encoder is best in every task. Validation-trained stacking improves further when strong encoders make complementary errors: a DINOv2–SigLIP 2 stack reaches 0.666 for houses, and averaging improves BreaKHis accuracy from 0.911 to 0.926. Stacking adds little for horses and facial age, and rice is at its performance ceiling. The rankings persist with nonlinear neural-network heads, so the gaps are not artifacts of linear extraction. These results support a simple workflow: benchmark plausible representations, select on locked validation data, combine only when validation evidence justifies the additional cost, and report paired uncertainty. We implement this workflow in a reproducible software framework using the "LookAgain-ML" package.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation selection, model selection, reproducibility, applied machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nKJMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors show that the choice of pre-trained image encoding model is an important factor in the success of a downstream classifier.

Critical strengths
  1. Careful experimental validation of the hypothesis (that the choice of pre-trained image encoding model is an important factor in the success of a downstream classifier).
  2. Introduces a practical workflow to help researchers choose a pre-trained image encoder.
Critical weaknesses
  1. The fact that the choice of image encoder has a high impact on downstream predictive models has been a central tenant in computer vision since the beginning of the field. The revolution of general pre-trained encoders may have led newer researchers to believe that visual encoding was a solved problem, and so it is good for them to be reminded that there is (at least not yet) one encoder that rules them all.
  2. The paper cites several recent papers making similar arguments. Some that are missing include Goldblum et al. (2023), "Battle of the Backbones," and Ericsson et al. (CVPR 2021), "How Well Do Self-Supervised Models Transfer?".
Questions for authors

Could you consider adding to the evaluation some models that practitioners would be most likely to use. For example, for medical images there is RAD-DINO or CheXzero. It would be nice to also include the original CLIP (although one would not expect it to be as strong as the others, it is still widely used).

Reviewer_G4fpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper argues that the choice of frozen image encoder is a large and under-reported source of uncertainty in prediction, and refers to this as representation risk. The authors extract features from ten encoders, reduce each to at most 128 dimensions with PCA, and fit the same ridge or logistic head on six tasks. They compare picking one encoder on validation data against four ways of combining encoders, with everything fixed before testing. The main findings are that the best encoder differs by task, and that combining encoders helps on some tasks but not others. The workflow is released as the LookAgain-ML package.

Critical strengths
  • The protocol is careful about leakage. Splits are grouped by patient or by horse identity (Appendix A), and stacking weights and subset choices come only from training folds (Appendix B). This makes the insights reliable.
  • The authors report results that do not favor them, such as the horse feature-union failure (Table 3, Appendix D.1) and the age stack doing worse than the single selected encoder on the fixed split.
  • Uncertainty is reported as paired differences against the selected encoder (Table 2), with repeated-split results shown separately (Table 3).
  • The robustness checks cover some of the important aspects like, an MLP head vs linear head, DINOv2 vs. ViT-B/16 at similar size, near-duplicate checks for age and rice, and LogME as a cheap screen.
  • The tasks span economic, medical, animal, demographic and agricultural images, and the house and horse tasks come with economic targets that are less common in ML transfer studies.
  • Code is released through an anonymous repository, with a public end-to-end example (Appendix J).
Critical weaknesses
  • The core finding is not new. That results depend on the frozen encoder, and that the best encoder varies by task, is already shown by Kornblith et al. (2019), VTAB, LogME, and two uncited papers: Ericsson et al. (CVPR 2021) and "Battle of the Backbones" (NeurIPS 2023). Perception Encoder (Bolya et al., 2025) goes further and shows that, even within one encoder, the best features for a task are often not at the output of the network. The paper does not cite it or test features from other layers.

  • The recommended procedure (try several encoders, pick on validation, combine only if validation supports it) is standard model selection, and the methods are standard too (ridge, logistic regression, stacking, concatenation). The paper offers no new insight into why encoders differ and no method that beats validation selection.

  • The main gaps reported in the paper are measured against ResNet50 (abstract, Table 2), which few people would pick today. Against the best modern encoder the gaps are much smaller. On houses, DINOv2 reaches 0.621 and SigLIP 2 0.629 (Table 5). On BreaKHis and pneumonia, the top encoders are within a few points of each other (Table 6), and rice is at ceiling. The paper should report gaps relative to a strong modern default such as DINOv2 or SigLIP 2.

  • The encoder set is narrow. All modern models are base size. There is no CLIP, no larger DINOv2 or SigLIP, and no domain-specific encoders that an applied researcher would actually try, such as pathology models (UNI, CONCH) for BreaKHis, chest X-ray models (BiomedCLIP, RAD-DINO) for pneumonia, or face models for age.

  • The fixed 128-dimension PCA cap could drive the rankings and is not ablated. ResNet50 goes from 2,048 to 128 dimensions, ViT, DINOv2 and SigLIP from 768. Ridge already controls capacity, so PCA is not needed for that. Results with no PCA, or with caps of 256 and 512, would show whether the gap holds.

  • The horse and house results might be confounded, and the authors must verify this. An R2 of about 0.10 for workout speed from a single photo is unusual. Photos from the same session may share background or lighting cues that relate to workout conditions. Grouping by horse does not rule this out, but grouping by sale session or masking the background would. For houses, the splits group only by exact image hash, so neighbors on the same street can end up in both train and test. A spatial split would show whether SigLIP 2 is reading the house or the neighborhood.

  • The horse feature-union failure is not explained. In one of twelve splits, the procedure picked on validation gave a test R2 of -49.1, and the authors say the cause is unverified (Section 5.3, Appendix D.1). Since the main contribution is a selection procedure and a software package, the paper should diagnose this failure and say what safeguard the workflow needs.

  • Two of the six tasks add little. Rice is near ceiling with 37% near-duplicate test images (Appendix F), and pneumonia sits at 0.95-0.98 for most modern encoders.

  • The writing needs improvement. In some parts, it reads like a workflow log rather than paper text. "Fixed in the implemented workflow" is repeated many times. There are unexplained mentions of "corrected" predictions (Figure 5, Appendix J.5), and Appendix A says "5,180 metric comparisons" were recomputed and "58 checks passed" without explaining how the two relate.

Questions for authors
  1. If you limit the candidates to the four modern encoders (DINOv2, SigLIP 2, ConvNeXt-B, ViT-B/16), what is the spread in test performance per task? How often does the selected encoder beat a fixed DINOv2 or SigLIP 2 default by more than its interval?
  2. How do the rankings change without PCA, or with caps of 256 or 512? Does the ResNet50 to SigLIP 2 gap on houses shrink?
  3. For horses, does the DINOv2 advantage survive when grouped by sale session or consignor, or masking the background?
  4. For houses, how does the ranking change under a spatial split (by neighborhood or postal code)?
  5. What caused the test R2 of -49.1 in the horse feature-union split? Can you confirm the near-constant-feature explanation, and what safeguard would you add to LookAgain-ML?
  6. Across the twelve splits, how much worse is the validation-selected encoder than the test-best encoder on each task, and how does that compare to always using DINOv2 or SigLIP 2?
  7. How many patients are in the BreaKHis test set, and how stable are the bootstrap intervals with so few groups?
  8. Did you consider domain-specific or larger encoders? If they were left out on purpose, why?
Reviewer_cj3gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the standard practice of probing pretrained features for predictive tasks. They frame the resulting uncertainty over the results as "representation risk." They then survey some methods for improving held out results using validation, ensembling etc.

Critical strengths

The application of frozen features to predictive tasks is relevant to the broader community to encourage best practices, and code is released as a package.

Critical weaknesses

The findings of this paper are generally obvious. Probing pretrained features is a commonly used methodology and this paper does not sufficiently reframe either the setup, evaluation, or methodology. Existing literature has already thoroughly explored model selection from pretrained backbones: Ericsson, Linus, Henry Gouk, and Timothy M. Hospedales. "How well do self-supervised models transfer?." 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2021. Goldblum, Micah, et al. "Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks." Advances in Neural Information Processing Systems 36 (2023): 29343-29371. Bolya, Daniel, Rohit Mittapalli, and Judy Hoffman. "Scalable diverse model selection for accessible transfer learning." Advances in Neural Information Processing Systems 34 (2021): 19301-19312. Pineda Arango, Sebastian, et al. "Quick-tune: Quickly learning which pretrained model to finetune and how." International Conference on Learning Representations. Vol. 2024. 2024.

The paper also does not really follow through on the story in the abstract/intro. The paper is motivated by how features change empirical decision making, but does not study or sufficiently discuss this.

Outside of highly specific medical tasks, the validation winners tend to be general purpose models DINOv or SigLIP2. This calls into question if newer and larger models like DINOv3 overcome issues in model selection by generally being better at everything through scale. And ensembling, previous work in this paper does not sufficiently acknowledge despite high similarity: Van der Laan, Mark J., Eric C. Polley, and Alan E. Hubbard. "Super learner." (2007). Caruana, Rich, et al. "Ensemble selection from libraries of models." Proceedings of the twenty-first international conference on Machine learning. 2004. Breiman, Leo. "Stacked regressions." Machine learning 24.1 (1996): 49-64/

The figures in the paper are crowded and uninviting to meaningfully follow.

Questions for authors

See weaknesses.

Also, for a paper largely relying on benchmarking, why were uncited and presumably unreleasable (according to reproducibility statement) datasets chosen?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.