acceptodds
Withdrawn from ICLR 2027

Per-Sample Routing of Pre-trained Encoders via Embedding Typicality

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Practitioners typically deploy pre-trained vision encoders by choosing a single backbone and layer, fitting a probe on its embeddings, and using that representation for every test sample. This fixed choice can be suboptimal because the best encoder and layer can vary across inputs in heterogeneous tasks. We study unsupervised per-sample routing over a pool of frozen pre-trained encoders, treating each (backbone, layer) pair as a candidate. Using only embedding geometry, the router either assigns a test sample to one candidate or combines predictions across the pool. It requires neither supervised router training, test labels, nor per-candidate calibration. For hard routing, we repurpose the existing Levy radial-typicality score: it reads typicality from the embedding norm and selects one candidate per sample. For soft ensembling, we introduce the local spectral normalized gap (LSNG), which measures local spectral typicality and produces dimension-comparable weights over the full pool. On a pool of 192 candidates from 2 general-purpose backbones and 6 finetuned experts, LSNG soft ensembling gains +7.8 to +54.1 percentage points (pp) in accuracy over the DINOv2-L last-layer default under a linear probe. This is within 1.0 pp of the test-selected best fixed candidate on 3/5 evaluations and above it on the other 2. Levy hard routing evaluates 1 probe per sample, matching the fixed default's probe cost, and improves accuracy by up to +31.1 pp on 3 evaluations. It fails when strongly finetuned off-domain experts imitate typical embedding norms and attract the argmax; soft ensembling is less sensitive to this failure. Across this heterogeneous library, training-free embedding geometry provides enough signal to route between frozen encoders.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, model routing, pretrained encoders, embedding geometry, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RSi5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles the problem of routing test examples in an unsupervised fashion to the best (backbone, layer) tuple, under the assumption that a set of backbones and per-layer linear probes are available. They propose two algorithms to do so:

  • Hard routing via the Levy radial-typicality score.
  • Soft-routing via the local spectral normalized gap, or LSNG.

They compare the two approaches using a candidate pool of 8 encoders, composed of 2 general purpose models, i.e., DINOv2 ViT-L/14 and MAE ViT-L, and 6 expert models, fine-tuned on Galaxy10, PathMNIST and Aircraft, where each dataset has both a DINOv2 and MAE expert. They compare their approaches to two main baselines:

  • Fixed backbone, i.e., a linear probe fitted on the last layer of DINOv2 ViT-L/14.
  • The per-dataset oracle, i.e., the best (backbone, layer) probe by test accuracy.

Levy hard routing improves over the fixed backbone across 3/5 datasets, except on Aircraft and PathMNIST, where it performs worse than the fixed backbone baseline. LSNG instead provides a consistent neutral to strongly positive performance improvement in linear probing, sometimes outperforming the per-dataset oracle.

The caveat is that both hard and soft routing require evaluating all 8 backbones. Additionally, soft routing must evaluate each of the 192 (backbone, layer) probes.

Investigating which probes contribute the most to LSNG reveals that the algorithm tends to prefer expert models, with some exceptions.

The authors further show that LSNG is somewhat robust to pool corruptions and that it significantly improves the worst-class accuracy.

Critical strengths
  1. The paper is well written, clear, and easy to read.
  2. The proposed routing and ensembling techniques are training-free and mathematically motivated.
Critical weaknesses

In my opinion, the main weakness of the paper is the evaluation protocol: under the assumption that an expert model is available, a practitioner would deploy such model, making the per-dataset oracle (when selected on validation) the realistic baseline to compare the method against.

Against such a baseline, the performance improvements due to the proposed methods are negative to modest. The only clear improvement is achieved by LSNG on the merged dataset, but even then, the per-dataset oracle is a model fine-tuned on one of the three datasets that compose the merged one, and is not clearly comparable.

Other weaknesses include:

  • While both methods indeed require no router and are fully unsupervised, they require a forward from every pool member.
  • It's unclear why MAE models were chosen for the pool. LevyScore [1] was designed for joint-embedding architectures, and MAE is not. The LevyScore paper also shows that the method does not work on MAEs. Furthermore, MAE is known to perform poorly in k-NN, limiting the conclusions that can be drawn from this evaluation.
  • No comparison is made with simple ensembling strategies on top of an expert model, e.g., test-time augmentation.

[1] LevyScore: A Fast Sample-Wise Confidence Score of Pretrained Joint Embedding Model, UniReps (NeurIPS Workshop) 2025

Questions for authors
  1. Why was MAE picked as the second backbone?
  2. How do Levy and LSNG compare to simple ensembling strategies?
  3. How would the per-dataset oracle perform if it was selected on the validation split instead of test?
Reviewer_yCknOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a method to route individual candidates to different pre-trained encoders. The paper presents two scores to make the routing decision, Levy score and LSNG. Levy score (proposed in a previous work) is used to make hard routing decisions, and this paper presents LSNG building on top of Levy score to have a soft aggregation across different layers. The paper presents this method for classification tasks and showcases impressive performance on image classification benchmarks using ViT-based encoders.

Critical strengths
  1. The paper presents a training-free approach to route candidates to different encoder layers.
Critical weaknesses
  1. The paper in its current state is quite difficult to understand and it's contributions are unclear. Several assumptions or claims made in the paper are not justified and requires more work. Many mathematical notations and claims are incorrect or isn't described properly. Please find the details in the subsequent comments.

  2. The paper's claims should be toned down to focus on only classification tasks. In Section 3.1, it is unclear how to extend this setup beyond classification tasks?

  3. Line 164: It is unclear how the cited works motivate or imply well-spread “embedding spread” in general. Some of the cited works analyze a specific model. In general, there are plenty of evidence showcasing that embeddings are anisotropic in nature [1, 2]

  4. Eq 6. isn’t well formatted. What does pdf mean? The paper should use a function to indicate the probability distribution under the chi-squared distribution.

  5. Section 3.3, are the embeddings generated from all layers or from specific ones for clustering?

  6. Line 188: “g⋆(x) is the cluster of candidate c’s train set whose centroid is closest to zc(x).”, closest in terms of what distance metric? Why are LSNG scores of other clusters not considered? Why is LSNG a good metric for candidate quality, there are a ton of measures that capture the skewness of spectral distributions? Why are the sets called the tangent and normal sets?

  7. Line 208: Use a numeric literal instead of “const.”

  8. Line 240: Inconsistent notation. LSNG in section 3.3 was defined as a function of z_c(x), which is missing in this setup.

  9. The section “Local Block Model” is very hard to understand. The paper switches between references in the appendix and then presenting equations in the main body, without providing an hierarchy or intuition. Many terms are not described properly like “label-relevant” and “label-irrelevant” signals are unclear, are those matrices? What are their dimensions? Overall, it is unclear how this section is related to the rest of the “LSNG” description and how wc(x)w_c(x) is computed.

[1] https://aclanthology.org/D19-1006/

[2] https://arxiv.org/pdf/2401.12143

Questions for authors

Please respond to the weakness section above.

Reviewer_kYMBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of test-time, per-sample unsupervised routing across a frozen library of pre-trained vision encoders and their intermediate layers (treating each backbone-layer pair as a candidate). Rather than committing to a single fixed backbone and layer, the authors propose selecting or weighting candidates without requiring ground-truth test labels, per-candidate score calibration, or supervised router training.

The paper studies two routing strategies. First, Levy hard routing uses the embedding norm and a chi-distribution-based typicality score to select one candidate per sample. Second, the authors introduce Local Spectral Normalized Gap (LSNG), which measures local covariance structure by comparing the leading and trailing eigenvalue blocks and uses the resulting scores to weight candidate predictions.

The experiments use 192 candidates drawn from eight backbones, including general-purpose pretrained models and task-specific fine-tuned experts. Evaluation covers Galaxy10, PathMNIST, Aircraft, a merged 119-class task, and Galaxy Zoo, whose classes were not seen by the experts during fine-tuning. The authors report substantial improvements over a DINOv2-L last-layer baseline, particularly for LSNG with linear probes. However, hard routing is vulnerable to misleading embedding norms, and soft ensembling incurs additional inference costs by evaluating multiple encoders and probes. Overall, the paper presents a promising study of label-free, sample-wise representation selection, but the strength of its claims depends on the evaluation protocol, the validity of its geometric assumptions, and the practical cost of routing.

Critical strengths
  • The paper addresses an important limitation of conventional transfer learning: selecting one backbone and layer for an entire dataset may be suboptimal when individual samples differ in domain or difficulty.

  • The proposed methods make routing decisions without training an additional supervised routing network or using test labels. This is attractive when labeled data for router training are unavailable.

  • The derivation of LSNG via local semantic-noise modeling (Theorems 7 and 8) establishes an intuitive link between local spectral volume ratios and label-aligned clusterability. The score's invariance to positive scaling and robustness to embedding dimensionality (Propositions 5 and 6) make it well-suited for weighting heterogeneous candidates.

  • The paper provides extensive experimental analysis, including corrupting the pool with random-init backbones, synthetic bad-probe injections, top-k gating sweeps, alternative probe architectures (Appendix P), non-parametric kNN evaluation (Appendix L), and scaling up to heterogeneous pools with different dimensions (Appendix S).

Critical weaknesses

Major Weaknesses

  • While the authors highlight that Levy hard routing evaluates just one probe per sample, every single candidate backbone must still run a full forward pass to generate embeddings. In the primary setup, that means computing feature maps across 8 ViT-Large models per test input—an ∼8×\sim 8\times compute increase over standard deployment. This severely undermines the practical utility of the method in real-world latency- or resource-constrained settings.

  • The large headline gains over the fixed DINOv2-L default are somewhat misleading. Because the pool is heavily populated with in-domain fine-tuned experts that beat the default by 8 to 54 percentage points, even an unweighted, plain uniform ensemble reaches nearly identical accuracy.

  • The core premise—that geometric typicality serves as a dependable proxy for candidate accuracy—proves brittle. As the Aircraft evaluation clearly illustrates, off-domain models can easily produce norm decoys with typical radial norms that trick Levy into picking incompetent representations on 83% of samples. Similarly, high local tangent variation in LSNG can easily reflect task-irrelevant nuisance factors rather than true semantic separation. The theoretical bounds rationalize why typicality might correlate with clustering under idealized models, but they do not provide formal guarantees of predictive reliability.

  • The effectiveness of LSNG soft ensembling appears tightly coupled to well-calibrated, smooth linear classifier posteriors. When evaluated with non-parametric kNN heads or multi-layer perceptrons, performance drops suddenly (e.g., losing 49.1 pp on Aircraft under kNN).

  • Because both Levy and LSNG evaluate embedding geometry in isolation, they cannot assess whether the attached classifier is actually competent. In the synthetic bad-probe experiments, attaching broken heads to valid embeddings degrades both LSNG and uniform ensembling indiscriminately. Geometry alone cannot detect or downweight poor predictions caused by misfitted probes, label noise, or training defects.

Minor Weaknesses

  • The term “training-free” could be misunderstood as requiring no fitting at all. LSNG fits unlabeled geometric statistics on training embeddings, and all candidates have separately fitted probes. “No supervised router training” would be more precise when describing the distinction.

  • In Appendix S.1 and S.2, when models of differing dimensions (384 to 2048) and coordinate scales are evaluated, Levy hard routing collapses completely (e.g., scoring 0.048 on Aircraft where the default is 0.387), severely restricting its real-world utility across arbitrary model zoos.

  • In Appendix Q (Tables 29 and 30), evaluating routing over Pythia language-model layers shows that Levy hard routing trails uniform averaging across every text benchmark, and LSNG yields only marginal gains (+0.4 pp over uniform average on the merged set). The findings appear almost entirely specific to vision representations.

  • While the proposed Lite variant (Appendix I) reduces fit-and-score overhead significantly via Gaussian projection and sample subsampling, it introduces additional hyperparameters whose stability across different domain shifts has not been broadly validated.

Questions for authors

Major Weaknesses

  • While the authors highlight that Levy hard routing evaluates just one probe per sample, every single candidate backbone must still run a full forward pass to generate embeddings. In the primary setup, that means computing feature maps across 8 ViT-Large models per test input—an ∼8×\sim 8\times compute increase over standard deployment. This severely undermines the practical utility of the method in real-world latency- or resource-constrained settings.

  • The large headline gains over the fixed DINOv2-L default are somewhat misleading. Because the pool is heavily populated with in-domain fine-tuned experts that beat the default by 8 to 54 percentage points, even an unweighted, plain uniform ensemble reaches nearly identical accuracy.

  • The core premise—that geometric typicality serves as a dependable proxy for candidate accuracy—proves brittle. As the Aircraft evaluation clearly illustrates, off-domain models can easily produce norm decoys with typical radial norms that trick Levy into picking incompetent representations on 83% of samples. Similarly, high local tangent variation in LSNG can easily reflect task-irrelevant nuisance factors rather than true semantic separation. The theoretical bounds rationalize why typicality might correlate with clustering under idealized models, but they do not provide formal guarantees of predictive reliability.

  • The effectiveness of LSNG soft ensembling appears tightly coupled to well-calibrated, smooth linear classifier posteriors. When evaluated with non-parametric kNN heads or multi-layer perceptrons, performance drops suddenly (e.g., losing 49.1 pp on Aircraft under kNN).

  • Because both Levy and LSNG evaluate embedding geometry in isolation, they cannot assess whether the attached classifier is actually competent. In the synthetic bad-probe experiments, attaching broken heads to valid embeddings degrades both LSNG and uniform ensembling indiscriminately. Geometry alone cannot detect or downweight poor predictions caused by misfitted probes, label noise, or training defects.

Minor Weaknesses

  • The term “training-free” could be misunderstood as requiring no fitting at all. LSNG fits unlabeled geometric statistics on training embeddings, and all candidates have separately fitted probes. “No supervised router training” would be more precise when describing the distinction.

  • In Appendix S.1 and S.2, when models of differing dimensions (384 to 2048) and coordinate scales are evaluated, Levy hard routing collapses completely (e.g., scoring 0.048 on Aircraft where the default is 0.387), severely restricting its real-world utility across arbitrary model zoos.

  • In Appendix Q (Tables 29 and 30), evaluating routing over Pythia language-model layers shows that Levy hard routing trails uniform averaging across every text benchmark, and LSNG yields only marginal gains (+0.4 pp over uniform average on the merged set). The findings appear almost entirely specific to vision representations.

  • While the proposed Lite variant (Appendix I) reduces fit-and-score overhead significantly via Gaussian projection and sample subsampling, it introduces additional hyperparameters whose stability across different domain shifts has not been broadly validated.

Comments on the paper

1 comment

We thank the reviewers for their time and detailed feedback. We have read the reviews carefully and appreciate the questions and suggestions. We plan to continue developing this work and will consider the points raised as we prepare the next iteration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.