- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper tackles the problem of routing test examples in an unsupervised fashion to the best (backbone, layer) tuple, under the assumption that a set of backbones and per-layer linear probes are available. They propose two algorithms to do so:
- Hard routing via the Levy radial-typicality score.
- Soft-routing via the local spectral normalized gap, or LSNG.
They compare the two approaches using a candidate pool of 8 encoders, composed of 2 general purpose models, i.e., DINOv2 ViT-L/14 and MAE ViT-L, and 6 expert models, fine-tuned on Galaxy10, PathMNIST and Aircraft, where each dataset has both a DINOv2 and MAE expert. They compare their approaches to two main baselines:
- Fixed backbone, i.e., a linear probe fitted on the last layer of DINOv2 ViT-L/14.
- The per-dataset oracle, i.e., the best (backbone, layer) probe by test accuracy.
Levy hard routing improves over the fixed backbone across 3/5 datasets, except on Aircraft and PathMNIST, where it performs worse than the fixed backbone baseline. LSNG instead provides a consistent neutral to strongly positive performance improvement in linear probing, sometimes outperforming the per-dataset oracle.
The caveat is that both hard and soft routing require evaluating all 8 backbones. Additionally, soft routing must evaluate each of the 192 (backbone, layer) probes.
Investigating which probes contribute the most to LSNG reveals that the algorithm tends to prefer expert models, with some exceptions.
The authors further show that LSNG is somewhat robust to pool corruptions and that it significantly improves the worst-class accuracy.
Critical strengths
- The paper is well written, clear, and easy to read.
- The proposed routing and ensembling techniques are training-free and mathematically motivated.
Critical weaknesses
In my opinion, the main weakness of the paper is the evaluation protocol: under the assumption that an expert model is available, a practitioner would deploy such model, making the per-dataset oracle (when selected on validation) the realistic baseline to compare the method against.
Against such a baseline, the performance improvements due to the proposed methods are negative to modest. The only clear improvement is achieved by LSNG on the merged dataset, but even then, the per-dataset oracle is a model fine-tuned on one of the three datasets that compose the merged one, and is not clearly comparable.
Other weaknesses include:
- While both methods indeed require no router and are fully unsupervised, they require a forward from every pool member.
- It's unclear why MAE models were chosen for the pool. LevyScore [1] was designed for joint-embedding architectures, and MAE is not. The LevyScore paper also shows that the method does not work on MAEs. Furthermore, MAE is known to perform poorly in k-NN, limiting the conclusions that can be drawn from this evaluation.
- No comparison is made with simple ensembling strategies on top of an expert model, e.g., test-time augmentation.
[1] LevyScore: A Fast Sample-Wise Confidence Score of Pretrained Joint Embedding Model, UniReps (NeurIPS Workshop) 2025
Questions for authors
- Why was MAE picked as the second backbone?
- How do Levy and LSNG compare to simple ensembling strategies?
- How would the per-dataset oracle perform if it was selected on the validation split instead of test?
We thank the reviewers for their time and detailed feedback. We have read the reviews carefully and appreciate the questions and suggestions. We plan to continue developing this work and will consider the points raised as we prepare the next iteration.