- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free framework for fine-grained recognition across multiple domains. It combines four frozen encoders. Five labeled references per class are used to calibrate three parts: domain-specific expert weights, covariance-based visual comparison, and the fusion of visual and semantic scores. At inference, the method estimates the domain through retrieval and uses this estimate to select the calibration statistics. All classes remain candidates. The evaluation covers 598 mixed-domain classes, 10,000 species, and 4,431 product identities.
Critical strengths
- Practical motivation and broad evaluation. The framework makes use of complementary pretrained models in several recognition settings. The gains on Aircraft and SOP are encouraging.
- Careful calibration. The nested held-out procedure recomputes the reference statistics after removing the outer held-out samples. This reduces self-evaluation bias.
- Useful ablations. The experiments study the main components, expert subsets, and the number of references. The authors also report negative results and discuss measurement limits.
Critical weaknesses
- Heuristic competence estimation and limited novelty. The “expert competence” is simply held-out reference accuracy turned into softmax weights. This is an empirical heuristic. The paper does not explain why it should produce a good expert combination. Competence is measured with cosine similarity, while the final classifier is covariance-based. This mismatch further weakens the rationale. Overall, the method reads as an engineering combination of existing models and statistical tools. It does not offer a clear methodological advance on the specific challenges of fine-grained recognition.
- Gains are not well isolated. The paper needs stronger controls that use the same classifier and the same calibration budget. Without them, the effect of the proposed calibration cannot be separated from a generic ensemble effect. Two baselines would help: a single expert selected on the references, and a globally weighted ensemble. These would show whether the full domain-conditioned design is necessary.
- Uncertainty is not reported. The gains on CUB, Flowers, and iNaturalist are small. However, the variance caused by reference sampling is not reported. On iNaturalist, the gain over APE is only 0.114 percentage points. This result needs matched-seed evaluation.
- No routing analysis. Domain prediction drives several core components. Yet the paper reports neither routing accuracy nor oracle-domain results. The impact of routing errors therefore remains unknown.
- Unclear benefit over a single model. Each query must pass through several encoders. This raises inference cost and deployment complexity. The reported efficiency, however, excludes feature extraction. The paper should compare end-to-end cost against strong single-expert and two-expert systems. Without this, it is hard to judge whether the accuracy gains justify the extra latency, memory, and maintenance.
- Incomplete related work. The paper overlooks vocabulary-free fine-grained recognition methods such as FineR [1] and FiNDR [2]. These works recognize categories without predefined names and are relevant to the paper’s broader motivation. The authors should explain how their setting differs, since it relies on a reference-defined vocabulary and labeled support samples. Direct numerical comparison would require aligned protocols.
[1] Liu et al. Democratizing Fine-grained Visual Recognition with Large Language Models. ICLR, 2024.
[2] Demidov et al. Thinking Beyond Labels: Vocabulary-Free Fine-Grained Recognition using Reasoning-Augmented LMMs. CVPR, 2026.
Questions for authors
- What justifies turning held-out accuracy into softmax expert weights? If competence is estimated with the final covariance-based classifier, do the expert rankings or the recognition accuracy improve?
- Under the same classifier and calibration procedure, how does the mixture compare with a reference-selected single expert and with global expert weights? The results on iNaturalist are of particular interest.
- How many reference seeds are used? Can the authors report matched-seed means and variance against the strongest baselines?
- What is the domain-routing accuracy? How large is the gap between predicted-domain and oracle-domain recognition?
- What are the end-to-end inference and calibration costs compared with single-expert and two-expert setups?
- How is MeanTop3 computed when nested replay leaves fewer than three references? This case arises in the three-reference experiment.
12 comments
W1&Q1 (continued)
Classifier-matched competence estimation.
We also investigated the reviewer's concern that visual competence is estimated using cosine similarity, whereas the final visual classifier uses regularized covariance geometry.
To examine this potential mismatch, we implemented a classifier-matched alternative in which expert competence is estimated using the same covariance-based comparison mechanism as the final classifier.
During nested reference replay, covariance statistics are recomputed from the retained reference images, ensuring that held-out samples are excluded from their corresponding parameter estimation.
The results are:
| Benchmark | Cosine Competence (%) | Covariance Competence (%) | Difference (pp) |
|---|---|---|---|
| SOP, default | 87.461 | 87.613 | +0.152 |
| SOP, | 87.552 | 87.770 | +0.218 |
| 4-Domain Joint, | 92.623 | 92.674 | +0.051 |
| iNat-10k, | 89.564 | 89.535 | -0.029 |
On SOP, classifier-matched covariance competence improves recognition by +0.152 pp at the default temperature and by +0.218 pp at , based on the displayed accuracies.
On the four-domain benchmark, it provides a smaller improvement of +0.051 pp. On iNat-10k, cosine competence remains slightly better by 0.029 pp.
The results show that classifier-matched competence can improve recognition in certain settings, while the default cosine-based estimator remains competitive and avoids additional covariance inversions in the nested competence-estimation stage.
Methodological contribution.
Our framework connects three reference-derived components:
- Domain-specific expert competence, which determines how heterogeneous pretrained encoders contribute to recognition.
- Domain-conditioned comparison geometry, which uses within-class reference variation to construct discriminative visual distances.
- Cross-fitted visual–semantic calibration, which determines the appropriate balance between visual and semantic evidence under the actual inference-time routing procedure.
These components are calibrated for the same reference-defined recognition task rather than treated as independent global ensemble parameters.
We will clarify the entropy-regularized interpretation and add the classifier-matched covariance competence experiments to the revised manuscript and supplementary material.
W2&Q2: Comparison of expert composition against independently deployed single experts and globally shared expert weights.
We evaluated the Full four-expert composition against independently deployed single-expert systems and globally shared expert-weight configurations under the same regularized covariance-classification framework and 5-shot reference budget.
An Independent Single Expert activates only one encoder during query inference. Where applicable, the encoder is selected using held-out reference performance, and its routing, covariance statistics, and calibration are constructed using that selected encoder.
A Globally Shared Expert Weights configuration activates all four experts but replaces domain-specific competence weights with one shared expert-weight vector.
1. Full composition versus independently deployed single experts.
We evaluated the independently deployed controls across five matched reference splits (seeds 1–5).
| Benchmark | Independent Single-Expert Selection / Deployment | Independent Single Expert (%) | Full Four-Expert Method (%) | Gain of Full (pp) | Paired 95% CI |
|---|---|---|---|---|---|
| 4-Domain Joint (598 classes) | Reference-selected: FG-CLIP2 (seed 1); DINOv3 (seeds 2–5) | 86.780 | 92.623 | +5.843 | [+5.356, +6.330] |
| SOP (4,431 identities) | Reference-selected: FG-CLIP2 (all five seeds) | 86.636 | 87.461 | +0.825 | [+0.677, +0.973] |
| iNat-10k (10,000 species) | Pre-specified BioCLIP2; independently deployed for all five seeds | 89.517 | 89.564 | +0.047 | [-0.034, +0.128] |
The four-domain and SOP single experts are selected using their corresponding reference splits, whereas BioCLIP2 is pre-specified as the independent single-expert deployment baseline on iNat-10k.
On the four-domain benchmark, Full improves over the independently deployed single expert by +5.843 pp. On SOP, the corresponding improvement is +0.825 pp.
On iNat-10k, BioCLIP2-only reaches 89.517%, compared with 89.564% for Full. The mean difference is +0.047 pp, with a five-split paired confidence interval crossing zero.
For clarity, the original single-backbone result in the initial submission reported 85.967% for BioCLIP2 on iNat-10k using seed 1. The updated local evaluation over seeds 0, 1, and 2 gives 86.023 ± 0.076%. These values correspond to different reference-split aggregation protocols. They should also be distinguished from the 89.517% independently deployed BioCLIP2-only result above, which uses the reference-calibrated covariance classification pipeline.
2. Full composition versus globally shared expert weights.
We next evaluated the contribution of domain-specific competence weighting by comparing Full with a four-expert configuration using one globally shared expert-weight vector.
| Benchmark | Global Shared Expert Weights (%) | Full Four-Expert Method (%) | Gain of Full (pp) |
|---|---|---|---|
| 4-Domain Joint | 91.979 | 92.623 | +0.644 |
| SOP | 87.473 | 87.461 | -0.012 |
| iNat-10k | 89.573 | 89.564 | -0.009 |
On the four-domain benchmark, the complete method improves over global expert weighting by +0.644 pp.
The four domains have different strongest individual encoders, as shown by the independent single-expert results. Domain-specific competence weighting can adapt to these changes, whereas one shared weight vector cannot explicitly represent different expert rankings across domains.
On SOP and iNat-10k, the global shared-weight configurations perform comparably to domain-specific competence weighting, with mean differences of -0.012 pp and -0.009 pp, respectively.
3. Full calibration versus jointly simplified configuration.
The globally shared expert-weight control changes the expert-weighting policy while retaining the other applicable calibration components. We additionally evaluated a jointly simplified configuration that replaces domain-conditioned covariance with global covariance, uses equal visual expert weights, and, where applicable, applies one globally shared visual–semantic branch weight.
| Benchmark | Jointly Simplified (%) | Full Method (%) | Gain of Full (pp) |
|---|---|---|---|
| 4-Domain Joint | 91.906 | 92.623 | +0.717 |
| SOP | 82.887 | 87.461 | +4.574 |
| iNat-10k | 88.888 | 89.564 | +0.676 |
The Full method improves over the jointly simplified configuration by +4.574 pp on SOP and +0.676 pp on iNat-10k.
These comparisons distinguish the contribution of domain-specific expert weighting from that of the complete calibration pipeline. On SOP and iNat-10k, globally shared expert weights alone are competitive, while the full combination of reference-derived geometry, competence estimation, and calibration provides additional gains over the jointly simplified configurations.
Response to Reviewer 5Fd6
We thank the reviewer for the detailed questions concerning competence weighting, single-expert comparisons, routing, and deployment efficiency. We conducted additional controlled experiments to examine these aspects and clarify the methodological design.
Our competence-weighting strategy is motivated by the observation that heterogeneous pretrained encoders exhibit different discriminative capabilities across recognition domains. A small labeled reference set provides direct evidence of these domain-dependent capabilities, enabling expert contributions to be calibrated without updating the pretrained models.
Rather than selecting a single expert through a hard decision rule, we transform the reference-derived competence scores into normalized weights:
πm,g=∑ℓ=1Mexp(rℓ,g/τ)exp(rm,g/τ).
Here, m indexes an expert, g denotes a coarse recognition domain, M is the number of available experts, and rm,g represents the discriminative competence of expert m on domain g, estimated from held-out reference predictions. The temperature τ>0 controls the concentration of the resulting expert weights. The semantic competence weights ωm,g are constructed analogously using reference-derived image–text matching performance.
Theoretical motivation: Entropy-regularized expert allocation.
The above softmax form is the exact solution to the following entropy-regularized optimization problem:
\pi_g^*=\underset{\pi_g\in\Delta^{M-1}}{\operatorname{argmax}}\left\\{\sum_{m=1}^{M}\pi_{m,g}r_{m,g}+\tau\mathcal{H}(\pi_g)\right\\}.
Here, πg=(π1,g,…,πM,g) is the expert-weight vector, and ΔM−1 denotes the probability simplex, requiring nonnegative weights that sum to one. The entropy term is defined as:
H(πg)=−∑m=1Mπm,glogπm,g.
The first term rewards assigning greater weight to experts with stronger reference-derived discrimination. The second term controls the concentration of the expert allocation and encourages a smooth distribution over potentially complementary experts.
To derive the optimizer, introduce a Lagrange multiplier λ for the normalization constraint:
L=∑m=1Mπm,grm,g−τ∑m=1Mπm,glogπm,g+λ(∑m=1Mπm,g−1).
Setting the derivative with respect to πm,g to zero gives:
rm,g−τ(logπm,g+1)+λ=0.
Rearranging yields:
πm,g∝exp(rm,g/τ).
Enforcing the simplex normalization constraint gives exactly:
πm,g∗=∑ℓ=1Mexp(rℓ,g/τ)exp(rm,g/τ).
Since the objective is strictly concave for τ>0, this solution is the unique maximizer.
This formulation balances competence-based specialization and expert diversity. Smaller temperatures emphasize experts with stronger reference-derived discrimination, whereas larger temperatures distribute weight more evenly. An expert with lower standalone competence can still retain a contribution when its representation provides complementary visual or semantic information.
Our framework uses this entropy-regularized allocation together with domain-conditioned comparison geometry and nested reference-only visual–semantic calibration. These components jointly determine how expert evidence is estimated, compared, and combined for the target recognition task.