acceptodds
Under review as a conference paper at ICLR 2027

Reference-Calibrated Expert Composition for Open-Domain Fine-Grained Recognition

Abstract

A long-standing goal of fine-grained visual recognition is to build a general-purpose system that recognizes fine-grained categories across a broad and growing range of domains. However, the visual cues needed to distinguish closely related categories vary across domains, and models specialized through task- or domain-specific training may perform less reliably outside their target domains. Extending coverage through this approach therefore requires repeated supervision and model adaptation, making broad fine-grained recognition difficult to scale. Visual models pretrained on large-scale data offer an alternative foundation, as their representations already capture rich fine-grained distinctions. Our analysis shows that the examined models have uneven capabilities across domains but exhibit complementary strengths. These observations motivate composing pretrained experts whose strengths differ across domains. We propose a training-free framework that reads domain-specific capability from a small set of labeled reference images and composes heterogeneous frozen experts accordingly. Specifically, references calibrate domain-specific expert contributions for visual comparison and image–name matching, while within-class feature statistics define the visual comparison geometry. Held-out reference predictions further calibrate the balance between visual and semantic evidence. All pretrained models remain frozen, allowing new domains and new experts to be incorporated without retraining. We evaluate the framework on 598 mixed-domain classes, 10,000 species, and 4,431 product identities, demonstrating the utility of reference-calibrated composition across heterogeneous recognition tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fine-Grained Visual Recognition, Open-Domain Recognition, Training-Free Learning, Expert Composition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5Fd6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework for fine-grained recognition across multiple domains. It combines four frozen encoders. Five labeled references per class are used to calibrate three parts: domain-specific expert weights, covariance-based visual comparison, and the fusion of visual and semantic scores. At inference, the method estimates the domain through retrieval and uses this estimate to select the calibration statistics. All classes remain candidates. The evaluation covers 598 mixed-domain classes, 10,000 species, and 4,431 product identities.

Critical strengths
  1. Practical motivation and broad evaluation. The framework makes use of complementary pretrained models in several recognition settings. The gains on Aircraft and SOP are encouraging.
  2. Careful calibration. The nested held-out procedure recomputes the reference statistics after removing the outer held-out samples. This reduces self-evaluation bias.
  3. Useful ablations. The experiments study the main components, expert subsets, and the number of references. The authors also report negative results and discuss measurement limits.
Critical weaknesses
  1. Heuristic competence estimation and limited novelty. The “expert competence” is simply held-out reference accuracy turned into softmax weights. This is an empirical heuristic. The paper does not explain why it should produce a good expert combination. Competence is measured with cosine similarity, while the final classifier is covariance-based. This mismatch further weakens the rationale. Overall, the method reads as an engineering combination of existing models and statistical tools. It does not offer a clear methodological advance on the specific challenges of fine-grained recognition.
  2. Gains are not well isolated. The paper needs stronger controls that use the same classifier and the same calibration budget. Without them, the effect of the proposed calibration cannot be separated from a generic ensemble effect. Two baselines would help: a single expert selected on the references, and a globally weighted ensemble. These would show whether the full domain-conditioned design is necessary.
  3. Uncertainty is not reported. The gains on CUB, Flowers, and iNaturalist are small. However, the variance caused by reference sampling is not reported. On iNaturalist, the gain over APE is only 0.114 percentage points. This result needs matched-seed evaluation.
  4. No routing analysis. Domain prediction drives several core components. Yet the paper reports neither routing accuracy nor oracle-domain results. The impact of routing errors therefore remains unknown.
  5. Unclear benefit over a single model. Each query must pass through several encoders. This raises inference cost and deployment complexity. The reported efficiency, however, excludes feature extraction. The paper should compare end-to-end cost against strong single-expert and two-expert systems. Without this, it is hard to judge whether the accuracy gains justify the extra latency, memory, and maintenance.
  6. Incomplete related work. The paper overlooks vocabulary-free fine-grained recognition methods such as FineR [1] and FiNDR [2]. These works recognize categories without predefined names and are relevant to the paper’s broader motivation. The authors should explain how their setting differs, since it relies on a reference-defined vocabulary and labeled support samples. Direct numerical comparison would require aligned protocols.

[1] Liu et al. Democratizing Fine-grained Visual Recognition with Large Language Models. ICLR, 2024.

[2] Demidov et al. Thinking Beyond Labels: Vocabulary-Free Fine-Grained Recognition using Reasoning-Augmented LMMs. CVPR, 2026.

Questions for authors
  1. What justifies turning held-out accuracy into softmax expert weights? If competence is estimated with the final covariance-based classifier, do the expert rankings or the recognition accuracy improve?
  2. Under the same classifier and calibration procedure, how does the mixture compare with a reference-selected single expert and with global expert weights? The results on iNaturalist are of particular interest.
  3. How many reference seeds are used? Can the authors report matched-seed means and variance against the strongest baselines?
  4. What is the domain-routing accuracy? How large is the gap between predicted-domain and oracle-domain recognition?
  5. What are the end-to-end inference and calibration costs compared with single-expert and two-expert setups?
  6. How is MeanTop3 computed when nested replay leaves fewer than three references? This case arises in the three-reference experiment.
12 comments

Response to Reviewer 5Fd6

We thank the reviewer for the detailed questions concerning competence weighting, single-expert comparisons, routing, and deployment efficiency. We conducted additional controlled experiments to examine these aspects and clarify the methodological design.

W1&Q1: The rationale for softmax competence weighting and classifier-matched covariance competence.

Our competence-weighting strategy is motivated by the observation that heterogeneous pretrained encoders exhibit different discriminative capabilities across recognition domains. A small labeled reference set provides direct evidence of these domain-dependent capabilities, enabling expert contributions to be calibrated without updating the pretrained models.

Rather than selecting a single expert through a hard decision rule, we transform the reference-derived competence scores into normalized weights:

πm,g=exp⁡(rm,g/τ)∑ℓ=1Mexp⁡(rℓ,g/τ).\pi_{m,g}=\frac{\exp(r_{m,g}/\tau)}{\sum_{\ell=1}^{M}\exp(r_{\ell,g}/\tau)}.

Here, mm indexes an expert, gg denotes a coarse recognition domain, MM is the number of available experts, and rm,gr_{m,g} represents the discriminative competence of expert mm on domain gg, estimated from held-out reference predictions. The temperature τ>0\tau>0 controls the concentration of the resulting expert weights. The semantic competence weights ωm,g\omega_{m,g} are constructed analogously using reference-derived image–text matching performance.

Theoretical motivation: Entropy-regularized expert allocation.

The above softmax form is the exact solution to the following entropy-regularized optimization problem:

\pi_g^*=\underset{\pi_g\in\Delta^{M-1}}{\operatorname{argmax}}\left\\{\sum_{m=1}^{M}\pi_{m,g}r_{m,g}+\tau\mathcal{H}(\pi_g)\right\\}.

Here, πg=(π1,g,…,πM,g)\pi_g=(\pi_{1,g},\ldots,\pi_{M,g}) is the expert-weight vector, and ΔM−1\Delta^{M-1} denotes the probability simplex, requiring nonnegative weights that sum to one. The entropy term is defined as:

H(πg)=−∑m=1Mπm,glog⁡πm,g.\mathcal{H}(\pi_g)=-\sum_{m=1}^{M}\pi_{m,g}\log\pi_{m,g}.

The first term rewards assigning greater weight to experts with stronger reference-derived discrimination. The second term controls the concentration of the expert allocation and encourages a smooth distribution over potentially complementary experts.

To derive the optimizer, introduce a Lagrange multiplier λ\lambda for the normalization constraint:

L=∑m=1Mπm,grm,g−τ∑m=1Mπm,glog⁡πm,g+λ(∑m=1Mπm,g−1).\mathcal{L}=\sum_{m=1}^{M}\pi_{m,g}r_{m,g}-\tau\sum_{m=1}^{M}\pi_{m,g}\log\pi_{m,g}+\lambda\left(\sum_{m=1}^{M}\pi_{m,g}-1\right).

Setting the derivative with respect to πm,g\pi_{m,g} to zero gives:

rm,g−τ(log⁡πm,g+1)+λ=0.r_{m,g}-\tau(\log\pi_{m,g}+1)+\lambda=0.

Rearranging yields:

πm,g∝exp⁡(rm,g/τ).\pi_{m,g}\propto\exp(r_{m,g}/\tau).

Enforcing the simplex normalization constraint gives exactly:

πm,g∗=exp⁡(rm,g/τ)∑ℓ=1Mexp⁡(rℓ,g/τ).\boxed{\pi_{m,g}^*=\frac{\exp(r_{m,g}/\tau)}{\sum_{\ell=1}^{M}\exp(r_{\ell,g}/\tau)}}.

Since the objective is strictly concave for τ>0\tau>0, this solution is the unique maximizer.

This formulation balances competence-based specialization and expert diversity. Smaller temperatures emphasize experts with stronger reference-derived discrimination, whereas larger temperatures distribute weight more evenly. An expert with lower standalone competence can still retain a contribution when its representation provides complementary visual or semantic information.

Our framework uses this entropy-regularized allocation together with domain-conditioned comparison geometry and nested reference-only visual–semantic calibration. These components jointly determine how expert evidence is estimated, compared, and combined for the target recognition task.

W1&Q1 (continued)

Classifier-matched competence estimation.

We also investigated the reviewer's concern that visual competence is estimated using cosine similarity, whereas the final visual classifier uses regularized covariance geometry.

To examine this potential mismatch, we implemented a classifier-matched alternative in which expert competence is estimated using the same covariance-based comparison mechanism as the final classifier.

During nested reference replay, covariance statistics are recomputed from the retained reference images, ensuring that held-out samples are excluded from their corresponding parameter estimation.

The results are:

BenchmarkCosine Competence (%)Covariance Competence (%)Difference (pp)
SOP, default τ=0.1\tau=0.187.46187.613+0.152
SOP, τ=0.05\tau=0.0587.55287.770+0.218
4-Domain Joint, τ=0.1\tau=0.192.62392.674+0.051
iNat-10k, τ=0.1\tau=0.189.56489.535-0.029

On SOP, classifier-matched covariance competence improves recognition by +0.152 pp at the default temperature and by +0.218 pp at τ=0.05\tau=0.05, based on the displayed accuracies.

On the four-domain benchmark, it provides a smaller improvement of +0.051 pp. On iNat-10k, cosine competence remains slightly better by 0.029 pp.

The results show that classifier-matched competence can improve recognition in certain settings, while the default cosine-based estimator remains competitive and avoids additional covariance inversions in the nested competence-estimation stage.

Methodological contribution.

Our framework connects three reference-derived components:

  1. Domain-specific expert competence, which determines how heterogeneous pretrained encoders contribute to recognition.
  2. Domain-conditioned comparison geometry, which uses within-class reference variation to construct discriminative visual distances.
  3. Cross-fitted visual–semantic calibration, which determines the appropriate balance between visual and semantic evidence under the actual inference-time routing procedure.

These components are calibrated for the same reference-defined recognition task rather than treated as independent global ensemble parameters.

We will clarify the entropy-regularized interpretation and add the classifier-matched covariance competence experiments to the revised manuscript and supplementary material.

W2&Q2: Comparison of expert composition against independently deployed single experts and globally shared expert weights.

We evaluated the Full four-expert composition against independently deployed single-expert systems and globally shared expert-weight configurations under the same regularized covariance-classification framework and 5-shot reference budget.

An Independent Single Expert activates only one encoder during query inference. Where applicable, the encoder is selected using held-out reference performance, and its routing, covariance statistics, and calibration are constructed using that selected encoder.

A Globally Shared Expert Weights configuration activates all four experts but replaces domain-specific competence weights with one shared expert-weight vector.

1. Full composition versus independently deployed single experts.

We evaluated the independently deployed controls across five matched reference splits (seeds 1–5).

BenchmarkIndependent Single-Expert Selection / DeploymentIndependent Single Expert (%)Full Four-Expert Method (%)Gain of Full (pp)Paired 95% CI
4-Domain Joint (598 classes)Reference-selected: FG-CLIP2 (seed 1); DINOv3 (seeds 2–5)86.78092.623+5.843[+5.356, +6.330]
SOP (4,431 identities)Reference-selected: FG-CLIP2 (all five seeds)86.63687.461+0.825[+0.677, +0.973]
iNat-10k (10,000 species)Pre-specified BioCLIP2; independently deployed for all five seeds89.51789.564+0.047[-0.034, +0.128]

The four-domain and SOP single experts are selected using their corresponding reference splits, whereas BioCLIP2 is pre-specified as the independent single-expert deployment baseline on iNat-10k.

On the four-domain benchmark, Full improves over the independently deployed single expert by +5.843 pp. On SOP, the corresponding improvement is +0.825 pp.

On iNat-10k, BioCLIP2-only reaches 89.517%, compared with 89.564% for Full. The mean difference is +0.047 pp, with a five-split paired confidence interval crossing zero.

For clarity, the original single-backbone result in the initial submission reported 85.967% for BioCLIP2 on iNat-10k using seed 1. The updated local evaluation over seeds 0, 1, and 2 gives 86.023 ± 0.076%. These values correspond to different reference-split aggregation protocols. They should also be distinguished from the 89.517% independently deployed BioCLIP2-only result above, which uses the reference-calibrated covariance classification pipeline.

2. Full composition versus globally shared expert weights.

We next evaluated the contribution of domain-specific competence weighting by comparing Full with a four-expert configuration using one globally shared expert-weight vector.

BenchmarkGlobal Shared Expert Weights (%)Full Four-Expert Method (%)Gain of Full (pp)
4-Domain Joint91.97992.623+0.644
SOP87.47387.461-0.012
iNat-10k89.57389.564-0.009

On the four-domain benchmark, the complete method improves over global expert weighting by +0.644 pp.

The four domains have different strongest individual encoders, as shown by the independent single-expert results. Domain-specific competence weighting can adapt to these changes, whereas one shared weight vector cannot explicitly represent different expert rankings across domains.

On SOP and iNat-10k, the global shared-weight configurations perform comparably to domain-specific competence weighting, with mean differences of -0.012 pp and -0.009 pp, respectively.

3. Full calibration versus jointly simplified configuration.

The globally shared expert-weight control changes the expert-weighting policy while retaining the other applicable calibration components. We additionally evaluated a jointly simplified configuration that replaces domain-conditioned covariance with global covariance, uses equal visual expert weights, and, where applicable, applies one globally shared visual–semantic branch weight.

BenchmarkJointly Simplified (%)Full Method (%)Gain of Full (pp)
4-Domain Joint91.90692.623+0.717
SOP82.88787.461+4.574
iNat-10k88.88889.564+0.676

The Full method improves over the jointly simplified configuration by +4.574 pp on SOP and +0.676 pp on iNat-10k.

These comparisons distinguish the contribution of domain-specific expert weighting from that of the complete calibration pipeline. On SOP and iNat-10k, globally shared expert weights alone are competitive, while the full combination of reference-derived geometry, competence estimation, and calibration provides additional gains over the jointly simplified configurations.

Reviewer_N8C2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework for fine-grained recognition over a shared vocabulary spanning multiple coarse domains. Given a small labeled reference set for each class, the method combines four frozen pretrained experts using domain-specific visual and semantic competence weights, a regularized within-class covariance metric, and a cross-fitted mixture of visual and class-name evidence. At inference time, reference retrieval predicts a coarse-domain context, which selects the calibration statistics while retaining the complete candidate vocabulary. Experiments cover a joint 598-class benchmark, 10,000 iNaturalist species, and 4,431 product identities. The strongest improvements are observed on Aircraft and SOP. The main contribution is the integration of reference-based expert composition, domain-conditioned comparison geometry, and visual–semantic calibration into a unified recognition framework.

Critical strengths
  1. A coherent framework with careful calibration. Separately estimating visual and image–name competence is sensible, since an expert’s ability to discriminate reference images need not match its image–text alignment quality. The nested reference-only procedure in Section 3.3 is a positive aspect: the relevant statistics are recomputed without the outer-held-out examples, and branch calibration incorporates the inference-time routing procedure.
  2. Meaningful improvements on Aircraft and product recognition. The reported gains over GDA are approximately 4.98 percentage points on Aircraft and 3.54 points on SOP. The SOP result provides useful evidence because identity-level text is unavailable, and improvements over GDA are observed across all twelve reported product groups. These results demonstrate practical value in settings where the method is effective.
  3. Relevant controlled ablations are already provided. Table 3 examines covariance geometry, expert weighting, and branch calibration under the same expert pool. Table 5 evaluates all fifteen nonempty visual expert subsets, with the single-expert variants retaining the covariance-based classifier. These experiments provide evidence beyond comparisons with unadapted backbones. The authors also appropriately acknowledge that adding every expert does not consistently improve performance.
Critical weaknesses
  1. The incremental contribution of the complete design is limited and unevenly supported. The method introduces several domain-specific weighting and calibration choices on top of established covariance-based adaptation and ensembling ideas. Although the Aircraft results support these choices, their additional benefits are much smaller elsewhere. For example, on Cars, the full method improves over global covariance, equal visual weights, and a global branch weight by only 0.08, 0.21, and 0.09 percentage points, respectively. On CUB, a global branch weight is only 0.02 points below the full method, while Flowers is essentially saturated across configurations (Table 3). These ablations are useful, but they leave open whether several simplifications could be combined with little loss. A jointly simplified configuration using the same expert pool would clarify how much additional value the complete calibration pipeline provides.
  2. The benefits of heterogeneous composition are not sufficiently isolated on the large-vocabulary tasks. The calibrated single-expert and subset analyses in Table 5 are limited to the four-domain benchmark. On iNat, the visual-only ensemble achieves 85.42%, below the BioCLIP2 prototype baseline at 85.97%, so the results do not directly establish a benefit from heterogeneous visual composition in this setting. On SOP, the improvement is substantial, but a single-expert control using the same comparison geometry would help distinguish gains from the classifier itself from gains due to combining experts. These controls would make the scope of the central expert-composition contribution clearer.
  3. Reference-sampling variability is insufficiently reported for the small performance margins. The main comparisons rely on a single reference seed, whereas Tables 3 and 5 aggregate matched reference splits without specifying the number of splits or reporting dispersion. This matters particularly for the 0.114-percentage-point improvement over APE on iNat, the approximately 0.14-point improvement over GDA on CUB, and the near-tied Flowers results. The paper does not claim statistical significance, but paired results across several reference samples are still needed to assess the stability of these small advantages. This concern does not negate the larger Aircraft and SOP improvements.
Questions for authors
  1. Using the same four experts and reference budget, how does a jointly simplified configuration—global covariance, equal visual expert weights, and one reference-selected global branch weight—compare with the full method? This would complement the existing individual component ablations and clarify the incremental value of the complete design.
  2. Could the authors report a BioCLIP2-only version on iNat and a reference-selected single-expert version on SOP, retaining the same comparison geometry and, where applicable, the same branch-calibration procedure? How much additional improvement does heterogeneous composition provide over these controls?
  3. How many reference splits are aggregated in Tables 3 and 5, which seeds are used, and what is the variability of the paired performance differences? Please also clarify how the aggregation protocol relates to the single-seed main comparisons.
8 comments

Response to Reviewer N8C2

We sincerely thank the reviewer for the careful assessment of our framework and for recognizing the strengths of our nested reference-only calibration procedure and the improvements on Aircraft and SOP.

We conducted additional controlled experiments to clarify the incremental value of the complete design, distinguish heterogeneous expert composition from single-expert recognition, and quantify reference-sampling variability. We also provide further evidence that the benefits of expert composition are particularly important in open-domain recognition, where the system must accommodate multiple heterogeneous recognition domains within a shared candidate vocabulary.

W1&Q1: Comparison with a jointly simplified configuration.

We implemented the requested jointly simplified baseline using the same four experts, the same candidate vocabulary, and the same 5-shot reference budget.

The simplified configuration simultaneously replaces domain-conditioned covariance with global covariance, uses equal visual expert weights, and selects one globally shared visual–semantic branch weight.

Benchmark / DomainJointly Simplified (%)Full Method (%)Accuracy Difference (pp)
4-Domain Joint Overall (598 classes)91.90692.623+0.717
Aircraft69.79972.877+3.078
Cars94.27794.991+0.714
CUB93.01393.165+0.152
Flowers99.74699.717-0.029
iNat-10k (10,000 species)88.88889.564+0.676
SOP (4,431 identities)82.88787.461+4.574

The complete framework achieves clear improvements on all three overall benchmarks. The gains are particularly substantial on Aircraft (+3.078 pp) and SOP (+4.574 pp), where domain-dependent comparison geometry and expert calibration are especially valuable.

To further understand the effect of joint simplification, we performed a 2×22\times2 covariance–competence decomposition on SOP:

Visual GeometryExpert WeightingAccuracy (%)
Domain CovarianceReference-Derived Competence87.461
Global CovarianceReference-Derived Competence83.762
Domain CovarianceEqual Visual Weights86.917
Global CovarianceEqual Visual Weights82.887

Replacing domain covariance alone reduces accuracy by 3.699 pp, while replacing competence weighting alone reduces accuracy by 0.544 pp. Applying both simplifications jointly results in a 4.574 pp reduction. The effects are therefore not simply additive, further demonstrating interaction between comparison geometry and expert calibration.

For SOP, identity-level semantic labels are unavailable, so the effective joint simplification concerns visual geometry and expert weighting.

These experiments demonstrate that the complete reference-calibration pipeline provides meaningful additional value beyond conventional global covariance and uniform expert weighting. We will include the jointly simplified baseline and the component interaction analysis in the revised manuscript and supplementary material.

W2&Q2: Independently deployable, reference-selected single-expert controls on SOP and iNat-10k.

We thank the reviewer for requesting a clearer separation between the contribution of heterogeneous expert composition and the underlying covariance-based classifier.

To address this question directly, we evaluated independently deployable single-expert systems, rather than single-expert variants that share the routing or calibration statistics of a multi-expert system.

For each reference split, the single expert is selected according to held-out reference performance, without using test labels, where reference-based selection is applicable. The selected encoder then independently performs the entire recognition pipeline, including reference retrieval, coarse-domain routing, covariance estimation, competence calibration, and final classification.

Importantly, these systems activate only one encoder during query inference, while retaining the same 5-shot reference budget, full candidate vocabulary, and regularized covariance-based classification framework.

Direct comparison with independently deployed single experts.

Across five matched reference splits (seeds 1–5), we obtain:

BenchmarkIndependent Single-Expert Selection / DeploymentIndependent Single Expert (%)Full Four-Expert Method (%)Gain of Full (pp)Paired 95% CI
4-Domain Joint (598 classes)Reference-selected: FG-CLIP2 (seed 1); DINOv3 (seeds 2–5)86.78092.623+5.843[+5.356, +6.330]
SOP (4,431 identities)Reference-selected: FG-CLIP2 (all five seeds)86.63687.461+0.825[+0.677, +0.973]
iNat-10k (10,000 species)Pre-specified BioCLIP2; independently deployed for all five seeds89.51789.564+0.047[-0.034, +0.128]

The selection strategy is explicitly distinguished across benchmarks. The four-domain and SOP single experts are selected using the corresponding reference splits, whereas BioCLIP2 is pre-specified as the independent single-expert deployment baseline on iNat-10k.

The results reveal an important task-dependent pattern.

On the four-domain joint benchmark, heterogeneous expert composition provides a substantial +5.843 pp improvement over the reference-selected independent single-expert system. This demonstrates the considerable additional recognition capability obtained by combining experts with different domain-specific strengths.

On SOP, the complete method also achieves a +0.825 pp improvement over the independently deployed FG-CLIP2 system.

On iNat-10k, the complete method reaches 89.564%, compared with 89.517% for independently deployed BioCLIP2-only. The difference is +0.047 pp, with the five-split paired confidence interval crossing zero. Thus, the additional composition benefit is limited on this particular specialist-dominated benchmark.

These comparisons directly demonstrate that our framework's contribution is not limited to improving the classification geometry of one pretrained backbone. In the heterogeneous four-domain setting, combining experts provides substantial additional recognition capability beyond a carefully reference-selected single encoder.

W2&Q2 (continued): Why heterogeneous composition is particularly important for open-domain recognition.

Our primary motivation is not to optimize a single specialized dataset in isolation, but to construct a general recognition system capable of accommodating multiple heterogeneous fine-grained domains.

The four-domain benchmark jointly evaluates 598 candidate classes spanning Aircraft, Cars, CUB, and Flowers. Each query is classified against the same complete 598-class vocabulary, rather than being restricted to a small domain-specific candidate set.

These domains exhibit markedly different discriminative requirements:

  • Aircraft recognition requires sensitivity to subtle structural differences among visually similar aircraft models.
  • Cars recognition benefits from detailed appearance information and semantic make/model knowledge.
  • Bird recognition requires fine-grained biological cues, for which specialized pretrained encoders can be particularly effective.
  • Flowers recognition involves fine-grained visual appearance and category-specific morphology.

Consequently, no single pretrained expert is uniformly dominant across all domains.

To illustrate this point, we report the independently calibrated single-expert visual configurations from Table 5. Each single-expert system uses its own routing and reference-derived comparison statistics, and all configurations retain the complete 598-class candidate vocabulary.

ConfigurationAircraft (%)Cars (%)CUB (%)Flowers (%)
DINOv366.617 ± 0.38087.124 ± 0.56385.658 ± 0.25999.664 ± 0.009
BioCLIP237.524 ± 1.26184.542 ± 0.56292.866 ± 0.38899.268 ± 0.059
FG-CLIP268.927 ± 0.81192.654 ± 0.09775.469 ± 0.08599.181 ± 0.057
SigLIP265.427 ± 1.73592.269 ± 0.12370.935 ± 1.04098.862 ± 0.086
Full Visual Composition72.727 ± 1.92392.990 ± 0.14892.797 ± 0.40099.718 ± 0.009
Full Visual + Semantic Method72.737 ± 1.93494.959 ± 0.10992.970 ± 0.50499.724 ± 0.016

These are the three-reference-split results (seeds 0–2). The full visual-composition row provides a visual-only comparison, while the complete method additionally incorporates semantic evidence where available.

The single-expert results show a clear redistribution of expertise across domains. FG-CLIP2 is strongest among the individual visual experts on Aircraft and Cars, BioCLIP2 is strongest on CUB, and DINOv3 is strongest on Flowers.

Crucially, the identity of the strongest expert changes across recognition domains.

Our reference-derived competence weighting is specifically designed to address this situation: it calibrates the relative contributions of heterogeneous experts according to the discriminative evidence available in each domain, while domain-conditioned geometry and visual–semantic calibration further improve the final recognition scores.

The full visual composition improves the strongest individual visual result on Aircraft and Cars, and is competitive with the strongest single model on CUB, where BioCLIP2 is already highly specialized. The complete visual–semantic method further improves recognition across the four domains.

This explains why heterogeneous expert composition offers particularly pronounced benefits in the joint open-domain setting.

Reviewer_fewYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a fine-grained recognition ensemble framework with several pretrained VLM/Vision encoder models. Their presented method is called reference-calibrated expert composition, which consists of a 2-stage approach. The first stage is precalculation/trianing for each expert model’s visual and semantic weights and domain-conditioned weights. Then, the second inference stage uses those trained/cached statistical priors for final prediction. They show significant improvement over each expert model and other existing methods.

Critical strengths

I think the strong contribution is how they ensemble strong expert models and achieve significant accuracy improvement, especially CUB 93.23, which is excellent without fine-tuning each model.

Critical weaknesses

It is a bit unclear which components are contributing most in Table 3. It may be beneficial to compare with an ordinary ensemble method and show how this proposed method improves more than such methods, such as expert models with embedding concatenation and MLP head training. Figure 3 is interesting for showing how different each expert is, but it would be more convincing if you could show a deeper understanding of the mechanical contributions of each component.

Questions for authors

Some expert models have relatively poor performance, for example, Aircraft BioCLIP2 and iNat Siglip2/FG-CLIP2. Is it better to simply remove those models, or do those models actually contribute with expert competence steps? It may be interesting to see those weights so that we know how those models are helping with inference, or just rely on better models. It would be helpful if you could show some evidence that the stage 1 process is better than a simple ensemble method.

3 comments

Response to Reviewer fewY

We thank the reviewer for the careful reading and helpful suggestions regarding conventional ensemble comparisons, component-level mechanisms, and the contributions of relatively weak experts.

1. Comparison with ordinary ensemble methods.

To distinguish reference-calibrated composition from conventional feature aggregation, we conducted controlled experiments using the same four frozen visual encoders, the same 5-shot reference budget, and the complete 598-class joint benchmark with 23,317 queries.

The concatenated feature dimension is 3,328. All methods in the following comparison use visual information only.

Method (4 Frozen Visual Encoders, 5-Shot)Accuracy (%)Gain of Ours (Visual-Only) (pp)
Concat + Cosine Prototype88.086+3.774
Concat + Ridge Regression89.681+2.179
Concat + 2-Layer MLP89.350+2.510
Ours (Visual-Only)91.860—

Our complete method, incorporating both visual and semantic evidence, further achieves 92.623%, corresponding to improvements of +4.537, +2.942, and +3.273 pp over the three conventional aggregation baselines, respectively.

The controlled comparisons favor our calibrated readout over the three concatenation baselines. Conventional concatenation places heterogeneous representations into a single feature space, whereas our framework preserves expert-specific feature geometry and calibrates their contributions using domain-conditioned covariance statistics and reference-derived competence.

The visual-only results are particularly informative: without introducing semantic information, our method improves over conventional prototype, ridge, and MLP classifiers by 2.179–3.774 percentage points.

2. Mechanistic contributions and interactions of the components in Table 3.

We analyzed the individual mechanisms and their interactions using the component ablations in Table 3, a two-factor covariance–competence experiment on SOP, and a complete 2×2×22\times2\times2 factorial experiment on the four-domain benchmark.

The Table 3 component ablations use three matched reference splits (seeds 0, 1, and 2), with paired differences reported as mean ± sample SD. The factorial analysis uses five matched reference splits (seeds 1–5), for which paired 95% confidence intervals are reported.

ComponentStatistical RoleMain Experimental Evidence
Cross-Fitted Branch CalibrationSelects a domain-specific visual–semantic mixture from nested held-out reference predictions.Aircraft: Full reaches 72.737 ± 1.934%, compared with 61.896 ± 0.483% for fixed 0.5 weighting (paired difference −10.841 ± 1.453 pp) and 68.137 ± 1.300% for global branch weighting (−4.600 ± 0.676 pp). Cars: Full reaches 94.959 ± 0.109%, improving over visual-only by +1.969 pp.
Domain-Conditioned Covariance GeometryUses class-centered reference residuals to downweight unstable feature directions and preserve discriminative visual evidence.Replacing domain-conditioned geometry with an isotropic metric reduces Aircraft accuracy by 9.751 pp; replacing it with global covariance reduces Aircraft by 1.560 pp and SOP by 3.699 pp.
Reference-Derived Expert CompetenceEstimates domain-specific visual and semantic discriminability, increasing the influence of relevant experts.Compared with equal visual weights, competence weighting improves Aircraft by 1.500 pp, SOP by 0.544 pp, and iNat-10k by 0.509 pp.
Factorial Component InteractionEvaluates geometry, visual competence weighting, and branch calibration jointly.Positive main effects for visual competence (+0.263 pp) and branch calibration (+0.352 pp), together with a positive Geometry × Branch interaction (+0.300 pp).

Cross-fitted branch calibration. The optimal contribution of class-name evidence varies substantially across domains.

For Aircraft, alphanumeric model designations provide limited image–text alignment. Semantic-only recognition reaches 48.245%, and cross-fitting selects a purely visual branch. For Cars, semantic make/model information provides complementary evidence: semantic-only and visual-only reach 93.939% and 92.990%, respectively, while their calibrated combination reaches 94.959%.

Domain-conditioned geometry. The largest geometric effect appears on Aircraft, where pose, background, and fine structural variation make a global or isotropic metric less effective. The 9.751 pp difference from isotropic geometry and the 3.699 pp difference from global covariance on SOP quantify the contribution of reference-derived comparison geometry.

Reference-derived competence. An encoder's standalone discriminability differs across recognition domains. Reference-derived weights adapt these contributions rather than assigning every encoder the same importance. The improvements on Aircraft, SOP, and iNat-10k show that competence calibration is useful across different recognition settings.

Component interactions. We evaluated all eight combinations of:

  • Domain versus global covariance.
  • Competence-based versus equal visual weighting.
  • Domain-specific versus global branch calibration.

The main effects of visual competence weighting (+0.263 pp) and branch calibration (+0.352 pp) have positive confidence intervals. The balanced geometry main effect is +0.057 pp, with its confidence interval crossing zero, while the Geometry × Branch interaction reaches +0.300 pp, with a positive confidence interval.

The positive interaction indicates that the contribution of visual–semantic calibration depends on the underlying visual comparison geometry. Domain-conditioned geometric calibration and evidence-level branch selection therefore provide coordinated benefits in the factorial evaluation.

The three-way interaction also exhibits a positive overall effect, although its domain-wise macro-average and individual-domain confidence intervals cross zero.

Together, the experiments characterize the respective roles of the three components: estimating which experts are useful, determining how their visual evidence should be compared, and calibrating the relative importance of visual and semantic information.

3. Should relatively weak experts be removed or retained?

An expert's usefulness varies across recognition domains. A model with weak standalone performance on one dataset may still preserve discriminative information that is important on another.

Our expert-subset experiments provide direct evidence:

  • Aircraft: BioCLIP2 achieves relatively low standalone accuracy. Removing it changes the full-method accuracy by only +0.312 pp.
  • CUB: Removing BioCLIP2 reduces accuracy by approximately 7.166 pp in the corresponding removal experiment. In the visual-only subset analysis, accuracy decreases from 92.797% to 85.709%, with a matched-reference difference of −7.088 ± 0.337 pp over seeds 0–2.
  • iNat-10k: Removing FG-CLIP2 or SigLIP2 changes Full accuracy by only +0.008 pp and +0.004 pp, respectively.

These results illustrate why standalone accuracy is insufficient for deciding whether an expert should be retained. BioCLIP2, for example, contributes little to Aircraft but provides important discriminative information for CUB. A manually fixed expert pool based on performance in one domain would therefore risk discarding useful capabilities needed in another.

Our reference-derived competence weighting addresses this issue by assigning domain-specific contributions. Experts with limited relevance to the current task receive smaller weights, while their representations remain available when they provide complementary information in other domains.

Additional evidence from an independent biological expert pool.

To examine composition when constituent models have relatively comparable biological recognition capabilities, we additionally evaluated BioTrove-M, BioCLIP, BioCAP, and BioDINO336.

Their standalone accuracies are 69.711%, 70.265%, 71.182%, and 65.033%, respectively. The first three models differ by only 1.471 pp in standalone accuracy.

Expert ConfigurationOurs (%)Strongest Single (%)Gain over Strongest Single (pp)Equal Weights (%)Gain over Equal Weights (pp)
BioTrove-M + BioCLIP + BioCAP81.00671.182+9.82476.921+4.085
BioTrove-M + BioCLIP + BioCAP + BioDINO33682.25071.182+11.06879.038+3.212

In this separate expert pool, reference-calibrated composition provides substantial gains over both the strongest individual encoder and conventional equal-weight composition.

The first three biological experts have relatively similar standalone capabilities, yet their combination reaches 81.006%. Adding BioDINO336 further increases the result to 82.250%.

These results provide additional evidence that expert composition can be especially beneficial when several pretrained models have useful, complementary representations and reasonably comparable individual recognition capabilities.

This independent biological expert-pool evaluation is supplementary exploratory evidence from a distinct model pool, rather than a replacement for the original four-expert experiments.

We will include the additional biological expert-pool analysis and a more detailed discussion of domain-dependent weak-expert contributions in the revised supplementary material.

4. Interpretation of Stage I calibrated expert weights.

To provide a direct view of how Stage I assigns expert contributions, we report representative domain-specific visual weights, semantic weights, and cross-fitted branch weights.

DomainVisual Weights (DINOv3 / BioCLIP2 / FG-CLIP2 / SigLIP2)Semantic Weights (BioCLIP2 / FG-CLIP2 / SigLIP2)Semantic Branch Weight
Aircraft0.17 / 0.03 / 0.36 / 0.440.11 / 0.44 / 0.450.00
Cars0.16 / 0.15 / 0.31 / 0.380.13 / 0.42 / 0.450.50
CUB0.23 / 0.62 / 0.08 / 0.070.68 / 0.16 / 0.160.25
Flowers0.28 / 0.26 / 0.23 / 0.230.48 / 0.27 / 0.250.00
iNat-10k (average)0.09 / 0.77 / 0.07 / 0.070.79 / 0.11 / 0.100.50

The iNat-10k row summarizes the weights across its 11 coarse taxonomic groups.

The calibrated weights reflect the observed domain-specific model capabilities. BioCLIP2 receives dominant contributions on CUB and iNat, whereas the Aircraft visual branch assigns larger weights to FG-CLIP2 and SigLIP2. The branch weights also reflect differences in the usefulness of class-name semantics across domains.

We will add these calibrated-weight results and their interpretation to the revised manuscript.

Closing

We will incorporate the conventional ensemble comparisons, component interaction analysis, expert-subset results, independent biological expert-pool experiment, and calibrated-weight interpretation into the revised manuscript and supplementary material.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary

Dear Area Chairs and Reviewers,

We sincerely thank you for your constructive engagement and for recognizing our practical motivation (5Fd6, N8C2), coherent nested calibration (N8C2, 5Fd6), controlled ablations (N8C2, 5Fd6), and strong training-free gains (fewY, N8C2, 5Fd6). Below we summarize our core contributions, additional analyses, and manuscript updates:

1. Differences from Existing Work (Methodological Novelty)

  • Open-domain inference under unknown query domains: Unlike single-backbone adaptation (Tip-Adapter, APE, GDA) or fixed-dataset VLM ensembles (ZS-En, TF-En), we tackle unknown-domain recognition over a shared open vocabulary. Dynamically routing domain-conditioned statistics while keeping all classes open outperforms routed GDA by +4.98 pp (Aircraft) and +3.54 pp (SOP), and surpasses TF-En by +27.09 pp (Aircraft) and +9.73 pp (SOP).
  • Asymmetric pools via modality-decoupled cross-fitting: Our framework natively coordinates vision-only (DINOv3) and VLM encoders via separate visual (πm,g\pi_{m,g}) and semantic (ωm,g\omega_{m,g}) weights plus out-of-fold branch calibration (wg∗w_g^*). This suppresses noisy text (w∗=0w^*=0 on Aircraft) and enables text-free recognition across 4,431 SOP identities (complementary to vocabulary-free discovery like FineR/FiNDR).
  • Low-variance 5-shot calibration: Pairing entropy-regularized allocation with within-class shrinkage covariance preserves per-expert geometry and avoids few-shot overfitting. Under identical 4-encoder visual inputs (5-shot, 598 classes), our visual-only readout achieves 91.86%, beating Concat + Cosine Prototype, Ridge Regression, and 2-Layer MLP by +3.77 pp, +2.18 pp, and +2.51 pp (92.62% with semantics).

2. Responses to Major Reviewer Questions

  • Component roles and weak-expert retention: Over a jointly simplified baseline (global covariance + equal visual weights + global branch weight), full calibration gains +0.72 pp on Four-Domain (+3.08 pp on Aircraft), +0.68 pp on iNat-10k, and +4.57 pp on SOP. Stage I automatically downweights domain-weak experts (0.03 for BioCLIP2 on Aircraft), while retaining them prevents a 7.09 pp drop on CUB.
  • Single-expert controls and practical scope: Our Calibrated Single baseline already incorporates core components of our method (domain-conditioned covariance, routing, and branch calibration), which substantially outperform plain cosine prototypes (+3.95 pp on Four-Domain, +10.72 pp on SOP); thus, this comparison strictly isolates the incremental benefit of multi-expert composition. Accordingly, four-expert gains over calibrated single experts are substantially larger on benchmarks with greater inter-domain heterogeneity—adding +5.84 pp on Four-Domain and +3.71 pp on the new 268-class Meta-Album task, versus +0.83 pp on single-supercategory SOP. At fixed 4-encoder cost, domain weighting also beats global weights by +0.64 pp and +0.86 pp on both heterogeneous tasks. On BioCLIP2-dominated iNat-10k, it preserves specialist accuracy without degradation (89.56% vs. 89.52%), while an exploratory pool of comparable biological encoders gains up to +11.07 pp over the best single model.
  • Routing robustness, end-to-end cost, and clarifications: Coarse-domain routing reaches 96.0%–99.96% accuracy (within 0.01–0.88 pp of an oracle router). On an NVIDIA L40, native four-encoder inference uses 5.6–9.0 GiB VRAM and 19.4–41.6 ms/image (B16) with <2.0 s offline calibration—far lighter than generative VLM Top-5 reranking (38.1 GiB, 44.5 ms/image). We also added the entropy-regularized derivation, covariance-competence control, and adaptive MeanTopmin⁡(3,navail)\text{MeanTop}_{\min(3,n_{\text{avail}})} specification.

3. Summary of Revisions

We have uploaded the revised PDF with all main-body updates highlighted in blue. Specifically, we refined our positioning and added FineR/FiNDR (Secs. 1–2), clarified adaptive MeanTop\text{MeanTop} and open-vocabulary routing (Sec. 3), and updated Secs. 4–5 with the Meta-Album benchmark, aligned seeds 0–2 ablations (Table 3), a unified accuracy–latency–VRAM table (Table 4), ensemble/routing/weight takeaways, and Limitations; full derivations, CIs, and control tables appear in Appendices A–F. Thank you again for your valuable feedback—we will release our code to facilitate further community exploration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.