acceptodds
Under review as a conference paper at ICLR 2027

VEFR: Visual Evidence Fusion Routing for Budgeted Skill Selection in Fine-Grained Visual Recognition with MLLMs

Abstract

Fine-grained recognition requires distinguishing subtle visual differences, an MLLM-based system can approach the same image through several recognition strategies. These strategies differ in cost and have complementary strengths across classes and individual images. We formulate recognition as budgeted skill routing: selecting an executable recognition strategy before observing its output or incurring its execution cost. We propose Visual Evidence Fusion Routing (VEFR), which estimates skill competence from two sources of visual evidence. Class-conditioned statistics capture recurring skill preferences, while correctness records from visually similar profiling images provide query-specific evidence. A soft visual class distribution weights the class statistics, and a validation-selected fixed weight combines the two estimates without training a separate routing network. We separate competence estimation from deployment control, allowing different prioritization of accuracy–cost without recalculating the evidence. Across eight image recognition benchmarks, VEFR achieves better accuracy compared with fixed skill and skill routing methods while consuming much fewer MLLM tokens. These results show that visual evidence can guide skill selection to improve the accuracy–cost trade-off.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fine-Grained Visual Recognition, Multimodal Large Language Models, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_UbD4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes VEFR, a skill competence estimation approach for fine-grained image classification tasks. It uses a combination of class-conditioned competence and query-specific competence estimate, called “fixed hybrid estimator”, to select a skill to be used at inference. Since the fixed hybrid estimator is done offline, it allows VEFR to perform different inference strategies based on accuracy-cost tradeoff. Experimental results using VEFR shows same/improved accuracy while using significantly fewer tokens across 8 fine-grained image classification datasets.

Critical strengths

The formulation of task as skill selection and combining different recognition approaches such as visual retrieval, direct prediction, VICL etc. as skills for fine-grained image classification is interesting.

Critical weaknesses
  1. The method section of the paper is unclear and hard to follow.

    1. Sec. 3.2 introduces Eq. (5) without explaining what p(c∣xi)p(c \mid x_i) represents. However, it is not clearly explained what this ratio represents and why it should be used to compute class-conditioned competence estimates.

    2. In Sec. 3.3, Eq. (9), what does NpN_p represent?

    3. The inference setting is also unclear. Eq. (10) uses A^i,s\hat{A}_{i,s} for skill selection, but it is not clear how A^i,s\hat{A}_{i,s} is estimated during inference. During training, yi,sy_{i,s} is available, but how is yi,sy_{i,s} computed during test-set evaluation?

    4. References to the appendix are not sufficiently specific. While I was happy to consult the appendix for additional details, the references do not correctly point to the relevant subsections, making the paper difficult to navigate.

  2. The paper mentions training at multiple places (e.g., Lines 323 and 354). However, it is unclear what exactly is being trained in the overall pipeline.

  3. Table 1 presents results for the "Oracle Skill Selector," but I could not find details explaining how the oracle skill is selected.

  4. While the reduction in token usage is beneficial, the accuracy gains are marginal. A variance analysis would help determine whether these improvements are statistically meaningful or simply due to noise.

Questions for authors
  1. While Fig. 1b shows specific examples, could the authors provide a more comprehensive analysis of how each skill performs across different classes? The main motivation of the paper relies on the assumption that different skills solve problems in complementary ways. Therefore, an inter-class and intra-class analysis of each skill would be helpful to validate this core assumption. I understand that this is indirectly demonstrated through the Oracle Skill Selector results; however, the details of how the oracle selects skills are unclear.

  2. Which model is used for skill execution in the results presented in Table 1?

  3. In batch-budget allocation, why is the method constrained to select exactly one skill?

11 comments

Response to ''Questions For Authors''

1. Inter-class and Intra-class Skill Complementarity

We thank the reviewer for this suggestion. We conduct additional analyses across all eight datasets to examine skill complementarity at both the class and instance levels.

Inter-class complementarity. We compute the accuracy of each skill for every class using the training split and identify the preferred skill for each class. All five skills are selected as the best-performing skill for some classes, with ICL. preferred for 47.9% of classes, followed by Desc. (14.7%), CoT. (14.8%), CLIP. (14.4%), and Direct. (8.2%). This shows that skill preferences vary across categories rather than being dominated by a single skill.

Table R1. Inter-class skill preferences and individual skill contributions.

SkillPreferred-class share (%) ↑Mean per-class test accuracy (%) ↑Exclusively correct images
CLIP.14.471.11,508 (1.9%)
Direct.8.262.9454 (0.6%)
Desc.14.767.5554 (0.7%)
CoT.14.863.9785 (1.0%)
ICL.47.981.92,396 (3.0%)

The last column counts test images correctly recognized by only the corresponding skill, showing that every skill contributes unique coverage beyond the others.

Intra-class complementarity. We compare a class-level selector with a sample-level oracle. The class-level selector chooses the best skill for each category using training-set accuracy and applies that fixed choice to test images. In contrast, the sample-level oracle selects, for each test image, any of the five skills that produces the correct prediction.

Table R2. Class-level selection versus sample-level oracle across eight datasets.

DatasetClassesClass-level selection (%)Sample-level Oracle (%)Oracle gain (pp)Oracle-rescued images
Aircraft10069.583.013.5451
Car19689.096.07.0563
CUB20082.091.19.1526
Dogs12082.090.48.5726
Flowers10297.599.62.1129
Food10193.197.03.9988
Pets3795.698.63.0111
SUN39783.391.58.31,640
Overall1,25387.7694.13+6.365,134

The sample-level oracle improves over class-level selection by 6.36 pp overall, with positive gains across all eight datasets. In total, 5,134 test images misclassified by the class-level selection can be correctly recognized by at least one alternative skill.

Together, these results show that skill complementarity exists both across categories and among images within the same category.

We will include these analyses and clarify the oracle definition in the revised manuscript.

2.Which model is used for skill execution in the results presented in Table 1?

All MLLM-based recognition skills in Table 1 use Qwen3-VL-8B-Instruct as the shared executor to ensure a consistent evaluation setting. The CLIP kNN Retrieval skill uses CLIP ViT-B/16.

We will explicitly clarify the MLLM executor used in Table 1 in its caption.

3. In batch-budget allocation, why is the method constrained to select exactly one skill?

Our formulation focuses on single-skill selection for each query image, rather than multi-skill execution or ensemble. The batch-budget optimizer jointly allocates different skills across queries, allowing more expensive skills to be assigned to images with higher estimated competence gains while keeping the total estimated cost within the given budget.

Response to ''Critical Weaknesses The method section of the paper is unclear and hard to follow.''

1. Analysis and explanation for Eq. (5)

(1) Interpretation and motivation.
p(c∣xi)p(c\mid x_i) represents a similarity-weighted soft distribution over the CLIP-selected candidate classes, constructed from the labels of the query image's visual neighbors. Specifically, the numerator measures the total similarity weight of neighbors belonging to class c, while the denominator normalizes over neighbors belonging to the candidate set. In Eq. (6), this distribution weights the class-level skill accuracies to estimate expected skill competence across multiple plausible classes, rather than assuming that a single Top-1 prediction is necessarily correct.

(2) Soft vs. hard class assignment.

To test whether retaining uncertainty over candidate classes improves skill selection, we compare class-assignment strategies using the same profiling data, visual encoder, retrieved neighbors, local evidence, and skill pool. In particular, Neighbor Top-1 takes the highest-probability class from the same distribution p(c∣xi)p(c\mid x_i) used by our method. This comparison tests whether using the full distribution offers an advantage over using its top class alone.

Table R3. Ablation of different class-assignment strategies.

Class AssignmentClass-only Acc. (%)Hybrid Acc. (%)Tokens/img
CLIP Top-179.0183.464,450
Neighbor Top-179.3083.164,309
Uniform Candidate83.7383.926,894
Soft Distribution (Ours)84.0384.175,802
Unweighted Neighbor83.9784.095,831

(3) Class uncertainty analysis.

We expect soft assignment to help most when the visual neighbors provide support for several plausible classes. To test this hypothesis, we group test queries by the normalized entropy of p(c∣xi)p(c\mid x_i) and compare Soft Distribution with Neighbor Top-1 within each group.

Table R4. Accuracy improvements of Soft Distribution over Neighbor Top-1 across different class-uncertainty levels.

Uncertainty GroupTest SamplesClass-only ΔAcc. (pp)Hybrid ΔAcc. (pp)
Low32,688+2.21+0.33
Medium25,606+3.69+0.93
High22,372+5.92+1.69

The improvements increase with class uncertainty, indicating that soft assignment is particularly beneficial for ambiguous queries. Accuracy in both tables is the eight-dataset macro average, while token usage is weighted by the number of test images.

2. Explanation for NPN_P

NPN_P denotes the number of images in the profiling set, and Cj,sC_{j,s} represents the realized execution cost of skill ss on the jj-th profiling image. Thus, Eq. (9) estimates the average execution cost of each skill over the profiling set. We will explicitly define NPN_P in Section 3.3 for clarity.

3. Explanation for inference setting

We thank the reviewer for pointing out this ambiguity. We clarify that yj,sy_{j,s} is obtained during offline profiling, rather than at test-time inference. Specifically, we execute all recognition skills on the labeled profiling images and record their prediction correctness: yj,s=1y_{j,s}=1 if skill ss correctly predicts image xjx_j, and yj,s=0y_{j,s}=0 otherwise. These correctness labels are precomputed and stored for subsequent inference.

At test time, the ground-truth label of a query image xix_i is unavailable. Therefore, yi,sy_{i,s} cannot be computed and is never required for skill selection. Instead, VEFR retrieves visually similar images from the profiling set and uses their pre-recorded correctness labels to estimate local competence via Eq. (7). Together with the class-conditioned competence estimated by Eq. (6), these estimates are combined in Eq. (8) to obtain the hybrid competence score A^i,s\hat{A}_{i,s}, which is then used for skill selection in Eq. (10).

Only the selected skill is executed on the test image. Its prediction is compared with the ground-truth label solely for final accuracy evaluation, not for skill selection or routing. We will clarify the distinction between offline profiling and test-time inference in Sections 3.1–3.3.

4. References to the appendix are not sufficiently specific

We thank the reviewer for pointing this out. We will carefully revise all appendix references throughout the manuscript to point to the exact relevant subsections, tables, and figures.

Response to ''Critical Weaknesses 2,3,4''

2. The paper mentions training at multiple places (e.g., Lines 323 and 354). However, it is unclear what exactly is being trained in the overall pipeline.

We thank the reviewer for pointing out this ambiguity. We clarify that VEFR does not involve training an additional router or fine-tuning any pretrained models (CLIP, DINOv3, or Qwen3-VL).

Instead, our pipeline consists of three steps: (1) Offline profiling: we execute the five recognition skills on labeled profiling images and record their correctness and inference costs. (2) Competence estimation: we compute class-level skill accuracy statistics and use visual neighbors to estimate query-specific competence. The hyperparameters (e.g., KK, TT, and γ\gamma) are selected on the validation set. (3) Inference: the resulting statistics and hyperparameters are fixed and directly used for skill selection, without additional training.

The term "training" in Lines 354–356 primarily refers to training the learned baseline routers (e.g., MLP Router, RouterDC, and the FrugalGPT scorer). After hyperparameter selection, VEFR recomputes its profiling statistics using the full training set.

We will clarify this distinction and revise the wording throughout the manuscript to distinguish offline profiling, hyperparameter selection, and actual model training.

3. Table 1 presents results for the "Oracle Skill Selector," but I could not find details explaining how the oracle skill is selected.

We thank the reviewer for pointing this out. The Oracle Skill Selector is a hindsight upper bound, not a deployable router. For each test image, it selects a correct skill if any of the five skills produces a correct prediction.

Let yi,s∈0,1y_{i,s}\in{0,1} indicate whether skill ss predicts the ground-truth class of test image xix_i, and let S\mathcal S denote the set of recognition skills. The oracle’s correctness on image xix_i is yioracle=max⁡s∈Syi,sy_i^{\mathrm{oracle}}=\max_{s\in\mathcal S}y_{i,s}. This value is 1 if at least one skill is correct and 0 if all five are incorrect. The reported 93.42% is the eight-dataset macro-average accuracy. Since it requires ground-truth labels and all skill outcomes, it is unavailable at inference time. We will clarify this in Section 4.2 and the Table 1 caption.

4. While the reduction in token usage is beneficial, the accuracy gains are marginal. A variance analysis would help determine whether these improvements are statistically meaningful or simply due to noise.

We thank the reviewer for pointing out the need for statistical analysis. We perform paired bootstrap analysis using the same test images for all methods, with 10,000 resampling iterations stratified by dataset and class. We report 95% confidence intervals (CIs) for both accuracy differences and relative token savings.

Table R5. Paired confidence intervals for accuracy differences and token savings (Qwen3-VL-8B).

BaselineAccuracy difference (pp) [95% CI]Token savings (%) [95% CI]
Visual ICL+0.747 [+0.558, +0.937]35.62 [35.40, 35.84]
Class-only+0.343 [+0.224, +0.462]0.75 [0.49, 1.01]
Class + Global Prior−0.042 [−0.173, +0.089]19.76 [19.55, 19.99]

The results show that VEFR significantly improves accuracy over Visual ICL and Class-only, as both confidence intervals exclude zero. Compared with Class + Global Prior, the accuracy difference is not statistically significant, indicating comparable accuracy. However, VEFR reduces online token consumption by 19.76%, with a narrow 95% CI of [19.55%, 19.99%].

Overall, the paired analysis supports the accuracy gains over Class-only and Visual ICL, while showing that the more consistent advantage of VEFR is its improved accuracy–efficiency trade-off.

We will include these confidence intervals in the revised manuscript.

Reviewer_yAXSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VEFR, a method that chooses among five recognition strategies before execution for fine-grained recognition with MLLMs. VEFR estimates each strategy’s competence from per-class accuracy and from the correctness records of visually similar profiling images. The two estimates are fused with a validation-tuned weight and then used for accuracy-first, cost-sensitive, or budget-constrained selection.

Critical strengths
  1. Practical problem. Choosing a strategy before running it addresses a real efficiency need. The paper shows that the strategies make complementary errors and reports tokens, requests, and image inputs alongside accuracy.
  2. Simple and interpretable design. The estimates are easy to inspect and need no trained routing network. Because estimation is separate from the controller, the same scores support different accuracy and cost goals.
  3. Broad supplementary evaluation. The appendices cover encoders, profiling-set size, ICL context size, executors, and profile transfer. Table 6 and Figure 9 analyze decision changes and failures, not just success cases.
Critical weaknesses
  1. The core mechanism lacks justification. The method is mainly a validation-tuned mix of empirical estimates. The paper does not show when class-level accuracy reflects instance-level competence, or when visual neighbors add useful information. The ablations show gains but cannot tell whether they come from instance-specific information, statistical smoothing, or a shift in strategy preference. Targeted experiments testing these hypotheses are needed.
  2. The main comparisons confound the router with its input features. Instance-kNN, MLP, and RouterDC use CLIP embeddings of MLLM-generated captions (Appendix A.3), while VEFR uses DINOv3 image features. Captions may lose fine visual details and add MLLM cost. Figure 5 tests a DINOv3-based MLP, but only on two datasets. A full eight-dataset comparison with matched features and controllers is needed.
  3. The value of local evidence is not isolated. In Appendix E.1, Class+global reaches 84.54% at 5,916 tokens per image, and VEFR reaches 84.50% at 4,747. This shows an efficiency gain at two operating points, not under a shared cost controller. Also, only the class branch filters neighbors by the candidate set, so the gap mixes two factors. Matched-cost curves and a control for filtering are needed.
  4. Related work is incomplete. The paper should discuss Demidov et al. (Thinking Beyond Labels, CVPR 2026), which uses reasoning-augmented LMMs for fine-grained recognition. A numerical comparison is not required, since the setting differs. The local estimator should also be linked to classical dynamic classifier selection based on local accuracy.
  5. Small gains lack statistical support. The main accuracy differences have no paired confidence intervals. This matters for the 0.35-point gain over Class-only and especially the 0.04-point margin over the strongest baseline with Qwen3-VL-32B. The efficiency gains look more robust.
  6. Cost claims need clearer accounting. The reported savings exclude profiling, validation, and updates. The limited-profiling experiments still rely on fixed strategies and a validation set, so they do not reduce total offline cost to the same degree. In addition, Table 1, Figure 3, and the no-violation replay use different cost settings, so the replay does not confirm budget compliance for the other two. Per-setting compliance results and an amortized cost analysis are needed.
Questions for authors
  1. How do VEFR, MLP, and a local-accuracy router compare on all eight datasets with the same DINOv3 features, profiling outcomes, and cost controller?
  2. Does local evidence carry within-class information? How does performance change if correctness vectors are shuffled among profiling images of the same class?
  3. Does VEFR beat separately tuned Class-only and Class+global at matched token budgets? Can candidate filtering be separated from the aggregation level?
  4. Can the authors report paired confidence intervals for the main accuracy differences, especially against Class-only, Class+global, and the strongest routing baseline?
  5. How does the reasoning setup relate to FiNDR by Demidov et al.? How does the local estimator go beyond classical local-accuracy-based selection?
  6. What are the offline costs of profiling, validation, and gallery construction, and at what volume do they pay off? What are the actual budget usage and violation rates under the cost settings of Table 1 and Figure 3?
12 comments

Response to "Does local evidence carry within-class information? How does performance change if correctness vectors are shuffled among profiling images of the same class? "

We thank the reviewer for this question. Local evidence captures instance-specific information beyond class-level statistics. To verify this, we conduct a within-class shuffling experiment: we randomly shuffle the complete skill-correctness vectors among profiling images of the same class. This preserves class-level accuracy statistics and inter-skill correlations, while breaking the correspondence between visual neighbors and their individual skill outcomes.

When does local evidence help? We compare VEFR with Class-only at five profiling sizes, with smoothing disabled for both methods. This tests how the contribution of local evidence changes as more observations become available for estimating class-level skill accuracy.

Table R1. Local-evidence gains without smoothing. Each entry is the paired test accuracy difference between VEFR and Class-only, in percentage points (pp), averaged over five class-balanced support galleries per dataset. Columns indicate the number of profiled images per class. The IQR reports the 25th and 75th percentiles across the eight dataset gains.

Dataset1 image2 images4 images8 images16 images
CUB+2.63+1.41+0.42+0.59−0.21
Aircraft+2.62+2.00+0.58+0.76+0.55
Cars+1.34+0.29+0.08+0.02+0.06
Pets+0.45+0.29−0.20+0.07+0.06
Dogs+4.16+2.42+1.34+0.63+1.06
Flowers+9.13+7.27+6.86+6.37−0.01
Food+0.11+0.11−0.09+0.05−0.03
SUN397+2.11+1.36+0.45+0.25+0.15
Macro mean+2.82+1.89+1.18+1.09+0.20
Dataset IQR[+1.12, +3.01][+0.29, +2.11][+0.04, +0.77][+0.06, +0.66][−0.02, +0.25]

Does local evidence capture instance-specific information? We shuffle complete skill-correctness vectors within each class, preserving class-level statistics and inter-skill correlations while breaking their correspondence with individual visual neighbors.

Table R2. Effect of within-class shuffling under matched token budgets.

Shared BudgetOriginal VEFR Acc. / TokensShuffled Acc. / TokensΔAcc. (pp) [95% CI]
30%82.903% / 2,16982.380% / 2,173+0.523 [0.397, 0.655]
50%84.129% / 3,55983.721% / 3,557+0.408 [0.294, 0.517]
70%84.466% / 4,65584.185% / 4,651+0.281 [0.180, 0.381]

The original VEFR consistently outperforms the shuffled variant, with nearly identical token consumption (within 5 tokens/image). The accuracy drops by 0.523, 0.408, and 0.281 percentage points after shuffling under the three budget settings, respectively. Since class-level statistics and inter-skill correlations are preserved, these results demonstrate that local evidence provides useful instance-specific information beyond class-level statistics.

We will include this analysis in the revised manuscript.

Response to "Does VEFR beat separately tuned Class-only and Class+global at matched token budgets? Can candidate filtering be separated from the aggregation level? "

We thank the reviewer for raising these concerns. We conduct additional experiments to compare the methods under matched token budgets and isolate the effect of candidate filtering.

(1) Comparison under matched token budgets. We evaluate Class-only, Class + Global Prior, and VEFR using the same batch-budget controller, skill-cost estimates, and five shared estimated budget caps.

Table R3. Accuracy and realized token use under five shared budget caps. Each cell reports eight-dataset macro accuracy (%) / realized tokens per test image.

Shared budget capClass-onlyClass + Global PriorVEFR
30%82.28 / 2,18682.25 / 2,18682.90 / 2,169
50%83.59 / 3,57283.58 / 3,67784.13 / 3,559
70%84.12 / 4,67484.36 / 5,09984.47 / 4,655
90%84.15 / 4,78384.50 / 5,86284.50 / 4,747
100%84.15 / 4,78384.54 / 5,91684.50 / 4,747

At the 30%, 50%, and 70% caps, VEFR achieves higher accuracy and uses fewer realized tokens than both alternatives. At the 90% cap, VEFR and Class + Global Prior both reach 84.50% to two decimal places, with VEFR using 19.0% fewer tokens (4,747 vs. 5,862 per image). At the 100% cap, VEFR uses 19.8% fewer tokens than Class + Global Prior, with a 0.04 percentage point lower accuracy (84.50% vs. 84.54%).

(2) Separating candidate filtering from evidence aggregation. We independently enable or disable candidate filtering in the class and local branches while keeping the initial Top-K neighbors, similarity weights, and routing rule unchanged.

Table R4. Ablation of candidate filtering in class and local evidence.

Class EvidenceLocal EvidenceFixed γ: Acc. / TokensRetuned γ: Acc. / Tokens
Candidate-filteredAll (K) (VEFR)84.496 / 4,74784.496 / 4,747
Candidate-filteredCandidate-filtered84.153 / 4,68884.229 / 4,723
All (K)All (K)84.528 / 4,87484.520 / 4,861
All (K)Candidate-filtered84.328 / 4,84384.582 / 4,901

Filtering local neighbors reduces accuracy by 0.272 pp (95% CI: [−0.321, −0.224]), averaged across the two class-branch settings. The reduction remains 0.103 pp (95% CI: [−0.151, −0.055]) after validation-based re-tuning of γ.

We further compare Class + Local and Class + Global Prior using the same unfiltered class evidence to eliminate differences in class-branch filtering.

Table R5. Class + Local versus Class + Global Prior under matched candidate filtering.

SettingClass + Local Acc. / TokensΔAcc. (pp) [95% CI]Token Savings
Fixed γ84.528 / 4,874+0.005 [−0.120, +0.133]13.98%
Retuned γ84.520 / 4,861−0.004 [−0.128, +0.124]14.20%

Class + Global Prior achieves 84.524% accuracy at 5,666 tokens/image. Thus, under matched candidate filtering, local evidence achieves comparable accuracy while reducing online token consumption by approximately 14%.

These experiments isolate the effects of budget allocation, candidate filtering, and evidence aggregation.

We will incorporate the results into the revised manuscript.

Response to "Can the authors report paired confidence intervals for the main accuracy differences, especially against Class-only, Class+global, and the strongest routing baseline? "

We thank the reviewer for highlighting the need for statistical significance analysis. We conduct paired bootstrap analysis with 10,000 replicates on the same 80,666 test images, resampling within each dataset and class. Accuracy differences are computed using an equal-weight average across the eight datasets. We also calculate paired 95% confidence intervals (CIs) for token savings.

(1) Comparison with Class-only and Class + Global Prior (Qwen3-VL-8B). VEFR achieves 84.496% accuracy with 4,747 tokens/image. Compared with Class-only, VEFR improves accuracy by 0.343 pp (95% CI: [+0.224, +0.462]), confirming a statistically significant improvement. Compared with Class + Global Prior, VEFR achieves comparable accuracy (−0.042 pp, 95% CI: [−0.173, +0.089]) while reducing token consumption by 19.76% (95% CI: [19.55%, 19.99%]).

(2) Comparison with routing baselines (Qwen3-VL-32B). Against the strongest routing baseline, Prompted VLM, VEFR achieves a small, statistically insignificant accuracy improvement of +0.032 pp (95% CI: [−0.155, +0.215]), while reducing token consumption by 38.44% (95% CI: [38.26%, 38.63%]). VEFR also achieves statistically significant accuracy improvements over MLP Router and RouterDC, with substantial token savings over all routing baselines.

Table R6. Paired accuracy differences and token savings with 95% confidence intervals. All accuracy differences are computed as VEFR minus the corresponding baseline; positive token savings favor VEFR.

Qwen3-VL-8B BaselineΔAcc. (pp) [95% CI]Token Savings (%) [95% CI]
Class-only+0.343 [+0.224, +0.462]0.75 [0.49, 1.01]
Class + Global Prior−0.042 [−0.173, +0.089]19.76 [19.55, 19.99]
Qwen3-VL-32B BaselineΔAcc. (pp) [95% CI]Token Savings (%) [95% CI]
Prompted VLM+0.032 [−0.155, +0.215]38.44 [38.26, 38.63]
FrugalGPT+0.049 [−0.163, +0.260]29.83 [29.62, 30.04]
MLP Router+0.246 [+0.046, +0.444]26.06 [25.82, 26.31]
RouterDC+0.650 [+0.452, +0.849]23.25 [22.98, 23.52]
Instance-kNN+1.414 [+1.195, +1.633]3.46 [3.03, 3.90]

Overall, the results confirm that VEFR significantly improves accuracy over Class-only and several learned routing baselines. Although its accuracy is statistically comparable to Class + Global Prior and the strongest routing baseline, it achieves substantial and statistically consistent token savings. These results highlight VEFR's favorable accuracy–cost trade-off. We will incorporate these confidence intervals into the revised manuscript.

Reviewer_Nw4tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a fine-grained MLLM recognition framework which consist of several recognition skills with budgeting routing method for skill selection/composition. Also, they present the Visual Evidence Fusion Routing, a competence estimation method that fuses class-conditioned evidence and local visual evidence. They show the accuracy-cost tradeoff controlled by the budgeting routing, and improvement with accuracy-first routing over existing methods.

Critical strengths

I think the strong contribution is accuracy improvement while suppressing token usage by allocating each skill at deployment, which potentially provides a huge computation reduction for the token-consuming reasoning part of estimation.

Critical weaknesses

The biggest concern is computation overhead. They precalculated each skill during offline skill profiling, and they don’t reveal the token usage for that part. They claimed their proposed method has no training process, but it is hard to judge without revealing this precomputing profiling cost because they use 80% of the training data for this profiling.

Second, the class-conditioned evidence and local visual evidence are unclear. The Fig 2 caption mentions both class evidence and local evidence are combined, but not sure if they do all skills or a selected skill. Also, it is unclear how they retrieve visually similar profiling images to calculate local visual evidence and what is different from CLIP-selected top K images.

Finally, the experimental results are not clear for fair comparison. I do not understand which CLIP model was used for retrieval and for each skill model. I need to see the comparison between existing methods more precisely. I could not understand each experimental setting, such as a pure methodology with the same base VLM, MLLM, or LLM. For example, VICL (Xiong et al. (2026)) paper’s results show CUB acc 82.17 and Flowers acc 98.99 by using Qwen2.5 -VL-7B on their paper, but those are different from their Table 1. This definitely needs more elaboration to understand their improvement over other methods.

Questions for authors

What is the check and cross mark in Figure 1 (c)? Need more explanation about this figure. Could you elaborate on how you select visually similar profiling images for local visual evidence, and what is different from the images that you use for class-conditioned evidence?

6 comments

Response to ''Questions For Authors''

What is the check and cross mark in Figure 1 (c)? Need more explanation about this figure. Could you elaborate on how you select visually similar profiling images for local visual evidence, and what is different from the images that you use for class-conditioned evidence?

We thank the reviewer for pointing out these ambiguities. We clarify the figure and the evidence retrieval process below.

(1) Meaning of the check and cross marks. In Fig. 1(c), ✓ and × indicate whether the corresponding recognition skill correctly or incorrectly classifies the given image, respectively. These examples illustrate the complementary performance of different skills across individual images.

(2) Retrieval of visually similar profiling images. We use a frozen DINOv3 ViT-B/16 encoder h(⋅)h(\cdot) to extract visual features for the query image xix_i and the profiling images xjx_j. For each query, we retrieve the top-KK profiling images ranked by cosine similarity rij=cos⁡(h(xi),h(xj))r_{ij} = \cos(h(x_i), h(x_j)). We convert these similarities into positive weights wij=exp⁡(rij/T)w_{ij} = \exp(r_{ij}/T), where the temperature TT controls how strongly the weighting favors more similar neighbors. We then use these weights to aggregate the profiling evidence for each recognition skill.

(3) Difference between the two evidence sources. Importantly, both class-conditioned and local visual evidence use the same retrieved top-K images, but aggregate their information differently. Class-conditioned evidence uses the labels of retrieved images to construct a soft distribution over the CLIP-selected candidate classes, which is then combined with class-level skill accuracy statistics computed from the profiling set. In contrast, local visual evidence directly aggregates the skill correctness records of all retrieved neighbors using their similarity weights, without restricting them to the candidate classes. This allows visually similar examples from other classes to provide complementary, query-specific evidence.

We will clarify these explanations in Section 3.2 and improve the caption of Fig. 1(c) to make the figure and the distinction between the two evidence sources easier to understand.

Response to ''Critical Weaknesses : Computation Overhead''

We thank the reviewer for raising this important concern. We therefore conduct an explicit analysis of offline costs, reduced profiling budgets, and amortized deployment costs.

VEFR trades one-time offline profiling computation for lower per-query inference cost, with its benefit depending on the deployment volume.

(1) Actual offline profiling cost.

Table R1. Offline MLLM token consumption under the original profiling setting.

Offline componentImagesTotal tokens
Router-Train profiling (80%)107,2991,188.58M
Validation skill execution26,826297.76M
Total134,1251,486.33M

(2) Reduced profiling budgets. We evaluate two complementary settings.

First, under Protocol A, we reduce the Router-Train profiling subset while retaining the full validation set and original recognition gallery. As shown in Table R2, using 10% of the profiling data achieves 84.00% accuracy, compared with 84.29% using 80%. This reduces offline token consumption by approximately 70%, with only a 0.29 pp accuracy difference, although online token consumption increases by approximately 30%.

Table R2. Profiling-data reduction under Protocol A.

Profiling fractionTotal collected imagesAccuracy (%) ↑Online tokens/test image ↓Offline total (M) ↓
10%40,23784.006,611446.80
80%134,12584.295,0861,486.33

Second, under the stricter Protocol B, we jointly limit profiling, validation, and recognition-gallery data. VEFR consistently outperforms the corresponding budget-specific Visual ICL baseline at 20%, 40%, and 80% budgets.

Table R3. Accuracy and computational costs under strict profiling budgets (Protocol B).

BudgetImagesVEFR Acc. (%) ↑VEFR online tokens/test image ↓Offline total (M) ↓Visual ICL Acc. / tokens/image
20%26,82379.574,565298.0078.96 / 6,984
40%53,64881.915,196594.9581.07 / 6,976
80%107,29983.735,2721,187.9582.75 / 6,979

At 20%, 40%, and 80% budgets, VEFR improves accuracy by 0.62, 0.84, and 0.98 pp, respectively, while reducing online token consumption by 34.6%, 25.5%, and 24.5%. The Visual ICL results use budget-specific recognition galleries.

(3) Amortized deployment cost. We further analyze cumulative MLLM token consumption as deployment volume increases. Total cost is calculated as the one-time offline profiling cost plus online token consumption over deployment queries.

Table R4. Cumulative MLLM token use and break-even volume. Cumulative totals are in billions of tokens (B).

SettingOffline tokens (M)Online tokens/imageTotal at 80,666 queries (B)Total at 400k queries (B)Total at 800k queries (B)Break-even queries
VEFR, strict 20%298.004,5650.6662.1243.950123,163
VEFR, strict 40%594.955,1961.0142.6734.752334,234
VEFR, Full1,524.654,7471.9083.4235.322580,631

The benchmark query distribution is held fixed. The Paper Visual ICL reference uses 7,373 tokens per test image, while the Paper Full curve uses a lower-bound estimate of offline preparation cost.

These results clarify that VEFR is free of additional neural router training, but not free of offline computation. Its initial profiling cost can be substantial, while reduced profiling budgets provide a practical way to lower this cost. At sufficiently large deployment volumes, the lower per-query token consumption can amortize the initial investment. We will revise the manuscript to explicitly distinguish offline profiling costs from online inference savings and report their trade-off.

Response to ''Critical Weaknesses : "the class-conditioned evidence and local visual evidence are unclear"

We thank the reviewer for pointing out these ambiguities. We clarify the two aspects below.

(1) Fusion across all skills. Both class-conditioned and local visual evidence are estimated for every recognition skill, rather than only for a pre-selected skill. Specifically, we first estimate the two competence scores for each skill and fuse them using a fixed weight (Eq. 8): A^i,shyb=(1−γ)A^i,scls+γA^i,sloc.\hat A_{i,s}^{hyb} = (1-\gamma)\hat A_{i,s}^{cls} + \gamma\hat A_{i,s}^{loc}.

The router then selects one skill based on the fused competence estimates. Thus, evidence fusion takes place before skill selection.

(2) Retrieval of visually similar profiling images. We use a frozen image encoder (DINOv3 ViT-B/16 in our main setting) to extract image features and retrieve the top-K nearest profiling images based on cosine similarity. Their similarity scores are converted into weights and used to estimate local competence from their recorded skill correctness. Importantly, this is different from CLIP-based candidate selection: CLIP selects candidate classes by image–text similarity, whereas DINOv3 retrieves visually similar profiling images by image–image similarity. The retrieved visual neighbors are shared by both evidence branches. Class-conditioned evidence aggregates class-level skill statistics over the CLIP-selected candidate classes, while local evidence directly aggregates skill correctness over all retrieved neighbors, including those whose labels may fall outside the candidate class set.

We will further clarify these distinctions in Section 3.2 and the caption of Figure 2.

Reviewer_ukqsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Visual Evidence Fusion Routing (VEFR), a framework for improving cost-aware skill selection in fine-grained visual recognition with MLLMs. VEFR estimates the competence of different recognition skills to guide skill selection under different deployment objectives, without requiring a separately trained routing network. Experiments on eight fine-grained recognition benchmarks demonstrate that VEFR achieves competitive or superior recognition accuracy while substantially reducing MLLM token consumption compared with fixed recognition strategies and existing routing baselines. Additional ablation studies examine the contributions of the proposed components and the robustness of the framework.

Critical strengths
  1. This work presents a practical and easy-to-follow framework for cost-aware recognition skill selection in practical MLLM fine-grained recognition.

  2. The proposed method achieves consistent accuracy–cost trade-offs across eight fine-grained recognition benchmarks. It also applies to different MLLM architectures.

  3. The paper provides comprehensive experiments across various baselines, different component alternatives, and deployment budgets.

Critical weaknesses
  1. This work focuses on budgeted skill selection for fine-grained visual recognition. Yet very few clues about what the skills exactly are were provided.

  2. The competence estimator is a fixed-weight combination of class-conditioned accuracy statistics and similarity-weighted neighboring outcomes. The components and fusion are straightforward

  3. The performance gain is limited compared to some simple alternatives. Table 2 shows that the fusion improves accuracy from 84.15% (Class-only) to only 84.50%. Appendix E.1 shows that a Class+global variant achieves 84.54% accuracy.

  4. My main concern about this work is that for fine-grained visual recognition, the cost can be less important than the capability like recognition accuracy, interpretability, and robustness, etc.

Questions for authors
  1. The router is trained per dataset. Can it generalize across datasets, or does it accumulate as the number of seen categories grows?

  2. Were the compared baselines evaluated using the same base MLLM?

  3. Other grammatical errors:

  • lines 43 to 44: which skill should be used for image be recognized? -> which skill should be used for an image to be recognized?

  • lines 217 to 218: when only few profiling examples are available -> when only a few profiling examples are available

6 comments
AuthorsOpenReview ↗

Response to Questions For Authors

Response to ''Questions For Authors''

1. The router is trained per dataset. Can it generalize across datasets, or does it accumulate as the number of seen categories grows?

We thank the reviewer for raising this important question. VEFR supports cross-dataset few-shot adaptation and incremental adaptation to new categories:

Cross-dataset few-shot adaptation. We conduct leave-one-dataset-out (LODO) evaluation across eight datasets. In each fold, hyperparameters are selected using the seven source validation sets and transferred to the held-out target dataset without target-specific validation tuning. Source-global selects a fixed skill based on average source competence.

Table R1. Cross-dataset few-shot adaptation under LODO.

MethodTarget records/classAccuracy (%) ↑Tokens/image ↓
Source-global083.757,841
Target-only VEFR1683.977,081
Source-prior + Target VEFR1684.187,393

With 16 target records per class, Source-prior + Target VEFR improves accuracy over Source-global by 0.43 pp (95% CI: [+0.31, +0.54]) while reducing token consumption by 5.7%. Compared with Target-only VEFR, the source prior provides an additional 0.21 pp improvement (95% CI: [+0.12, +0.30]), at a cost of 313 additional tokens/image.

Incremental category expansion. We evaluate SUN397 by expanding the active vocabulary from 50 to 100, 200, and 397 classes. Hyperparameters selected on the initial 50 classes are reused throughout. Here, 0-shot means no competence records are available for newly introduced classes; 16-shot provides 16 records per new class, while Full uses all available profiling records.

Table R2. Few-shot adaptation under incremental category expansion on SUN397.

Active classesNew classes0-shot Acc. (%)16-shot Acc. (%)Full Acc. (%)Records (16/class)
500———800
1005085.7289.6890.881,600
20010077.8483.0283.783,200
39719771.6277.7178.386,352

At 397 classes, 16 records per new class improve accuracy by 6.08 pp (95% CI: [+5.45, +6.72]), bringing it within 0.67 pp of Full profiling. The profiling requirement grows with the number of categories, but no additional trainable routing parameters are introduced.

Overall, these results demonstrate that VEFR supports cross-dataset few-shot adaptation and incremental adaptation to new categories.

2. Were the compared baselines evaluated using the same base MLLM?

Yes, all MLLM-based recognition skills are executed using the same base MLLM, Qwen3-VL-8B-Instruct, to ensure a fair comparison. As stated in Section 4.1 (Experimental Setup), all routing baselines use the same five-skill pool and are evaluated using identical pre-recorded skill outcomes, ensuring that the comparison isolates the effectiveness of different routing strategies rather than differences in the underlying recognition models. We acknowledge that this setup could be stated more explicitly and will further clarify this point in the main text.

3. Other grammatical errors

We thank the reviewer for these helpful suggestions. We will correct the two grammatical errors as suggested. We will also carefully proofread the manuscript to improve its overall grammatical correctness.

Response to ''Critical Weaknesses 1''

1. This work focuses on budgeted skill selection for fine-grained visual recognition. Yet very few clues about what the skills exactly are were provided.

We thank the reviewer for this comment. The five skills are concrete and executable recognition strategies, with detailed implementations and prompts provided in Appendix A.2 (Executable Recognition Skills and Prompts). Briefly, they include:

  1. CLIP kNN Retrieval (Zhang et al., 2022): Retrieves similar labeled images using CLIP features for classification.
  2. Direct MLLM (Conti et al., 2025): Directly asks the MLLM to select a category from candidate labels given the query image.
  3. Category Description (Yang et al., 2026): Provides textual descriptions of candidate categories to facilitate fine-grained discrimination.
  4. CoT(Conti et al., 2025; Lawrence et al., 2026): First generates an unrestricted prediction, then maps it to a candidate label through reasoning.
  5. Visual ICL (Xiong et al., 2026): Uses retrieved labeled image–text examples to guide the MLLM's recognition.

These skills exploit complementary recognition strategies while incurring different inference costs, making them suitable candidates for budget-aware skill selection.

Response to ''Critical Weaknesses 2''

2.The competence estimator is a fixed-weight combination of class-conditioned accuracy statistics and similarity-weighted neighboring outcomes. The components and fusion are straightforward.

We thank the reviewer for this insightful comment. We agree that the two competence components and their weighted combination are mathematically straightforward. We would like to clarify that our contribution is not to introduce a novel fusion operator, but to develop a framework for estimating the competence of heterogeneous recognition skills before executing them, using two complementary sources of visual evidence. Class-conditioned statistics capture category-level skill preferences, while similarity-weighted outcomes from visually neighboring profiling images capture instance-level variations in skill competence within a category.

Appendix C.1 shows that VEFR changes 9.22% of the skill-selection decisions made by Class-only. On these queries, the accuracy increases from 73.96% to 77.68%, suggesting that local evidence enables selective corrections to class-conditioned routing decisions:

Appendix C.1. Table 6: Effect of evidence fusion on routing decisions. Statistics use frozen Test routes across eight datasets; the last two rows condition on queries whose Class-only choice is changed by VEFR.

StatisticValue (%)
Class and Local choose different skills49.13
VEFR changes the Class-only choice9.22
VEFR accuracy among changed choices77.68
Class-only accuracy among changed choices73.96

Comparison with learned fusion alternatives:

  1. Skill-wise Learned Linear Fusion: learns an independent fusion coefficient for each recognition skill, shared across queries.
  2. Instance-adaptive Fusion: employs an MLP to predict image-specific fusion coefficients for each skill from the image embedding and the two competence vectors.

Both variants use leave-one-out evidence construction to avoid self-information leakage and are trained using binary cross-entropy on profiling outcomes.

Table R3. Comparison of fixed and learned competence fusion .

Fusion strategyAccuracy (%) ↑Online tokens/image ↓ΔAcc. vs. Fixed (pp) [95% CI]Added offline time (s) ↓
Fixed Hybrid (VEFR)84.504,747—0.0
Learned Linear84.544,789+0.046 [−0.061, +0.151]176.7
Adaptive MLP84.564,833+0.063 [−0.045, +0.171]222.6

The added offline fitting times are summed across all eight datasets and include leave-one-out evidence preparation, model fitting, and validation-based model selection.

Although the learned alternatives achieve slightly higher accuracy point estimates, neither improvement is statistically significant, as both paired 95% confidence intervals include zero. Moreover, both incur higher online token consumption and additional offline fitting costs.

We therefore retain fixed-weight fusion as a deliberate design choice: it effectively integrates complementary competence evidence without requiring an additional trainable routing network.

We will revise the method discussion and experimental analysis to clarify that the contribution lies in visual-evidence-based competence estimation and pre-execution routing, rather than in the mathematical novelty of the weighted-sum operation itself.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Area Chairs

We sincerely thank all reviewers for their careful evaluations and constructive feedback, and for recognizing the practical value of VEFR in improving the accuracy–cost trade-off for fine-grained visual recognition.

We are encouraged that Reviewer yAXS recognizes our work as addressing "a practical problem" with "a simple and interpretable design," that Reviewer ukqs appreciates the consistent performance across multiple benchmarks and comprehensive experiments, that Reviewer Nw4t identifies accuracy improvement with reduced token consumption as "a strong contribution," and that Reviewer UbD4 finds our formulation of recognition as skill selection "interesting."

We would also like to respectfully clarify several potential misunderstandings, particularly those underlying the concerns raised by Reviewers UbD4 and Nw4t.

First, VEFR's efficiency gains do not simply come from selecting smaller models. All MLLM-based recognition skills use the same underlying MLLM, and their differences arise from recognition strategies rather than model sizes. Our additional experiments further confirm that the improvements come from effective image-specific skill selection.

Second, not requiring an additional trained routing network does not mean that VEFR has no offline computation cost. We have explicitly reported profiling costs, evaluated reduced-profiling settings, and analyzed how these upfront costs can be amortized over large-scale deployment. We have also clarified that the two sources of visual evidence estimate competence for all candidate skills before routing, without requiring test-time ground-truth labels.

In response to the broader concerns, we conducted extensive additional experiments, including matched-feature routing comparisons, within-class shuffling, matched-budget evaluations, statistical significance analysis, and further investigations of skill complementarity and generalization. These results provide stronger evidence for VEFR's effectiveness and clarify the contributions of its individual components.

We are particularly grateful that Reviewer yAXS explicitly acknowledged that several concerns had been addressed, found our additional explanations convincing, and subsequently increased scores. We have also uploaded a revised manuscript incorporating the additional experiments and methodological clarifications.

We hope these clarifications help the reviewers and Area Chairs distinguish potential misunderstandings of our methodology and experimental settings from substantive limitations of the work. We respectfully invite the reviewers to reconsider their assessments in light of our detailed rebuttal and revised manuscript.

Once again, we sincerely thank all reviewers and Area Chairs for their time and constructive feedback, and we welcome any further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.