- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes VEFR, a skill competence estimation approach for fine-grained image classification tasks. It uses a combination of class-conditioned competence and query-specific competence estimate, called “fixed hybrid estimator”, to select a skill to be used at inference. Since the fixed hybrid estimator is done offline, it allows VEFR to perform different inference strategies based on accuracy-cost tradeoff. Experimental results using VEFR shows same/improved accuracy while using significantly fewer tokens across 8 fine-grained image classification datasets.
Critical strengths
The formulation of task as skill selection and combining different recognition approaches such as visual retrieval, direct prediction, VICL etc. as skills for fine-grained image classification is interesting.
Critical weaknesses
-
The method section of the paper is unclear and hard to follow.
-
Sec. 3.2 introduces Eq. (5) without explaining what represents. However, it is not clearly explained what this ratio represents and why it should be used to compute class-conditioned competence estimates.
-
In Sec. 3.3, Eq. (9), what does represent?
-
The inference setting is also unclear. Eq. (10) uses for skill selection, but it is not clear how is estimated during inference. During training, is available, but how is computed during test-set evaluation?
-
References to the appendix are not sufficiently specific. While I was happy to consult the appendix for additional details, the references do not correctly point to the relevant subsections, making the paper difficult to navigate.
-
-
The paper mentions training at multiple places (e.g., Lines 323 and 354). However, it is unclear what exactly is being trained in the overall pipeline.
-
Table 1 presents results for the "Oracle Skill Selector," but I could not find details explaining how the oracle skill is selected.
-
While the reduction in token usage is beneficial, the accuracy gains are marginal. A variance analysis would help determine whether these improvements are statistically meaningful or simply due to noise.
Questions for authors
-
While Fig. 1b shows specific examples, could the authors provide a more comprehensive analysis of how each skill performs across different classes? The main motivation of the paper relies on the assumption that different skills solve problems in complementary ways. Therefore, an inter-class and intra-class analysis of each skill would be helpful to validate this core assumption. I understand that this is indirectly demonstrated through the Oracle Skill Selector results; however, the details of how the oracle selects skills are unclear.
-
Which model is used for skill execution in the results presented in Table 1?
-
In batch-budget allocation, why is the method constrained to select exactly one skill?
11 comments
Response to ''Critical Weaknesses The method section of the paper is unclear and hard to follow.''
1. Analysis and explanation for Eq. (5)
(1) Interpretation and motivation.
represents a similarity-weighted soft distribution over the CLIP-selected candidate classes, constructed from the labels of the query image's visual neighbors. Specifically, the numerator measures the total similarity weight of neighbors belonging to class c, while the denominator normalizes over neighbors belonging to the candidate set. In Eq. (6), this distribution weights the class-level skill accuracies to estimate expected skill competence across multiple plausible classes, rather than assuming that a single Top-1 prediction is necessarily correct.
(2) Soft vs. hard class assignment.
To test whether retaining uncertainty over candidate classes improves skill selection, we compare class-assignment strategies using the same profiling data, visual encoder, retrieved neighbors, local evidence, and skill pool. In particular, Neighbor Top-1 takes the highest-probability class from the same distribution used by our method. This comparison tests whether using the full distribution offers an advantage over using its top class alone.
Table R3. Ablation of different class-assignment strategies.
| Class Assignment | Class-only Acc. (%) | Hybrid Acc. (%) | Tokens/img |
|---|---|---|---|
| CLIP Top-1 | 79.01 | 83.46 | 4,450 |
| Neighbor Top-1 | 79.30 | 83.16 | 4,309 |
| Uniform Candidate | 83.73 | 83.92 | 6,894 |
| Soft Distribution (Ours) | 84.03 | 84.17 | 5,802 |
| Unweighted Neighbor | 83.97 | 84.09 | 5,831 |
(3) Class uncertainty analysis.
We expect soft assignment to help most when the visual neighbors provide support for several plausible classes. To test this hypothesis, we group test queries by the normalized entropy of and compare Soft Distribution with Neighbor Top-1 within each group.
Table R4. Accuracy improvements of Soft Distribution over Neighbor Top-1 across different class-uncertainty levels.
| Uncertainty Group | Test Samples | Class-only ΔAcc. (pp) | Hybrid ΔAcc. (pp) |
|---|---|---|---|
| Low | 32,688 | +2.21 | +0.33 |
| Medium | 25,606 | +3.69 | +0.93 |
| High | 22,372 | +5.92 | +1.69 |
The improvements increase with class uncertainty, indicating that soft assignment is particularly beneficial for ambiguous queries. Accuracy in both tables is the eight-dataset macro average, while token usage is weighted by the number of test images.
2. Explanation for
denotes the number of images in the profiling set, and represents the realized execution cost of skill on the -th profiling image. Thus, Eq. (9) estimates the average execution cost of each skill over the profiling set. We will explicitly define in Section 3.3 for clarity.
3. Explanation for inference setting
We thank the reviewer for pointing out this ambiguity. We clarify that is obtained during offline profiling, rather than at test-time inference. Specifically, we execute all recognition skills on the labeled profiling images and record their prediction correctness: if skill correctly predicts image , and otherwise. These correctness labels are precomputed and stored for subsequent inference.
At test time, the ground-truth label of a query image is unavailable. Therefore, cannot be computed and is never required for skill selection. Instead, VEFR retrieves visually similar images from the profiling set and uses their pre-recorded correctness labels to estimate local competence via Eq. (7). Together with the class-conditioned competence estimated by Eq. (6), these estimates are combined in Eq. (8) to obtain the hybrid competence score , which is then used for skill selection in Eq. (10).
Only the selected skill is executed on the test image. Its prediction is compared with the ground-truth label solely for final accuracy evaluation, not for skill selection or routing. We will clarify the distinction between offline profiling and test-time inference in Sections 3.1–3.3.
4. References to the appendix are not sufficiently specific
We thank the reviewer for pointing this out. We will carefully revise all appendix references throughout the manuscript to point to the exact relevant subsections, tables, and figures.
Response to ''Critical Weaknesses 2,3,4''
2. The paper mentions training at multiple places (e.g., Lines 323 and 354). However, it is unclear what exactly is being trained in the overall pipeline.
We thank the reviewer for pointing out this ambiguity. We clarify that VEFR does not involve training an additional router or fine-tuning any pretrained models (CLIP, DINOv3, or Qwen3-VL).
Instead, our pipeline consists of three steps: (1) Offline profiling: we execute the five recognition skills on labeled profiling images and record their correctness and inference costs. (2) Competence estimation: we compute class-level skill accuracy statistics and use visual neighbors to estimate query-specific competence. The hyperparameters (e.g., , , and ) are selected on the validation set. (3) Inference: the resulting statistics and hyperparameters are fixed and directly used for skill selection, without additional training.
The term "training" in Lines 354–356 primarily refers to training the learned baseline routers (e.g., MLP Router, RouterDC, and the FrugalGPT scorer). After hyperparameter selection, VEFR recomputes its profiling statistics using the full training set.
We will clarify this distinction and revise the wording throughout the manuscript to distinguish offline profiling, hyperparameter selection, and actual model training.
3. Table 1 presents results for the "Oracle Skill Selector," but I could not find details explaining how the oracle skill is selected.
We thank the reviewer for pointing this out. The Oracle Skill Selector is a hindsight upper bound, not a deployable router. For each test image, it selects a correct skill if any of the five skills produces a correct prediction.
Let indicate whether skill predicts the ground-truth class of test image , and let denote the set of recognition skills. The oracle’s correctness on image is . This value is 1 if at least one skill is correct and 0 if all five are incorrect. The reported 93.42% is the eight-dataset macro-average accuracy. Since it requires ground-truth labels and all skill outcomes, it is unavailable at inference time. We will clarify this in Section 4.2 and the Table 1 caption.
4. While the reduction in token usage is beneficial, the accuracy gains are marginal. A variance analysis would help determine whether these improvements are statistically meaningful or simply due to noise.
We thank the reviewer for pointing out the need for statistical analysis. We perform paired bootstrap analysis using the same test images for all methods, with 10,000 resampling iterations stratified by dataset and class. We report 95% confidence intervals (CIs) for both accuracy differences and relative token savings.
Table R5. Paired confidence intervals for accuracy differences and token savings (Qwen3-VL-8B).
| Baseline | Accuracy difference (pp) [95% CI] | Token savings (%) [95% CI] |
|---|---|---|
| Visual ICL | +0.747 [+0.558, +0.937] | 35.62 [35.40, 35.84] |
| Class-only | +0.343 [+0.224, +0.462] | 0.75 [0.49, 1.01] |
| Class + Global Prior | −0.042 [−0.173, +0.089] | 19.76 [19.55, 19.99] |
The results show that VEFR significantly improves accuracy over Visual ICL and Class-only, as both confidence intervals exclude zero. Compared with Class + Global Prior, the accuracy difference is not statistically significant, indicating comparable accuracy. However, VEFR reduces online token consumption by 19.76%, with a narrow 95% CI of [19.55%, 19.99%].
Overall, the paired analysis supports the accuracy gains over Class-only and Visual ICL, while showing that the more consistent advantage of VEFR is its improved accuracy–efficiency trade-off.
We will include these confidence intervals in the revised manuscript.
Response to ''Questions For Authors''
1. Inter-class and Intra-class Skill Complementarity
We thank the reviewer for this suggestion. We conduct additional analyses across all eight datasets to examine skill complementarity at both the class and instance levels.
Inter-class complementarity. We compute the accuracy of each skill for every class using the training split and identify the preferred skill for each class. All five skills are selected as the best-performing skill for some classes, with ICL. preferred for 47.9% of classes, followed by Desc. (14.7%), CoT. (14.8%), CLIP. (14.4%), and Direct. (8.2%). This shows that skill preferences vary across categories rather than being dominated by a single skill.
Table R1. Inter-class skill preferences and individual skill contributions.
The last column counts test images correctly recognized by only the corresponding skill, showing that every skill contributes unique coverage beyond the others.
Intra-class complementarity. We compare a class-level selector with a sample-level oracle. The class-level selector chooses the best skill for each category using training-set accuracy and applies that fixed choice to test images. In contrast, the sample-level oracle selects, for each test image, any of the five skills that produces the correct prediction.
Table R2. Class-level selection versus sample-level oracle across eight datasets.
The sample-level oracle improves over class-level selection by 6.36 pp overall, with positive gains across all eight datasets. In total, 5,134 test images misclassified by the class-level selection can be correctly recognized by at least one alternative skill.
Together, these results show that skill complementarity exists both across categories and among images within the same category.
We will include these analyses and clarify the oracle definition in the revised manuscript.
2.Which model is used for skill execution in the results presented in Table 1?
All MLLM-based recognition skills in Table 1 use Qwen3-VL-8B-Instruct as the shared executor to ensure a consistent evaluation setting. The CLIP kNN Retrieval skill uses CLIP ViT-B/16.
We will explicitly clarify the MLLM executor used in Table 1 in its caption.
3. In batch-budget allocation, why is the method constrained to select exactly one skill?
Our formulation focuses on single-skill selection for each query image, rather than multi-skill execution or ensemble. The batch-budget optimizer jointly allocates different skills across queries, allowing more expensive skills to be assigned to images with higher estimated competence gains while keeping the total estimated cost within the given budget.