acceptodds
Under review as a conference paper at ICLR 2027

Vision Verification Enhanced Fusion of VLMs for Efficient Visual Reasoning

Abstract

Ensemble of either large language models or large vision models has been studied extensively to date. However, there is an increasing demand on tackling the problem of ensemble of diverse Vision-Language Models (VLMs) based on both vision and language modalities jointly. This paper presents V3Fusion, a Vision-Verification enhanced VLM Fusion framework for high performance visual reasoning with three unique contributions. First, V3Fusion introduces focal error diversity to capture complementary reasoning across VLMs by combining language-based focal error diversity with a Focal Central Kernel Alignment (CKA) based focal diversity metric (CKA-focal) to measure disagreement in visual embeddings. Second, on the constructed ensemble surface from a pool of candidate VLMs, we applied a Genetic Algorithm to effectively prune out those component VLMs that do not add value to the fusion performance. Finally, we identify the best combination for each reasoning task and resolve conflict among the outputs of multiple VLMs in the ensemble model pool, enabling heterogeneous models to capture epistemic uncertainty dynamically while mitigating hallucinations. Our V3Fusion approach is capable of producing dual focal-diversity fused predictions with high performance for vision-language reasoning, even when there is no majority consensus or the majority of VLMs make incorrect predictions. Extensive experiments with four popular VLM benchmarks (A-OKVQA, MMMU, MMMU-Pro, and OCR-VQA) validate the effectiveness of V3Fusion. The results show that V3Fusion outperforms the best-performing VLM on MMMU by 4.54% and MMMU-Pro by 2.29% gain in accuracy. For generative tasks, V3Fusion outperforms Intern-VL2-8b and Qwen2.5-VL-7b, the top-2 VLM performers on both A-OKVQA and OCR-VQA. Code is available at anonymous.4open.science/r/v3fusion-30D1/

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
VLM, fusion learning, error diversity, ensemble pruning, epistemic uncertainty, verification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_DNYtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides a framework for fusing heterogeneous VLMs by selecting sub-ensembles using visual and error diversity via genetic search, training fusion models, and applying an epistemic uncertainty-based rectification stage. The paper reports performance gains over individual VLMs and several baselines.

Critical strengths

First, the authors practically motivated the realistic problem of heterogeneous VLM fusion, combining diverse VLMs with separate metrics for visual and linguistic diversity. Multimodal fusion remains underexplored, so their proposed dual-diversity approach is reasonable.

Second, the authors perform comprehensive experiments and ablations. Specifically, broad evaluation across multiple-choice questions and open-ended tasks, comparisons to multiple baselines, seed robustness testing, and GA optimality analysis strengthen the paper's empirical case. As such, the presented results appear reproducible and not driven by a single baseline.

Critical weaknesses

First, the paper does not clearly specify which data split is used for diversity computation, ensemble pruning, GA hyperparameter selection, fusion training, threshold selection, and final evaluation. If ensemble selection and fusion are supervised on overlapping validation/test splits, performance gains reflect benchmark-specific stacking rather than general fusion capability.

Second, the paper has relatively weak justification for uncertainty and hallucination-detection claims. Specifically, treating disagreement among independently pretrained VLMs as epistemic uncertainty uses the mutual-information framework without establishing theoretical validity. Furthermore, no calibration curves, selective-prediction metrics, or standard confidence baselines are provided.

Questions for authors
  1. What is the exact specification of train/validation/test usage for pruning, GA, fusion training, and threshold selection?
  2. Can you provide error-detection AUROC, risk-coverage curves, and comparison to max-probability and entropy baselines?
  3. For open-ended tasks, is the fallback truly random selection? How does this support hallucination correction?
3 comments

Q1

We sincerely thank the reviewer for raising this important concern and apologize for not clearly specifying the data allocation across different stages. We have clarified the complete protocol in the accompanying table. Focal-Diversity computation and GA-based ensemble pruning use only the validation split, while the fitness weights are selected through focal model's errors in these validation episodes. The fusion model is trained using the designated training data, with validation data used for early stopping and OCR-VQA checkpoint selection. Importantly, the rectification threshold is estimated from a held-out calibration subset of the training data, without using correctness labels, and remains frozen during final evaluation. Thus, ensemble selection, fusion training, and threshold calibration are completed before evaluating the final model, without using evaluation labels to optimize these components. We appreciate the reviewer's observation and will include the detailed split-allocation table to make the protocol transparent and reproducible.

StageRevised protocol
Focal-Div + Focal-CKA + GA ensemble selectionUses the validation split
Fusion-head trainingThe same validation and triaining splits
Early stoppingBased on the validation set
Rectification threshold τ (Alg. 1)Fitted on held-out training data
OCR-VQA LED checkpointChosen on validation

Q2

We sincerely thank the reviewer for this valuable observation. To further strengthen the empirical justification of our epistemic uncertainty measure, we conducted an additional error-detection experiment on 1,145 A-OKVQA questions, holding the fused predictions fixed and treating incorrect answers as positive instances. As shown in our new ROC analysis, the proposed epistemic uncertainty achieves the highest AUROC of 0.886 (95% CI [0.862, 0.909]), compared with mean-member entropy (0.878), maximum-probability confidence (0.857), fused entropy (0.857), and vote disagreement (0.819). Importantly, at a 10% false-positive rate, our method detects 67.1% of errors, compared with 58.7% for maximum probability, 55.5% for mean-member entropy, and 54.8% for vote disagreement. Paired comparisons further confirm improvements over maximum probability (+0.029, 95% CI [0.011, 0.048]) and vote disagreement (+0.067, 95% CI [0.046, 0.087]). At the threshold selected by Algorithm 1, our method identifies 87.6% of errors, directing 34.3% of questions to rectification. These additional results provide strong empirical evidence that our uncertainty-based approach effectively identifies unreliable predictions, particularly at low false-positive rates, supporting its role in targeted error detection and rectification.

DatasetEpistemic Uncertainty1 − max pFused entropyMean-member entropyVote disagreement
A-OKVQA (14.8% errors)0.8860.8570.8570.8780.819

We plot the ROC curve and include it in our revised paper. We would like to thank the reviewer for their helpful comments.

Q3

We sincerely thank the reviewer for raising this concern and apologize for the imprecise description of our fallback mechanism. Random selection does not refer to choosing an arbitrary answer. Instead, for rejected predictions, we aggregate the base models' answer-level scores over a shared candidate-answer space and select the highest-scoring answer. Random selection is employed only as a tie-breaking mechanism when multiple candidate answers share the maximum aggregated score. We will clarify this distinction in Section 7.

To further substantiate our hallucination-mitigation claims, we conducted additional experiments on POPE (9,000 samples), an object-hallucination benchmark covering random, popular, and adversarial splits, and RealWorldQA (765 samples), which evaluates real-world visual understanding and spatial reasoning. We use the same six base models and an image-disjoint 5-fold cross-validation protocol, ensuring that ensemble pruning, fusion training, and threshold calibration use only training images. Fitness weights are selected through inner cross-validation, and results are averaged across three fold assignments and five random seeds.

ModelRealWorldQA Acc. (%) ↑POPE Acc. (%) ↑
LLaVA-1.6-7B58.0787.06
LLaVA-1.6-13B56.0786.90
Qwen2.5-VL-7B70.5587.64
InternVL2-8B64.2387.81
DeepSeek-VL2-Tiny69.3889.20
DeepSeek-VL2-Small69.5589.39
V3Fusion (Pruning + MLP + Rectification)74.57 ± 1.0390.20 ± 0.21
Gain over strongest base model+4.02 pp+0.81 pp

These results demonstrate that V3Fusion outperforms the strongest individual VLM on both benchmarks, providing additional empirical evidence for the effectiveness of our complete framework in reducing object-hallucination errors and improving visual reasoning.

I want to thank the authors for addressing my comments. You provided appropriate clarification for each of the questions, and I appreciate the additional experimentation performed.

Dear Reviewer DNYt,

We would like to take this opportunity to let you and AC know that we highly appreciate your insightful and constructive comments and your positive encouragement on our rebuttal.

Sincerely,

The authors

Reviewer_Jn8FOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select and fuse heterogeneous VLMs for multiple-choice and open-ended visual question answering. It proposes Focal-CKA and Focal-Diversity for ensemble pruning, learned MLP and LED fusion models, and uncertainty-based rectification, evaluated on MMMU, MMMU-Pro, A-OKVQA, and OCR-VQA.

Critical strengths
  • The empirical fusion results are consistently competitive across the main tasks.

  • The paper provides meaningful evidence that the MLP fuser is not merely exploiting one fortunate seed.

Critical weaknesses

In Eq(2), the paper defines X~i∈RK×di\tilde{\mathbf X}_i\in\mathbb R^{K\times d_i} and X~j∈RK×dj\tilde{\mathbf X}_j\in\mathbb R^{K\times d_j}, but then sets K=X~iX~j⊤\mathbf K=\tilde{\mathbf X}_i\tilde{\mathbf X}_j^\top. This product is undefined when di≠djd_i\neq d_j, precisely the heterogeneous-encoder setting the method is intended to handle. Moreover, for linear CKA, the representation-specific sample Gram matrices should be K=X~iX~i⊤\mathbf K=\tilde{\mathbf X}_i\tilde{\mathbf X}_i^\top and L=X~jX~j⊤\mathbf L=\tilde{\mathbf X}_j\tilde{\mathbf X}_j^\top, rather than a cross-representation product used as one of the kernels in HSIC.

There is a related inconsistency in Eq (4): The text defines P(1)P(1) as the probability that one uniformly selected member of an ensemble of size SS fails, and the subsequent S=3S=3 example explicitly uses probabilities 1/31/3, 2/32/3, and 11. Yet the equation uses jNpj\frac{j}{N}p_j, where NN is the global model-pool size, rather than jSpj\frac{j}{S}p_j. Further, although ϕdiv(Mi;E)\phi_{\mathrm{div}}(\mathcal M_i;\mathcal E) is described as focal-model-specific, pjp_j is defined only through the number of ensemble members failing on an episode, with no dependence on ii. As written, the score is therefore not focal-model-specific.

As a result, I tend to believe that reader cannot faithfully reproduce the proposed procedure, nor determine whether the reported selected ensembles correspond to the equations in the paper. The authors should state exactly what the implementation computes, and verify whether correcting/changing these expressions changes the selected ensembles or reported ablations.

Questions for authors

In seciton 7, how is the distribution P(y∣x,θ(i))P(y\mid \mathbf{x},\theta^{(i)}) placed in a common outcome space for heterogeneous models on OCR-VQA, where vocabularies and free-form outputs differ? The MCQ choice distribution is clear, but the open-ended implementation of the entropy and mutual-information calculation is not.

2 comments

Weaknesses

We sincerely thank the reviewer for carefully examining our mathematical formulations and apologize for the ambiguities.

Regarding Equation 2 (CKA), the Gram matrices are correctly defined in the manuscript as (K=X~iX~i⊤K=\widetilde{X}_i\widetilde{X}_i^\top) and (L=X~jX~j⊤L=\widetilde{X}_j\widetilde{X}_j^\top). Both are sample-by-sample Gram matrices with dimensions independent of the respective visual encoder feature sizes. Therefore, CKA remains well-defined even for heterogeneous VLMs with different embedding dimensions. We apologize if the notation caused confusion and will ensure these definitions are unambiguous.

Regarding Equation 4, we acknowledge a typographical error: (j/Nj/N) should be (j/Sj/S), where (S=∣E∣S=|\mathcal{E}|) denotes the candidate ensemble size. We sincerely apologize for this oversight.

Regarding focal-model specificity, both Focal-CKA and Focal-Diversity are computed on the failure episodes of each focal model, rather than on the entire validation dataset. Specifically, for each focal model (Mi\mathcal{M}_i), we identify the subset (Di−\mathcal{D}_i^{-}) containing only examples where (Mi\mathcal{M}_i) predicts incorrectly. Both the CKA similarities and the joint failure probabilities are then calculated on this focal-specific subset. Consequently, these statistics depend explicitly on the selected focal model. As defined in Equation 1, we repeat this procedure for every member of the candidate ensemble and average the resulting focal-specific diversity scores to obtain the ensemble-level score.

We recognize that the manuscript does not sufficiently emphasize this conditioning and will revise Equations 3–4 to explicitly include the focal-model indices and failure subsets. We greatly appreciate the reviewer's attention to these details and would be happy to provide any further clarification.

Questions

We sincerely thank the reviewer for highlighting this missing implementation detail. We apologize for not explicitly describing the probability construction for open-ended questions in Section 7. Unlike MCQs, where probabilities are naturally aligned over the available choices, for open-ended questions we construct a shared answer space using empirical answer frequencies. Specifically, for each image-question pair, we sample each base VLM (k) times and collect the distinct generated answers across all ensemble members. These answers form a common categorical outcome space, independent of the models' tokenizers or vocabulary sizes. For each model, we estimate the probability of an answer by dividing its occurrence count by (k), assigning zero probability to answers not generated by that model. Consequently, all models have probability distributions over the same answer space, allowing us to compute the entropy and mutual-information terms in Equation 5 without requiring token-level vocabulary alignment. We will explicitly incorporate this frequency-based probability estimation procedure into Section 7 and Appendix C to improve clarity and reproducibility. We sincerely appreciate the reviewer's careful observation.

For each VLM Mi\mathcal M_i, generate kk responses to the same image-question pair:

$ \mathcal A_i={a_i^{(1)},a_i^{(2)},\ldots,a_i^{(k)}}. $

Define the common answer space as the union of distinct generated answers:

$ \mathcal A=\bigcup_{i=1}^{S}\operatorname{Unique}(\mathcal A_i). $

For every answer �2��2�, estimate:

$ \boxed{ \hat p_i(a\mid x,I)= \frac{1}{k}\sum_{t=1}^{k} \mathbb{1}[a_i^{(t)}=a] } $

For example, if a model generates Harry Potter three times and The Hobbit twice across five samples, their empirical probabilities are 0.6 and 0.4.

With a uniformly weighted model ensemble, the predictive distribution becomes

$ \bar p(a)=\frac{1}{S}\sum_{i=1}^{S}\hat p_i(a), $

and the corresponding disagreement-based mutual information is

$ U_{\mathrm{epi}} = H(\bar p)-\frac{1}{S}\sum_{i=1}^{S}H(\hat p_i). $

Therefore, we can capture the disagreement and uncertainty of the open-ended models in this way. However, we accept the limitations of this methodology, which is that the generative type of questions can produce very long answers, and counting the frequency of the same answers is not feasible. The method is more based on questions whose answers are solution spaces of a number (like math questions) or a couple of words.

Thank you for clarifying the focal-failure conditioning and shared answer-space construction. About Eq. (2), I withdraw that objection. I retain my weak-accept recommendation, but please specify the open-ended sampling and answer-normalization protocol, including its inference cost.

Reviewer_4CRvOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+2)
Read review
Paper summary

This paper proposes V3Fusion, a framework for combining multiple VLMs to improve visual question answering. It uses visual representation similarity (CKA) and error diversity to select complementary models, then trains an MLP or LED to fuse their predictions. An uncertainty-based mechanism is further introduced to revise uncertain predictions. Experiments on four VQA benchmarks show improvements over individual models and several ensemble baselines.

Critical strengths

I appreciate that the paper addresses a practical problem: how to combine heterogeneous VLMs while reducing redundant computation. The authors also provide evaluations across multiple benchmarks, including comparisons with ensemble baselines and ablation studies. These experiments suggest that learning from complementary model predictions can be useful.

Critical weaknesses
  1. The core diversity metrics are not clearly justified.

My main concern is that the proposed metrics do not seem to match their stated motivation. The paper emphasizes measuring diversity conditioned on a focal model's failures. However, Equation 4 uses overall error statistics without explicitly conditioning on the focal model. Its normalization also appears inconsistent with the stated range of the metric. Similarly, Focal-CKA is computed over all samples rather than the focal model's failure cases. I am therefore not convinced that these metrics capture the claimed failure-conditioned complementarity.

  1. The experiments do not sufficiently isolate the proposed contributions.

In Table 4, pruning and fusion together improve MMMU-Pro by only 0.36 points, while the full method improves it by 2.29 points. Most of the gain therefore comes from the rectification stage, which falls back to aggregating base-model predictions. It remains unclear how much improvement actually comes from the proposed diversity metrics. I also notice inconsistencies in the strongest base-model scores between Tables 2 and 3, which makes the reported gains difficult to interpret.

Questions for authors

Q1. Could you clarify how Equations 3 and 4 implement conditioning on the focal model's failures? In particular, why does Equation 4 use (N) rather than (S) in the denominator of (P^{(1)})? Is this a notation error or the actual implementation?

Q2. Could you compare the full method against simpler probability fusion and uncertainty-based fallback methods under the same supervision and inference budget? This would help establish whether Focal-CKA and Focal Diversity provide meaningful improvements beyond standard ensemble methods.

Q3. Could you explain the different strongest base-model scores in Tables 2 and 3? Which results should be used to calculate the final improvements?

7 comments

W1, Q1

We sincerely thank the reviewer for carefully examining our diversity metrics and identifying ambiguities in their mathematical formulation. We apologize for the lack of clarity and the typographical error in Equation 4: (j/Nj/N) should be (j/Sj/S), where (S=∣E∣S=|\mathcal{E}|) is the candidate ensemble size. Importantly, both Focal-Diversity and Focal-CKA are computed exclusively on the validation episodes where the focal model (Mi\mathcal{M}_i) makes incorrect predictions. To clarify Equation 1, for each candidate ensemble (E\mathcal{E}), we successively designate each member (Mi\mathcal{M}_i) as the focal model and calculate its diversity relative to the remaining ensemble members using only its own failure episodes. We then average these focal-specific scores across all (SS) members to obtain the ensemble-level diversity score. Thus, each member contributes equally to the final score, but its contribution is evaluated on its own failure subset, capturing complementarity precisely where that model struggles. We acknowledge that this conditioning was insufficiently specified in Equations 3–4 and will revise the definitions accordingly. We sincerely apologize for the confusion, appreciate the reviewer's careful feedback, and would be happy to provide further clarification.

W2, Q2 and Q3

We sincerely thank the reviewer for this valuable observation. We agree that rectification contributes substantially to the final improvement on MMMU-Pro. However, the contribution of diversity-guided pruning is best understood in conjunction with learned fusion rather than as an isolated stage. Our additional ablation shows that, using identical pruned ensembles, MLP fusion outperforms plurality voting by 2.16 percentage points on MMMU and 2.15 on MMMU-Pro, demonstrating its ability to exploit complementary model predictions beyond simple voting. More importantly, Figure 4 directly validates our proposed diversity metrics: combining Focal-CKA with Focal-Diversity improves accuracy over Focal-Diversity alone by 1.83 points on MMMU and 3.27 on A-OKVQA, outperforming either metric individually. Furthermore, our additional rectification ablation shows that epistemic uncertainty consistently outperforms maximum-probability and entropy-based alternatives across all three datasets, with all methods using the same selected ensembles and adaptive threshold procedure (Algorithm 1). These results help distinguish the contributions of diversity-guided selection, learned fusion, and uncertainty-based rectification.

Additional Ablation Study :

MethodPruningIDsMMMU*MMMU-Pro'A-OKVQA^
Majority Voting✗12345647.5844.1287.86
Plurality Voting✗12345654.9547.5385.76
Mean choice probabilities✗12345649.6945.9588.73
V3Fusion (MLP)✗12345654.9547.5385.76
V3Fusion (Voting + Pruning)✓235*|234'|123^52.9145.1988.31
V3Fusion (MLP + Pruning)✓235*|234'|123^55.0747.3488.31
V3Fusion (MLP + Pruning + Rectification )(max-probability)✓235*|234'|123^53.9446.7987.79
V3Fusion (MLP + Pruning + Rectification )(entropy)✓235*|234'|123^53.9446.8488.86
V3Fusion (MLP + Pruning + Rectification )(epistemic uncertainty)(paper's version)✓235*|234'|123^56.0949.2789.17

Rectification (max-probability) : The score is 1 − max_k p_k, so the lower the MLP's confidence in its top answer, the higher the score. The question is rejected when the score is above the threshold. Rectification (max-probability) : The score is H(p) = −Σ_k p_k log p_k, which measures how spread out the whole distribution is. The question is rejected when the score is above the threshold. Rectification (epistemic uncertainty) : The version in the paper. It measures the epistemic uncertainty, which indicates that when the model is unsure, it hasn’t learned the data distribution well, and eventually will help us to prevent hallucination.

The threshold selection for all of the Rectification methods of different metrics is done through the gate-threshold selection process we introduced in Alg. 1.

We also sincerely apologize for the inconsistent strongest-base scores in Tables 2 and 3. Table 2 contains the correct results, which we have confirmed against our implementation and repository: 51.55% (MMMU), 46.98% (MMMU-Pro), and 87.24% (A-OKVQA). The strongest-base row in Table 3 was incorrectly transcribed. The reported improvements of +4.54, +2.29, and +1.93 percentage points, respectively, were calculated using the correct Table 2 values and therefore remain unchanged. We deeply regret this oversight and will correct Table 3 accordingly.

Thank you for the response. The clarification of Table 3 and the correction of Eq. 4 address my concerns about the reported gains and the normalization, so I have raised my Soundness score from 1 to 2. I am keeping my overall rating for two reasons. First, the new ablation shows that pruning adds little to the MLP on MMMU and MMMU-Pro, and the simplest baseline, mean choice probabilities with the same rectification, is still missing. Second, Table 9 shows that the fused ensemble uses about 1.7× the memory of Qwen2.5-VL-7B. A single larger or more recent VLM with a similar budget may match or exceed these gains, so the practical value of fusing these models remains unclear.

We deeply appreciate the Reviewer's quick feedback and raising our soundness score to 2. We would like to address the remaining concerns:

First

We sincerely thank the reviewer for this constructive suggestion. We agree that comparing against mean choice probabilities with identical pruning and rectification settings is important for isolating the contributions of our framework. We would first like to clarify that our rectification mechanism replaces rejected predictions with the mean of the selected base models' probability distributions. Therefore, when the initial prediction already uses mean probability aggregation over the same ensemble, rectification leaves the prediction unchanged, regardless of the threshold. Consequently, Mean Probabilities + Rectification is mathematically equivalent to Mean Probabilities under the same selected subset.

To provide a more comprehensive comparison, we additionally evaluate mean probability aggregation both before and after diversity-guided pruning, including two new benchmarks, POPE and RealWorldQA.

MethodA-OKVQA ↑MMMU ↑MMMU-Pro ↑POPE ↑RealWorldQA ↑
Mean probabilities88.7349.6945.9589.1672.21
Mean probabilities + pruning (+ rectification)88.4753.5447.2089.1571.94
V3Fusion (Pruning + MLP + Rectification)89.1756.0949.2790.2074.57

These results provide further insight into the role of pruning. Even with simple mean probability aggregation, diversity-guided pruning improves MMMU by 3.85 percentage points and MMMU-Pro by 1.25 points, demonstrating that selecting complementary models can be beneficial independently of learned fusion. Furthermore, compared with the pruned mean-probability baseline, the complete V3Fusion framework achieves additional improvements of 2.55 points on MMMU, 2.07 on MMMU-Pro, 1.05 on POPE, and 2.63 on RealWorldQA, outperforming this baseline on four of the five benchmarks. These findings demonstrate that, although the isolated contribution of pruning varies across datasets, diversity-guided selection combined with learned fusion and uncertainty-based rectification provides consistent benefits across multiple visual reasoning tasks. We appreciate the reviewer's suggestion and will incorporate these additional comparisons to clarify the individual and combined contributions of our framework.

Second

We sincerely thank the reviewer for raising this important practical consideration. Table 9 reports the memory requirements of our fully local implementation, where three selected VLMs are deployed together, rather than an inherent memory requirement of V3Fusion. A key advantage of our framework is its efficient diversity-guided pruning, which eliminates redundant models and reduces the number of VLMs required for subsequent inference. As demonstrated in Table 1, pruning a pool of 20 models takes less than one minute, while our extended experiment with 12 models, including substantially larger VLMs (Table 11 and Figure 8), achieves a 90% reduction in ensemble-search time compared with exhaustive search.

ModelPoolMMMU (Acc %)
gemma-4-31B-itAdded78.34
Qwen3-VL-235B-A22BAdded75.92
InternVL3.5-241B-A28BAdded70.45
Qwen2-VL-72BAdded61.15
pixtral-12b-2409Added53.63
InternVL2-8BOriginal51.08
Qwen2.5-VL-7BOriginal50.32
granite-vision-4.1-4bAdded47.52
deepseek-vl2-tinyOriginal37.71
llava-v1.6-vicuna-13bOriginal36.69
llava-v1.6-vicuna-7bOriginal35.54
deepseek-vl2-smallOriginal31.72
V3Fusion (Pruning + Fusion + Rectification)-79.13

Importantly, V3Fusion does not require jointly hosting or fine-tuning the selected VLMs. Once the pruning is performed, the selected models can potentially be accessed through API providers such as Together AI or DeepInfra, provided their prediction scores are available. This avoids the need to host all selected models locally, although API inference costs remain. Furthermore, V3Fusion is model-agnostic and is not restricted to small VLMs: stronger and larger models can also be incorporated into the candidate pool, allowing the framework to exploit complementary capabilities rather than relying exclusively on a single model. Our uncertainty-based rectification additionally identifies potentially incorrect predictions and revises them using existing ensemble outputs, without additional VLM inference calls. Overall, V3Fusion can substantially reduce local memory requirements by leveraging a small set of complementary VLMs through API-based inference, without requiring their local deployment, while improving upon the performance of the strongest model in the candidate pool.

Reviewer_BpdPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author propose V3Fusion, which selects a subset of six candidate vision-language models using two measurements: similarity between their image features, and how often they answer the same labeled questions incorrectly. It trains an answer combiner: a neural network reads all selected models’ multiple-choice probabilities, or a text model reads the question and their generated answers. If an uncertainty score exceeds a threshold, the system replaces the combined answer with an answer computed directly from the selected models’ outputs. It reaches 56.09% accuracy on MMMU, versus 51.55% for the strongest reported single model and 54.95% for plurality voting.

Critical strengths

V3Fusion leverage information discarded by a simple vote. For example in figure 3, an option receives substantial probability after combining even when no model ranks it first. A vote discards that information.

The experiment design and evaluation setup is sound.

V3Fusion improves over relevant ensemble baselines. (e.g. On MMMU, rectified fusion reaches 56.09% vs plurality voting’s 54.95%. On MMMU-Pro, it reaches 49.27% vs PairRanker’s 48.02%.)

Critical weaknesses

The performance-cost comparison seems incomplete. The OCR performance and timing tables list different model subsets, plus offline selection and training also use resources.

Will the uncertainty gate still work on a different incoming batch? A threshold fitted successfully on one batch may not maintain the same error rate elsewhere.

Questions for authors

See weakness

2 comments

W1

We thank the reviewer for highlighting the need for clearer cost accounting. The manuscript already reports analysis and selection timings in Figure 2(b) and Table 1, training times in Appendix F.1, Tables 7–8, and inference latency and memory in Table 9. For a fixed dataset and model configuration, selection and fusion training are offline setup operations whose costs can be amortized across subsequent queries, although they remain part of the total resource budget. We agree that the different subsets listed in the OCR performance and timing tables prevent a direct performance–cost comparison as currently presented. We will verify the underlying configurations, standardize the model identifiers, and report OCR performance, latency, and memory for the same selected ensemble. We will also consolidate the offline cost breakdown, explicitly accounting for base-model output and visual-feature collection, diversity computation, ensemble search, and fusion training, so that the distinction between setup cost and recurring inference cost is transparent.

Table 1. Ensemble-search time: brute force versus genetic algorithm

The reported reduction compares genetic-algorithm (GA) search time with brute-force (BF) search time.

Number of candidate VLMsBrute-force time (s) ↓Genetic-algorithm time (s) ↓Time reduction (%) ↑
59.409.90−5.31
10228.2024.5089.26
15508.9941.8091.78
2016,201.5154.9099.66

Table 7. V3Fusion-LED versus supervised fine-tuned VLM aggregators

These SFT models are trained to aggregate candidate outputs. The accompanying text reports 35 minutes of training for V3Fusion-LED and 1.47 hours for the Qwen2.5-7B aggregator; those timings are not columns in the original table.

MethodModel IDsOCR-VQA BLEU-1 ↑OCR-VQA EM (%) ↑OCR-VQA F1 ↑
LLaVA-v1.5-Vicuna-7B (SFT)12345669.2558.9871.24
Qwen2-7B-VL-Instruct (SFT)12345684.9975.2786.49
Qwen2.5-7B-VL-Instruct (SFT)12345685.6575.5787.04
V3Fusion-LED45687.7974.9188.83

Table 8. Individually fine-tuned base models versus V3Fusion

This is the separate comparison in which the base models are fine-tuned individually, without the ensemble aggregation objective

MethodTrainingtime(h)↓OCR-VQABLEU-1↑OCR-VQAEM(%)↑OCR-VQAF1↑MMMUaccuracy(%)↑
LLaVA-v1.6-Vicuna-7B4.4050.2730.0053.8829.06
LLaVA-v1.6-Vicuna-13B7.0260.2344.0063.9830.55
Qwen2.5-VL-7B-Instruct2.1785.1274.8986.5027.95
V3Fusion1.3986.2471.9186.8256.09

Table 9. Inference time and memory

The two memory columns reproduce the paper’s “Allocated” and “Consumed” labels. Model IDs follow Table 9’s own mapping.

Model / MethodModel ID(s)Mean inference time (ms) ↓Allocated memory (MB)Consumed memory (MB)
LLaVA-v1.6-Vicuna-7B0350.7113,472.437.43
LLaVA-v1.6-Vicuna-13B1511.2525,587.727.26
Qwen2.5-VL-7B-Instruct2551.4831,774.134.02
InternVL2-8B3558.6615,917.261.08
DeepSeek-VL2-Tiny4206.146,548.170.08
DeepSeek-VL2-Small5727.0230,928.160.06
V3Fusion234768.0254,239.565.18

W2

We thank the reviewer for raising this important question. V3Fusion is designed for task-specific adaptation, where a validation dataset is used to identify complementary VLMs through ensemble pruning and train the fusion model. The resulting ensemble is then applied to incoming batches, with Algorithm 1 adaptively determining the rectification threshold from their uncertainty distributions without requiring ground-truth labels (Appendix H). We acknowledge that substantial domain shifts may alter both model complementarity and uncertainty distributions; in such cases, re-pruning and retraining the fusion model using a small representative labeled dataset may be necessary. Importantly, V3Fusion has relatively low data requirements: Appendix G, Table 10 shows that using only 5% of the training data already achieves competitive performance, including 83.34 F1 on OCR-VQA and 87.78% accuracy on A-OKVQA. Thus, adapting to new domains can be relatively inexpensive, while the uncertainty gate itself remains label-free at inference time. We will clarify these adaptation assumptions and acknowledge that maintaining a fixed error rate across shifted batches is not guaranteed.

Dear Reviewer BpdP,

Thank you for your constructive comments. We would very much appreciate your comments on whether our rebuttal addresses your questions and comments.

Sincerely,

The authors

Reviewer_wWFcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes V3Fusion, an ensemble framework for heterogeneous VLMs. Candidate ensembles are evaluated with Focal-CKA, based on CKA between visual embeddings, and Focal-Diversity, based on error correlation. A Genetic Algorithm then selects the sub-ensemble. For MCQ, each model’s option probabilities are fused with an MLP; for OEQ, output texts are fused with LED. A rectification stage applies a GMM-based adaptive threshold to ensemble epistemic uncertainty, rejects uncertain samples, and replaces their predictions with the average of the base logits. Evaluation is conducted on MMMU, MMMU-Pro, A-OKVQA, and OCR-VQA using a pool of six small VLMs.

Critical strengths
  • The motivating experiment clearly shows that different VLMs exhibit complementary error patterns.

  • The analysis in Appendix H is clean: the threshold is selected from the uncertainty distribution alone, without ground-truth labels, and then validated post hoc with labels.

Critical weaknesses
  • The substantive contribution of the proposed method is unclear: before rectification, the MLP performs almost identically to plurality voting.

  • The verification stage relies only on logit disagreement without using any visual information, so some claims appear overstated.

  • Although the paper claims hallucination mitigation, it provides no evaluation on hallucination benchmarks such as POPE or CHAIR.

Questions for authors
  • Could you also compare performance on hallucination benchmarks and spatial reasoning benchmarks?

  • Could you explain the advantages of V3Fusion compared to a single large VLM with a similar memory or compute budget?

3 comments

W1

We thank the reviewer for this helpful observation. We agree that the unpruned MLP matches plurality voting; our contribution centers on combining diversity-guided model selection with epistemic-uncertainty-based rectification. Our ablation helps distinguish these contributions: with identical pruned three-model subsets, MLP fusion improves over plurality voting by 2.16 percentage points on MMMU and 2.15 on MMMU-Pro, while matching its A-OKVQA accuracy. Most importantly, with MLP fusion, V3fusion can even find the correct answers even for cases when the majority of the models make mistakes. Epistemic rectification then improves the pruned MLP by a further 1.02, 1.93, and 0.86 percentage points, respectively.

MethodPruningIDsMMMU*MMMU-Pro'A-OKVQA^
MajorityVoting✗12345647.5844.1287.86
PluralityVoting✗12345654.9547.5385.76
Meanchoiceprobabilities✗12345649.6945.9588.73
V3Fusion(MLP)✗12345654.9547.5385.76
V3Fusion(Voting+Pruning)✓235*|234'|123^52.9145.1988.31
V3Fusion(MLP+Pruning)✓235*|234'|123^55.0747.3488.31
V3Fusion(MLP+Pruning+Rectification)✓235*|234'|123^56.0949.2789.17

The pruning is the result of the ensemble set selection with the Genetic Algorithm run using a combination of focal-diversity + focal-CKA metrics. We do also observe that the rectification stage is working well in simple dataset due to fall back rejection policy is mostly correct, but when it comes down to hard datasets such as MMMU and MMMU-pro, it requires learn-to-combine fusion enhanced by the pruning elimination of the noise of wrong models.

W2

We thank the reviewer for highlighting this important distinction. We agree that the final uncertainty gate operates on prediction disagreement and does not independently re-examine the image. Visual information contributes explicitly through Focal-CKA, which compares the visual representations extracted by candidate VLMs and, together with Focal-Diversity, guides the selection of models with complementary perceptual capabilities. This selection aims to reduce representational redundancy and provide broader coverage of task-relevant visual information; consequently, fusion and uncertainty estimation operate on an ensemble whose composition is informed by visual diversity. Figure 4 supports (we repeat this in the Table) the usefulness of this design. The subsequent verification and rectification stages reuse the probabilities and candidate answers already produced by this ensemble to flag uncertain predictions and revise the final answer. This enables **fast, self-sustained verification and rectification."

Ensemble-selection objectiveMMMU accuracy(%)↑A-OKVQA accuracy(%)↑
V3Fusion (Focal-CKA only)50.86±0.4686.57±0.24
V3Fusion (Focal-Diversity only)54.00±0.6985.34±0.56
V3Fusion**(Focal-CKA+Focal-Diversity)**55.83±1.0088.61±0.14
Gain from adding Focal-CKA to Focal-Diversity+1.83pp+3.27pp

W3 - Q1

We further evaluate V3Fusion on POPE (9000 samples), an object-hallucination benchmark with random, popular and adversarial splits, and on RealWorldQA (765 Samples), which tests real-world spatial understanding, using the same six base models, prompts and greedy decoding as in our main experiments. Neither benchmark provides a training split, so we use image-disjoint 5-fold cross-validation, fitting ensemble pruning, the fusion head and the rectification threshold only on the questions about the training images. We repeat the procedure with three fold assignments and five seeds, and report mean accuracy.

ModelRealWorldQAPOPE
LLaVA-1.6-7b58.0787.06
LLaVA-1.6-13b56.0786.90
Qwen2.5-VL-7b70.5587.64
InternVL2-8b64.2387.81
DeepSeek-VL2-tiny69.3889.20
DeepSeek-VL2-small69.5589.39
V3Fusion(Pruning+MLP+Rectify)74.57±1.0390.20±0.21
Compared to strongest basemodel+4.02pp+0.81pp

#Q2 Thank you for this thoughtful question. We compare with the strongest VLMs and improve their performance by up to 5%. Moreover, V3Fusion’s principal advantage is inexpensive task-specific adaptation while exploiting complementary pretrained models. For V3Fusion-MLP, optimization trains only two 100-unit hidden layers (Appendix D). This avoids repeatedly running and backpropagating through the base VLMs during fusion training, whereas adapting a large VLM, including through LoRA, requires repeated forward and backward computation. Appendix F.1 provides empirical support: V3Fusion-LED requires 35 minutes of training versus 1.47 hours for the Qwen2.5-7B aggregator—approximately 60% less training time—while achieving higher BLEU-1 and F1 scores (Table 7). Table 8 likewise reports 1.39 hours for V3Fusion compared with 2.17–7.02 hours for the individually fine-tuned baselines. Furthermore, verification and rectification reuse existing model outputs without additional VLM inference calls.

Dear Reviewer wWFc,

Thank you for your constructive comments. We would very much appreciate your comments on whether our rebuttal addresses your questions and comments.

Sincerely,

The authors

I appreciate the authors' detailed responses and additional experiments. The new ablation studies clarify the contribution of MLP fusion beyond voting, and the evaluations on POPE and RealWorldQA substantially address my concerns regarding benchmark coverage.

Although some concerns remain regarding the scope of visual verification and the lack of a comparison with a single larger VLM under a matched inference budget, I believe the additional evidence has sufficiently strengthened the paper.

Therefore I raise my score to Weak Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.