- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper provides a framework for fusing heterogeneous VLMs by selecting sub-ensembles using visual and error diversity via genetic search, training fusion models, and applying an epistemic uncertainty-based rectification stage. The paper reports performance gains over individual VLMs and several baselines.
Critical strengths
First, the authors practically motivated the realistic problem of heterogeneous VLM fusion, combining diverse VLMs with separate metrics for visual and linguistic diversity. Multimodal fusion remains underexplored, so their proposed dual-diversity approach is reasonable.
Second, the authors perform comprehensive experiments and ablations. Specifically, broad evaluation across multiple-choice questions and open-ended tasks, comparisons to multiple baselines, seed robustness testing, and GA optimality analysis strengthen the paper's empirical case. As such, the presented results appear reproducible and not driven by a single baseline.
Critical weaknesses
First, the paper does not clearly specify which data split is used for diversity computation, ensemble pruning, GA hyperparameter selection, fusion training, threshold selection, and final evaluation. If ensemble selection and fusion are supervised on overlapping validation/test splits, performance gains reflect benchmark-specific stacking rather than general fusion capability.
Second, the paper has relatively weak justification for uncertainty and hallucination-detection claims. Specifically, treating disagreement among independently pretrained VLMs as epistemic uncertainty uses the mutual-information framework without establishing theoretical validity. Furthermore, no calibration curves, selective-prediction metrics, or standard confidence baselines are provided.
Questions for authors
- What is the exact specification of train/validation/test usage for pruning, GA, fusion training, and threshold selection?
- Can you provide error-detection AUROC, risk-coverage curves, and comparison to max-probability and entropy baselines?
- For open-ended tasks, is the fallback truly random selection? How does this support hallucination correction?
3 comments
I want to thank the authors for addressing my comments. You provided appropriate clarification for each of the questions, and I appreciate the additional experimentation performed.
Dear Reviewer DNYt,
We would like to take this opportunity to let you and AC know that we highly appreciate your insightful and constructive comments and your positive encouragement on our rebuttal.
Sincerely,
The authors
Q1
We sincerely thank the reviewer for raising this important concern and apologize for not clearly specifying the data allocation across different stages. We have clarified the complete protocol in the accompanying table. Focal-Diversity computation and GA-based ensemble pruning use only the validation split, while the fitness weights are selected through focal model's errors in these validation episodes. The fusion model is trained using the designated training data, with validation data used for early stopping and OCR-VQA checkpoint selection. Importantly, the rectification threshold is estimated from a held-out calibration subset of the training data, without using correctness labels, and remains frozen during final evaluation. Thus, ensemble selection, fusion training, and threshold calibration are completed before evaluating the final model, without using evaluation labels to optimize these components. We appreciate the reviewer's observation and will include the detailed split-allocation table to make the protocol transparent and reproducible.
Q2
We sincerely thank the reviewer for this valuable observation. To further strengthen the empirical justification of our epistemic uncertainty measure, we conducted an additional error-detection experiment on 1,145 A-OKVQA questions, holding the fused predictions fixed and treating incorrect answers as positive instances. As shown in our new ROC analysis, the proposed epistemic uncertainty achieves the highest AUROC of 0.886 (95% CI [0.862, 0.909]), compared with mean-member entropy (0.878), maximum-probability confidence (0.857), fused entropy (0.857), and vote disagreement (0.819). Importantly, at a 10% false-positive rate, our method detects 67.1% of errors, compared with 58.7% for maximum probability, 55.5% for mean-member entropy, and 54.8% for vote disagreement. Paired comparisons further confirm improvements over maximum probability (+0.029, 95% CI [0.011, 0.048]) and vote disagreement (+0.067, 95% CI [0.046, 0.087]). At the threshold selected by Algorithm 1, our method identifies 87.6% of errors, directing 34.3% of questions to rectification. These additional results provide strong empirical evidence that our uncertainty-based approach effectively identifies unreliable predictions, particularly at low false-positive rates, supporting its role in targeted error detection and rectification.
We plot the ROC curve and include it in our revised paper. We would like to thank the reviewer for their helpful comments.
Q3
We sincerely thank the reviewer for raising this concern and apologize for the imprecise description of our fallback mechanism. Random selection does not refer to choosing an arbitrary answer. Instead, for rejected predictions, we aggregate the base models' answer-level scores over a shared candidate-answer space and select the highest-scoring answer. Random selection is employed only as a tie-breaking mechanism when multiple candidate answers share the maximum aggregated score. We will clarify this distinction in Section 7.
To further substantiate our hallucination-mitigation claims, we conducted additional experiments on POPE (9,000 samples), an object-hallucination benchmark covering random, popular, and adversarial splits, and RealWorldQA (765 samples), which evaluates real-world visual understanding and spatial reasoning. We use the same six base models and an image-disjoint 5-fold cross-validation protocol, ensuring that ensemble pruning, fusion training, and threshold calibration use only training images. Fitness weights are selected through inner cross-validation, and results are averaged across three fold assignments and five random seeds.
These results demonstrate that V3Fusion outperforms the strongest individual VLM on both benchmarks, providing additional empirical evidence for the effectiveness of our complete framework in reducing object-hallucination errors and improving visual reasoning.