- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes MAVER, a label-free, training-free method for ranking black-box multimodal models by combining cross-model semantic agreement, visual dependence measured through image interventions, and semantic-entropy-based sample weighting, and evaluates it across 32 models and 15 datasets.
Critical strengths
- The paper is well structured, with clear problem formulation and coherent presentation of both the motivation and its method.
- The idea of combining cross-model agreement with visual dependence for label-free black-box ranking is interesting, supported by extensive experiments across 32 models and 15 datasets.
Critical weaknesses
- The validity of the visual-dependence signal requires further justification. Keeping the question fixed while randomly replacing the image may introduce irrelevant visual distractors or image–question mismatches. Consequently, contradictions between the original and intervened responses may reflect sensitivity to noise or hallucination under mismatched inputs, rather than meaningful visual grounding. Although the paper acknowledges that visual dependence does not directly measure correctness, it remains unclear why this signal should increase the reliability weight of the original answer cluster.
- Potential benchmark contamination is not discussed. Since the evaluation relies on public benchmarks, some candidate models may have encountered evaluation samples or answers during training or fine-tuning. Such overlap could affect both the reference rankings and cross-model agreement, with agreement potentially reflecting shared memorization rather than independent task solving. While this does not necessarily invalidate ranking against observed benchmark scores, it leaves unclear whether the reported effectiveness generalizes to genuinely unseen target data.
- Ranking robustness to candidate-pool composition is insufficiently evaluated. The test model pool is unevenly distributed across model families, including 6 InternVL variants and 4 LLaVA-OneVision variants, but only one Qwen2-VL and one DeepSeek-VL2 model. Closely related models may share errors and disproportionately influence agreement scores, yet ranking stability under different pool compositions is not examined. Also, although MAVER targets black-box model ranking and motivates this setting using commercial LMMs, all evaluated candidates are open models. Treating these models as black boxes is valid, but lack of closed-source API models leaves its effectiveness in the motivating deployment setting insufficiently validated.
Questions for authors
- How does the proposed visual-dependence signal distinguish meaningful use of visual evidence from susceptibility to irrelevant images or image–question mismatches? Could you provide further justification or experimental evidence that response contradictions under random image replacement reliably improve the assessment of the original answers? A convincing clarification addressing this concern would make me reconsider my score upward.
- Given that the evaluation relies on public benchmarks, could potential benchmark contamination affect both the reference rankings and cross-model agreement? Is the ranking performance remains still reliable on newly collected or less contaminated data?
Response to the authors
Thank you for the additional analyses.
Q1. The family-removal and family-balancing results (mean flip rates of 2.94%, 1.76%, and 4.87%) address my main concern within the tested pools. MathVista and MMMU-Pro remain sensitive (12.05% and 11.28% flips under family balancing), and Proposition 1 covers Degree, not the full MAVER estimator.
Q2. Thank you for confirming the C.4 implementation and reporting the contradiction rates (12.31% of all groups; 38.72% of groups with two or more distinct answers). The multicut effect depends on the dataset: on WildVision ρ drops from 0.81 to 0.30, while MathVista and MMMU improve. Please describe this effect per dataset, add a mean row, and use "compatibility groups" in Sec. 4.1 and Eq. 7. Please also report per-dataset flip rates and state the scope of Proposition 1.
I have raised my soundness and informativeness scores. I keep the overall rating at 2 because the model-selection claim is supported within the evaluated pool but not yet beyond it: rankings shift on some datasets when the pool changes, and after adding the two API models, their pairwise agreement with the reference ranking is 32.81% on MMMU and 29.69% on MMMU-Pro.