acceptodds
Under review as a conference paper at ICLR 2027

Ranking Black-Box LMMs Without Labels via Visually Grounded Cross-Model Agreement

Abstract

Ranking large multimodal models (LMMs) on a new task typically requires labeled evaluation data, yet in practice target data may be unlabeled and candidate models may be accessible only as black boxes. Existing unsupervised ranking methods commonly rely on model internals, labeled proxy data, or repeated generations, which are costly and can favor models that are consistently wrong. We ask whether candidate LMMs can instead provide supervision for ranking one another. We introduce Cross-Model Agreement with Visual Evidence for Ranking (MAVER), a label-free and training-free framework for LMM ranking. Given one response from each candidate model, MAVER constructs an NLI-weighted response graph and characterizes agreement using semantic-cluster prevalence and within-cluster connectivity. However, agreement alone can be misleading when multiple models share the same language prior and converge on the same incorrect answer. MAVER therefore intervenes on the image while keeping the question fixed, and measures the resulting semantic response change to quantify how strongly each response depends on the visual input. This visual-dependence signal is used to refine the support assigned to each semantic answer, while cross-model semantic entropy gives greater weight to image–question pairs that better distinguish the candidate models. Across 32 LMMs and 15 datasets, MAVER outperforms 14 comparison methods in average ranking performance, showing that cross-model agreement becomes a reliable signal for label-free LMM ranking when complemented by evidence of visual dependence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
unsupervised model ranking, cross-model agreement, visual grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Wj3JOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MAVER, a label-free, training-free method for ranking black-box multimodal models by combining cross-model semantic agreement, visual dependence measured through image interventions, and semantic-entropy-based sample weighting, and evaluates it across 32 models and 15 datasets.

Critical strengths
  • The paper is well structured, with clear problem formulation and coherent presentation of both the motivation and its method.
  • The idea of combining cross-model agreement with visual dependence for label-free black-box ranking is interesting, supported by extensive experiments across 32 models and 15 datasets.
Critical weaknesses
  • The validity of the visual-dependence signal requires further justification. Keeping the question fixed while randomly replacing the image may introduce irrelevant visual distractors or image–question mismatches. Consequently, contradictions between the original and intervened responses may reflect sensitivity to noise or hallucination under mismatched inputs, rather than meaningful visual grounding. Although the paper acknowledges that visual dependence does not directly measure correctness, it remains unclear why this signal should increase the reliability weight of the original answer cluster.
  • Potential benchmark contamination is not discussed. Since the evaluation relies on public benchmarks, some candidate models may have encountered evaluation samples or answers during training or fine-tuning. Such overlap could affect both the reference rankings and cross-model agreement, with agreement potentially reflecting shared memorization rather than independent task solving. While this does not necessarily invalidate ranking against observed benchmark scores, it leaves unclear whether the reported effectiveness generalizes to genuinely unseen target data.
  • Ranking robustness to candidate-pool composition is insufficiently evaluated. The test model pool is unevenly distributed across model families, including 6 InternVL variants and 4 LLaVA-OneVision variants, but only one Qwen2-VL and one DeepSeek-VL2 model. Closely related models may share errors and disproportionately influence agreement scores, yet ranking stability under different pool compositions is not examined. Also, although MAVER targets black-box model ranking and motivates this setting using commercial LMMs, all evaluated candidates are open models. Treating these models as black boxes is valid, but lack of closed-source API models leaves its effectiveness in the motivating deployment setting insufficiently validated.
Questions for authors
  • How does the proposed visual-dependence signal distinguish meaningful use of visual evidence from susceptibility to irrelevant images or image–question mismatches? Could you provide further justification or experimental evidence that response contradictions under random image replacement reliably improve the assessment of the original answers? A convincing clarification addressing this concern would make me reconsider my score upward.
  • Given that the evaluation relies on public benchmarks, could potential benchmark contamination affect both the reference rankings and cross-model agreement? Is the ranking performance remains still reliable on newly collected or less contaminated data?
Reviewer_iWsuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAVER for ranking LMMs on visual question answering without ground-truth labels or access to model internals. The authors compare answers from different models using an NLI graph and group them into semantic clusters. Each answer is scored using its cluster size and agreement with answers in the same cluster. They also replace the image while keeping the question fixed, then use the response change to adjust cluster support. Examples with more disagreement receive higher weight in the final model score. The evaluation uses 32 models on 15 datasets. MAVER improves average ranking correlations over the baselines, and the visual intervention further improves over the agreement-only version on most of the six additional datasets.

Critical strengths
  1. Ranking models without target labels is a useful problem, particularly when only generated responses are available. MAVER can work in this setting with one additional response per model and example for the visual intervention.

  2. On the six additional datasets, average Spearman correlation increases from 0.69 for Degree to 0.78 for the agreement-only method and 0.82 for MAVER. Weighted Kendall correlation increases from 0.55 to 0.62 and 0.66.

  3. Adding the visual signal improves both correlations on five of these six datasets. It also helps on the analyzed subsets where agreement fails.

Critical weaknesses
  1. All 15 datasets use the same 32 models, including six InternVL checkpoints. Since the models themselves provide the agreement signal, adding more related checkpoints could change the rankings of the others. I would like to know how much the result depends on this particular pool. The current evaluation does not test smaller pools or different proportions of model families, which would be common when choosing models in practice.
  2. UPME [1] also uses visual peer evaluation, but is only discussed and not compared. Its original setup generates questions, so a direct comparison would need adaptation. The peer-scoring component could still be evaluated on the same target questions with a comparable budget. Without this comparison, I am not sure how much MAVER gains over asking the models to assess each other's answers with access to the image.
  3. Section 4.1 allows neutral NLI pairs in the clustering. Two contradictory answers can then end up in the same cluster through a third answer compatible with both. Since cluster size and entropy are used in the score, I would like to know how often this happens and whether a stricter grouping rule changes the rankings.
  4. Replacing the image at random may make the original question invalid, for example if the object in the question is no longer present. The response change can then reflect how the model handles this mismatch. The authors correctly note that sensitivity is not correctness. The ranking improvement shows that the signal helps, but does not tell us whether it separates visually grounded answers from shared language priors. Replacements that remain compatible with the question would be a cleaner check.

[1] UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation, CVPR 2025.

Questions for authors
  1. How much do the rankings change when related checkpoints are added or removed? Does the method still work with a much smaller pool?

  2. How does MAVER compare with UPME peer scoring adapted to the same questions and budget?

  3. How often do clusters contain contradictory answers? Does a stricter clustering rule change the result?

  4. Does the visual signal still improve ranking when the replacement image remains compatible with the question?

Reviewer_sb5WOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

MAVER ranks output-only LMMs on unlabeled VQA data by combining NLI-based peer support, response changes after random image replacement, and disagreement-weighted aggregation. The full method uses two responses per model–example: the original answer and one image-intervened answer. The contribution is this particular visually reweighted estimator and its evaluation, rather than reference-free peer ranking in general. Experiments cover 15 datasets using a common pool of 32 models.

Critical strengths
  • The comparisons help explain where the gains come from. Degree, MAVER*, and MAVER separate basic agreement from clustering/aggregation and the additional visual signal (Tables 1–2). The agreement-failure analysis and free-form versus multiple-choice comparison provide useful evidence beyond aggregate correlations (Tables E-1 and E-3). The appendix also reports module ablations, intervention-seed variation, and sensitivity to the visual weight.

  • The method addresses a practical black-box setting. It needs neither target-task labels nor candidate-model probabilities. The additional generation cost is limited to one intervention response, and Appendix F includes measured runtime with NLI scoring rather than relying only on query counts. These measurements support an efficiency advantage in the tested settings, while the output-only design makes the approach applicable to a different access regime from logit-based estimators.

Critical weaknesses
  • Stability under changes to the peer pool is not established. All 15 datasets use the same, unevenly represented 32-model pool (Table C-2), while both peer support and the final aggregation depend on that population (Eqs. 1 and 4–7). Proposition 1 establishes sufficient conditions for expected Degree ordering, not for the complete MAVER estimator. Consequently, the results support useful ranking within the evaluated pool but do not yet establish that a fixed pair retains its ordering when correlated peers are added or removed. The paper discusses correlated errors, and its task coverage and ablations remain valuable; unequal family counts alone do not demonstrate dependence or an actual ranking failure. The unresolved issue is the scope of the model-selection claim.
  • The cluster construction permits a broader relation than semantic equivalence. Appendix C.4 accepts bidirectional non-contradiction, including neutral relations, and updates labels by overwriting a later response's label. Under the stated rule, compatible A–B and B–C pairs can place contradictory A–C responses together. This matters because cluster mass represents semantic support and cluster entropy weights examples. However, the frequency and ranking impact are unreported, and within-cluster degree can downweight weakly supported answers. Approximate NLI order dependence is also acknowledged in prior semantic-uncertainty work. The concern is therefore whether this implementation's clusters justify their interpretation on the observed outputs, not that nontransitivity by itself invalidates the method.
Questions for authors
  • Using existing responses and NLI scores, does a fixed target pair or subset retain its ordering after one family-balanced or family-removal reaggregation?
  • Could the authors confirm the C.4 overwrite implementation and question/options supplied to NLI, then report the prevalence of internally contradictory clusters and whether they materially affect rankings? If their effect is negligible, clarifying that these are compatibility groups may resolve much of the interpretation concern.

Comments on the paper

2 comments
Reviewer_sb5WOpenReview ↗

Response to the authors

Thank you for the additional analyses.

Q1. The family-removal and family-balancing results (mean flip rates of 2.94%, 1.76%, and 4.87%) address my main concern within the tested pools. MathVista and MMMU-Pro remain sensitive (12.05% and 11.28% flips under family balancing), and Proposition 1 covers Degree, not the full MAVER estimator.

Q2. Thank you for confirming the C.4 implementation and reporting the contradiction rates (12.31% of all groups; 38.72% of groups with two or more distinct answers). The multicut effect depends on the dataset: on WildVision ρ\rho drops from 0.81 to 0.30, while MathVista and MMMU improve. Please describe this effect per dataset, add a mean row, and use "compatibility groups" in Sec. 4.1 and Eq. 7. Please also report per-dataset flip rates and state the scope of Proposition 1.

I have raised my soundness and informativeness scores. I keep the overall rating at 2 because the model-selection claim is supported within the evaluated pool but not yet beyond it: rankings shift on some datasets when the pool changes, and after adding the two API models, their pairwise agreement with the reference ranking is 32.81% on MMMU and 29.69% on MMMU-Pro.

Thanks for the detailed responses and additional experiments. The family balancing and smaller pool results address much of my concern about relying on a fixed candidate pool. The UPME comparison is helpful, and the compatible image replacement experiment gives better support for the visual signal. I also appreciate the additional analyses in the responses to the other reviewers.

I have two remaining questions.

  1. On MathVista and MMMU-Pro, do the remaining ranking flips mostly involve models with similar true performance, or do they change which model would be selected? This would help show how much the sensitivity to pool composition matters in practice.
  2. On WildVision, Spearman correlation drops from 0.81 to 0.30 with multicut grouping. Could you provide a few examples supporting the explanation that stricter grouping splits informative responses and favors generic ones? The explanation sounds reasonable, but examples would make it easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.