acceptodds
Under review as a conference paper at ICLR 2027

SPAR: LEARNING VLM RISK FROM STRUCTURED PEER AGREEMENT

Abstract

Vision–language models can be confidently wrong, and their confidence scales transfer poorly across architectures and tasks. We study fixed-focal selective prediction, where heterogeneous peer outputs help decide whether to accept or abstain on a focal VLM answer without changing its content. Common multi-VLM uncertainty scores compress this evidence into scalar agreement, even though the same mean disagreement can arise when peers agree with one another but oppose the focal answer, or when they disagree among themselves. We introduce SPAR, a Structured Peer-Agreement-based Risk estimator for fixed-focal selective prediction. It decomposes focal disagreement into focal isolation and peer ambiguity, and combines this decomposition with relative peer confidence. Percentile-rank alignment makes continuous features comparable across models and datasets, allowing a single risk scorer to be applied unchanged across different settings. Across five focal VLMs and three multimodal benchmarks, SPAR achieves the best aggregate AURC and consistently outperforms the evaluated peer-based and repeated-sampling alternatives. Under simultaneous model-family and dataset shift, it reduces macro-AURC by 7.05% over the strongest baseline (Consensus Entropy), with gains in every setting, while achieving up to a 10.1× generation-time speedup over the strongest repeated-sampling comparison. These results establish structured output relations as an effective and transferable reliability signal for selective VLM prediction.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Vision-Language Models, Selective Prediction, Uncertainty Quantification, Failure Prediction

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZMC9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fixed-focal selective prediction for vision-language models (VLMs) and proposes SPAR, a risk estimation framework that exploits the relational structure between focal and peer model outputs. By distinguishing focal isolation from peer ambiguity, SPAR provides more informative reliability signals than scalar disagreement measures. Experimental results demonstrate consistent, though relatively modest, improvements over competitive baselines.

Critical strengths
  1. The research motivation is clear and well justified. Within the scope of fixed-focal selective prediction, distinguishing focal–peer disagreement from peer–peer agreement is a reasonable and meaningful approach to reliability estimation.

  2. The experimental evaluation is comprehensive, covering diverse VLM architectures, datasets, transfer settings, and ablation studies, providing substantial empirical support for the proposed method.

Critical weaknesses

The proposed framework relies on multiple peer VLMs and requires training an additional risk-scoring model. Despite its efficiency improvements over repeated-sampling methods, deploying and querying multiple models still introduces considerable computational and deployment overhead, which may limit its practical applicability.

Questions for authors

None beyond the concerns raised above.

Reviewer_DDcPOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for VLM selective prediction, using a principled procedure involving multiple heterogeous peer models to estimate the risk of a single 'focal' model output. The method only needs output logits from each of the constituent models, and computes multiple metrics that correspond to structured answer agreement / certainty features. These features are combined to produce a 'focal risk' score for a model output. The authors systematically test their method against existing models in various benchmarks, as well as evaluating transfer performance and robustness to changes to the intermediate computed statistics. The method consistently outperforms the strongest baseline (Consensus Entropy), across model scales, and usage settings. The authors also systematically quantify the effect of the different architectural choices of the method, showing that leveraging relational structure improves performance.

Critical strengths

The paper presents a principled, scalable, and relatively elegant method for estimating VLM output risk using a set of heterogenous models. The authors offer clear motivation for the design, thoroughly evaluate it against benchmark methods in a variety of tasks, and perform systematic analysis of the operating characteristics of the method. The method consistently outperforms existing methods across model scales and benchmarks.

Critical weaknesses

I would like to point out that the meaning of the term 'focal' is never explicitly explained in the paper, and this caused significant confusion on my first reading. It is not a bad wording to use per se, but associations with concepts like focal length and focal attention mechanisms made it difficult to disambiguate. I do not believe that this is an established usage of the term, since I could not find instances of it with web searches or in the multi-model method papers you cite. Please consider adding a clear definition in the abstract or the introduction. As far as I can tell, the 'focal VLM' is simply the model chosen as the anchor of comparision to the other models in the set.

Questions for authors

Is the use of likelihood-derived risk the best way to quantify per-model uncertainty, or are there other options that might improve upon the accuracy of output certainty?

Reviewer_zGzuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a risk estimation method called SPAR, which utilizes the output results of the heterogeneous peer model to determine whether to accept or reject a specific fixed VLM answer. It combines the answer relationships between the focus-peer and peer-peer, as well as the likelihood-based confidence signals, employs percentile alignment techniques, and uses the source-trained logistic regression scorer to estimate the failure risk. Experiments show that, in the presence of both the focus model family and dataset shift, SPAR outperforms the evaluated baseline methods in terms of risk ranking.

Critical strengths
  1. The method provides a useful way to distinguish coherent peer opposition from disagreement among peers when deciding whether to accept or abstain.
  2. The experimental evaluation is extensive, covering multiple VLM families, model scales, and datasets.
Critical weaknesses
  1. The authors use the token-level F1 as the answer-similarity measure. However, lexical overlap may fail to capture semantically equivalent answers or distinguish similar wording with contradictory meanings, potentially affecting the interpretation of focal isolation and peer ambiguity. A comparison with semantic similarity measures, such as cosine similarity between answer embeddings, would help assess the robustness of the proposed relations.
  2. Although the paper evaluates unseen focal-model families, the primary transfer experiments retain the same peer panel. It therefore remains unclear whether the frozen scorer generalizes to other model combinations, particularly when the peer panel is replaced with previously unseen model families. Evaluating multiple alternative peer panels would strengthen the generalization claims.

Minor weakness: Some essential feature definitions and key experimental results referenced in the main text are deferred to the appendix, requiring frequent cross-referencing to understand and assess the method. Including a concise feature-definition table and the main joint-transfer results in the main text would improve readability

Questions for authors

Please address points 1 and 2 in the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.