- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether visual evidence in vision-language models contributes to the answer candidate it actually supports, introducing the concept of candidate-bound visual contribution. The authors construct CROSS-Bench, a benchmark comprising 28,000 decision problems with matched evidence invalidation and rebinding tests, to evaluate evidence utility, specificity, and decision ownership. They further propose RIVET, a structured evidence-to-decision interface that preserves evidence identity, semantic bindings, and uncertainty through candidate-conditioned composition and scalar response control. Experiments demonstrate that prediction accuracy and evidence ownership can diverge. Compared with a capacity-matched baseline, RIVET improves normalized evidence-effect transfer from 0.512 to 0.651, while achieving an average accuracy improvement of 5.70 percentage points over frozen base models on CROSS-Bench. Additional evaluations on external VQA benchmarks demonstrate improvements in task accuracy.
Critical strengths
- The paper introduces a meaningful and underexplored perspective on visual evidence utilization. By distinguishing evidence utility from decision ownership, the work identifies a limitation of conventional accuracy-based evaluations: a model may produce correct predictions without assigning evidence contributions to the candidates they actually support. The proposed invalidation and rebinding criteria provide a useful framework for examining this issue.
- CROSS-Bench provides a structured evaluation framework specifically designed for evidence ownership. The benchmark covers eight visual evidence families and constructs matched clean, invalid, and valid-rebinding conditions. Its explicit source–entity–role–state relations enable more targeted evaluation of evidence contributions than conventional VQA benchmarks.
- The proposed architecture is well structured and supported by informative experimental controls. RIVET explicitly preserves evidence identity, semantic bindings, candidate alignment, and uncertainty, while separating evidence composition from response-strength control. Capacity-matched comparisons, component ablations, cross-backbone experiments, and evaluations on independent benchmarks provide useful evidence supporting the proposed design.
Critical weaknesses
- The benchmark's applicability is constrained by its predefined evidence structure and evaluation scope. Although CROSS-Bench contains 28,000 problems, the primary evidence-ownership evaluation relies on a subset of 960 examples, of which only 707 support valid rebinding. The benchmark requires explicit, annotatable source–entity–role–state relations and a fixed set of 2–5 answer candidates.
- The paper demonstrates a dissociation between accuracy and ownership but provides limited insight into its underlying causes. The experiments show that highly accurate fusion models may exhibit poor evidence ownership. However, the analysis does not fully explain why such predictions remain correct, how much they depend on base-model priors versus misallocated evidence, or under which conditions incorrect ownership causes actual prediction failures. This limits the mechanistic understanding of the central observation.
- The practical benefits of improving evidence ownership are not sufficiently demonstrated. While RIVET achieves higher transfer scores, its matrix accuracy is slightly lower than Capacity-matched Fusion (83.33% versus 84.27%) and Unconstrained Fusion (84.58%). The paper does not sufficiently establish whether improved ownership translates into better robustness, reliability, or reduced reasoning errors in realistic downstream applications.
Questions for authors
- Mechanistic explanation of the accuracy–ownership dissociation. The paper shows that some models achieve high answer accuracy despite poor evidence ownership. However, what mechanisms actually enable these correct predictions? Can the authors distinguish the contributions of base-model priors, alternative visual cues, and incorrectly assigned evidence to explain when and why accuracy becomes disconnected from evidence ownership?
- Practical significance of improving evidence ownership. Beyond improving the proposed ownership metrics, what concrete benefits does correct evidence ownership provide in real-world VQA applications? Could the authors demonstrate that higher ownership leads to fewer evidence-grounding errors, stronger robustness under misleading visual information, or more reliable reasoning on distribution-shifted examples?
- Computational efficiency and inference overhead. RIVET introduces several additional stages, including evidence acquisition, binding, specialized visual reading, and ensemble-based composition. What are the additional end-to-end inference latency, GPU memory consumption, and computational costs compared with the frozen Base and capacity-matched fusion baselines?
- Necessity of the scalar Bound controller. The ablation results show that removing Bound increases evidence transfer from 0.651 to 0.676 and reduces original-owner leakage from 0.205 to 0.194, although accuracy decreases slightly. Could the authors further clarify why the learned scalar controller is necessary and whether its accuracy improvements justify the degradation in the primary ownership metrics?
Thank the authors for their detailed response. I appreciate the time and effort they have devoted to addressing my concerns.
I agree with this intuition. However, similar conclusions can be found in previous works [1] [2], and I think the intuition itself can't be hold as a contribution. Also, the role of RIVET is still unclear. Directly leveraging RIVET as the teacher model is basically amortizing this pipeline into a neural network. However, I still don't think RIVET is general and robust enough as a caption model.
I think 0.643 and 0.548 can't show a strong correlation with only 8 data points. With these two numbers I'm not quite convinced that CROSS-Bench could play an important role for VLM evaluation.
Another suggestion is about the presentation. The current Figure 3 is quite complicated and hard to understand. A good paper should be easy to read and understand. Current figures are quite fancy, but really hard for an audience to read. Also, some terminologies require more illustration and examples (say utility, specificity and ownership) otherwise it would only increase the cognitive load of the reader.
Overall, I appreciate the efforts from the authors, but I will maintain my rating as 2.
[1] Yuxiao Qu et al. “POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration.” in TMLR 2026
[2] Zhaoyang Li et al. "ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models" in CVPR 2026