acceptodds
Under review as a conference paper at ICLR 2027

RewardLens: Measuring the Behavioral Resolution of Static Multimodal Judge Scores

Abstract

Static preference accuracy is a common way to compare multimodal judges. A tie shows that two judges were equally accurate on the evaluated examples, but says little about how they respond when decision-relevant visual evidence changes. We call the extent to which a static score constrains such intervention behavior its behavioral resolution. RewardLens measures this property with matched visual triplets that hold the question and candidate responses fixed: relevant edits reverse the programmed gold preference, while matched irrelevant edits preserve it. We quantify the two responses with Relevant Adaptation (RA) and Irrelevant Invari- ance (II). On an independent same-domain CLEVR static evaluation, the same five judges score 100% on both Attribute and Spatial, yet their RA spans are 0 and 43.5 percentage points. In Count, Qwen and Skywork score 100.0% and 99.5% statically but differ by 45.73 points in RA on shared correct bases while remaining nearly perfectly invariant to irrelevant edits. Across all 42 frozen one-point same- domain matches, 16 have zero RA gap and 16 have gaps of at least 10 points. Static accuracy summarizes correctness on fixed examples. When responses to changing evidence matter, that summary should be paired with controlled inter- vention measurements.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal judge evaluation, multimodal reward models, LLM-as-a-judge, preference evaluation, behavioral evaluation, intervention-based evaluation, vision-language models, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WhZ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether multimodal judges with similar static preference accuracy respond similarly to changes in visual evidence. RewardLens constructs image triplets with relevant edits that reverse the correct preference and irrelevant edits that preserve it, while keeping the question and candidate responses fixed. Experiments across eight judges and four visual factors show that similar static scores can conceal substantial differences in intervention performance.

Critical strengths
  • The Count comparison provides concrete evidence that static scores can miss substantial behavioral differences. Qwen and Skywork have nearly identical static accuracy, yet their RA differs by 45.73 percentage points on shared base-correct examples. The gap persists under the available candidate-order and human-agreement checks, and this provides the strongest empirical support for the paper’s central claim.
  • The paired relevant/irrelevant edit design makes the observed failures more interpretable. Keeping the question and candidate responses fixed while testing both adaptation and invariance helps distinguish failures to respond to relevant evidence from instability under unrelated changes. This gives the audit practical value as a diagnostic complement to aggregate accuracy.
Critical weaknesses
  • The Spatial human validation remains inconclusive. In Table 17, annotator A2 agrees with all three programmed labels on 35/35 triplets when the base gold is A, but only 2/45 when it is B. Since all sampled Spatial base answers are semantically "yes," this pattern raises concerns about confusion between candidate labels and answer content. The checks in Appendix B.3 do not establish whether annotators understood the response format. This limits confidence in the perceptual validity of a headline result in the abstract and Figure 1. A targeted recheck using direct answer-content selection or swapped candidate orders is needed to resolve this concern.
  • The contribution beyond existing perturbation evaluations is limited. The paper identifies independent static-score matching as its main distinction from existing perturbation evaluations (Section 2). However, 12 of the 32 same-domain static scores are 100%, and the main Attribute–Spatial comparison involves five judges at this ceiling. For judges with perfect audit-based accuracy, RA also reduces to ordinary accuracy on the relevant-image set. The reported gaps may therefore reflect limited coverage of the static set, leaving unclear what score matching adds beyond jointly reporting static accuracy and perturbation performance. An independent static control matched to the relevant scenes in answer distribution and complexity would help establish whether the same judges remain score-matched. Without this evidence, I find the additional contribution too limited to support acceptance.
Questions for authors
  • Qwen and InternVL both answer all 200 Count bases correctly, with RA values of 17% and 18%, respectively (Tables 3 and 5). Can the authors report how many relevant-edit examples are answered correctly by both judges and how many by only one?
  • Appendix C.1 provides the prompt for the non-reward judges but only briefly describes Skywork’s reward-head adapter. Can the authors specify its exact input format and preference decision rule, including whether each candidate is scored independently or with both candidates included in the input?
Reviewer_eQukOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RewardLens, which evaluates whether multimodal judges with similar static accuracy respond similarly to controlled visual changes. It measures adaptation to preference-changing edits and invariance to preference-preserving edits across eight judges and four factors.

Critical strengths
  1. The question focused by this paper is clear and useful, where static accuracy can leave practically relevant differences unresolved.
  2. The design of the intervention is thoughtful, because relevant and irrelevant branches help distinguish selective responses from general instability.
  3. The static matching is independent, where the intervention outcomes are not used to form the score-matched groups.
Critical weaknesses
  1. Figure 1 would benefit from a substantial redesign to improve readability.
  2. Section 2 identifies independent static-score matching as the main contribution, but equal accuracy can hide different error patterns. The authors should have compared directly with VisualFLIP and MM-JudgeBias, and explain what score matching reveals beyond standard intervention evaluation.
  3. The wording "behaviorally tight" seems to be very vague in terms of Equal RA.
  4. Count uses additions, Presence uses removals, and all sampled Presence and Spatial base answers are “yes.” Would answering preferences may therefore affect RA?
  5. Appendices C.4–C.5 describe frozen plans and records without providing enough details.
Questions for authors

Please refer to the weakness section.

Reviewer_kbhKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RewardLens, a framework for examining whether multimodal judges with similar static preference accuracy respond similarly to changes in visual evidence. It constructs CLEVR triplets consisting of a base image, a relevant edit that reverses the preferred answer, and an irrelevant edit that preserves it. Relevant Adaptation and Irrelevant Invariance are measured conditional on correct base judgments. Experiments evaluate eight open-source models.

Critical strengths
  • The method is clearly described, and the paired relevant/irrelevant intervention design is straightforward to follow.
Critical weaknesses
  • The experimental setting has limited practical relevance to multimodal judging. The experiments focus on counting, object presence, attributes, and spatial relations, which admit short, objectively verifiable answers and can be evaluated directly against available references. Model-based judges are particularly valuable for open-ended visual question answering and nuanced evaluation of detailed responses, where simple answer matching is insufficient. The absence of these settings makes it unclear how the findings inform practical applications of multimodal judges.

  • Evaluation scope is limited. All evaluated models are open-source models with at most 8B parameters; frontier models (GPT, Gemini) and finetuned judge models [1,2,3,4] are not well covered.

    [1] Wang, Yibin, et al. "Unified multimodal chain-of-thought reward model through reinforcement fine-tuning." Advances in neural information processing systems, 2025

    [2] Xiong, Tianyi, et al. "LLaVA-critic: Learning to evaluate multimodal models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

    [3] Wang, Xiyao, et al. "Llava-critic-r1: Your critic model is secretly a strong policy model." arXiv preprint arXiv:2509.00676 (2025).

    [4] Zhang, Yifan, et al. "R1-reward: Training multimodal reward model through stable reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.

  • The analysis provides limited insight beyond model-specific performance differences. The experiments demonstrate that similar static scores can coexist with different intervention scores, but the broader empirical takeaway remains unclear. The findings provide limited guidance for evaluating and improving multimodal judges.

  • The writing lacks clarity and consistency. For example, "frozen CLEVR static protocol v2.1" introduces a versioned protocol without clarifying the significance of the version number.

Questions for authors

Please refer to the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.