- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether multimodal judges with similar static preference accuracy respond similarly to changes in visual evidence. RewardLens constructs image triplets with relevant edits that reverse the correct preference and irrelevant edits that preserve it, while keeping the question and candidate responses fixed. Experiments across eight judges and four visual factors show that similar static scores can conceal substantial differences in intervention performance.
Critical strengths
- The Count comparison provides concrete evidence that static scores can miss substantial behavioral differences. Qwen and Skywork have nearly identical static accuracy, yet their RA differs by 45.73 percentage points on shared base-correct examples. The gap persists under the available candidate-order and human-agreement checks, and this provides the strongest empirical support for the paper’s central claim.
- The paired relevant/irrelevant edit design makes the observed failures more interpretable. Keeping the question and candidate responses fixed while testing both adaptation and invariance helps distinguish failures to respond to relevant evidence from instability under unrelated changes. This gives the audit practical value as a diagnostic complement to aggregate accuracy.
Critical weaknesses
- The Spatial human validation remains inconclusive. In Table 17, annotator A2 agrees with all three programmed labels on 35/35 triplets when the base gold is A, but only 2/45 when it is B. Since all sampled Spatial base answers are semantically "yes," this pattern raises concerns about confusion between candidate labels and answer content. The checks in Appendix B.3 do not establish whether annotators understood the response format. This limits confidence in the perceptual validity of a headline result in the abstract and Figure 1. A targeted recheck using direct answer-content selection or swapped candidate orders is needed to resolve this concern.
- The contribution beyond existing perturbation evaluations is limited. The paper identifies independent static-score matching as its main distinction from existing perturbation evaluations (Section 2). However, 12 of the 32 same-domain static scores are 100%, and the main Attribute–Spatial comparison involves five judges at this ceiling. For judges with perfect audit-based accuracy, RA also reduces to ordinary accuracy on the relevant-image set. The reported gaps may therefore reflect limited coverage of the static set, leaving unclear what score matching adds beyond jointly reporting static accuracy and perturbation performance. An independent static control matched to the relevant scenes in answer distribution and complexity would help establish whether the same judges remain score-matched. Without this evidence, I find the additional contribution too limited to support acceptance.
Questions for authors
- Qwen and InternVL both answer all 200 Count bases correctly, with RA values of 17% and 18%, respectively (Tables 3 and 5). Can the authors report how many relevant-edit examples are answered correctly by both judges and how many by only one?
- Appendix C.1 provides the prompt for the non-reward judges but only briefly describes Skywork’s reward-head adapter. Can the authors specify its exact input format and preference decision rule, including whether each candidate is scored independently or with both candidates included in the input?