- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper highlights an issue in judge rewards for visual agents. The utility of the correct crop compared to an equal sized but incorrect crop is clear, but existing judges can score correct crops below solid black images. This would punish the model for finding the correct crop. To solve this, many types of reward types are explored both in evaluation and reinforcement. The paper shows that blind scores aren't necessarily neutral and that to improve the reward system, the correct term must be rewarded and the overall reward structure needs to me changed.
Critical strengths
- Very interesting problem and the example issue draws attention well. The paper shows an example where the reward completely contradicts the desired results. This is clearly an important topic with room for improvement.
- The paper is written fairly well with slight room for improved clarity. Figure and table captions are very informative.
- Figures clearly demonstrate the issues and modifications suggested by the paper and are easy to follow
- The experiments and results provide valuable insights for the problem.
Critical weaknesses
- It is difficult to see in the tables what the previous baseline/SOTA is for comparison with the suggested alterations in reward scheme
- The arrows in figure 1 are a bit misleading, as the crop turn is followed by the judge evaluation, but then the suggested tuning of the knobs for altering the reward is not part of the process.
Questions for authors
- Can you please provide a more thorough explaination for figure 3?
Response to Reviewer teNr
Thank you for the suggestions on baseline identification and figure clarity. We have revised the figure explanations and clarified the comparison groups.
W1. Baselines and reward comparisons.
A: Outcome-only is the RL baseline; VIG-style adapts the prior VIG reward to the shared training setup.
Table 3 compares reward schemes using the same Qwen2.5-VL-3B backbone, training data, and GRPO budget. Its reference points are the initial policy, outcome-only RL, and a prior-work reward adapted to this setup:
The trained rows are the initial three-run means reported in the submitted Table 3. Opt-Turn improves over Outcome-only by 3.5/3.5 percentage points on V*-Bench/HR-Bench-4K and over VIG-style by 1.8/2.0 points. The Raw-Turn comparison holds turn-level credit fixed: changing the belief target to the option posterior improves accuracy by 4.2 points on each benchmark.
Table 3 now identifies the baseline roles and distinguishes the belief-target and credit-scope comparisons; Sec. 4.2 specifies the source of VIG-style, and Table 4 identifies the reward-placement controls. These tables measure reward-design gains under a shared training setup; they do not report a benchmark-wide SOTA ranking.
W2. Figure 1: design choices and execution.
A: In the original Figure 1, the two knobs represented choices in reward design: the belief target determines which answer event is scored, and the credit scope determines which intervention contrast supplies the reward. They are fixed for each reward variant before training.
In the revised Figure 1, we have removed the arrows between the three panels. Panels (a–c) now separately show the crop turn, scoring contrast, and reward-design choices; panel (c) labels the outcomes "After GRPO training." We have clarified in the caption that the panels connect the scoring problem to reward design, rather than depicting knobs adjusted after each judge evaluation.
Q1. Reading Figure 3.
A: Figure 3 separates whole-turn tagged-belief support from the fixed-slot pixel contrast and shows that high whole-turn scores can coexist with incorrect final answers.
Figure 3 analyzes Qwen2.5-VL-3B/7B on V*-Bench using the judge's log-probability of the complete gold tagged-answer string, measured in nats. Both panels compare the same two scores:
Panel (a) reports question-weighted means over the recorded crop turns: turns are averaged within each question, then questions receive equal weight. Whole-turn gain versus real-minus-black is +8.001 versus +0.193 nats at 3B, and +1.399 versus -0.404 at 7B (Table 8a). At 7B, the negative contrast means that the gold tagged answer has lower log-probability on average with the real crop than with its black replacement. The whole-turn gain includes changes beyond this fixed-slot pixel comparison and therefore cannot be read as pixel-specific credit.
Panel (b) restricts the analysis to crop turns in trajectories whose final answer is incorrect. For each question, we compute the fraction of these crop turns whose score exceeds 0.5 nats, then average equally across questions with at least one eligible turn. The rates are 96.1% versus 15.8% at 3B and 65.8% versus 6.7% at 7B, for whole-turn versus real-minus-black scores. These percentages describe local score increases within unsuccessful trajectories, not answer accuracy. A positive whole-turn score can therefore coexist with an incorrect final answer.
We have redrawn Figure 3 and expanded its caption to state the reference states, aggregation rules, and incorrect-trajectory conditioning. Sec. 4.1 now distinguishes whole-turn gain from the fixed-slot pixel contrast.