acceptodds
Under review as a conference paper at ICLR 2027

Beyond Pixels: What Blind Replacement Measures in Visual Agents

Abstract

Visual agents gather evidence through crops and other intermediate observations. A common process score compares a judge's belief under a real image and a blind reference, then interprets the contrast as visual credit. We show that this interpretation mixes answer parameterization with the structure of a crop turn. On V*-Bench, annotated crops improve answer accuracy over matched distractors by 8.9–10.5 points, yet the tagged-sequence score places them below black replacements. Across 32 serialization, path, and judge settings, structural increments exceed pixel increments in 27 cases; their median magnitudes are 4.72 and 0.31 nats. We separate two choices hidden by the original score: the belief target and the credited part of the turn. Under a shared GRPO setup, option-posterior turn credit reaches 52.5% and 48.0% exact match on V*-Bench and HR-Bench-4K. Matching total reward mass retains nearly all of this performance, whereas uniform or shuffled placement does not. Pixel-level credit yields the strongest target retrieval, and its advantage remains after controlling for crop area. Blind replacement is therefore an operator-specific comparison, not a self-defining measure of visual credit.

Primary area:
interpretability and explainable AI
Keywords:
Process Supervision, Vision-Language Agents, RLHF, Information Gain

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_teNrOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper highlights an issue in judge rewards for visual agents. The utility of the correct crop compared to an equal sized but incorrect crop is clear, but existing judges can score correct crops below solid black images. This would punish the model for finding the correct crop. To solve this, many types of reward types are explored both in evaluation and reinforcement. The paper shows that blind scores aren't necessarily neutral and that to improve the reward system, the correct term must be rewarded and the overall reward structure needs to me changed.

Critical strengths
  • Very interesting problem and the example issue draws attention well. The paper shows an example where the reward completely contradicts the desired results. This is clearly an important topic with room for improvement.
  • The paper is written fairly well with slight room for improved clarity. Figure and table captions are very informative.
  • Figures clearly demonstrate the issues and modifications suggested by the paper and are easy to follow
  • The experiments and results provide valuable insights for the problem.
Critical weaknesses
  • It is difficult to see in the tables what the previous baseline/SOTA is for comparison with the suggested alterations in reward scheme
  • The arrows in figure 1 are a bit misleading, as the crop turn is followed by the judge evaluation, but then the suggested tuning of the knobs for altering the reward is not part of the process.
Questions for authors
  • Can you please provide a more thorough explaination for figure 3?
1 comment
AuthorsOpenReview ↗

Response to Reviewer teNr

Thank you for the suggestions on baseline identification and figure clarity. We have revised the figure explanations and clarified the comparison groups.

W1. Baselines and reward comparisons.

Which table entries provide the baseline/SOTA reference for evaluating the proposed reward changes?

A: Outcome-only is the RL baseline; VIG-style adapts the prior VIG reward to the shared training setup.

Table 3 compares reward schemes using the same Qwen2.5-VL-3B backbone, training data, and GRPO budget. Its reference points are the initial policy, outcome-only RL, and a prior-work reward adapted to this setup:

MethodRole in the comparisonV* EM (%)HR-4K EM (%)
Initial checkpointPolicy before reward training46.142.0
Outcome-onlyRL baseline without process rewards49.044.5
VIG-styleAdapted VIG entropy reward (Luo et al., 2026)50.746.0
Raw-TurnTagged-answer whole-turn credit48.343.8
Opt-TurnOption-posterior whole-turn credit52.548.0

The trained rows are the initial three-run means reported in the submitted Table 3. Opt-Turn improves over Outcome-only by 3.5/3.5 percentage points on V*-Bench/HR-Bench-4K and over VIG-style by 1.8/2.0 points. The Raw-Turn comparison holds turn-level credit fixed: changing the belief target to the option posterior improves accuracy by 4.2 points on each benchmark.

Table 3 now identifies the baseline roles and distinguishes the belief-target and credit-scope comparisons; Sec. 4.2 specifies the source of VIG-style, and Table 4 identifies the reward-placement controls. These tables measure reward-design gains under a shared training setup; they do not report a benchmark-wide SOTA ranking.

W2. Figure 1: design choices and execution.

Figure 1’s arrows suggest that reward-design choices follow judge evaluation. How do these choices relate to the actual process?

A: In the original Figure 1, the two knobs represented choices in reward design: the belief target determines which answer event is scored, and the credit scope determines which intervention contrast supplies the reward. They are fixed for each reward variant before training.

In the revised Figure 1, we have removed the arrows between the three panels. Panels (a–c) now separately show the crop turn, scoring contrast, and reward-design choices; panel (c) labels the outcomes "After GRPO training." We have clarified in the caption that the panels connect the scoring problem to reward design, rather than depicting knobs adjusted after each judge evaluation.

Q1. Reading Figure 3.

Please explain what Figure 3 measures and how to interpret its two panels.

A: Figure 3 separates whole-turn tagged-belief support from the fixed-slot pixel contrast and shows that high whole-turn scores can coexist with incorrect final answers.

Figure 3 analyzes Qwen2.5-VL-3B/7B on V*-Bench using the judge's log-probability of the complete gold tagged-answer string, measured in nats. Both panels compare the same two scores:

  • Whole-turn gain: B(Ht)−B(Ht−1)B(H_t)-B(H_{t-1}), comparing the history after the crop action and returned image with the history before that turn.
  • Real-minus-black contrast: B(Ht)−B(Htblack)B(H_t)-B(H_t^{\mathrm{black}}), replacing the returned crop with a same-size black image while holding the action text, image slot, and patch budget fixed.

Panel (a) reports question-weighted means over the recorded crop turns: turns are averaged within each question, then questions receive equal weight. Whole-turn gain versus real-minus-black is +8.001 versus +0.193 nats at 3B, and +1.399 versus -0.404 at 7B (Table 8a). At 7B, the negative contrast means that the gold tagged answer has lower log-probability on average with the real crop than with its black replacement. The whole-turn gain includes changes beyond this fixed-slot pixel comparison and therefore cannot be read as pixel-specific credit.

Panel (b) restricts the analysis to crop turns in trajectories whose final answer is incorrect. For each question, we compute the fraction of these crop turns whose score exceeds 0.5 nats, then average equally across questions with at least one eligible turn. The rates are 96.1% versus 15.8% at 3B and 65.8% versus 6.7% at 7B, for whole-turn versus real-minus-black scores. These percentages describe local score increases within unsuccessful trajectories, not answer accuracy. A positive whole-turn score can therefore coexist with an incorrect final answer.

We have redrawn Figure 3 and expanded its caption to state the reference states, aggregation rules, and incorrect-trajectory conditioning. Sec. 4.1 now distinguishes whole-turn gain from the fixed-slot pixel contrast.

Reviewer_cfEeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines a family of process rewards for crop-tool VLM agents. These rewards score an intermediate crop turn by the change in a frozen judge’s teacher-forced log-probability of the gold answer, either across the whole turn or as a contrast between the real returned crop and a blind replacement. The paper distinguishes two reward-design choices: the belief target, such as the likelihood of a tagged answer sequence or the posterior over valid options, and the credit scope, such as the entire action–observation turn, the observation, or the real-versus-replacement contrast. Experiments on V*-Bench and HR-Bench-4K show that tagged-sequence scores are sensitive to action formatting, message structure, replacement content, and image resolution. Annotated crops improve accuracy relative to distractors even when their tagged-sequence scores fall below black replacements. Under a shared GRPO setup, option-posterior turn rewards achieve the highest reported answer accuracy, while pixel-level rewards achieve the strongest reported target retrieval.

Critical strengths
  1. The paper examines an important measurement problem in process-reward design. A reward intended to measure useful visual evidence may also respond to answer formatting, conversational structure, or the presence of an image message. Distinguishing these effects is important since optimizing the score could encourage behavior that looks like visual search without improving evidence acquisition.
  2. The separation of belief target and credit scope provides a useful framework. The distinction between tagged-sequence likelihood, answer-token likelihood, and option-posterior likelihood is conceptually clear. Separately distinguishing whole-turn, observation-level, and pixel-level comparisons helps explain why similar rewards can encourage different behavior.
  3. The diagnostic experiments vary several plausible sources of score sensitivity. The paper examines black, noise, blur, and mean-color replacements; deletion; patch-budget equalization; different action serializations; and alternative orders of adding action and observation information. These experiments are strong to demonstrate the potential source of score sensitivity.
  4. The gold-crop controls help separate crop usefulness from the behavior of a particular score. Comparing annotated crops with target-free crops from the same image provides a meaningful test of whether the retrieved region contains useful evidence. The reported accuracy improvements of around10.5 and 8.9 percentage points for Qwen and SmolVLM2 show that the crops are useful relative to these distractors. Moreover, the additional ranking comparisons help demonstrate that raw tagged-sequence scores do not consistently reflect that usefulness.
Critical weaknesses
  1. Several conclusions depend on differences below or around one percentage point. For example, Opt-Turn exceeds Opt-Observation by 0.6 points on V-Bench and 0.5 points on HR-Bench-4K*. Mass-Matched Opt-Turn exceeds Uniform-Turn by 0.9 points on both benchmarks, while Opt-Pixel exceeds Opt-Observation in target coverage by 0.6 points. Moreover, the paper does not report training-seed variation and confidence intervals.
  2. The reward-mass controls are not very clear to support the temporal-alignment conclusion. Table 4 mainly claim that correct placement of process rewards matters beyond total reward magnitude, but reward mass is not mathematically defined. It is unclear whether this refers to signed reward, absolute reward, squared magnitude, or the total contribution after assigning rewards to policy tokens.
  3. The usefulness comparison and the negative blind-reference comparison use different baselines. The paper demonstrates that gold crops improve accuracy over distractors while receiving negative real-versus-black scores. However, improvement relative to a distractor does not establish improvement relative to a black image. The gold-versus-distractor ranking results provide more direct evidence of score misalignment, especially for SmolVLM2. To strengthen the central argument, it would be better to report absolute accuracy and option-posterior scores under gold, same-image distractor, cross-question distractor, and black conditions, with paired differences and uncertainty.
  4. The “structural” decomposition does not fully separate message structure from replacement-image content. The black-minus-delete contrast adds both an image message and a particular image: a black canvas with a specified patch budget. It therefore measures the combined effect of the message, its visual representation, and its interaction with the preceding context.
Questions for authors
  1. How many independent training seeds were used? Possible to provide seed-wise results and paired confidence intervals for the comparisons in Tables 3–5?
  2. What is the exact definition of “process-reward mass”? At what stage is it matched relative to standardization, clipping, and token-level assignment?
  3. How are Uniform-Turn and Shuffled-Turn constructed when trajectories have different numbers of crops or different turn lengths? What properties of the original reward sequence are preserved?
6 comments
AuthorsOpenReview ↗

Response to Reviewer cfEe (Part 1/4)

Thank you for the detailed comments on statistical reliability, reward controls, and the interpretation of our intervention contrasts.

W1/Q1. Training variability and paired comparisons.

Are the small accuracy and coverage gains reliable? Please report the number of training seeds, seed-wise results and paired intervals for Tables 3–5.

A: The trained-variant results in submitted Tables 3–5 average three independent training runs per variant. We added ten independent seed blocks for all eleven variants (110 additional runs). The M−Uniform advantage raised in W1 remains significant on both benchmarks, as does M−Shuffled. The T−O EM and P−O coverage differences are not significant after correction. We have revised the corresponding rankings in the abstract, introduction, Sec. 4.2 and conclusion to distinguish supported comparisons from mean rankings.

The new Appendix F in the revised PDF reports seed-wise EM for the ten additional runs, means, SDs and all paired comparisons. Tables 3–5 retain the initial three-run means alongside the additional-run summaries. Tests use the ten additional blocks; each policy is evaluated with four rollouts per question on the same benchmark questions.

Differences are in percentage points, with unadjusted two-sided 95% paired t intervals (df = 9). T = Opt-Turn; O = Opt-Observation; P = Opt-Pixel; M = Mass-Matched Opt-Turn.

ComparisonMetricV*HR
T−OEM+0.50 [-0.53, +1.52]+0.19 [-0.72, +1.10]
T−PEM+0.98 [+0.18, +1.78]+1.15 [+0.58, +1.72]†
M−UniformEM+1.03 [+0.47, +1.59]†+1.06 [+0.65, +1.47]†
M−ShuffledEM+1.64 [+0.99, +2.28]†+1.74 [+1.16, +2.32]†
M−TEM-0.09 [-0.79, +0.61]-0.01 [-0.32, +0.30]
P−TCoverage+1.64 [+0.83, +2.44]†—
P−OCoverage+0.10 [-0.71, +0.92]—
P−OArea-std. coverage+0.10 [-0.76, +0.96]—

† Holm-adjusted p < 0.05 across all 32 comparisons in Table F.4; intervals are unadjusted. Comparisons added during rebuttal are marked exploratory in Appendix F.

Under the scalar-mass calibration specified in Appendix B, M exceeds both Uniform and Shuffled on both benchmarks. Its mean EM differences from ordinary T are −0.09/−0.01 points (V*/HR), with the paired intervals shown above. For the belief target, T exceeds Raw-Turn by 4.08/3.99 points; both Raw→Ans and Ans→Opt steps also remain significant after correction.

For credit scope, T exceeds P in HR accuracy after correction. P improves IoU and area-standardized coverage over T, with mean crop areas of 0.1830/0.1833 (P/T). Relative to O, P improves IoU by 0.0158 and pooled precision by 0.0164, both significant after correction; its coverage advantage is not established. The P−O real–black gap difference is +0.31 [−0.40, +1.03] points and is not significant.

The three scopes define different scoring contrasts over the interaction. The revised text distinguishes their supported empirical contrasts from the observed mean rankings. Tables 3–5 now report ten-run mean ± SD alongside the initial three-run means. Scope statements identify the benchmark, metric and supported comparison, including crop–target overlap and the real–black gap.

AuthorsOpenReview ↗

Response to Reviewer cfEe (Part 2/4)

W2/Q2. Definition and matching.

How is process-reward mass defined, and when is it matched relative to standardization, clipping and token assignment?

A: Process-reward mass is the mean per-trajectory sum of absolute scalar process rewards, measured after scaling and before token assignment or advantage computation:

Mv(S;cv)=1N∑i=1N∑t∈Ti(v)∣λcvr~it(v)∣,λ=0.3.M_v(\mathcal S;c_v)=\frac{1}{N}\sum_{i=1}^{N}\sum_{t\in\mathcal T_i^{(v)}}|\lambda c_v\widetilde r_{it}^{(v)}|,\qquad\lambda=0.3.

r~\widetilde r is the reward after prompt-group standardization, clipping to [−3,3][-3,3], and redistribution where applicable. Credited turns are crop-action turns for turn-level variants and the pre-answer turn for Terminal-Opt. NN includes all trajectories; zero-crop trajectories contribute zero crop-reward mass. Absolute values count both reward signs without cancellation.

We calibrate on 4,096 shared trajectories from 512 held-out training prompts, eight rollouts each, collected from the common initial 3B checkpoint before its first GRPO update. V*-Bench and HR-Bench-4K are excluded. Within each prompt group, crop-turn gains are standardized across valid crop turns in its eight rollouts; Terminal-Opt uses the eight pre-answer gains. We set

Av=Mv(Scal;1),cv=ATerminal-Opt/Av.A_v=M_v(\mathcal S_{\mathrm{cal}};1),\qquad c_v=A_{\mathrm{Terminal\text{-}Opt}}/A_v.

Terminal-Opt's coefficient is one. Calibration equates scalar mass on these shared trajectories. The fixed coefficients are 0.7443 (Mass-Matched Opt-Turn), 0.9202 (Uniform-Turn), and 0.7443 (Shuffled-Turn); ordinary Opt-Turn uses cv=1c_v=1. Permutation preserves absolute mass, giving Shuffled and Mass-Matched the same coefficient; Uniform's coefficient accounts for cancellation from averaging.

The order is standardization → clipping → redistribution → scaling by λcv\lambda c_v → mass measurement → token assignment → advantage computation.

Table 4 reports realized mass: within each run, we divide absolute scalar rewards accumulated over all unique training rollouts by their count, normalize by Terminal-Opt in the same seed block, and average these ratios separately over the initial three and additional ten runs.

VariantRelative massV* EM (%)HR EM (%)
Terminal-Opt1.0051.346.8
Mass-Matched Opt-Turn1.0252.347.8
Uniform-Turn1.0151.446.9
Shuffled-Turn0.9950.946.3
Opt-Turn1.1852.548.0

Q3. Construction and sequence properties.

How do Uniform-Turn and Shuffled-Turn handle different crop counts and turn lengths, and which reward-sequence properties do they preserve?

A: Uniform replaces the crop-turn rewards by their trajectory mean; Shuffled permutes the rewards within that same trajectory.

Each variant computes the standardized, clipped Opt-Turn signal ritr_{it} on its own policy rollouts during training. Shared rollouts are used for calibration. For ni>0n_i>0 crop turns in trajectory ii,

\widetilde r_{it}^{\mathrm{shuffled}}=r_{i,\pi_i(t)}.$$ $\pi_i$ is a uniform permutation of $\{1,\ldots,n_i\}$, sampled independently per rollout. Redistribution stays within each trajectory. With one crop, both operations leave the reward unchanged before calibration; zero-crop trajectories receive no crop reward. Before calibration, write $S_i=\sum_t r_{it}$, $A_i=\sum_t|r_{it}|$, and $Q_i=\sum_t r_{it}^2$: | Property | Uniform | Shuffled | |---|---|---| | Signed sum | $S_i$ | $S_i$ | | Absolute sum | $\lvert S_i\rvert\leq A_i$ | $A_i$ | | Squared sum | $S_i^2/n_i\leq Q_i$ | $Q_i$ | | Reward values | Trajectory mean | Original multiset | Uniform removes within-trajectory variation; Shuffled randomizes the preserved reward multiset. After scaling, every valid policy token in destination crop-action turn $t$ receives $\lambda c_v\widetilde r_{it}^{(v)}$. With $L_{it}$ tokens, its absolute assigned total is $L_{it}|\lambda c_v\widetilde r_{it}^{(v)}|$. Token-weighted totals therefore depend on destination turn lengths. Section F.6 of the new Appendix F in the revised PDF reports the mass diagnostics using this scalar definition. Mass-Matched Opt-Turn exceeds both controls on both benchmarks after correction (Table F.4), supporting reward–turn correspondence under this scalar-mass control. Appendix B now specifies the protocol, and Table 4 defines the scalar mass being matched.
AuthorsOpenReview ↗

Response to Reviewer cfEe (Part 3/4)

W3. Crop utility and belief scores under matched baselines.

Does crop usefulness disagree with belief scores under the same baseline? Please compare gold, same-image distractor, cross-question distractor and black conditions with paired uncertainty.

A: Gold exceeds black in accuracy by 10.53/8.95 pp, while tagged belief decreases by 0.492/1.462 nats (Qwen/Smol). All four contrasts remain significant after correction: tagged belief orders these two conditions oppositely to answer accuracy. With BoptB_{\mathrm{opt}}, the gold-minus-black contrast is positive for both models (+0.140/+0.104 nats), also after correction.

We compare gold crops directly with black replacements, alongside both distractor controls, on the same 190 V*-Bench questions. All four conditions retain the overview, question, options, and complete Appendix A oracle sentence followed by the gold box. Gold crops use native image coordinates; replacement images match their dimensions, image slot, and patch budget. The evaluation protocol uses greedy forced answers with a 16-token limit. Qwen and Smol denote Qwen2.5-VL-3B and SmolVLM2-2.2B.

The table reports absolute accuracy and mean belief scores. BoptB_{\mathrm{opt}} is the log posterior of the gold option normalized over valid options; tagged BB scores the complete gold answer string.

ModelConditionAccuracy (%)BoptB_{\mathrm{opt}} (nats)Tagged BB (nats)
QwenGold57.37-1.145-5.873
QwenSame-image distractor46.84-1.286-5.704
QwenCross-question distractor44.21-1.418-5.812
QwenBlack46.84-1.285-5.381
SmolGold48.42-1.275-4.594
SmolSame-image distractor39.47-1.335-3.908
SmolCross-question distractor38.42-1.425-3.919
SmolBlack39.47-1.379-3.132

Paired differences below are gold minus the indicated reference. Intervals are unadjusted 95% intervals from 20,000 paired question-bootstrap resamples. For accuracy, (b,c)(b,c) counts questions answered correctly only under gold and only under the reference, respectively.

ModelReferenceAccuracy (pp)(b,c)(b,c)Tagged BB (nats)BoptB_{\mathrm{opt}} (nats)
QwenBlack+10.53 [+6.32, +15.26]†(21, 1)-0.492 [-0.649, -0.336]†+0.140 [+0.100, +0.182]†
SmolBlack+8.95 [+3.68, +14.21]†(22, 5)-1.462 [-1.618, -1.306]†+0.104 [+0.064, +0.142]†
QwenSame-image+10.53 [+5.26, +15.79]†(24, 4)-0.169 [-0.310, -0.028]+0.141 [+0.082, +0.201]†
SmolSame-image+8.95 [+3.16, +14.74]†(26, 9)-0.686 [-0.846, -0.530]†+0.060 [+0.001, +0.120]
QwenCross-question+13.16 [+7.89, +18.42]†(28, 3)-0.061 [-0.210, +0.088]+0.273 [+0.187, +0.356]†
SmolCross-question+10.00 [+3.68, +16.32]†(30, 11)-0.675 [-0.833, -0.523]†+0.150 [+0.062, +0.237]†

† Holm-adjusted p<0.05p<0.05 across all 18 comparisons (two models × three references × three metrics). Accuracy uses exact McNemar tests; belief scores use paired t tests. Intervals are unadjusted and the significance markers use adjusted tests.

Same-image distractors and black replacements have equal correct totals but different answer sets. Their answer agreement is 154/190 (81.05%) for Qwen and 157/190 (82.63%) for Smol; their correctness discordances are (11,11) and (13,13), respectively.

The Smol gold-minus-same-image BoptB_{\mathrm{opt}} difference is not significant after correction. The comparison therefore depends on both the belief target and the reference condition.

The reported scores use the complete oracle action specified above; Table 7 reports contrasts across eight action serializations. Here, the gold-minus-same-image BoptB_{\mathrm{opt}} gain is +0.141/+0.060 nats (Qwen/Smol). Sec. 4.1 and Table 1 now include the direct gold–black results and paired uncertainty. The new Appendix H in the revised PDF reports all four conditions and paired comparisons.

Reviewer_5ovPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the validity of blind-state contrast rewards, comparing a model’s belief under a real returned crop against a "blind" replacement such as a black image, which are widely used as process rewards for crop-tool vision-language agents. The central claim is that these contrasts are not self-defining measures of visual credit: they conflate structural changes (action text, message-slot existence, vision-token budget) with the actual contribution of pixels. The authors decompose the reward into two explicit design axes: the belief target (tagged sequence vs. answer token vs. option posterior) and the credited intervention scope (whole turn vs. returned observation vs. pixels). Through extensive diagnostics on V*-Bench and HR-Bench-4K with Qwen2.5-VL and SmolVLM2, they show that structural increments dominate pixel increments in 84.4% of settings (median 4.72 nats vs. 0.31 nats). End-to-end GRPO training confirms the framework’s practical relevance: option-posterior turn-level credit (Opt-Turn) yields the best final-answer accuracy, while pixel-level credit (Opt-Pixel) yields the highest target coverage without enlarging crops.

Critical strengths

The paper does not merely assert that blind-state contrasts are noisy - it provides a controlled, multi-axis decomposition that separates action text, message structure, patch budget, and pixels. The gold-region message ladder (Figure 4a) and the ordered increment analysis (Figure 4b) are particularly compelling: adding a permuted <box> tag contributes +4.068 nats, while the true crop contributes essentially zero. This is a consequential diagnostic because it reveals that a reward commonly interpreted as “visual credit” is predominantly driven by discourse structure. The result is not a minor measurement artifact—it changes what the reward actually incentivizes.

The next one strong point is that the paper moves beyond critique to offer a prescriptive design space. The finding that Opt-Turn (option-posterior turn credit) achieves the highest EM (52.5% / 48.0%) while Opt-Pixel achieves the highest target coverage (46.2%) and real–black gap (17.2 points) is a non-trivial trade-off that practitioners can act on. The reward-matched training control (Table 4) further strengthens the causal claim: merely matching reward mass does not recover the benefit of turn alignment, so the improvement is genuinely attributable to the placement of credit rather than to a reward scale. This level of empirical rigor is rare in papers that critique reward design.

Critical weaknesses

The headline structural-dominance result is partly a consequence of the deployed belief target, and the paper does not fully confront this tension. The main diagnostics use a tagged-sequence belief B that mixes answer content with five surrounding tag tokens. When the authors switch to an option-posterior belief B_opt, the four operator contrasts compress to a range of 0.016 nats at 3B (Section 4.3). This is a double-edged result. On one hand, it supports the paper’s recommendation to normalize over options. On the other hand, it suggests that the enormous structural increments reported in the main text are partly an artifact of a belief target that the paper itself argues is suboptimal. A skeptical reader could conclude that the paper’s strongest evidence against blind-state contrasts is really evidence against tagged-sequence beliefs, not against blind replacement per se. The paper acknowledges this in Section 4.3 but does not sufficiently explore whether a well-chosen belief target (e.g., B_opt) might make blind-state contrasts more meaningful. Addressing this would require reporting the structural-vs-pixel decomposition under B_opt as a co-equal analysis, not a secondary robustness check.

The entire diagnostic and training study is restricted to multiple-choice crop-tool agents with a gold-conditioned judge. The authors explicitly state this scope limitation, but the framework’s practical value for open-ended VLM agents, where the answer event is not a single letter token, remains untested. Similarly, the training experiments use a single 3B checkpoint and the DeepEyes/VeRL GRPO setup. While the authors show consistent patterns across V*-Bench and HR-Bench-4K, the absence of a larger model (e.g., 7B training) or a different RL algorithm makes it difficult to assess whether the Opt-Turn vs. Opt-Pixel trade-off is a general phenomenon or an artifact of the specific optimization landscape. Given that the paper’s core recommendation is about reward design, this is a material limitation.

Questions for authors
  1. How do the structural and pixel increments compare under B_opt on the gold-region ladder and the first-crop decomposition? The paper reports that B_opt compresses operator contrasts to a 0.016-nat range, but does not show whether the relative contribution of action tags versus pixels changes. If the structural dominance persists under B_opt, the paper’s claim is much stronger. If not, the recommendation may need to be reframed as "use B_opt" rather than "blind-state contrasts are fundamentally flawed".

  2. Why was the end-to-end training only performed at 3B? The diagnostic rollouts include 7B, but the training study does not. Does the Opt-Turn advantage persist at 7B, or does the larger model’s stronger visual grounding change the trade-off? This is directly relevant to the paper’s practical guidance.

  3. How does the framework extend to open-ended agents where the answer event is not a single option token? The paper states that open-ended agents "require a different answer event", but does not specify what that event should be. A concrete proposal would significantly increase the paper’s impact.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5ovP (Part 1/3)

Thank you for the thoughtful comments on the belief target, credit scope, and generalizability of our findings.

W1/Q1. Belief Target and Structural–Pixel Decomposition

Does structural dominance persist under option-posterior belief on the gold-region ladder and first-crop path, and how does this affect the paper’s recommendation?

A: Our claim is that visual credit requires an explicit belief target and intervention scope, not that every blind-reference contrast is invalid.

Under BoptB_{\mathrm{opt}}, the action-text increment remains larger than the pixel contrast on the matched first-crop path; the large positive black-message effect does not persist. Option normalization substantially reduces the tagged-belief increments and improves answer-relevant scoring. Whole-turn and fixed-slot pixel scores measure different interventions. We recomputed the gold-region ladder (191 questions) and 3B first-crop decomposition (635 turns) under BoptB_{\mathrm{opt}}, retaining the same inputs, addition paths, and question weighting as tagged BB (Fig. 4; Sec. 4.3; App. C; Table 8b–c).

The 0.016-nat range measures variation across four pixel-replacement references, not the action–pixel difference (Sec. 4.3; Table 9).

First-crop increment (nats)Tagged BBBoptB_{\mathrm{opt}} (95% CI)
Action text AA6.441+0.0616 [0.0580, 0.0652]
Black-reference image-message RR1.729−0.0094 [−0.0119, −0.0069]
Fixed-slot real−black PP0.246+0.0128 [0.0111, 0.0145]
Whole turn A+R+PA+R+P8.416+0.0650 [0.0605, 0.0696]

We computed these intervals using 20,000 paired question-bootstrap resamples and the 2.5th/97.5th percentiles. For the first-crop analysis, events are averaged within each contributing question before resampling; every contrast uses the same sampled question IDs. Each resample recomputes A−PA-P and the ratio of mean AA to mean PP. The ladder resamples its 191 questions in pairs.

Here, RR is black-minus-delete: image-message insertion with the reference image and contextual interactions. PP is real-minus-black at the same image slot.

The action/pixel ratio of question-weighted means decreases from 26.2 to 4.81 (95% CI [4.19, 5.61]), while the paired A−PA-P difference remains +0.0488 nats (95% CI [0.0448, 0.0529]). Option normalization substantially reduces the action increment and reverses the black-reference message increment from positive to negative. The action-text increment remains larger than the pixel contrast on this matched path.

On the gold-region ladder, under BoptB_{\mathrm{opt}}, the action-tag increment decreases from +4.068 to +0.057 (95% CI [0.0511, 0.0629]), while the real-image observation increment changes from −0.109 to +0.030 (95% CI [0.0259, 0.0341]). The template and coordinate-correction increments are +0.004 and −0.001, respectively. The ladder observation step includes message insertion; first-crop PP instead holds the action and image slot fixed.

Table 7 shows how the message and pixel terms change under BoptB_{\mathrm{opt}}. Across the eight Qwen serializations, real-minus-black is +0.036 to +0.060 nats, while black-minus-delete ranges from −0.023 to +0.020. The large positive black-message contribution under tagged belief therefore does not persist in these comparisons.

BoptB_{\mathrm{opt}} also improves gold-over-distractor ranking from 59.21% to 65.79% for Qwen and 39.21% to 57.37% for SmolVLM2 (Sec. 4.1; Table 1; App. C). These results support using BoptB_{\mathrm{opt}} for more answer-relevant crop scoring. The belief target specifies the answer event; credit scope specifies whether the reward measures whole-turn support or a real-minus-reference pixel contrast. The residual action increment explains why whole-turn and fixed-slot pixel credit remain distinct. We have revised the abstract, introduction and discussion to identify the 27/32 structural-dominance summary and its 4.72/0.31-nat median magnitudes as tagged-belief results. The BoptB_{\mathrm{opt}} conclusion distinguishes the action-text, black-reference message and pixel increments on the tested paths. Table 6 now presents the two belief targets side by side with these uncertainty estimates. The new Appendix H in the revised PDF gives the matched protocols.

AuthorsOpenReview ↗

Response to Reviewer 5ovP (Part 2/3)

W2/Q2. End-to-end validation at 7B.

Why was training limited to 3B, and does the accuracy–retrieval pattern persist at 7B?

A: The 7B runs preserve the mean ordering: Opt-Turn gives higher answer accuracy and Opt-Pixel higher target coverage among the tested methods.

The initial 3B study held the checkpoint and training setup fixed across reward variants. We extended the policy to Qwen2.5-VL-7B-Instruct while keeping Qwen2.5-VL-3B-Instruct as the frozen reward judge. This changes policy capacity while preserving the judge used in the 3B training study.

Training uses the DeepEyes/VeRL GRPO setup in Sec. 4.2 and Appendix B, with eight rollouts per training prompt and an eight-turn cap. All four reward variants use the same training data and update budget. Each run is evaluated at the common final training step on the same 191 V*-Bench and 200 HR-Bench-4K questions, averaging four temperature-1 rollouts per question with equal question weight.

The table reports mean ± sample SD over five independent training runs. EM and coverage are percentages. Coverage uses the first crop and requires at least 50% of the annotated target area to lie inside the first crop, assigning zero to rollouts without a crop.

MethodV* EM (%)HR EM (%)V* coverage (%)
Outcome-only54.1 ± 1.0149.2 ± 1.0944.7 ± 1.14
Raw-Turn53.6 ± 0.6548.4 ± 0.6344.1 ± 1.66
Opt-Turn57.9 ± 0.5652.8 ± 0.9548.4 ± 0.97
Opt-Pixel56.2 ± 1.0151.4 ± 0.6051.7 ± 0.32

Belief target. With turn-level credit fixed, Opt-Turn improves over Raw-Turn by 4.3 percentage points on V*-Bench and 4.4 on HR-Bench-4K.

Credit scope. With BoptB_{\mathrm{opt}} fixed, Opt-Turn has the highest mean answer accuracy among the tested methods, exceeding Opt-Pixel by 1.7 points on V*-Bench and 1.4 on HR-Bench-4K. Opt-Pixel has the highest mean target coverage, exceeding Opt-Turn by 3.3 points, compared with 1.6 points in the submitted 3B Table 3.

Across the five matched seed blocks, the Opt-Turn−Opt-Pixel EM differences are +1.70 [0.72, 2.68] points on V*-Bench and +1.40 [−0.34, 3.14] on HR-Bench-4K (unadjusted two-sided 95% paired t intervals, df = 4, on the fixed question sets).

These mean directions agree across policy sizes. In the additional 3B runs, Opt-Turn exceeds Opt-Pixel in HR accuracy, and Opt-Pixel exceeds Opt-Turn in V* coverage after correction; their V* accuracy difference is not significant. The new Appendix F in the revised PDF reports the 3B means, SDs and complete paired comparisons.

The new Appendix G in the revised PDF reports the 7B results, per-run scores and evaluation settings.

AuthorsOpenReview ↗

Response to Reviewer 5ovP (Part 3/3)

W2/Q3. Open-ended answer events and validation.

How can the framework define an answer event for open-ended agents and extend beyond multiple-choice tasks and GRPO?

A: For short-answer VQA, we replace the single gold-option event with a verified set of complete answers and retain the same intervention definitions. Across the tested GRPO/PPO and policy-size settings, Set-Turn has higher mean answer accuracy and Set-Pixel higher mean target coverage. We score the probability mass of the answer set:

BA(H)=log⁡∑y∈A(q,y∗)PM(y⊕τ∣H,π).B_{\mathcal A}(H)=\log\sum_{y\in\mathcal A(q,y^*)}P_M(y\oplus\tau\mid H,\pi).

A\mathcal A contains the canonical answer and verified aliases. Candidates come from the dataset’s human reference answers and number-word/numeral variants, such as “2” and “two”. Annotators retain candidates that preserve the canonical answer in the question and image context, including the written form when the question asks for transcription. We fix these sets before model comparison and deduplicate token sequences. With the fixed prefix π\pi (Answer: ) and end-of-sequence token τ\tau, we sum the teacher-forced probabilities of the complete answer sequences. The answer set, prefix and ending token stay fixed across interventions.

The score is the log probability of a verified answer set under the model's sequence distribution. BoptB_{\mathrm{opt}} instead conditions on a finite set of valid options. This is another answer event within the same framework, retaining the turn, observation and fixed-slot pixel intervention definitions.

Controlled comparisons. Single-Turn scores the canonical answer alone; Set-Turn uses the verified answer set. Within each model/optimizer setting, they share the policy initialization, frozen judge, training data, rollout and update budgets, reward coefficient, reward-processing rules, and evaluation protocol. The same rules govern standardization, clipping, token assignment and advantage computation. Set-Pixel uses the same answer set as Set-Turn and changes the scoring contrast to fixed-slot real-minus-black. The ranking comparison likewise holds the judge, crop pair, prefix and ending token fixed while changing the answer set.

Ranking diagnostics. We use 1,000 fixed target-localizable validation questions from each of TextVQA and VQA v2. Human annotators mark the question-relevant target regions before model comparison. Same-image distractors exclude the annotated target. The 3B and 7B labels below identify frozen Qwen2.5-VL-Instruct judges of the corresponding size.

Gold-over-distractor ranking improves with verified-set versus single-answer scoring: TextVQA 62.7→67.0% (3B), 65.8→68.7% (7B); VQA v2 64.1→66.2% (3B), 67.9→69.3% (7B).

End-to-end evaluation. We train on the TextVQA training split. Final-answer accuracy uses the official multi-reference VQA scoring on all 5,000 validation questions; first-crop coverage uses the fixed 1,000-question TextVQA subset above. Both average four temperature-1 rollouts per question. Coverage requires the first crop to contain at least half of the annotated target area; no-crop rollouts contribute zero. Validation questions are excluded from training. The policy retains the eight-turn crop-tool interface, and every training setting uses a frozen Qwen2.5-VL-3B-Instruct reward judge.

Entries are mean ± sample SD across five independent training runs, in percentages. “Turn / Pixel” reports Set-Turn and Set-Pixel coverage.

Model / RLSingle-TurnSet-TurnSet-PixelCoverage: Turn / Pixel
3B / GRPO73.0 ± 1.0376.1 ± 0.9474.8 ± 0.8249.3 ± 1.81 / 52.7 ± 0.86
3B / PPO72.9 ± 0.4575.1 ± 0.6774.0 ± 0.4047.9 ± 0.41 / 50.8 ± 1.12
7B / GRPO79.7 ± 0.6482.4 ± 0.9181.2 ± 0.8251.5 ± 0.71 / 54.9 ± 1.15

We have added the answer-event definition, protocol and results to the new Appendix G in the revised PDF.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank the Area Chair and reviewers for their helpful feedback. We have added experiments, clarified the claims, and improved the writing.

Core contribution. A crop can improve answer accuracy yet score lower than a black image. We observe this when the judge scores the full tagged answer. We distinguish the belief target (what the judge scores) from the credit scope (which part of the interaction receives credit). Our analyses and training experiments show how these choices affect score interpretation and learned behavior.

Reviewer 5ovP: belief target and broader settings. We compare tagged answer and option posterior score breakdowns on matched paths. Normalization reduces the score changes. On the tested first crop path, the action effect remains larger than the pixel contrast with the image slot fixed. We clarify the scope of this finding. We also add five independent runs of 7B policy training and short-answer experiments across three model and optimizer settings.

Reviewer cfEe: reliability and controls. We add ten independent runs per 3B variant, paired uncertainty estimates, and tests corrected for multiple comparisons. We state which differences are statistically significant. We define scalar reward mass and its calibration, and explain the Uniform and Shuffled controls. We directly compare accuracy and belief scores against the same black baseline. We also clarify that the black image message term includes the reference image and its interactions with the context.

Reviewer teNr: baselines and figures. We clarify the roles of the baselines and controls. We remove the arrows between panels in Figure 1, explain the scores and averaging in Figure 3, and correct the labels in Figure 4.

Writing and revision details. We simplify Sec. 3.1 to present the agent, judge, and three belief definitions in order. Implementation details are in Appendices A and B. Appendices F, G, and H report the additional statistics, extension experiments, and matched crop analyses, respectively. Revised headings, captions, and key text are marked in blue. Detailed responses remain in each reviewer's thread.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.