- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present VisCA, a self-distilled RL framework that breaks down rewards in vision-language models by assigning credit to specific visual regions. Their approach constructs two types of views: evidence-centered crops (privileged) and degraded counterfactual versions. By checking whether these conditions agree, they verify which tokens actually depend on fine-grained visual evidence rather than just correlating with it. This verified signal adjusts outcome-based advantages from a verifier while keeping the reward-driven optimization direction intact. Testing on Qwen3.5 models (4B and 9B parameters) shows 6-7 point average improvements across six fine-grained visual understanding benchmarks, with performance holding steady on separate multimodal tasks. VisCA outperforms GRPO and several self-distillation baselines when trained on the same synthetic dataset.
Critical strengths
-
The counterfactual verification mechanism tackles a real problem in using privileged information for credit assignment. It's a more careful attempt at separating correlation from causation than what I've seen in prior work on this topic.
-
Results hold up across multiple benchmarks and two model scales, and the gains transfer to held-out tasks. This suggests the method isn't just memorizing the training distribution.
-
The semantic analysis in Section 4.4 offers solid evidence that verified credit actually lands on visually grounded tokens instead of linguistic scaffolding. This validates the method's intended behavior rather than just showing aggregate metrics.
Critical weaknesses
-
Training exclusively on 6.2K synthetic examples from Vision-OPD makes me question whether these gains reflect real capability improvements or just overfitting to a narrow distribution. There's no comparison against models trained on more diverse or larger-scale data.
-
The core assumption, that directional agreement between privileged and counterfactual differences indicates visual attribution, lacks rigorous justification. Why does this particular consistency check reliably filter out confounding factors? The paper doesn't really explain the mechanism.
-
Experimental validation sticks to tasks where fine-grained visual details are explicitly required. What about scenarios where attention and reward relevance might diverge, like tasks requiring the model to deliberately ignore salient distractors? Those cases aren't tested.
Questions for authors
-
How does VisCA perform when the model needs to actively ignore visually salient but task-irrelevant features? Does counterfactual verification still produce reliable credit in those adversarial cases?
-
Can you compare against simpler spatial decomposition schemes, uniform grid-based reward attribution or gradient-based saliency, for instance, to show whether the counterfactual machinery is actually necessary?
-
What fraction of tokens typically pass the counterfactual consistency check? How does this vary across different task types or training stages? Does the sparsity of verified credit shift as the model improves?
-
How sensitive is the method to the choice of privileged region? If the bounding box is slightly misaligned or includes irrelevant background, does the verification mechanism degrade gracefully or collapse?
5 comments
4. Comparisons with Simpler Attribution Methods (Q2)
Thank you for suggesting these comparisons. We are making every effort to complete comparisons with uniform grid-based reward attribution and gradient-based saliency to assess the added value of counterfactual verification. We will update this response as soon as the results become available.
5. Fraction and Evolution of Verified Credit (Q3)
Across training stages, the fraction of tokens passing the counterfactual consistency check starts higher early in training and gradually decreases to 29.3% at the final checkpoint. This shift in sparsity reflects how the model evolves as it improves. Early in training, the full-image policy remains close to the base model and has not yet internalized fine-grained perception, so differences between full-image and privileged views are spread across many tokens with small yet sign-consistent margins. As training progresses and the policy internalizes localized visual understanding, full-image predictions align more closely with the privileged view on routine tokens, concentrating verified visual credit onto a sparser subset of decisive tokens.
6. Sensitivity to the Privileged Region (Q4)
At inference the model receives only the full image, and all benchmarks are box-free. The privileged region is a training-time construct, so a misaligned or missing box at test time has no effect. During training, the region is used only to build the pair and . Since the two views share the same region, comparing them may reduce the influence of mild misalignment or shared irrelevant background when these affect both views similarly. The gate remains a consistency check, and the credit becomes less specific rather than invalid. As shown in the table below, the matched 0.10 degradation is best at 77.73. A random crop gives 76.17, and removing the image gives 76.80. Even the worst setting remains above the Base (72.54) and close to the raw visual-evidence variant (76.27). The mechanism therefore degrades gracefully rather than collapsing, and the reward anchor keeps the update direction unchanged.
| Counterfactual View | V*Bench | ZoomBench | HR-Bench 4K | HR-Bench 8K | MME-RW EN | MME-RW CN | Avg |
|---|---|---|---|---|---|---|---|
| No Image | 88.48 | 60.00 | 86.00 | 82.62 | 72.51 | 71.18 | 76.80 |
| Random Crop | 88.48 | 58.82 | 85.12 | 81.12 | 72.55 | 70.91 | 76.17 |
| Mask | 89.01 | 57.28 | 87.25 | 83.75 | 72.60 | 72.25 | 77.02 |
| Blur_0.05 | 91.10 | 59.17 | 85.88 | 83.88 | 72.93 | 71.57 | 77.42 |
| Blur_0.10 | 92.15 | 60.47 | 87.25 | 82.62 | 72.69 | 71.18 | 77.73 |
| Blur_0.25 | 88.48 | 60.83 | 84.62 | 83.25 | 71.38 | 70.81 | 76.56 |
Thank you once again for your thoughtful review and helpful feedback. We hope the explanations above resolve your concerns and clarify the issues you highlighted. We will revise the manuscript to include these clarifications and the additional analyses. We would be glad to provide further details should you have any remaining questions.
[1] Yuan, Q., et al. (2026). Vision-OPD: Learning to See Fine-Grained Details for Multimodal LLMs via On-Policy Self-Distillation. NeurIPS.
4. Comparisons with Simpler Attribution Methods (Q2)
Thank you for suggesting these comparisons. We have now completed experiments with uniform grid attribution and gradient-based saliency to assess the added value of VisCA’s counterfactual verification.
Uniform grid attribution. We partition the full image into a fixed (2×2) grid. For each generated token, we compute the difference between the teacher’s log probability under each grid crop and that under the full image, keeping the question and response prefix fixed. We then average these differences over the four crops to obtain token-level visual credit.
Gradient-based saliency. For each generated token, we compute the gradient of the teacher’s log probability with respect to the visual embeddings under the full-image input. We then average the gradient norms across visual positions to obtain a token-level sensitivity score.
As shown below, VisCA improves the average score by 3.12 and 2.08 points over uniform grid attribution and gradient-based saliency, respectively, highlighting the advantage of counterfactual verification for token-level visual credit assignment.
| Method | V*Bench | ZoomBench | HR-Bench 4K | HR-Bench 8K | MME-RW EN | MME-RW CN | Avg |
|---|---|---|---|---|---|---|---|
| Uniform grid attribution | 87.96 | 57.99 | 82.00 | 80.25 | 70.69 | 68.78 | 74.61 |
| Gradient-based saliency | 89.01 | 58.70 | 83.25 | 81.38 | 71.58 | 69.97 | 75.65 |
| VisCA (Ours) | 92.15 | 60.47 | 87.25 | 82.62 | 72.69 | 71.18 | 77.73 |
Thank you for your review and constructive feedback. We address generalization from the 6.2K training set, explain the rationale for counterfactual verification, and describe comparisons with simpler attribution methods. We examine verified-credit sparsity across training stages, robustness to task-irrelevant visual features, and sensitivity to the privileged region.
1. Training-Data Rationale and Generalization (W1)
Training-data rationale. We use the 6.2K examples released by Vision-OPD [1], which are constructed around questions answerable from localized visual evidence. This design addresses a common difficulty in fine-grained perception, where a model may recognize a detail in an isolated crop but fail to use it effectively in the full image. Pairing full images with privileged regional views aims to improve the model’s use of local evidence, rather than to tailor training examples to the six evaluation benchmarks. VisCA builds on these paired views to derive token-level visual credit, aiming to translate the model’s existing local recognition ability into more effective use of task-relevant evidence during full-image reasoning.
Evidence of generalization. To assess whether the gains extend beyond fine-grained visual tasks, we evaluate on three held-out visual tasks, MMVP, CV-Bench, and MMStar, which were not used for VisCA post-training or hyperparameter selection. VisCA achieves higher average scores on the held-out tasks than the base model at both scales, with 80.69 versus 78.99 at 4B and 82.98 versus 82.44 at 9B, and outperforms GRPO, OPSD, RLSD, and Vision-OPD on the same held-out average. All methods are trained on the same 6.2K samples using the same training recipe. If these gains were driven by overfitting to this distribution, the baselines trained on the same data would also show similar improvements, but they do not.
2. Mechanistic Justification of the Consistency Check (W2)
Decomposition of the visual contrasts. For a sampled token yt, we hold the teacher, question, and response prefix fixed and denote its log-probability under view v by ℓt(v). We compute
e_t^{\mathrm{vis}} = \ell_t(x^+) - \ell_t(x), \qquad e_t^{\mathrm{ctr}} = \ell_t(x^+) - \ell_t(x^-). \end{xalignat*}$$ These satisfy the exact identity $$\begin{xalignat*}{1} e_t^{\mathrm{vis}} = e_t^{\mathrm{ctr}} + c_t, \qquad c_t = \ell_t(x^-) - \ell_t(x). \end{xalignat*}$$ Here, $e_t^{\mathrm{ctr}}$ measures sensitivity to detail suppression within the same crop, while $c_t$ captures the remaining difference between the degraded crop and the full image, including region and context changes.Why directional agreement. The gate retains a signal only when etvisetctr>0. If the privileged crop increases a token’s likelihood, retaining positive credit requires that suppressing its fine details decreases that likelihood. Conversely, negative credit is retained when the details suppress the token. In the decomposition above, when ct opposes and outweighs etctr, the privileged–full difference reverses direction and is rejected. The check therefore filters signals whose direction conflicts with the matched detail contrast. Using signs tests directional consistency without requiring the two contrasts to have equal magnitudes.
3. Robustness to Task-Irrelevant Visual Distractors (W3 & Q1)
To evaluate VisCA’s robustness to visually salient but task-irrelevant features, we conduct a distractor test on ZoomBench. Its full images and question-relevant regional crops allow us to place high-contrast solid-color blocks outside the evidence region while keeping the questions and reference answers unchanged. As shown below, VisCA’s accuracy decreases slightly from 60.47% on clean images to 60.00% with distractors, a drop of 0.47 percentage points. VisCA remains effective under the tested distractions, suggesting that salient but irrelevant visual content does not readily disrupt its task performance.