acceptodds
Under review as a conference paper at ICLR 2027

Reward Hacking in Multimodal Reinforcement Learning via Linguistic Priors

Abstract

Reinforcement learning (RL) has become a powerful post-training paradigm for multimodal large language models (MLLMs), which are increasingly deployed in safety-critical domains. However, we reveal a critical yet overlooked phenomenon: models optimized via RL may exploit linguistic priors encoded in their language backbone to predict answers without truly processing image content. We term this behavior Visually-bypassed Shortcut Learning (VSL). Unlike reward hacking in text-only models, VSL is invisible to existing monitoring, as both pathways yield superficially identical reasoning traces. We show that this shortcut is systematically favored during RL optimization due to its lower computational cost, reduced reward variance, and the rich statistical structures of the language backbone, causing RL training to actively degrade visual grounding. Using image-perturbation sensitivity metrics, we verify the pervasiveness of VSL across multiple MLLMs and task types in clean training environments, and find its severity strongly correlates with task-specific linguistic prior strength. These findings expose a silent trade-off: benchmark scores keep improving while visual grounding steadily erodes, yet no existing signal indicates when the degradation becomes unacceptable. We therefore propose Visual Information Evaluation (VIE), a low-cost detector requiring only forward passes and no training modification, which exploits the insight that shortcut-reliant models need less visual information to maintain high rewards. Experiments show that VIE reliably detects visual reward hacking under both in-distribution and out-of-distribution settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reward Hacking, Reinforcement learning, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W2YXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the Visually-bypassed Shortcut Learning problem in MLLM post-training, where multimodal models become less sensitive to image perturbations after reinforcement learning despite improved task accuracy. The authors evaluate five models on three VQA datasets and propose the Visual Information Evaluation metric, which measures reward retention under progressive image degradation. Additional experiments examine alternative RL algorithms, process-based rewards, and image captioning.

Critical strengths
  1. The paper addresses an important question: whether improvements in answer accuracy correspond to improvements in visual grounding.
  2. Experiments cover several model families and scales, with supplementary results using an alternative RL algorithm.
  3. Multiple perturbation families and intermediate checkpoints provide complementary views of behavioral changes during training.
Critical weaknesses
  1. Reduced visual sensitivity does not necessarily indicate harmful visual degradation. If questions can be directly answered accurately from linguistic priors, this is a problem or characteristic of the dataset itself, and reduced image dependence can even be a reasonable adaptation to the dataset. The key experiment should separate RL training data from an independently validated, image-essential probing set and test whether training impairs visual performance there. Appendix F’s model-defined subset and Appendix D’s detector-transfer results do not establish this distinction.
  2. The task coverage is limited relative to the claims. The main experiments cover three VQA datasets. This is insufficient to establish VSL as an inherent or broadly task-independent property of multimodal RL. Experiments on representative post-training tasks with stronger visual requirements, such as visual mathematical reasoning, chart/document understanding, or spatial grounding, would strengthen the conclusions.
  3. VIE’s practical value for model training remains unclear. The paper does not show that using VIE improves training decisions or downstream performance. A useful experiment would filter or reweight training data using VIE and compare against unfiltered data, size-matched random selection, and a simple question-only-based selection baseline under comparable compute. Evaluation should use an independent image-essential benchmark.
  4. The reward-variance mechanism is insufficiently justified. For binary correctness rewards, Var(R)=p(1-p), so equal success probabilities imply equal reward variance regardless of the answer pathway. Lower reward variance also does not automatically imply lower policy-gradient variance or preferential optimization by GRPO.
Questions for authors

How do you verify that a perturbation removes information necessary for the particular question, rather than information that a correctly grounded model can safely ignore?

7 comments
AuthorsOpenReview ↗

Response to Reviewer W2YX (Part1)

We really appreciate your time and insightful comments. Thank you so much for acknowledging notable performance gain and detailed analysis of our work. The response to your concerns are summarized as follows.


W1: Reduced visual sensitivity does not necessarily indicate harmful visual degradation.

A1: We thank the reviewer for this important distinction. We agree that, for a sample that can be answered from the question alone, answering from linguistic priors is not wrong in itself. The question that matters is whether the reliance learned on such data carries over to samples where the image is indispensable. We address this in three ways.

(1). Image-essential probe set, separate from the RL training data and independent of the model. Following the reviewer's suggestion, we train on OK-VQA as before and evaluate on data in which the image is verified to be essential without using the model under study: (a) MMStar, a human-curated benchmark whose samples are selected to be vision-indispensable; (b) a VQAv2 subset of 1,000 questions that two text-only LLMs both fail to answer from the question alone; and (c) the 500 counterfactual pairs described below, in which the answer changes with the image by construction.

Evaluation setBefore RLAfter RL
OK-VQA (RL training distribution), accuracy ↑0.32130.4218
MMStar, accuracy ↑0.6980.681
VQAv2 text-unanswerable subset, accuracy ↑0.6120.604
Counterfactual pairs, prior-locked rate ↓4.6%15.8%

While accuracy on the training distribution increases, accuracy decreases on every image-essential set, and the prior-locked rate increases. The reduced visual reliance is therefore not confined to samples where priors suffice: it transfers to, and impairs, performance on data where the image is necessary.

(2) VQAv2 is itself designed to suppress language priors: each question is paired with complementary images that lead to different answers. If reduced visual sensitivity were merely a reasonable adaptation to prior-answerable data, it should be weakest on such a dataset. Yet VS decreases on VQAv2 for all five models.

(3) Reliance on priors is driven by the optimization, not only by the dataset. On the same OK-VQA data, process-based rewards reduce the VS decline by 69% compared with outcome-only rewards. If lower visual dependence were simply a property of the data, changing the reward function on identical data would not change it. This indicates that the pathway the model adopts is shaped by the RL objective, which is the core claim of VSL.


W2: Task coverage is limited relative to the claims.

A2: (1). We thank the reviewer for this suggestion, which we had also considered. Appendix G already extends the study beyond VQA to image captioning on COCO Captions, where there is no question text to exploit. The same signature appears: the benchmark reward (CIDEr) rises from 1.08 to 1.19, while image grounding decreases (CLIPScore 0.71 → 0.68, VS_gen 0.58 → 0.42), and VIE detects the shortcut-reliant checkpoint with an AUC of 0.76.

(2) Following the reviewer's suggestion, we add experiments on visual mathematical reasoning, a task with strong visual requirements. We train Qwen3-VL-4B with GRPO on Geometry3K, in which answering requires reading lengths, angles and relations from the diagram, using the same protocol as the main experiments:

Geometry3KBefore RLAfter RL
Accuracy ↑0.4680.521
VS, Patch Shuffle0.7020.661
VS, Gaussian Noise0.7450.698
VS, Blur (σ = 10)0.6870.617
VS, Average0.7120.659
VIE detection AUC–0.76

The VSL signature is present: accuracy increases while VS decreases under all three perturbations, and VIE separates the two checkpoints. The average VS drop is clearly smaller than on OK-VQA , comparable to the low-prior CLEVR setting. This matches the prediction stated in Appendix H that tasks strictly anchored in fine-grained visual structure, such as geometric reasoning, should exhibit milder rather than absent VSL, and it is consistent with our finding that VSL severity scales with the strength of task-specific linguistic priors.


AuthorsOpenReview ↗

Response to Reviewer W2YX (Part2)

W3: VIE's practical value for model training remains unclear.

A3: (1). The purpose of VIE is to tell users, during typical RL training such as GRPO, at which step the decline in visual sensitivity has become unacceptable, even though continued training may still improve benchmark scores. Monitoring this trade-off requires a low-cost detector that does not modify the training process. VIE detects the step at which the model's dependence on visual information has dropped to a non-negligible degree, and signals that developers should adjust the training strategy or the training data, rather than blindly continuing RL training.

(2). In our current design, VIE monitoring does not affect training at all. Different domains and methods may require different training strategies, so we keep the monitor independent of the training procedure; this ensures that VIE remains usable under any condition, and can be applied to training runs with different models, datasets and RL methods.

(3). The reviewer's suggestion is nevertheless reasonable, and we conduct the proposed experiment. Using the pre-RL model, we compute the per-sample VSL-Score of each training sample and remove those samples whose reward is stably retained under visual degradation. We compare against unfiltered training, a size-matched random selection, and a question-only-based selection that removes the same number of samples answerable without the image. All variants use the same number of training steps.

Training dataAcc ↑VS ↑
Before RL0.32130.6690
RL, unfiltered0.42180.4200
RL, size-matched random0.41960.4235
RL, question-only selection0.42470.4458
RL, VIE-based filtering0.43040.5169

After VIE-based filtering, the trained model retains substantially more visual dependence and its accuracy on OK-VQA is essentially unchanged (0.4304 vs. 0.4218; the small gain may be due to run-to-run randomness). Question-only selection helps partially, because it removes samples that are answerable without the image but not samples whose reward is stably retained under partial degradation. This shows that VIE can also inform training decisions, beyond monitoring. We will add this experiment to the paper.


W4: The reward-variance mechanism is insufficiently justified.

A4: (1). We agree with the reviewer. For a binary reward, Var(R) = p(1 − p), so two pathways with the same success probability have the same reward variance. Moreover, under GRPO, the advantage is normalized by the reward standard deviation within each group, so a difference in reward variance alone does not imply preferential optimization. The formulation in the paper is imprecise, and we will revise it.

(2). The reward-standard-deviation gap reported in Table 7 arises because the per-sample success probability differs between the two subsets: on prior-answerable samples, the model answers consistently and p is close to 1. The mechanism we intend is not about reward variance, but about the gradient signal that the two pathways provide, which is what the other rows of Table 7 measure: gradient SNR (1.5× vs. 1.0×), gradient-direction consistency (0.41 vs. 0.19) and convergence speed (∼110 vs. ∼370 steps).

(3). To separate the two, we repeat the analysis of Table 7 on subsets of S_lang and S_vis matched in their pre-RL success probability p:

StatisticS_langS_visS_lang (p-matched)S_vis (p-matched)
Reward std σ_R0.170.260.220.22
Gradient SNR (relative)1.5×1.0×1.4×1.0×
Gradient-direction cosine consistency0.410.190.380.20
Steps to 90% of final reward∼110∼370∼140∼350

Once p is matched the reward variance becomes equal. The gradient-level asymmetry, however, persists: the linguistic pathway still produces a higher gradient SNR, more consistent gradient directions across rollouts, and faster convergence. The optimization preference is therefore explained by the policy-gradient signal, not by the reward variance.

AuthorsOpenReview ↗

Response to Reviewer W2YX (Part3)

Q1: How do you verify that a perturbation removes information necessary for the particular question?

AQ1: We examined this question from several angles.

(1). We use multiple perturbation families to study how stable the model's answers are when visual information is unavailable. For the same perturbation family, we strictly vary the degradation strength, from the weakest setting, which may still retain most of the visual information, a pure noise image that contains no usable visual content. For information-free perturbations, every piece of information necessary for the question is removed by construction, so the question of necessity does not arise. Under Gaussian noise substitution, the post-RL model still answers more questions correctly (Table 1: 0.2737 → 0.3111). This multi-level statistical analysis shows that the unchanged answers are not explained by sufficient residual visual information.

(2). For partial perturbations such as blur, we do not assume at the sample level that every perturbation removes necessary information. VS is compared on the same samples under the same perturbation before and after RL, so for a sample where blur leaves sufficient information, the comparison is controlled; the change in VS reflects a change in the model rather than in the information available. Additionally, in Appendix D, under randomly mismatched images, which contain no information relevant to the question, the post-RL model keeps its answer far more often (behavioral shortcut index 0.05 → 0.19), showing that it indeed relies on textual priors.

(3). To verify necessity directly, we curated 500 counterfactual image pairs in which the question stays the same but the correct answer changes. In these pairs, the information that distinguishes the two images is, by construction, necessary for the question.

MetricBefore RLAfter RL
Prior-locked rate ↓4.6%15.8%
VS on counterfactual pairs ↑0.4120.368

After RL, the model gives the same question-determined answer to both images more than three times as often (4.6% → 15.8%). Under this setting, the lower VS after RL indicates both reduced reliance on visual information and greater reliance on linguistic priors.


Thank you again for your valuable suggestions. We value the opportunity to discuss our work with the research community and welcome any further questions or comments.

Reviewer_iQ86OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper investigates changes in multimodal models’ visual dependence during outcome-reward reinforcement learning. It introduces Visually-bypassed Shortcut Learning (VSL), measures answer changes under image perturbations using Visual Sensitivity (VS), and proposes Visual Information Evaluation (VIE), which aggregates normalized reward losses under progressive image degradation. Experiments cover five models and three VQA datasets, with supplementary analyses of GSPO, process rewards, intermediate checkpoints, and image captioning. The paper reports increased task accuracy alongside decreased visual sensitivity during RL and evaluates VIE for in-distribution and out-of-distribution visual-shortcut detection.

Critical strengths
  1. The paper examines a consequential failure mode beyond benchmark accuracy. Comparing original-image accuracy with image sensitivity before and after RL highlights that an improved task score need not imply improved visual reliance. This is a useful auditing perspective for multimodal post-training.

  2. The empirical investigation has useful breadth. Table 2 covers five models and three VQA datasets, with reduced visual sensitivity observed across the reported model–dataset combinations. This breadth makes the observed sensitivity changes more informative than a single-model example.

Critical weaknesses
  1. The central interpretation does not distinguish visual shortcuts from improved robustness. Blur may leave sufficient task-relevant visual evidence, so maintained reward can reflect better use of that evidence rather than reliance on language priors. Reduced perturbation sensitivity therefore does not by itself establish degraded visual grounding or reward hacking.

  2. The checkpoint-level validation in Appendix D contains an apparent statistical inconsistency. Table 5 (p.18) reports per-sample VSL-Score means and standard deviations of 0.79 ± 0.08 at step 0 and 0.18 ± 0.06 at step 10000, alongside an AUC of 0.87 for separating these checkpoints. If the AUC uses those same score distributions, a distribution-free one-sided variance bound implies an AUC of at least approximately 0.974, with lower scores identifying the post-RL checkpoint. These quantities therefore require reconciliation. This inconsistency weakens the evidence that VIE tracks shortcut reliance across RL checkpoints.

Questions for authors
  1. What evidence distinguishes language shortcuts from improved visual robustness? On matched image pairs with the same question but different correct answers, how often do the pre- and post-RL models answer both images correctly?

  2. In Appendix D, Table 5, are the reported means, standard deviations, and AUCs calculated from the same per-sample VSL-Scores? Please clarify the statistical definitions and evaluation protocol, and explain or correct the apparent inconsistency.

3 comments
AuthorsOpenReview ↗

Response to Reviewer iQ86

We really appreciate your time and insightful comments. The response to your concerns are summarized as follows.


W1 & Q1: The central interpretation does not distinguish visual shortcuts from improved robustness.

A1: We thank the reviewer for this insightful comment. We considered this question carefully during our study: a lower VS means that the model depends less on visual information, but does it also mean that the model depends more on linguistic shortcuts, rather than having become more robust? The two explanations make different predictions, and we tested them from three angles.

(1). For the same perturbation family, we strictly vary the degradation strength, from the weakest setting, which may still retain most of the visual information, to Gaussian noise substitution, a pure noise image with no usable visual content. Improved robustness can explain a maintained reward under mild blur, but it cannot explain a maintained reward when no visual evidence remains. Under Gaussian noise substitution, the post-RL model still answers more questions correctly (Table 1: 0.2737 → 0.3111), and its VS drops from 0.3730 to 0.2000. Furthermore, VS is compared on the same samples under the same perturbation before and after RL, so the amount of task-relevant evidence that blur leaves for each sample is held fixed; the change in VS therefore reflects a change in the model.

(2). A randomly mismatched image is a natural image that contains no information about the question. A more robust model has no reason to keep its answer when the image is replaced in this way, whereas a shortcut-reliant model does. In Appendix D (Table 5), the fraction of answers that remain unchanged under a mismatched image rises from 0.05 to 0.19 along the RL trajectory.

(3). Counterfactual image pairs, as reviewer suggests. We manually curated 500 pairs in which the question is identical, the images are similar, and the correct answers differ. A model with improved robustness should answer both images correctly more often; a shortcut-reliant model gives the same answer to both.

MetricBefore RLAfter RL
Prior-locked rate ↓4.6%15.8%
VS on counterfactual pairs ↑0.4120.368

The prior-locked rate is the fraction of pairs in which the model gives the same answer to both images and this answer is the correct answer for only one of them, i.e., the answer is determined by the question rather than by the image. On these pairs, residual visual information cannot explain an unchanged answer, because the visual evidence points to a different answer. Under this setting, lower VS indicates both reduced reliance on visual information and increased reliance on linguistic shortcuts after RL.


W2 & Q2: Apparent statistical inconsistency in Table 5 of Appendix D.

A2: We thank the reviewer for this careful reading, and we apologize for the confusion caused by our description.

(1). The standard deviations in Row 1 of Table 5 are not the dispersion of the per-sample VSL-Score distribution. They quantify the variability across three independent RL training runs with different random seeds under the identical configuration. Each entry of Row 1 is the dataset-level VSL-Score of one run, and we report the mean and the cross-run standard deviation over the three runs. We reported this specifically to show that the trend is stable across runs.

(2). The detection AUCs in Row 2, by contrast, are computed from the full per-sample score distributions under the protocol of Section 4.4. The per-sample distributions are much wider than the run-level means, because averaging over the evaluation set removes most of the sample-level variation. For completeness, the revised table reports both statistics:

RL steps01000300010000 (final)
Dataset-level VSL-Score (mean ± std over 3 runs)0.79 ± 0.080.52 ± 0.110.37 ± 0.090.18 ± 0.06
Per-sample VSL-Score (mean ± std over samples)0.73 ± 0.300.57 ± 0.330.42 ± 0.340.20 ± 0.29
Detection AUC vs. step 0 (per-sample)–0.630.750.87
Behavioral shortcut index0.050.090.110.19

With per-sample standard deviations of this magnitude, the two distributions overlap substantially, and an AUC of 0.87 is consistent with them. The two statistics are therefore complementary rather than contradictory.

(3). We further note that the run-level statistic strengthens our claim rather than weakening it: the differences in mean VSL-Score between adjacent checkpoints all exceed the cross-run standard deviations, so the monotonic decline is robust to seed-level training variance and is not an artifact of a single run.


Thank you again for your valuable suggestions. We value the opportunity to discuss our work with the research community and welcome any further questions or comments.

Thanks for the detailed response and extra experiments. The added analyses and statistical clarifications resolve my main concerns, so I'm happy to raise my rating.

We are excited to hear that most of the concerns are addressed and the reviewer raises the score. Thanks to the reviewer very much again for valuable suggestions, which significantly helped us improve the quality and clarity of the paper.

Reviewer_XjbGOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies visually-bypassed shortcut learning (VSL), where multimodal language models gradually rely more on language-based shortcuts and less on visual information during reinforcement learning. To better understand this issue, the authors introduce two evaluation methods: Visual Sensitivity (VS), which measures how model predictions change when images are modified, and Visual Information Evaluation (VIE), which examines how model performance changes when visual information is gradually removed. The authors conduct experiments on five MLLMs and three VQA datasets. Their results show that while reinforcement learning can improve overall accuracy, it can also make models less sensitive to visual information. They also explore how linguistic priors, model size, and different RL methods affect this behavior. Finally, they show that VIE can help identify models that rely more heavily on shortcuts, both on familiar datasets and in out-of-distribution settings.

Critical strengths
  1. Addresses an important problem in multimodal RL. The paper highlights an important issue: improving answer accuracy through RL does not necessarily mean that a model is making better use of visual information. This is particularly relevant because higher scores on VQA benchmarks may give the impression that a model's visual understanding has improved, even when it is relying more on language-based shortcuts. I think this is a meaningful problem to study, especially for evaluating multimodal models and improving the reliability of RL-based post-training.
  2. Solid empirical analysis and a potentially useful detection method. The paper provides a fairly thorough empirical study, covering five models, three datasets, and several types of image perturbations. One interesting result is shown in Table 1: for Qwen3-VL-4B on OK-VQA, accuracy increases from 0.3213 to 0.4218 after RL, while average visual sensitivity drops from 0.4730 to 0.3063. This provides concrete evidence that accuracy improvements can come with reduced sensitivity to visual inputs.
Critical weaknesses
  1. Visual sensitivity may not fully capture linguistic shortcut learning. My main concern is whether lower visual sensitivity necessarily means that a model is relying more on linguistic shortcuts. The VS metric measures how much a model's predictions change when the input image is perturbed, but there could be other explanations for the observed behavior. For example, a model might give the same answer after an image perturbation simply because enough relevant visual information is still available. On the other hand, a prediction change could be caused by the perturbation itself introducing a distribution shift, rather than reflecting meaningful visual reasoning. The authors do try to address this issue through different perturbation methods and mismatched-image experiments, which I appreciate. However, I am not fully convinced that these analyses can clearly separate linguistic shortcut learning from other reasons why predictions remain unchanged. Since this distinction is central to the paper's main argument, I think it deserves more careful discussion and stronger evidence.
  2. The evaluation of VIE could benefit from more independent validation. Another concern is how VIE is validated. Much of the evaluation compares models before and after RL, using these two stages as indicators of different levels of visual grounding. While this comparison is reasonable, it does not necessarily establish that VIE is specifically detecting visually-bypassed shortcut learning. The additional behavioral labeling and checkpoint analyses in Appendix D are helpful, but I would like to see stronger independent evidence that VIE measures shortcut reliance rather than more general changes in model behavior caused by RL.
Questions for authors
  1. Visual sensitivity and shortcut learning: How can we tell whether a lower VS score really means the model is relying more on linguistic shortcuts? For example, the model might still have enough visual information to answer correctly after an image perturbation, or the perturbation itself might cause a distribution shift. Have the authors considered testing with counterfactual image pairs where the question stays the same but the correct answer changes? I think this would provide stronger evidence for the paper's main claim.
  2. Validation of VIE: How well does VIE detect shortcut learning when the ground-truth labels are determined independently of whether a model has undergone RL training? The behavioral labeling analysis in Appendix D is helpful, but it would be useful to clarify how those labels were obtained and whether they are independent of the image degradation process used by VIE.
  3. Calculation of VSL-Score: Could the authors clarify how Equation 5 handles individual samples where the original-image reward is zero? I'm also curious whether the way these cases are handled could affect the reported AUC results or the VSL-Score distributions across training checkpoints.
6 comments
AuthorsOpenReview ↗

Response to Reviewer XjbG (Part1)

We really appreciate your time and insightful comments. Thank you so much for acknowledging notable performance gain and detailed analysis of our work. The response to your concerns are summarized as follows.


W1 & Q1: Lower visual sensitivity may not fully capture linguistic shortcut learning.

A1: We thank the reviewer for this insightful and inspiring comment. We considered this question in depth during our study: a lower VS means that the model depends less on visual information, but does it also mean that the model depends more on linguistic shortcuts? Reviewer raises two alternative explanations, namely (a) that the answer is unchanged because sufficient visual information survives the perturbation, and (b) that a changed answer reflects a distribution shift introduced by the perturbation. We examined this question from three angles.

(1). We use multiple perturbation families, and for the same family we strictly vary the degradation strength: the weakest setting may still retain most of the visual information, whereas the strongest setting, is a pure noise image that contains no usable visual content. If unchanged answers were explained only by surviving visual information (explanation a), the gap between the pre-RL and post-RL models should vanish once no information remains. It does not. Under Gaussian noise substitution, the post-RL model still answers more questions correctly than the pre-RL model. Since this image carries no information about the question, these correct answers cannot come from visual evidence. In addition, VS is always compared on the same samples under the same perturbation before and after RL; the amount of information a perturbation leaves for a given sample is therefore held fixed, and the change in VS reflects a change in the model rather than in the residual information.

(2). Regarding explanation (b), a mismatched image is a natural image from the same dataset, so replacing the image with it introduces no low-level distribution shift, yet it removes all information relevant to the question. In Appendix D, the fraction of samples whose answer remains unchanged under a random mismatched image rises from 0.05 to 0.19. Patch Shuffle, which preserves local texture and color statistics and therefore introduces minimal distribution shift, shows the same VS decline. Statistically, across perturbations that differ in how much information they remove and in how much distribution shift they introduce, the post-RL model consistently relies less on visual input and more on textual priors.

(3). Following the reviewer's suggestion, we manually curated 500 counterfactual pairs, in which the question is identical, the two images are similar, and the correct answers differ. For a model that uses the image, the answer must change with the image; for a model that relies on the question, it will not.

MetricBefore RLAfter RL
Prior-locked rate ↓4.6%15.8%
VS on counterfactual pairs ↑0.4120.368

The prior-locked rate is the fraction of pairs in which the model gives the same answer to both images and this answer is the correct answer for only one of them, i.e., the answer is determined by the question rather than by the image. On these pairs, residual visual information cannot explain an unchanged answer, because the visual evidence points to a different answer. Under this setting, lower VS indicates both reduced reliance on visual information and increased reliance on linguistic shortcuts after RL.

AuthorsOpenReview ↗

Response to Reviewer XjbG (Part2)

W2 & Q2: VIE would benefit from more independent validation.

A2: (1) The core mechanism of VIE is to monitor how much a model depends on visual information. Typical RL training gradually reduces this dependence, so VIE can be used to monitor the training process and indicate the point at which a model should not be trained further in the same way. It is a low-cost and efficient way to monitor whether training remains well-founded. Its validation therefore requires evidence that VIE tracks shortcut reliance itself, rather than any change that RL happens to introduce.

(2) Appendix D provides three pieces of evidence for this, and we clarify how the labels were obtained.

  • ① VIE is entirely training-free. No parameter is fitted to the Before/After labels; the labels are used only to compute the AUC afterwards, in the same way that in-distribution and out-of-distribution partitions serve as ground truth in OOD detection.

  • ② The behavioral labels are obtained independently of VIE's degradation process. A sample is labelled as shortcut-reliant if it is answered correctly when the image is replaced by a blank image and when it is replaced by a randomly mismatched image (the S_lang criterion in Appendix F). This criterion uses image replacement, involves no reward curve, and does not depend on which checkpoint produced the answer. Re-labelling with this criterion yields an AUC of 0.88, nearly identical to the training-state labels.

  • ③ VIE does not merely detect general behavioral changes caused by RL. If it did, other black-box signals that also change between the two checkpoints would separate them as well. Appendix E shows that they do not: question-only accuracy, blank-image accuracy, single severe perturbation, and mismatched-image sensitivity all remain close to chance (AUC 0.498–0.586), while VIE reaches 0.877. Moreover, along the training trajectory (Table 5), question-only accuracy stays flat, whereas the VSL-Score decreases and the mismatched-image behavioral index increases in exact anti-phase. What changes is which pathway the model uses to answer, which is precisely what VIE measures.


Q3: How does Eq. 5 handle samples whose original-image reward is zero?

A3: (1). For the dataset-level VSL-Score in Eq. 5, C_θ(1) is the mean reward over the dataset and is strictly positive in all our settings (the smallest value is 0.226, on VQAv2), so the ratio is always well defined.

(2). For the per-sample VSL-Score used in Appendix D and in the AUC computation, C_θ(α) for a sample is the mean reward over 4 sampled responses. When all responses on the original image receive zero reward, the sample has no reward to retain, the ratio C_θ(α)/C_θ(1) is undefined, and the sample carries no information about reward retention. Such samples are excluded from the per-sample score distribution. We will state this explicitly below Eq. 5.

(3). Since the post-RL checkpoint answers more samples correctly, excluding these samples could in principle change the set of samples compared between the two checkpoints. As a check, we recompute the AUC on the common subset of samples that have a nonzero original reward under both checkpoints, so that the two checkpoints are compared on exactly the same samples. This requires no additional inference, only a re-selection of the existing per-sample scores. The AUC on the common subset is 0.859, compared with 0.877 in the reported setting, indicating that the handling of these cases does not significantly affect the reported AUC or the checkpoint-level trends.


Thank you again for your valuable suggestions. We value the opportunity to discuss our work with the research community and welcome any further questions or comments.

AuthorsOpenReview ↗

This Is a Kind Inquiry to Reviewer XjbG

Dear Reviewer XjbG,

On November 14, we received an email showing your new comments on our rebuttal, which appear to have been subsequently deleted. In that comment, we noted the new questions you raised:

“That said, I think there is still a gap between showing reduced visual sensitivity and concluding that the model increasingly relies on linguistic priors. The observed changes could also reflect weaker visual discrimination or other shifts in prediction behavior. It would be helpful if the authors could clarify how the 500 counterfactual pairs were constructed and whether the conclusions still hold when controlling for accuracy on the original images.”

First, we thank the reviewer for the careful follow-up and for finding the counterfactual, Gaussian-noise and mismatched-image analyses helpful. We address the reviewer's questions in turn.

—

Q1: How are the 500 counterfactual pairs constructed?

A1: We followed reviewer's suggestion, each pair shares one question and differs only in the image, such that the correct answer changes.

(1). We start from questions whose answer is determined by visible content. For a question that already has a natural counterpart image in VQAv2's complementary pairs, we use it directly.

(2) Many questions have no suitable second image, since a naturally occurring image that keeps the scene but changes the relevant content is rare. For these we construct the counterpart by image editing: we edit only the region that the answer depends on, leaving the scene, style, composition and lighting unchanged. The question and the prompt template are therefore identical within a pair, and the only difference between the two inputs is the visual evidence that determines the answer.

(3) Each edited counterpart was checked to confirm two things: that the correct answer indeed changes after the edit, and that no artifacts were introduced outside the edited region that could make the image recognizable as manipulated. Pairs failing either check were discarded.


Q2: Do the conclusions still hold when controlling for accuracy on the original images?

A2: Yes. Since RL raises accuracy on the original images, we recompute the analyses on the subset of samples that both the pre-RL and the post-RL model answer correctly on the original image, so that original-image accuracy is identical by construction:

Metric (common-correct subset)Before RLAfter RL
Average VS0.4520.318
Unchanged answers under mismatched images0.060.17
Prior-locked rate (counterfactual pairs)4.1%14.2%

On this subset the two models are equally accurate on the original images, yet the post-RL model is markedly less sensitive to image changes and gives the question-determined answer to both images of a pair more than three times as often. The conclusions therefore do not depend on the accuracy difference on the original images.


Q3: How do we distinguish weaker visual discrimination from reliance on linguistic priors?

A3: Weaker discrimination and prior reliance make different predictions.

  • ① Under information-free inputs, weaker discrimination cannot raise accuracy. With Gaussian noise substitution the image carries no information about the question, so a model that has merely become worse at reading images should be no more accurate than before. Instead, accuracy rises from 0.2737 to 0.3111 after RL (Table 1). These additional correct answers can only come from the question text.

  • ② Weaker discrimination produces scattered errors; prior reliance produces convergent ones. If the model discriminated images less precisely, its answers on the two images of a counterfactual pair would become noisier and diverge in arbitrary ways. We observe the opposite: the model increasingly gives the same answer to both images (prior-locked rate 4.6% → 15.8%), and that answer is consistent with the question for only one of the two images.

  • ③ The same convergence appears with natural images. Under a randomly mismatched image, which is a natural image with no distribution shift but no information about the question, the fraction of unchanged answers rises from 0.05 to 0.19 along the RL trajectory (Table 5).


We would be very happy to provide any further details if the reviewer has remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Work and the Discussion

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their time, careful reading and constructive feedback, and the Area Chair for handling our submission.

I. Paper Recognized Strengths

Strengths acknowledged by the reviewers:

  • An important and under-studied problem. Higher accuracy after RL does not imply better use of visual information; this is "a meaningful problem to study" (XjbG), "a consequential failure mode beyond benchmark accuracy" and "a useful auditing perspective for multimodal post-training" (iQ86), and "an important question" (W2YX).
  • Breadth and solidity of the empirical study. Five models, three datasets and several perturbation types (XjbG, iQ86); multiple model families and scales, an alternative RL algorithm (W2YX).
  • Concrete evidence of the accuracy–grounding divergence. (XjbG).
  • A potentially useful, low-cost detection method (XjbG).

II. Main Concerns and Our Responses

1. Does lower VS really indicate reliance on linguistic priors? (XjbG-W1/Q1, iQ86-W1/Q1, W2YX-Q1; follow-up by XjbG)

Our response:

  • Information-free inputs. Under pure Gaussian-noise images, the post-RL model still answers more questions correctly (0.2737 → 0.3111). Since the image carries no information, these answers can only come from the question text, which neither residual information, robustness nor weaker discrimination can explain.
  • Natural mismatched images (no distribution shift). The fraction of answers unchanged under a randomly mismatched natural image rises from 0.05 to 0.19 along the RL trajectory.
  • New: 500 counterfactual image pairs (same question, different correct answers; natural complementary pairs from VQAv2, plus image-edited counterparts with only the answer-relevant region modified and manually verified). The prior-locked rate (same answer to both images, correct for only one) rises from 4.6% to 15.8%, and VS on these pairs drops from 0.412 to 0.368.
  • New: controlled for original-image accuracy. On the subset both models answer correctly on the original image, the post-RL model still shows lower VS, more unchanged answers under mismatched images, and a ~3× higher prior-locked rate.

2. Is the reduced visual reliance actually harmful, or dataset adaptation / forgetting? (W2YX-W1; follow-up by W2YX)

Our response:

  • New: image-essential probe sets independent of the training data and of the model (MMStar, a text-unanswerable VQAv2 subset, counterfactual pairs): accuracy rises on the training distribution but decreases on every image-essential set.
  • The small accuracy change is the phenomenon itself, not a weakness of the measurement. Answers from the image and from the prior are indistinguishable on accuracy whenever the prior is correct; this is exactly why VSL is invisible to accuracy-based monitoring. The effect is carried by VS (−35% relative) and the prior-locked rate (×3.4).
  • Forgetting cannot explain the pattern. Forgetting makes a model wrong more often, not systematically wrong toward the prior; and on identical data, a process-based reward reduces the VS decline by 69% (Appendix A.3), so the effect is driven by the RL objective rather than by the dataset or the evaluation shift.

3. Independent validation of VIE (XjbG-W2/Q2)

Our response:

  • VIE is training-free; the Before/After labels are used only to compute AUC.
  • Behavioral labels independent of VIE's degradation process (blank and mismatched-image replacement, no reward curve) yield AUC 0.88, nearly identical to 0.877.
  • VIE does not merely detect generic RL-induced change: all other black-box signals stay near chance (AUC 0.498–0.586), and along training, question-only accuracy stays flat while the VSL-Score and the behavioral shortcut index move in exact anti-phase.

4. Task coverage (W2YX-W2)

Our response:

Beyond the image-captioning study in Appendix G (CIDEr ↑ while grounding ↓, VIE AUC 0.76), we added visual mathematical reasoning (Geometry3K): accuracy increases while VS decreases under all perturbations, and VIE separates the checkpoints. The milder effect matches our prediction that VSL scales with linguistic-prior strength.

5. Practical value of VIE for training (W2YX-W3)

Our response:

New: VIE-based training-data filtering, compared with unfiltered training, size-matched random selection and question-only selection under the same budget. VIE filtering retains substantially more visual dependence (VS 0.5169 vs. 0.4200) with no loss in accuracy (0.4304 vs. 0.4218), outperforming both baselines.


We believe the revisions address the main concerns, and we would be glad to provide any further clarification if needed. Thank you again for your time and consideration.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.