acceptodds
Under review as a conference paper at ICLR 2027

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for advancing complex reasoning in large language models, and recent work extends RLVR to multimodal large language models (MLLMs). This transfer, however, surfaces a faithfulness challenge: faithful perception of task-relevant visual evidence and faithful use of that evidence during reasoning. Existing perception supervision often operates on textual descriptions rather than natively on image regions, while answer-level rewards provide limited guidance on when to attend to visual evidence during reasoning. We introduce Faithful-MR1, an anchor-then-reinforce training framework that guides where and when to attend to visual evidence through question-relevant spatial supervision and intervention-guided attention reinforcement. The Anchoring stage turns perception into an explicit pre-reasoning subtask, supervising a dedicated <Focus> token's attention directly against image regions rather than through textual descriptions. The Reinforcing stage uses counterfactual image intervention to identify vision-dependent response tokens, rewarding answer-correct trajectories that concentrate visual attention at those positions. Extensive experiments demonstrate that Faithful-MR1 outperforms recent multimodal reasoning baselines on both Qwen2.5-VL-Instruct 3B and 7B backbones while using substantially less training data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Reasoning, Visual Evidence Utilization, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tCzDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Faithful-MR1, a two-stage training framework designed to improve faithful visual reasoning in multimodal large language models. The first stage, Anchoring, introduces a dedicated <Focus> token and supervises its attention toward question-relevant image regions using bounding-box annotations. The second stage, Reinforcing, identifies vision-dependent reasoning tokens through counterfactual image masking and encourages stronger visual attention at these positions using an auxiliary reward during GRPO training. Experiments on Qwen2.5-VL-3B and 7B demonstrate consistent improvements over GRPO and VPPO across seven multimodal reasoning benchmarks. Additional ablation studies suggest that both training stages contribute to the improvements.

Critical strengths
  1. Considers an important question in VLM reasoning, that how to make VLM focus on relevant areas of image.
  2. The proposed method achieves consistent improvements across seven benchmarks and two model scales. The ablation studies also demonstrate the complementary benefits of the Anchoring and Reinforcing stages.
Critical weaknesses
  1. The paper encourages stronger visual attention at vision-dependent reasoning tokens by optimizing the ratio between their attention to visual tokens and that of other response tokens. However, this relative attention ratio can be increased either by enhancing attention at vision-dependent tokens or by suppressing attention at other positions. Consequently, an improvement in this metric does not necessarily indicate more effective utilization of visual evidence. Furthermore, the attention reward includes both image patch tokens and the <Focus> token as visual tokens. It remains unclear whether the resulting improvements primarily come from attending to the actual image content or from attending to the learned <Focus> representation. More targeted analyses or interventions would help clarify how the proposed mechanism improves visual reasoning.
  2. The Reinforcing stage identifies vision-dependent tokens by selecting the top 10% of response tokens ranked by the KL divergence between predictions conditioned on original and masked images. However, this fixed selection ratio lacks sufficient justification. Even when a response is largely insensitive to visual interventions, the method still selects 10% of its tokens as vision-dependent. This raises concerns about whether the selected tokens genuinely represent positions where visual evidence is required. The paper would benefit from an analysis of the KL divergence distribution and ablation studies on different selection ratios or absolute sensitivity thresholds.
  3. Faithful-MR1 generates shorter reasoning responses than VPPO (325 versus 382 tokens on average), which may influence how frequently visual facts are explicitly reported and subsequently evaluated. Since the Faithful Use metric is conditioned on correctly reported facts, differences in response length and reporting behavior may also affect the results.
  4. The training dataset is annotated with question-relevant bounding boxes using Gemini-3-Flash, a powerful multimodal model. These annotations identify the visual regions most relevant to solving each problem. This raises a fundamental question: To what extent do the improvements come from the proposed training mechanisms, rather than simply encouraging the model to attend to regions identified by a stronger model? Although the paper includes GRPO and VPPO baselines trained on the same examples, it lacks a simpler baseline that directly increases attention to the annotated regions without introducing the proposed two-stage framework or counterfactual token identification.
Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer tCzD [W1-W2] (Part 1/2): Attention reward and VD selection

W1. Absolute visual attention and the role of <Focus>

Thank you for the suggestion. We provide two targeted analyses to clarify the attention reward.

First, Figure 5 (Appendix C.1) tracks the auxiliary attention reward and the absolute visual attention at vision-dependent positions, avda^{\mathrm{vd}}, over the first 150 training steps. The reward increases alongside avda^{\mathrm{vd}}, showing that its improvement is accompanied by stronger visual attention at these positions, rather than solely by reduced attention elsewhere.

Second, we exclude <Focus> from the attention statistics used to compute the auxiliary reward, counting only image-patch attention:

Attention included in the rewardMath Avg.General Avg.
Image patches + <Focus>41.649.9
Image patches only41.150.7

The results show small changes in both directions, with no consistent degradation. This suggests that directly rewarding attention to <Focus> is not essential to maintaining this performance level. This ablation retains <Focus> as a learned representation and isolates only its inclusion in the reward; it does not establish that the representation itself is unnecessary.

W2. Absolute KL threshold and candidate selection ratio

Thank you for pointing this out. Our manuscript inadvertently omitted the absolute KL threshold (Dt≥0.3D_t \geq 0.3), which was already present in the implementation. We apologize for this omission and have corrected Section 3.3 (p. 4), with the clarification marked in blue. We first select the top-10% candidate positions by KL divergence and retain only those satisfying this threshold. Consequently, the final VD set can be empty; the method does not force every response to contribute 10% VD tokens.

Our analysis of training responses shows that 4.81% of response tokens are selected as VD tokens on average, while 11.20% of responses contain no VD tokens. Separately, we examine the KL distribution of 205,128 top-10% candidates from 10,240 responses before thresholding. The CDF in Figure 6 (Appendix C.2) shows that 55.52% have KL < 0.3 and would be excluded by the implemented threshold.

To examine the threshold’s contribution and sensitivity to the candidate ratio, we consider three selection rules:

Selection ruleMathVisionMathVerseMathVistaWeMathDynaMathMath Avg.MMMU-ProHallusionBenchGeneral Avg.
Top 10% + KL ≥ 0.323.640.961.962.818.641.631.768.049.9
Top 10% only22.740.062.763.217.741.331.166.748.9
Top 20% + KL ≥ 0.323.240.662.363.517.741.431.268.950.1

The first two settings examine the effect of removing the absolute threshold; the first and third examine increasing the candidate ratio while retaining the threshold. The results support the role of the absolute threshold in filtering low-sensitivity candidates, while the small changes between 10% and 20% suggest limited sensitivity to the candidate ratio.

AuthorsOpenReview ↗

Response to Reviewer tCzD [W3-W4] (Part 2/2): Fact-use evaluation and region supervision

W3. Response length and fact-reporting effects

Thank you for raising this concern. We agree that response length and reporting behavior can affect Faithful Use, and therefore conduct two complementary analyses.

  • Length matching: Across 198 same-question pairs with response-length differences within 10%, average lengths are 355.9 vs. 359.0 tokens, while Faithful Use is 69.92% vs. 65.13% for Faithful-MR1 and VPPO.
  • Shared facts: Separately, on 1,274 facts correctly reported by both models, Faithful-MR1 achieves 67.50% vs. 65.46%, using an identical fact denominator.

Faithful-MR1 also achieves higher answer accuracy (45.6% vs. 42.4%) across all 788 questions, independently of the fact-reporting denominator.

Together, these results support our claim that Faithful-MR1 more consistently uses correctly reported visual evidence and achieves better task performance: the Faithful Use advantage persists under length matching and remains directionally consistent when comparing the same facts.

W4. Contribution beyond region annotations

Thank you for raising this attribution question. We add GRPO + Region-Attention Loss on Qwen2.5-VL-3B-Instruct as a controlled baseline using the same training examples and Gemini-3-Flash region annotations as Faithful-MR1. It directly encourages response-token attention to the annotated patches, without the proposed <Focus> token or counterfactual token identification.

MethodUses region annotationsMath Avg.General VQA Avg.Overall
GRPONo36.048.739.6
GRPO + Region-Attention LossYes38.948.941.7
Faithful-MR1Yes41.649.943.9

The improvement from 39.6 to 41.7 shows that directly exploiting the region annotations is beneficial. However, with access to the same annotations, Faithful-MR1 reaches 43.9, a further 2.2 percentage-point gain. This supports an additional benefit of the full framework beyond directly increasing attention to regions identified by Gemini-3-Flash.

AuthorsOpenReview ↗

Follow-up on W1–W4 and Appendix C

Dear Reviewer tCzD,

Our two-part response covers W1–W4. The new figures on visual attention and token selection are in Appendix C of the revised manuscript. We’d welcome your thoughts on whether our response addresses your main concerns and whether there is anything we could clarify further.

AuthorsOpenReview ↗

Final Note Before the Discussion Closes

Dear Reviewer tCzD,

Thank you for your thoughtful questions. They helped shape the additional analyses in our rebuttal.

With only a few hours left in the discussion period, we would welcome your thoughts on whether our response addresses your concerns. We hope the new evidence will be helpful as you finalize your rating, and we’d be happy to clarify any remaining points.

Reviewer_mNfUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents Faithful-MR1, an anchor-then-reinforce training framework that guides where and when to attend to visual evidence through question-relevant spatial supervision and intervention-guided attention reinforcement. 19.2K bounding‑box annotated dataset is constructed. Qwen2.5‑VL‑Instruct (3B / 7B) is evaluated across seven multimodal benchmarks covering mathematical visual reasoning and general VQA, demonstrating the effectiveness.

Critical strengths

The identified problem is interesting, namely the critical perception‑reasoning disconnect (PRD) — a known but under‑addressed failure pattern for multimodal reasoning RLVR methods.

The training design is interesting and novel, where the two‑stage pipeline treats “where to look” and “when to use visual evidence” as separate but complementary training objectives.

Critical weaknesses
  1. The method highly relies on the human-level bounding-box. The whole framework relies on Gemini‑3‑Flash to generate question‑relevant bounding boxes. Annotation noise or imperfect grounding will degrade both Anchoring supervision and counterfactual masking in Reinforcing. As such, the real-world applications with no high-quality box annotations may not be applicable.

  2. The experimental results are not sufficient. The paper mainly compares against GRPO and VPPO. The paper should compared to other approaches, such Ground-R1 and so on. which can further demonstrate the performance gains originality from the proposed <Focus>+counterfactual‑reward pipeline or the simpler region‑supervision recipes.

  3. The experiments lack of cross‑model backbone generalisation evidence. All main experiments are built exclusively on Qwen2.5‑VL‑Instruct. What about the performance on Qwen3.5 or other VLMs?

  4. The auxiliary attention reward is threshold‑based binary reward, which may induce bias, such as boosting attention weights but not performing correct reasoning.

Questions for authors

Check the detailed comments in weaknesses part.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mNfU [W1-W4]: Annotations, baselines, backbones, and reward

W1. Region annotations and applicability

We thank the reviewer for highlighting this limitation. As acknowledged in our limitations section, Faithful-MR1 relies on question-relevant region annotations during training, whose quality can affect both Anchoring supervision and counterfactual masking in Reinforcing. However, these annotations are generated automatically by a VLM, requiring no manual box annotation. Moreover, at inference, Faithful-MR1 requires only the image and question, with no bounding boxes or external annotator. Thus, the absence of box annotations at deployment does not itself prevent application of the trained model. Obtaining reliable question-relevant region annotations in challenging domains remains an open problem, which may limit applicability in such settings. A practical mitigation could be to filter training annotations based on agreement across multiple annotators.

W2. Additional comparisons and simpler region supervision

Thank you for the suggestion. Beyond GRPO and VPPO, we compare against Perception-R1, which rewards consistency with textual visual annotations, and Vision-SR1, which uses self-rewards to encourage visual descriptions sufficient for correct answering. Both introduce explicit perception signals beyond final-answer rewards. Faithful-MR1-7B achieves an overall score of 51.3, compared with 49.8 for Perception-R1 and 49.0 for Vision-SR1 (Table 1).

To examine whether simpler region supervision can explain the gains, we additionally evaluate GRPO + Region-Attention Loss on Qwen2.5-VL-3B-Instruct. This baseline uses the same training examples and region annotations as our full method, directly supervising response-token attention toward the annotated patches without the proposed <Focus> token or counterfactual token identification.

MethodMath AvgGeneral AvgOverall
Qwen2.5-VL-3B-Instruct35.846.438.8
+ GRPO36.048.739.6
+ VPPO37.748.740.8
+ GRPO + Region-Attention Loss38.948.941.7
+ Faithful-MR141.649.943.9

Direct region supervision improves the overall score from 39.6 to 41.7, confirming the value of the annotations. With access to the same region annotations, Faithful-MR1 still achieves a further 2.2 percentage-point gain, with improvements in both mathematical reasoning and general VQA. These results support the additional benefit of the proposed pipeline beyond simply encouraging attention to regions identified by the annotating model.

W3. Evaluation on additional backbones

Thank you for highlighting the need for broader backbone evaluation. We extend our experiments to Qwen3-VL-4B-Instruct and Qwen3.5-4B, using the same training data and evaluation protocol as in the main experiments. For each backbone, we compare GRPO with Faithful-MR1 to assess whether the improvements transfer beyond Qwen2.5-VL.

MethodMathVisionMathVerseMathVistaWeMathDynaMathMath AvgMMMU-ProHallusionBenchGeneral AvgOverall
Qwen3-VL-4B-Instruct + GRPO30.254.267.672.431.951.241.270.155.752.5
Qwen3-VL-4B-Instruct + Faithful-MR131.255.268.175.232.452.441.670.856.253.5
Qwen3.5-4B + GRPO34.263.974.377.839.958.049.371.560.458.7
Qwen3.5-4B + Faithful-MR135.565.174.178.543.359.350.275.062.660.3

Faithful-MR1 achieves higher overall scores on both additional backbones, improving all seven benchmarks on Qwen3-VL-4B-Instruct and six of seven on Qwen3.5-4B. Both backbones also show improvements in Math Avg and General Avg. These results provide additional evidence that the benefits of our framework extend beyond Qwen2.5-VL to newer Qwen backbones.

W4. Correctness-conditioned attention reward

Thank you for raising this concern. The attention reward is explicitly conditioned on answer correctness (Sec. 3.3), tying it to the reasoning outcome: a trajectory that increases attention but produces an incorrect answer receives no auxiliary reward. Thus, higher attention alone is insufficient to earn this reward.

The binary threshold further limits the incentive to inflate attention: once g>τgg > \tau_g, increasing the ratio yields no additional reward. It encourages meeting a relative-attention criterion rather than continually maximizing the ratio.

The gains also extend to task performance: replacing Vanilla GRPO with Reinforcing improves the overall score by 1.1 points after Vanilla SFT (48.6 → 49.7) and 2.1 points after Anchoring (49.2 → 51.3; Table 2).

AuthorsOpenReview ↗

Follow-up on W1–W4

Dear Reviewer mNfU,

Our full response includes the comparison with direct region supervision and the results on newer Qwen backbones. We’d be glad to hear whether these additions address your main concerns and whether any points need further clarification.

The authors mostly address my comments, with additional experiments on the new VLMs as well as the simple regional annotation.

Reviewer_knWsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the perception–reasoning disconnect in multimodal reasoning: a model may identify a visual fact correctly but ignore or misuse it later in its derivation. Faithful-MR1 trains visual attention in two stages. Anchoring uses question-relevant bounding boxes to supervise where a <Focus> token attends before reasoning. Reinforcing masks those regions to identify visually sensitive response positions, then rewards correct responses that attend more to visual information at those positions. With matched backbones and training data, the method outperforms GRPO and VPPO on average across seven benchmarks at both 3B and 7B scales. Ablations suggest that the two stages are complementary.

Critical strengths
  1. The paper usefully separates recognizing visual facts from using them in reasoning. It highlights a failure mode that supervision of visual descriptions or final answers may miss, and the two training stages align well with evidence localization and evidence use.
  2. The evaluation includes matched-backbone, matched-data baselines and ablations of Anchoring and Reinforcing. On the 7B model, the overall score rises from 49.5 with VPPO to 51.3, and the full method exceeds variants using either stage alone. These results support the method’s effectiveness and the complementarity of its components.
Critical weaknesses
  1. The evidence for faithful use of visual information remains indirect. A higher attention ratio alone does not establish that a visual fact causally affects the answer. The fact-use measure relies on a model-based judge and is conditioned on facts already stated correctly. The results therefore support improved consistency in the written reasoning more directly than a stronger claim about the model’s internal mechanism.
  2. The method requires key-region annotations and an additional masked-image forward pass during reinforcement learning. These costs are not fully quantified, and compute-matched comparisons and variability across runs are missing. Thus, the smaller training set and some modest performance gains do not yet establish the method’s practical efficiency or stability.
Questions for authors
  1. How do performance and fact-use scores change if the auxiliary reward counts attention only to image patches and excludes <Focus>? Could the authors manually audit a sample of the fact-use judgments to assess the reliability of the automatic evaluation?
  2. Could the authors report the additional training time and compute required by each stage, together with results across multiple seeds? Does the performance advantage remain under a comparable training-compute budget?
2 comments
AuthorsOpenReview ↗

Response to Reviewer knWs [W1, W2, Q1, Q2]: Faithfulness, compute, and evaluation

W1. Interpretation of faithfulness evidence

Thank you for highlighting this distinction. In Section 4.4, we describe the fact-use metrics as behavioral measures rather than direct measurements of internal perception or computation. Our central contribution is a training framework supported by benchmark improvements and component ablations, with attention and fact-use analyses providing complementary evidence of its effectiveness. We will clarify this distinction throughout the discussion so that our interpretations remain consistent with what these analyses directly support.

W2 / Q2. Training cost, matched compute, and variability

Thank you for these related comments. We report training costs and results across three random seeds, including a compute-matched comparison.

Training cost. For the cost comparison below, all methods use Qwen2.5-VL-3B-Instruct and 250 RL steps on 32 H20 GPUs. Faithful-MR1 requires 1.5 GPU-hours for Anchoring and 494.5 GPU-hours for Reinforcing, totaling 496 GPU-hours.

MethodTotal GPU-hoursAdditional cost vs. GRPO
GRPO395—
VPPO479+84 (+21.3%)
Faithful-MR1496+101 (+25.6%)

Faithful-MR1 requires 3.5% more total compute than VPPO at 250 RL steps.

Matched compute and variability. We additionally evaluate VPPO at 496 GPU-hours, matching Faithful-MR1’s total training budget. The scores below are mean ± standard deviation across three random seeds (1, 2, and 3).

MethodMathVisionMathVerseMathVistaWeMathDynaMathMath Avg.MMMU-ProHallusionBenchGeneral Avg.Overall
VPPO, 250 steps20.4 ± 1.636.3 ± 0.660.3 ± 2.252.4 ± 2.214.9 ± 0.236.9 ± 1.329.6 ± 0.666.0 ± 2.747.8 ± 1.540.0 ± 1.3
VPPO, 496 GPU-hours20.8 ± 0.535.7 ± 0.0457.1 ± 0.649.6 ± 1.216.7 ± 0.436.0 ± 0.630.4 ± 0.164.7 ± 1.047.5 ± 0.539.3 ± 0.5
Faithful-MR1, 496 GPU-hours24.0 ± 0.542.2 ± 1.662.9 ± 1.262.0 ± 1.118.7 ± 0.342.0 ± 0.532.2 ± 0.868.4 ± 0.850.3 ± 0.544.3 ± 0.5

We acknowledge that key-region annotation introduces an additional, one-time preparation cost. We will make this cost explicit alongside statements about using less training data, distinguishing reduced data volume from lower overall preparation cost.

At the same total training budget, Faithful-MR1 achieves an overall advantage of approximately 5 points with the Qwen2.5-VL-3B-Instruct backbone. Its overall standard deviation of 0.5 points indicates modest variability across seeds relative to this performance gap.

Q1. Image-patch-only reward and manual evaluation audit

We evaluated a variant in which the auxiliary reward aggregates attention only over image patches, excluding <Focus> from the aggregation.

Attention ScopeMath Avg.General Avg.Perception CoverageFaithful UseAnswer Acc.
Image patches + <Focus>41.649.952.1764.5039.34
Image patches only41.150.751.5065.6138.58

Perception Coverage, Faithful Use, and Answer Accuracy are evaluated on the MathVerse Vision-Only split.

The reported scores show small changes in both directions across task performance and fact-use metrics, with no consistent improvement or degradation when <Focus> is excluded from the reward aggregation.

We conducted a manual audit of 30 questions across Faithful-MR1, VPPO, and Vision-SR1 (90 responses; 273 valid fact–response pairs). Annotators assessed whether task-relevant facts were correctly stated (P) and faithfully used in the derivation (U), and we compared the results with the model-based automatic evaluation. Agreement between the model and manual judgments was 95.6% for P (261/273) and 91.9% for U (251/273).

AuthorsOpenReview ↗

Follow-up on W1/W2 and Q1/Q2

Dear Reviewer knWs,

In our full response, we report the compute-matched comparison across three seeds and the manual audit of fact-use judgments. We’d welcome your thoughts on whether our response addresses your main concerns and whether there is anything we could clarify further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and Area Chair,

Thank you for your careful reviews and constructive feedback. In response to your comments, we have conducted additional experiments and revised the manuscript as follows:

  • Training cost and variability. We quantified the training cost of each stage. On Qwen2.5-VL-3B-Instruct, Faithful-MR1 retained its advantage in mean overall score across random seeds under a matched compute budget. (knWs)

  • Additional backbones. We extended the evaluation to Qwen3-VL-4B-Instruct and Qwen3.5-4B. On each backbone, Faithful-MR1 achieved higher average scores than GRPO in both math reasoning and general VQA. (mNfU)

  • Attention reward and token selection. Absolute visual attention at vision-dependent positions rose alongside the auxiliary reward. Task performance remained similar when the reward counted only image-patch attention. The revised manuscript also explains how the absolute KL threshold filters candidate tokens and why the selected set can be empty. (knWs, mNfU, tCzD)

  • Value beyond region supervision. Using the same training examples and region annotations, Faithful-MR1 outperformed a GRPO baseline that directly supervises attention to the annotated regions, with higher average scores in both math reasoning and general VQA. (mNfU, tCzD)

  • Reliability of fact-use evaluation. A manual audit supported the reliability of the automated evaluation. The Faithful Use advantage persisted when we matched responses by length and, separately, when we compared the same correctly stated facts. (knWs, tCzD)

We hope these additional experiments and clarifications address the main concerns raised in the reviews. We have not yet heard whether any concerns remain and would welcome any further feedback. As the discussion period draws to a close, we would appreciate the reviewers and Area Chair taking this evidence into account in their assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.