acceptodds
Under review as a conference paper at ICLR 2027

When VLMs Rely Less on Their Reasoning: Adaptive KL Regularization for RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown promise for improving reasoning in vision-language models (VLMs). Productive exploration is important for continued performance gains, motivating efforts to relax reference-policy constraints. However, removing reference-policy KL regularization in VLM RLVR can degrade reasoning–answer consistency even when final-answer accuracy remains comparable. Our analyses suggest that RLVR reduces VLMs' reliance on generated reasoning relative to visual inputs during answer prediction. To address this issue, we propose PEAK (Position- and Entropy-Aware KL Anchoring), a token-adaptive KL regularization method that uses the pre-RLVR model as a reference to help preserve reliance on generated reasoning while encouraging productive exploration. PEAK is guided by two observations: (1) reasoning–answer inconsistent responses exhibit larger actor–reference entropy gaps, particularly without KL regularization; so PEAK assigns stronger regularization to tokens with larger entropy gaps; (2) large positive entropy gaps are concentrated in a subset of tokens, while token entropy is elevated at later response positions; PEAK adapts its regularization strength to both token-level entropy gaps and response position. Across seven multimodal reasoning benchmarks, PEAK outperforms the evaluated RLVR methods in average final-answer accuracy. It also improves reasoning–answer consistency and Pass@ over both uniform-KL and no-KL baselines, with gains of 21.04 and 7.90 percentage points in consistency and Pass@, respectively, over standard DAPO (without KL regularization). These joint improvements support more productive exploration while better preserving reasoning behavior.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Models, Reinforcement Learning with Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LbggOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how reference-policy KL regularization affects reasoning behavior during reinforcement learning with verifiable rewards for vision-language models. The authors report that removing KL regularization decreases reasoning–answer consistency and changes sensitivity to interventions on generated reasoning and visual inputs, despite similar final-answer accuracy. They also find an association between inconsistent responses and positive actor–reference entropy gaps, particularly at later response positions. Motivated by these observations, the paper proposes PEAK, which weights the reference-policy KL penalty using a clipped, normalized positive entropy gap and relative token position. The weights are detached during policy optimization. Experiments primarily train Qwen2.5-VL-7B-Instruct on ViRL39K and evaluate seven multimodal benchmark views. PEAK improves average accuracy, judge-assessed consistency, and Pass@k relative to uniform-KL and no-KL baselines. Additional experiments examine GRPO, smaller VLMs, Qwen3-VL-2B-Thinking, and text-only models.

Critical strengths
  1. The controlled comparisons show that similar final-answer accuracy can coexist with substantially different consistency and sampling coverage. For example, the DAPO uniform-KL and no-KL baselines achieve 55.43% and 54.96% accuracy, but their reported consistency rates are 66.40% and 49.17%. Their Pass@64 scores also differ substantially: 84.16% versus 79.94%. This is a useful empirical observation because it shows why answer accuracy and token entropy alone are insufficient to characterize the effects of RLVR. The additional GRPO results strengthen the observation beyond one optimization configuration.

  2. The method requires neither additional rollouts nor an auxiliary reward model, and its weighting rule is easy to integrate into reference-KL training. On the main 7B setting, PEAK achieves 57.06% average accuracy, compared with 55.43% for the default uniform-KL baseline and 55.80% for the best uniform coefficient reported in Appendix C.1.

Critical weaknesses
  1. Appendix B.4 introduces several confounds. In the image-intervention condition, the model receives reasoning generated from the original image, and the authors explicitly acknowledge that verbalized answers may remain in this reasoning. In the reasoning-intervention condition, that reasoning is replaced by a trace from an unrelated question. The comparison therefore changes not only the input modality but also the relevance and potentially answer-bearing content of the supplied text. Stronger sensitivity to unrelated reasoning could reflect susceptibility to distracting text rather than more useful reasoning dependence.

  2. Each model also supplies its own reasoning, so the comparisons combine differences in how models use reasoning with differences in the information contained in their traces. Furthermore, clean accuracy is measured from stored full responses, while interventions regenerate answer suffixes using a forced answer prefix. A matched clean-regeneration control is needed to separate intervention effects from changes in the generation protocol.

  3. In Table 7, removing reasoning reduces accuracy by 8.49 points for the pre-RLVR model and 9.17 points for DAPO without KL. Thus, this measurement does not show a smaller absolute benefit from reasoning after RLVR; the reported pattern primarily concerns its importance relative to the image under this protocol.

  4. Consistency is assessed using one GPT-OSS-120B judge at temperature 0.6, without a repeated-judgment reliability analysis. Missing reasoning or answer content is counted as inconsistent. Since response lengths differ substantially across training conditions, the aggregate score may combine genuine contradictions with missing content, formatting failures, and insufficiently explicit explanations. Figure 8 documents length differences but does not control for them. This matters particularly for PEAK’s incremental consistency gain over uniform KL, which is 3.81 points; much of the 21.04-point improvement over no KL is already achieved by restoring uniform regularization.

Questions for authors
  1. Please include a clean condition that regenerates the answer using exactly the same prefix construction and decoding procedure as the intervention conditions. Can you also remove explicit answer statements from retained reasoning and compare models using a shared set of reasoning traces?

  2. Can PEAK be compared with a uniform coefficient that follows its observed batch-mean effective coefficient over training, and with shuffled weights preserving the weight distribution? Please clarify how coefficients were selected for the entropy-only and position-only ablations, and provide uncertainty estimates for the main comparisons. Was hyperparameter selection performed on a separate validation set?

  3. Equation (6) uses response-wise normalization, whereas Appendix A.1 states token-level aggregation. Which reduction was used for the policy and KL losses in each configuration? Also, Appendix A.4 describes entropy weights computed during the experience phase and then held constant: are they based on the rollout policy or recomputed from the current actor during optimization?

7 comments

We thank the reviewer for their thoughtful evaluation and for recognizing both our empirical findings on consistency and coverage and PEAK’s practical simplicity. We also appreciate the detailed suggestions for additional analyses and ablations, and address each comment below.

W1, W2, Q1(2) Explicit answer statements and shared reasoning traces

Our original protocol captures both 1) what information a model preserves in its own reasoning and 2) how it uses that reasoning to produce the final answer. While both aspects are relevant to the model’s use of textual and visual information, we agree that the original experiment does not isolate reasoning utilization from differences in trace content. We therefore conducted two complementary analyses.

Explicit answer statements in retained reasoning.

First, we measured how frequently each model explicitly states its final answer within the retained reasoning, using an LLM-based detector(GPT6-luna).

ModelReasoning containing explicit answer statements (%)
Pre-RLVR80.6
DAPO + KL73.4
DAPO w/o KL40.1
PEAK (+DAPO)73.5

RLVR-trained models state their final answers less frequently within the reasoning than the pre-RLVR model, with the largest reduction observed without KL regularization. Together with our qualitative observation of no-KL responses that omit an explicit derivation of the final answer, this suggests that the textual reasoning may provide a less complete basis for answering, potentially leaving a greater role for visual information.

Shared reasoning traces with explicit answers removed.

To distinguish these differences in reasoning content from differences in how models use the supplied reasoning, we evaluated all models using a shared set of reasoning traces after removing explicit answer statements. We used a two-stage LLM-based procedure: detecting sentences containing explicit answer statements, then rewriting those sentences to remove the answer-bearing portions while aiming to preserve the supporting reasoning. We collected 200 traces from each of four source models for each of the seven benchmark views, yielding 5,600 shared question–trace pairs. For each pair, all four evaluated models received the same reasoning trace.

ModelClean regenerationMismatched reasoningMismatched imageGap (image − reasoning)
Pre-RLVR38.2817.17 (−21.11)31.49 (−6.79)+14.32
DAPO + KL50.0242.37 (−7.65)33.30 (−16.72)−9.07
DAPO w/o KL50.2144.01 (−6.20)33.57 (−16.64)−10.44
PEAK (+DAPO)50.4242.98 (−7.44)34.13 (−16.29)−8.85

The relative sensitivity pattern persists under these controls. The pre-RLVR model is more sensitive to mismatched reasoning than to mismatched images. All three RLVR-trained models exhibit the opposite ordering. This asymmetry is largest without KL regularization and is somewhat smaller with uniform KL and PEAK.

These results indicate that the observed pattern is not solely attributable to explicit answer statements or differences in the supplied reasoning across models. Nevertheless, mismatched reasoning may still act as distracting text. We therefore interpret these results as evidence of relative sensitivity to the two interventions under this protocol.

W2, Q1(1) Matched clean-regeneration control

To address this concern, we conducted additional experiments with a matched clean-regeneration control and length/format-matched donor reasoning. For the clean control, we removed the original boxed answer and regenerated the answer using the same prefix construction and decoding procedure as in the intervention conditions.

Donor reasoning was drawn from a different question and matched on four criteria: (1) benchmark, (2) token length within ±10% of the original reasoning, (3) answer type—multiple-choice or open-ended—and (4) reasoning format, categorized by rules as numbered steps, bullet points, equation-centered reasoning, or prose paragraphs. All intervention drops below are measured relative to clean-regenerated accuracy.

ModelCleanClean regenMismatched reasoningMismatched image
Pre-RLVR40.9442.4815.50 (−26.98)38.81 (−3.67)
DAPO + KL55.4355.3143.54 (−11.77)42.55 (−12.76)
DAPO w/o KL54.9655.0345.37 (−9.66)40.59 (−14.44)
PEAK (+DAPO)57.0656.6044.36 (−12.24)44.01 (−12.59)

Clean regeneration changes accuracy by less than 0.5 percentage points for all RLVR-trained models and by 1.54 points for the pre-RLVR model. The matched-control experiments reproduce the relative sensitivity pattern reported in the original submission. Specifically, the pre-RLVR model is more sensitive to mismatched reasoning than to a mismatched image. After RLVR, this asymmetry largely disappears or reverses, particularly without KL.

W3 Relative sensitivity in the removal experiments

We agree with the reviewer’s reading of the absolute accuracy drops in Table 7. As stated in the main analysis, our claim concerns sensitivity to generated reasoning relative to visual input under the evaluated intervention protocol. For the pre-RLVR model, removing reasoning reduces accuracy by 8.49 percentage points, compared with 2.73 points for image removal. For DAPO without KL, the respective drops are 9.17 and 12.03 points, reversing the ordering.

We will make this distinction explicit throughout the camera-ready version.

W4(1) Validation of reasoning-answer consistency

To address the concern that our consistency results rely on a single judge, we conducted additional evaluations using three other LLM judges, human annotations, and a complementary answer-continuation protocol.

We first re-evaluated the same responses using three additional judges. As shown below, all four judges assign PEAK higher consistency scores than both uniform KL and no KL. Scores are averaged equally across the seven benchmark views.

ModelGPT-OSS-120B (paper)gpt-5.6-lunagpt-6-lunaSonnet-5.5
Pre-RLVR76.4373.4673.0373.20
DAPO + KL66.4068.3666.3566.01
DAPO w/o KL49.1750.9449.1450.29
PEAK (+DAPO)70.2174.0770.3770.53

We also assessed agreement among the four judges. Across responses rated by all four LLM judges, Fleiss’ κ was 0.78, providing evidence of inter-judge agreement.

ModelFleiss’ κ
Pre-RLVR0.71
DAPO + KL0.81
DAPO w/o KL0.82
PEAK (+DAPO)0.73
All0.78

To assess agreement with human judgments, we randomly sampled 200 responses, with 50 per model. Four human annotators labeled these responses, with each response evaluated by one annotator. Human–judge agreement ranges from 84.5% to 87.0%, with Cohen’s κ ranging from 0.64 to 0.70.

JudgeAgreement with human (%)Cohen’s κ
GPT-OSS-120B87.00.70
gpt-5.6-luna84.50.64
gpt-6-luna85.00.65
Sonnet-5.586.00.68

On this human-annotated sample, consistency rates are 82% for PEAK, 76% for uniform KL, 54% for no KL, and 74% for pre-RLVR. Although the sample is small, the human results are directionally consistent with PEAK’s improvement over both RLVR baselines and provide additional support for our consistency judgments.

As a complementary check, we tested whether the retained reasoning could be used to reproduce the original answer when the image was withheld. We provided the original text prompt and each model’s retained reasoning to Qwen3-8B(text-only LLM) without the image, removed boxed answer spans, and greedily completed the answer from an open \boxed{. We measured semantic agreement between the completed answer and the original VLM answer. The following results are averages across the seven benchmark views.

Source modelOriginal-answer match: boxed spans removed (%)
Pre-RLVR72.97
DAPO + KL71.67
DAPO w/o KL54.36
PEAK (+DAPO)73.02

PEAK achieves a higher original-answer match rate than both uniform KL and no KL. This provides complementary evidence of answer reproducibility from the retained reasoning.

W4(2) Missing content and Reasoning Length

Missing content

We additionally examined whether missing content or differences in response length could account for the consistency results.

First, missing reasoning or final-answer content is rare among the RLVR-trained models:

ModelMissing reasoning (%)Missing final answer (%)Missing either (%)
DAPO + uniform KL0.200.000.20
DAPO w/o KL0.880.000.88
PEAK (+DAPO)0.060.000.06

After excluding questions for which either model has missing content, PEAK’s consistency advantage remains +3.76 percentage points over uniform KL and +20.84 points over no KL, close to the original gains of 3.81 and 21.04 points. Thus, missing content does not account for the reported improvements.

Reasoning length

Second, we compared the original GPT-OSS-120B consistency judgments for responses to the same questions, restricting each response pair to the same reasoning-length range or to lengths differing by at most 10%. Length is measured in tokens.

PEAK vs no KLPEAK (%)No KL (%)Difference
Both <150 tokens82.672.2+10.4
Both 150–250 tokens77.761.7+16.0
Both 250–400 tokens67.756.8+10.9
Both ≥400 tokens58.233.0+25.3
Lengths within 10%79.066.8+12.2

PEAK maintains a positive consistency advantage over no KL in every reported length range. These results indicate that no-KL’s lower consistency is not solely explained by its shorter responses.

PEAK vs uniform KLPEAK (%)Uniform KL (%)Difference
Both <150 tokens89.191.0−1.8
Both 150–250 tokens79.778.5+1.2
Both 250–400 tokens67.962.3+5.7
Both ≥400 tokens45.044.2+0.9
Lengths within 10%68.366.6+1.7

Relative to uniform KL, the differences are vary across length ranges. PEAK generally scores higher than uniform KL across length ranges, with the largest observed advantage in the 250–400-token range.

Q2(1) KL Coefficient schedule, Shuffled weights, and Ablation settings

KL coefficient schedule and Shuffled weights

We thank the reviewer for suggesting these additional controls. We conducted two experiments to distinguish PEAK’s token-specific weighting from the overall regularization schedule and weight distribution.

First, we trained a uniform-KL baseline whose coefficient at each training step follows the observed batch-mean effective KL coefficient from PEAK. This tests whether PEAK’s performance can be explained by the evolution of its average regularization strength alone.

Second, we randomly shuffled the full PEAK weights wi,t=ei,tpi,tw_{i,t}=e_{i,t}p_{i,t} among tokens within each response. This preserves the weight distribution and mean within each response while disrupting the assignment of weights to specific tokens.

We additionally tested inverted entropy factors, replacing ei,te_{i,t} with 1−ei,t1-e_{i,t} while retaining the positional factor. This reverses the entropy-based weighting preference, assigning stronger regularization to tokens with smaller entropy excess at a given position.

MethodAverage accuracy (%)Decrease from PEAK (pp)
PEAK57.06—
Uniform KL following PEAK’s batch-mean coefficient54.872.19
Shuffled PEAK weights within each response55.491.57
Inverted entropy factors (1−ei,t)(1-e_{i,t})55.052.01
DAPO + uniform KL55.431.63

PEAK outperforms both the batch-mean schedule and shuffled-weight controls. These results support the importance of assigning regularization weights according to token-specific entropy and position, beyond the average coefficient schedule or weight distribution alone.

Inverting the entropy factors also reduces accuracy. Together with the entropy-decrease ablation in Table 3, this provides additional support for PEAK’s preference for regularizing positive entropy excess.

Ablataion settings

For the entropy-only and position-only ablations, we retained PEAK’s training settings, including the base KL coefficient β=0.05\beta=0.05. Starting from wi,t=ei,tpi,tw_{i,t}=e_{i,t}p_{i,t}, the entropy-only ablation sets pi,t=1p_{i,t}=1, and the position-only ablation sets ei,t=1e_{i,t}=1.

We will clarify this protocol and include the additional results in the camera-ready version.

Reviewer_fUjyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors observe that removing reference-policy KL regularization substantially increases reasoning–answer inconsistency despite similar final-answer accuracy. They associate inconsistent responses with larger actor–reference entropy gaps and higher entropy at later response positions, then propose PEAK, which scales token-level KL regularization by the clipped, normalized positive entropy gap and relative token position.

Critical strengths
  1. Systematic diagnostic analysis: Figures 2 and 9–11 associate reasoning–answer inconsistency with larger actor–reference entropy gaps and elevated late-response entropy. Figure 9 additionally separates consistency from final-answer correctness. Figures 10 and 11 further show that PEAK reduces positive entropy excess and late-response entropy in both consistent and inconsistent responses.

2.The objective is straightforward and generalizes across optimization backbones and models, including accuracy improvements on Qwen3-VL-Thinking.

Critical weaknesses

Biggest concerns:

  1. Entropy-based token selection is not appear to be directly validated. Figures 10–11 establish associations but do not locate the first inconsistent step or determine whether entropy spikes precede or follow it. Controlled step perturbations and shuffling the entropy factors �0��0� among tokens within the same response would strengthen the justification for the weighting rule (Equation 7; Table 3).

  2. There are no training-seed variability or confidence intervals are reported.

  3. Appendix E already acknowledges incomplete recovery and proposes jointly controlling entropy growth/collapse and refining positional weighting. However, it does not establish which residual errors explain the consistency and Pass@64 gaps.

  4. I would be interested in seeing whether PEAK’s regularization can complement, for example, PAPO-D’s visual grounding. Given PEAK’s modest accuracy advantage over the evaluated PAPO-D checkpoint, could the authors discuss whether the methods address complementary failures and whether combining them improves both grounding and reasoning–answer consistency (Table 2; Figure 7)?

  5. Authors should include discussion about why they believe there is still a gap between PEAK and baseline and what future work could do to close the gap could do.

Smallest concerns: Almost all of the important comparisons and stratified diagnostics results are in the appendix. Please add the consistent and inconsistent (Fig 7), table 7, replace figure 2b and 2c with figure 10 and 11 to show readers clearly that there is a gap between entropy in consistent and inconsistent responses. Otherwise, readers can attribute higher entropy to other factors like generally degraded continuation.

Questions for authors

1.There could be a more in depth exploration of the types of inconsistency in the reasoning chains and how PEAK affects them. Are issues in consistency in responses predominantly only reasoning-answer inconsistency? Are there also inconsistencies in the reasoning itself? Is there any situation where the model answers correctly but the reasoning is wrong? How do the entropy gap differ in these situation? Does the position in which the inconsistent information exist in the reasoning step affect the model's final answer and how is the entropy gap affected by the different placements? You have an ablation where you replace the reasoning with a "mismatched reasoning," but the placement of inconsistency in the response can affect final answer. 2. Does the position of inconsistent information affect the final answer, and how does its placement affect the entropy gap? The mismatched-reasoning experiment replaces the trace with reasoning from another question; introducing controlled inconsistencies at different steps would help test the positional rationale. 3. Can PEAK improve agreement by making the final answer follow incorrect reasoning? How do the proportions of consistent/correct, consistent/incorrect, inconsistent/correct, and inconsistent/incorrect responses change under PEAK? 4. Since Equation 7 assigns zero anchoring weight when actor entropy does not exceed reference entropy, can anchoring impede useful corrections when the reference policy is wrong? 4. Are there any situations where PEAK can collapse and cause hacking? For example, the model focuses more on consistency at the cost of correctness or ignores?

8 comments

We thank the reviewer for their insightful feedback and for recognizing the value of our systematic diagnostic analysis as well as the simplicity and generalizability of our objective. We address the reviewer's comments and questions in detail below.

W1(1) Entropy around the first identified inconsistency

We agree that our original analysis did not establish how entropy gaps evolve around the first inconsistent step. To address this concern, we conducted a finer-grained analysis using GPT-6-astra with high reasoning effort to inspect 200 responses per model that were originally labeled inconsistent. The judge model identified the earliest step containing a contradiction with earlier reasoning or the task text, an invalid calculation or inference, or a mismatch with the final answer.

We then measured the actor–reference entropy gap before, at, and after the identified step. We compared these measurements with steps from consistent responses generated by the same model on the same benchmark and matched by relative-position.

DAPO w/o KL

GroupEarlier stepsPreceding stepFocal stepFollowing stepLater steps
First-inconsistency alignment0.0840.4110.5531.3312.292
Matched consistent-response controls0.0710.2310.3750.5410.747
Paired difference+0.002+0.172+0.178+0.830+1.542

For DAPO without KL, the mean actor–reference entropy gap is already larger than in the matched controls at the step immediately preceding the first identified inconsistency, with the largest separation occurring after that step. DAPO with uniform KL and PEAK show a similar pattern.

These results extend the response-level association to the neighborhood of the first identified inconsistency. The analysis provides finer-grained evidence for PEAK’s weighting motivation.

W1(2) Entropy weighting ablation

To further examine the weighting rule, we trained an ablation that replaces the entropy-excess factor ei,te_{i,t} with 1−ei,t1-e_{i,t}, while retaining the positional factor. This reverses the weighting preference, assigning stronger regularization to tokens with smaller entropy excess.

MethodAverage accuracy (%)Decrease from PEAK (pp)
PEAK57.06—
Inverted entropy factors (1−ei,t1-e_{i,t})55.052.01
DAPO + uniform KL55.431.63

Inverting the entropy factors reduces average accuracy by 2.01 percentage points. This result complements the entropy-decrease ablation in Table 3, which assigns positive entropy-based weights only when actor entropy is below reference entropy. Together, these ablations support the preference for stronger regularization at larger positive entropy gaps.

W2 Training-seed variability

We thank the reviewer for highlighting the need to quantify training-seed variability. The original results used one training seed. We have now expanded the evaluation to three independent training seeds per method, comparing PEAK with KL-Cov and the strongest uniform-KL setting from the Appendix C.1 sweep, β=0.005\beta=0.005.

The table below reports the mean and standard deviation across training seeds. For each seed, we first average the metric equally across the seven benchmark views.

MethodAvg@8 (%)Pass@8 (%)Consistency (%)
KL-Cov56.30 ± 0.5374.16 ± 0.8660.55 ± 0.45
Uniform KL, β=0.00555.61 ± 0.1972.99 ± 0.9360.93 ± 1.59
PEAK56.82 ± 0.2176.40 ± 0.9968.97 ± 1.10

PEAK achieves higher mean accuracy than KL-Cov and the tuned uniform-KL baseline by 0.52 and 1.21 percentage points. PEAK’s benefits also extend beyond average accuracy. It improves Pass@8 by 2.24 points over KL-Cov and 3.41 points over uniform KL, and consistency by 8.42 and 8.04 points, respectively. These results support our contribution of improving reasoning–answer consistency and solution coverage while achieving accuracy gains.

We will incorporate these results and the corresponding per-benchmark statistics into the camera-ready version, explicitly reporting the number of training seeds and the uncertainty measure.

W3, W5 Remaining gaps in consistency and solution coverage

Prior work has reported that RLVR can improve sampling efficiency while reducing solution coverage at larger kk, consistent with shifting probability toward successful trajectories already available to the pretrained model [1]. In our VLM experiments, all four RLVR baselines fall below the pre-RLVR model in Pass@kk by k=16k=16 (Figure 5). This motivates the hypothesis that multimodal answer shortcuts may exacerbate the concentration of successful behavior in our setting.

Specifically, a VLM can use visual information to produce a correct final answer even when its generated reasoning does not substantively support that answer. Since the reward evaluates only final-answer correctness, it can reinforce such responses without requiring a stronger connection between reasoning and the answer. RLVR may therefore favor readily rewarded visual answering strategies without preserving alternative reasoning-supported solution paths. This offers a possible connection between the reduced reasoning–answer consistency and narrower solution coverage observed in our experiments.

Future work could pursue two directions. First, a finer-grained analysis of when and how visual information enables correct answers despite inconsistent reasoning could inform more targeted prevention of shortcut-based answering. Second, moving beyond a fixed reference could help preserve useful reasoning without retaining the reference’s limitations. This could involve incorporating verified improvements from the evolving actor or selectively relaxing anchoring when the reference is unreliable.

[1] Yue et al. “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”. NeurIPS 2025

W4 Combining PEAK with PAPO-D

We thank the reviewer for this suggestion. PAPO-D and PEAK target different aspects of multimodal reasoning. PAPO-D encourages visual grounding, whereas PEAK uses adaptive reference regularization to mitigate the loss of reasoning–answer consistency during RLVR. Their objectives therefore offer a plausible basis for complementarity.

To investigate this, we applied PEAK’s adaptive KL regularization to the PAPO-D training setup. We retained the PAPO-D settings and used the same base KL coefficient as PEAK, β=0.05\beta=0.05, without additional hyperparameter tuning. The results averaged across the seven benchmark views are shown below

MethodAvg@8 accuracy (%)Consistency (%)
PAPO-D56.6559.16
PEAK (+DAPO)57.0670.21
PAPO-D + PEAK57.8370.17

The combined model exceeds the evaluated PAPO-D checkpoint by 1.18 percentage points in accuracy and 11.01 points in consistency. It also achieves 0.77 points higher accuracy than PEAK, while retaining a similar consistency score.

These results support the compatibility of PEAK’s regularization with PAPO-D’s visual-grounding objective. We will include these results and the discussion of complementarity in the camera-ready version.

Presentation

We thank the reviewer for this helpful suggestion. We originally placed these analyses in the appendix to cover a broader range of experiments, and we will move the key results into the main text as suggested in the camera-ready version.

Q1(1), Q3 Reasoning-answer consistency and final answer correctness

We jointly analyzed consistency and final-answer correctness for the same responses across the seven benchmark views.

Among responses with correct final answers, PEAK has a higher consistency rate than both uniform KL and no KL:

ModelConsistent given correctInconsistent given correct
Pre-RLVR83.06%16.94%
No KL63.71%36.29%
Uniform KL78.40%21.60%
PEAK79.42%20.58%

PEAK also increases reasoning–answer agreement among responses with incorrect final answers:

ModelConsistent given incorrectInconsistent given incorrect
Pre-RLVR71.86%28.14%
No KL29.34%70.66%
Uniform KL50.18%49.82%
PEAK56.09%43.91%

Thus, PEAK’s consistency improvement occurs in both correctness groups.

For completeness, the four joint proportions among all responses are:

ModelConsistent / correctConsistent / incorrectInconsistent / correctInconsistent / incorrect
Pre-RLVR34.45%41.98%6.20%17.37%
No KL35.70%13.47%19.40%31.43%
Uniform KL43.51%22.89%12.14%21.47%
PEAK45.34%24.86%11.68%18.12%

Relative to uniform KL, PEAK increases the consistent/correct fraction by 1.83 percentage points and the consistent/incorrect fraction by 1.98 points.

We further extend the analysis in Appendix B.5 to PEAK using the same response-level actor–reference entropy gap:

ModelConsistent / correctConsistent / incorrectInconsistent / correctInconsistent / incorrect
No KL0.21930.36090.61940.6585
Uniform KL0.05770.08790.12600.1322
PEAK0.01850.04450.05320.0629

The same association holds for PEAK: within both correct and incorrect responses, inconsistent responses exhibit larger mean actor–reference entropy gaps than consistent responses. Meanwhile, PEAK exhibits smaller mean gaps than uniform KL in all four categories.

Q1(2), Q2 Position of the first identified inconsistency

We conducted an additional analysis of the position of the first inconsistent step, using the GPT-6-astra annotations described in our response to W1.

The distribution of first-inconsistency positions is shown below. Percentages are calculated among responses with a localized inconsistent step

Model0–20%20–40%40–60%60–80%80–100%
Pre-RLVR6.522.023.823.224.4
DAPO + KL7.018.826.921.525.8
DAPO w/o KL3.111.516.731.237.5
PEAK4.718.318.323.734.9

First inconsistencies are concentrated toward the later portions of the reasoning.

We then measured the mean actor–reference entropy gap at the first inconsistent step, grouped by its relative position.

Model0–20%20–40%40–60%60–80%80–100%
DAPO + KL0.0480.0680.0930.1120.252
DAPO w/o KL0.1890.0730.1060.2871.148
PEAK0.0360.0160.0450.0490.129

Across all three trained models, the larger gaps associated with late inconsistencies provide additional motivation for stronger regularization at later positions.

Finally, we examined final-answer accuracy within the position groups.

Model20–40%40–60%60–80%80–100%
Pre-RLVR29.722.515.441.5
DAPO + KL42.942.040.037.5
DAPO w/o KL27.350.043.336.1
PEAK38.732.337.539.0

We do not observe a consistent trend in final-answer accuracy across positions. These groups contain different questions and naturally occurring inconsistencies, so this comparison does not isolate the effect of moving the same inconsistency to a different position.

Q4 Potential failure modes of PEAK

PEAK’s entropy-based weight is positive only when the current actor has higher entropy than the reference on the same prefix. The reviewer’s concern can arise when the actor favors a more appropriate continuation than the reference but also has higher entropy.

As shown in Figure 9 of Appendix B.5, consistent responses exhibit smaller mean actor–reference entropy gaps than inconsistent responses within both correctness groups. Likewise, correct responses exhibit smaller mean gaps than incorrect responses within both consistency groups. These observations support using entropy excess to allocate anchoring.

We additionally evaluated model performance on questions where the sampled pre-RLVR response was incorrect:

ModelAccuracy on questions with an incorrect pre-RLVR response (%)
No KL39.89
Uniform KL40.97
PEAK42.02

These conditional accuracies are computed within each benchmark and averaged across the seven benchmark views. PEAK achieves the highest observed accuracy among the three methods, demonstrating that it can overcome the reference’s sampled errors despite anchoring. These results complement the entropy analysis, while not excluding the possibility that anchoring impedes some individual corrections.

We have not observed training collapse with PEAK in the runs examined so far.

PEAK does not introduce a consistency-based reward. The training reward remains final-answer correctness, so it does not directly incentivize optimizing the consistency judge’s score.

Empirically, we do not observe an aggregate trade-off between consistency and correctness. Compared with uniform KL, PEAK increases accuracy from 55.65% to 57.02% and consistency from 66.40% to 70.21% on the same evaluated responses. As reported in our response to Q1/Q3, consistency also increases within both correct and incorrect responses.


We hope these additional results and clarifications address the reviewer’s concerns. We would be happy to clarify any remaining points.

Reviewer_v52VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how reference-policy KL regularization affects reasoning behavior in vision-language model RLVR. The authors find that removing KL can increase token entropy and reduce reasoning–answer consistency despite comparable final-answer accuracy. Intervention experiments suggest a shift in answer prediction toward visual inputs relative to generated reasoning. Motivated by these observations, they propose PEAK, which weights reference-policy KL by positive normalized actor–reference entropy gaps and relative response position, with the weights treated as constants during optimization. On seven multimodal benchmark views, PEAK achieves 57.06% Avg@8 accuracy, versus 55.43% for the default uniform-KL baseline, 54.96% without KL, and 56.56% for KL-Cov. It also improves the reported consistency and Pass@k metrics over the uniform-KL and no-KL baselines. Additional experiments examine transfer across model sizes, RL backbones, and text-only models.

Critical strengths
  1. The paper addresses a relevant gap between answer accuracy and reasoning–answer consistency. The paired KL/no-KL comparisons and complementary accuracy, Pass@k, and consistency evaluations provide useful empirical observations.
  2. PEAK is simple to integrate and requires no additional rollouts, auxiliary reward models, or perception annotations. Component ablations and experiments across model sizes, RL backbones, and modalities support its practical applicability.
Critical weaknesses
  1. Reasoning reliance is not convincingly isolated. Different clean/intervention generation procedures and unmatched donor reasoning complicate the intervention results (Appendix B.4). The consistency analysis also relies on one GPT-OSS-120B judge without human or independent-judge calibration, leaving the proposed explanation insufficiently validated.

  2. The accuracy gains lack robustness evidence. PEAK gains only 0.50 percentage points over KL-Cov and 1.26 over the best uniform-KL setting reported in Appendix C.1. Independent training-seed counts and uncertainty estimates are not reported, and the hyperparameter-selection protocol remains unclear.

Questions for authors
  1. Can the reliance trend be reproduced with a matched clean-regeneration control and length/format-matched donors, and can a small human-annotated or independently judged sample corroborate the consistency gain?
  2. How many independent training seeds were used, how uncertain are the gains over KL-Cov and uniform KL, and which data and criteria determined the reported hyperparameters?
3 comments

We thank the reviewer for their careful reading and for suggesting additional experiments to further validate our claims. Below, we present the results of the additional experiments and analyses.

W1, Q1(1) Reasoning reliance under matched controls

To address this concern, we conducted additional experiments with a matched clean-regeneration control and length/format-matched donor reasoning. For the clean control, we removed the original boxed answer and regenerated the answer using the same prefix construction and decoding procedure as in the intervention conditions.

Donor reasoning was drawn from a different question and matched on four criteria: (1) benchmark, (2) token length within ±10% of the original reasoning, (3) answer type (multiple-choice or open-ended) and (4) reasoning format, categorized by rules as numbered steps, bullet points, equation-centered reasoning, or prose paragraphs. All intervention drops below are measured relative to clean-regenerated accuracy.

ModelCleanClean regenMismatched reasoningMismatched image
Pre-RLVR40.9442.4815.50 (−26.98)38.81 (−3.67)
DAPO + KL55.4355.3143.54 (−11.77)42.55 (−12.76)
DAPO w/o KL54.9655.0345.37 (−9.66)40.59 (−14.44)
PEAK (+DAPO)57.0656.6044.36 (−12.24)44.01 (−12.59)

Clean regeneration changes accuracy by less than 0.5 percentage points for all RLVR-trained models and by 1.54 points for the pre-RLVR model. The matched-control experiments reproduce the relative sensitivity pattern reported in the original submission. Specifically, the pre-RLVR model is more sensitive to mismatched reasoning than to a mismatched image. After RLVR, this asymmetry largely disappears or reverses, particularly without KL

W1, Q1(2) Validation of reasoning-answer consistency

To address the concern that our consistency results rely on a single judge, we conducted additional evaluations using three other LLM judges, human annotations, and a complementary answer-continuation protocol.

We first re-evaluated the same responses using three additional judges. As shown below, all four judges assign PEAK higher consistency scores than both uniform KL and no KL. Scores are averaged equally across the seven benchmark views.

ModelGPT-OSS-120B (paper)gpt-5.6-lunagpt-6-lunaSonnet-5.5
Pre-RLVR76.4373.4673.0373.20
DAPO + KL66.4068.3666.3566.01
DAPO w/o KL49.1750.9449.1450.29
PEAK (+DAPO)70.2174.0770.3770.53

We also assessed agreement among the four judges. Across responses rated by all four LLM judges, Fleiss’ κ was 0.78, providing evidence of inter-judge agreement.

ModelFleiss’ κ
Pre-RLVR0.71
DAPO + KL0.81
DAPO w/o KL0.82
PEAK (+DAPO)0.73
All0.78

To assess agreement with human judgments, we randomly sampled 200 responses, with 50 per model. Four human annotators labeled these responses, with each response evaluated by one annotator. Human–judge agreement ranges from 84.5% to 87.0%, with Cohen’s κ ranging from 0.64 to 0.70.

JudgeAgreement with human (%)Cohen’s κ
GPT-OSS-120B87.00.70
gpt-5.6-luna84.50.64
gpt-6-luna85.00.65
Sonnet-5.586.00.68

On this human-annotated sample, consistency rates are 82% for PEAK, 76% for uniform KL, 54% for no KL, and 74% for pre-RLVR. Although the sample is small, the human results are directionally consistent with PEAK’s improvement over both RLVR baselines and provide additional support for our consistency judgments.

As a complementary check, we tested whether the retained reasoning could be used to reproduce the original answer when the image was withheld. We provided the original text prompt and each model’s retained reasoning to Qwen3-8B(text-only LLM) without the image, removed boxed answer spans, and greedily completed the answer from an open \boxed{. We measured semantic agreement between the completed answer and the original VLM answer. The following results are averages across the seven benchmark views.

Source modelOriginal-answer match: boxed spans removed (%)
Pre-RLVR72.97
DAPO + KL71.67
DAPO w/o KL54.36
PEAK (+DAPO)73.02

PEAK achieves a higher original-answer match rate than both uniform KL and no KL. This provides complementary evidence of answer reproducibility from the retained reasoning.

W2, Q2 Training-seed variability and hyperparameter selection

Training-seed variability We thank the reviewer for highlighting the need to quantify training-seed variability. The original results used one training seed. We have now expanded the evaluation to three independent training seeds per method, comparing PEAK with KL-Cov and the strongest uniform-KL setting from the Appendix C.1 sweep, β=0.005\beta=0.005.

The table below reports the mean and standard deviation across training seeds. For each seed, we first average the metric equally across the seven benchmark views.

MethodAvg@8 (%)Pass@8 (%)Consistency (%)
KL-Cov56.30 ± 0.5374.16 ± 0.8660.55 ± 0.45
Uniform KL, β=0.00555.61 ± 0.1972.99 ± 0.9360.93 ± 1.59
PEAK56.82 ± 0.2176.40 ± 0.9968.97 ± 1.10

PEAK achieves higher mean accuracy than KL-Cov and the tuned uniform-KL baseline by 0.52 and 1.21 percentage points. PEAK’s benefits also extend beyond average accuracy. It improves Pass@8 by 2.24 points over KL-Cov and 3.41 points over uniform KL, and consistency by 8.42 and 8.04 points, respectively. These results support our contribution of improving reasoning–answer consistency and solution coverage while achieving accuracy gains.

We will incorporate these results and the corresponding per-benchmark statistics into the camera-ready version, explicitly reporting the number of training seeds and the uncertainty measure.

Hyperparameter selection Following PAPO [1], we used MMK12 [2] as the validation dataset during training. We selected β=0.05\beta=0.05 based on MMK12 accuracy and the scale of the mean effective KL coefficient. Specifically, because PEAK’s adaptive weights attenuate the token-level KL coefficient, we increased the base coefficient to compensate for this attenuation and maintain a scale comparable to the default uniform-KL baseline (Figure 12 in Appendix C.1). We used the final checkpoint after a fixed two epochs, without validation-based checkpoint selection. We thank the reviewer for pointing out the need for clarification. We will explicitly describe this hyperparameter-selection protocol in the camera-ready version

[1] Wang et al. Perception-Aware Policy Optimization for Multimodal Reasoning. ICLR 2026

[2] Meng et al. MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning. arXiv 2025


We hope these additional results and clarifications address the reviewer’s concerns. We would be happy to clarify any remaining points.

Dear Reviewer v52V,

With the discussion deadline approaching, we would like to briefly summarize our responses and additional experiments addressing your main concerns.

  • Isolating reasoning reliance: We repeated the interventions with matched clean regeneration and length/format-matched donor reasoning. The original relative sensitivity pattern persists under these controls.
  • Validating reasoning–answer consistency: Three additional LLM judges, human annotations on 200 responses, and an answer-continuation evaluation all support PEAK’s consistency improvement over uniform KL and no KL.
  • Accuracy robustness and hyperparameter selection: We expanded the comparison to three independent training seeds per method and reported means and standard deviations. PEAK retains higher mean accuracy than KL-Cov and the strongest reported uniform-KL setting, alongside larger improvements in Pass@8 and consistency. We also clarified the hyperparameter selection.

We would be grateful to hear whether these additions address your concerns and whether any points need further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.