- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how reference-policy KL regularization affects reasoning behavior during reinforcement learning with verifiable rewards for vision-language models. The authors report that removing KL regularization decreases reasoning–answer consistency and changes sensitivity to interventions on generated reasoning and visual inputs, despite similar final-answer accuracy. They also find an association between inconsistent responses and positive actor–reference entropy gaps, particularly at later response positions. Motivated by these observations, the paper proposes PEAK, which weights the reference-policy KL penalty using a clipped, normalized positive entropy gap and relative token position. The weights are detached during policy optimization. Experiments primarily train Qwen2.5-VL-7B-Instruct on ViRL39K and evaluate seven multimodal benchmark views. PEAK improves average accuracy, judge-assessed consistency, and Pass@k relative to uniform-KL and no-KL baselines. Additional experiments examine GRPO, smaller VLMs, Qwen3-VL-2B-Thinking, and text-only models.
Critical strengths
-
The controlled comparisons show that similar final-answer accuracy can coexist with substantially different consistency and sampling coverage. For example, the DAPO uniform-KL and no-KL baselines achieve 55.43% and 54.96% accuracy, but their reported consistency rates are 66.40% and 49.17%. Their Pass@64 scores also differ substantially: 84.16% versus 79.94%. This is a useful empirical observation because it shows why answer accuracy and token entropy alone are insufficient to characterize the effects of RLVR. The additional GRPO results strengthen the observation beyond one optimization configuration.
-
The method requires neither additional rollouts nor an auxiliary reward model, and its weighting rule is easy to integrate into reference-KL training. On the main 7B setting, PEAK achieves 57.06% average accuracy, compared with 55.43% for the default uniform-KL baseline and 55.80% for the best uniform coefficient reported in Appendix C.1.
Critical weaknesses
-
Appendix B.4 introduces several confounds. In the image-intervention condition, the model receives reasoning generated from the original image, and the authors explicitly acknowledge that verbalized answers may remain in this reasoning. In the reasoning-intervention condition, that reasoning is replaced by a trace from an unrelated question. The comparison therefore changes not only the input modality but also the relevance and potentially answer-bearing content of the supplied text. Stronger sensitivity to unrelated reasoning could reflect susceptibility to distracting text rather than more useful reasoning dependence.
-
Each model also supplies its own reasoning, so the comparisons combine differences in how models use reasoning with differences in the information contained in their traces. Furthermore, clean accuracy is measured from stored full responses, while interventions regenerate answer suffixes using a forced answer prefix. A matched clean-regeneration control is needed to separate intervention effects from changes in the generation protocol.
-
In Table 7, removing reasoning reduces accuracy by 8.49 points for the pre-RLVR model and 9.17 points for DAPO without KL. Thus, this measurement does not show a smaller absolute benefit from reasoning after RLVR; the reported pattern primarily concerns its importance relative to the image under this protocol.
-
Consistency is assessed using one GPT-OSS-120B judge at temperature 0.6, without a repeated-judgment reliability analysis. Missing reasoning or answer content is counted as inconsistent. Since response lengths differ substantially across training conditions, the aggregate score may combine genuine contradictions with missing content, formatting failures, and insufficiently explicit explanations. Figure 8 documents length differences but does not control for them. This matters particularly for PEAK’s incremental consistency gain over uniform KL, which is 3.81 points; much of the 21.04-point improvement over no KL is already achieved by restoring uniform regularization.
Questions for authors
-
Please include a clean condition that regenerates the answer using exactly the same prefix construction and decoding procedure as the intervention conditions. Can you also remove explicit answer statements from retained reasoning and compare models using a shared set of reasoning traces?
-
Can PEAK be compared with a uniform coefficient that follows its observed batch-mean effective coefficient over training, and with shuffled weights preserving the weight distribution? Please clarify how coefficients were selected for the entropy-only and position-only ablations, and provide uncertainty estimates for the main comparisons. Was hyperparameter selection performed on a separate validation set?
-
Equation (6) uses response-wise normalization, whereas Appendix A.1 states token-level aggregation. Which reduction was used for the policy and KL losses in each configuration? Also, Appendix A.4 describes entropy weights computed during the experience phase and then held constant: are they based on the rollout policy or recomputed from the current actor during optimization?
7 comments
W3 Relative sensitivity in the removal experiments
We agree with the reviewer’s reading of the absolute accuracy drops in Table 7. As stated in the main analysis, our claim concerns sensitivity to generated reasoning relative to visual input under the evaluated intervention protocol. For the pre-RLVR model, removing reasoning reduces accuracy by 8.49 percentage points, compared with 2.73 points for image removal. For DAPO without KL, the respective drops are 9.17 and 12.03 points, reversing the ordering.
We will make this distinction explicit throughout the camera-ready version.
W4(1) Validation of reasoning-answer consistency
To address the concern that our consistency results rely on a single judge, we conducted additional evaluations using three other LLM judges, human annotations, and a complementary answer-continuation protocol.
We first re-evaluated the same responses using three additional judges. As shown below, all four judges assign PEAK higher consistency scores than both uniform KL and no KL. Scores are averaged equally across the seven benchmark views.
| Model | GPT-OSS-120B (paper) | gpt-5.6-luna | gpt-6-luna | Sonnet-5.5 |
|---|---|---|---|---|
| Pre-RLVR | 76.43 | 73.46 | 73.03 | 73.20 |
| DAPO + KL | 66.40 | 68.36 | 66.35 | 66.01 |
| DAPO w/o KL | 49.17 | 50.94 | 49.14 | 50.29 |
| PEAK (+DAPO) | 70.21 | 74.07 | 70.37 | 70.53 |
We also assessed agreement among the four judges. Across responses rated by all four LLM judges, Fleiss’ κ was 0.78, providing evidence of inter-judge agreement.
| Model | Fleiss’ κ |
|---|---|
| Pre-RLVR | 0.71 |
| DAPO + KL | 0.81 |
| DAPO w/o KL | 0.82 |
| PEAK (+DAPO) | 0.73 |
| All | 0.78 |
To assess agreement with human judgments, we randomly sampled 200 responses, with 50 per model. Four human annotators labeled these responses, with each response evaluated by one annotator. Human–judge agreement ranges from 84.5% to 87.0%, with Cohen’s κ ranging from 0.64 to 0.70.
| Judge | Agreement with human (%) | Cohen’s κ |
|---|---|---|
| GPT-OSS-120B | 87.0 | 0.70 |
| gpt-5.6-luna | 84.5 | 0.64 |
| gpt-6-luna | 85.0 | 0.65 |
| Sonnet-5.5 | 86.0 | 0.68 |
On this human-annotated sample, consistency rates are 82% for PEAK, 76% for uniform KL, 54% for no KL, and 74% for pre-RLVR. Although the sample is small, the human results are directionally consistent with PEAK’s improvement over both RLVR baselines and provide additional support for our consistency judgments.
As a complementary check, we tested whether the retained reasoning could be used to reproduce the original answer when the image was withheld. We provided the original text prompt and each model’s retained reasoning to Qwen3-8B(text-only LLM) without the image, removed boxed answer spans, and greedily completed the answer from an open \boxed{. We measured semantic agreement between the completed answer and the original VLM answer. The following results are averages across the seven benchmark views.
| Source model | Original-answer match: boxed spans removed (%) |
|---|---|
| Pre-RLVR | 72.97 |
| DAPO + KL | 71.67 |
| DAPO w/o KL | 54.36 |
| PEAK (+DAPO) | 73.02 |
PEAK achieves a higher original-answer match rate than both uniform KL and no KL. This provides complementary evidence of answer reproducibility from the retained reasoning.
W4(2) Missing content and Reasoning Length
Missing content
We additionally examined whether missing content or differences in response length could account for the consistency results.
First, missing reasoning or final-answer content is rare among the RLVR-trained models:
| Model | Missing reasoning (%) | Missing final answer (%) | Missing either (%) |
|---|---|---|---|
| DAPO + uniform KL | 0.20 | 0.00 | 0.20 |
| DAPO w/o KL | 0.88 | 0.00 | 0.88 |
| PEAK (+DAPO) | 0.06 | 0.00 | 0.06 |
After excluding questions for which either model has missing content, PEAK’s consistency advantage remains +3.76 percentage points over uniform KL and +20.84 points over no KL, close to the original gains of 3.81 and 21.04 points. Thus, missing content does not account for the reported improvements.
Reasoning length
Second, we compared the original GPT-OSS-120B consistency judgments for responses to the same questions, restricting each response pair to the same reasoning-length range or to lengths differing by at most 10%. Length is measured in tokens.
| PEAK vs no KL | PEAK (%) | No KL (%) | Difference |
|---|---|---|---|
| Both <150 tokens | 82.6 | 72.2 | +10.4 |
| Both 150–250 tokens | 77.7 | 61.7 | +16.0 |
| Both 250–400 tokens | 67.7 | 56.8 | +10.9 |
| Both ≥400 tokens | 58.2 | 33.0 | +25.3 |
| Lengths within 10% | 79.0 | 66.8 | +12.2 |
PEAK maintains a positive consistency advantage over no KL in every reported length range. These results indicate that no-KL’s lower consistency is not solely explained by its shorter responses.
| PEAK vs uniform KL | PEAK (%) | Uniform KL (%) | Difference |
|---|---|---|---|
| Both <150 tokens | 89.1 | 91.0 | −1.8 |
| Both 150–250 tokens | 79.7 | 78.5 | +1.2 |
| Both 250–400 tokens | 67.9 | 62.3 | +5.7 |
| Both ≥400 tokens | 45.0 | 44.2 | +0.9 |
| Lengths within 10% | 68.3 | 66.6 | +1.7 |
Relative to uniform KL, the differences are vary across length ranges. PEAK generally scores higher than uniform KL across length ranges, with the largest observed advantage in the 250–400-token range.
Q2(1) KL Coefficient schedule, Shuffled weights, and Ablation settings
KL coefficient schedule and Shuffled weights
We thank the reviewer for suggesting these additional controls. We conducted two experiments to distinguish PEAK’s token-specific weighting from the overall regularization schedule and weight distribution.
First, we trained a uniform-KL baseline whose coefficient at each training step follows the observed batch-mean effective KL coefficient from PEAK. This tests whether PEAK’s performance can be explained by the evolution of its average regularization strength alone.
Second, we randomly shuffled the full PEAK weights among tokens within each response. This preserves the weight distribution and mean within each response while disrupting the assignment of weights to specific tokens.
We additionally tested inverted entropy factors, replacing with while retaining the positional factor. This reverses the entropy-based weighting preference, assigning stronger regularization to tokens with smaller entropy excess at a given position.
| Method | Average accuracy (%) | Decrease from PEAK (pp) |
|---|---|---|
| PEAK | 57.06 | — |
| Uniform KL following PEAK’s batch-mean coefficient | 54.87 | 2.19 |
| Shuffled PEAK weights within each response | 55.49 | 1.57 |
| Inverted entropy factors | 55.05 | 2.01 |
| DAPO + uniform KL | 55.43 | 1.63 |
PEAK outperforms both the batch-mean schedule and shuffled-weight controls. These results support the importance of assigning regularization weights according to token-specific entropy and position, beyond the average coefficient schedule or weight distribution alone.
Inverting the entropy factors also reduces accuracy. Together with the entropy-decrease ablation in Table 3, this provides additional support for PEAK’s preference for regularizing positive entropy excess.
Ablataion settings
For the entropy-only and position-only ablations, we retained PEAK’s training settings, including the base KL coefficient . Starting from , the entropy-only ablation sets , and the position-only ablation sets .
We will clarify this protocol and include the additional results in the camera-ready version.
We thank the reviewer for their thoughtful evaluation and for recognizing both our empirical findings on consistency and coverage and PEAK’s practical simplicity. We also appreciate the detailed suggestions for additional analyses and ablations, and address each comment below.
W1, W2, Q1(2) Explicit answer statements and shared reasoning traces
Our original protocol captures both 1) what information a model preserves in its own reasoning and 2) how it uses that reasoning to produce the final answer. While both aspects are relevant to the model’s use of textual and visual information, we agree that the original experiment does not isolate reasoning utilization from differences in trace content. We therefore conducted two complementary analyses.
Explicit answer statements in retained reasoning.
First, we measured how frequently each model explicitly states its final answer within the retained reasoning, using an LLM-based detector(GPT6-luna).
RLVR-trained models state their final answers less frequently within the reasoning than the pre-RLVR model, with the largest reduction observed without KL regularization. Together with our qualitative observation of no-KL responses that omit an explicit derivation of the final answer, this suggests that the textual reasoning may provide a less complete basis for answering, potentially leaving a greater role for visual information.
Shared reasoning traces with explicit answers removed.
To distinguish these differences in reasoning content from differences in how models use the supplied reasoning, we evaluated all models using a shared set of reasoning traces after removing explicit answer statements. We used a two-stage LLM-based procedure: detecting sentences containing explicit answer statements, then rewriting those sentences to remove the answer-bearing portions while aiming to preserve the supporting reasoning. We collected 200 traces from each of four source models for each of the seven benchmark views, yielding 5,600 shared question–trace pairs. For each pair, all four evaluated models received the same reasoning trace.
The relative sensitivity pattern persists under these controls. The pre-RLVR model is more sensitive to mismatched reasoning than to mismatched images. All three RLVR-trained models exhibit the opposite ordering. This asymmetry is largest without KL regularization and is somewhat smaller with uniform KL and PEAK.
These results indicate that the observed pattern is not solely attributable to explicit answer statements or differences in the supplied reasoning across models. Nevertheless, mismatched reasoning may still act as distracting text. We therefore interpret these results as evidence of relative sensitivity to the two interventions under this protocol.
W2, Q1(1) Matched clean-regeneration control
To address this concern, we conducted additional experiments with a matched clean-regeneration control and length/format-matched donor reasoning. For the clean control, we removed the original boxed answer and regenerated the answer using the same prefix construction and decoding procedure as in the intervention conditions.
Donor reasoning was drawn from a different question and matched on four criteria: (1) benchmark, (2) token length within ±10% of the original reasoning, (3) answer type—multiple-choice or open-ended—and (4) reasoning format, categorized by rules as numbered steps, bullet points, equation-centered reasoning, or prose paragraphs. All intervention drops below are measured relative to clean-regenerated accuracy.
Clean regeneration changes accuracy by less than 0.5 percentage points for all RLVR-trained models and by 1.54 points for the pre-RLVR model. The matched-control experiments reproduce the relative sensitivity pattern reported in the original submission. Specifically, the pre-RLVR model is more sensitive to mismatched reasoning than to a mismatched image. After RLVR, this asymmetry largely disappears or reverses, particularly without KL.