acceptodds
Under review as a conference paper at ICLR 2027

Deeper Is Not Always Wiser: Evidential GRPO for Overthinking in Large Vision-Language Models

Abstract

Existing approaches to overthinking in large vision-language models (LVLMs) mainly control reasoning length at the token-sequence level, with limited attention to how entropy changes across LLM depth. We observe that entropy can reach a trough at intermediate layers but rebound as LLM depth increases, leading to unnecessary additional reasoning. Motivated by this observation, we propose Evidential GRPO EviGRPO, a new GRPO variant that regulates reasoning through a Trigger–Verify process. It consists of an entropy trigger, which monitors entropy variations across LLM layers to identify the layer where entropy rebounds from its trough, and an evidence verifier, which detects rebound-induced token preference shifts and learns to assess whether the shift is sufficiently supported by evidence. For an evidence-unsupported preference shift, the final-layer token is replaced with the token predicted at the entropy-trough layer, while evidence-supported tokens are retained. Next, we formulate the EviGRPO reward based on the changes in reasoning length and final-answer correctness under trough-token substitution, while deriving evidential supervision from token-shift outcomes for verifier optimization. In this way, EviGRPO rectifies abnormal final-layer token shifts, thereby mitigating their propagation into redundant reasoning. Experiments across seven reasoning benchmarks demonstrate that EviGRPO consistently improves reasoning accuracy across 3B, 7B, and 32B models, achieving accuracy gains of 1.2–5.7 percentage points over standard GRPO while reducing reasoning length by up to 256.4 tokens, effectively balancing reasoning accuracy and efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Vision-Language Models, Efficient Reasoning, Evidential Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kHL3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies overthinking in large vision-language models from a layer-wise perspective. The key idea is to detect entropy troughs near the final layers and identify cases where the preferred token changes between the trough layer and the final layer. An Evidence Verifier is then used to decide whether the final-layer token shift should be retained or replaced by the trough-layer token. During training, the method further incorporates changes in answer correctness and reasoning length into a GRPO-based objective.

Critical strengths

The paper studies overthinking from a layer-wise perspective rather than only controlling sequence length, which is an interesting and potentially useful direction. The Trigger–Verify design is intuitive and explicitly accounts for the fact that intermediate-layer predictions are not always better than final-layer predictions. The empirical results and ablations generally support the usefulness of the proposed components, including the verifier and the correctness- and length-aware rewards. The case studies are also informative, as they analyze concrete token substitutions made by EviGRPO rather than only reporting aggregate performance.

Critical weaknesses

The current experiments do not fully establish that the entropy trough itself is the key factor. It remains unclear whether the gains come from entropy-based layer selection or from a more general late-intermediate-layer intervention. The intermediate-layer distributions are obtained through the shared final LM head, so the cross-layer entropy may be affected by layer-dependent calibration rather than directly reflecting uncertainty. The Evidence Verifier is supervised by trajectory-level group-relative advantage instead of direct token-level outcomes. This may introduce credit-assignment issues when multiple token shifts occur in one trajectory, and the supervision target also changes as the policy evolves during RL training. The paper introduces several method-specific hyperparameters. Although the appendix reports sensitivity analyses, it is not fully clear whether these hyperparameters were selected on independent validation data.

Questions for authors
  1. The intermediate-layer distributions are obtained using the shared final LM head. How robust is the observed entropy-trough phenomenon to layer-wise calibration or alternative readout mechanisms? Does the same behavior hold consistently across different model families?
  2. The Evidence Verifier is trained from rollout-level relative advantages while the underlying policy is continuously changing. How stable is the verifier under this non-stationary supervision? Is there any evaluation of verifier accuracy or calibration during training, especially when multiple token shifts occur within one trajectory?
  3. What is the practical inference overhead of the additional intermediate-layer LM-head projections and verifier calls? It would be useful to report the trigger frequency, verifier activation frequency, and actual latency overhead.
8 comments

Thank you for recognizing our layer-wise perspective on overthinking as an interesting and potentially useful direction, and for finding the Trigger–Verify design intuitive. We address your concerns and questions below.

W1. Entropy-Based Layer Selection, Layer-Wise Calibration, Verifier Supervision, and Hyperparameter Selection

Thank you for these comments. We address these concerns below.

1. Entropy-Based Layer Selection Analysis. To examine whether the improvements come from entropy-based layer selection or general late-layer intervention, we conduct additional comparisons in Appendix D.4. Specifically, Table 15 compares entropy-trough selection with ten fixed intermediate layers (L−1L-1 to L−10L-10) on Qwen2.5-VL-7B. The best-performing fixed layer differs across benchmarks, indicating that a fixed layer cannot consistently capture the optimal token predictions across different reasoning tasks. Compared with Layer L−3L-3, which achieves the highest average accuracy among fixed-layers, entropy-trough selection improves average accuracy by 2.7 points while reducing reasoning length by 32.5 tokens. Furthermore, it outperforms the best fixed-layer accuracy on each of the four benchmarks by 1.2–3.2 points. These comparisons demonstrate the advantages of adaptive entropy-based layer selection over fixed late-layer intervention.

Table 15: Comparison of late-layer token substitution and entropy-trough token substitution on Qwen2.5-VL-7B. Acc. denotes accuracy (%), and Len. denotes the average number of reasoning tokens.

Layer SelectionMathVista Acc.MathVista Len.WeMath Acc.WeMath Len.MM-Vet Acc.MM-Vet Len.MMK12 Acc.MMK12 Len.Avg. Acc.Avg. Len.
Layer L−1L-175.2188.167.7222.671.5117.974.3390.472.2229.8
Layer L−2L-274.6198.766.9237.572.0112.473.6399.271.8237.0
Layer L−3L-375.8182.968.1213.870.6127.475.2373.672.4224.4
Layer L−4L-473.1218.466.3249.671.9121.873.6411.771.2250.4
Layer L−5L-574.7210.268.4227.669.9145.374.0395.171.8244.6
Layer L−6L-672.9236.164.5262.470.8134.773.9427.570.5265.2
Layer L−7L-774.1219.365.9250.868.9148.572.3452.270.3267.7
Layer L−8L-874.5215.867.1241.670.1127.873.1431.971.2254.3
Layer L−9L-971.2260.762.9292.168.1156.870.9467.368.3294.2
Layer L−10L-1072.8237.465.3273.670.4134.172.6437.970.3270.8
Entropy-Trough Layer77.0166.471.6178.173.6100.478.3322.875.1191.9

2. Layer-Wise Calibration Analysis. We further examine the effect of intermediate-layer calibration in Appendix D.5. We introduce learnable temperature parameters for the last ten layers and optimize them on LLaVA-CoT while keeping the LLM and verifier parameters frozen. As shown in Table 16, entropy-trough matching rates reach 93.2% and 93.6% for Qwen2.5-VL-3B and 7B, respectively. Most trough locations remain unchanged after temperature calibration.

Table 16: Robustness of the entropy-trough phenomenon to layer-wise temperature calibration on Qwen2.5-VL-3B and Qwen2.5-VL-7B. Entropy-trough matching rate measures the percentage of calibrated entropy troughs that match those obtained using the shared final LM head at the same token positions and layers.

ModelReadout MechanismMathVistaWeMathMM-VetMMK12Avg.
Qwen2.5-VL-3BFinal LM Head Readout100.0100.0100.0100.0100.0
Qwen2.5-VL-3BTemperature-Calibrated Readout96.488.794.193.593.2
Qwen2.5-VL-7BFinal LM Head Readout100.0100.0100.0100.0100.0
Qwen2.5-VL-7BTemperature-Calibrated Readout89.496.895.592.893.6

3. Evidence Verifier Supervision. We acknowledge that trajectory-level advantages provide coarse supervision when multiple token shifts occur within the same rollout. Eq. (11) combines the group-relative advantage with each token-selection action, but cannot fully isolate individual token contributions. As the policy evolves, the verifier is updated using the latest rollout outcomes to adapt to changing supervision. Fig. 10 shows that the verifier loss gradually stabilizes during training. Furthermore, Table 10 reports 86.2% token-level verification accuracy using independently evaluated token choices. These results suggest that the verifier learns useful token preferences despite coarse supervision. However, we recognize that coarse supervision is not optimal and acknowledge this limitation. Introducing more precise token-level supervision for entropy-trough token selection may further improve EviGRPO.

Table 10: Token-level verification accuracy of the evidence verifier on Qwen2.5-VL-7B.

BenchmarkAccuracy (%)
MathVista89.3
WeMath77.6
MM-Vet93.1
MMK1284.8
Avg.86.2

4. Hyperparameter Selection Details. As clarified in Appendix A.1, we reserve 10% of ViRL39K as an independent validation set and use the remaining 90% for policy optimization. Hyperparameters are selected based on validation accuracy and reasoning efficiency.

We have added these analyses and clarifications to the revised manuscript on Page 23, Lines 1206–1220 and 1227–1241 (Appendix D.4 and Table 15), Pages 23–24, Lines 1222–1225 and 1243–1261 (Appendix D.5 and Table 16), Page 19, Lines 1000–1024 (Appendix C.3 and Table 10), Page 22, Lines 1164–1169 (Appendix D.2 and Fig. 10), and Page 14, Lines 744–747 (Appendix A.1).

Q1. Robustness to Layer-Wise Calibration

Thank you for this question. As discussed in our response to Weakness 2, we conduct additional experiments in Appendix D.5 to examine the robustness of entropy troughs to layer-wise calibration. Specifically, we introduce learnable temperature parameters for the last ten layers and optimize them on LLaVA-CoT while keeping the LLM and verifier parameters frozen. As shown in Table 16, the entropy-trough matching rates reach 93.2% and 93.6% for Qwen2.5-VL-3B and 7B, respectively. These results indicate that most entropy troughs remain stable after calibration across both model scales.

Q2. Evidence Verifier Stability and Token-Level Evaluation

1. Evidence Verifier Stability Analysis. To examine the verifier's stability, we analyze its training dynamics in Appendix D.2. As shown in Fig. 10, the evidential loss decreases during training for both Qwen2.5-VL-3B and 7B. It then gradually stabilizes with moderate fluctuations. We attribute this stability to the supervision mechanism, which uses only the sign of the group-relative advantage and excludes zero-advantage samples, which may reduce the sensitivity to variations in reward magnitude.

2. Evidence Verifier Evaluation. We acknowledge that verifier accuracy and calibration are not separately evaluated during training. As defined in Eq. (11), the supervision signal is derived from the sign of the group-relative advantage and the corresponding token-selection action. When multiple token shifts occur within the same trajectory, their supervision shares the same rollout-level outcome, making it difficult to isolate the contribution of each token decision. We recognize this limitation and believe that introducing more precise token-level supervision could further improve the reliability of the verifier.

Q3. Inference Overhead of the Entropy Trigger and Evidence Verifier

Thank you for this question. We provide a detailed computational overhead analysis in Appendix D.1 (Tables 12 and 13).

As shown in Table 13, during inference, the entropy trigger is activated at 45.03% and 39.42% of reasoning positions for Qwen2.5-VL-3B and 7B, respectively, while the evidence verifier is activated at 32.44% and 30.74%. The corresponding trigger and verifier latencies are 0.21/0.10 hours for 3B and 0.27/0.12 hours for 7B. Despite these additional computations, EviGRPO reduces total inference cost by 35.58% and 34.62% compared with standard GRPO. Peak GPU memory increases by only 2.22% and 3.20%.

Table 12: Computational overhead analysis of GRPO and EviGRPO during training on Qwen2.5-VL-3B and 7B using 8 NVIDIA H100 GPUs.

ModelEntropy Trigger Frequency (%)Entropy Trigger Latency (h)Evidence Verifier Frequency (%)Evidence Verifier Latency (h)Total Training Cost (GPU-Hours)Peak GPU Memory (GiB)
Qwen2.5-VL-3B + GRPO--------83.7553.62
Qwen2.5-VL-3B + EviGRPO53.853.3242.701.9497.68 (+16.63%)55.74 (+3.95%)
Qwen2.5-VL-7B + GRPO--------113.5570.90
Qwen2.5-VL-7B + EviGRPO46.574.8237.142.35136.73 (+20.41%)73.65 (+3.88%)

Table 13: Computational overhead analysis of GRPO and EviGRPO on the MathVision benchmark using Qwen2.5-VL-3B and 7B with 8 NVIDIA H100 GPUs.

ModelEntropy Trigger Frequency (%)Entropy Trigger Latency (h)Evidence Verifier Frequency (%)Evidence Verifier Latency (h)Total Inference Cost (GPU-Hours)Peak GPU Memory (GiB)
Qwen2.5-VL-3B + GRPO--------9.644.05
Qwen2.5-VL-3B + EviGRPO45.030.2132.440.106.21 (-35.58%)4.14 (+2.22%)
Qwen2.5-VL-7B + GRPO--------12.484.37
Qwen2.5-VL-7B + EviGRPO39.420.2730.740.128.16 (-34.62%)4.51 (+3.20%)

Table 12 additionally reports the training overhead, where EviGRPO increases total training cost by 16.63% and 20.41% for the 3B and 7B models, respectively. The additional training cost mainly arises from the entropy trigger and evidence verifier, as well as the generation of both standard and entropy-trough-substituted responses to calculate the answer correctness and reasoning length rewards. However, inference requires only a shorter entropy-trough-substituted reasoning trajectory, reducing the overall inference GPU-hours despite the additional trigger and verifier computations.

We have added these computational overhead analyses to the revised manuscript on Pages 20–21, Lines 1051–1095 (Appendices D.1.2–D.1.3 and Tables 12–13).

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Respectfully and sincerely,
Your Authors

Reviewer_PTB4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies overthinking in LVLMs from the perspective of cross-layer token dynamics. Using the logit lens, the authors observe that token entropy often reaches a trough at a late intermediate layer and then rebounds toward the final layer. Based on this, they propose EviGRPO. An entropy trigger locates the entropy trough within the last K layers. An evidence verifier, implemented as a lightweight MLP, then uses the "evidential change" between the two layers to decide whether to keep the final-layer token or fall back to the trough-layer token. During training, relative correctness and relative length rewards, computed by comparing responses before and after substitution, are added to the accuracy and format rewards. The verifier is trained jointly with the policy via a Dirichlet evidential loss, using the sign of the group advantage as its label. On Qwen2.5-VL 3B/7B/32B and seven multimodal reasoning benchmarks, EviGRPO improves accuracy over GRPO by 1.2–5.7 points and reduces reasoning length by up to 256.4 tokens.

Critical strengths
  1. Approaching overthinking through cross-layer entropy dynamics (an entropy trough followed by a rebound) and turning this into a trainable decoding-time gate inside GRPO is a fresh angle. It complements the prevailing sequence-level length control.
  2. The experiments cover 3B/7B/32B models and seven benchmarks, and EviGRPO improves both accuracy and reasoning length over GRPO in all 21 settings. The paper also includes ablations, hyperparameter sensitivity, and latency analysis, and candidly presents a failure case.
Critical weaknesses
  1. The Verifier column in Fig. 5 matches EviGRPO in Table 1 exactly. When the same model decodes with final-layer tokens only, it averages about 72.2% on the four benchmarks, on par with FAST-GRPO (about 72.0%), and on WeMath it is even below GRPO (67.5% vs. 68.3%). The gains over the baselines therefore come mainly from inference-time trough substitution rather than from GRPO training itself, yet all baselines in Table 1 use standard decoding. Missing controls include GRPO/FAST-GRPO with inference-time substitution, layer-selection decoding such as DoLa, and GRPO with a conventional length penalty.

  2. The paper does not specify whether the policy gradient is applied to ystdy^{\text{std}} or ytry^{\text{tr}}. In the former case, part of its reward comes from a different trajectory. In the latter case, the substituted tokens are the argmax of an intermediate layer rather than samples from πθold\pi_{\theta_\text{old}}, yet the clipped ratio is still computed from final-layer probabilities with no correction. Moreover, the difference-based rewards RansR^{\text{ans}} and RlenR^{\text{len}} can be inflated by making ystdy^{\text{std}} worse or longer. In Fig. 9, RansR^{\text{ans}} reaches about 0.2–0.4 late in training. That would mean substitution adds 20–40 points of net accuracy, which is clearly inconsistent with the 1.5–4 point test-time gap in Fig. 5.

  3. The verifier's label is sgn⁡(Ai)\operatorname{sgn}(A_i) multiplied by the verifier's own action. All substitutions in a trajectory share one sign, and decisions are deterministic with no exploration, so the signal is coarse and self-reinforcing. The so-called evidential modeling is essentially binary classification on the scalar GtEG^E_t, and the Dirichlet uncertainty is never used. Fig. 6 does not state where its labels come from, the two sides of the decision boundary overlap noticeably, and there is no token-level accuracy evaluation. In addition, 97.34% of samples in Fig. 1b contain an entropy trough, so the trigger is barely selective, yet token-level trigger and substitution rates are not reported.

  4. Logit-lens distributions at intermediate layers are biased; this is precisely the point of the tuned lens work the authors cite. The final-layer entropy rebound also already has an explanation as confidence regulation. Layer-wise overthinking and DoLa-style layer-contrastive decoding have been studied before, so the "overlooked phenomenon" claim does not hold. The core evidence, Fig. 1c, is a 1.3-point gain on a single benchmark with no variance reported. The case studies compare full outputs from different models, so the length reduction cannot be attributed to the few substituted tokens.

Questions for authors

Please refer to Weaknesses.

4 comments

We appreciate your recognition of the novelty of our cross-layer entropy perspective, consistent performance improvements, and comprehensive experimental evaluation. For the concerns, we have provided additional comparisons, quantitative analyses, and detailed methodological clarifications on policy optimization, verifier supervision, and entropy dynamics in the revised manuscript. We hope these additions address your concerns.

W1. Comparison of Decoding Strategies and Length Penalty

Thank you for highlighting this issue. We conduct additional comparisons of GRPO, FAST-GRPO, and EviGRPO under four decoding strategies: standard decoding, DoLa decoding, trough-token substitution, and verifier-guided decoding. We also compare EviGRPO with GRPO using a conventional length penalty. The results are presented in Tables 6 and 7.

1. Decoding Strategies. As shown in Table 6, EviGRPO achieves 1.3% higher average accuracy than GRPO under standard decoding, reflecting the benefit of policy optimization. Under identical trough-substitution decoding, EviGRPO further outperforms GRPO and FAST-GRPO by 2.0% and 0.8%, respectively. Within EviGRPO, verifier-guided decoding improves average accuracy by 1.4% while reducing reasoning length by 16.6 tokens compared with direct trough substitution, highlighting the importance of preserving beneficial final-layer token shifts. In contrast, DoLa produces mixed results, improving WeMath accuracy by 0.7% but reducing MM-Vet accuracy by 0.8% relative to standard decoding. Compared with DoLa, our verifier-guided decoding achieves 2.7% higher average accuracy with 28.9 fewer reasoning tokens, providing a better balance between reasoning accuracy and efficiency.

Table 6: Comparison of GRPO, FAST-GRPO, and EviGRPO under different decoding strategies on Qwen2.5-VL-7B. Acc. denotes accuracy (%), and Len. denotes average reasoning length.

Training MethodDecoding StrategyMathVista Acc.MathVista Len.WeMath Acc.WeMath Len.MM-Vet Acc.MM-Vet Len.MMK12 Acc.MMK12 Len.Avg. Acc.Avg. Len.
GRPOStandard Decoding74.4194.068.3285.168.2154.772.8466.770.9275.1
GRPODoLa Decoding73.8208.769.1298.667.4163.573.9454.671.1281.4
GRPOTrough Substitution75.0180.668.7254.368.5144.174.4412.971.7248.0
GRPOVerifier-guided Decoding74.6169.369.2234.870.7133.775.6399.472.5234.3
FAST-GRPOStandard Decoding74.8141.467.6160.571.0119.374.5386.072.0201.8
FAST-GRPODoLa Decoding75.1128.766.8173.972.5112.474.0345.672.1190.2
FAST-GRPOTrough Substitution75.4133.668.2150.971.9109.776.2357.872.9188.0
FAST-GRPOVerifier-guided Decoding75.9122.169.7141.972.8103.777.1333.873.9175.4
EviGRPOStandard Decoding75.5188.967.5226.771.3116.874.5385.272.2229.4
EviGRPODoLa Decoding75.8194.368.2191.470.5120.775.0376.972.4220.8
EviGRPOTrough Substitution76.1180.069.2199.273.0112.476.4342.473.7208.5
EviGRPOVerifier-guided Decoding77.0166.471.6178.173.6100.478.3322.875.1191.9

2. Comparison with Conventional Length Penalty. Table 7 shows that incorporating a conventional length penalty [1] into GRPO reduces average reasoning length from 275.1 to 201.4 tokens but decreases accuracy from 70.9% to 70.0%. In contrast, EviGRPO achieves 75.1% accuracy with a shorter reasoning length of 191.9 tokens. These findings suggest that EviGRPO offers a more effective approach to reducing redundant reasoning while preserving answer correctness.

[1] Arora, D., and Zanette, A. “Training Language Models to Reason Efficiently.” Advances in Neural Information Processing Systems (NeurIPS), vol. 38, pp. 67629–67667, 2025.

Table 7: Comparison with a conventional length penalty on Qwen2.5-VL-7B.

MethodMathVista Acc.MathVista Len.WeMath Acc.WeMath Len.MM-Vet Acc.MM-Vet Len.MMK12 Acc.MMK12 Len.Avg. Acc.Avg. Len.
GRPO74.4194.068.3285.168.2154.772.8466.770.9275.1
GRPO + Length Penalty73.6158.966.5195.667.896.771.9354.270.0201.4
EviGRPO77.0166.471.6178.173.6100.478.3322.875.1191.9

We have included these comparisons in the revised manuscript on Pages 16–17, Lines 850–898 (Appendices B.4–B.5, Tables 6–7).

W2. Policy Optimization and Relative Reward Design

We thank you for these concerns. We clarify the policy optimization procedure and the interpretation of the relative rewards below.

1. Policy Gradient Computation. As specified in Section 4.4 (Page 6, Lines 284–296), the policy gradient is applied to the standard trajectory ystdy^{\mathrm{std}}, which is sampled from the final-layer policy πθold\pi_{\theta_{\mathrm{old}}}. The substituted trajectory ytry^{\mathrm{tr}} is used to compute RansR^{\mathrm{ans}} and RlenR^{\mathrm{len}}. Both the current and old policy probabilities are evaluated on ystdy^{\mathrm{std}}, ensuring that the importance ratio corresponds to the sampled trajectory.

2. Relative Reward Design. We acknowledge that RansR^{\mathrm{ans}} and RlenR^{\mathrm{len}} can be inflated if ystdy^{\mathrm{std}} becomes less accurate or longer. Nevertheless, the ablation results in Table 2 demonstrate the overall benefits of both rewards. Specifically, removing RansR^{\mathrm{ans}} decreases average accuracy by 2.1% and increases reasoning length by 59.2 tokens, while removing RlenR^{\mathrm{len}} reduces accuracy by 1.3% and increases reasoning length by 11.8 tokens. Furthermore, Table 6 shows that EviGRPO achieves 1.3% higher accuracy and 45.7 fewer reasoning tokens than GRPO under standard decoding, suggesting that the improvements are not simply achieved by degrading ystdy^{\mathrm{std}}.

3. Interpretation of Fig. 9. The values of RansR^{\mathrm{ans}} in Fig. 9 represent the average correctness difference between standard and substituted responses during training on ViRL39K. In contrast, Fig. 5 reports accuracy under different decoding strategies on four benchmarks. Therefore, the reward values measured during training and the accuracy measured during evaluation represent distinct quantities. The increasing RansR^{\mathrm{ans}} reflects the relative correctness improvements in EviGRPO training rollouts, not an improvement in benchmark accuracy.

We have added clarifications on policy gradient computation to the revised manuscript on Page 6, Lines 286–289 and 294–296 (Section 4.4).

W3. Verifier Labels, Fig. 6, and Token-Level Triggering and Verification

Thank you for these comments. We clarify the verifier labels, Fig. 6, and token-level triggering and verification below.

1. Verifier Labels. In Eq. (11), the verifier label qi,t=sgn(Ai)ρi,tq_{i,t}=\mathrm{sgn}(A_i)\rho_{i,t} combines the group-relative advantage with the token-selection action. We acknowledge that this provides coarse supervision because token decisions within the same trajectory share the advantage sign. However, the action indicator distinguishes retaining final-layer tokens from adopting trough-layer tokens, and zero-advantage samples are excluded from verifier optimization. For Dirichlet uncertainty, Eqs. (12)–(13) construct Dirichlet parameters αi,t=ei,t+1\alpha_{i,t} = e_{i,t}+1, where the total evidence strength is Si,t=∑k=12αi,t,kS_{i,t}=\sum_{k=1}^{2}\alpha_{i,t,k}. The corresponding uncertainty is defined as ui,t=2/Si,tu_{i,t}=2/S_{i,t}. Since Eq. (13) contains ψ(Si,t)=ψ(2/ui,t)\psi(S_{i,t})=\psi(2/u_{i,t}), Dirichlet uncertainty is implicitly incorporated into the evidential loss through the evidence strength.

2. Fig. 6. In Fig. 6, Final-Supported and Trough-Supported represent two categories of token shifts, corresponding to supervision of the final-layer token and entropy-trough token, respectively, as defined by qi,tq_{i,t} in Eq. (11). We acknowledge that some samples overlap near the decision boundary GtE=0G_t^E=0, where similar evidential changes make token shifts difficult to distinguish. Nevertheless, most samples are separated, indicating that the verifier can effectively distinguish final-supported and trough-supported token shifts. To quantitatively evaluate the verifier, we conduct additional token-level experiments on Qwen2.5-VL-7B in Appendix C.3. Specifically, given the same reasoning prefix, we generate two responses by retaining the final-layer token or replacing it with the trough-layer token at a single position, while using standard decoding at all other positions. As reported in Table 10, the verifier achieves accuracies of 89.3%, 77.6%, 93.1%, and 84.8% on MathVista, WeMath, MM-Vet, and MMK12, respectively, with an average of 86.2%. This evaluation directly measures the verifier's ability to identify beneficial token choices.

Table 10: Token-level verification accuracy of the evidence verifier on Qwen2.5-VL-7B.

BenchmarkAccuracy (%)
MathVista89.3
WeMath77.6
MM-Vet93.1
MMK1284.8
Avg.86.2

3. Token-Level Triggering and Verification. The 97.34% in Fig. 1b refers to the proportion of samples containing at least one entropy trough. Since a sample is counted even if only one trough occurs throughout its reasoning trajectory, the sample-level proportion is high. In contrast, the token-level trigger and verification frequencies are substantially lower, as reported in Tables 12 and 13. Tables 12 and 13 quantify these frequencies for the 3B/7B models. During training, trigger frequencies are 53.85%/46.57%, while verification frequencies are 42.70%/37.14%. During inference, the corresponding frequencies are 45.03%/39.42% and 32.44%/30.74%. These results highlight the distinction between sample-level trough occurrence and token-level triggering and verification frequencies.

We have added these analyses to the revised manuscript on Page 19, Lines 1000–1025 (Appendix C.3 and Table 10), and Pages 20–21, Lines 1051–1095 (Appendix D.1 and Tables 12–13).

W4. Intermediate-Layer Calibration, Entropy-Trough Clarification, Fig. 1c, and Case Study Interpretation

Thank you for these detailed comments. We address the four concerns below.

1. Intermediate-Layer Logit Calibration. We agree that applying the shared final LM head to intermediate layers may introduce calibration bias. To examine this issue, we conduct additional experiments in Appendix D.5. Specifically, we introduce learnable temperature parameters for the last ten layers and optimize them on LLaVA-CoT using token-level cross-entropy, while keeping all other model parameters frozen. We then compare the entropy-trough locations before and after calibration. As shown in Table 16, the average matching rates reach 93.2% and 93.6% for Qwen2.5-VL-3B and 7B, respectively. These results show that most entropy troughs remain consistent after layer-wise calibration.

Table 16: Robustness of the entropy-trough phenomenon to layer-wise temperature calibration on Qwen2.5-VL-3B and Qwen2.5-VL-7B. Entropy-trough matching rate measures the percentage of calibrated entropy troughs that match those obtained using the shared final LM head at the same token positions and layers.

ModelReadout MechanismMathVistaWeMathMM-VetMMK12Avg.
Qwen2.5-VL-3BFinal LM Head Readout100.0100.0100.0100.0100.0
Qwen2.5-VL-3BTemperature-Calibrated Readout96.488.794.193.593.2
Qwen2.5-VL-7BFinal LM Head Readout100.0100.0100.0100.0100.0
Qwen2.5-VL-7BTemperature-Calibrated Readout89.496.895.592.893.6

2. Entropy-Trough Clarification. We have revised our first contribution to state: "We investigate layer-wise entropy dynamics in LVLMs," rather than presenting entropy rebound as an overlooked phenomenon.

3. Fig. 1c Clarification. Fig. 1c presents an initial observation on Qwen2.5-VL-7B using MathVision. Trough-token substitution improves accuracy by 1.3 points and reduces reasoning length by 53.2 tokens. We acknowledge that this figure does not report variance and is intended to motivate our method rather than provide conclusive evidence. Table 1 further evaluates seven benchmarks across three model scales, while Fig. 5 and Table 6 provide additional evidence of the benefits of trough-token substitution and evidence-based verification.

4. Case Study Interpretation. We conduct an additional case study in Fig. 14 to examine the effect of token substitution on reasoning efficiency. Both versions of EviGRPO produce the correct answer. With token substitution, reasoning length decreases by 113 tokens (25.1%) and latency by 4.20 s (27.2%), which highlights the importance of token substitution in reducing redundant reasoning.

We have added these revisions and analyses to the revised manuscript on Page 2, Line 93 (First Contribution), Pages 23–24, Lines 1222–1261 (Appendix D.5 and Table 16), and Pages 27–28, Lines 1415–1418 (Appendix E and Fig. 14).

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Respectfully and sincerely,
Your Authors

Reviewer_HvwNOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Evidential GRPO (EviGRPO) to mitigate overthinking in LVLMs. The method identifies entropy troughs across intermediate layers and uses a trained evidence verifier to determine whether final-layer token preference shifts are justified. Unsupported shifts are corrected by substituting tokens from the entropy-trough layer. The authors further introduce a GRPO reward that considers reasoning length and answer correctness. Experiments on seven reasoning benchmarks show improvements in accuracy and reductions in reasoning length compared to standard GRPO.

Critical strengths
  • The idea of training and reward design is interesting. It improves the performance of GRPO with no further external supervision

  • The paper is well organized and easy to follow.

  • The authors conduct extensive experiments on different model sizes.

  • The performance is remarkable compared to baselines.

Critical weaknesses
  • Limited novelty of the entropy trough idea.

    The idea of an entropy trough is not entirely new. Prior work on LLMs [1] has already shown that decoding from the most confident intermediate layer can achieve strong performance.

  • Limited connection to LVLMs

    The proposed method does not appear to be particularly specific to LVLMs. Why do the authors focus on LVLMs rather than investigating the method on LLMs?

  • Potential issue with vanishing advantages

    In GRPO, the advantage can vanish when all samples within a group receive the same reward. Similarly, the proposed RansR^{ans} appears to always be zero when the std and tr rollouts have the same correctness outcome. The authors should report the distribution of different RansR^{ans} values during rollout generation throughout training to clarify how frequently this issue occurs.

  • Unclear computational overhead

    It is unclear whether the proposed method introduces substantial computational overhead, particularly during rollout generation and training. The authors should provide an analysis of the additional computational cost.

[1]: Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding

Questions for authors
  • Could the authors report the training initialized on thinking models? e.g., Qwen3-VL-Thinkinig. This may support the efficiency of the methods,

  • I'm a little bit curious about the motivation on efficient reasoning. Why are verifier and eviGRPO considered related to efficiency? It seems like the length reward is the key.

  • Since prior work on LLMs has demonstrated that decoding from the layer with the lowest entropy at each token position can improve performance, it would be particularly valuable to investigate whether the trained verifier can generalize to other models with similar architectures.

  • Could the authors provide additional ablation studies on the reward design? For example, how would removing the accuracy reward or the format reward affect performance?

  • According to the definition of trough entropy, multiple layers among the last KK layers may satisfy the criterion. In such cases, how is the layer selected for verification?

7 comments

We are grateful to you for recognizing the interesting training and reward design of our approach, the clear writing and organization, the extensive experimental evaluations across different model sizes, and the performance improvements over baselines and your valuable feedback and constructive suggestions. Below, we address your concerns and questions in turn.

W1. Novelty of the Entropy-Trough

We thank you for highlighting this closely related work. We agree that prior work [1] has investigated low-entropy intermediate layers in LLMs. However, our work investigates the entropy-trough phenomenon from a different perspective. Specifically, we find that entropy troughs in LVLMs can be used to improve both reasoning accuracy and efficiency. As shown in Fig. 1, directly replacing final-layer tokens with entropy-trough tokens improves accuracy from 25.4% to 26.7% while reducing reasoning length from 457.1 to 403.9 tokens. These observations reveal the potential of entropy troughs to mitigate overthinking in multimodal reasoning.

[1] Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding

W2. Generalization to LLM Reasoning Models

We thank you for this valuable comment. Although our work focuses on mitigating overthinking in LVLMs, the proposed Trigger–Verify can also applicable to text-only LLMs. To address this concern, we conduct additional experiments on DeepSeek-R1-Distill-Qwen-1.5B and 7B, evaluating EviGRPO across five mathematical and scientific reasoning benchmarks. The results are presented in Table 8.

As shown in Table 8, EviGRPO consistently improves reasoning accuracy and reduces reasoning length across both models. For the 1.5B model, EviGRPO outperforms Thinkless on all five benchmarks, achieving 88.3% and 65.5% accuracy on MATH500 and AMC2023, exceeding Thinkless by 6.9% and 7.3%, respectively. Meanwhile, reasoning length decreases from 5344.1 to 2016.8 tokens on MATH500 and from 10445.0 to 5149.5 tokens on AMC2023 compared with the original model.

For the 7B model, EviGRPO achieves 97.4% on MATH500, 89.1% on AMC2023, and 52.5% on GPQA-Diamond, outperforming VeriThinker by 2.4%, 4.0%, and 2.7%, respectively. It also reduces reasoning length on AIME2025 and GPQA-Diamond from 13794.0/7750.6 to 8428.6/3886.3 tokens compared with the original model. These results demonstrate that EviGRPO is not restricted to LVLMs and can effectively improve reasoning accuracy and efficiency in text-only LLMs.

Table 8: Comparison of text-only LLMs on mathematical and scientific reasoning benchmarks. Acc. denotes accuracy (%), and Len. denotes the average number of reasoning tokens.

DeepSeek-R1-Distill-Qwen-1.5B

BenchmarkBase Acc.Base Len.Thinkless Acc.Thinkless Len.EviGRPO Acc.EviGRPO Len.
GSM8K83.31946.284.5607.984.7811.3
MATH50086.45344.181.42486.588.32016.8
AMC202362.410445.058.26840.165.55149.5
AIME202524.615692.824.511052.629.48466.7
GPQA-Diamond37.018063.429.55584.037.87448.0

DeepSeek-R1-Distill-Qwen-7B

BenchmarkBase Acc.Base Len.VeriThinker Acc.VeriThinker Len.EviGRPO Acc.EviGRPO Len.
GSM8K91.5981.893.0774.692.7547.2
MATH50094.23815.495.02103.797.42547.0
AMC202386.45874.285.14476.089.15060.9
AIME202538.513794.041.29812.441.28428.6
GPQA-Diamond50.97750.649.85049.552.53886.3

We have added these experiments to the revised manuscript on Pages 17–18, Lines 903–964 (Appendix C.1 and Table 8).

W3. Distribution of RansR^{\mathrm{ans}}

We thank you for raising this concern. We agree that Rans=0R^{\mathrm{ans}}=0 when the standard and entropy-trough-substituted responses have the same answer correctness. To investigate how frequently this occurs, we analyze the distribution of RansR^{\mathrm{ans}} during training. The results are presented in Table 14.

As shown in Table 14, nonzero rewards account for 22.95% and 38.19% of rollouts for the 3B and 7B models, respectively. Notably, positive rewards substantially outnumber negative rewards (19.51% vs. 3.44% and 31.47% vs. 6.72%), indicating that trough-token substitution more frequently corrects erroneous answers than disrupts correct predictions. These nonzero rewards provide feedback that encourages beneficial substitutions while penalizing harmful substitutions during policy optimization.

Furthermore, Rans=0R^{\mathrm{ans}}=0 does not necessarily imply vanishing GRPO advantages. Although the two responses have the same answer correctness, trough-token substitution may change the reasoning trajectory and its length, providing additional feedback through RlenR^{\mathrm{len}}.

Table 14: Distribution of relative answer correctness rewards during training.

ModelRans=−1R^{\mathrm{ans}}=-1Rans=0R^{\mathrm{ans}}=0Rans=+1R^{\mathrm{ans}}=+1
Qwen2.5-VL-3B3.44%77.05%19.51%
Qwen2.5-VL-7B6.72%61.81%31.47%

We have added this analysis to the revised manuscript on Page 23, Lines 1188–1204 (Appendix D.3 and Table 14).

W4. Computational Overhead During Training and Inference

We thank you for this valuable suggestion. To evaluate the additional computational overhead of EviGRPO, we conduct efficiency analyses on Qwen2.5-VL-3B and 7B using 8 NVIDIA H100 GPUs. We report the trigger frequencies, the latencies of the entropy trigger and evidence verifier, total GPU-hours and peak GPU memory. The results are presented in Tables 12 and 13.

1. Training Overhead. As shown in Table 12, the entropy trigger and evidence verifier are activated at 53.85%/42.70% for the 3B model and 46.57%/37.14% for the 7B model. EviGRPO additionally generates standard and entropy-trough-substituted responses during training to compute RansR^{\mathrm{ans}} and RlenR^{\mathrm{len}}. Compared with standard GRPO, the total training cost increases by 16.63% and 20.41% for the 3B and 7B models, while peak GPU memory increases by only 3.95% and 3.88%. These results quantify the additional computational cost introduced by EviGRPO during training.

Table 12: Computational overhead analysis during training on 8 NVIDIA H100 GPUs.

ModelTrigger Freq. (%)Trigger Latency (h)Verifier Freq. (%)Verifier Latency (h)GPU-HoursPeak Memory (GiB)
Qwen2.5-VL-3B + GRPO--------83.7553.62
Qwen2.5-VL-3B + EviGRPO53.853.3242.701.9497.68 (+16.63%)55.74 (+3.95%)
Qwen2.5-VL-7B + GRPO--------113.5570.90
Qwen2.5-VL-7B + EviGRPO46.574.8237.142.35136.73 (+20.41%)73.65 (+3.88%)

2. Inference Overhead. As shown in Table 13, inference does not require additional rollout generation for reward computation. Although the entropy trigger and evidence verifier introduce additional computation, EviGRPO substantially reduces reasoning length. Compared with standard GRPO, EviGRPO reduces total inference cost by 35.58% and 34.62% for the 3B and 7B models, respectively, while peak GPU memory increases by only 2.22% and 3.20%. These results demonstrate that the efficiency gains from shorter reasoning sequences exceed the additional computational overhead during inference.

Table 13: Computational overhead analysis on MathVision using 8 NVIDIA H100 GPUs.

ModelTrigger Freq. (%)Trigger Latency (h)Verifier Freq. (%)Verifier Latency (h)GPU-HoursPeak Memory (GiB)
Qwen2.5-VL-3B + GRPO--------9.644.05
Qwen2.5-VL-3B + EviGRPO45.030.2132.440.106.21 (-35.58%)4.14 (+2.22%)
Qwen2.5-VL-7B + GRPO--------12.484.37
Qwen2.5-VL-7B + EviGRPO39.420.2730.740.128.16 (-34.62%)4.51 (+3.20%)

We have added these analyses to the revised manuscript on Page 20, Lines 1051–1064 and 1072–1079 (Appendix D.1.2 and Table 12), and Pages 20–21, Lines 1066–1095 (Appendix D.1.3 and Table 13).

Q1. Generalization to Thinking Models

We appreciate this suggestion. To address this question, we conduct additional experiments from Qwen3-VL-8B-Thinking, trained on ViRL39K with a maximum response length of 4096 tokens. We compare EviGRPO with standard GRPO and FAST-GRPO across seven reasoning benchmarks.

As shown in Table 9, EviGRPO consistently outperforms GRPO in accuracy and reasoning length across all seven benchmarks. Specifically, on MathVerse and MathVision, accuracy improves from 78.1%/45.7% to 81.4%/50.2%, while reasoning length decreases from 1593.5/2718.5 to 562.4/1084.5 tokens. Compared with FAST-GRPO, EviGRPO achieves higher accuracy on six benchmarks and shorter reasoning length on five benchmarks. These results demonstrate that EviGRPO also improves reasoning accuracy and efficiency when initialized from Qwen3-VL-8B-Thinking.

Table 9: Main results on Qwen3-VL-8B-Thinking. Acc. denotes accuracy (%), and Len. denotes the average number of reasoning tokens.

ModelMathVista Acc.MathVista Len.MathVerse Acc.MathVerse Len.MathVision Acc.MathVision Len.LogicVista Acc.LogicVista Len.WeMath Acc.WeMath Len.MM-Vet Acc.MM-Vet Len.MMK12 Acc.MMK12 Len.
+ GRPO67.6688.478.11593.545.72718.556.81822.077.91247.677.4694.670.31883.1
+ FAST-GRPO69.1357.279.5644.250.61386.463.3786.777.3366.179.3455.373.8746.5
+ EviGRPO72.0411.381.4562.450.21084.567.1708.879.5513.180.8424.777.2582.7

We have added these experiments to the revised manuscript on Pages 18–19, Lines 966–998 (Appendix C.2 and Table 9).

Q2. Contributions of the Evidence Verifier and Length Reward

We appreciate your comment on the sources of reasoning efficiency in EviGRPO. We agree that the reasoning-length reward directly encourages shorter reasoning sequences. However, the evidence verifier also contributes to efficiency by identifying unsupported final-layer token shifts that may propagate into redundant reasoning.

In Table 2 of our manuscript, removing RlenR^{\mathrm{len}} increases the average reasoning length from 191.9 to 203.7 tokens and reduces accuracy from 75.1% to 73.8%, confirming the effectiveness of the length reward. Notably, removing the evidence verifier further increases reasoning length to 236.3 tokens and decreases accuracy to 72.3%, even with the length reward retained. This is because the verifier evaluates whether final-layer token shifts are sufficiently supported by evidence, preventing unsupported shifts from extending reasoning while preserving beneficial corrections. Therefore, the length reward encourages shorter reasoning, while the evidence verifier improves token selection to prevent unnecessary reasoning. Their complementary contributions enable EviGRPO to improve both reasoning accuracy and efficiency.

Comments on the paper

1 comment

General Response to All Reviewers

We sincerely thank the Area Chair and all reviewers for their constructive feedback and valuable suggestions. We have strengthened the manuscript through additional experiments, quantitative analyses, and methodological clarifications. We have uploaded a revised version with all changes highlighted in blue. The major updates are summarized below.


(i) Entropy-trough motivation and robustness. (Section 1; Appendices D.4–D.5; Tables 15–16)

We clarified the motivation of entropy-based layer selection and compared it with ten fixed late-layer strategies. As shown in Table 15, EviGRPO improves average accuracy by 2.7 points and reduces reasoning length by 32.5 tokens compared with the best fixed-layer strategy on average. We further evaluated layer-wise temperature calibration. Table 16 shows entropy-trough matching rates of 93.2% and 93.6% for Qwen2.5-VL-3B and 7B, respectively.

(ii) Decoding strategies and reward design. (Section 5.3; Appendices B.4–B.5; Tables 2, 6–7)

We added comparisons across four decoding strategies and additional reward ablations. As shown in Table 6, EviGRPO outperforms GRPO by 1.3 points under standard decoding, while verifier-guided decoding further improves accuracy by 1.4 points and reduces reasoning length by 16.6 tokens compared with trough substitution. Table 7 shows that EviGRPO achieves higher accuracy with shorter reasoning sequences than conventional length penalties. Table 2 further validates the contributions of the accuracy and format rewards.

(iii) Generalization to LLMs and thinking models. (Appendices C.1–C.2; Tables 8–9)

We extended EviGRPO to DeepSeek-R1-Distill-Qwen-1.5B/7B and Qwen3-VL-8B-Thinking. As shown in Tables 8 and 9, EviGRPO improves reasoning accuracy and efficiency over standard baselines across mathematical, scientific, and multimodal reasoning benchmarks, demonstrating its applicability to text-only LLMs and pretrained thinking models.

(iv) Evidence verifier supervision and evaluation. (Section 4.4; Appendices C.3 and D.2; Table 10; Fig. 10)

We clarified the verifier's rollout-derived supervision and acknowledged its limitations in token-level credit assignment. As shown in Table 10, the verifier achieves an average token-level verification accuracy of 86.2%. Fig. 10 further shows that the verifier loss gradually stabilizes during training across both 3B and 7B models.

(v) Policy optimization and relative reward analysis. (Sections 4.3–4.4; Appendices A.1 and D.3; Table 14; Fig. 9)

We clarified that policy gradients are computed using standard trajectories. As shown in Table 14, nonzero relative correctness rewards account for 22.95% and 38.19% of rollouts for 3B and 7B, respectively. We also clarified the interpretation of the training reward dynamics in Fig. 9 and the independent validation procedure for hyperparameter selection.

(vi) Computational overhead and inference efficiency. (Appendices D.1.2–D.1.3; Tables 12–13)

We added computational overhead analyses, including trigger frequencies, verifier activation frequencies, latency, GPU-hours, and memory consumption. As shown in Table 12, EviGRPO increases training cost by 16.63% and 20.41% for the 3B and 7B models, respectively. However, Table 13 shows that shorter reasoning sequences reduce total inference GPU-hours by 35.58% and 34.62%, outweighing the additional computational overhead.


We hope these revisions address the main concerns raised during review. We are grateful for the reviewers' guidance, which has substantially improved the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.