acceptodds
Under review as a conference paper at ICLR 2027

Efficient LLM Reasoning via Answer-Guided Posterior-to-Prior Transfer

Abstract

Although large language models achieve strong reasoning capabilities through reinforcement learning, they often generate unnecessarily long reasoning trajectories, leading to substantial inference costs. Conventional length-based reward engineering curtails redundancy but overlooks the fundamental sampling bottleneck: correct-yet-concise trajectories remain severely sparse under question-only exploration, hindering further gains in reasoning efficiency. Inspired by cognitive science, where knowledge of outcomes suppresses ineffective trial-and-error, we leverage training-time reference answers as privileged signals to guide exploration. We theoretically prove that, under a non-negative covariance condition between correctness likelihood and comprehensive utility, the ideal Bayesian answer-conditioned posterior yields no lower expected utility than the question-only prior, with strict improvement when the covariance is positive, providing a principled foundation for answer-guided exploration. To bridge the posterior-prior gap and prevent the transfer of privileged shortcuts, we propose VPG-EA (Variational Posterior Guidance and Efficiency Awareness). Motivated by a variational lower-bound perspective, VPG-EA uses Cross-View Validation to select trajectories that remain predictive under the question-only view, and Advantage-Gated Posterior-to-Prior Transfer to selectively distill validated efficient reasoning patterns into the question-only prior. Experiments across 1.5B and 7B models demonstrate consistent accuracy-efficiency improvements: VPG-EA improves the comprehensive efficiency metric by 8.5% and 17.2%, respectively, reduces token consumption by 29–44%, maintains or improves accuracy, and generalizes to non-mathematical scientific domains.

Primary area:
reinforcement learning
Keywords:
Large Language Models;Reinforcement Learning;Efficient Reasoning;

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o8bXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies efficient reasoning and proposes leveraging reference answers as privileged signals to guide exploration during training. The proposed training objective selects trajectories predictive under the question-only view and distills reasoning patterns into the question-only prior, demonstrating consistent accuracy-efficiency improvements.

Critical strengths

The paper is well-motivated and clearly written, giving a clear distinction between reward design and exploration. The logic flows naturally, making it easy to read and understand. I also find the idea of improving reasoning efficiency through better exploration, rather than relying solely on length penalties, interesting.

Critical weaknesses
  1. RAVR [1] is a close baseline and also leverages a similar variational approach and reward designs, using the "priors" and answer-conditioned "posteriors". It's important to clarify the key differences in terms of motivation and implementation and include it as a major baseline.
  2. On the math reasoning tasks, the results come from a single training run with one sample per question, which makes them noisy, especially on the 30-question AIME benchmarks. Reporting avg@k or pass@k would be more informative. On the OOD tasks, the proposed method performs similarly to other baselines, without clear gains.

[1]. Lin, T., Zhao, X., Zhang, X., Long, R., Xu, Y., Jiang, Z., Su, W., & Zheng, B. (2025). RAVR: Reference-Answer-guided Variational Reasoning for Large Language Models. ArXiv, abs/2510.25206.

Questions for authors

Can the authors give a more intuitive interpretation of the final objective? How do the two terms contribute to the final goal?

3 comments

We sincerely thank Reviewer for the positive feedback on our motivation, writing clarity, and core idea of improving reasoning efficiency via better exploration rather than length penalties alone! Due to the 5,000-character limit per comment, we present our response in 2 parts.

W1: Key Differences from RAVR and Direct Experimental Comparison

  1. Essential Differences in Motivation & Implementation (Appendix A, Table 3): While both RAVR and VPG-EA use a reference-answer posterior and question-only prior, they differ fundamentally in motivation, target regime, and implementation. RAVR targets pure accuracy without length constraints, whereas VPG-EA compresses length while preserving accuracy. Under length pressure, an answer-conditioned posterior easily bypasses naive rewards: trajectories may appear brief and reach y∗y^* with high surface reward, yet rely on answer hints and cannot be independently derived by the question-only prior. Thus, VPG-EA introduces three key designs beyond RAVR:
    • Cross-View Validation (CVV): We strip the final answer and check if the posterior chain remains predictive of y∗y^* under the question-only view relative to prior rollouts (Rcorrect=max⁡(0,Upost−Uˉprior)R_{\mathrm{correct}} = \max(0, U_{\mathrm{post}} - \bar{U}_{\mathrm{prior}})), multiplied by relative efficiency H(zpost)H(z^{\mathrm{post}}).
    • Advantage-Gated One-Sided Transfer: VPG-EA gates transfer by 1(Ai>0)\mathbf{1}(A_i > 0) and applies a stop-gradient sg⁡[⋅]\operatorname{sg}[\cdot] on the posterior in Ltransfer\mathcal{L}_{\mathrm{transfer}} so the unconverged prior cannot degrade the posterior.
    • Guarantee under Length Decay: Proposition 1 (verified in Appendix B.5) establishes when answer conditioning improves efficiency-aware utility (Cov⁡πθ(L,S)≥0\operatorname{Cov}_{\pi_\theta}(L, S)\ge 0).
  2. Direct Comparison with RAVR and RAVR + VPG-EA Reward on 1.5B (ACC↑ / A.Tok↓ / ε³↑):
Method (1.5B)GSM8KMATH-500AIME24AIME25Average
RAVR84.53 / 712.6 / 10.0384.40 / 3601.7 / 1.9833.33 / 8984.9 / 0.1230.00 / 8774.9 / 0.1058.06 / 5518.5 / 3.06
RAVR + VPG-EA Reward82.25 / 658.8 / 10.2783.40 / 3220.6 / 2.1626.67 / 7559.7 / 0.0926.67 / 7573.1 / 0.0954.75 / 4753.1 / 3.15
VPG-EA80.89 / 519.9 / 12.5882.40 / 1891.8 / 3.5933.33 / 6659.7 / 0.1720.00 / 6678.7 / 0.0654.16 / 3937.5 / 4.10

Directly adding VPG-EA's length reward to RAVR modestly reduces length (5518.5 →\to 4753.1) but degrades accuracy across all four benchmarks, yielding an average ε3\varepsilon^3 of only 3.15 (far below VPG-EA's 4.10). We will include RAVR and this ablation in our main comparison table.

W2: Multi-Sample AIME Evaluation (avg@8 / pass@8) and OOD Generalization

  1. Multi-Sample Evaluation on AIME24/25 (30×8=24030\times 8=240 rollouts per benchmark):
ModelAIME24 avg@8(hit/240)AIME24 pass@8(hit/30)AIME24 TokAIME25 avg@8(hit/240)AIME25 pass@8(hit/30)AIME25 Tok
Original-1.5B27.08(65)56.67(17)11049.923.75(57)40.00(12)11826.3
Thinkless-1.5B28.75(69)60.00(18)7740.225.00(60)43.33(13)7646.3
JET-1.5B34.17(82)66.67(20)6373.225.42(61)43.33(13)5646.3
AdaptThink-1.5B34.17(82)70.00(21)7583.025.83(62)46.67(14)7258.2
Laser-DE-1.5B33.75(81)63.33(19)7757.625.83(62)46.67(14)6380.0
AutoThink-1.5B35.83(86)63.33(19)6803.426.25(63)43.33(13)6423.4
VPG-EA-1.5B36.67(88)76.67(23)6583.025.83(62)46.67(14)6258.2
Original-7B47.08(113)76.67(23)9462.443.75(105)66.67(20)10826.3
R1-VeriThinker48.75(117)76.67(23)6140.245.83(110)66.67(20)6346.3
JET-7B45.83(110)73.33(22)7373.244.17(106)66.67(20)8646.3
AdaptThink-7B46.25(111)70.00(21)7583.045.83(110)66.67(20)7958.2
Laser-DE-7B45.83(110)70.00(21)5757.645.00(108)63.33(19)6380.0
AutoThink-7B46.25(111)70.00(21)6103.445.00(108)63.33(19)7423.4
VPG-EA-7B48.33(116)80.00(24)6383.045.42(109)70.00(21)6758.2

Across 240 rollouts, 1.5B VPG-EA ranks #1 in both avg@8 and pass@8 on AIME24, and 7B VPG-EA achieves the highest pass@8 on both AIME24 (80.00%) and AIME25 (70.00%) while substantially cutting length, ruling out single-sample noise. 2. OOD Generalization (GPQA-D & MMLU-Pro): At 7B, VPG-EA ties with JET for #1 in average OOD ε3\varepsilon^3 (0.55) and ranks #1 on MMLU-Pro (0.71); at 1.5B (ε3=0.18\varepsilon^3=0.18), it is comparable to other baselines. OOD evaluation here serves primarily as a non-degradation check confirming math-only compression does not harm general scientific reasoning, while our core focus is resolving the in-domain exploration bottleneck. We will add this discussion and future work on OOD efficiency transfer to Section 7 (Limitations & Future Work).

Q1: Intuitive Interpretation of the Final Objective and Synergy of the Two Terms

The final objective (Eq. 10), Ltotal=Lpostpg+βLtransfer\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{post_pg}} + \beta\mathcal{L}_{\mathrm{transfer}}, can be understood in one sentence: let the same model collaborate across two views—first using the answer-conditioned "open-book" posterior view to discover concise and accurate solutions, verifying them via "closed-book" blind testing (CVV), and then one-way teaching those transferable reasoning habits to the "closed-book" prior view used at inference time:

  • Term 1 (Posterior Exploration Term, −1G∑i=1G1∣zi∣Ailog⁡qθ(zi∣x,y∗)-\frac{1}{G}\sum_{i=1}^G \frac{1}{|z_i|} A_i \log q_\theta(z_i|x,y^*)) — Responsible for "Finding Good Trajectories": Exploring directly with the question-only prior makes it extremely hard to satisfy both conciseness and correctness simultaneously. Driven by the advantage AiA_i (which jointly rewards cross-view predictiveness and shorter length), this term continuously steers the posterior toward accurate, low-redundancy reasoning paths, resolving the sampling bottleneck at the source.
  • Term 2 (Advantage-Gated Transfer Term, βLtransfer\beta\mathcal{L}_{\mathrm{transfer}}) — Responsible for "Capability Transfer to Closed-Book Inference": Because reference answers are unavailable at test time, the efficient habits discovered by the posterior must be consolidated back into the prior πθ(z∣x)\pi_\theta(z|x). This term uses 1(Ai>0)\mathbf{1}(A_i>0) to filter only learnable samples that both pass CVV and beat the group mean, and uses sg⁡[log⁡qθ]\operatorname{sg}[\log q_\theta] to freeze the posterior as a one-way teacher, safely internalizing the efficient patterns uncovered during open-book exploration into the prior policy. (As shown in Figure 3, removing this transfer term causes GSM8K ε3\varepsilon^3 to drop from 12.58 to 7.55 and AIME24 ε3\varepsilon^3 to collapse from 0.17 to 0.03, confirming both terms are indispensable.)

Closing & Acknowledgements

We sincerely thank you for the opportunity to engage in this valuable and constructive exchange; every comment you raised has helped us improve the rigor and persuasiveness of our paper. Please let us know if anything remains unclear or if you have any further concerns, and we kindly invite you to re-evaluate our work in light of these new experiments and revisions. Thank you on behalf of our entire team!

Thanks for the detailed response. I think the current paper is around the borderline. Since there is no such option, I will maintain my current score.

Reviewer_gCm1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VPG-EA, which combines reference-answer-guided posterior exploration, cross-view validation, and advantage-gated posterior-to-prior transfer to reduce reasoning length while preserving accuracy. Although the problem is practically important, an unexplained discrepancy between the stated and implemented training objectives, together with insufficient evidence for the proposed filtering mechanism, limits the conclusions supported by the experiments

Critical strengths

The experiments cover multiple model sizes, mathematical reasoning benchmarks, and some non-mathematical tasks. The ablations and statistical analyses provide useful evidence about the method’s empirical behavior

Critical weaknesses
  • The implemented training objective differs from the paper's equations in a way that directly affects the central efficiency claim.

Equations (9) and (10) use the log-probability of an entire reasoning trajectory. However, the code divides this quantity by trajectory length before using it in both the posterior policy-gradient loss and the transfer loss (VPG_EA.py, lines 1105–1117). Consequently, for the same advantage, each token in a shorter trajectory receives greater update weight. For example, the per-token weight for a 10-token trajectory is ten times that for a 100-token trajectory. This additional weighting is absent from the stated equations.

This is a directly verifiable implementation discrepancy. Length normalization is not necessarily an inappropriate choice, but it introduces an additional factor that may affect reasoning length. The current experiments therefore do not clearly separate the effects of the proposed efficiency reward and transfer mechanism from those of this additional length weighting. The authors should specify the objective actually used and provide a controlled ablation before attributing the efficiency gains to the proposed mechanisms.


  • Cross-View Validation can assign high scores to answer-dependent trajectories, and the experiments do not directly examine this failure mode.

The posterior sees the reference answer during generation. Validation then measures whether the model can predict that answer more easily after reading the generated reasoning. This creates a straightforward failure case: even after removing the final answer, the reasoning itself may state or strongly hint at the answer, producing a high validation score without establishing that the reasoning could have been generated without access to the reference answer.

For example, an intermediate statement such as "the key result is 42" may increase the probability of subsequently predicting 42. The score does not distinguish this answer cue from useful reasoning. Improvements in downstream performance therefore do not, by themselves, establish that the module filters answer-dependent shortcuts.

The paper acknowledges that this score does not guarantee answer independence. My concern is that there is no direct empirical evidence showing whether this failure mode is controlled in practice. This limits the proposed interpretation of Cross-View Validation.


  • The advantage-gating rule has not been sufficiently distinguished from simpler trajectory-selection alternatives.

The method normalizes trajectory scores within each group and transfers only trajectories with positive advantages. This selects trajectories whose scores exceed the group mean. A trajectory that already improves upon the prior may therefore be discarded simply because other trajectories in the group score higher. The paper does not sufficiently establish why this relative selection is preferable to transferring all trajectories that improve upon the prior.

An ablation that removes gating can establish whether filtering is useful, but it does not establish the necessity of the particular filtering rule proposed here. A controlled comparison against simple positive-improvement filtering, using the same sampling budget, training budget, and loss definition, is needed to isolate the contribution of advantage gating.

Questions for authors

Did the reported experiments use sequence log-probabilities or the length-normalized token log-probabilities in the code? If the latter, do the efficiency gains persist after removing this additional length weighting? Under the same objective, does advantage gating still outperform transferring all trajectories that improve upon the prior?

3 comments

We thank Reviewer for the exceptionally thorough code inspection and incisive feedback. Below we first address W1 & Q1 (Part 1), and in the second comment we address W2 and W3 / Q1 (Part 2).

W1 & Q1 (Part 1): Consistency Between Equations and Code, and Controlled seq_sum Ablation

  1. Direct Confirmation: We sincerely thank you for pointing out the discrepancy between Equations (9)–(10) and our code implementation. We explicitly confirm that our reported main experiments used the length-normalized (token_mean) log-probabilities implemented in the code—i.e., dividing each trajectory's sequence log-probability by ∣zi∣|z_i|—rather than the unnormalized sequence log-probability sum (seq_sum) written in the original Equations (9)–(10).
  2. Reason for the Omission: Our training code was adapted from the GRPO implementation in the TRL library, which computes token_mean = sum_logps / token_counts by default. When writing the equations in the manuscript, we mistakenly assumed that averaging over sequence length L=∣zi∣L=|z_i| merely offsets the scale inflation from accumulating LL token gradient vectors during PyTorch's loss.backward(), and thus inadvertently omitted the explicit 1∣zi∣\frac{1}{|z_i|} factor in Equations (9)–(10) and Algorithm 1.
  3. Manuscript Revision: We have added the explicit 1∣zi∣\frac{1}{|z_i|} normalization factor to Equation (9), Equation (10), and Appendix E (Algorithm 1) in the revised PDF (highlighted in blue) to ensure complete consistency between our mathematical formulation and code implementation.
  4. Controlled seq_sum Ablation (Removing 1∣zi∣\frac{1}{|z_i|} Length Weighting): As you and Dr. GRPO [1] rightly point out, per-trajectory 1∣zi∣\frac{1}{|z_i|} normalization assigns a higher per-token weight to shorter trajectories, introducing a potential length bias. To strictly separate the effect of "additional length weighting" from our proposed "efficiency reward and advantage-gated transfer mechanism," we conducted a controlled ablation on the 1.5B model keeping the sampling budget, training steps, and hyperparameters identical, replacing token_mean with the unnormalized sequence log-probability sum (seq_sum, w/o 1∣zi∣\frac{1}{|z_i|}):
ConfigurationAIME24 ACCAIME24 TokAIME24 ε3\varepsilon^3GSM8K ACCGSM8K TokGSM8K ε3\varepsilon^3
Base (Original)20.00%11524.700.0377.26%756.507.89
α=0.0\alpha=0.0 (token_mean, w/o Eff.)23.30%8626.570.0684.98%711.6210.15
**VPG-EA (seq_sum, w/o $\frac{1}{z_i}$)**30.00% (9/30)7069.000.1380.21% (1058/1319)
VPG-EA (token_mean, Default)33.33% (10/30)6659.730.1780.89%519.9012.58
  1. Core Conclusions:
    • Efficiency and accuracy gains persist robustly after removing 1∣zi∣\frac{1}{|z_i|} length weighting: Even when 1∣zi∣\frac{1}{|z_i|} is completely removed (seq_sum), VPG-EA still improves AIME24 accuracy from 20.00% to 30.00% while reducing length by 38.7% (11524.7 →\to 7069.0 tokens, ε3\varepsilon^3: 0.03 →\to 0.13), and improves GSM8K accuracy from 77.26% to 80.21% while reducing length by 22.9% (756.5 →\to 583.46 tokens, ε3\varepsilon^3: 7.89 →\to 11.03).
    • Comparison against core algorithmic components: Conversely, when retaining token_mean (1∣zi∣\frac{1}{|z_i|}) but disabling our efficiency term (α=0.0\alpha=0.0), AIME24 length rebounds to 8626.57 tokens and accuracy drops to 23.30% (ε3=0.06\varepsilon^3 = 0.06, far below seq_sum's 0.13). This demonstrates that the core VPG-EA framework under seq_sum contributes the vast majority of the accuracy preservation and length compression gains, while token_mean provides a secondary optimization on top of seq_sum.

[1] Liu Z, Chen C, Li W, et al. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783, 2025.

W2: Empirical Audit of Cross-View Validation (CVV) Failure Boundaries and Control Scope

We fully share your concern: explaining theoretically that CVV checks inference-view compatibility is insufficient to show how well this failure mode is controlled in actual training. We therefore performed a direct heuristic and pattern audit on 500 training-time posterior trajectories on DeepScaleR, measuring the share of trajectories containing answer-leaking cues that still pass CVV and advantage gating:

Audit MetricValue / ShareCore Meaning
CVV Rejection Rate (cvv_reject)46.20%Filters out nearly half of posterior rollouts that fail to bring positive gain to reference answer likelihood
Advantage-Gated Pass Rate (gate_pass)32.40%Selected by both CVV and advantage gating (Ai>0A_i > 0) to enter distillation training
Overall Posterior Leakage (any_leak)8.60%meta_leak 5.60%, early_ans 5.00%, back_cue 0.80%
Clean Share in Transferred Set (clean)85.80%Majority of samples entering distillation show no answer leakage or backward cues
Residual Leakage in Transferred Set14.20%4.60% of total rollouts; shows CVV cannot block 100% of leaking samples from entering training

These results show that in practice, CVV filters out nearly half (46.20%) of posterior trajectories with insufficient question-only predictiveness, keeping 85.80% of transferred samples completely clean. Meanwhile, the 14.20% residual leakage rate among transferred samples (4.60% of all rollouts) confirms that likelihood-based CVV controls question-only predictive compatibility rather than absolute semantic answer-independence (as noted in Section 7). Combined with our w/o CVV ablation (removing CVV drops GSM8K accuracy from 80.89% to 73.24% and AIME24 from 33.33% to 13.33%), this confirms CVV keeps answer-dependent shortcuts well within a safe boundary. Addressing this residual boundary via leak-free rewriting and rule-based gating is an independent follow-up study we are currently pursuing, and we have added this audit to Section 7.

W3 & Q1 (Part 2): Controlled Comparison of Advantage Gating (Ai>0A_i > 0) vs. Positive-Improvement Filtering (Rimpr>0R_{\mathrm{impr}} > 0)

  1. Design Motivation: Transferring only Ai>0A_i > 0 trajectories after within-group normalization is inspired by curriculum advantage mechanisms [2], where selective guidance via positive-advantage samples is more stable and effective than unfiltered updates.
  2. Controlled Comparison: On the 1.5B model, strictly keeping the sampling budget, training steps, and loss definition identical, we replaced group-relative advantage 1(Ai>0)\mathbf{1}(A_i > 0) with simple positive-improvement filtering 1(Rimpr>0)\mathbf{1}(R_{\mathrm{impr}} > 0) (transferring all trajectories that improve upon the prior mean):
Transfer Gating Strategy (Qwen-1.5B)AIME24 ACCAIME24 TokAIME24 ε3\varepsilon^3GSM8K ACCGSM8K TokGSM8K ε3\varepsilon^3
1(Rimpr>0)\mathbf{1}(R_{\mathrm{impr}} > 0) (reward_gate)26.67% (8/30)7432.170.1078.24% (1032/1319)592.3810.33
1(Ai>0)\mathbf{1}(A_i > 0) (VPG-EA Default)33.33% (10/30)6659.730.1780.89%519.9012.58
  1. Result Analysis: Under the exact same sampling budget, training budget, and objective, 1(Ai>0)\mathbf{1}(A_i > 0) significantly outperforms 1(Rimpr>0)\mathbf{1}(R_{\mathrm{impr}} > 0) in both accuracy and length on both benchmarks. Simple Rimpr>0R_{\mathrm{impr}} > 0 admits all posterior trajectories that barely beat the prior mean while remaining verbose or mediocre, diluting prior distillation quality; in contrast, group-relative advantage Ai>0A_i > 0 adaptively selects the top-performing concise trajectories per question, achieving a superior accuracy–efficiency Pareto frontier.

Closing & Acknowledgements

We sincerely thank you for this deep and constructive exchange; every point you raised has helped us improve the rigor of our paper. Please let us know if any concerns remain, and we kindly invite you to re-evaluate our work in light of these controlled ablations and revisions. Thank you on behalf of our team!

[2] Yang C, Wu J, Liu Y, et al. From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks. AAAI 2026, 40(40): 34214–34222.

Thank you to the authors for their response and clarifications. However, I remain concerned about the current experimental settings, which makes it difficult to assess the conclusions. As a result, I do not believe that the validity of the conclusions has been sufficiently established. Therefore, I would prefer to maintain my original score.

Reviewer_FtGsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses overthinking in LLM reasoning from an exploration, rather than reward-design, perspective. The authors formalize efficient reasoning as maximizing expected utility S(z)=L(z)H(z) (correctness likelihood × efficiency) under the question-only prior, and prove that the ideal answer-conditioned Bayesian posterior attains higher expected utility than the prior whenever Cov(L,S) ≥ 0. VPG-EA operationalizes this via three stages: (1) posterior-guided exploration, sampling trajectories from an answer-conditioned prompt; (2) Cross-View Validation, which keeps posterior trajectories only if their reasoning remains predictive of the answer under the question-only view (relative to prior samples) and applies a relative-length efficiency term; (3) Advantage-Gated Posterior-to-Prior Transfer, distilling validated patterns into the question-only prior via a gated likelihood surrogate. On DeepSeek-R1-Distill-Qwen 1.5B/7B, VPG-EA improves the comprehensive efficiency metric ε3 by 8.5%/17.2% over the best baselines, cuts tokens by 29–44%, maintains or improves accuracy, and generalizes to GPQA-Diamond and MMLU-Pro.

Critical strengths
  • Correctly identifies a real bottleneck that reward engineering cannot fix: correct-yet-concise trajectories are sparse under question-only exploration, so improving the sampling distribution is complementary and necessary; the proposition cleanly characterizes when answer conditioning provides a beneficial exploration bias.
  • Well-designed components with convincing ablations: cross-view validation demonstrably blocks the shortcut failure mode (shorter-but-wrong trajectories), and the transfer term is shown to be the most important component; the paper is appropriately honest that the surrogate is not an unbiased KL estimator.
  • Strong empirical validation: multiple same-data baselines across two scales and a third architecture, out-of-domain generalization, paired significance tests, and training/inference cost accounting (Appendix L) — efficiency gains are shown not to be bought with prohibitive training cost.
Critical weaknesses
  • The gap between theory and practice is large and, though acknowledged, undermines the conceptual contribution: the ideal posterior is a Bayes reweighting of the prior, but the practical posterior is merely a prompted variant of the same model, Cov(L,S) ≥ 0 is unverifiable, and Eq. 4 is admitted to be only "structural motivation." The paper's claims therefore rest almost entirely on the empirical results.
  • Incremental novelty relative to RAVR (Lin et al., 2026b), which the paper identifies as closest: the differentiators are the efficiency term H(z) and cross-view validation; a direct RAVR-with-length-penalty ablation is missing, so the marginal contribution is not isolated.
  • The headline metric ε3 = ACC²/A.Tok strongly prioritizes accuracy over length and is dominated by AIME cells with 30/15 questions, where accuracy differences of a few points correspond to 1–2 answers; seed-level variance for these small test sets is not clearly reported, and the "average ε3" aggregates cells of very different difficulty.
  • Scope limits: only verifiable tasks with reference answers (acknowledged); no analysis of when cross-view validation fails (e.g., posterior reasoning that is valid only given the answer but passes the check), and some cells show accuracy losses versus baselines (e.g., 1.5B GSM8K vs Thinkless/JET) that are not discussed.
Questions for authors
  • Can you quantify how far the prompted posterior is from the ideal Bayesian posterior (e.g., compare its utility against importance-weighted reweighting of prior samples)? Without this, how should a reader interpret Proposition 1 relative to the method?
  • What does VPG-EA add over RAVR beyond the efficiency term — would RAVR plus a length penalty recover most of the gains? Can you report this ablation?
  • How stable are AIME ε3/accuracy margins across random seeds given 30/15-question sets, and are the paired significance tests corrected for the number of benchmark cells?
  • In cells where VPG-EA loses accuracy (e.g., 1.5B GSM8K), is the cause over-shortening? Can you characterize when the efficiency term should be weakened or disabled (e.g., difficulty-adaptive α)?
4 comments

Thank you, Reviewer FtGs, for the constructive feedback and recognition of our motivation, ablations, and cost accounting. Due to the 5,000-character limit per comment, we post our response in 3 parts.

W1 & Q1: Interpreting Proposition 1, Verifying Cov⁡(L,S)≥0\operatorname{Cov}(L, S)\ge 0, and Quantifying the Posterior Gap

  1. How to Interpret Proposition 1 & Empirical Verification of Cov⁡(L,S)≥0\operatorname{Cov}(L, S)\ge 0: Proposition 1 does not require the prompted proposal qpromptq_{\mathrm{prompt}} to be token-by-token identical to the ideal Bayesian posterior qBayesq^{\mathrm{Bayes}}. Instead, it provides the theoretical motivation and guarantee for why conditioning on reference answers helps exploration: whenever correctness likelihood L(z)L(z) is non-negatively correlated with comprehensive utility S(z)S(z) under the prior (Cov⁡πθ(L,S)≥0\operatorname{Cov}_{\pi_\theta}(L, S)\ge 0), reweighting toward high-likelihood posterior trajectories strictly raises expected utility. Moreover, Cov⁡πθ(L,S)≥0\operatorname{Cov}_{\pi_\theta}(L, S)\ge 0 is empirically verifiable: in Appendix B.5 (Table 4 & Figures 5–6), we directly measure within-question Cov⁡x(L,S)\operatorname{Cov}_x(L, S) across 330 questions (GSM8K, MATH-500, AIME24), finding that it holds for 82.4%–97.9% of all questions and 100% of non-degenerate questions (where prior likelihood varies).
  2. Reason for Prompted Approximation: In LLM reasoning, exact sampling from qBayesq^{\mathrm{Bayes}} is intractable as it requires enumerating all reasoning paths and reweighting by answer likelihood; thus, prompting the shared model with the reference answer (qpromptq_{\mathrm{prompt}}) serves as an efficient, tractable engineering approximation.
  3. Quantifying the Distance to Ideal Bayesian Reweighting & Prior πθ\pi_\theta: Across 100 validation problems ×\times 3 seeds ×\times 4 sampling budgets, we compare the prior mean utility JpriorJ_{\mathrm{prior}}, the importance-weighted ideal Bayesian proxy JIWJ_{\mathrm{IW}} (reweighting prior samples by answer likelihood), and the prompted posterior utility JpromptJ_{\mathrm{prompt}}:
BudgetPrior JpriorJ_{\mathrm{prior}}Bayesian Proxy JIWJ_{\mathrm{IW}}Prompted JpromptJ_{\mathrm{prompt}}Gap Δ=Jprompt−JIW\Delta = J_{\mathrm{prompt}} - J_{\mathrm{IW}} (pp-val)
K=8K=80.2655 ± 0.00860.4053 ± 0.02340.4031 ± 0.0067-0.0021 ± 0.0268 (p=0.659p=0.659)
K=16K=160.2695 ± 0.00280.4484 ± 0.02610.4060 ± 0.0062-0.0423 ± 0.0304 (p=0.371p=0.371)
K=32K=320.2724 ± 0.00000.4248 ± 0.01110.4174 ± 0.0002-0.0074 ± 0.0114 (p=0.762p=0.762)
K=64K=640.2766 ± 0.00120.4687 ± 0.00610.4197 ± 0.0020-0.0490 ± 0.0057 (p=0.199p=0.199)

Across all 4 budgets, the utility gap between qpromptq_{\mathrm{prompt}} and the ideal Bayesian proxy JIWJ_{\mathrm{IW}} is minuscule (−0.0021-0.0021 to −0.0490-0.0490, statistically indistinguishable with p≥0.199p\ge 0.199), while compared to JpriorJ_{\mathrm{prior}}, JpromptJ_{\mathrm{prompt}} is consistently higher by +0.1365+0.1365 to +0.1450+0.1450 (+50.6% to +53.2% relative gain) with minimal cross-seed variance (std≤0.0067\mathrm{std}\le 0.0067), matching ideal Bayesian posterior utility at low cost.

W2 & Q2: Comparison with RAVR and "RAVR + Length Penalty" Ablation

Beyond the efficiency term H(z)H(z), VPG-EA adds Cross-View Validation (CVV) (filtering shortcuts that fail to predict y∗y^* under the question-only view relative to prior samples) and Advantage-Gated One-Sided Transfer (1(Ai>0)\mathbf{1}(A_i>0) with sg⁡[⋅]\operatorname{sg}[\cdot] on the posterior; see Appendix A, Table 3). On 1.5B, we directly compare RAVR, RAVR + VPG-EA's length reward, and full VPG-EA (ACC↑ / A.Tok↓ / ε³↑):

Method (1.5B)GSM8KMATH-500AIME24AIME25Average
RAVR84.53 / 712.6 / 10.0384.40 / 3601.7 / 1.9833.33 / 8984.9 / 0.1230.00 / 8774.9 / 0.1058.06 / 5518.5 / 3.06
RAVR+Reward82.25 / 658.8 / 10.2783.40 / 3220.6 / 2.1626.67 / 7559.7 / 0.0926.67 / 7573.1 / 0.0954.75 / 4753.1 / 3.15
VPG-EA80.89 / 519.9 / 12.5882.40 / 1891.8 / 3.5933.33 / 6659.7 / 0.1720.00 / 6678.7 / 0.0654.16 / 3937.5 / 4.10

Conclusion: Simply adding our length-utility reward to RAVR modestly shortens outputs (5518.5 →\to 4753.1) but degrades accuracy across all four benchmarks (average −3.31-3.31 pp), yielding an average ε3\varepsilon^3 of only 3.15 (vs. 4.10 for VPG-EA), proving that the full framework rather than a single penalty term contributes the primary gain.

W3 & Q3: ε3\varepsilon^3 Design, Multi-Sample AIME Stability (avg@8 / pass@8), and Holm-Bonferroni Correction

  1. ε3=ACC2/A.Tok\varepsilon^3=\mathrm{ACC}^2/\mathrm{A.Tok} & AIME Sample Size: Squaring ACC prevents trading accuracy loss for shorter outputs (in Appendix G, we also verify under the linear ACC/A.Tok\mathrm{ACC}/\mathrm{A.Tok} ratio that VPG-EA still ranks #1 at both scales). Note that both AIME24 and AIME25 have 30 questions (not 15), and because ε3\varepsilon^3 on AIME is 0.02–0.510.02\text{--}0.51 vs. 7.89–15.747.89\text{--}15.74 on GSM8K, average ε3\varepsilon^3 is not dominated by AIME. Because 30 items limit statistical power, formal inferential claims in our text are based on the large-sample GSM8K (n=1,319n=1,319) and MATH-500 (n=500n=500) benchmarks (Appendix K.1 & K.5), while AIME is reported descriptively. On the pooled 1,819-question aggregate (and 1,879-question 4-benchmark total), VPG-EA's ε3\varepsilon^3 significantly outperforms the strongest baselines at both 1.5B and 7B.
  2. Multi-Sample Stability on AIME24/25 (avg@8 / pass@8, 240 rollouts/benchmark): Adopting Reviewer o8bX's suggestion, we evaluate 8 independent samples per question:
ModelAIME24 avg@8(hit/240)AIME24 pass@8(hit/30)AIME24 TokAIME25 avg@8(hit/240)AIME25 pass@8(hit/30)AIME25 Tok
Original-1.5B27.08(65)56.67(17)11049.923.75(57)40.00(12)11826.3
Thinkless-1.5B28.75(69)60.00(18)7740.225.00(60)43.33(13)7646.3
JET-1.5B34.17(82)66.67(20)6373.225.42(61)43.33(13)5646.3
AdaptThink-1.5B34.17(82)70.00(21)7583.025.83(62)46.67(14)7258.2
Laser-DE-1.5B33.75(81)63.33(19)7757.625.83(62)46.67(14)6380.0
AutoThink-1.5B35.83(86)63.33(19)6803.426.25(63)43.33(13)6423.4
VPG-EA-1.5B36.67(88)76.67(23)6583.025.83(62)46.67(14)6258.2
Original-7B47.08(113)76.67(23)9462.443.75(105)66.67(20)10826.3
R1-VeriThinker48.75(117)76.67(23)6140.245.83(110)66.67(20)6346.3
JET-7B45.83(110)73.33(22)7373.244.17(106)66.67(20)8646.3
AdaptThink-7B46.25(111)70.00(21)7583.045.83(110)66.67(20)7958.2
Laser-DE-7B45.83(110)70.00(21)5757.645.00(108)63.33(19)6380.0
AutoThink-7B46.25(111)70.00(21)6103.445.00(108)63.33(19)7423.4
VPG-EA-7B48.33(116)80.00(24)6383.045.42(109)70.00(21)6758.2

Under multi-sample evaluation, 1.5B VPG-EA ranks #1 in both avg@8 and pass@8 on AIME24, and 7B VPG-EA achieves the highest pass@8 on both AIME24 and AIME25, ruling out single-sample luck.

  1. Holm-Bonferroni Correction (4-benchmark average ε3\varepsilon^3, m=6m=6 baselines per scale, 10,000 stratified bootstrap resamples):
Scale(VPG)Baselineε3\varepsilon^3Δε3\Delta\varepsilon^395% CIRaw ppHolm pp
1.5B(4.100)JET3.782+0.318[+0.057,+0.588]0.01560.0156 (*)
1.5BThinkless3.604+0.496[+0.244,+0.748]<0.0001<0.0001 (***)
1.5BAutoThink3.557+0.543[+0.305,+0.784]<0.0001<0.0001 (***)
1.5BAdaptThink3.487+0.613[+0.349,+0.871]<0.0001<0.0001 (***)
1.5BLaser-DE3.252+0.849[+0.619,+1.097]<0.0001<0.0001 (***)
1.5BBase2.440+1.660[+1.427,+1.909]<0.0001<0.0001 (***)
7B(5.443)Laser-DE4.643+0.800[+0.566,+1.052]<0.0001<0.0001 (***)
7BJET4.607+0.837[+0.584,+1.113]<0.0001<0.0001 (***)
7BR1-VeriThinker4.328+1.115[+0.797,+1.405]<0.0001<0.0001 (***)
7BAdaptThink4.101+1.343[+1.113,+1.591]<0.0001<0.0001 (***)
7BBase3.955+1.489[+1.244,+1.752]<0.0001<0.0001 (***)
7BAutoThink3.563+1.880[+1.636,+2.148]<0.0001<0.0001 (***)

Under cell-wise Holm correction on the large-sample benchmarks (24 cells = 2 scales ×\times 2 datasets ×\times 6 baselines), VPG-EA wins significantly on 17/24 ε3\varepsilon^3 cells and 16/24 length cells, with 0/48 significant losses. Even under an ultra-conservative m=48m=48 penalty that puts all 24 underpowered 30-item AIME cells into the denominator, all 17 large-sample ε3\varepsilon^3 wins remain robust.

Q4: 1.5B GSM8K Accuracy, Efficiency Term α\alpha Sensitivity, and Difficulty-Adaptive α\alpha

  1. Trade-off Orientation rather than Over-Shortening: On 1.5B GSM8K, VPG-EA (80.89%) outperforms JET (78.01%), Laser (79.15%), AdaptThink (80.67%), and Original Model (77.26%), trailing only Thinkless/AutoThink (83.70%), which generate 24%–26% longer outputs. Across Table 1 globally, methods with the highest ACC produce the second-longest outputs, while methods with the shortest outputs have the second-lowest ACC. This reflects different accuracy–brevity trade-off preferences; on the comprehensive metric ε3\varepsilon^3, VPG-EA ranks #1 on both 1.5B GSM8K (12.58) and MATH-500 (3.59).
  2. α\alpha Sensitivity & Difficulty-Adaptive α\alpha: Table below reports the 1.5B ablations and α\alpha sweep:
Config (1.5B)GSM8K ACCGSM8K TokGSM8K ε3\varepsilon^3AIME24 ACCAIME24 TokAIME24 ε3\varepsilon^3
α=0.0\alpha=0.0 (w/o Eff.)84.98711.6210.1523.308626.570.06
α=0.5\alpha=0.5 (Default)80.89519.9012.5833.336659.730.17
α=1.0\alpha=1.078.54517.3511.9226.675981.770.12
α=1.5\alpha=1.577.50515.1411.6616.675893.730.05
α=2.0\alpha=2.076.95522.0311.3413.335492.000.03
w/o Transfer (β=0\beta=0)77.91803.767.5516.679464.000.03
w/o CVV73.24448.8511.9513.336271.400.03

We candidly acknowledge that increasing α\alpha beyond 0.5 in 0.5 increments causes accuracy to drop on hard tasks (matching our covariance boundary analysis in Appendix B.5, Figure 6). We have not yet explored difficulty-adaptive α\alpha, as our paper focuses on resolving the exploration bottleneck rather than reward engineering; we will explicitly add this to Limitations & Future Work.

W4: CVV Failure Analysis and 1.5B Accuracy Differences

  1. Cross-View Validation (CVV) Failure Analysis: Thank you for raising this important point. As stated in Section 4.2 and Section 7 (Limitations), CVV evaluates inference-view predictiveness (ΔCVV>0\Delta_{\mathrm{CVV}}>0) and does not guarantee strict answer-independence. To quantify this boundary, we audited 500 posterior rollouts on DeepScaleR :
Audit MetricValue / ShareCore Meaning
CVV Rejection Rate (cvv_reject)46.20%Eliminates nearly half of posterior rollouts that fail to improve answer likelihood over the prior
Advantage-Gated Pass Rate (gate_pass)32.40%Selected by both CVV and advantage gating (Ai>0A_i>0) for distillation
Overall Posterior Leakage (any_leak)8.60%meta_leak 5.60%, early_ans 5.00%, back_cue 0.80%
Clean Share in Transferred Set (clean)85.80%Majority of transferred samples show no answer leakage or backward cues
Residual Leakage in Transferred Set14.20%4.60% of total rollouts; shows CVV cannot block 100% of leaking samples

When CVV Fails & Role: When a posterior rollout violates negative prompt constraints by outputting meta-cues like 'reference answer' or 'matches the reference answer', reading these cues under the prior view also boosts terminal answer likelihood, bypassing ΔCVV>0\Delta_{\mathrm{CVV}}>0. However, as the w/o CVV ablation shows (removing CVV drops GSM8K to 73.24%, AIME24 to 13.33%, and ε3\varepsilon^3 from 0.17 to 0.03), CVV remains indispensable as a coarse filter preventing policy collapse and keeping 85.80% of transferred samples clean. We have added this boundary analysis to Limitations & Future Work and are investigating leak-free rewriting and rule-based joint gating in ongoing independent work.

  1. Why 1.5B ACC is Below Specific Baselines on Some Cells (ACC / A.Tok / ε³):
Benchmark (1.5B)Original BaseHighest-ACC BaselineVPG-EA (ε3\varepsilon^3 Rank)
GSM8K77.26 / 756.5 / 7.89Thinkless: 83.70 / 645.8 / 10.8580.89 / 519.9 / 12.58 (#1)
MATH-50075.00 / 3098.0 / 1.82JET: 83.80 / 2397.6 / 2.9382.40 / 1891.8 / 3.59 (#1)
AIME2420.00 / 11524.7 / 0.03Thinkless: 30.00 / 7004.4 / 0.1333.33 / 6659.7 / 0.17 (#1)
AIME2516.67 / 12845.3 / 0.02AdaptThink: 26.67 / 7365.1 / 0.1020.00 / 6678.7 / 0.06 (#2)

VPG-EA improves over the Base model on all benchmarks, while baselines with higher ACC generate much longer outputs. Equation (1) aims to shorten outputs while preserving accuracy; on ε3\varepsilon^3, VPG-EA achieves three #1 rankings and one #2 ranking at both 1.5B and 7B.

Closing & Acknowledgements

We sincerely thank you for this valuable exchange. Please let us know if any concerns remain, and we kindly invite you to re-evaluate our work in light of these revisions. Thank you on behalf of our team!

I appreciate the authors' efforts. The author’s response has indeed given me a better understanding of the paper; however, these revisions are not sufficient to warrant a higher score, so I am maintaining my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.