- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies efficient reasoning and proposes leveraging reference answers as privileged signals to guide exploration during training. The proposed training objective selects trajectories predictive under the question-only view and distills reasoning patterns into the question-only prior, demonstrating consistent accuracy-efficiency improvements.
Critical strengths
The paper is well-motivated and clearly written, giving a clear distinction between reward design and exploration. The logic flows naturally, making it easy to read and understand. I also find the idea of improving reasoning efficiency through better exploration, rather than relying solely on length penalties, interesting.
Critical weaknesses
- RAVR [1] is a close baseline and also leverages a similar variational approach and reward designs, using the "priors" and answer-conditioned "posteriors". It's important to clarify the key differences in terms of motivation and implementation and include it as a major baseline.
- On the math reasoning tasks, the results come from a single training run with one sample per question, which makes them noisy, especially on the 30-question AIME benchmarks. Reporting avg@k or pass@k would be more informative. On the OOD tasks, the proposed method performs similarly to other baselines, without clear gains.
[1]. Lin, T., Zhao, X., Zhang, X., Long, R., Xu, Y., Jiang, Z., Su, W., & Zheng, B. (2025). RAVR: Reference-Answer-guided Variational Reasoning for Large Language Models. ArXiv, abs/2510.25206.
Questions for authors
Can the authors give a more intuitive interpretation of the final objective? How do the two terms contribute to the final goal?
3 comments
Q1: Intuitive Interpretation of the Final Objective and Synergy of the Two Terms
The final objective (Eq. 10), , can be understood in one sentence: let the same model collaborate across two views—first using the answer-conditioned "open-book" posterior view to discover concise and accurate solutions, verifying them via "closed-book" blind testing (CVV), and then one-way teaching those transferable reasoning habits to the "closed-book" prior view used at inference time:
- Term 1 (Posterior Exploration Term, ) — Responsible for "Finding Good Trajectories": Exploring directly with the question-only prior makes it extremely hard to satisfy both conciseness and correctness simultaneously. Driven by the advantage (which jointly rewards cross-view predictiveness and shorter length), this term continuously steers the posterior toward accurate, low-redundancy reasoning paths, resolving the sampling bottleneck at the source.
- Term 2 (Advantage-Gated Transfer Term, ) — Responsible for "Capability Transfer to Closed-Book Inference": Because reference answers are unavailable at test time, the efficient habits discovered by the posterior must be consolidated back into the prior . This term uses to filter only learnable samples that both pass CVV and beat the group mean, and uses to freeze the posterior as a one-way teacher, safely internalizing the efficient patterns uncovered during open-book exploration into the prior policy. (As shown in Figure 3, removing this transfer term causes GSM8K to drop from 12.58 to 7.55 and AIME24 to collapse from 0.17 to 0.03, confirming both terms are indispensable.)
Closing & Acknowledgements
We sincerely thank you for the opportunity to engage in this valuable and constructive exchange; every comment you raised has helped us improve the rigor and persuasiveness of our paper. Please let us know if anything remains unclear or if you have any further concerns, and we kindly invite you to re-evaluate our work in light of these new experiments and revisions. Thank you on behalf of our entire team!
Thanks for the detailed response. I think the current paper is around the borderline. Since there is no such option, I will maintain my current score.
We sincerely thank Reviewer for the positive feedback on our motivation, writing clarity, and core idea of improving reasoning efficiency via better exploration rather than length penalties alone! Due to the 5,000-character limit per comment, we present our response in 2 parts.
W1: Key Differences from RAVR and Direct Experimental Comparison
ACC↑ / A.Tok↓ / ε³↑):Directly adding VPG-EA's length reward to RAVR modestly reduces length (5518.5 → 4753.1) but degrades accuracy across all four benchmarks, yielding an average ε3 of only 3.15 (far below VPG-EA's 4.10). We will include RAVR and this ablation in our main comparison table.
W2: Multi-Sample AIME Evaluation (
avg@8/pass@8) and OOD GeneralizationAcross 240 rollouts, 1.5B VPG-EA ranks #1 in both
avg@8andpass@8on AIME24, and 7B VPG-EA achieves the highestpass@8on both AIME24 (80.00%) and AIME25 (70.00%) while substantially cutting length, ruling out single-sample noise. 2. OOD Generalization (GPQA-D & MMLU-Pro): At 7B, VPG-EA ties with JET for #1 in average OOD ε3 (0.55) and ranks #1 on MMLU-Pro (0.71); at 1.5B (ε3=0.18), it is comparable to other baselines. OOD evaluation here serves primarily as a non-degradation check confirming math-only compression does not harm general scientific reasoning, while our core focus is resolving the in-domain exploration bottleneck. We will add this discussion and future work on OOD efficiency transfer to Section 7 (Limitations & Future Work).