- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to use RL to train LLMs to acquire and apply knowledge from textbooks when solving mathematical proof problems. The authors construct a textbook-based proof task and introduce Book Navigator, which allows models to search and read relevant textbook sections during reasoning.
The results show substantial improvements on mathematical proof tasks and some gains on two cross-task benchmarks.
Critical strengths
1. Good results on mathematical proof tasks.
The method achieves clear improvements over No-Book RL and fixed-retrieval baselines, with good performance on unseen textbooks and mathematical subjects.
2. Thorough experimental analysis.
The paper compares different textbook conditions and includes ablation studies on the exploration bonus and tool credit. These experiments help explain how textbook access and the different reward components contribute to the performance gains.
Critical weaknesses
1. Limited independent validation of the proof evaluation.
The paper uses GLM-5.2 to judge proof correctness, both for RL training rewards and final evaluation. Although the authors use eight independent judgments, I am still concerned about the reliability of LLM-as-Judge for informal mathematical proofs, especially without human evaluation or other independent validation.
2. Limited comparison with related methods.
The paper mainly compares against No-Book and fixed-retrieval baselines. While the authors include an ablation with correctness-only GRPO, they do not provide direct comparisons with existing interactive retrieval RL methods that use step-level rewards or credit assignment, such as StepSearch, SLATE, and IG-Search. As a result, it is difficult to assess how much the proposed reward design improves upon existing approaches.
Questions for authors
Q1. Reliability of Proof Evaluation
Have the authors evaluated the reliability of GLM-5.2 in judging informal mathematical proofs, for example through human evaluation? If so, could you report the agreement rate between the judge and human evaluators? Since the same judge is used for both training rewards and final evaluation, I am concerned about the potential risk of reward hacking.
Q2. Comparison with Existing Methods
Could the authors compare the proposed approach with existing methods that use step-level rewards or credit assignment under the same experimental setting? If direct comparisons are difficult, could you clarify how the proposed approach differs from these methods and what advantages it offers?
2 comments
Thank you for the detailed response.
Q1 (Proof evaluation). The human calibration study and the re-evaluation with GPT-6-astra and Claude-Opus-5.5 as independent judges are convincing, and they address my concerns about the reliability of the proof evaluation and the risk of reward hacking.
Q2 (Comparison with related methods). Thank you for clarifying the differences from StepSearch, SLATE, and IG-Search in terms of step-level signals and credit assignment. The explanation is reasonable, although direct empirical comparisons would further strengthen the paper.
Remaining concern. The improvements on BrowseComp-Plus and CL-bench over the base model are only 1.7 and 2.3 points, so the cross-task gains are limited, and the claim of a domain-agnostic self-learning strategy seems somewhat overstated.
Since my main concern has been addressed, I will raise my score.
Response to Reviewer WAWL: Independent Evaluation and Interactive Retrieval RL
Thank you for recognizing the mathematical results and the attribution analyses. We address independent validation and the comparison with interactive retrieval RL below.
W1/Q1: Human calibration and independent judges. We calibrated judge accuracy before finalizing the evaluation protocol. Four senior mathematics PhD researchers annotated 90 base-model proofs following the protocol of AdvancedMathBench (Kong et al., 2026). The sample contained 30 proofs each from ID-Book, OOD-Book, and OOD-Subject, yielding 49 correct and 41 incorrect proofs. GLM-5.2 judged each proof eight times with the reference solution. Against these human labels, TPR and TNR measure accuracy on correct and incorrect proofs, respectively; Bal. F1 denotes their harmonic mean:
The pessimistic rule balances acceptance of valid proofs with rejection of invalid ones and achieves strong agreement with human experts (Bal. F1: 92.1%), supporting its use for supervision. We attribute this reliability to three factors: (1) the capacity of the 744B GLM-5.2 to evaluate textbook-level proofs generated by InternS2-35B; (2) reference solutions grounding its judgments; and (3) pessimistic aggregation improving TNR (Huang et al., 2025). Together, these results support GLM-5.2 as a reliable source of proof-correctness supervision.
Independent reevaluation supports our conclusion, with no clear sign of judge-specific reward hacking. We reevaluated the main results with GPT-6-astra and Claude-Opus-5.5 as independent judges. Overall accuracy (%):
Across all Base/+RL entries, scores change by at most 3.4 points. Crucially, Self-Learning retains both the highest post-RL accuracy and the largest RL gain: +28.0/+28.6/+28.9 points under GLM/GPT/Claude. The main conclusion therefore persists beyond the training judge.
W2/Q2: Differences from related interactive retrieval RL. We are happy to clarify how our method differs from prior approaches, as suggested. The key differences lie in supervision and credit assignment:
Our design targets sustained, useful exploration of long textbooks. It requires neither gold intermediate retrieval paths nor gold-answer likelihood calculations for tool credit, and evaluates how actual tool returns are used in subsequent reasoning. Centering preserves the trajectory's token-weighted mean advantage while redistributing credit across rounds. The correctness-gated bonus sustains exploration; conservative credit then discourages harmful or redundant calls while preserving reasonable search attempts. This accommodates multiple valid reading paths in open-ended proof construction.
We have added human calibration and independent-judge reevaluation in Appendix D.1–D.2 (Tables 7–8), and expanded the comparison with related interactive retrieval RL methods in Section 4.