- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
PTPO builds a library of about 7.9K theorems extracted by an LLM from solved problems, and rewrites each OpenR1-Math training solution into a “theorem-grounded CoT” that labels each step with the theorem it uses. During GRPO training, a prefix of this CoT is appended to the problem as a hint. The prefix length is set to 1 − α(q), where α(q) is the problem’s rollout success rate in the previous epoch, so harder problems receive more guidance and the hint is gradually withdrawn as the model improves. The reward remains the standard answer-correctness reward plus a format term. The authors report the best average accuracy on nine math benchmarks with two 1.5B backbones, gains over GRPO on Qwen3.5-2B and Qwen3-8B.
Critical strengths
- Adaptive hinting yields a large gain over GRPO in a matched setting. On OpenMath-Nemotron-1.5B, the full method raises the four-benchmark average from 74.97 (GRPO) to 80.77, and on Qwen3.5-2B from 38.03 to 49.54. The schedule is simple and adds no online retrieval cost, which makes it a practical curriculum for sparse-reward math RL.
Critical weaknesses
- The core mechanism is not new, and the closest prior work is neither cited nor compared. Adaptively revealing a prefix of the reference solution based on per-problem reward is essentially AdaBack (Amani et al., ICLR 2026). SEELE (Li et al., 2025) and UFT (Liu et al., NeurIPS 2025) are also closely related.
- The comparisons with external methods are not controlled, and the evaluation protocol is unclear. The baselines in Table 1 differ in training data, compute and length budget, and the margin over the most comparable method, QuestA, is only 0.78 points. The 30-question AIME/HMMT/BRUMO/CMIMC sets come with no reported variance. Fig. 4 tracks AIME 2024 during training, but the paper does not state how the final checkpoint was chosen. The claim of “fewer training resources” is not supported by any compute numbers.
Questions for authors
- How was the final checkpoint selected? Was AIME 2024 or any other evaluation benchmark used for checkpoint or hyperparameter selection?
- What is the training compute (GPU-hours, epochs, rollouts) of PTPO and the matched GRPO baseline?
2 comments
Thank you for the response. I have increased my soundness score.
We thank the reviewer for recognizing PTPO's gains over GRPO.
W1. Relationship to AdaBack, SEELE, and UFT
We agree that adaptive solution-prefix guidance has precedents. PTPO's main contribution is the integration of a structured theorem library, theorem-grounded solution rewriting, and problem-level guidance during RL.
AdaBack updates the hint-ratio sampling interval using per-problem rewards. SEELE uses multi-round sampling and item response theory to select hint lengths. UFT anneals prefix sampling with training progress. SEELE and UFT also supervise hint tokens. PTPO constructs theorem-grounded guidance offline and sets the next-epoch hint ratio to 1−α(q) using each problem's success rate from existing training rollouts, without an auxiliary supervised loss on hint tokens. We will add these references and clarify these distinctions.
The generic-CoT control uses original OpenR1-Math solutions with the same progressive schedule and filtering/resampling as PTPO. Across training seeds 1, 17, and 42, PTPO outperforms generic CoT on all four benchmarks, improving average accuracy by 1.81 percentage points (Table 1). This supports the additional benefit of the theorem-grounded guidance pipeline under a shared schedule.
Table 1. Accuracy (%) across three training seeds: mean ± sample SD.
Avg. is the four-benchmark mean within each seed, then aggregated across seeds.
We additionally compare PTPO directly with AdaBack scheduling. Both methods use OpenMath-Nemotron-1.5B and train on the same 11,853 training records (Table 2). PTPO outperforms AdaBack on all four benchmarks, increasing average accuracy from 75.82% to 80.80%, a gain of 4.98 percentage points.
**Table 2. Direct comparison with AdaBack **
W2. External comparisons and evaluation uncertainty
External baselines in the paper's Table 1 are reported results with different training data, compute, and token budgets; we will label their sources and settings explicitly.
Alongside the reported 0.78-point average margin over QuestA, we highlight the training setup. QuestA reports "eight NVIDIA H800 (80GB) nodes," 2,000 training steps, batch size 128, and 16 rollouts per prompt. PTPO achieves competitive performance on four RTX A6000 GPUs in approximately 48 hours. QuestA uses 50%→25% solution prefixes with repeated difficulty filtering; PTPO adjusts each problem's hint length using existing rollout feedback, avoiding these stage transitions and associated re-filtering.
All our benchmark evaluations are unhinted: temperature 0.7, top-p 0.9, and at most 31,744 generated tokens. We sample 32 responses per question on AIME24/25, AMC23, HMMT25, BRUMO25, and CMIMC25, and 4 on MATH-500, OlympiadBench, and Minerva. Accuracy is the mean per-question fraction of correct responses. We will correct Section 5.1 accordingly.
For HMMT25 and BRUMO25, we report the accuracy of our PTPO model across 32 decoding seeds. Table 3 summarizes fixed-checkpoint results; Table 1 reports training-seed variability.
Table 3. Accuracy (%) across 32 decoding seeds: mean ± sample SD.
Q1. Checkpoint selection
We use the final checkpoint after 10 epochs. Neither AIME 2024 nor any other evaluation benchmark was used for checkpoint or hyperparameter selection.
Q2. Training compute
PTPO and the matched GRPO baseline each use four RTX A6000 GPUs for 10 epochs, taking approximately 48 hours (192 GPU-hours) per 1.5B run. Both use learning rate 10−6, batch size 64, mini-batch size 16, and 16 rollouts per problem per sampled group. PTPO's offline theorem extraction, library construction, and CoT rewriting cost approximately US$50 in API fees; RL training reuses cached guidance without online retrieval or teacher API calls.