acceptodds
Under review as a conference paper at ICLR 2027

PTPO: Progressive Theorem-Guided Policy Optimization for LLM-Based Mathematical Reasoning

Abstract

Despite recent great advances in mathematical reasoning, large language models still struggle with competition-level problems that require selecting appropriate mathematical theorems and applying them under the correct conditions. Existing reinforcement learning methods mainly optimize final-answer correctness or generic reasoning trajectories, often providing limited guidance on the theorem-level principles. To address this limitation, we propose Progressive Theorem-guided Policy Optimization (PTPO), a theorem-guided reinforcement learning framework with input-adaptive progressive prompt construction. We first construct a structured theorem library and generates theorem-grounded chain-of-thought prompts that explicitly connect problems with relevant mathematical principles. Based on the theorem library, we then adaptively inject theorem-grounded prompts according to the question difficulty and model's current ability, offering rich hints for hard questions and progressively reducing hints as the model improves. This progressive schedule encourages the model to transition from external theorem guidance to autonomous theorem application. Extensive experiments on nine benchmarks show that our proposed PTPO achieves the best overall performance, and it enables faster convergence and stronger reasoning. Our source code and theorem library are available at https://anonymous.4open.science/r/PTPO-FED5.

Primary area:
reinforcement learning
Keywords:
computer science, reinforce learning, large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yYrQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

PTPO builds a library of about 7.9K theorems extracted by an LLM from solved problems, and rewrites each OpenR1-Math training solution into a “theorem-grounded CoT” that labels each step with the theorem it uses. During GRPO training, a prefix of this CoT is appended to the problem as a hint. The prefix length is set to 1 − α(q), where α(q) is the problem’s rollout success rate in the previous epoch, so harder problems receive more guidance and the hint is gradually withdrawn as the model improves. The reward remains the standard answer-correctness reward plus a format term. The authors report the best average accuracy on nine math benchmarks with two 1.5B backbones, gains over GRPO on Qwen3.5-2B and Qwen3-8B.

Critical strengths
  1. Adaptive hinting yields a large gain over GRPO in a matched setting. On OpenMath-Nemotron-1.5B, the full method raises the four-benchmark average from 74.97 (GRPO) to 80.77, and on Qwen3.5-2B from 38.03 to 49.54. The schedule is simple and adds no online retrieval cost, which makes it a practical curriculum for sparse-reward math RL.
Critical weaknesses
  1. The core mechanism is not new, and the closest prior work is neither cited nor compared. Adaptively revealing a prefix of the reference solution based on per-problem reward is essentially AdaBack (Amani et al., ICLR 2026). SEELE (Li et al., 2025) and UFT (Liu et al., NeurIPS 2025) are also closely related.
  2. The comparisons with external methods are not controlled, and the evaluation protocol is unclear. The baselines in Table 1 differ in training data, compute and length budget, and the margin over the most comparable method, QuestA, is only 0.78 points. The 30-question AIME/HMMT/BRUMO/CMIMC sets come with no reported variance. Fig. 4 tracks AIME 2024 during training, but the paper does not state how the final checkpoint was chosen. The claim of “fewer training resources” is not supported by any compute numbers.
Questions for authors
  1. How was the final checkpoint selected? Was AIME 2024 or any other evaluation benchmark used for checkpoint or hyperparameter selection?
  2. What is the training compute (GPU-hours, epochs, rollouts) of PTPO and the matched GRPO baseline?
2 comments

We thank the reviewer for recognizing PTPO's gains over GRPO.

W1. Relationship to AdaBack, SEELE, and UFT

We agree that adaptive solution-prefix guidance has precedents. PTPO's main contribution is the integration of a structured theorem library, theorem-grounded solution rewriting, and problem-level guidance during RL.

AdaBack updates the hint-ratio sampling interval using per-problem rewards. SEELE uses multi-round sampling and item response theory to select hint lengths. UFT anneals prefix sampling with training progress. SEELE and UFT also supervise hint tokens. PTPO constructs theorem-grounded guidance offline and sets the next-epoch hint ratio to 1−α(q)1-\alpha(q) using each problem's success rate from existing training rollouts, without an auxiliary supervised loss on hint tokens. We will add these references and clarify these distinctions.

The generic-CoT control uses original OpenR1-Math solutions with the same progressive schedule and filtering/resampling as PTPO. Across training seeds 1, 17, and 42, PTPO outperforms generic CoT on all four benchmarks, improving average accuracy by 1.81 percentage points (Table 1). This supports the additional benefit of the theorem-grounded guidance pipeline under a shared schedule.

Table 1. Accuracy (%) across three training seeds: mean ± sample SD.

MethodAIME24AIME25AMC23MATH-500Avg.
GRPO64.07 ± 0.2552.33 ± 0.4290.15 ± 0.4192.76 ± 0.3274.83 ± 0.21
Generic CoT68.02 ± 0.1760.54 ± 0.1794.00 ± 0.1693.39 ± 0.1178.99 ± 0.03
PTPO71.08 ± 0.1662.20 ± 0.1296.02 ± 0.2593.90 ± 0.1380.80 ± 0.05

Avg. is the four-benchmark mean within each seed, then aggregated across seeds.

We additionally compare PTPO directly with AdaBack scheduling. Both methods use OpenMath-Nemotron-1.5B and train on the same 11,853 training records (Table 2). PTPO outperforms AdaBack on all four benchmarks, increasing average accuracy from 75.82% to 80.80%, a gain of 4.98 percentage points.

**Table 2. Direct comparison with AdaBack **

MethodAIME24AIME25AMC23MATH-500Avg.
AdaBack60.5056.6793.6092.5075.82
PTPO71.0862.2096.0293.9080.80

W2. External comparisons and evaluation uncertainty

External baselines in the paper's Table 1 are reported results with different training data, compute, and token budgets; we will label their sources and settings explicitly.

Alongside the reported 0.78-point average margin over QuestA, we highlight the training setup. QuestA reports "eight NVIDIA H800 (80GB) nodes," 2,000 training steps, batch size 128, and 16 rollouts per prompt. PTPO achieves competitive performance on four RTX A6000 GPUs in approximately 48 hours. QuestA uses 50%→25% solution prefixes with repeated difficulty filtering; PTPO adjusts each problem's hint length using existing rollout feedback, avoiding these stage transitions and associated re-filtering.

All our benchmark evaluations are unhinted: temperature 0.7, top-p 0.9, and at most 31,744 generated tokens. We sample 32 responses per question on AIME24/25, AMC23, HMMT25, BRUMO25, and CMIMC25, and 4 on MATH-500, OlympiadBench, and Minerva. Accuracy is the mean per-question fraction of correct responses. We will correct Section 5.1 accordingly.

For HMMT25 and BRUMO25, we report the accuracy of our PTPO model across 32 decoding seeds. Table 3 summarizes fixed-checkpoint results; Table 1 reports training-seed variability.

Table 3. Accuracy (%) across 32 decoding seeds: mean ± sample SD.

BenchmarkQuestionsMean ± SD
HMMT253038.02 ± 4.05
BRUMO253065.00 ± 4.15
CMIMC254041.22 ± 3.89

Q1. Checkpoint selection

We use the final checkpoint after 10 epochs. Neither AIME 2024 nor any other evaluation benchmark was used for checkpoint or hyperparameter selection.

Q2. Training compute

PTPO and the matched GRPO baseline each use four RTX A6000 GPUs for 10 epochs, taking approximately 48 hours (192 GPU-hours) per 1.5B run. Both use learning rate 10−610^{-6}, batch size 64, mini-batch size 16, and 16 rollouts per problem per sampled group. PTPO's offline theorem extraction, library construction, and CoT rewriting cost approximately US$50 in API fees; RL training reuses cached guidance without online retrieval or teacher API calls.

Thank you for the response. I have increased my soundness score.

Reviewer_9b34OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PTPO builds a structured theorem library, rewrites reference solutions into theorem-grounded chains of thought, and adaptively injects their prefixes during RL according to each problem's previous rollout accuracy. It reports improvements across nine mathematical benchmarks, with additional experiments on 2B and 8B models.

Critical strengths
  • The cached theorem repository and simple problem-level guidance rule provide a practical way to support difficult training examples without online retrieval or additional teacher calls during RL.
  • Experiments cover multiple backbones and include useful component, generic-CoT, retrieval-depth, and random-theorem controls. The controlled comparisons show consistent improvements over standard GRPO.
Critical weaknesses
  • The theorem-specific mechanism is only partly established. The advantage over generic CoT is 1.75 points (Table 4), without reported uncertainty or explicit hint-length matching. The judge-estimated theorem counts in Appendix A.3 measure breadth rather than application correctness, as acknowledged, and offer only indirect support for better theorem reasoning.
  • The ability estimate is computed from already hinted rollouts (Algorithm 1). Success with a long reference-solution prefix need not indicate unassisted competence; reducing hints could reverse that success. The paper does not show whether hint ratios actually decrease or whether this feedback oscillates.
  • Important reproduction and efficiency details are missing: the extraction/rewriting model, RL learning rate, batch size, and end-to-end preprocessing/training cost. Appendix A.1's retrieval latency alone does not substantiate the resource-efficiency claim, and the evaluation protocol should explicitly state whether test prompts contain any hints.
Questions for authors
  1. Are all benchmark results obtained without theorem or solution hints? Can you provide full training settings and comparisons matched for teacher, hint length, and total computation?
  2. How do per-problem hint ratios and independently measured unhinted accuracy evolve? Does estimating ability from occasional unhinted rollouts improve the curriculum?
2 comments

We thank the reviewer for the constructive comments.

Q1. Evaluation protocol, training settings, and comparisons

All benchmarks are evaluated without theorem or solution hints using the final epoch-10 checkpoint, with no benchmark-based checkpoint or hyperparameter selection.

For benchmark evaluation, we use temperature 0.7, top-p 0.9, and a 31,744-token generation limit. We sample 32 responses per question for AIME24/25, AMC23, HMMT25, BRUMO25, and CMIMC25, and 4 for MATH-500, OlympiadBench, and Minerva. Accuracy is the mean per-question fraction of correct responses.

For 1.5B training, all methods use learning rate 10−610^{-6}, batch size 64, mini-batch size 16, 16 rollouts per problem, and 10 epochs. Each GRPO, generic CoT, and PTPO run takes approximately 48 hours on four NVIDIA RTX A6000 GPUs (192 GPU-hours). Generic CoT and PTPO share progressive prompting and filtering: all-correct/all-incorrect groups are discarded and resampled to refill the batch.

For data construction, generic CoT uses original OpenR1-Math solutions, while PTPO uses deepseek-v4-flash for theorem extraction, library construction, and CoT rewriting. Full reference CoTs average 4,000 tokens for generic CoT and 3,500 for PTPO; actual hints are adaptively selected prefixes. Table 2 reports PTPO's hint usage.

For computational cost, offline preprocessing costs approximately US$50 in API fees. RL reuses cached guidance without online retrieval or additional teacher calls. API fees exclude GPU costs. External baselines in Table 1 are from prior reports, not rerun under matched budgets.

Across seeds 1, 17, and 42, PTPO outperforms both baselines on all benchmarks (Table 1), using shorter reference CoTs at comparable RL training costs.

Table 1. Accuracy (%) across three training seeds (mean ± SD).

MethodAIME24AIME25AMC23MATH-500Avg.
GRPO64.07 ± 0.2552.33 ± 0.4290.15 ± 0.4192.76 ± 0.3274.83 ± 0.21
Generic CoT68.02 ± 0.1760.54 ± 0.1794.00 ± 0.1693.39 ± 0.1178.99 ± 0.03
PTPO71.08 ± 0.1662.20 ± 0.1296.02 ± 0.2593.90 ± 0.1380.80 ± 0.05

Q2. Hint dynamics and occasional unhinted feedback

The success rate α(q)\alpha(q) measures correctness under the current hint and determines subsequent hint allocation. At α(q)=0\alpha(q)=0, PTPO provides the full reference CoT, including the answer. Success rates are recorded before filtering, including all-correct/all-incorrect groups. Each epoch contains 11,853 instances. Unhinted accuracy is independently evaluated on held-out AIME.

Table 2. Training hint usage and unhinted AIME accuracy (%).

EpochMean actual hint ratioFull-CoT rateUnhinted AIME accuracy
254.1424.6360.25
535.5513.3768.56
1033.5611.1471.08

Hint ratios and full-CoT exposure decline as unhinted accuracy improves. Errors still occur with full-CoT guidance. We observed no all-incorrect → all-correct → all-incorrect sequence across three consecutive visits to the same problem.

We further tested unhinted feedback in a single-seed OpenMath-Nemotron-1.5B continuation study. A reuses the original PTPO run; B resumes the complete epoch-5 state (optimizer, scheduler, and α\alpha memory) through epoch 10. In B, 25% of groups per batch are randomly selected before generation for 16 extra unhinted rollouts per problem. Their accuracy updates α\alpha; other groups retain the original rule.

Both methods retain 16 regular rollouts and the same filtering/resampling rules. B's extra probes use the pre-update model and affect only future hint allocation, not policy updates, filtering, or batch filling.

Table 3. Unhinted accuracy (%) at epoch 10.

MethodAIME24AIME25AMC23MATH-500Avg.Time (h)
A: Standard feedback71.0862.1096.0193.9080.7723
B: Unhinted feedback72.4062.8696.1294.1881.3924

Unhinted feedback improves all four benchmarks, raising average accuracy by 0.62 percentage points at one additional hour (4.3%) over epochs 6–10. This supports occasional unhinted ability estimates as a useful refinement of PTPO's hint scheduling.

Thank you for the detailed response and experiments. I have raised my score.

Reviewer_pyaGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces PTPO, a reinforcement learning approach to improve mathematical reasoning in LLMs. The main idea is to provide theorem-based reasoning hints during training and gradually reduce them as the model gets better at solving problems. The authors build a library of 7,875 mathematical theorems and use it to generate hints based on how well the model performs on each problem. They evaluate the method on nine math benchmarks using models of different sizes (1.5B, 2B, and 8B). The results show that PTPO generally performs better than standard GRPO and is competitive with other existing methods. The paper also includes ablation studies to examine how different components contribute to the overall performance.

Critical strengths

S1. The idea is simple: give more theorem hints on hard problems and fewer as the model improves. Everything is precomputed offline, so it adds almost no overhead on top of GRPO.

S2. PTPO has the best average on nine benchmarks for both 1.5B models and beats GRPO at 2B and 8B, with the biggest gains on weaker models.

S3. The ablations are thorough and cover each component, retrieval depth, and random theorems. The code and theorem library are also released.

Critical weaknesses

W1. It is unclear whether the gains come from theorems specifically. Random theorems (78.13) and generic CoT (79.02) already get most of the way from GRPO (74.97) to PTPO (80.77). The theorem-specific gain is ~1.7 points from a single run.

W2. When α=0\alpha=0, the full rewritten reference solution, answer included, goes into the prompt. Also, α\alpha is measured on the hinted prompt, so a problem can flip between no hint and the full solution from one epoch to the next. Neither issue is analyzed.

W3. Some numbers do not add up: Table 3 (77.59) vs Table 4 (79.02) for what looks like the same setting, and Table 5’s Top-0 exactly matches GRPO despite supposedly using progressive prompting.

W4. No seeds or variance are reported, and the number of samples averaged per test problem is unclear.

Questions for authors

Q1. Does the theorem-specific gain hold over multiple seeds (GRPO vs random theorems vs generic CoT vs PTPO)?

Q2. How often is the full solution/answer injected during training, and does the hint ratio actually go down over epochs or oscillate?

Q3. How does PTPO compare to simply giving question-retrieved theorems to the base or GRPO model at inference time?

Q4. Were the Table 1 baselines re-run? What is PTPO’s training compute? And What explains the Table 3/4/5 mismatches in W3?

Q5. Which LLM built the theorem library and CoTs, from what corpus, at what cost, and were the theorems checked for correctness?

2 comments

We thank the reviewer for the constructive comments.

Q1. Robustness across training seeds

PTPO outperforms GRPO and generic CoT on all four benchmarks across three training seeds (Table 1), supporting the benefit of theorem-grounded guidance over the original-data CoT baseline. Due to RL retraining costs, we prioritized these comparisons.

Table 1. Accuracy (%) across training seeds.

MethodSeedAIME24AIME25AMC23MATH-500Avg.
GRPO164.0952.3990.5592.8674.97
GRPO1763.8252.7190.1893.0274.93
GRPO4264.3151.8889.7392.4174.58
Generic CoT168.0860.5494.0593.4179.02
Generic CoT1767.8360.7193.8293.4978.96
Generic CoT4268.1660.3894.1293.2778.98
PTPO171.0862.1096.0193.9080.77
PTPO1770.9262.3495.7794.0380.76
PTPO4271.2362.1696.2793.7880.86

Q2. Full-solution exposure and hint dynamics

Each epoch contains 11,853 training instances. When α(q)=0\alpha(q)=0, PTPO provides the full reference CoT, including the answer. Success rates are recorded before filtering; thus, all-correct and all-incorrect groups also inform subsequent hint allocation.

Table 2. Training hint usage and unhinted AIME accuracy (%).

EpochMean actual hint ratioFull-CoT rateUnhinted AIME accuracy
254.1424.6360.25
535.5513.3768.56
1033.5611.1471.08

Hint usage and full-CoT exposure decline as unhinted accuracy improves. Incorrect responses also occur despite access to the full reference CoT and answer. In the examined trajectories, we observed no all-incorrect → all-correct → all-incorrect sequence over three consecutive visits to the same problem.

Q3. Inference-time theorem retrieval

We evaluated Base, GRPO, and PTPO with and without Top-5 theorem retrieval (Table 3). Retrieval uses only questions, without reference solutions or answers. All six settings share test questions and decoding configurations; the three retrieval-enabled models use identical retrieved theorems.

Table 3. Accuracy (%) with and without inference-time theorem retrieval.

ModelTop-5 at inferenceAIME24AIME25AMC23MATH-500Avg.
BaseNo58.7548.4490.5592.4072.54
BaseYes56.6747.6089.4590.3571.02
GRPONo64.0952.3990.5592.8674.97
GRPOYes64.1351.6989.9693.1274.73
PTPONo71.0862.1096.0193.9080.77
PTPOYes67.0861.1596.0493.9579.56

Retrieval does not improve average accuracy for any model. PTPO without inference-time hints remains strongest, indicating that retrieval alone cannot reproduce PTPO's gains in this setting.

Q4. Baselines, training compute, and ablation settings

External baselines in the paper's Table 1 are cited from prior reports, not rerun. We will explicitly identify their sources.

Our 1.5B PTPO and GRPO experiments used four NVIDIA RTX A6000 GPUs, 10 epochs, batch size 64, mini-batch size 16, and 16 rollouts per problem. Each run took approximately 48 hours (192 GPU-hours).

The paper's Tables 3–5 use the following settings:

  • Paper Table 3, 77.59: GRPO with group filtering and resampling, without CoT hints. All-correct and all-incorrect groups are discarded and replaced with additional samples.
  • Paper Table 4, 79.02: The same filtering and resampling, with progressive hints from original OpenR1-Math CoTs.
  • Paper Table 5, Top-0: Plain GRPO without hints or additional filtering/resampling. Its identical score reflects this distinction; it is not a controlled retrieval-depth ablation. We will relabel it as GRPO and rerun Top-0 under the same settings as other Top-k variants.

Q5. Theorem-library construction and verification

We used the deepseek-v4-flash API for theorem extraction and CoT rewriting. The corpus comprised OpenR1-Math training problems and reference solutions. An LLM translated each theorem's symbolic representation into Lean 4 for verification; all entries passed. Total offline API cost was approximately US$50.

Thank you for your responses and additional experiments. The multi-seed results, inference-time comparisons, and methodological clarifications address some of my original concerns. However, I remain concerned that the theorem-specific contribution is not sufficiently isolated from the effects of generic CoT guidance and LLM-based rewriting. The continued exposure to full reference solutions during training also raises concerns. While I acknowledge the improvements and have increased my soundness score, I will maintain my overall rating of 2.

Comments on the paper

1 comment

Dear Reviewers 9b34 and pyaG,

As we are approaching the end of the discussion period, could you read the authors' rebuttal and see if it has addressed your concerns and questions? Thanks again for your time!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.