acceptodds
Under review as a conference paper at ICLR 2027

PO: Joint Policy and Prompt Optimization

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) enhances Large Language Model (LLM) reasoning but is suffer from advantage collapse: when all rollouts of a query receive identical rewards, the group variance vanishes, most damagingly on hard samples, where scaling rollout budgets yields little. We introduce Joint Policy and Prompt Optimization (PO) to mitigate this collapse by alternating continuous policy updates with discrete prompt evolution. PO mines hard samples with a success-rate threshold, evolves reasoning prompts for them with GEPA, and internalizes the elicited trajectories via context distillation, which optimizes each trajectory under the original query and thus removes inference-time prompting, with a Context Ratio Mask (CRM) filtering out extreme likelihood ratios. PO restores critical advantage signals and surpasses the GRPO baseline by up to 8.2 points in average accuracy on six held-out benchmarks across all training datasets and backbones, while also outperforming DAPO and other baselines. The gains are especially pronounced on hard benchmarks, reaching up to 16.3 points above GRPO on average across AIME24 and AIME25. Our findings expose the limits of standard exploration in sparse-reward environments, illuminating the potential of unifying evolutionary algorithms with reinforcement learning. This integration of discrete semantic search and continuous parameter updates provides a self-reinforcing framework that facilitates more effective LLM alignment.

Primary area:
generative models
Keywords:
RLVR, LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UT3iOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces P²O. The idea of jointly optimizing prompts and model parameters is insightful, and the method is sufficiently mature. However, some supporting evidence is missing and should be added.

Critical strengths
  • The goal of this work is well motivated: jointly optimizing prompts and model parameters is desirable, and this problem has not been adequately addressed by others.
  • Context Distillation precisely targets an aspect overlooked by related methods.
  • The discussion of off-policy / distribution mismatch makes the method more complete and the paper more insightful.
  • The performance improvements are significant.
Critical weaknesses
  • The paper does not discuss the effect of optimizing only the prompt side, nor does it provide examples showing to what extent GEPA itself can improve performance. If introducing prompt optimization during training leads to substantial gains, the paper should show where these gains come from and rule out the effect of using prompt optimization alone. In addition, a single piece of data generally includes a question and a prompt. Only optimizing the prompt's upper limit is somewhat concerning, because I don't think merely optimizing prompts like "reasoning step by step" can bring much benefit. I hope the author demonstrates this through case studies. These are real limitations, but despite that, I think this work is insightful.

  • Most of the experiments focus on accuracy and appear relatively late in the paper, while the motivation of the method is not sufficiently supported by evidence in the earlier sections. Some of these points may reasonably be treated as common knowledge in a submission version, but they should be supplemented in a later version. For example, training only with improved prompts may have poor generalization, and the authors should demonstrate this empirically. Similarly, the Context Ratio Mask should be supported by a quantitative analysis of the degree of off-policy / distribution mismatch. These analyses should ideally be presented before the accuracy experiments to improve the readability of the paper, especially for less experienced readers.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer UT3i

Thank you for recognizing the value of joint policy and prompt optimization and for suggesting clearer evidence for the method's motivation.

W1(a). What does prompt optimization alone achieve? We evaluated GEPA with the initial policy weights frozen. On DeepMath-5K/Qwen3-4B, the following results use the same 500-example evaluation set:

MethodOptimized componentsEvolved templates at evaluationAccuracy (%)
BaseNoneNo41.20
GEPA-onlyPromptsYes55.20
GRPOPolicyNo72.20
P²OPolicy and promptsNo77.00

With frozen weights, GEPA improves accuracy by 14.00 points over Base. It uses the training-time search configuration, with the rollout cap removed and early stopping after eight search steps without dev improvement.

W1(b). What does joint training add beyond prompt optimization alone?

On the same evaluation set, P²O exceeds GEPA-only by 21.80 points and GRPO by 4.80 points while using original queries at evaluation. Its final performance therefore cannot be explained by retaining optimized inference-time prompts: the policy has learned to perform better without them.

W1(c). What do the templates contribute beyond “think step by step”? Generic step-by-step guidance is already included in the input for the base model, GRPO and P²O, all run in non-thinking mode. Please refer to our response to Reviewer MoGR, Part 1, Q3, for this shared setting. The evolved templates add substantive problem-solving guidance. In Figure 7, the question asks for the smallest sphere enclosing four unit spheres. The bare-query response returns the incorrect radius 1+21+\sqrt{2}. The evolved template directs the model to consider regular-polyhedron configurations and supplies the tetrahedron's centroid-to-vertex relation, R=a3/8R=a\sqrt{3/8}. With center-to-center edge length a=2a=2, the guided response obtains the correct enclosing radius 1+6/21+\sqrt{6}/2. The useful intervention is this geometric structure and relation, beyond a generic reasoning instruction.

This case illustrates the mechanism. Please refer to our response to Reviewer MoGR, Part 2, W2(d)/Q5, for aggregate evidence from the corrected Table 6 analysis. Across 1,980 historical hard records, templates raise mean reward from 0.1495 to 0.2419 and nonzero-variance groups from 876 (44.24%) to 1,025 (51.77%). Context distillation trains the assisted trajectories under the original query. The R1-hint and single-turn-reflection controls further support evolving guidance over the tested simpler alternatives. Please refer to our response to Reviewer MoGR, Part 1, W1(a)/Q2 and W1(d)/Q4, for these comparisons.

Prior work also demonstrates substantial gains from prompt optimization alone: GEPA improves Qwen3-8B’s six-benchmark score from 45.23 to 54.85, while OPRO (Yang et al., ICLR 2024) raises PaLM 2-L’s GSM8K accuracy from 71.8% to 80.2%.

W2. Supporting Evidence and Presentation

Thank you for the suggestion. We will revise the manuscript to present the method’s motivation more clearly and better highlight the relevant supporting evidence. Specifically, we address the concerns about generalization and distribution mismatch as follows:

  1. Generalization without optimized prompts. Table 2 already provides a matched ablation addressing this concern. Optimizing template-assisted trajectories under the augmented context rather than the original query reduces template-free benchmark accuracy from 65.2% (P$^2$O) to 55.6% (P$^2$O$_{\textnormal{w/o-Context-Distillation}}$), below GRPO’s 60.5%. This supports the importance of context distillation for transferring prompt-assisted gains to the original-query setting.

  2. Quantifying context-induced mismatch. We measure the original-query to template-augmented token probability ratio in Eq. 6 and track the fraction of response tokens outside the CRM interval [0.01, 10] throughout training.

    The excluded fraction rises from 0.22% at step 40 to 2.10% at step 200, as shown below:

    StepResponse tokens excluded by CRM (%)
    400.22
    600.28
    800.38
    1000.64
    1200.88
    1400.94
    1601.29
    1801.74
    2002.10

    In very rare cases late in training, the context ratio exceeds 2 million, indicating severe mismatch between template-assisted rollouts and the original-query learning context and include training instability. CRM excludes tokens outside [0.01, 10] from the policy objective, retaining most response tokens while filtering these outliers. Please refer to our response to Reviewer MoGR, Part 2, W2(c), for the collapse evidence. Together, these results support CRM's role in stabilizing context distillation.

    We will make these analyses more prominent when introducing the corresponding components, improving the connection between the method’s motivation, design, and empirical evidence.

Thank you for the detailed response and additional analyses. These results largely address my concerns about the contribution of prompt optimization and the motivation for context distillation.

My understanding is that the GEPA-evolved templates provide hint-like privileged information during training, and P²O distills the benefits of this guidance into the policy so that it can perform better without the templates at inference time. I find this a useful and well-motivated approach. The prompt-only baseline and context-distillation ablation help clarify this interpretation.

The use of the non-thinking setting does reduce my confidence somewhat, as it remains unclear how much of the benefit would carry over to models operating in thinking mode. Nevertheless, I find the method clever and its progression from prompt-assisted exploration to context distillation coherent. Overall, I remain inclined to accept the paper and maintain my accept rating.

AuthorsOpenReview ↗

Response to Reviewer UT3i

Thank you for the thoughtful follow-up. Your interpretation captures our intended mechanism: GEPA-evolved templates provide auxiliary, hint-like guidance during training, and context distillation updates the policy under the original query so that it can benefit from this guidance without the templates at inference time.

To address the concern about dependence on a hybrid thinking/non-thinking backbone, we added an experiment using Qwen2.5-7B-Instruct on DeepMath-5K. Qwen2.5-7B-Instruct is a conventional instruction-tuned model without a separate thinking mode. We compare P²O with GRPO-8 on the same six benchmarks, with P²O evaluated on original queries without evolved templates.

All scores are accuracy (%); AVG is the unweighted mean across the six benchmarks.

MethodAIME24AIME25AMCMATH500Minerva MathOlympiadBenchAVG
GRPO-811.407.9050.9077.2038.6039.9037.65
P²O17.9010.4059.1077.4040.1041.3041.03

P²O raises average accuracy from 37.65% to 41.03%, a gain of 3.38 percentage points, and achieves higher scores on all six benchmarks.

These results demonstrate that P²O's benefits also hold for a conventional Instruct model, addressing the concern that a hybrid thinking/non-thinking backbone is necessary for the observed improvement. The gains are achieved without evolved templates at inference, consistent with your interpretation that prompt-assisted exploration and context distillation translate training-time guidance into improved policy performance.

Based on the experimental results reported in our previous responses and the reviewers' feedback, we have submitted a revised manuscript. The revision updates the experimental results and conclusions, corrects Table 6 and its associated analyses, and includes additional experiments. All changes are highlighted in blue for ease of review. We invite you to review the revised manuscript.

Reviewer_9GwAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes P²O to mitigate advantage collapse on hard samples in reinforcement learning with verifiable rewards. It alternates policy optimization with GEPA-based prompt evolution and uses context distillation to internalize prompt-induced reasoning without requiring optimized prompts at inference time. A Context Ratio Mask filters extreme likelihood ratios. Experiments on six mathematical reasoning benchmarks show up to an 8.2-percentage-point improvement in average accuracy over GRPO.

Critical strengths
  1. Addresses an important exploration problem in RLVR. P²O combines prompt evolution with policy optimization to improve learning on hard samples where standard GRPO suffers from advantage collapse.
  2. Effective context distillation without inference-time prompt dependency. The method internalizes prompt-induced reasoning into model parameters, with ablation results demonstrating the importance of context distillation.
  3. Strong empirical performance across multiple settings. P²O achieves consistent improvements across different model sizes and training datasets, particularly on challenging benchmarks such as AIME, outperforming GRPO and other baselines.
Critical weaknesses
  1. Insufficient baselines to isolate GEPA's contribution. Although single-turn reflection is evaluated, comparisons with simpler prompt selection or fixed prompt strategies under comparable computational budgets are missing.
  2. Inconsistency in the reported reward variance. The variance values in Table 6 appear inconsistent with the stated within-query variance calculation, weakening the evidence for recovering non-degenerate advantages.
  3. Limited explanation of Self-Ref's superiority. Although Appendix G provides indirect evidence, the claim that Teacher-Ref induces trajectories that are harder to internalize lacks direct measurements of their distance from the student policy.
  4. Mixed effectiveness of CRM. Table 3 shows that CRM reduces accuracy in some Qwen3-4B settings, and its relationship with model size and training stability is not sufficiently validated.
Questions for authors
  1. Could the authors compare GEPA with simpler prompt optimization methods, such as fixed prompts or random search, under the same context distillation procedure and computational budget?
  2. Could the authors clarify how the reward variance in Table 6 is calculated and report the actual within-query variance and proportion of groups with non-degenerate advantages?
  3. Could the authors provide direct evidence that Teacher-Ref trajectories are harder to internalize, such as trajectory log-likelihood or distribution-distance measurements under the original student policy?
  4. Could the authors report how the number of hard samples changes across epochs when evaluated using original queries without optimized prompts, to better distinguish prompt-assisted success from actual policy improvement?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9GwA

Thank you for recognizing the exploration problem and the value of context distillation. We address each concern below.

W1/Q1. Contribution of prompt evolution. Please refer to our response to Reviewer MoGR, Part 1, W1(a)/Q2, for the controlled R1-hint baseline, which replaces evolved templates with supplied R1 reasoning traces while keeping context distillation, CRM and the remaining training procedure unchanged. On DeepMath-5K/Qwen3-4B, it scores 56.5%, compared with 50.4% for single-turn reflection, 63.0% for P²O-Self-Ref and 58.5% for P²O-Teacher-Ref. These results support iterative prompt evaluation and refinement over the tested guidance substitutes.

W2/Q2. Correcting reward variance. We found two errors in Table 6 in the Appendix: pooled variance was reported as within-query variance, and thinking was inadvertently enabled. We reran both conditions with thinking disabled. Please refer to our response to Reviewer MoGR, Part 2, W2(d)/Q5, for the full corrected table. Across the fixed 1,980 records, mean reward increases from 0.1495 to 0.2419, within-query variance from 0.0831 to 0.0987, and nonzero-variance groups from 876 (44.24%) to 1,025 (51.77%). Among 1,098 all-incorrect bare-query groups, 394 (35.88%) have non-degenerate advantages with templates. The 1,479-query near-collapsed subgroup shows a 2.25-fold variance increase. These results support the role of evolved guidance in alleviating advantage collapse. We will update Table 6 and Appendix E.

W3/Q3. Direct trajectory-likelihood evidence. We froze the same student checkpoint and used the same questions and decoding settings to generate six responses per query under Self-Ref and Teacher-Ref templates. Both sets of responses are generated by the student; the teacher supplies templates. We then score response tokens under the original query, without the template:

ℓ(y∣x)=1∣y∣∑tlog⁡πθ(yt∣x,y<t).\ell(y\mid x)=\frac{1}{|y|}\sum_t\log\pi_\theta(y_t\mid x,y_{<t}).

The diagnostic includes all response tokens and excludes prompt/padding tokens.

Template sourceLog-likelihood (nat/token)
Self-Ref−2.117077
Teacher-Ref−2.521148

Teacher-Ref trajectories receive 0.404 nat/token lower likelihood under the original-query student policy. This directly supports lower compatibility with that policy. Table 7 (w/o CRM) shows the same inversion: Teacher-Ref has higher mean dev-hard accuracy (24.4% vs. 16.2%) but lower downstream accuracy (57.9% vs. 64.2%). Together, these observations support the internalization explanation. The three-seed results on DeepMath-5K/4B also show higher mean downstream accuracy for Self-Ref: 62.52 ± 0.53%, versus 59.38 ± 1.49% for Teacher-Ref, a 3.14-point difference. Please refer to our response to Reviewer MoGR, Part 2, W2(a)/Q1, for the full multi-seed results.

W4. CRM: accuracy and training stability.

We selected the two Qwen3-4B configurations in Table 3 where the original single-run result was higher without CRM than with CRM: DeepMath-5K/Self-Ref and DeepScaler-5K/Teacher-Ref. We reran the w/o-CRM variants with three seeds to check whether that apparent advantage persists. On DeepMath-5K, Self-Ref with CRM reaches 62.52 ± 0.53%, versus 61.52 ± 3.33% without it; all three unmasked runs collapsed late in training. On DeepScaler-5K, Teacher-Ref w/o CRM averages 63.45 ± 1.72%, versus the 64.0% with CRM. The reruns no longer show the original mean advantage without CRM.

Q4. Policy improvement without templates. We tracked a fixed set of 1,980 initially mined hard samples across checkpoints, using only original queries, non-thinking mode and six fresh rollouts per query. A sample remains hard if at most one rollout succeeds.

StepAll incorrectAt most one successMean reward
40108514880.147811
8068810100.349327
1205657890.452609
1604806680.526936
2005476920.520034

From step 40 to 200, the number of hard samples falls from 1,488 to 692 and mean reward rises from 0.147811 to 0.520034, showing improved policy performance without templates.

Reviewer_9GwAOpenReview ↗

Response to Author

Thank you for the additional experiments. The new results address most of my previous concerns, particularly regarding reward variance, trajectory internalization, and policy improvement without optimized prompts. However, onw concern remain:

GEPA's contribution. The R1-hint baseline is helpful, but it does not directly show that GEPA's evolutionary search is better than simpler alternatives. A comparison with random prompt search or fixed reasoning prompts under a matched computational budget would help isolate its contribution.

Overall, I appreciate the authors' efforts to address the previous concerns. The paper is substantially improved, although above point remain insufficiently addressed.

AuthorsOpenReview ↗

Response to Reviewer 9GwA

Thank you for this constructive suggestion and for acknowledging that the additional experiments address your concerns about reward variance, trajectory internalization, and policy improvement without optimized prompts. We agree that a fixed-prompt control helps clarify the contribution of GEPA's prompt evolution.

On DeepMath-5K/Qwen3-4B, we added P²O-Random-Template, which replaces GEPA-evolved templates with a fixed pool of 27 generic mathematical reasoning templates generated by an LLM before training. For each hard training sample, we randomly select one template from this pool. The templates remain fixed throughout training, while context distillation and CRM are retained. One example template is:

Start by translating the problem into a precise target: an exact value, a set of solutions, a bound, or a statement to prove or disprove. Record the domain and the hypotheses that constrain the answer. Identify a short chain of intermediate claims that would imply the target, and work through them in order. Keep exact expressions whenever possible. Before finishing, check that the result answers the original question and respects every restriction. Present a concise derivation and place the final answer in \boxed{}.

All scores are accuracy (%); AVG is the unweighted mean across the six benchmarks.

MethodAIME24AIME25AMCMATH500Minerva MathOlympiadBenchAVG
GRPO-834.134.179.988.541.056.955.8
P²O-Single-Turn-Ref27.122.174.285.641.252.150.4
R1 hints + CD + CRM (GHPO-inspired)34.430.484.889.842.657.056.5
P²O-Random-Template38.531.780.688.440.856.456.07
P²O-Self-Ref53.040.685.890.843.861.262.5

P²O-Self-Ref achieves 62.5% average accuracy versus 56.07% for P²O-Random-Template, a gap of 6.43 percentage points. This gap highlights the essential role of prompt optimization in achieving P²O's full gains: combining iterative prompt refinement with policy optimization produces greater improvements than policy training with fixed templates alone.

Based on the experimental results reported in our previous responses and the reviewers' feedback, we have submitted a revised manuscript. The revision updates the experimental results and conclusions, corrects Table 6 and its associated analyses, and includes additional experiments. All changes are highlighted in blue for ease of review. We invite you to review the revised manuscript.

Reviewer_MoGROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets a known weakness of GRPO-style RL with verifiable rewards: when every rollout of a hard prompt fails, the group-normalized advantage is zero and the prompt contributes no learning signal.

P2O alternates two phases. First, after each epoch it marks prompts solved at most once in six rollouts as hard, and uses GEPA to evolve instruction templates that make them solvable more often, with reflection by the frozen pre-RL model (Self-Ref) or by Kimi-K2 (Teacher-Ref). Second, in the next epoch it rolls out hard prompts with a template but takes the gradient as if only the bare prompt had been seen (context distillation), so no template is needed at inference. A Context Ratio Mask drops tokens whose likelihood differs too much between the two contexts.

On math reasoning with Qwen3-4B and Qwen3-8B, P2O beats GRPO in all six dataset and model settings, with the largest gains on AIME. The ablations show that context distillation is essential, since training with the template left in the context ends below plain GRPO, and that the model's own reflections work better than the stronger teacher's.

Critical strengths

S1. A simple, practical mechanism, backed by one convincing ablation. P2O gives hard prompts a learning signal again and leaves nothing extra at inference. The key experiment (Table 2): training on the templated question and its answer, (x', y'), scores 55.6, which is 4.9 points below plain GRPO (60.5). Scoring the same answers as if only the bare question was seen scores 65.2. So adding hints during RL can actively hurt unless the gradient is taken in the inference context. That is a useful, general lesson.

S2. Large, consistent gains, with rollouts matched where it matters most. P2O beats GRPO-6 in all six dataset/backbone settings, by +3.9 to +8.2 average. On the two 5K sets with Qwen3-4B, AIME24 improves by +12.1 and +21.5, and P2O also beats the rollout-matched GRPO-8 and GRPO-12. The teacher result (Table 7) is surprising and informative: the teacher's templates make hard dev problems more solvable (+8.2 mean dev-hard accuracy), yet the model trained with them ends 6.3 points lower (57.9 vs 64.2).

Critical weaknesses

W1. The experiments do not test the central claim, that evolving prompts with GEPA is the right way to rescue hard prompts.

The natural alternative, a simple hint under the same context distillation and mask, is never run. Prior hint- and guidance-based methods for hard prompts (QuestA, BREAD, LUFFY, GHPO, Adaptive Guidance, Critique-GRPO) are cited but not compared, although partial-solution hints are cheap here: DeepMath-103K ships three R1 solutions per problem. Work that alternates prompt optimization with weight updates is not cited (BetterTogether, Soylu et al., EMNLP 2024; Ziems et al., 2025). The two cheaper variants the paper does run, Single-Turn-Ref and Teacher-Distill-SFT, end only 1.9-4.5 points above the base model and 4.4-12.7 points below GRPO-6, which suggests the results depend heavily on template quality. Compute is also matched only for the 4B 5K settings, and the rollout count does not charge reflection calls (Q7).

W2. The statistical and measurement support is thinner than the claims.

Every number is a single run with best-checkpoint selection, and GRPO alone moves 4.1 points across its own budgets (60.5 vs 56.4 on DeepScaler-5K). The headline gains probably survive this, but most secondary claims sit within it: Self-Ref beats Teacher-Ref by only 0.3-1.2 in three of four 4B settings, and the mask averages about zero on 4B (though it helps by 2.4-3.4 on 8B in all four settings). Two pieces of evidence also do not show what they claim. For the mask, the 4B run without it scores higher in Table 9 (65.2 vs 64.0), and the collapse in Figure 3 appears only in training reward over the last 20 or so of 230 steps, not in validation. For the mechanism, every "within-group variance" value in Table 6 equals mean x (1 - mean), the pooled variance of the bucket, so the column adds nothing beyond the mean. Under the per-group definition the 0.2494 value is impossible: 28.2% of groups have zero variance, which caps the average at 0.18.

Questions for authors

Q1. Seeds. Can you report mean and standard deviation over at least 3 seeds for GRPO-6, GRPO-8 and P2O-Self-Ref on one 5K setting?

Q2. Hint baseline. Can you add one hint baseline with the same context distillation and mask, such as QuestA-style partial solutions or a fixed hand-written strategy prompt? Does GEPA's evolution add anything over a reasonable hint?

Q3. Thinking mode. The base-model scores (Qwen3-4B: 23.8/19.4 on AIME24/25) match the Qwen3 report's non-thinking mode (25.0/19.1), not its thinking mode (73.8/65.6). Which mode did you use? If non-thinking, how much of the gain comes from templates such as Figure 7's "Think step-by-step" switching reasoning back on?

Q4. Weak variants. What exactly do Single-Turn-Ref and Teacher-Distill-SFT run, and why do both end near the base model?

Q5. Table 6. Please report the fraction of hard-sample groups with nonzero within-group variance, with and without templates, and reconcile the 0.4765 mean with the at-most-one-in-six hard-set definition.

Q6. Training length. 10 epochs at N = 5,000 and batch 128 is about 390 steps, but Figures 3-5 end at 230-240 steps. How many steps do the reported checkpoints have, and is the dev set used to select them disjoint from GEPA's 300-sample dev split?

Minor suggestions (do not affect the score).

(a) The DeepScaler-5K Qwen3-4B Self-Ref average should be 64.4, not 64.7, in Tables 1 and 9 and in the dependent text (+4.2, 0.7, 2.3).

(b) Fix the Figure 3 legend ("w/o-TRM") and the Table 2 Olympiad bold (63.0 is the column maximum, not 62.2), and label Table 2's "P2O" row as w/o-CRM.

4 comments
AuthorsOpenReview ↗

Response to MoGR Part-1

Thank you for the detailed review.

W1(a)/Q2. Does evolved guidance improve over a reasonable hint?

We added a control that replaces GEPA templates with reasoning traces sampled from each problem's three supplied R1 solutions. Context distillation, CRM and the remaining training components are unchanged. This GHPO-inspired control isolates the guidance source within P²O framework. Results on DeepMath-5K/Qwen3-4B are:

MethodAIME24AIME25AMCMATH500MinervaOlympiadAVG
GRPO-835.628.379.789.040.156.454.9
Single-Turn-Ref27.122.174.285.641.252.150.4
R1 hints + CD + CRM (GHPO-inspired)34.430.484.889.842.657.056.5
P²O-Self-Ref55.340.086.291.243.062.463.0
P²O-Teacher-Ref46.532.582.388.641.559.358.5

CD denotes context distillation. Self-Ref and Teacher-Ref exceed the R1-hint control by 6.5 and 2.0 points. Single-Turn-Ref tests one reflection per problem instead of iterative search (GEPA). These controls support evaluating and refining guidance as the student policy evolves. The variants are explained separately under Q4.

W1(b). Relation to earlier joint prompt and weight optimization.

We will cite both works and distinguish the shared ideas from P²O's contribution.

BetterTogether [1] uses SFT on filtered successful trajectories, while its prompt optimization selects successful self-generated trajectories as few-shot demonstrations. Its approach thus centers on trajectory selection and imitation, akin to rejection sampling. P²O instead combines reflective prompt evolution with on-policy RL to recover learning signals on the current policy's hard samples.

The combined method in Ziems et al. [2] optimizes prompts first and keeps them fixed during subsequent mmGRPO training. P²O iteratively alternates policy learning and prompt evolution, allowing guidance to adapt to the policy's changing hard samples and support further improvement.

[1] Soylu et al. Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together. EMNLP 2024.

[2] Ziems et al. Multi-module GRPO: Composing Policy Gradients and Prompt Optimization for Language Model Programs. 2025.

W1(c)/Q7. Reflection-call accounting and the 10K-data comparison.

Appendix B and Table 5 include reflection calls, mini-batch rollouts and development evaluations in GEPA's 10K-generation cap. However, Algorithm 3 in the appendix omitted explicit deductions for reflection and the initial empty-template evaluation. We will correct the pseudocode by initializing the budget before template initialization, since initialization itself consumes budget, and deducting both costs.

We added GRPO-7 results on DeepMath-10K/Qwen3-4B, matching the per-epoch generation budget of P²O with six rollouts per query on this 10K dataset:

MethodSix-benchmark AVG (%)
GRPO-757.39
P²O-Self-Ref60.60
P²O-Teacher-Ref60.33

The gains are 3.21 and 2.94 points under the same generation-count budget.

Q3. Thinking mode and generic reasoning instructions.

Training and benchmark evaluation use non-thinking mode. The base model, GRPO and P²O all receive “Please reason step by step, and put your final answer within \boxed{}.” during training and evaluation. Thus, generic step-by-step guidance is already present in the baseline.

The templates also contain substantive guidance. Figure 7 supplies a regular-tetrahedron construction and its centroid-to-vertex relation for a sphere-packing problem. Please refer to our response to Reviewer UT3i, W1(c), for the detailed case study.

W1(d)/Q4. What do the two weaker variants test?

Single-Turn-Ref. It tests whether one reflection can substitute for iterative candidate evaluation and selection. At each epoch's end, each hard sample receives one self-reflection to obtain a template, replacing GEPA while retaining the P²O training procedure. It scores 50.4/47.8% on DeepMath/DeepScaler-5K, compared with the base model's 45.9%. Its limited gains support the importance of prompt optimization.

Teacher-Distill-SFT. It provides static teacher supervision while consuming substantially more teacher tokens than P²O-Teacher-Ref. The teacher generates two answers per training problem; ground-truth verification filters them, and the student is fine-tuned on accepted answers. It achieves 50.1/47.9% on DeepMath/DeepScaler-5K , versus Teacher-Ref's 58.5/64.0%, despite using about 14M teacher tokens versus <0.5M (Appendix B). By comparison, P²O-Teacher-Ref makes more efficient use of teacher supervision, achieving higher accuracy.

AuthorsOpenReview ↗

Response to MoGR Part-2

W2(a)/Q1. Results across random seeds.

We report three independent training seeds. Values are mean ± sample SD of six-benchmark accuracy (%).

Across seeds, the main performance trends and conclusions hold. CRM's benefit is clearer: both 4B ablations that previously favored w/o CRM now favor CRM; W2(c) details these comparisons.

Dataset / modelMethodMean ± SD
DeepScaler-5K / 4BGRPO-659.71 ± 2.20
DeepScaler-5K / 4BGRPO-858.70 ± 3.27
DeepScaler-5K / 4BTeacher-Ref w/o CRM63.45 ± 1.72
DeepMath-5K / 4BGRPO-654.62 ± 1.05
DeepMath-5K / 4BGRPO-855.78 ± 0.82
DeepMath-5K / 4BSelf-Ref62.52 ± 0.53
DeepMath-5K / 4BTeacher-Ref59.38 ± 1.49
DeepMath-5K / 4BSelf-Ref w/o CRM61.52 ± 3.33

On DeepMath-5K/4B, Self-Ref's three-seed mean exceeds GRPO-6/8 by 7.90/6.73 points, respectively.

W2(b). Self-Ref versus Teacher-Ref.

The paired DeepMath seeds in W2(a)/Q1 give 62.52% for Self-Ref versus 59.38% for Teacher-Ref, a 3.14-point gap. Please refer to our response to Reviewer 9GwA, W3/Q3, for trajectory-likelihood evidence supporting the proposed internalization explanation.

W2(c). CRM and late-stage collapse.

On DeepMath-5K, CRM improves Self-Ref's three-seed mean by 1.00 point and reduces SD from 3.33 to 0.53; all three w/o-CRM runs collapsed late. On DeepScaler-5K, Teacher-Ref w/o CRM averages 63.45 ± 1.72%, versus 64.0% with CRM in Table 3 (single run).

To address the concern that Figure 3 shows collapse only in training reward, we evaluated its DeepScaler-5K/4B Teacher-Ref w/o-CRM run at step 240 on the six benchmarks:

Checkpoint / statisticAIME24AIME25AMCMATH500MinervaOlympiadAVG
Best-checkpoint mean (3 seeds)53.8945.2987.8991.6741.4060.6363.46
Step 240 (Figure 3 run)17.4023.7447.9774.2025.7034.4037.24

The step-240 checkpoint performs poorly across all six benchmarks, with only 37.24% average accuracy, confirming the late-stage collapse.

W2(d)/Q5. Correcting Table 6 and clarifying the hard set.

1. Corrected evaluation. We found two errors in Appendix Table 6: rewards were pooled across queries for variance, and thinking was inadvertently enabled. We reran both conditions with thinking disabled using the frozen step-40 checkpoint, the same 1,980 historical hard records and six saved template assignments per query. The conclusion remains unchanged: evolved guidance alleviates advantage collapse.

2. Corrected results. Variance is computed per query over six rewards, then averaged. Nonzero-variance groups have 1–5 successes and non-degenerate advantages; percentages use each bucket's size.

Bucket (queries)Mean reward (w/o template)Mean reward (w/ template)Within-query variance (w/o template)Within-query variance (w/ template)Nonzero-variance groups, n (%) (w/o template)Nonzero-variance groups, n (%) (w/ template)
Historical hard set (1980)0.14950.24190.08310.0987876 (44.24%)1025 (51.77%)
All incorrect (1098)0.00000.15030.00000.06670 (0.00%)394 (35.88%)
At most one success (1479)0.04290.17850.03580.0805381 (25.76%)637 (43.07%)

For all 1,980 queries, templates raise mean reward from 0.1495 to 0.2419 and nonzero-variance groups from 876 to 1,025. Mean within-query variance also increases from 0.0831 to 0.0987. Of the 1,098 all-incorrect groups, 416 improve in reward: 394 recover nonzero variance. The ≤1-success subgroup's variance increases 2.25-fold. Together, these results support the role of evolved guidance in alleviating advantage collapse by providing within-group reward differences usable by GRPO on more queries.

We will correct Table 6, its caption and Appendix E. We apologize for these errors in the appendix; but the overall experimental conclusions remain unchanged.

Q6. Training length and development splits.

10 epochs is the maximum training budget, with early stopping enabled. At N=5,000 and batch size 128, 230–240 steps correspond to approximately six epochs. The best checkpoints for the plotted 4B configurations are:

DatasetTeacher-Refw/o context distillationw/o CRMGRPO-8
DeepScaler-5K180210210210
DeepMath-5K180210180240

The dev set used for checkpoint selection is disjoint from GEPA's 300-example dev subset, which is sampled from training data and used for prompt selection.

Minor suggestions.

The AIME25 entry in Tables 1 and 9 should be 44.2, not 42.2; the correct six-benchmark average remains 64.7. We will correct this entry, change Figure 3's “w/o-TRM” label to “w/o-CRM,” bold the Olympiad maximum of 63.0 in Table 2, and label its P²O row as w/o CRM.

Thank you for the thorough response. The R1-hint control under the same context distillation and mask (56.5 vs. 63.0 for Self-Ref) addresses my main concern that GEPA's evolution was untested against a simple hint. The three-seed results (62.52 +/- 0.53 vs. 54.62 +/- 1.05 for GRPO-6) and the rollout-matched GRPO-7 run show the gains are robust. I also appreciate the corrected Table 6 and the clarification on checkpoints and dev splits. Two points remain: all results use non-thinking mode, so whether the gains carry over to thinking mode is open, and the hint control covers one setting. Please include the corrected Table 6 and the new controls in the revision. I am keeping my score of weak acceptance.

AuthorsOpenReview ↗

Response to Reviewer MoGR

Thank you for the follow-up and for recognizing that the additional controls and corrections address your main concerns.

  1. Effectiveness on a conventional Instruct backbone.

To address the concern that P²O's gains might depend on Qwen3's hybrid thinking/non-thinking capabilities, we added an experiment on DeepMath-5K with Qwen2.5-7B-Instruct, a conventional instruction-tuned backbone without a separate thinking mode. We trained GRPO-8 and P²O on this dataset and evaluated both on the same six benchmarks. P²O uses the original queries without evolved templates at inference time.

All scores are accuracy (%); AVG is the unweighted mean across the six benchmarks.

MethodAIME24AIME25AMCMATH500Minerva MathOlympiadBenchAVG
GRPO-811.407.9050.9077.2038.6039.9037.65
P²O17.9010.4059.1077.4040.1041.3041.03

P²O improves average accuracy by 3.38 percentage points, with higher scores on all six benchmarks. The gains include 6.50 points on AIME24, 2.50 on AIME25, and 8.20 on AMC.

These results demonstrate that P²O remains effective on a conventional Instruct backbone. A hybrid thinking/non-thinking backbone is not required for the observed gains, addressing the concern that the method's effectiveness depends on Qwen3's hybrid-mode capabilities. This provides additional evidence for the applicability of P²O's prompt-assisted exploration and context distillation to standard instruction-tuned models.

  1. Extending the hint control to a second training dataset.

To address your observation that the hint control covered one setting, we added the same GHPO-inspired control on DeepScaler-5K/Qwen3-4B. Our DeepScaler training data do not include the reference solution traces needed for this control, so we generated a reference solution for each training problem using Kimi. Following the earlier R1-hint control, this variant uses reference-solution guidance in place of GEPA-evolved templates while retaining context distillation (CD) and CRM. The hint source here is Kimi, whereas the earlier DeepMath control used the supplied R1 solutions.

MethodAIME24AIME25AMCMATH500Minerva MathOlympiadBenchAVG
GRPO-843.436.783.489.343.256.258.7
Kimi hints + CD + CRM (GHPO-inspired)39.031.976.686.844.950.754.98
P²O-Self-Ref59.044.288.492.042.662.164.7

P²O-Self-Ref achieves 64.7% average accuracy, exceeding the hint control's 54.98% by approximately 9.7 percentage points and GRPO-8's 58.7% by 6.0 points. Together with the earlier DeepMath-5K comparison, this extends the evidence for GEPA-evolved guidance over the tested reference-solution control to a second training dataset.

Based on the experimental results reported in our previous responses and the reviewers' feedback, we have submitted a revised manuscript. The revision updates the experimental results and conclusions, corrects Table 6 and its associated analyses, and includes additional experiments. All changes are highlighted in blue for ease of review. We invite you to review the revised manuscript.

Comments on the paper

1 comment

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.