acceptodds
Under review as a conference paper at ICLR 2027

PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning

Abstract

Reinforcement learning algorithms such as group-relative policy optimization (GRPO) have shown strong potential for improving the mathematical reasoning capabilities of large language models. While a growing body of work seeks to improve training entropy, rollout diversity, and exploration, most existing methods still train models with a single fixed reasoning prompt or template, which can encourage prompt-specific overfitting and unstable training dynamics. In this work, we introduce Prompt Augmented Policy Optimization (PrAg-PO), a simple policy optimization method that mixes prompt templates with template-specific format rewards during training. By encouraging models to generate reasoning traces under diverse instructions and output formats, PrAg-PO increases rollout diversity and improves robustness. Compared with GRPO and DAPO, PrAg-PO achieves significantly higher reasoning accuracy while mitigating premature training collapse. Empirically, experiments on DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-Math-1.5B, Qwen3-1.7B, and Qwen3-8B show that PrAg-PO consistently outperforms strong baselines and achieves competitive performance against recent methods on mathematics benchmarks, using only a fixed MATH Level 3-5 training set of 8.5K problems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Prompt Augmentation, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MxN9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PrAg-PO, a GRPO-style method for reinforcement-learning post-training on mathematical reasoning. During training, the model randomly selects a prompt template for each question and generates a group of rollouts conditioned on that template. It then combines answer-accuracy rewards with template-specific format rewards to compute group-relative advantages and update a shared policy. The templates cover free-form, tagged, explicit chain-of-thought, and reflection-based reasoning. The authors evaluate PrAg-PO on four models, training on a fixed set of MATH Level 3–5 problems and testing on multiple mathematical benchmarks. The results show that PrAg-PO outperforms the reproduced GRPO and DAPO baselines in accuracy, increases rollout diversity, and makes training more stable. The authors also use ablation studies to examine the role of format rewards and evaluate the models with different prompt templates, including an out-of-distribution prompt.

Critical strengths

PrAg-PO modifies the training prompt and reward setup without requiring new model components or additional training data. Across four model checkpoints, the authors report improved benchmark accuracy over their reproduced GRPO and DAPO baselines while evaluating with a single default prompt. This makes the approach practically relevant: the reported gains do not depend on test-time prompt ensembling or additional data.

It examines training stability, prompt-format adherence, and rollout diversity, and includes an ablation showing that removing format rewards leads to earlier collapse and weaker adherence to the requested formats. This provides useful evidence for why the combined prompt-and-reward design matters, beyond reporting benchmark scores alone.

Critical weaknesses

The experimental section does not report results from independent training runs with different random seeds. The authors state that evaluation is performed over five inference rounds using the same random seed and that average accuracy is reported. However, this reflects variability at inference time and does not measure the stochasticity of the training process itself. I believe it is necessary to conduct training runs with different random seeds and report the mean and variance of final accuracy and metrics such as the training step at which collapse occurs.

The paper does not systematically evaluate the individual effects of different prompt templates on the training process. Although the authors train with a mixture of templates and examine how different templates affect performance at evaluation time in their ablation studies, they do not conduct experiments to isolate or explain the effect of each template on training.

Questions for authors

Could the authors run experiments with different random seeds during training and report the corresponding results?

Could the authors conduct ablation studies to investigate how using different prompt templates individually during training affects the training process and final evaluation results?

Could the authors clarify how the accuracy and format rewards are combined to form the total reward used to compute group-relative advantages?

Could the authors clarify whether the prompt-template augmentation used in PrAg-PO can be applied to other RL methods, such as DAPO and Dr. GRPO?

If the authors address my questions, I will raise my score.

4 comments

We thank the reviewer MxN9 for the constructive feedback and for recognizing the practical simplicity of PrAg-PO and the value of our analyses of training stability, format adherence, and rollout diversity. We address the concerns and questions below.

1. Independent training runs with different random seeds

We agree that repeated inference on a fixed checkpoint does not measure variability arising from training. The five inference rounds reported in the submission were intended to account for inference-engine nondeterminism under greedy decoding, rather than to establish robustness across training seeds. We will clarify this distinction in the manuscript.

To address this concern and due to limited time during rebuttal, we conducted experiments with three independent training seeds (0, 42, None) on Qwen2.5-Math-1.5B (each takes ~50 hours on 8 L40S GPUs). We report mean accuracy and standard deviation across the three training runs in the table below:

Qwen2.5-Math-1.5BAIME24AMCMATH500MinervaOlympiadAVG benchmarkAVG question
PrAg-PO (Seed=42)23.353.578.431.339.445.250.9
PrAg-PO (Seed=1)23.354.278.629.640.045.251.0
PrAg-PO (Seed=None)23.352.076.828.241.944.550.9
PrAg-PO23.3 ± 0.053.2 ± 1.177.9 ± 1.029.7 ± 1.640.4 ± 1.345.0 ± 0.450.9 ± 0.1

As seen from the table above, although the accuracy within each benchmark shows some fluctuations among different training seeds, the overall performance is stable and has reasonable reproducibility. All 3 seeded run outperforms other baselines compared in our paper. We will make sure to extend the seeded experiments to all our evaluated models in our final manuscript.

2. Effects of individual prompt templates during training

We appreciate this suggestion and agree that evaluation under different prompts does not isolate the effect of the templates used during training. Our existing single-template baselines (i.e., DAPO) provide a reference under the default training prompts. We conducted an additional experiment on Qwen2.5-Math-1.5B based on DeepSeek template (i.e., respond in the format <think>...</think><answer>...</answer>), together with its corresponding format reward.

Qwen2.5-Math-1.5BAIME24AMCMATH500MinervaOlympiadAVG benchmarkAVG question
Default Template23.354.976.926.039.444.149.6
DeepSeek Template20.047.277.527.939.942.549.9
Augmented Template23.353.578.431.339.445.250.9

As shown in the table above, the augmented template outperforms both the default template and the Deepseek template, which shows that the collective effect of template augmentation contributes to the improvement in model's performance, rather than a single particular template. Thanks for suggesting this ablation, we will incorporate ablation to more alternative templates to demonstrate the effect in oure revised manuscript.

3. Combination of accuracy and format rewards

Thank you for pointing out that the reward combination should be stated explicitly. For a response oio_i generated under template TkT_k, the total reward is the unweighted sum:

Rk(oi)=Racc(oi)+Rfmt,k(oi).R_k(o_i)=R_{\mathrm{acc}}(o_i)+R_{\mathrm{fmt},k}(o_i).

Here, Racc∈0,1R_{\mathrm{acc}}\in \\{0,1\\} indicates whether the extracted final answer is correct, while Rfmt,k∈[0,1]R_{\mathrm{fmt},k}\in[0,1] measures adherence to the format requested by template TkT_k. The total reward therefore lies in [0,2][0,2]. The two components are combined before computing the group-relative advantage; they are not normalized separately:

A^ik=Rk(oi)−mean⁡j=1GRk(oj)std⁡j=1GRk(oj)\displaystyle \hat{A}_i^k=\frac{R_k(o_i)-\operatorname{mean}_{j=1}^{G}R_k(o_j)}{\operatorname{std}_{j=1}^{G}R_k(o_j)}

All responses within a rollout group share the same question and template, so their rewards are computed using the same format criterion. Different groups within a training batch can use different templates.

For templates without explicit format constraints, Rfmt,k=1R_{\mathrm{fmt},k}=1 for every response. This constant cancels under group mean subtraction and does not change the group standard deviation, so the resulting advantages depend only on answer accuracy. We will add the explicit reward-combination equation to Section 3.4.

4. Applicability to DAPO and Dr. GRPO

Yes. Prompt augmentation only modifies the conditioning prompts and their associated reward functions, and can be incorporated into other group-based policy optimization methods. For each question, one samples a template, generates the rollout group under that template, and computes the corresponding rewards. The underlying algorithm then applies its own advantage estimator and optimization objective.

In fact, our current formulation already incorporates DAPO-style asymmetric clipping and token-level loss aggregation, and omitting the KL penalty. It also supports dynamic sampling as the filtering criterion can be applied within each template-conditioned rollout group to retain groups with nonzero variance in the combined accuracy and format reward. Thus, prompt augmentation is already compatible with the DAPO training objective.

For Dr. GRPO, the same template sampling and reward construction can be retained while using its group-mean-centered advantages without standard-deviation normalization and its constant loss normalization. Prompt augmentation does not require restoring the normalization terms that Dr. GRPO removes.

We will make sure to highlight this broad applicability of PrAg-PO to various policy optimization methods.

We thank the reviewer again for these suggestions. We will incorporate the completed experimental results and the above clarifications into the revised manuscript and retain them in the final version. We remain engaged and willing to provide further clarification during the ongoing discussion.

Thank you for the additional experiments. Your response has addressed my concerns, and I will raise my score to 3.

We sincerely thank you for taking the time to review the additional experiments and our rebuttal. We are glad they addressed your concerns, and we really appreciate your constructive feedback.

Reviewer_gTNUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PrAg-PO, a prompt augmentation method for RL. Although the authors provide a series of empirical results demonstrating the effectiveness of their method, I still do not recommend acceptance. There are issues throughout the paper, from the motivation to the claims and the method itself. First, it is questionable whether prompt augmentation truly promotes reasoning trajectory diversity and training stability, and both diversity and stability can also be improved through other approaches. The practical need for using multiple templates is also unclear; in principle, a single template is sufficient as long as the model performs well. Finally, the proposed method is overly intuitive and lacks sophisticated design or particularly novel insights. Therefore, I currently lean toward rejection.

Critical strengths

The experimental results are comprehensive and relatively stable across multiple model families, benchmarks, and training settings.

Critical weaknesses
  • First, it is questionable whether prompt augmentation truly promotes reasoning trajectory diversity and training stability, and both diversity and stability can also be improved through other approaches.
  • The practical need for using multiple templates is also unclear; in principle, a single template is sufficient as long as the model performs well.
  • Finally, the proposed method is overly intuitive and lacks sophisticated design or particularly novel insights.
Questions for authors

See Weaknesses.

5 comments

We thank the reviewer gTNU for acknowledging the comprehensive and consistent experimental results across model families, benchmarks, and training settings. We respectfully clarify the motivation and contribution of PrAg-PO below.

1. Evidence for reasoning diversity and training stability

Our claims are supported by complementary quantitative analyses and ablation studies.

For diversity, Figure 7 compares rollouts generated by the same PrAg-PO checkpoints with and without prompt augmentation, using the same sampling settings and rollout count. Augmented prompts consistently produce higher semantic diversity, measured through pairwise distances between rollout embeddings. Figure 4 additionally illustrates differences in reasoning structure and mathematical exposition when the same model answers the same question under different templates. These results both quantitatively and qualitatively show the PrAg-PO enables the generation of more diverse trajectories.

For stability, Figure 1 directly compares evaluation accuracy throughout training. PrAg-PO delays the premature collapse observed in single-template baselines and supports continued improvement over longer training horizons. Appendix A.3 provides complementary evidence through smoother policy-entropy trajectories. Furthermore, Figure 5 shows that removing template-specific format rewards while retaining prompt augmentation leads to earlier collapse and lower accuracy, demonstrating that the reward component contributes to the observed benefits.

We agree that other approaches can also improve diversity and stability. Our contribution is not that prompt augmentation is the only way to achieve these goals, but that diversifying the prompt–reward interface provides a simple and effective additional mechanism. The existence of alternative approaches does not resolve whether this mechanism is useful; our comparisons and ablations investigate precisely that question.

2. The practical need for using multiple templates is unclear

We agree that strong performance under a single template is desirable. However, PrAg-PO does not require multiple templates at inference time. All main benchmark results use the model’s single default evaluation template. Thus, the improvements in Table 1 demonstrate that training with diverse templates can benefit deployment even when only one inference template is used.

Beyond this setting, robustness to prompt variation is practically important. Users and downstream applications may request different reasoning structures, explicit verification steps, or particular output formats. High accuracy under one fixed prompt does not establish reliable performance under these variations. Figure 6 examines performance across templates, while Appendix A.5 evaluates an out-of-distribution prompt and shows that PrAg-PO maintains higher average accuracy than GRPO and DAPO across the three evaluated models. These experiments do not cover every deployment scenario, but they directly address robustness beyond a single evaluation interface.

The motivation is therefore twofold: improving optimization and accuracy under standard evaluation, and reducing sensitivity to how reasoning is requested.

3. Simplicity and the contribution of the method

PrAg-PO’s simplicity makes the approach easier to implement and adopt; its scientific value rests on the question it investigates and the evidence supporting its effectiveness.

Specifically, we study whether varying reasoning templates and their associated format rewards can improve policy optimization under a fixed training dataset. The results show that this training-time intervention improves performance even under a single default evaluation template, delays collapse, and improves robustness to prompt variation. The format-reward ablation further reveals that simply varying prompts is insufficient to obtain the same benefits: without the associated rewards, the model tends to ignore the requested formats and collapses earlier.

These findings go beyond the intuition that “more prompts provide more diversity.” We identify an effective training intervention, establish the importance of its reward component, and characterize its effects across multiple models and benchmarks. We hope these clarifications help distinguish the method’s implementation simplicity from the empirical insights it provides..

We sincerely thank you for your feedback. We will remain available during the remaining discussion period.

I would like to emphasize that the main reason for my negative assessment is not whether the proposed method is useful, but whether the research question it addresses is sufficiently important and whether the work provides meaningful insights to the broader research community.

While I acknowledge the potential utility of the proposed method, I remain unconvinced that the research question and findings are sufficiently fundamental or insightful to attract broad interest from the ICLR community.

In my view, this work might have been competitive at NeurIPS 2025, when LLM-based reinforcement learning was still in a relatively early exploratory stage. However, for ICLR 2027, I believe the community should expect research that addresses more fundamental questions and offers deeper, more generalizable insights, rather than primarily demonstrating methodological improvements.

Therefore, while I appreciate the authors' efforts and clarifications, my core concerns remain unresolved, and I will maintain my original score.

Thank you for clarifying your concern. We agree that an important question is what the community can learn from these experiments beyond the performance gains of PrAg-PO itself.

Earlier RL work has shown that the behaviors an agent practices during training can matter even when its evaluation objective is fixed. Riedmiller et al. (ICML 2018) found that a robot learned to stack objects more reliably when it also practiced auxiliary behaviors such as touching and moving them. Similarly, Badia et al. (ICLR 2020) found that training a shared network under multiple exploration incentives improved game scores over a single-incentive baseline on four of six hard-exploration Atari games. Recent work continues to investigate how agents can learn different ways to solve the same task. In continuous control, Shitanda et al. (ICLR 2026) found that regulating diversity among exploratory policies improved both sample efficiency and final performance. In robotic manipulation, Braun et al. (RLC 2026) showed that explicitly searching for distinct, high-reward trajectories yielded a wider range of successful behaviors.

PrAg-PO asks whether this principle extends to LLM reasoning: when the training questions and their correct answers are held fixed, does varying how the model is asked and rewarded to solve them improve accuracy under a single default prompt? Across the models and benchmarks in Table 1, training with multiple reasoning templates improves accuracy even when evaluation uses only the model’s default prompt. This suggests that an important design choice in RL with verifiable rewards is which behaviors the training prompts elicit and which of those behaviors the reward reinforces. In practical settings where labeled training data are limited, our results suggest that template-level augmentation can support more stable training over longer horizons and improve accuracy without requiring additional labeled examples.

We hope this connection to the broader RL literature clarifies why the question matters and what our findings contribute to the study of exploration in LLM reasoning. Thank you again for your time in reviewing our work and for encouraging us to articulate the broader significance of our work more clearly.

  1. Riedmiller et al. (2018). ICML. Learning by Playing - Solving Sparse Reward Tasks from Scratch

  2. Badia et al. (2020). ICLR. Never Give Up: Learning Directed Exploration Strategies

  3. Shitanda et al. (2026). ICLR. Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning

  4. Braun et al. (2026). RLC. Trajectory First: A Curriculum for Discovering Diverse Policies

Reviewer_t8JeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issues of prompt overfitting and unstable training that often occur when large language models are trained using reinforcement learning for mathematical reasoning tasks. It proposes a new method called Prompt Enhancement Strategy Optimization. This method effectively enhances the diversity of the reasoning trajectories generated by the model by mixing multiple prompt templates during the training phase and introducing format rewards for specific templates. Experimental results show that this method significantly improves the accuracy of mathematical benchmark tests on models of various sizes, while avoiding the collapse phenomenon in the early training stage, demonstrating excellent robustness and generalization ability.

Critical strengths
  1. The current reinforcement learning methods based on group relative strategy optimization do have the problem of over-reliance on a single prompt template. The authors have keenly identified this pain point and provided a concise and effective solution from the perspectives of data augmentation and reward design.

  2. This method does not introduce complex network structures or huge additional computational costs. Instead, it ingeniously utilizes the specific format of the template rewards to guide the model. This lightweight improvement is highly in line with the current research trend of efficient fine-tuning.

  3. The authors conducted extensive comparative experiments on multiple models with different parameter scales. They not only proved the superiority of this method in terms of accuracy but also verified its effectiveness in alleviating training crashes through dynamic training analysis, thereby enhancing the persuasiveness of the conclusion.

Critical weaknesses
  1. The paper only used a fixed number of moderately difficult mathematical problems as the training set. Although it verified the effectiveness of the method, it failed to fully demonstrate the generalization potential of this method in more complex or cross-domain reasoning tasks.

  2. Although a template-specific format reward was introduced, the paper's sensitivity analysis of the weights of different format rewards was rather lacking. Readers find it difficult to determine whether this method remains stable when faced with extremely imbalanced template distributions.

  3. Introducing multiple prompt template inevitably affects the batch composition and computational efficiency during the training stage. The paper failed to provide specific comparative data on training time or resource consumption, which is an important consideration when deploying the actual system.

Questions for authors

Lack of discussion with related works for stable/robust RFT:

  1. Historical Relative Policy Optimization for Bootstrapping LLM Reasoning

  2. Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning

  3. Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning

  4. Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization in LLM Reasoning

  5. Dynamic Important Example Mining for Reinforcement Finetuning

  6. Understanding Data Influence in Reinforcement Finetuning

1 comment

We thank the reviewer t8Je for the constructive feedback and positive assessment of PrAg-PO’s simplicity, effectiveness, and improved training stability. We address the concerns below.

1. Training data and generalization to more complex or cross-domain tasks

Our use of a fixed training dataset allows us to study the effect of prompt augmentation while controlling for changes in training data. Evaluation includes challenging benchmarks such as AIME24 and OlympiadBench, alongside AMC, MATH500, and Minerva Math. Thus, the experiments assess performance across multiple mathematical benchmarks and difficulty levels under a fixed training-data budget.

We also examine generalization to prompt variations. Figure 6 shows consistent performance across different templates, while Appendix A.5 evaluates a minimal out-of-distribution prompt, “{question}\nAnswer:”, without a system message or explicit reasoning instructions. PrAg-PO achieves higher average accuracy than GRPO and DAPO under this prompt across all three evaluated models.

Nevertheless, we agree that these results establish generalization within mathematical reasoning and across prompt interfaces, rather than cross-domain generalization. Extending the approach to domains such as coding and scientific reasoning is an important direction for future work.

2. Sensitivity to format weights and imbalanced template distributions

Thank you for raising this concern. We would like to clarify that PrAg-PO samples templates uniformly from the template pool, giving each template an equal probability of selection. Thus, extreme sampling imbalance is not part of our experimental setup. Nevertheless, single-template training can be viewed as the limiting case of an imbalanced template distribution, in which all probability mass is assigned to one template. Our comparisons with single-template baselines show that these methods are more susceptible to training instability and premature collapse, supporting the value of maintaining template diversity. Systematically investigating nonuniform sampling distributions would be a valuable direction for future work, although a comprehensive sweep would require substantial computational resources.

3. Batch composition and computational efficiency

Thanks for raising this concern. We would like to clarify that PrAg-PO randomly samples one template per training example within a batch, rather than expanding each example into all available templates. The prompt batch size remains 128, the mini-batch size remains 32, and each example has eight rollouts (Section 4.1). Increasing the template pool therefore does not multiply the number of rollouts.

The additional operations consist of template sampling, prompt formatting, and simple string-matching or tag-counting rewards, which incur minimal computational overhead. In practice, we observe no noticeable increase in per-step latency compared with DAPO and Dr. GRPO. The total training cost is primarily determined by the number of optimization steps. As shown in Figure 1, PrAg-PO incurs additional computation only when training continues beyond the point at which competing methods collapse, allowing it to achieve further performance improvements.

4. Additional related literature

Thank you for recommending these relevant works on stable and robust reinforcement fine-tuning. We have incorporated all six suggested papers into the Related Work section. The additions are highlighted in red text in the revised manuscript. We will make sure to retain these discussions in the final version of the manuscript.

We truly appreciate the feedback and believe the changes have made our submission much stronger. We will remain available for conversation until the discussion period ends.

Comments on the paper

2 comments

Dear all, The discussion phase will end soon. The authors have provided their responses to your comments. Some of you have started involving discussion, but all of us can take action now. If the provided responses do not address (even part of) your concerns, this is your chance to raise it. Please engage with the authors as soon as possible. We really appreciate your efforts! Thanks, AC

Thank you for your reminder, but I currently seem unable to see the author's reply, and I'm not sure if an error has occurred.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.