- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PrAg-PO, a GRPO-style method for reinforcement-learning post-training on mathematical reasoning. During training, the model randomly selects a prompt template for each question and generates a group of rollouts conditioned on that template. It then combines answer-accuracy rewards with template-specific format rewards to compute group-relative advantages and update a shared policy. The templates cover free-form, tagged, explicit chain-of-thought, and reflection-based reasoning. The authors evaluate PrAg-PO on four models, training on a fixed set of MATH Level 3–5 problems and testing on multiple mathematical benchmarks. The results show that PrAg-PO outperforms the reproduced GRPO and DAPO baselines in accuracy, increases rollout diversity, and makes training more stable. The authors also use ablation studies to examine the role of format rewards and evaluate the models with different prompt templates, including an out-of-distribution prompt.
Critical strengths
PrAg-PO modifies the training prompt and reward setup without requiring new model components or additional training data. Across four model checkpoints, the authors report improved benchmark accuracy over their reproduced GRPO and DAPO baselines while evaluating with a single default prompt. This makes the approach practically relevant: the reported gains do not depend on test-time prompt ensembling or additional data.
It examines training stability, prompt-format adherence, and rollout diversity, and includes an ablation showing that removing format rewards leads to earlier collapse and weaker adherence to the requested formats. This provides useful evidence for why the combined prompt-and-reward design matters, beyond reporting benchmark scores alone.
Critical weaknesses
The experimental section does not report results from independent training runs with different random seeds. The authors state that evaluation is performed over five inference rounds using the same random seed and that average accuracy is reported. However, this reflects variability at inference time and does not measure the stochasticity of the training process itself. I believe it is necessary to conduct training runs with different random seeds and report the mean and variance of final accuracy and metrics such as the training step at which collapse occurs.
The paper does not systematically evaluate the individual effects of different prompt templates on the training process. Although the authors train with a mixture of templates and examine how different templates affect performance at evaluation time in their ablation studies, they do not conduct experiments to isolate or explain the effect of each template on training.
Questions for authors
Could the authors run experiments with different random seeds during training and report the corresponding results?
Could the authors conduct ablation studies to investigate how using different prompt templates individually during training affects the training process and final evaluation results?
Could the authors clarify how the accuracy and format rewards are combined to form the total reward used to compute group-relative advantages?
Could the authors clarify whether the prompt-template augmentation used in PrAg-PO can be applied to other RL methods, such as DAPO and Dr. GRPO?
If the authors address my questions, I will raise my score.
4 comments
4. Applicability to DAPO and Dr. GRPO
Yes. Prompt augmentation only modifies the conditioning prompts and their associated reward functions, and can be incorporated into other group-based policy optimization methods. For each question, one samples a template, generates the rollout group under that template, and computes the corresponding rewards. The underlying algorithm then applies its own advantage estimator and optimization objective.
In fact, our current formulation already incorporates DAPO-style asymmetric clipping and token-level loss aggregation, and omitting the KL penalty. It also supports dynamic sampling as the filtering criterion can be applied within each template-conditioned rollout group to retain groups with nonzero variance in the combined accuracy and format reward. Thus, prompt augmentation is already compatible with the DAPO training objective.
For Dr. GRPO, the same template sampling and reward construction can be retained while using its group-mean-centered advantages without standard-deviation normalization and its constant loss normalization. Prompt augmentation does not require restoring the normalization terms that Dr. GRPO removes.
We will make sure to highlight this broad applicability of PrAg-PO to various policy optimization methods.
We thank the reviewer again for these suggestions. We will incorporate the completed experimental results and the above clarifications into the revised manuscript and retain them in the final version. We remain engaged and willing to provide further clarification during the ongoing discussion.
Thank you for the additional experiments. Your response has addressed my concerns, and I will raise my score to 3.
We sincerely thank you for taking the time to review the additional experiments and our rebuttal. We are glad they addressed your concerns, and we really appreciate your constructive feedback.
We thank the reviewer MxN9 for the constructive feedback and for recognizing the practical simplicity of PrAg-PO and the value of our analyses of training stability, format adherence, and rollout diversity. We address the concerns and questions below.
1. Independent training runs with different random seeds
We agree that repeated inference on a fixed checkpoint does not measure variability arising from training. The five inference rounds reported in the submission were intended to account for inference-engine nondeterminism under greedy decoding, rather than to establish robustness across training seeds. We will clarify this distinction in the manuscript.
To address this concern and due to limited time during rebuttal, we conducted experiments with three independent training seeds (0, 42, None) on Qwen2.5-Math-1.5B (each takes ~50 hours on 8 L40S GPUs). We report mean accuracy and standard deviation across the three training runs in the table below:
As seen from the table above, although the accuracy within each benchmark shows some fluctuations among different training seeds, the overall performance is stable and has reasonable reproducibility. All 3 seeded run outperforms other baselines compared in our paper. We will make sure to extend the seeded experiments to all our evaluated models in our final manuscript.
2. Effects of individual prompt templates during training
We appreciate this suggestion and agree that evaluation under different prompts does not isolate the effect of the templates used during training. Our existing single-template baselines (i.e., DAPO) provide a reference under the default training prompts. We conducted an additional experiment on Qwen2.5-Math-1.5B based on DeepSeek template (i.e., respond in the format <think>...</think><answer>...</answer>), together with its corresponding format reward.
As shown in the table above, the augmented template outperforms both the default template and the Deepseek template, which shows that the collective effect of template augmentation contributes to the improvement in model's performance, rather than a single particular template. Thanks for suggesting this ablation, we will incorporate ablation to more alternative templates to demonstrate the effect in oure revised manuscript.
3. Combination of accuracy and format rewards
Thank you for pointing out that the reward combination should be stated explicitly. For a response oi generated under template Tk, the total reward is the unweighted sum:
Rk(oi)=Racc(oi)+Rfmt,k(oi).
Here, Racc∈0,1 indicates whether the extracted final answer is correct, while Rfmt,k∈[0,1] measures adherence to the format requested by template Tk. The total reward therefore lies in [0,2]. The two components are combined before computing the group-relative advantage; they are not normalized separately:
A^ik=stdj=1GRk(oj)Rk(oi)−meanj=1GRk(oj)
All responses within a rollout group share the same question and template, so their rewards are computed using the same format criterion. Different groups within a training batch can use different templates.
For templates without explicit format constraints, Rfmt,k=1 for every response. This constant cancels under group mean subtraction and does not change the group standard deviation, so the resulting advantages depend only on answer accuracy. We will add the explicit reward-combination equation to Section 3.4.