- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether the generalization benefits of on-policy LLM post-training can be transferred to supervised fine-tuning through parameter update directions. The authors observe that SFT updates remain relatively consistent across training stages, whereas GRPO and on-policy distillation exhibit more variation. Motivated by this observation, they propose OPSFT, which extracts the sign of each parameter’s cumulative change during on-policy training and uses it to constrain SFT updates. Across math and code reasoning benchmarks, the authors report improvements over standard SFT and performance comparable to the on-policy methods that provide the directions. They also explore two practical applications: using a short GRPO run to identify directions before switching to OPSFT, and using OPSFT to incorporate additional high-quality demonstrations into models that have already undergone on-policy training.
Critical strengths
-
The paper turns an observation about optimization into a concrete intervention. Rather than only comparing parameter updates across training methods, the authors use the cumulative update signs from on-policy training to constrain SFT. The location-only and random-sign controls in Table A13 provide useful evidence that the learned signs contain information beyond the update locations.
-
The empirical evaluation covers several relevant settings. The experiments include multiple model sizes, two model families, and both mathematical reasoning and code generation. The additional OPD experiment also provides some evidence that the approach is not restricted to GRPO.
-
The proposed applications address practical post-training problems. Using a short on-policy run to guide subsequent SFT, and incorporating additional demonstrations into an already post-trained model, are both useful settings. The latter is particularly interesting given the performance degradation observed when ordinary SFT is applied after GRPO.
Critical weaknesses
-
The evidence supports the usefulness of the constraints more clearly than the proposed explanation of generalization. Table A13 shows that on-policy signs outperform random signs, but this does not establish that the benefit is specific to on-policy training. Signs extracted from another effective training procedure might also encode useful task information. Moreover, random signs can change the proportion and magnitude of retained gradients, so their weaker performance does not fully rule out other regularization effects.
-
The runtime results need a clearer breakdown. Section 4.1 describes a 100-step GRPO baseline and a 50-step GRPO stage followed by 100 OPSFT steps. However, all OPSFT totals in Table 2 are below half the corresponding GRPO times. For Qwen3-8B, the reported total is 8.9 hours, whereas half the GRPO time is 9.65 hours. Nonuniform step costs could explain this, but the paper does not provide the measurements needed to check it. Appendix Table A7 also lists 150 GRPO steps, leaving the correspondence between configurations and reported times unclear.
-
The stability of the reported advantages is not established. The paper does not report independent training repetitions or confidence intervals for the key comparisons. This is especially relevant to the small advantages over GRPO, such as the 0.16- and 0.19-point mean differences in Table 1. Multiple generations per problem do not capture variation across direction extraction and training runs.
Questions for authors
-
Could you break down the Table 2 runtime into the GRPO stage, direction extraction and checkpoint handling, and subsequent OPSFT training? In my experience training with GRPO, runtime scales approximately linearly with the number of steps, so I am having difficulty reconciling the reported times with the stated training schedule. Do the baseline times correspond to 100 or 150 GRPO steps? How much time do the first 50 GRPO steps actually take?
-
Would directions extracted from a competitive SFT checkpoint provide similar benefits? A comparison with the same support set and matched effective update magnitude would help distinguish on-policy-specific information from more general task information or regularization.
-
What weights initialize OPSFT in Figure 3 and Table 2? Please specify whether OPSFT starts from the original base model or the direction-providing GRPO checkpoint, whether optimizer states are inherited, and which checkpoint serves as the projection reference. If OPSFT restarts from the base model, I can see how this helps isolate the contribution of the update directions. From a practical perspective, however, the initial GRPO stage has already produced a potentially useful checkpoint. Could the authors clarify the benefit of transferring its update signs to SFT from the base model, compared with continuing SFT or OPSFT directly from the 50-step GRPO checkpoint? A comparison would help clarify the practical motivation for this choice.
If the authors can address most of my concerns with convincing experimental evidence, I would be happy to raise my score.
5 comments
W2 & Q1: More analysis of the runtime results
Thank you for your constructive comments, which have helped us refine our comparisons. To fully address your concerns, we first report and analyze the training time of GRPO at different stages, followed by a detailed breakdown of the training time for each stage of OPSFT.
GRPO training time. We report the training time of GRPO at different stages on Qwen3-8B in the table below.
| Training stage | Training time (h) | Average time per step (s) | Average response length (tokens) |
|---|---|---|---|
| Steps 1–50 | 7.3 | 525.4 | 1,773.5 |
| Steps 51–100 | 11.9 | 859.7 | 2,779.6 |
| Steps 101–150 | 18.0 | 1,296.3 | 4,152.3 |
The average time per GRPO step increases as training progresses. This is because we set a large maximum response length (16,384 tokens) following previous work [1]. This setting allows the model to generate increasingly longer responses as training progresses, a phenomenon also widely observed in prior studies like DeepSeek-R1 [2]. The increasing response length leads to greater computations during rollouts and policy updating, thereby resulting in longer training time in later GRPO stages.
Additionally, we would like to clarify that the training step reported in Table A7 indicates the configuration in Figure 3, whereas the runtime of the GRPO baseline in Table 2 is measured over 100 training steps.
Detailed training time of OPSFT. We report a detailed breakdown of the training time in Table 2 across different stages within OPSFT on Qwen3-8B.
| Stage | Time | Share of total time (%) |
|---|---|---|
| Initial GRPO training | 7.3 h | 81.9 |
| Direction extraction | 1 min | 0.2 |
| Continual OPSFT | 1.6 h | 17.9 |
| Total | 8.9h | 100.0 |
First, the initial GRPO training accounts for the majority of the total runtime. Second, the direction extraction incurs negligible overhead as it requires no additional training. Finally, OPSFT training itself accounts for only 17.9% of the total runtime.
Revision. We have illustrated the nonuniform step costs of GRPO in Section 4.1 (L413-416), explicitly clarified the optimization step in Table A7, and incorporated the above experiments & analysis into Appendix B (L1024-1054).
[1] Yang, W., Liu, W., Xie, R., Yang, K., Yang, S., & Lin, Y. (2026). Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125.
[2] Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
W3: Advantage stability
Thank you for your valuable comments that help us further improve the experiments. Following your suggestions, we conducted three independent runs of GRPO training, direction extraction, and OPSFT training on Qwen3-8B with the Deepmath dataset. We report the mean performance and standard deviation across multiple runs on four math tasks and three held-out tasks as follows.
| Method | AIME24 | AIME25 | HMMT25-Feb | HMMT25-Nov | Mean |
|---|---|---|---|---|---|
| SFT | 50.69 2.44 | 43.88 2.92 | 22.63 2.13 | 32.36 2.95 | 37.39 |
| GRPO | 57.50 3.01 | 45.27 3.78 | 26.94 2.29 | 37.36 1.96 | 41.77 |
| OPSFT | 55.69 2.54 | 47.63 2.71 | 27.22 2.09 | 38.47 3.12 | 42.25 |
| Method | ARC | HaluEval | Winogrande | Mean |
|---|---|---|---|---|
| SFT | 87.79 0.22 | 73.23 0.21 | 66.11 0.19 | 75.71 |
| GRPO | 87.88 0.17 | 73.53 0.29 | 66.42 0.27 | 75.94 |
| OPSFT | 87.96 0.17 | 73.83 0.15 | 66.48 0.16 | 76.09 |
The results of multiple runs are consistent with our original conclusions: OPSFT largely exceeds vanilla SFT and consistently achieves performance comparable to on-policy paradigms that provide the update direction, further demonstrating the robustness of our findings across three independent runs.
Revision. We have incorporated the above experiment in Appendix B (L1054-1061).
Q3: The initialization of each experiment
Thank you for your careful feedback that helps us better illustrate the experimental settings. We adopt different initializations and directions in Figure 3 and Table 2 depending on their investigation objectives.
Optimizer states. OPSFT does not inherit optimizer states from GRPO training across all experiments, including Figure 3 and Table 2.
In Figure 3, we try to isolate the contribution of the update directions. Therefore, both OPSFT and SFT are initialized from the base model for comparisons under the same training procedure. The directions are identified from a 150-step GRPO checkpoint with strong generalization performance.
In Table 2, we try to improve the training efficiency. Thus, OPSFT is initialized from the 50-step GRPO checkpoint for fast convergence. The directions are also extracted from this checkpoint, as getting the directions via 150-step GRPO is expensive. To better illustrate the effect of this initialization strategy on performance, we provide experiments as follows:
-
Experiments. We evaluate SFT and OPSFT initialized from the base model and the GRPO checkpoint.
Initialization Method AIME24 AIME25 HMMT25-Feb HMMT25-Nov Mean Base - 21.15 21.67 12.08 7.81 15.68 GRPO50 - 33.13 27.71 15.83 16.67 23.33 Base SFT 40.42 46.67 24.58 27.92 34.90 GRPO50 SFT 42.50 40.00 20.83 29.58 33.23 Base OPSFT 47.08 48.33 24.58 34.17 38.54 GRPO50 OPSFT 52.92 50.42 25.83 31.25 40.11 -
Analysis. OPSFT achieves better performance when initialized from the GRPO checkpoint (40.11 vs. 38.54), whereas SFT exhibits the opposite trend (33.23 vs. 34.90). This is because vanilla SFT conflicts with on-policy updates, leading to weaker performance with a stronger initialization. These findings explain our choice to initialize OPSFT from the GRPO checkpoint and SFT from the base model in Table 2.
Revision. We have clarified the initialization settings of OPSFT across different experiments in Sections 3 and 4 (L261-262, L402-403). Additionally, we have included the above experiments comparing different initialization choices in Section 4.3 (L455-463).
Thank you very much for your feedback, which has been of great help to our work. We hope the following responses can clarify the missing points and address your concerns.
W1 & Q2: More ablations of the on-policy update directions
Thank you for your insightful suggestions, which have helped us improve the understanding of the sign constraint. To fully address your concern, we first investigate the effectiveness of the SFT direction. Then, we provide comparisons between OPSFT and magnitude-based constraints.
Effectiveness of competitive SFT directions. Following your suggestion, we construct an SFT checkpoint using trajectories generated by Qwen3-30B-A3B. Then, we use these directions to guide a weaker SFT baseline on trajectories generated by Qwen3-14B.
Experiments. We report the performance of OPSFT with constraints from different checkpoints (100-step GRPO, SFT) using Qwen3-8B on the Deepmath dataset.
Analysis.
The directions of a competitive SFT checkpoint can provide slight benefits, but these benefits are less pronounced than those of GRPO checkpoints. First, the directions of a competitive SFT checkpoint slightly improve a weaker SFT baseline (from 15.52% to 17.29%). Second, the GRPO direction achieves nearly 3% more improvement than the SFT direction (20.10% vs. 17.29%), with the GRPO checkpoint outperforming the SFT checkpoint by only about 2% (40.31% vs. 38.41%).Practical value comparisons.
Beyond the greater performance benefits, the constraint with on-policy directions is also more practically meaningful than using SFT directions. First, on-policy paradigms generally achieve stronger generalization performance than SFT, making it easier to obtain powerful checkpoints through on-policy training. Second, using a stronger SFT model to improve a weaker SFT baseline has less obvious practical value. Unlike our application that combines the strong generalization of on-policy paradigms with the high training efficiency of SFT, strong and weak SFT baselines lack such complementary advantages.Magnitude-based sign constraints. We report the performance of magnitude-based constraints with the same number of updated parameters as on-policy sign constraints at each training step.
The magnitude-based constraints lead to a significant performance drop compared to on-policy direction constraints(41.15% vs. 33.64%). Unlike on-policy directions that introduce information from another stronger training paradigm, the magnitude-based constraint only compresses the original SFT gradients. These findings further support the importance of incorporating on-policy directional information, rather than merely regularizing updates.Revision. We have incorporated the above experiments on SFT directions in Appendix B (L909-969). Moreover, the ablations of magnitude-based sign constraints are incorporated into Section 4.3 (L455-463).