acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training

Abstract

The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, overlooking their potential to serve as optimization principles for improving the generalization of other paradigms such as supervised fine-tuning (SFT). To address this limitation, we investigate whether there exists a specific on-policy update behavior that can achieve such improvements. First, our theoretical and experimental analyses reveal that SFT updates parameters along consistent directions, while the on-policy paradigm continuously adjusts the direction during training. This difference inspires us to focus on the cumulative update direction of each parameter as a promising behavior. Then, we evaluate its effectiveness for improving generalization by proposing On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains SFT updates to the direction identified by on-policy paradigms. The strong performance of OPSFT indicates that the generalization advantage of on-policy paradigms can be transferred to SFT through the parameter update direction. Once such a direction is identified, even SFT can generalize with its updates constrained to this direction. This finding offers two practical benefits by combining the strong generalization of on-policy paradigms with the advantages of SFT, including the high training efficiency and ability to leverage high-quality trajectories. For efficiency, we identify update directions that support strong generalization using a few on-policy training steps, and subsequently apply OPSFT to achieve high training efficiency. For leveraging high-quality trajectories, OPSFT can utilize these trajectories to continue improving a post-trained model along its update direction without disrupting the ability learned from on-policy training. Codes are provided in the supplementary material.

Primary area:
reinforcement learning
Keywords:
LLM post-training, On-policy Training, LLM Fine-tuning.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UxHuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether the generalization benefits of on-policy LLM post-training can be transferred to supervised fine-tuning through parameter update directions. The authors observe that SFT updates remain relatively consistent across training stages, whereas GRPO and on-policy distillation exhibit more variation. Motivated by this observation, they propose OPSFT, which extracts the sign of each parameter’s cumulative change during on-policy training and uses it to constrain SFT updates. Across math and code reasoning benchmarks, the authors report improvements over standard SFT and performance comparable to the on-policy methods that provide the directions. They also explore two practical applications: using a short GRPO run to identify directions before switching to OPSFT, and using OPSFT to incorporate additional high-quality demonstrations into models that have already undergone on-policy training.

Critical strengths
  1. The paper turns an observation about optimization into a concrete intervention. Rather than only comparing parameter updates across training methods, the authors use the cumulative update signs from on-policy training to constrain SFT. The location-only and random-sign controls in Table A13 provide useful evidence that the learned signs contain information beyond the update locations.

  2. The empirical evaluation covers several relevant settings. The experiments include multiple model sizes, two model families, and both mathematical reasoning and code generation. The additional OPD experiment also provides some evidence that the approach is not restricted to GRPO.

  3. The proposed applications address practical post-training problems. Using a short on-policy run to guide subsequent SFT, and incorporating additional demonstrations into an already post-trained model, are both useful settings. The latter is particularly interesting given the performance degradation observed when ordinary SFT is applied after GRPO.

Critical weaknesses
  1. The evidence supports the usefulness of the constraints more clearly than the proposed explanation of generalization. Table A13 shows that on-policy signs outperform random signs, but this does not establish that the benefit is specific to on-policy training. Signs extracted from another effective training procedure might also encode useful task information. Moreover, random signs can change the proportion and magnitude of retained gradients, so their weaker performance does not fully rule out other regularization effects.

  2. The runtime results need a clearer breakdown. Section 4.1 describes a 100-step GRPO baseline and a 50-step GRPO stage followed by 100 OPSFT steps. However, all OPSFT totals in Table 2 are below half the corresponding GRPO times. For Qwen3-8B, the reported total is 8.9 hours, whereas half the GRPO time is 9.65 hours. Nonuniform step costs could explain this, but the paper does not provide the measurements needed to check it. Appendix Table A7 also lists 150 GRPO steps, leaving the correspondence between configurations and reported times unclear.

  3. The stability of the reported advantages is not established. The paper does not report independent training repetitions or confidence intervals for the key comparisons. This is especially relevant to the small advantages over GRPO, such as the 0.16- and 0.19-point mean differences in Table 1. Multiple generations per problem do not capture variation across direction extraction and training runs.

Questions for authors
  1. Could you break down the Table 2 runtime into the GRPO stage, direction extraction and checkpoint handling, and subsequent OPSFT training? In my experience training with GRPO, runtime scales approximately linearly with the number of steps, so I am having difficulty reconciling the reported times with the stated training schedule. Do the baseline times correspond to 100 or 150 GRPO steps? How much time do the first 50 GRPO steps actually take?

  2. Would directions extracted from a competitive SFT checkpoint provide similar benefits? A comparison with the same support set and matched effective update magnitude would help distinguish on-policy-specific information from more general task information or regularization.

  3. What weights initialize OPSFT in Figure 3 and Table 2? Please specify whether OPSFT starts from the original base model or the direction-providing GRPO checkpoint, whether optimizer states are inherited, and which checkpoint serves as the projection reference. If OPSFT restarts from the base model, I can see how this helps isolate the contribution of the update directions. From a practical perspective, however, the initial GRPO stage has already produced a potentially useful checkpoint. Could the authors clarify the benefit of transferring its update signs to SFT from the base model, compared with continuing SFT or OPSFT directly from the 50-step GRPO checkpoint? A comparison would help clarify the practical motivation for this choice.

If the authors can address most of my concerns with convincing experimental evidence, I would be happy to raise my score.

5 comments

Thank you very much for your feedback, which has been of great help to our work. We hope the following responses can clarify the missing points and address your concerns.

W1 & Q2: More ablations of the on-policy update directions

Thank you for your insightful suggestions, which have helped us improve the understanding of the sign constraint. To fully address your concern, we first investigate the effectiveness of the SFT direction. Then, we provide comparisons between OPSFT and magnitude-based constraints.

Effectiveness of competitive SFT directions. Following your suggestion, we construct an SFT checkpoint using trajectories generated by Qwen3-30B-A3B. Then, we use these directions to guide a weaker SFT baseline on trajectories generated by Qwen3-14B.

  • Experiments. We report the performance of OPSFT with constraints from different checkpoints (100-step GRPO, SFT) using Qwen3-8B on the Deepmath dataset.

    MethodTrajectory sourceAIME24 (%)AIME25 (%)HMMT25-Feb (%)HMMT25-Nov (%)Mean (%)
    SFT referenceQwen3-30B-A3B52.6044.4823.8532.7138.41
    GRPO reference-54.1744.5828.3334.1740.31
    Vanilla SFTQwen3-14B23.7520.429.588.3315.52
    OPSFT (SFT-direction)Qwen3-14B27.0818.3312.0811.6717.29
    OPSFT (GRPO-direction)Qwen3-14B28.3324.1713.3314.5820.10
  • Analysis. The directions of a competitive SFT checkpoint can provide slight benefits, but these benefits are less pronounced than those of GRPO checkpoints. First, the directions of a competitive SFT checkpoint slightly improve a weaker SFT baseline (from 15.52% to 17.29%). Second, the GRPO direction achieves nearly 3% more improvement than the SFT direction (20.10% vs. 17.29%), with the GRPO checkpoint outperforming the SFT checkpoint by only about 2% (40.31% vs. 38.41%).

  • Practical value comparisons. Beyond the greater performance benefits, the constraint with on-policy directions is also more practically meaningful than using SFT directions. First, on-policy paradigms generally achieve stronger generalization performance than SFT, making it easier to obtain powerful checkpoints through on-policy training. Second, using a stronger SFT model to improve a weaker SFT baseline has less obvious practical value. Unlike our application that combines the strong generalization of on-policy paradigms with the high training efficiency of SFT, strong and weak SFT baselines lack such complementary advantages.

Magnitude-based sign constraints. We report the performance of magnitude-based constraints with the same number of updated parameters as on-policy sign constraints at each training step.

ConstraintAIME24AIME25HMMT25-FebHMMT25-NovMean
Null (SFT)46.6739.7923.0227.4034.22
Random41.6734.1724.1723.3330.84
Magnitude45.8338.7523.3326.6633.64
On-policy52.5048.3331.2532.5041.15

The magnitude-based constraints lead to a significant performance drop compared to on-policy direction constraints (41.15% vs. 33.64%). Unlike on-policy directions that introduce information from another stronger training paradigm, the magnitude-based constraint only compresses the original SFT gradients. These findings further support the importance of incorporating on-policy directional information, rather than merely regularizing updates.

Revision. We have incorporated the above experiments on SFT directions in Appendix B (L909-969). Moreover, the ablations of magnitude-based sign constraints are incorporated into Section 4.3 (L455-463).

W2 & Q1: More analysis of the runtime results

Thank you for your constructive comments, which have helped us refine our comparisons. To fully address your concerns, we first report and analyze the training time of GRPO at different stages, followed by a detailed breakdown of the training time for each stage of OPSFT.

GRPO training time. We report the training time of GRPO at different stages on Qwen3-8B in the table below.

Training stageTraining time (h)Average time per step (s)Average response length (tokens)
Steps 1–507.3525.41,773.5
Steps 51–10011.9859.72,779.6
Steps 101–15018.01,296.34,152.3

The average time per GRPO step increases as training progresses. This is because we set a large maximum response length (16,384 tokens) following previous work [1]. This setting allows the model to generate increasingly longer responses as training progresses, a phenomenon also widely observed in prior studies like DeepSeek-R1 [2]. The increasing response length leads to greater computations during rollouts and policy updating, thereby resulting in longer training time in later GRPO stages.

Additionally, we would like to clarify that the training step reported in Table A7 indicates the configuration in Figure 3, whereas the runtime of the GRPO baseline in Table 2 is measured over 100 training steps.

Detailed training time of OPSFT. We report a detailed breakdown of the training time in Table 2 across different stages within OPSFT on Qwen3-8B.

StageTimeShare of total time (%)
Initial GRPO training7.3 h81.9
Direction extraction1 min0.2
Continual OPSFT1.6 h17.9
Total8.9h100.0

First, the initial GRPO training accounts for the majority of the total runtime. Second, the direction extraction incurs negligible overhead as it requires no additional training. Finally, OPSFT training itself accounts for only 17.9% of the total runtime.

Revision. We have illustrated the nonuniform step costs of GRPO in Section 4.1 (L413-416), explicitly clarified the optimization step in Table A7, and incorporated the above experiments & analysis into Appendix B (L1024-1054).

[1] Yang, W., Liu, W., Xie, R., Yang, K., Yang, S., & Lin, Y. (2026). Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125.

[2] Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

W3: Advantage stability

Thank you for your valuable comments that help us further improve the experiments. Following your suggestions, we conducted three independent runs of GRPO training, direction extraction, and OPSFT training on Qwen3-8B with the Deepmath dataset. We report the mean performance and standard deviation across multiple runs on four math tasks and three held-out tasks as follows.

MethodAIME24AIME25HMMT25-FebHMMT25-NovMean
SFT50.69 ±\pm 2.4443.88 ±\pm 2.9222.63 ±\pm 2.1332.36 ±\pm 2.9537.39
GRPO57.50 ±\pm 3.0145.27 ±\pm 3.7826.94 ±\pm 2.2937.36 ±\pm 1.9641.77
OPSFT55.69 ±\pm 2.5447.63 ±\pm 2.7127.22 ±\pm 2.0938.47 ±\pm 3.1242.25
MethodARCHaluEvalWinograndeMean
SFT87.79 ±\pm 0.2273.23 ±\pm 0.2166.11 ±\pm 0.1975.71
GRPO87.88 ±\pm 0.1773.53 ±\pm 0.2966.42 ±\pm 0.2775.94
OPSFT87.96 ±\pm 0.1773.83 ±\pm 0.1566.48 ±\pm 0.1676.09

The results of multiple runs are consistent with our original conclusions: OPSFT largely exceeds vanilla SFT and consistently achieves performance comparable to on-policy paradigms that provide the update direction, further demonstrating the robustness of our findings across three independent runs.

Revision. We have incorporated the above experiment in Appendix B (L1054-1061).

Q3: The initialization of each experiment

Thank you for your careful feedback that helps us better illustrate the experimental settings. We adopt different initializations and directions in Figure 3 and Table 2 depending on their investigation objectives.

Optimizer states. OPSFT does not inherit optimizer states from GRPO training across all experiments, including Figure 3 and Table 2.

In Figure 3, we try to isolate the contribution of the update directions. Therefore, both OPSFT and SFT are initialized from the base model for comparisons under the same training procedure. The directions are identified from a 150-step GRPO checkpoint with strong generalization performance.

In Table 2, we try to improve the training efficiency. Thus, OPSFT is initialized from the 50-step GRPO checkpoint for fast convergence. The directions are also extracted from this checkpoint, as getting the directions via 150-step GRPO is expensive. To better illustrate the effect of this initialization strategy on performance, we provide experiments as follows:

  • Experiments. We evaluate SFT and OPSFT initialized from the base model and the GRPO checkpoint.

    InitializationMethodAIME24AIME25HMMT25-FebHMMT25-NovMean
    Base-21.1521.6712.087.8115.68
    GRPO50-33.1327.7115.8316.6723.33
    BaseSFT40.4246.6724.5827.9234.90
    GRPO50SFT42.5040.0020.8329.5833.23
    BaseOPSFT47.0848.3324.5834.1738.54
    GRPO50OPSFT52.9250.4225.8331.2540.11
  • Analysis. OPSFT achieves better performance when initialized from the GRPO checkpoint (40.11 vs. 38.54), whereas SFT exhibits the opposite trend (33.23 vs. 34.90). This is because vanilla SFT conflicts with on-policy updates, leading to weaker performance with a stronger initialization. These findings explain our choice to initialize OPSFT from the GRPO checkpoint and SFT from the base model in Table 2.

Revision. We have clarified the initialization settings of OPSFT across different experiments in Sections 3 and 4 (L261-262, L402-403). Additionally, we have included the above experiments comparing different initialization choices in Section 4.3 (L455-463).

Reviewer_ucitOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides empirical insights on how to leverage on-policy post-training methods to improve SFT, and proposes OPSFT, which constrains SFT updates to the parameter update direction identified by the on-policy method. OPSFT combines the benefits of SFT (leveraging high-quality trajectories) with the generalization of on-policy methods.

Critical strengths
  1. The paper offers an interesting perspective by comparing SFT and on-policy methods through their parameter update directions.
  2. The paper's core finding is consistent: OPSFT beats vanilla SFT and roughly matches on-policy methods across models and domains (math and code)
Critical weaknesses
  1. App. A.2 mentions "Table 4.1", which does not exist. Which table does this refer to, and what learning rate did the SFT baseline in that table use?
  2. Comparison with related works like RL's Razor https://arxiv.org/abs/2509.04259 and Beyond Magnitude https://arxiv.org/abs/2603.22117 could be provided to help clarify the difference and strength of OPSFT vs prior works
Questions for authors

See Weaknesses

3 comments

We would like to thank you for the valuable comments! We will diligently follow your guidance to improve our manuscripts.

W1: Clarification of App.A.2 and Table 4.1

Thank you very much for your careful feedback, which helped us identify and correct this typo. We would like to clarify that "Table 4.1" should refer to "Section 4.1." Regarding the learning rate, the SFT baseline uses the same learning rate of 1×10−61\times10^{-6} as OPSFT.

Revision. We have corrected these typos and explicitly clarified in Appendix A.2 (L803-804) that both SFT and OPSFT in Section 4.1 use the same learning rate of 1×10−61\times10^{-6}.

W2: More discussions and comparisons with previous studies

Thank you very much for bringing these insightful studies to us. We provide a detailed comparison with each of the two studies below, highlighting their similarities and differences with our work.

RL's Razor.

  • Similarities. Both RL's Razor and OPSFT analyze the differences between reinforcement learning and SFT, and their findings contribute to a deeper understanding of the distinctions between these two training paradigms.
  • Differences. (i) For motivation, RL's Razor primarily focuses on RL's resistance to catastrophic forgetting on held-out tasks, whereas OPSFT investigates RL's strong generalization capabilities within the target domain. (ii) For method design, RL's Razor focuses on the training objective, while OPSFT directly constrains parameter-wise update directions. (iii) For application, while RL's Razor focuses on analyzing and understanding on-policy training paradigms, OPSFT further translates these insights into a practical intervention that improves the generalization performance of SFT.

Beyond Magnitude.

  • Similarities. Both studies investigate the direction of RL updates and explore how such directional information can be leveraged to improve existing post-training paradigms. Their findings mutually reinforce the insight that update directions play an important role in effective post-training.
  • Differences. (i) For the analysis objective, this paper focuses on the directions of changes in token log-probabilities, whereas our OPSFT investigates parameter-wise update directions. (ii) For method design, this paper modifies training objectives at the token level, while our OPSFT directly constrains update directions at the parameter level. (iii) For application, while this paper aims to improve RL itself, OPSFT transfers the beneficial optimization principles of RL to SFT, integrating both advantages of RL and SFT.

Revision. We have incorporated both studies in Section 2 (L112-116) and provided detailed discussions in Appendix C.2 (L1104-1126).

Thank you for clarifying that Appendix A.2 refers to Section 4.1 and that SFT and OPSFT use the same learning rate. The added discussion of RL’s Razor and Beyond Magnitude also makes it clearer how OPSFT differs from these prior approaches. This addresses both points I raised.

Dear reviewer, we sincerely appreciate your feedback and are glad that our responses have addressed your concerns. Thank you again for your time and constructive comments.

Reviewer_3fE3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces OPSFT, which extracts the signs of cumulative parameter changes from on-policy training and uses them to constrain supervised fine-tuning through gradient masking and parameter projection. Experiments on mathematical reasoning and code generation demonstrate improved accuracy over vanilla SFT, reduced training time relative to GRPO, and benefits when continuing to train models that have already undergone reinforcement learning.

Critical strengths
  • Simple and practical approach. The method retains the standard cross-entropy objective and uses information available from two checkpoints.
  • Meaningful empirical benefits. Results cover several models and both mathematics and code. The efficiency comparison includes the cost of identifying the direction through GRPO (Table 2).
  • Relevant ablations. Table A13 provides useful evidence that learned update signs contribute beyond selecting update locations or imposing random signs.
Critical weaknesses
  1. The theoretical interpretation is stronger than the derivation supports. A fixed teacher distribution in Eq. (6) does not imply fixed SFT gradient directions, since the score function changes with the parameters. Likewise, the covariance identity in Eq. (5) does not establish continuous direction changes or improved generalization. Figure 2 provides empirical observations, but the connection between evolving on-policy updates and reusing a fixed cumulative sign pattern remains incomplete. These observations and mathematical deductions should be clearly distinguished.

  2. The limits of transferring historical signs are insufficiently examined. A direction that was useful during earlier on-policy training need not remain suitable for later teacher demonstrations, particularly those containing strategies rarely generated by the starting model. The constraint may suppress useful SFT updates. The paper does not directly investigate the resulting trade-off between retaining existing behavior and learning from substantially different demonstrations. Claims about continuing SFT without disrupting acquired capabilities should be qualified accordingly.

  3. The evaluation does not sufficiently resolve the proposed mechanism. Held-out accuracy gains support empirical generalization, but they do not distinguish preservation of existing solutions, acquisition of new strategies, and task-specific regularization. Table A13 establishes the value of informed signs, while the behavioral proxies in Figure 7 do not directly measure output entropy or solution diversity. The average accuracy reported in Appendix A.3 also does not characterize performance across different sampling budgets. These limitations matter for the paper's mechanistic interpretation, rather than invalidating its reported accuracy improvements.

Questions for authors
  1. Which statements in Section 3.1 follow mathematically, and which are empirical observations? Please explain the connection between continuously changing on-policy updates and a fixed cumulative sign constraint.

  2. In the efficiency experiments in Section 4.1, does OPSFT start from the original base model or the GRPO checkpoint used to identify the direction? Please specify the initialization of each baseline and clarify whether the comparisons isolate the contribution of the sign constraint.

  3. How much SFT gradient norm is removed by the support constraint and by the sign constraint, respectively? Please provide a targeted analysis of correct teacher trajectories with low likelihood under the initial student, controlling for trajectory length, to examine whether useful supervision is disproportionately filtered.

  4. Can the authors provide a representative pass@k comparison across sampling budgets and a direct analysis of solution diversity or retained versus newly solved examples? Such evidence would help characterize the accuracy-coverage trade-off beyond the proxies in Figure 7.

5 comments

Thank you very much for the detailed comments and suggestions, which have greatly benefited our research. We hope the following responses can address your concerns.

W1 & Q1: Re-clarification of our theoretical analysis results

Thank you for your insightful comments, which have helped us refine our theoretical interpretation. To fully address your concerns, we first interpret the theoretical findings and their connections with the proposed cumulative sign constraint. Then we clarify the consistency between theoretical and experimental findings.

Theoretical interpretation. We agree with your insight that the score function evolves with the model parameters. Beyond refining our interpretations in revision, we would like to clarify that the original motivation behind Eqs. (5)–(6) is leveraging gradient-based derivations to compare the target distributions of SFT and on-policy paradigms, thereby providing insights into their cumulative update directions rather than per-step gradients. From this perspective, the cumulative update direction of SFT is relatively consistent as the model is optimized from the base model toward a fixed teacher distribution. In contrast, the cumulative update directions of on-policy paradigms continuously adjust as the target distribution evolves throughout training.

Connection between evolving directions and fixed sign constraints. The fixed sign constraints for SFT capture the outcome of the direction evolution in on-policy training. The evolution of cumulative update directions is typically accompanied by increasing generalization performance during on-policy training. Therefore, cumulative update directions from the base model to later checkpoints are expected to capture more benefits than those from earlier checkpoints. Therefore, we extract the direction from the final checkpoint (i.e., the outcome of direction evolution in on-policy training) as the fixed sign constraint.

Consistency between theoretical and experimental findings. According to the above re-clarified interpretations, the experiments in Section 3.1 validate the theoretical findings rather than introduce new statements. First, the experiments on cumulative update directions yield observations consistent with the theoretical findings (i.e., the continuous adjustment of cumulative on-policy update directions). Second, the experiments on interval update directions highlight how the evolving target distribution in on-policy paradigms influences the update direction.

Revision. We have reorganized the theoretical and experimental analysis in Section 3.1 (L179-187). Additionally, we have clarified the connection between evolving directions and the fixed sign constraint in Section 3.2 (L203-205). These revisions illustrate the behavior of on-policy update directions more clearly without affecting our proposed method or subsequent experiments.

W2: The trade-off introduced by sign constraints

Thank you for your valuable comments. To fully address your concern, we first analyze this trade-off and then clarify the claims in Section 1.

Trade-off Analysis. We appreciate your insight regarding the trade-off between historical on-policy training and later teacher trajectories. From this perspective, we view SFT (following teacher guidance) and OPSFT (following directions of historical on-policy training) as two ends of this trade-off. To investigate this trade-off, we relax the direction constraint in OPSFT through a hyperparameter β\beta and modify Eq.7 of our paper as: gs=I(sign(−g)=v)⊙g+β⋅I(sign(−g)≠v)⊙gg_s = \mathbb{I}\left( \textit{sign}(-g) = v \right) \odot g +\beta\cdot\mathbb{I}\left( \textit{sign}(-g) \ne {v} \right) \odot {g}.

  • Experiments. We conduct experiments with different β\beta on Qwen3-8B.

    β\betaMath Mean ↑\uparrowTeacher NLL ↓\downarrow
    0.00 (OPSFT)39.690.3413
    0.2539.380.3386
    0.5038.850.3363
    0.7536.880.3344
    1.00 (SFT)36.560.3326
  • Analysis. The student achieves lower NLL (negative log-likelihood) on teacher trajectories as the constraints are progressively relaxed, indicating that the constraints limit the student from fully receiving teacher supervision. However, the accompanying performance drop suggests that the supervision filtered out by OPSFT does not contribute to better generalization. At least under the SFT training framework, incorporating such supervision hinders further performance improvements.

  • Future investigation. Although the above results suggest that the supervision filtered by OPSFT may be harmful to performance, we agree that this trade-off deserves further investigation by considering the quality of teacher trajectories. Inspired by your suggestion, we will further explore OPSFT under teacher demonstrations with different qualities in future work.

Clarification of capability retention: We would like to clarify that the statement "without disrupting acquired capabilities" in Section 1 specifically refers to the domain-specific generalization capabilities acquired through on-policy training, rather than capabilities that could be learned from teacher trajectories.

Revision. We have added the trade-off analysis to Appendix B (L970-1003) and incorporated the discussion of future research directions into Section 5 (L538-539). Additionally, we have qualified the claim regarding "without disrupting acquired capabilities" in Section 1 (L096-097).

W3 & Q4: Evaluations on more perspectives

Thank you for your detailed feedback that helps us improve our evaluations. Following your suggestions, we provide additional comparisons of pass@k, retained&newly solved examples, and solution diversity using Qwen3-8B.

Pass@k. OPSFT exhibits a similar trend to GRPO, with its improvements over vanilla SFT being most pronounced at smaller values of k. This suggests that OPSFT and GRPO reduce the sampling cost required to obtain correct solutions, consistent with findings from prior work [1]. Moreover, OPSFT consistently outperforms vanilla SFT across different values of k.

MethodPass@1Pass@2Pass@4Pass@8
Vanilla SFT37.5048.3355.8361.67
GRPO42.5050.0057.5061.67
OPSFT43.3352.5058.3362.50

Retained&newly solved examples of pass@8. OPSFT exhibits a pattern similar to GRPO. Both methods retain the ability to handle problems that are solvable for base models, whereas vanilla SFT tends to forget existing capabilities while learning to solve new problems.

MethodRetainedNewly solvedLost
Vanilla SFT38362
GRPO42321
OPSFT42331

Diversity. We compare the average token-level logit entropy and textual diversity (measured by the 4-gram Jaccard distance between different responses to the same problem) across different methods. OPSFT and GRPO exhibit higher token-level entropy than vanilla SFT, while all three methods achieve comparable levels of textual diversity.

MethodEntropy (per token)Textual diversity
Vanilla SFT0.43960.9083
GRPO0.54630.8904
OPSFT0.54600.9110

Revision. We have incorporated the above experiments and analysis in Appendix B (L1004-1016).

[1] Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., & Huang, G. (2026). Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. Advances in Neural Information Processing Systems, 38, 57654-57689.

Q2: The initialization of OPSFT in Section 4.1.

Thank you very much for your careful feedback, which has helped us clarify our experimental settings and better demonstrate the effectiveness of OPSFT.

Initialization of each method. First, GRPO and SFT are initialized from the same base model (e.g., Qwen3-4B). Second, OPSFT starts from the GRPO checkpoint used to identify the direction. Note that the training time of OPSFT in Table 2 includes both the initial GRPO and the subsequent OPSFT to ensure a fair comparison.

Contribution of the sign constraint. We isolate the contribution of the sign constraint by reporting the performance of OPSFT's initialization (the GRPO 50-step checkpoint) and SFT with this checkpoint on Qwen3-4B.

InitializationMethodAIME24AIME25HMMT25-FebHMMT25-NovMean
GRPO50-33.1327.7115.8316.6723.33
BaseSFT46.6739.7923.0227.4034.22
GRPO50SFT42.5040.0020.8329.5833.23
GRPO50OPSFT52.9250.4225.8331.2540.11

(i) OPSFT achieves significant performance improvement compared to its initialization (40.11 vs. 23.33), demonstrating the effectiveness of the proposed sign constraint. (ii) Notably, SFT initialized from the GRPO checkpoint performs worse than both OPSFT (33.23 vs. 40.11) and SFT initialized from the base model (33.23 vs. 34.22). This observation further highlights the ability of OPSFT to leverage GRPO capabilities.

Revision. We have clarified the initialization procedure of OPSFT in Section 4.1 (L402-403) and incorporated the above experiments into Section 4.3 (L464-469).

Q3: Analysis of filtered SFT gradient norm.

Thank you for your constructive comments that help us better illustrate the internal mechanism of OPSFT. To fully address your concern, we first analyze the SFT gradient norms filtered by the proposed constraint and then investigate the contribution of the filtered part to generalization performance.

Filtered gradient norm. Following your suggestion, we group teacher trajectories according to their sequence length and negative log-likelihood (NLL) under the initial student, respectively. Then, we quantify the proportion of gradient norms filtered by the support and sign constraints in OPSFT using Qwen3-8B.

Length (tokens)NLL GroupSupport (%)Support+Sign (%)
0–2048Low NLL72.0876.93
0–2048Middle NLL71.5877.17
0–2048High NLL72.7075.95
2048–4096Low NLL70.1575.62
2048–4096Middle NLL71.3276.53
2048–4096High NLL71.0475.73
4096–8192Low NLL69.4775.94
4096–8192Middle NLL70.9775.81
4096–8192High NLL71.4775.30

First, the support constraint filters out a major portion of the gradient norm (around 70%). The sign constraint further filters out an additional 3%–7% of the gradient norm. Second, OPSFT exhibits consistent filtering ratios across different trajectory groups (approximately 75%). This suggests that the SFT supervision is filtered uniformly rather than disproportionately.

Contributions the filtered part. We gradually relax the constraint in OPSFT through a hyperparameter β\beta and modify Eq.7 of our paper:

gs=I(sign(−g)=v)⊙g+β⋅I(sign(−g)≠v)⊙g{g}_s = \mathbb{I}\left( \textit{sign}(-{g}) = {v} \right) \odot {g} +\beta\cdot\mathbb{I}\left( \textit{sign}({-g}) \ne {v} \right) \odot {g}

The proportion of filtered gradient norm and corresponding performance on math tasks with Qwen3-8B are reported as follows:

Length (tokens)NLL Groupβ\beta=0 (OPSFT)β\beta=0.25β\beta=0.50β\beta=0.75β\beta=1 (SFT)
0–2048Low NLL76.93%57.72%38.49%19.24%0.00%
0–2048Middle NLL77.17%57.74%38.59%19.29%0.00%
0–2048High NLL75.95%56.99%37.98%18.99%0.00%
2048–4096Low NLL75.62%56.66%37.78%18.90%0.00%
2048–4096Middle NLL76.53%57.47%38.32%19.16%0.00%
2048–4096High NLL75.73%56.80%37.85%18.93%0.00%
4096–8192Low NLL75.94%56.96%37.96%18.96%0.00%
4096–8192Middle NLL75.81%56.89%37.92%18.93%0.00%
4096–8192High NLL75.30%56.50%37.66%18.83%0.00%
Math Mean Acc.-39.69%39.38%38.85%36.88%36.56%

As the constraints are progressively relaxed, the proportion of retained SFT gradient norm increases, while performance gradually declines. This suggests that OPSFT filters out gradients that hinder the performance improvement on the target domain, rather than those providing useful supervision.

Revision. We have incorporated the above experiments into Appendix B (L1017-1023).

Comments on the paper

1 comment

We would like to thank all the reviewers for their diligent efforts and valuable suggestions, which have greatly contributed to improving the quality of our manuscript.

Summary of strengths:

We sincerely appreciate that you find our method:

  • useful or interesting (reviewers 3fE3, ucit, UxHu);

  • offers meaningful empirical benefits (reviewers 3fE3, ucit, UxHu);

  • clear and easy to follow (reviewers 3fE3, UxHu);

  • provides relevant ablations (reviewers 3fE3, UxHu);

  • simple and practical (reviewer 3fE3);

  • addresses practical post-training problems (reviewer UxHu);

  • conducts intervention rather than only observation (reviewer UxHu);

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.