acceptodds
Under review as a conference paper at ICLR 2027

Sharp at the Peak, Rich in the Prior: Peak-Preserving Prior Distillation for RL-Ready Large Language Models

Abstract

Supervised fine-tuning (SFT) for large language models should acquire demonstrated behavior while retaining useful pretrained preferences and alternatives for subsequent reinforcement learning (RL). Conventional one-hot supervision promotes the demonstrated token without specifying which relationships among alternative continuations should be preserved. We introduce **Peak-Preserving Prior Distillation ()**, a principled approach that specifies the desired supervisory distribution before optimization. Guided by **Greedy Guarantee** and **Minimal Intervention**, constructs a closed-form target that promotes demonstrated-token dominance while preserving relative probabilities among retained non-target alternatives from the current student. This directly embeds acquisition and retention into the supervision target, providing dense feedback without teacher probability vectors or an additional teacher forward pass. Experiments across three models from two families on mathematical and medical reasoning demonstrate stronger SFT and subsequent GRPO performance alongside strong general capabilities. Compared with the respective HardSFT baselines, domain-averaged greedy accuracy improves by up to 6.1 percentage points after SFT and 10.3 points after GRPO. Subsequent analysis further support effective task adaptation with limited predictive drift and retained prior preferences. These findings highlight principled target design as a foundation for learning demonstrated behavior while preserving the capacity for further reinforcement learning. Our code has been open-sourced at the anonymous link https://anonymous.4open.science/P3D-433F.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Supervised Fine-tuning, Distillation, Reinforce Learning, Prior Distribution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ujuQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

P3D is an SFT objective for the SFT-then-RL pipeline. At each token it builds a soft target from the model’s own detached prediction. The demonstrated token is raised to a margin above the point where it would tie its strongest competitor, and all other tokens are rescaled by one common factor so their ratios are kept. The model is fit to this target with cross-entropy. Across Qwen3-4B/8B and Llama3-8B on math and medical reasoning, P3D beats standard SFT in greedy accuracy, both after SFT and after GRPO.

Critical strengths

The method is simple and cheap with a closed-form target with no reference model or extra forward pass. Its greedy gains over HardSFT hold across three models and two domains

Critical weaknesses
  • Unless I am missing something, the soft target reduces to a reweighted NLL. QQ is built from the model’s own detached prediction at the current parameters, so the logit gradient of the P3D loss is just wtw_t times the NLL gradient, with wt=max⁡(0, b+δ−p⋆)/(1−p⋆)w_t=\max(0,\,b+\delta-p^\star)/(1-p^\star). The prior term in Eq. 26 adds nothing new either. Its gradient points opposite to the NLL gradient and simply makes the step smaller. So the ratio preservation in Eq. 6 holds for the target but not for the update, and the update only uses the demonstrated token plus one scalar per position. I don’t see where the dense feedback comes from. This makes P3D a confidence-based weighting scheme, the same family the paper criticizes (DFT, EAFT), and it weakens contribution 1 (target design instead of loss design).

  • I am not convinced by the explanation of the gains. Only the scalar cc ends up in the gradient, so the derivation in Appendix B is effectively choosing a weight. In fact, uniform rescaling is exactly what turns the target into a weighted one-hot. The exact capped solution would not reduce this way. The resulting weight behaves like a hinge. With delta = 0.3, every token with p⋆≥0.8p^\star \ge 0.8 gets zero weight, and by Table 3 that is more than half of the tokens at the start of training. A simpler story then fits the retention and RL results just as well: P3D trains less on tokens the model already predicts. The paper needs an argument or a control that rules this out, such as NLL masked at the same threshold, or HardSFT with a smaller learning rate.

Questions for authors
  • Do you agree that the per-token logit gradient is wtw_t times the NLL gradient? If so, what does P3D add over NLL with this weight?

  • How does P3D compare with NLL masked at the same threshold, and with HardSFT at a lower learning rate?

  • Smaller points: was delta chosen on held-out data, how many seeds underlie Table 1, and why does the shared GRPO recipe lower HardSFT on Qwen3-8B and Llama3-8B Math?

6 comments
AuthorsOpenReview ↗

Response to Reviewer ujuQ (Part 1/4): Target Design

W1 & Q1: Weighted NLL equivalence, target design, and dense supervision

Thank you for the careful derivation and insightful comments. Takeaway. We agree with the gradient identity. We believe our contribution is to turn “learn the demonstration, retain the prior” into a concrete supervision rule. We connect this target design to adaptive learning and the practical value of soft supervision, clarifying both the weighting rule and the meaning of dense feedback. We will release the exact KL projection, P³D (Capped), alongside our current uniform-rescaling implementation, P³D (Uniform), with comparative experiments to guide their use.

At each active token, P³D (Uniform)'s gradient follows the NLL direction, as in HardSFT. However, at the sequence or step level, the relative weights assigned to different tokens can change the aggregate direction compared with uniformly weighted HardSFT. Figure 4 provides complementary evidence that the methods produce different final predictive distributions and learning outcomes.

Our contribution is the target-design framework and the supervision rule it induces. For P³D (Uniform), NLL with these same detached weights is an equivalent implementation of that design. P³D starts from two requirements, “Greedy Guarantee” and “Minimal Intervention,” and constructs a supervisory distribution from the student's current predictions and the demonstrated token. We first solve for a target under these requirements, then fit it using cross-entropy. Uniform's resulting weight depends on both the demonstrated-token probability and its strongest competitor, carrying the target-design information into optimization (§§4.1 to 4.3; Appendices B.2 to B.5).

This is also where P³D differs from DFT's target-confidence weighting and EAFT's entropy gating. Entropy alone can be a misleading learning signal [1][2]. P³D explicitly specifies the desired relationship between acquisition and retention in the supervisory target. Table 1, Figure 4, and the SFT dynamics provide empirical evidence for the resulting behavior; Appendix F details the mechanism comparisons.

We agree that a shared-parameter update need not preserve the target's non-target ratios. Soft-target supervision can still outperform hard-label training without exact target matching. Hinton et al. noted that students typically cannot exactly match soft targets, yet showed that transferring relationships among classes improves generalization over hard labels alone [3]. GKD likewise demonstrates useful distributional supervision despite limited student capacity [4]. In P³D, target design determines the adaptive weighting rule. Our domain gains, general-capability retention, and smaller policy shift demonstrate its practical value (Table 1; Figure 4; Q2).

Preserving non-target ratios is what yields the weighted-NLL form. For cross-entropy with detached targets on the same normalized support, this constraint is necessary for gradient equivalence to weighted NLL; Appendix B motivates this choice. The original P³D (Capped) projection can change these ratios and produce corrections beyond the NLL direction, but requires bisection to construct the target. Both constructions follow the same target-design principles, with different trade-offs. We will release both implementations and have added experiments to guide their use. Below, we compare them on Qwen3-4B at δ=0.30 with matched top-200 support and leading gaps. Scores are domain-average greedy / pass@5 (%); Gen. Avg. covers seven benchmarks (Appendix B.9, Table 2).

DomainP³D constructionSFT: greedy / pass@5SFT Gen. Avg.GRPO: greedy / pass@5GRPO Gen. Avg.
MathUniform54.0 / 69.954.562.7 / 77.453.1
MathCapped53.6 / 70.454.762.0 / 80.253.5
MedicalUniform54.7 / 86.052.970.0 / 87.453.4
MedicalCapped52.5 / 86.753.369.3 / 88.153.1

P³D (Uniform) retains competitive performance with much lower target-construction cost: 0.257 ms versus 6.248 ms for Capped on 8,192 Math logit positions (medians; Appendix B.9, Table 3). The two constructions are highly similar and both balance retention and acquisition; Appendices B.6 and B.9 provide the detailed comparison. For practical use, consider Capped for sampled coverage and limited policy shift, or Uniform for faster target construction and higher greedy accuracy at δ=0.30; Appendix I.1 guides δ for both.

AuthorsOpenReview ↗

Response to Reviewer ujuQ (Part 2/4): Target Design and Update Controls

To address your concern about “dense” feedback, we distinguish two aspects. First, the design motivation is distributional: the supervisory target explicitly considers non-target candidate structure alongside the demonstrated preference. The added P³D (Capped) experiments test this motivation beyond the P³D (Uniform) implementation.

Second, both constructions carry dense distributional information into the learning signal. Capped provides candidate-level supervision by fitting the projected distribution. Uniform preserves the relative probabilities of non-target candidates and translates the demonstrated-token probability and its strongest competitor into an adaptive learning weight. Although Uniform reduces to weighted NLL, its weight carries information about competing alternatives beyond the demonstrated label. The distributional design therefore remains effective through a simple, efficient optimization rule.

W2 & Q2: Do masking or smaller updates explain the gains?

Thank you for suggesting these comparisons. Takeaway. We believe the key is how learning is allocated, including among tokens that remain active. P³D adjusts that allocation as the model changes; we use the masking, learning-rate, stopping and norm controls to test this explanation, then connect the performance differences to policy shift.

We tested whether P³D's gains could be explained by masking tokens, lowering the learning rate, matching gradient norms, or stopping earlier. The key distinction is how much each active token learns. Related RL work also finds that a minority of uncertain tokens can contribute disproportionately to learning [5]. This provides a rationale for why reducing updates where the demonstrated token already has high probability can be beneficial. P³D adjusts each active token's weight using its demonstrated-token probability and strongest competitor; a binary mask applies standard NLL to the tokens it keeps. Matching the active set therefore does not reproduce P³D's allocation of learning within it.

Throughout SFT, P³D recomputes each token’s weight from the model’s current predictions, reducing or stopping updates when the required preference is met while continuing to correct unresolved tokens. A smaller global learning rate scales all updates, and early stopping ends them together. For the dynamic-mask baseline, we use the current model's predictions at each step to mask out the tokens that P³D's rule would skip, then apply standard NLL to the remaining tokens.

We ran these controls on Qwen3-4B with the same normalized top-200 support and GRPO recipe, using δ=0.30 for P³D. Norm control approximately matches the mean gradient norm across training phases. Each benchmark cell gives post-GRPO greedy accuracy / pass@5 (%); Domain Avg. follows the paper, and Gen. Avg. averages seven general benchmarks (Appendix F.7, Table 12).

Math, after GRPO. EAFT, GEM, and LS norm controls keep each method's default coefficient (Appendix F.7, Table 13).

SFT initializationMATH500GSM8KAIME24AIME25MinervaDomain Avg.Gen. Avg.
NLL (top-200)78.4 / 87.092.2 / 96.420.0 / 30.016.7 / 33.374.1 / 82.856.3 / 65.953.2
NLL + mask (p*<0.8)77.8 / 87.691.7 / 95.723.3 / 33.326.7 / 43.367.4 / 78.857.4 / 67.852.7
NLL + activation mask80.0 / 88.291.7 / 96.520.0 / 40.026.7 / 43.368.3 / 79.257.3 / 69.552.8
NLL (LR 5×10⁻⁶)76.2 / 86.285.8 / 73.613.3 / 26.713.3 / 43.374.3 / 83.252.6 / 62.653.1
NLL (LR 2.5×10⁻⁶)76.2 / 86.887.1 / 94.510.0 / 33.310.0 / 33.374.1 / 82.151.5 / 66.052.7
NLL + norm matching77.8 / 87.491.9 / 96.420.0 / 33.323.3 / 43.374.3 / 83.357.5 / 68.753.7
NLL (1 SFT epoch)78.0 / 86.693.7 / 95.513.3 / 36.713.3 / 43.374.9 / 83.254.7 / 69.052.7
EAFT (α=1)76.8 / 85.492.3 / 96.516.7 / 30.020.0 / 33.374.3 / 83.456.0 / 65.753.4
EAFT + norm matching79.2 / 85.490.9 / 95.220.0 / 36.720.0 / 43.374.2 / 83.656.8 / 68.853.4
GEM (β=0.7)74.4 / 84.092.5 / 95.623.3 / 33.320.0 / 36.779.4 / 87.557.9 / 67.450.0
GEM + norm matching78.0 / 86.691.1 / 94.423.3 / 33.316.7 / 43.373.2 / 82.656.5 / 68.153.1
LS (ε=0.1)76.0 / 87.690.4 / 96.413.3 / 33.316.7 / 33.369.8 / 84.053.2 / 66.953.0
LS + norm matching78.0 / 87.091.6 / 93.526.7 / 36.723.3 / 43.373.7 / 82.758.7 / 68.652.2
P³D (Uniform)82.0 / 92.294.2 / 97.730.0 / 60.030.0 / 50.077.5 / 87.362.7 / 77.453.1
AuthorsOpenReview ↗

Response to Reviewer ujuQ (Part 3/4): Medical Controls and Policy Shift

Medical, after GRPO.

SFT initializationMedQAMedMCQAPubMedQADomain Avg.Gen. Avg.
NLL (top-200)73.6 / 88.861.1 / 81.468.4 / 84.467.7 / 84.952.4
NLL + mask (p*<0.8)73.1 / 90.762.1 / 80.915.0 / 50.650.1 / 74.153.1
NLL + activation mask72.8 / 91.361.5 / 81.270.2 / 85.468.2 / 85.952.7
NLL (LR 5×10⁻⁶)72.7 / 88.059.8 / 81.969.0 / 86.267.2 / 85.452.9
NLL (LR 2.5×10⁻⁶)69.4 / 88.857.2 / 80.766.0 / 85.064.2 / 84.852.3
NLL + norm matching75.3 / 91.862.6 / 81.769.8 / 84.869.2 / 86.152.4
NLL (1 SFT epoch)73.1 / 89.259.1 / 81.064.2 / 84.065.5 / 84.752.0
P³D (Uniform)74.5 / 90.362.9 / 82.972.4 / 89.070.0 / 87.453.4

We used masking, early stopping, lower learning rates, and approximate adaptive norm matching to control the amount and strength of learning. The comparisons include NLL and norm-controlled EAFT, GEM, and LS. P³D remains ahead in domain-average greedy accuracy and pass@5, showing that its gains are not explained simply by fewer or smaller updates.

The policy-shift plot supports the same conclusion: the tested learning-rate and norm controls do not reproduce P³D's smaller distribution shift together with its performance gains (Appendix F.7, Figure 9).

Reviewer_xKxJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a P³D supervised fine-tuning method: it increases the probability of demonstration words while preserving the relative preferences among non-target words, maintaining exploration space for subsequent reinforcement learning. Experiments show that it outperforms the standard SFT on both mathematical and medical tasks and significantly improves the performance after GRPO.

Critical strengths
  • The overall idea is excellent, and a detailed theoretical analysis has been provided.
  • A clear paradigm for SFT target design has been proposed.
  • Extensive experiments have been conducted to verify the effectiveness of the method.
Critical weaknesses
  • The paper only presents entropy curves for the RL phase; it lacks the corresponding curves for the SFT phase, as well as comparisons across different methods.
  • Experiments on more powerful models—such as those in the Qwen3.5 series with comparable parameter counts—are missing.
  • The optimal value for δ\delta can only be determined through large-scale experimentation; there is no guidance provided on how to set δ\delta for practical, real-world applications.
Questions for authors
  • Does P³D still have an effect in more powerful models, such as the Qwen3.5 model?

  • Will using different delta settings for different models yield better results? In practical applications, how should delta be set specifically?

  • In Figure 4a, ASFT and DFT appear to perform worse than the base model (judging from the colors). Is there a problem?

5 comments
AuthorsOpenReview ↗

Response to Reviewer xKxJ (Part 1/3): SFT Dynamics

Thank you for the positive assessment and constructive suggestions. We have added the requested SFT dynamics and newer-model comparisons, and clarified the practical margin guidance and Figure 4.

W1: SFT entropy curves and comparisons across methods

We have added the SFT curves you requested. Takeaway. We believe the important behavior is learning the demonstration while retaining predictive uncertainty. The SFT curves let us examine both together; combined with the existing training curves and GRPO results, they explain why this supervision provides a useful RL initialization.

They track predictive entropy, demonstrated-token probability, and unweighted NLL for Math and Medical, comparing P³D, HardSFT, DFT, EAFT, ASFT, and EntropyReg. Measurements use fixed demonstration contexts and response positions. The final-checkpoint entropy values below illustrate the difference between P³D and HardSFT (Appendix H.1, Figure 14).

DomainHardSFT final entropyP³D final entropy
Math0.3750.644
Medical0.9991.488

P³D learns the demonstrations while retaining greater predictive uncertainty. The probability and NLL panels show demonstrated-token learning alongside the retained entropy, while Figure 5(a) records the SFT learning dynamics. Together with the subsequent GRPO results, these observations support P³D's role as an effective RL initialization.

AuthorsOpenReview ↗

Response to Reviewer xKxJ (Part 2/3): Qwen3.5 Results

W2 & Q1: Transfer to Qwen3.5

Takeaway. We see the Qwen3.5 results as a test of whether the supervision rule transfers beyond the original backbones. Its pass@5 advantage holds across both sizes, both domains and both training stages with the same margin.

Following your suggestion, we expanded the evaluation to Qwen3.5-4B-Base and Qwen3.5-9B-Base on Math and Medical, comparing HardSFT, LS, EAFT, and P³D after both SFT and GRPO. P³D uses the same δ=0.30 throughout. The tables show greedy accuracy / pass@5 (%) for each domain benchmark and the domain average; Gen. Avg. averages seven general benchmarks. Within each model and domain, all methods use the same GRPO recipe and budget (Appendix K, Table 24).

Qwen3.5-4B, Math.

MethodStageMATH500GSM8KAIME24AIME25MinervaDomain Avg.Gen. Avg.
HardSFTSFT82.8 / 93.286.4 / 94.943.3 / 50.036.7 / 50.073.7 / 84.064.6 / 74.453.2
LSSFT78.8 / 86.685.4 / 95.026.7 / 36.723.3 / 36.773.6 / 76.557.6 / 66.353.3
EAFTSFT80.4 / 88.885.2 / 94.943.3 / 46.736.7 / 50.074.1 / 82.963.9 / 72.653.1
P³DSFT84.8 / 92.488.4 / 95.746.7 / 63.336.7 / 56.774.5 / 86.866.2 / 79.053.6
HardSFTGRPO73.4 / 88.685.4 / 95.136.7 / 63.346.7 / 60.077.5 / 88.863.9 / 79.253.4
LSGRPO76.8 / 91.087.0 / 95.546.7 / 66.740.0 / 53.378.9 / 89.565.9 / 79.253.3
EAFTGRPO71.6 / 88.886.9 / 95.736.7 / 53.340.0 / 63.376.7 / 88.362.4 / 77.953.5
P³DGRPO82.2 / 94.891.1 / 96.653.3 / 73.343.3 / 63.378.2 / 88.669.6 / 83.352.4

Qwen3.5-4B, Medical.

MethodStageMedQAMedMCQAPubMedQADomain Avg.Gen. Avg.
HardSFTSFT83.0 / 91.167.8 / 83.370.2 / 86.273.6 / 86.955.5
LSSFT84.6 / 92.568.5 / 82.470.6 / 87.274.6 / 87.354.8
EAFTSFT84.2 / 92.268.3 / 83.069.4 / 86.474.0 / 87.255.7
P³DSFT85.5 / 93.268.7 / 84.472.4 / 87.675.6 / 88.455.7
HardSFTGRPO85.1 / 92.669.9 / 83.173.2 / 85.676.1 / 87.155.2
LSGRPO85.5 / 92.370.0 / 85.072.6 / 85.876.0 / 87.754.4
EAFTGRPO85.1 / 92.769.8 / 83.774.6 / 85.476.5 / 87.355.3
P³DGRPO87.0 / 94.569.1 / 84.675.6 / 87.477.2 / 88.855.5

Qwen3.5-9B, Math.

MethodStageMATH500GSM8KAIME24AIME25MinervaDomain Avg.Gen. Avg.
HardSFTSFT85.6 / 93.088.5 / 95.346.7 / 50.033.3 / 56.773.9 / 81.565.6 / 75.356.3
LSSFT76.2 / 83.477.3 / 92.216.7 / 40.020.0 / 36.771.5 / 75.352.3 / 65.555.8
EAFTSFT86.4 / 93.688.6 / 95.933.3 / 56.730.0 / 60.074.0 / 81.762.4 / 77.656.2
P³DSFT86.6 / 93.889.3 / 95.543.3 / 60.033.3 / 63.374.2 / 82.965.4 / 79.156.4
HardSFTGRPO82.4 / 94.887.8 / 95.746.7 / 60.046.7 / 56.776.1 / 87.667.9 / 78.956.5
LSGRPO81.6 / 92.086.7 / 95.543.3 / 63.343.3 / 53.373.2 / 85.665.6 / 77.955.8
EAFTGRPO80.6 / 95.090.9 / 97.043.3 / 76.743.3 / 56.774.6 / 88.466.5 / 82.857.2
P³DGRPO85.0 / 96.091.0 / 97.050.0 / 76.756.7 / 66.778.1 / 90.072.2 / 85.356.1

Qwen3.5-9B, Medical.

MethodStageMedQAMedMCQAPubMedQADomain Avg.Gen. Avg.
HardSFTSFT88.9 / 92.672.2 / 84.369.2 / 88.276.8 / 88.457.0
LSSFT87.4 / 89.070.1 / 82.466.4 / 88.074.6 / 86.557.0
EAFTSFT89.4 / 93.173.1 / 83.372.8 / 88.678.4 / 88.357.3
P³DSFT89.7 / 93.273.5 / 84.673.4 / 89.478.9 / 89.157.3
HardSFTGRPO88.5 / 93.272.8 / 84.873.6 / 88.478.3 / 88.856.9
LSGRPO83.6 / 93.072.6 / 85.469.8 / 87.675.3 / 88.756.5
EAFTGRPO89.2 / 93.673.7 / 84.576.6 / 87.279.8 / 88.557.1
P³DGRPO89.6 / 93.573.7 / 86.377.2 / 89.280.2 / 89.757.3

P³D achieves the highest domain-average pass@5 after SFT in all four settings, and the highest domain-average greedy accuracy and pass@5 after GRPO. It also retains the highest post-GRPO Gen. Avg. at both Medical model sizes. These results support transfer to newer backbones with the same default margin.

AuthorsOpenReview ↗

Response to Reviewer xKxJ (Part 3/3): Margin Guidance and Figure 4

W3 & Q2: Selecting δ in practice

We apologize for not clearly explaining how we selected δ. Takeaway. Our practical argument is that margin choice should follow the balance between correcting the demonstration and retaining alternatives. We explain the short-probe criterion and use the full Llama results to show how that criterion can guide a model-specific adjustment.

We fixed δ=0.30 before the main experiments and full sensitivity sweep, using a ten-step Qwen3-4B pilot on randomly sampled training-corpus demonstrations. We inspected correction and retained alternative probability mass, without using test-benchmark performance. We then applied this default across models and domains. “The lower end of this range” describes its position in the subsequent sensitivity results, rather than selection by the full sweep. Figure 6 shows that 0.30 is not always optimal and that competitive performance extends across a broad range, including 0.30 to 0.60.

For this short pilot, we checked two quantities. “Strict correction” is the fraction of positions where the demonstration becomes uniquely top-ranked after training, restricted to positions where Base originally ranked it below the strongest competitor. “Alternative probability mass” measures the remaining non-target probability at positions where Base already ranked the demonstration uniquely first. The Base-defined groups remain fixed across margins. Equal δ increments on Qwen3-4B give the following changes (Appendix I.1, Table 17).

Change in δStrict-correction gain (pp)Alternative-mass reduction (pp)
0.00→0.156.27.2
0.15→0.304.96.3
0.30→0.451.54.7
0.45→0.600.93.8

The additional correction gains diminish markedly beyond 0.30, while alternative mass continues to decrease. This supports 0.30 as a moderate trade-off. We also tested this diagnostic on Llama to examine model-specific adjustment.

On 512 Llama3-8B Math demonstrations, increasing δ from 0.30 to 0.45 still yields a 3.59-point correction gain, whereas increasing it from 0.45 to 0.60 yields only 1.70 points while reducing alternative mass by another 3.87 points (Appendix I.1, Table 18). We then checked the full SFT and GRPO performance across all four margins. The table reports domain-average scores (%) and Gen. Avg. (Appendix I.1, Table 19).

δSFT greedySFT pass@5SFT Gen. Avg.GRPO greedyGRPO pass@5GRPO Gen. Avg.
0.1535.030.647.240.060.248.1
0.3044.157.847.743.361.547.9
0.4543.557.547.645.661.847.9
0.6044.857.447.745.061.047.1

δ=0.45 gives the best post-GRPO greedy accuracy and pass@5, while maintaining a similar Gen. Avg. to δ=0.30.

Our practical recommendation is to use δ=0.30 when no additional tuning is desired. If adaptation to a new model or domain is useful, compare a few candidates on a fixed training subset with the same initialization and short training budget. Inspect where additional correction gains diminish while alternative mass continues to shrink, and choose a suitable trade-off. This provides a low-cost way to choose a useful balance between acquisition and retention.

Q3: Interpreting the color at Base in Figure 4(a)

Thank you for pointing out this ambiguity. Takeaway. We believe the apparent issue comes from treating an extrapolated background color as a measured Base score. We explain this distinction, the intended policy comparison, and why Base was not included as a scored generation baseline.

The background colors in Figure 4 interpolate or extrapolate task scores of evaluated checkpoints. Base is the predictive-distribution reference, not a measured generation-task score. Because it lies away from the evaluated SFT points, its surrounding color depends on extrapolation and may have larger error. It therefore cannot establish whether ASFT or DFT performs below Base. The figure is intended to relate the distributional shifts of SFT methods to their measured task performance.

We tried the same instruction-based generation evaluation for Base, but it did not reliably follow the required answer format, making it difficult to separate task ability from formatting errors. We therefore did not report it as the generation-task baseline in this figure.

We have clarified the distinction between measured scores and the extrapolated background in the Figure 4 caption.

Reviewer_L6YROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how supervised fine-tuning can acquire demonstrated behavior while retaining model preferences that may support subsequent reinforcement learning. The proposed method, P3DP^3D, constructs a soft target from the current student’s predictions by increasing the demonstrated token’s probability and uniformly rescaling the remaining candidate probabilities. The authors derive a constrained KL projection and evaluate the method on three base models across mathematical and medical reasoning, including subsequent GRPO training. P3DP^3D improves greedy accuracy and downstream RL performance over HardSFT in several settings, while distributional probes indicate smaller changes in the model’s predictions.

Critical strengths
  1. The method is simple and computationally practical. Its target can be constructed from the student’s existing forward pass without an additional teacher model. The paper also distinguishes the original KL projection from the restricted projection that preserves non-target probability ratios, rather than presenting the latter as the exact solution to the former.

  2. The evaluation examines both supervised adaptation and subsequent RL. Results across three models and two domains, together with general-capability evaluations and distributional probes, provide meaningful evidence of practical utility.

Critical weaknesses
  1. The distinction from token-weighted SFT is not sufficiently established. For the ideal construction, let pp be the current prediction, p⋆=p(y⋆)p^\star=p(y^\star), and cc the constructed target probability of the demonstrated token. The logit gradient satisfies p−Q=c−p⋆1−p⋆(p−ey⋆).p-Q=\frac{c-p^\star}{1-p^\star}\left(p-e_{y^\star}\right). Thus, on the same normalized support, the update is equivalent to hard-label cross-entropy with a detached token-specific weight. Appendix B.8 discusses a mixture-loss decomposition but does not adequately address this more direct equivalence. Moreover, preserving non-target ratios in the constructed target does not guarantee that a shared-parameter update preserves those ratios in the trained model. The current evidence therefore does not clearly separate the proposed retention mechanism from adaptive attenuation of SFT updates.

  2. The comparisons do not adequately isolate tuning and implementation effects. Appendix D uses shared optimization settings and hyperparameters from the baseline papers, whereas P3DP^3D receives an extensive margin sweep. The independent validation procedure for selecting these settings is insufficiently documented, and the main results lack uncertainty estimates across training seeds. P3DP^3D also operates on a renormalized support of 200 candidates, while some baselines use different candidate-support configurations. These factors should be controlled before attributing the improvements specifically to the target construction.

Questions for authors
  1. Mechanism clarification: Do the authors agree with the weighted cross-entropy gradient identity above? With identical candidate support and numerical handling, does a direct implementation of this weighted objective reproduce P3DP^3D? If so, should the contribution be framed more explicitly as a constraint-derived weighting rule?

  2. Decisive comparison: Does P3DP^3D retain an advantage over simple weighted SFT, a smaller learning rate, or earlier stopping when candidate support and average update strength are controlled?

  3. Result reliability: How were the margin and baseline hyperparameters selected, and was an independent validation set used? Are the improvements over stronger baselines such as EAFT consistent across training seeds?

6 comments
AuthorsOpenReview ↗

Response to Reviewer L6YR (Part 1/4): Mechanism and Target Design

Thank you for recognizing the method's practical value and for the thoughtful questions about its mechanism and the reliability of the comparisons.

W1 & Q1: Gradient equivalence and the contribution of target design

Thank you for the careful derivation. Takeaway. Your weighted-NLL gradient derivation is correct. We believe our contribution is to turn “learn the demonstration, retain the prior” into a concrete supervision rule. We explain how target constraints determine useful adaptive weights, and why soft supervision can improve learning despite imperfect target matching. We will release the exact KL projection, P³D (Capped), alongside our current uniform-rescaling implementation, P³D (Uniform), with comparative experiments to guide their use.

At each active token, P³D (Uniform) follows the NLL direction, as in HardSFT. However, token-dependent weights can change the aggregate sequence- or step-level direction relative to uniformly weighted HardSFT. Figure 4 provides complementary evidence of different final predictive distributions. A direct implementation using the same detached weights reproduces Uniform's gradient when candidate support, numerical handling, and loss reduction match: over 5,632 real-logit positions, the maximum gradient discrepancy is 4.72×10⁻⁷ (§4.3; Appendix B.8).

We agree that a shared-parameter update need not preserve the target's non-target ratios. Soft-target supervision can still outperform hard-label training without exact target matching. Hinton et al. noted that students typically cannot exactly match soft targets, yet showed that transferring relationships among classes improves generalization over hard labels alone [1]. GKD likewise demonstrates useful distributional supervision despite limited student capacity [2]. In P³D, target design determines the adaptive weighting rule. Our domain gains, general-capability retention, and smaller policy shift demonstrate its practical value (Table 1; Figure 4; Q2).

P³D contributes a framework for deriving supervision from explicit target requirements. We start from “Greedy Guarantee” and “Minimal Intervention,” construct a target using the demonstrated token and the student's current distribution, and then fit it with cross-entropy. The scalar weight in P³D (Uniform) depends on both the demonstrated-token probability and its strongest competitor. Thus, the target-design information remains in the optimization rule (§§4.1 to 4.3; Appendices B.2 to B.5).

Compared with DFT's target-confidence weighting and EAFT's entropy gating, P³D derives its rule from the demonstrated preference and minimal intervention. Entropy alone can be a misleading learning signal [3][4]. Our empirical comparisons examine how the resulting rules balance acquisition and retention; Appendix F provides the detailed mechanism comparison.

We therefore agree that P³D (Uniform) translates our distributional target constraints into an adaptive weighting rule that balances acquisition and retention.

Preserving non-target ratios is what yields the weighted-NLL form. For cross-entropy with detached targets on the same normalized support, this constraint is necessary for gradient equivalence to weighted NLL; Appendix B motivates this choice. The original P³D (Capped) projection can change these ratios and produce corrections beyond the NLL direction, but requires bisection to construct the target. Both constructions follow the same target-design principles, with different trade-offs. We will release both implementations and have added experiments to guide their use. Below, we compare them on Qwen3-4B at δ=0.30 with matched top-200 support and leading gaps. Scores are domain-average greedy / pass@5 (%); Gen. Avg. covers seven benchmarks (Appendix B.9, Table 2).

DomainP³D constructionSFT: greedy / pass@5SFT Gen. Avg.GRPO: greedy / pass@5GRPO Gen. Avg.
MathUniform54.0 / 69.954.562.7 / 77.453.1
MathCapped53.6 / 70.454.762.0 / 80.253.5
MedicalUniform54.7 / 86.052.970.0 / 87.453.4
MedicalCapped52.5 / 86.753.369.3 / 88.153.1

P³D (Uniform) retains competitive performance with much lower target-construction cost: 0.257 ms versus 6.248 ms for Capped on 8,192 Math logit positions (medians; Appendix B.9, Table 3). The two constructions are highly similar and both balance retention and acquisition; Appendices B.6 and B.9 provide the detailed comparison. For practical use, consider Capped for sampled coverage and limited policy shift, or Uniform for faster target construction and higher greedy accuracy at δ=0.30; Appendix I.1 guides δ for both. We have revised the contribution statements to make the framework and its two realizations explicit.

AuthorsOpenReview ↗

Response to Reviewer L6YR (Part 2/4): Update Controls

W2 & Q2: Controlling candidate support and update strength

We ran the controlled comparisons you requested. Takeaway. Our point is that P³D adapts each active token's correction to the model's evolving needs. Your proposed controls let us distinguish this allocation from support choice or reduced learning: we use shared top-200 support, control training amount and update strength, and extend norm matching to EAFT, GEM and LS. The performance and policy-shift results show the value of this finer allocation.

P³D retains its advantage under the support and update-strength controls below. The distinction is that a mask selects which tokens to update, while P³D also decides how much each active token learns from the demonstration and its competitors. We recompute these weights throughout SFT, reducing or ceasing local updates as the required preference is met. For the dynamic-mask baseline, we use the current model's predictions at each step to mask out the tokens that P³D's rule would skip, then apply standard NLL to the remaining tokens. We also test lower learning rates, approximate phase-average gradient-norm matching, and early stopping.

We ran these comparisons on Qwen3-4B with the same normalized top-200 support and reduction over valid response tokens. P³D uses δ=0.30. Each benchmark cell gives greedy accuracy / pass@5 (%) after the shared GRPO stage; Domain Avg. retains the paper-reported mean, and Gen. Avg. is the single mean over the seven general benchmarks (Appendix F.7, Table 12 for the domain averages).

Math, after GRPO. EAFT, GEM, and LS norm controls keep each method's default coefficient (Appendix F.7, Table 13).

SFT initializationMATH500GSM8KAIME24AIME25MinervaDomain Avg.Gen. Avg.
NLL (top-200)78.4 / 87.092.2 / 96.420.0 / 30.016.7 / 33.374.1 / 82.856.3 / 65.953.2
NLL + mask (p*<0.8)77.8 / 87.691.7 / 95.723.3 / 33.326.7 / 43.367.4 / 78.857.4 / 67.852.7
NLL + activation mask80.0 / 88.291.7 / 96.520.0 / 40.026.7 / 43.368.3 / 79.257.3 / 69.552.8
NLL (LR 5×10⁻⁶)76.2 / 86.285.8 / 73.613.3 / 26.713.3 / 43.374.3 / 83.252.6 / 62.653.1
NLL (LR 2.5×10⁻⁶)76.2 / 86.887.1 / 94.510.0 / 33.310.0 / 33.374.1 / 82.151.5 / 66.052.7
NLL + norm matching77.8 / 87.491.9 / 96.420.0 / 33.323.3 / 43.374.3 / 83.357.5 / 68.753.7
NLL (1 SFT epoch)78.0 / 86.693.7 / 95.513.3 / 36.713.3 / 43.374.9 / 83.254.7 / 69.052.7
EAFT (α=1)76.8 / 85.492.3 / 96.516.7 / 30.020.0 / 33.374.3 / 83.456.0 / 65.753.4
EAFT + norm matching79.2 / 85.490.9 / 95.220.0 / 36.720.0 / 43.374.2 / 83.656.8 / 68.853.4
GEM (β=0.7)74.4 / 84.092.5 / 95.623.3 / 33.320.0 / 36.779.4 / 87.557.9 / 67.450.0
GEM + norm matching78.0 / 86.691.1 / 94.423.3 / 33.316.7 / 43.373.2 / 82.656.5 / 68.153.1
LS (ε=0.1)76.0 / 87.690.4 / 96.413.3 / 33.316.7 / 33.369.8 / 84.053.2 / 66.953.0
LS + norm matching78.0 / 87.091.6 / 93.526.7 / 36.723.3 / 43.373.7 / 82.758.7 / 68.652.2
P³D (Uniform)82.0 / 92.294.2 / 97.730.0 / 60.030.0 / 50.077.5 / 87.362.7 / 77.453.1

Medical, after GRPO.

SFT initializationMedQAMedMCQAPubMedQADomain Avg.Gen. Avg.
NLL (top-200)73.6 / 88.861.1 / 81.468.4 / 84.467.7 / 84.952.4
NLL + mask (p*<0.8)73.1 / 90.762.1 / 80.915.0 / 50.650.1 / 74.153.1
NLL + activation mask72.8 / 91.361.5 / 81.270.2 / 85.468.2 / 85.952.7
NLL (LR 5×10⁻⁶)72.7 / 88.059.8 / 81.969.0 / 86.267.2 / 85.452.9
NLL (LR 2.5×10⁻⁶)69.4 / 88.857.2 / 80.766.0 / 85.064.2 / 84.852.3
NLL + norm matching75.3 / 91.862.6 / 81.769.8 / 84.869.2 / 86.152.4
NLL (1 SFT epoch)73.1 / 89.259.1 / 81.064.2 / 84.065.5 / 84.752.0
P³D (Uniform)74.5 / 90.362.9 / 82.972.4 / 89.070.0 / 87.453.4

We used masking, early stopping, lower learning rates, and approximate adaptive norm matching to control the amount and strength of learning. The comparisons include NLL and norm-controlled EAFT, GEM, and LS. P³D remains ahead in domain-average greedy accuracy and pass@5, showing that its gains are not explained simply by fewer or smaller updates.

The policy-shift plot supports the same conclusion: the tested learning-rate and norm controls do not reproduce P³D's smaller distribution shift together with its performance gains (Appendix F.7, Figure 9).

AuthorsOpenReview ↗

Response to Reviewer L6YR (Part 3/4): Hyperparameter Choices

W2 & Q3: Hyperparameter selection, fairness, and robustness

We apologize for not clearly explaining how we selected δ. Takeaway. Our case for reliability rests on a reusable default and consistent results under reasonable variations. We chose δ by balancing correction and retained alternatives in a short training-corpus pilot before the sweep, then reused it across models and domains. Published baseline recommendations, shared-support parameter controls and three training seeds provide complementary checks of fairness and stability.

We fixed δ=0.30 before the main experiments and full sensitivity sweep. A ten-step pilot on randomly sampled Qwen3-4B training-corpus demonstrations compared strict correction and retained alternative probability mass; test-benchmark performance was not used. We then shared the default across models and domains. “The lower end of this range” refers to its position in the subsequent sensitivity results, rather than selection by the full sweep. Figure 6 shows that 0.30 is not always optimal and that performance remains competitive over a broad range, including 0.30 to 0.60.

The pilot examines correction at positions where Base initially ranks the demonstration below its strongest competitor, and retained alternative mass where Base already ranks the demonstration uniquely first. For δ increases of 0.15→0.30 and 0.30→0.45, correction gains decrease from 4.9 to 1.5 percentage points while alternative mass still falls by 6.3 and 4.7 points. This supports a moderate default; Appendix I.1, Table 17 provides the definitions and complete results.

To address your concern about candidate support, the NLL controls in Q2 match candidate selection, normalization, and token reduction. The EAFT, GEM, and LS parameter and norm comparisons also use the same normalized, target-inclusive top-200 support as P³D (Appendix F.7).

For the baseline settings, we followed the recommendations in the original papers: GEM uses β=0.7 in its main experiments and parameter analysis [5], while EAFT recommends linear entropy gating [6]. We also tested alternative parameters and norm controls. The tables show every configuration after SFT and GRPO on Qwen3-4B Math, using the same top-200 support. Norm variants keep the default coefficient. Cells report greedy / pass@5 (%), with domain and general averages (Appendix F.7, Table 13).

Qwen3-4B Math, after SFT.

Method / settingMATH500GSM8KAIME24AIME25MinervaDomain Avg.Gen. Avg.
EAFT (α=1)73.0 / 88.489.6 / 97.923.3 / 46.723.3 / 40.057.6 / 78.453.4 / 70.352.9
EAFT + norm72.8 / 87.285.3 / 95.916.7 / 40.023.3 / 36.758.2 / 78.351.3 / 67.654.5
EAFT (α=2)72.6 / 87.284.1 / 95.516.7 / 33.323.3 / 30.056.8 / 78.850.7 / 65.054.4
GEM (β=0.7)69.8 / 82.482.2 / 96.220.0 / 40.013.3 / 30.060.4 / 76.349.2 / 65.049.5
GEM + norm75.0 / 87.086.1 / 96.420.0 / 36.726.7 / 33.359.1 / 79.753.4 / 66.654.5
GEM (β=0.35)73.0 / 31.683.3 / 35.316.7 / 30.023.3 / 13.358.1 / 30.650.9 / 28.252.6
LS (ε=0.1)71.4 / 86.489.4 / 97.023.3 / 46.723.3 / 33.355.8 / 69.952.6 / 66.753.8
LS + norm70.2 / 87.078.9 / 94.613.3 / 40.030.0 / 43.357.8 / 78.550.0 / 68.753.7
LS (ε=0.2)73.2 / 87.085.8 / 94.720.0 / 33.326.7 / 23.358.8 / 78.352.9 / 63.353.6
P³D (δ=0.30)73.8 / 88.290.5 / 97.220.0 / 46.726.7 / 43.359.0 / 74.354.0 / 69.954.5

Comments on the paper

2 comments
AuthorsOpenReview ↗

Humble Response to All Reviewers

We sincerely thank all reviewers for their thoughtful and constructive comments. Your feedback has helped us clarify our contribution and strengthen the experimental evidence for P³D.

In the revised manuscript, we clarify the relationship between target-distribution design and weighted NLL, and provide comparisons and practical guidance for the Capped and Uniform constructions of P³D. We have added masking, learning-rate, gradient-norm, and baseline-parameter controls, comparisons across three training seeds, and Qwen3.5-4B/9B experiments in both Math and Medical. We also expand the SFT dynamics and subsequent GRPO analyses, clarify the training-corpus diagnostic used to choose δ before the main experiments, and supplement the margin analysis with Llama experiments and practical selection guidance. Both P³D implementations have been released.

These additional analyses and results are concentrated in Appendices B, D, F, H, I, and K, with corresponding clarifications in the main text. For ease of reference, newly added content is highlighted in blue, while revisions to existing text are highlighted in red. Our individual responses explain how these changes address each reviewer’s questions and point to the relevant sections, figures, and tables.

We hope these revisions address your concerns, and we sincerely appreciate the time and care you have devoted to reviewing our work.

AuthorsOpenReview ↗

General Rebuttal Summary

We sincerely thank all reviewers for their thoughtful assessments and constructive comments, which helped us clarify P³D's contribution and strengthen the manuscript.

We are grateful that Reviewer L6YR described our method as “simple and computationally practical,” highlighting target construction from the student's existing forward pass without an additional teacher model and the distinction between the original KL projection and the restriction preserving non-target ratios; they also recognized “meaningful evidence of practical utility” from our evaluation across three models and two domains, covering SFT, subsequent RL, general capabilities, and distributional probes. We also greatly appreciate Reviewer xKxJ's explicit support for acceptance following the rebuttal.

In response to the reviewers' questions, we made the following additions and clarifications:

  1. Target design and weighted-NLL equivalence (ujuQ, L6YR; §4.3, Appendix B.8). We acknowledged and numerically verified the gradient identity and clarified P³D's contribution as deriving adaptive supervision from distributional target constraints.
  2. Two constructions of P³D (ujuQ, L6YR; Appendix B.9). We added performance, construction-cost, and practical-guidance comparisons for P³D (Capped) and P³D (Uniform), with both implementations included in our code release.
  3. Masking and update-strength controls (ujuQ, L6YR; Appendix F.7). Under shared normalized top-200 support, P³D retains its domain-average advantage over masking, learning-rate, norm-matching, and early-stopping controls, with policy-shift comparisons providing complementary evidence.
  4. Baseline settings and training-seed robustness (ujuQ, L6YR; Appendices D.5 and F.7). We documented baseline recommendations, tested alternative parameters and norm controls, and added three independent training seeds, with P³D retaining the highest mean post-GRPO greedy accuracy and pass@5 in both domains among P³D, HardSFT, EAFT, and GEM.
  5. Margin selection and practical guidance (All three reviewers; Appendix I.1). We clarified the training-corpus selection of δ=0.30 before the full sweep and added Llama diagnostics and performance comparisons to guide practical choices.
  6. Transfer to Qwen3.5 (xKxJ; Appendix K). The new Qwen3.5-4B/9B comparisons use δ=0.30 throughout, with P³D leading domain-average pass@5 across both sizes, both domains, and both SFT and GRPO stages among the compared methods.
  7. SFT and subsequent GRPO dynamics (xKxJ, ujuQ; Appendix H). We added SFT entropy, demonstrated-token probability, and NLL curves across methods, together with diagnostics of the model-specific HardSFT changes after GRPO.
  8. Figure 4 interpretation (xKxJ; Figure 4 and §5.3). We made the Base region neutral, labeled it “reference only,” and added a footnote explaining the omitted Base task score and the answer-format issue under the shared evaluation protocol.

These additions strengthen the theoretical and empirical basis of P³D and provide direct tests of the alternative explanations raised in the reviews; new content is highlighted in blue and revisions to existing text in red.

Regarding the discussion stage, Reviewer xKxJ confirmed that their concerns had been addressed and explicitly stated, “I think this paper should be accepted.” This assessment followed our responses on SFT dynamics, Qwen3.5 transfer, and practical margin guidance, with the margin-selection clarification also relevant to the questions raised by ujuQ and L6YR.

We fully appreciate the demands on reviewers' time. In response to the concerns raised by ujuQ and L6YR, we completed substantial additional experiments and provided detailed theoretical analyses and benchmark results specifically addressing their questions. These additions directly examine gradient equivalence, implementation choices, and whether the gains reflect simply fewer or smaller updates. At the time of this summary, we have not yet had the opportunity to discuss these new results with them. We would appreciate the AC's assessment of which initial concerns remain in light of the revised manuscript and this additional evidence, alongside xKxJ's explicit support for acceptance.

We thank the reviewers and the AC, SAC, and PC teams for their time and valuable feedback, and remain available for further questions or discussion.

Warmest regards,

The Authors of Submisssion #3315

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.