- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
P3D is an SFT objective for the SFT-then-RL pipeline. At each token it builds a soft target from the model’s own detached prediction. The demonstrated token is raised to a margin above the point where it would tie its strongest competitor, and all other tokens are rescaled by one common factor so their ratios are kept. The model is fit to this target with cross-entropy. Across Qwen3-4B/8B and Llama3-8B on math and medical reasoning, P3D beats standard SFT in greedy accuracy, both after SFT and after GRPO.
Critical strengths
The method is simple and cheap with a closed-form target with no reference model or extra forward pass. Its greedy gains over HardSFT hold across three models and two domains
Critical weaknesses
-
Unless I am missing something, the soft target reduces to a reweighted NLL. is built from the model’s own detached prediction at the current parameters, so the logit gradient of the P3D loss is just times the NLL gradient, with . The prior term in Eq. 26 adds nothing new either. Its gradient points opposite to the NLL gradient and simply makes the step smaller. So the ratio preservation in Eq. 6 holds for the target but not for the update, and the update only uses the demonstrated token plus one scalar per position. I don’t see where the dense feedback comes from. This makes P3D a confidence-based weighting scheme, the same family the paper criticizes (DFT, EAFT), and it weakens contribution 1 (target design instead of loss design).
-
I am not convinced by the explanation of the gains. Only the scalar ends up in the gradient, so the derivation in Appendix B is effectively choosing a weight. In fact, uniform rescaling is exactly what turns the target into a weighted one-hot. The exact capped solution would not reduce this way. The resulting weight behaves like a hinge. With delta = 0.3, every token with gets zero weight, and by Table 3 that is more than half of the tokens at the start of training. A simpler story then fits the retention and RL results just as well: P3D trains less on tokens the model already predicts. The paper needs an argument or a control that rules this out, such as NLL masked at the same threshold, or HardSFT with a smaller learning rate.
Questions for authors
-
Do you agree that the per-token logit gradient is times the NLL gradient? If so, what does P3D add over NLL with this weight?
-
How does P3D compare with NLL masked at the same threshold, and with HardSFT at a lower learning rate?
-
Smaller points: was delta chosen on held-out data, how many seeds underlie Table 1, and why does the shared GRPO recipe lower HardSFT on Qwen3-8B and Llama3-8B Math?
6 comments
Response to Reviewer ujuQ (Part 2/4): Target Design and Update Controls
To address your concern about “dense” feedback, we distinguish two aspects. First, the design motivation is distributional: the supervisory target explicitly considers non-target candidate structure alongside the demonstrated preference. The added P³D (Capped) experiments test this motivation beyond the P³D (Uniform) implementation.
Second, both constructions carry dense distributional information into the learning signal. Capped provides candidate-level supervision by fitting the projected distribution. Uniform preserves the relative probabilities of non-target candidates and translates the demonstrated-token probability and its strongest competitor into an adaptive learning weight. Although Uniform reduces to weighted NLL, its weight carries information about competing alternatives beyond the demonstrated label. The distributional design therefore remains effective through a simple, efficient optimization rule.
W2 & Q2: Do masking or smaller updates explain the gains?
Thank you for suggesting these comparisons. Takeaway. We believe the key is how learning is allocated, including among tokens that remain active. P³D adjusts that allocation as the model changes; we use the masking, learning-rate, stopping and norm controls to test this explanation, then connect the performance differences to policy shift.
We tested whether P³D's gains could be explained by masking tokens, lowering the learning rate, matching gradient norms, or stopping earlier. The key distinction is how much each active token learns. Related RL work also finds that a minority of uncertain tokens can contribute disproportionately to learning [5]. This provides a rationale for why reducing updates where the demonstrated token already has high probability can be beneficial. P³D adjusts each active token's weight using its demonstrated-token probability and strongest competitor; a binary mask applies standard NLL to the tokens it keeps. Matching the active set therefore does not reproduce P³D's allocation of learning within it.
Throughout SFT, P³D recomputes each token’s weight from the model’s current predictions, reducing or stopping updates when the required preference is met while continuing to correct unresolved tokens. A smaller global learning rate scales all updates, and early stopping ends them together. For the dynamic-mask baseline, we use the current model's predictions at each step to mask out the tokens that P³D's rule would skip, then apply standard NLL to the remaining tokens.
We ran these controls on Qwen3-4B with the same normalized top-200 support and GRPO recipe, using δ=0.30 for P³D. Norm control approximately matches the mean gradient norm across training phases. Each benchmark cell gives post-GRPO greedy accuracy / pass@5 (%); Domain Avg. follows the paper, and Gen. Avg. averages seven general benchmarks (Appendix F.7, Table 12).
Math, after GRPO. EAFT, GEM, and LS norm controls keep each method's default coefficient (Appendix F.7, Table 13).
| SFT initialization | MATH500 | GSM8K | AIME24 | AIME25 | Minerva | Domain Avg. | Gen. Avg. |
|---|---|---|---|---|---|---|---|
| NLL (top-200) | 78.4 / 87.0 | 92.2 / 96.4 | 20.0 / 30.0 | 16.7 / 33.3 | 74.1 / 82.8 | 56.3 / 65.9 | 53.2 |
| NLL + mask (p*<0.8) | 77.8 / 87.6 | 91.7 / 95.7 | 23.3 / 33.3 | 26.7 / 43.3 | 67.4 / 78.8 | 57.4 / 67.8 | 52.7 |
| NLL + activation mask | 80.0 / 88.2 | 91.7 / 96.5 | 20.0 / 40.0 | 26.7 / 43.3 | 68.3 / 79.2 | 57.3 / 69.5 | 52.8 |
| NLL (LR 5×10⁻⁶) | 76.2 / 86.2 | 85.8 / 73.6 | 13.3 / 26.7 | 13.3 / 43.3 | 74.3 / 83.2 | 52.6 / 62.6 | 53.1 |
| NLL (LR 2.5×10⁻⁶) | 76.2 / 86.8 | 87.1 / 94.5 | 10.0 / 33.3 | 10.0 / 33.3 | 74.1 / 82.1 | 51.5 / 66.0 | 52.7 |
| NLL + norm matching | 77.8 / 87.4 | 91.9 / 96.4 | 20.0 / 33.3 | 23.3 / 43.3 | 74.3 / 83.3 | 57.5 / 68.7 | 53.7 |
| NLL (1 SFT epoch) | 78.0 / 86.6 | 93.7 / 95.5 | 13.3 / 36.7 | 13.3 / 43.3 | 74.9 / 83.2 | 54.7 / 69.0 | 52.7 |
| EAFT (α=1) | 76.8 / 85.4 | 92.3 / 96.5 | 16.7 / 30.0 | 20.0 / 33.3 | 74.3 / 83.4 | 56.0 / 65.7 | 53.4 |
| EAFT + norm matching | 79.2 / 85.4 | 90.9 / 95.2 | 20.0 / 36.7 | 20.0 / 43.3 | 74.2 / 83.6 | 56.8 / 68.8 | 53.4 |
| GEM (β=0.7) | 74.4 / 84.0 | 92.5 / 95.6 | 23.3 / 33.3 | 20.0 / 36.7 | 79.4 / 87.5 | 57.9 / 67.4 | 50.0 |
| GEM + norm matching | 78.0 / 86.6 | 91.1 / 94.4 | 23.3 / 33.3 | 16.7 / 43.3 | 73.2 / 82.6 | 56.5 / 68.1 | 53.1 |
| LS (ε=0.1) | 76.0 / 87.6 | 90.4 / 96.4 | 13.3 / 33.3 | 16.7 / 33.3 | 69.8 / 84.0 | 53.2 / 66.9 | 53.0 |
| LS + norm matching | 78.0 / 87.0 | 91.6 / 93.5 | 26.7 / 36.7 | 23.3 / 43.3 | 73.7 / 82.7 | 58.7 / 68.6 | 52.2 |
| P³D (Uniform) | 82.0 / 92.2 | 94.2 / 97.7 | 30.0 / 60.0 | 30.0 / 50.0 | 77.5 / 87.3 | 62.7 / 77.4 | 53.1 |
Response to Reviewer ujuQ (Part 3/4): Medical Controls and Policy Shift
Medical, after GRPO.
| SFT initialization | MedQA | MedMCQA | PubMedQA | Domain Avg. | Gen. Avg. |
|---|---|---|---|---|---|
| NLL (top-200) | 73.6 / 88.8 | 61.1 / 81.4 | 68.4 / 84.4 | 67.7 / 84.9 | 52.4 |
| NLL + mask (p*<0.8) | 73.1 / 90.7 | 62.1 / 80.9 | 15.0 / 50.6 | 50.1 / 74.1 | 53.1 |
| NLL + activation mask | 72.8 / 91.3 | 61.5 / 81.2 | 70.2 / 85.4 | 68.2 / 85.9 | 52.7 |
| NLL (LR 5×10⁻⁶) | 72.7 / 88.0 | 59.8 / 81.9 | 69.0 / 86.2 | 67.2 / 85.4 | 52.9 |
| NLL (LR 2.5×10⁻⁶) | 69.4 / 88.8 | 57.2 / 80.7 | 66.0 / 85.0 | 64.2 / 84.8 | 52.3 |
| NLL + norm matching | 75.3 / 91.8 | 62.6 / 81.7 | 69.8 / 84.8 | 69.2 / 86.1 | 52.4 |
| NLL (1 SFT epoch) | 73.1 / 89.2 | 59.1 / 81.0 | 64.2 / 84.0 | 65.5 / 84.7 | 52.0 |
| P³D (Uniform) | 74.5 / 90.3 | 62.9 / 82.9 | 72.4 / 89.0 | 70.0 / 87.4 | 53.4 |
We used masking, early stopping, lower learning rates, and approximate adaptive norm matching to control the amount and strength of learning. The comparisons include NLL and norm-controlled EAFT, GEM, and LS. P³D remains ahead in domain-average greedy accuracy and pass@5, showing that its gains are not explained simply by fewer or smaller updates.
The policy-shift plot supports the same conclusion: the tested learning-rate and norm controls do not reproduce P³D's smaller distribution shift together with its performance gains (Appendix F.7, Figure 9).
Response to Reviewer ujuQ (Part 1/4): Target Design
W1 & Q1: Weighted NLL equivalence, target design, and dense supervision
Thank you for the careful derivation and insightful comments. Takeaway. We agree with the gradient identity. We believe our contribution is to turn “learn the demonstration, retain the prior” into a concrete supervision rule. We connect this target design to adaptive learning and the practical value of soft supervision, clarifying both the weighting rule and the meaning of dense feedback. We will release the exact KL projection, P³D (Capped), alongside our current uniform-rescaling implementation, P³D (Uniform), with comparative experiments to guide their use.
At each active token, P³D (Uniform)'s gradient follows the NLL direction, as in HardSFT. However, at the sequence or step level, the relative weights assigned to different tokens can change the aggregate direction compared with uniformly weighted HardSFT. Figure 4 provides complementary evidence that the methods produce different final predictive distributions and learning outcomes.
Our contribution is the target-design framework and the supervision rule it induces. For P³D (Uniform), NLL with these same detached weights is an equivalent implementation of that design. P³D starts from two requirements, “Greedy Guarantee” and “Minimal Intervention,” and constructs a supervisory distribution from the student's current predictions and the demonstrated token. We first solve for a target under these requirements, then fit it using cross-entropy. Uniform's resulting weight depends on both the demonstrated-token probability and its strongest competitor, carrying the target-design information into optimization (§§4.1 to 4.3; Appendices B.2 to B.5).
This is also where P³D differs from DFT's target-confidence weighting and EAFT's entropy gating. Entropy alone can be a misleading learning signal [1][2]. P³D explicitly specifies the desired relationship between acquisition and retention in the supervisory target. Table 1, Figure 4, and the SFT dynamics provide empirical evidence for the resulting behavior; Appendix F details the mechanism comparisons.
We agree that a shared-parameter update need not preserve the target's non-target ratios. Soft-target supervision can still outperform hard-label training without exact target matching. Hinton et al. noted that students typically cannot exactly match soft targets, yet showed that transferring relationships among classes improves generalization over hard labels alone [3]. GKD likewise demonstrates useful distributional supervision despite limited student capacity [4]. In P³D, target design determines the adaptive weighting rule. Our domain gains, general-capability retention, and smaller policy shift demonstrate its practical value (Table 1; Figure 4; Q2).
Preserving non-target ratios is what yields the weighted-NLL form. For cross-entropy with detached targets on the same normalized support, this constraint is necessary for gradient equivalence to weighted NLL; Appendix B motivates this choice. The original P³D (Capped) projection can change these ratios and produce corrections beyond the NLL direction, but requires bisection to construct the target. Both constructions follow the same target-design principles, with different trade-offs. We will release both implementations and have added experiments to guide their use. Below, we compare them on Qwen3-4B at δ=0.30 with matched top-200 support and leading gaps. Scores are domain-average greedy / pass@5 (%); Gen. Avg. covers seven benchmarks (Appendix B.9, Table 2).
P³D (Uniform) retains competitive performance with much lower target-construction cost: 0.257 ms versus 6.248 ms for Capped on 8,192 Math logit positions (medians; Appendix B.9, Table 3). The two constructions are highly similar and both balance retention and acquisition; Appendices B.6 and B.9 provide the detailed comparison. For practical use, consider Capped for sampled coverage and limited policy shift, or Uniform for faster target construction and higher greedy accuracy at δ=0.30; Appendix I.1 guides δ for both.