acceptodds
Under review as a conference paper at ICLR 2027

Self-Privileged Fine-Tuning

Abstract

Supervised fine-tuning (SFT) is a key component of language model post-training, enabling effective task adaptation and serving as an important foundation for subsequent reinforcement learning. Recent methods seek to improve SFT by introducing token-level weights into the training objective, where these weights are typically derived from a model’s output distribution at each token position. To better understand how such weighting schemes affect SFT, we analyze their optimization objectives and theoretically characterize the target distributions they induce. Our analysis shows that dynamic weighting methods, such as Dynamic Fine-Tuning (DFT), induce a heavily sharpened distribution, whereas static weighting methods fit a target distribution proportional to the product of the data distribution and a frozen prior. These findings suggest that static weighting provides a more stable training target, while its effectiveness critically depends on the quality of the prior distribution. Building on this observation, we propose Self-Privileged Fine-Tuning (SPFT), which uses a frozen copy of the initial model conditioned on privileged information (such as ground truth for mathematical questions) as a prior to provide guidance for token-level weighting during fine-tuning. Through a Bayesian decomposition, we further show that SPFT reweights the standard static prior according to how informative each target token is for predicting the ground-truth outcome, thereby assigning larger weights to more predictive tokens. Experiments on mathematical reasoning and code generation show that SPFT improves over standard SFT and other baselines across the main settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Supervised Fine-Tuning, LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zPC9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Privileged Fine-Tuning (SPFT), which uses a frozen copy of the initial model conditioned on privileged information as a prior to provide guidance for token-level weighting during fine-tuning.

Critical strengths

S1: This paper is well written and easy to follow.

S2: This paper presents comprehensive experiments to evaluate the proposed method.

Critical weaknesses

W1 (marginal improvement): The empirical results seem to have at most very marginal improvement over baselines. Considering the large variance of the avg@8 and pass@8 metrics, it is unclear whether this method is truly outperforming existing methods.

W2 (unrealistic assumption): It seems unrealistic to assume that language data has a unique maximizer in Proposition 1. The proof of Proposition 1 seems to crucially rely on this unrealistic assumption, so whether Proposition 1 holds in practice is questionable. It is not hard to prove a similar result without this assumption, so it is a bit unclear why this paper decides to use this assumption.

W3 (inconsistent performance): The results show no clear ranking between SPFT and SPFT-SR. Consequently, it is a bit unclear how to decide which variant to use in practice.

Questions for authors

See weaknesses.

8 comments

Part 1/2

Thank you for the review and the kind words on the writing and experiments. The concerns led us to run a multi-seed study and to simplify Prop. 1. Full per-benchmark tables, with standard deviations for every metric, are in Part 2.

W1: Are the improvements larger than the variance? With one training run and one evaluation seed, the submission indeed made this hard to judge. We have now re-evaluated every Qwen2.5-Math model with 3 evaluation seeds, keeping all other settings unchanged. Macro Avg@8 over the five benchmarks (mean ± std):

MethodQwen2.5-Math-1.5BQwen2.5-Math-7B
SFT25.55±0.4134.56±0.67
DFT29.65±0.0135.54±0.34
ASFT27.31±0.3536.70±0.19
TALR26.36±0.3132.45±0.63
PriFT27.90±0.3239.08±0.35
SPFT32.68±0.3242.04±0.72
SPFT-SR33.92±0.6142.33±0.45

Evaluation-seed stds stay at about 0.7 or below, while SPFT-SR leads the strongest baseline (DFT on 1.5B, PriFT on 7B) by 4.27 and 3.25, and SFT by about 8 points.

Evaluation seeds do not capture training variance, so on 1.5B we also retrained SPFT-SR and SPFT with 3 training seeds. Each run is averaged over its 3 evaluation seeds:

1.5Bseed 1seed 2seed 3mean ± std
SPFT-SR33.9234.5034.3934.27±0.31
SPFT32.6832.3132.1732.39±0.26

SPFT-SR is ahead of SPFT in every run, by 1.88 on average. This gap is several times the run-to-run std. We will report variance in all tables and base our claims on margins that hold across runs.

W2: The unique-maximizer assumption in Prop. 1. This is correct: uniqueness is not needed, and we will remove it. Let M=arg⁡max⁡vpdata(v∣h)M=\arg\max_v p_{\mathrm{data}}(v\mid h) be the set of all maximizers. Assume pθ0(M∣h)>0p_{\theta_0}(M\mid h)>0 and write p∗=max⁡vpdata(v∣h)p^*=\max_v p_{\mathrm{data}}(v\mid h). Then

pθk(v∣h)=pθ0(v∣h) (pdata(v∣h)/p∗)k∑upθ0(u∣h) (pdata(u∣h)/p∗)k.p_{\theta_k}(v\mid h)=\frac{p_{\theta_0}(v\mid h)\,\big(p_{\mathrm{data}}(v\mid h)/p^*\big)^k}{\sum_u p_{\theta_0}(u\mid h)\,\big(p_{\mathrm{data}}(u\mid h)/p^*\big)^k}.

For v∉Mv\notin M the ratio pdata(v∣h)/p∗p_{\mathrm{data}}(v\mid h)/p^* is below 1, so its kk-th power vanishes; on MM it equals 1. Hence

pθk(v∣h)  ⟶  pθ0(v∣h) 1[v∈M]pθ0(M∣h).p_{\theta_k}(v\mid h)\;\longrightarrow\;\frac{p_{\theta_0}(v\mid h)\,\mathbb{1}[v\in M]}{p_{\theta_0}(M\mid h)}.

DFT therefore still collapses all mass onto the data modes, and when there are ties it keeps the initial model's relative preferences among them. The unique-mode case is just ∣M∣=1|M|=1. Only one step of the proof changes, and the conclusion of Prop. 1 stays the same. We will update the proposition and its proof.

W3: Which variant should one use? We recommend SPFT-SR and treat SPFT as an ablation. With multiple seeds, the ranking that looked mixed in the submission becomes clearer:

  • On 1.5B, adding SR improves SPFT by 1.23, and it does so in every training run (+1.23, +2.20, +2.21).
  • On 7B, SPFT-SR is slightly ahead (42.33 vs 42.04). The single-seed submission had shown the opposite order.
  • SR also improves SPFT when the privileged information is a richer hint (31.80 → 34.18, Part 2).

The reason is the design of SR. A hard problem lowers the teacher's probabilities for every token of its solution together, so raw weights partly reflect problem difficulty. Standardizing within each response removes this shared shift and keeps the token-level differences (Section 4). We will state SPFT-SR as the default in the method section.

Part 2/2: Appendix: per-benchmark results

Each cell is the mean ± std over 3 evaluation seeds (in the training-seed table, ± is across the 3 training runs). The Avg@8 and Pass@8 columns are macro averages over the five benchmarks.

Qwen2.5-Math-1.5B

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
SFT56.41±0.4715.50±0.3121.80±0.513.33±1.9130.73±2.0825.55±0.4149.90±1.41
DFT64.46±1.3421.52±0.8224.58±0.405.28±0.6432.40±1.5729.65±0.0146.72±0.55
ASFT58.20±0.9218.72±0.8823.43±0.394.44±1.2731.77±0.9527.31±0.3549.46±1.06
TALR57.19±0.8816.82±1.0522.17±0.383.75±2.0831.87±0.3226.36±0.3149.21±0.93
PriFT57.48±0.2812.07±1.0527.41±0.675.28±1.0537.29±1.4427.90±0.3253.38±1.40
SPFT64.59±0.1420.28±0.5828.93±0.506.39±0.6443.23±1.1032.68±0.3254.76±1.34
SPFT-SR67.70±0.3820.82±0.2330.22±0.606.25±3.4144.58±1.0033.92±0.6155.33±1.06

Qwen2.5-Math-7B

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
SFT68.49±0.8124.97±0.5331.72±0.666.25±1.8141.35±0.7934.56±0.6758.48±1.19
DFT68.15±0.1625.37±0.3833.11±0.166.25±0.4244.79±1.7235.54±0.3450.04±0.51
ASFT70.59±0.2922.32±0.3833.99±0.487.64±1.3448.96±1.9936.70±0.1955.95±0.31
TALR65.93±0.4022.59±0.1829.60±0.446.11±0.8638.02±2.0832.45±0.6355.34±2.01
PriFT71.75±0.3219.95±0.4432.49±0.8516.53±2.1054.69±0.6339.08±0.3560.50±0.62
SPFT74.11±0.2525.71±0.7937.20±0.2614.31±1.4658.86±1.1942.04±0.7262.95±1.62
SPFT-SR72.09±0.0829.47±0.8436.55±0.3614.17±2.3259.38±1.5742.33±0.4562.14±0.84

Qwen2.5-Math-1.5B, 3 training seeds

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
SPFT64.64±0.2320.44±0.1728.84±0.146.30±0.1641.73±1.3232.39±0.2654.68±0.13
SPFT-SR68.34±0.6321.20±0.4130.47±0.275.88±0.6445.45±0.7734.27±0.3154.92±0.36

SPFT vs SPFT-SR with a hint as privileged information (Qwen2.5-Math-1.5B)

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
SPFT (hint)64.34±0.5120.50±0.8028.35±0.366.25±1.1039.58±1.7231.80±0.7754.16±2.83
SPFT-SR (hint)68.27±0.2720.97±0.1030.22±0.536.53±1.3444.90±2.9634.18±0.3755.40±0.58

Thanks for the response. W2 and W3 have been addressed.

Regarding W1, I am a bit confused by the provided tables. In the first table (3 eval seeds), Qwen2.5-Math-1.5B has mean Avg@8 = 32.68 and 33.92 for SPFT and SPFT-SR, respectively. However, the two numbers are exactly the same as the seed 1 results in the second table, which seems to be different from the description of the first table (i.e., the 3-seed results coincide with the 1-seed results). Could you please clarify the meanings of the two tables? Thanks!

Reviewer_RHPHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Privileged Fine-Tuning (SPFT), which leverages answers or structured prompts available during training to enable a frozen initial model to generate static training weights for tokens within reference answers. This approach retains the training paradigm of offline supervised fine-tuning, requires no online rollouts or additional external teachers, and incurs no extra cost during inference.

Critical strengths
  1. SPFT preserves the offline SFT pipeline and uses a frozen initial model to precompute weights. It requires neither online rollouts nor an external teacher and introduces no additional inference cost, making it relatively easy to adopt.

  2. The paper examines both average correctness and success under multiple samples. This provides a broader assessment of reasoning performance and helps investigate whether training preserves useful sampling diversity.

Critical weaknesses
  1. The experiments do not sufficiently disentangle the contribution of privileged information from the token selection and normalization introduced by SPFT-SR. Applying the same weight transformation without privileged context would help establish the added value of the central proposal.

  2. Higher token probabilities under privileged context do not necessarily indicate greater reasoning importance. The theoretical analysis offers useful intuition under idealized assumptions, but does not establish that the resulting weights reliably capture token-level contributions to task success.

Questions for authors

Please refer to weaknesses.

3 comments

Part 1/2

Thank you for the careful review and for noting that SPFT keeps the offline SFT pipeline simple. The two questions led us to a new ablation and a clearer reading of Prop. 3. Full per-benchmark tables are in Part 2. All numbers are macro Avg@8, as mean ± std over 3 evaluation seeds.

W1: Is the gain due to privileged information or to the SR transformation? This is a fair point. Compared with PriFT, SPFT-SR differs in two ways: it adds privileged information, and it applies SR, which standardizes the weights within each response, sets negative scores to zero, and rescales them to mean 1. Either difference could explain the gain. To separate them, we trained a new baseline, PriFT-SR. It starts from PriFT, which weights each token by the frozen initial model's own probability without any privileged information, and applies exactly the same SR transformation. The four methods then form a 2×2 design: with or without privileged information, and with or without SR. Because of a labeling error in the submission, the 8B model in the submission was in fact the post-trained Qwen3-8B rather than Qwen3-8B-Base, so we re-ran all four methods from Qwen3-8B-Base. For Qwen3-1.7B, which is a small model and not special for math, so the privileged information wo use is a short hint, a two-to-three-sentence summary of the reference solution followed by the answer.

BackbonePriFTSPFTPriFT-SR (new)SPFT-SR
Qwen2.5-Math-1.5B27.90±0.3232.68±0.3232.46±0.4933.92±0.61
Qwen2.5-Math-7B39.08±0.3542.04±0.7240.49±0.3042.33±0.45
Qwen3-8B-Base39.50±0.5240.44±0.5242.10±0.6943.96±0.49
Qwen3-1.7B-Base26.30±0.5226.97±0.6129.00±0.2030.41±0.09

The table shows three things:

  • Privileged information helps beyond SR. With SR applied on both sides, SPFT-SR leads PriFT-SR on all four backbones: by 1.46 (1.5B), 1.84 (7B), 1.85 (Qwen3-8B) and 1.40 (Qwen3-1.7B).
  • It also helps without SR. SPFT is ahead of PriFT on all four backbones.
  • SR improves both weightings, but it does not substitute for privileged information. PriFT-SR stays below SPFT-SR on every backbone.

W2: Does higher probability under privileged context mean reasoning importance? We agree that a raw probability is not, by itself, a measure of importance. Prop. 3, however, shows that the privileged weight has a more specific meaning: it is the prior multiplied by a process-advantage term. Define the value of a prefix as the frozen model's probability of eventually reaching the correct answer cc:

V(y<t):=pθ0(c∣x,y<t).V(y_{<t}) := p_{\theta_0}(c\mid x, y_{<t}).

By Bayes' rule, Prop. 3 then reads

q(yt∣x,c,y<t)=pθ0(yt∣x,y<t) exp⁡(At),At=log⁡V(y≤t)−log⁡V(y<t).q(y_t\mid x,c,y_{<t}) = p_{\theta_0}(y_t\mid x,y_{<t})\,\exp(A_t),\qquad A_t = \log V(y_{\le t}) - \log V(y_{<t}).

Appending a token is a deterministic step, so V(y≤t)V(y_{\le t}) is the action value Q(y<t,yt)Q(y_{<t},y_t) and At=log⁡(Q/V)A_t=\log(Q/V). In words, AtA_t measures how much emitting yty_t changes the chance of reaching the correct answer. This is the notion of step importance behind progress-based process rewards. Two exact properties support reading it as credit:

  1. Unit-mean ratio. By total probability, V(y<t)=∑vpθ0(v∣x,y<t) V(y<t,v)V(y_{<t})=\sum_{v} p_{\theta_0}(v\mid x,y_{<t})\,V(y_{<t},v), so Eyt∼pθ0[exp⁡(At)]=1\mathbb{E}_{y_t\sim p_{\theta_0}}[\exp(A_t)]=1. A weight above the prior therefore marks a token that raises the success probability above the average over the model's own next-token choices.
  2. Telescoping credit. ∑t=1TAt=log⁡pθ0(c∣x,y)−log⁡pθ0(c∣x)\sum_{t=1}^{T} A_t = \log p_{\theta_0}(c\mid x,y) - \log p_{\theta_0}(c\mid x). The per-token advantages split the total evidence that the response provides for the correct answer.

This signal comes from a single forward pass of the frozen model with the answer in its prompt, and it needs no rollouts or trained verifier. It is the frozen model's own estimate, and the weight also keeps the prior factor pθ0(yt)p_{\theta_0}(y_t). Whether the estimate is useful in practice is the question W1 answers: with SR fixed, adding the privileged signal improves accuracy on all four backbones.

We will add this interpretation of Prop. 3, with the derivation in the appendix, and replace "reasoning importance" with "process advantage under the frozen model".

Part 2/2: Appendix: per-benchmark results

Each cell is the mean ± std over 3 evaluation seeds. The Avg@8 and Pass@8 columns are macro averages over the five benchmarks.

Qwen2.5-Math-1.5B

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
PriFT57.48±0.2812.07±1.0527.41±0.675.28±1.0537.29±1.4427.90±0.3253.38±1.40
SPFT64.59±0.1420.28±0.5828.93±0.506.39±0.6443.23±1.1032.68±0.3254.76±1.34
PriFT-SR (new)60.69±0.5017.10±0.4829.51±0.329.16±0.7245.83±2.0332.46±0.4956.46±1.52
SPFT-SR67.70±0.3820.82±0.2330.22±0.606.25±3.4144.58±1.0033.92±0.6155.33±1.06

Qwen2.5-Math-7B

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
PriFT71.75±0.3219.95±0.4432.49±0.8516.53±2.1054.69±0.6339.08±0.3560.50±0.62
SPFT74.11±0.2525.71±0.7937.20±0.2614.31±1.4658.86±1.1942.04±0.7262.95±1.62
PriFT-SR (new)70.98±0.7123.13±0.2834.93±0.2315.70±0.4857.71±0.1840.49±0.3058.39±0.92
SPFT-SR72.09±0.0829.47±0.8436.55±0.3614.17±2.3259.38±1.5742.33±0.4562.14±0.84

Qwen3-8B-Base

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
PriFT69.95±0.1330.90±0.3136.61±0.4210.55±0.4849.48±2.0839.50±0.5260.17±0.27
SPFT72.37±0.4831.79±0.4937.45±0.3410.70±0.4849.89±2.9040.44±0.5260.05±0.67
PriFT-SR (new)74.08±0.3030.92±0.5339.86±0.1513.47±2.1352.19±0.5442.10±0.6959.76±0.78
SPFT-SR76.15±0.1334.58±0.8839.70±0.1213.61±1.2755.73±0.6543.96±0.4961.29±1.29

Qwen3-1.7B-Base

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
PriFT56.39±0.4418.20±0.4922.44±0.363.05±0.4831.46±1.9026.30±0.5247.58±1.26
SPFT58.25±0.2318.92±0.5223.46±0.853.47±1.8830.73±0.7226.97±0.6148.77±0.26
PriFT-SR (new)60.47±0.4921.17±0.5623.91±0.305.42±0.8434.06±1.6229.00±0.2048.05±0.52
SPFT-SR61.52±0.4223.49±0.2824.98±0.435.69±0.2436.36±0.4830.41±0.0949.68±0.78

The new PriFT-SR ablation largely addresses my first concern, but the second remains only partially resolved. Although the authors agree to narrow their claims, they do not establish that providing the answer in the prompt yields the same distribution as conditioning the original model on eventual success. The accuracy gains support the method’s empirical usefulness, but do not validate the proposed process-advantage interpretation of the actual token weights.

Consequently, I will maintain my original score.

Reviewer_fGEGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies token-weighted SFT. At the objective level it shows that dynamic weighting (e.g., DFT), under the idealization that each update reaches the pointwise optimum, induces progressive distribution sharpening pθk(v∣h)∝pθ0(v∣h) pdata(v∣h)kp_{\theta_k}(v\mid h)\propto p_{\theta_0}(v\mid h)\,p_{\mathrm{data}}(v\mid h)^k (Prop. 1), whereas static/frozen-prior weighting targets a fixed product of the data distribution and a frozen prior (Prop. 2). It then proposes Self-Privileged Fine-Tuning (SPFT): a frozen copy of the initial model, conditioned on training-only privileged information (verified answers for math, AST-derived rules for code), produces static token weights, and a Bayesian decomposition (Prop. 3) shows this prior equals the base prior times an outcome-likelihood ratio, upweighting outcome-predictive tokens. SPFT and its hard-selection variant SPFT-SR are evaluated on math reasoning and code generation against SFT, DFT, ASFT, TALR, and PriFT under a shared protocol, with frozen-vs-online (Table 3) and privileged-information (Table 4) ablations.

Critical strengths
  1. The objective-level theory separates the two method classes and is then used to design the method. Prop. 1 derives the DFT sharpening collapse and Prop. 2 the stationary product target of static weighting, and the frozen-vs-online ablation (Table 3, frozen +3.12 Avg@8) matches the stationary-target prediction. This is the paper's main claim and it is supported.

  2. SPFT is a simple, well-motivated use of training-only privileged information with a precise interpretation: Prop. 3 / Eqs. 12–15 write the privileged prior as the base prior times an outcome-likelihood ratio qt=pθ0(c∣x,y≤t)pθ0(c∣x,y<t) pθ0(yt∣x,y<t)q_t=\frac{p_{\theta_0}(c\mid x,y_{\le t})}{p_{\theta_0}(c\mid x,y_{<t})}\,p_{\theta_0}(y_t\mid x,y_{<t}), explaining how conditioning on the answer or code rule reweights target tokens. Under a matched protocol it yields encouraging gains, notably +10.67 on code over SFT (Table 2) and a supportive privileged-information ablation (Table 4).

Critical weaknesses
  1. No seed or variance is reported, so the main empirical claims are not well supported. All headline results use a single training run and a single evaluation seed (Section 5.1), yet the paper says SPFT "consistently outperforms" the baselines. Several margins are small: on Qwen3-8B (Table 1) SPFT averages 47.80 vs PriFT's 48.63 Avg@8 and only wins through SPFT-SR (49.74), and on Qwen3-1.7B (Table 7) the variants trail on several benchmarks. Without repeated runs or confidence intervals, the claims of consistent superiority and more stable optimization are stronger than the tables show. This is fixable: limit the claims to the settings actually won, and report multi-seed or bootstrap statistics.

  2. The comparison against the closest alternatives is incomplete, so it is hard to isolate where the gains come from. The core mechanism, a self-teacher conditioned on the gold answer, is basically gold-conditioned self-distillation (e.g., OPSD, Zhao et al., 2026). The paper shows that full-vocabulary KD is worse (Table 5), but it does not include a direct offline comparison against gold-conditioned distillation-style weighting, nor against several token-selection/reweighting methods. So it is unclear whether the gains come from the specific Bayesian prior or simply from having any gold-conditioned signal. The paper should add the most relevant comparisons where compute allows, or else soften its novelty and empirical claims.

Questions for authors
  1. [Clarification] How many independent training runs and evaluation seeds underlie Tables 1, 2, and 7? If the numbers are single-seed, do the principal SPFT/SPFT-SR vs PriFT and SFT gaps persist across seeds? Even multi-seed estimates on one backbone would substantially affect my confidence.

  2. [Clarification] On Qwen3-8B, SPFT-SR is the stronger variant while SPFT trails PriFT on aggregate Avg@8/Pass@8. Which variant is the principal method, and can the "consistently outperforms" language be revised accordingly?

2 comments

Part 1/2

Thank you for the thorough review and for recognizing the objective-level theory and the frozen-vs-online ablation. The suggestions on variance and on gold-conditioned distillation led to several new experiments, summarized below. Full per-benchmark tables are in Part 2. All numbers are macro Avg@8 over the five benchmarks.

W1 / Q1: Seeds and variance. We have now re-evaluated every Qwen2.5-Math model with 3 evaluation seeds (mean ± std) and kept all other settings unchanged.

MethodQwen2.5-Math-1.5BQwen2.5-Math-7B
SFT25.55±0.4134.56±0.67
DFT29.65±0.0135.54±0.34
ASFT27.31±0.3536.70±0.19
TALR26.36±0.3132.45±0.63
PriFT27.90±0.3239.08±0.35
SPFT32.68±0.3242.04±0.72
SPFT-SR33.92±0.6142.33±0.45

To also capture training variance, we retrained SPFT-SR and SPFT on 1.5B with 3 training seeds. Each run is averaged over its 3 evaluation seeds:

1.5Bseed 1seed 2seed 3mean ± std
SPFT-SR33.9234.5034.3934.27±0.31
SPFT32.6832.3132.1732.39±0.26

SPFT-SR is ahead of SPFT in every run, by 1.88 on average, which is several times the run-to-run std. The frozen-vs-online gap (Table 3) also holds with 3 evaluation seeds (+3.40). As suggested, we will report variance in all tables and replace "consistently outperforms" with claims tied to these margins.

Q2: Which variant is principal? SPFT-SR is the principal method, and SPFT is an ablation. SR standardizes the weights within each response, sets negative scores to zero, and rescales them to mean 1. With multiple seeds the picture is consistent: adding SR improves SPFT on 1.5B (+1.23, and in every training run: +1.23, +2.20, +2.21) and slightly on 7B (42.33 vs 42.04).

W2: Comparison with gold-conditioned distillation. To compare directly with OPSD in the offline setting, we extended both the objectives and the privileged information.

  • More objectives. Beyond the full-vocabulary forward and reverse KL of Table 5, we added k1, the sampled-token approximation used by OPSD. It applies the log-probability difference between teacher and student on each reference token, sg(log⁡q(yt∣x,c)−log⁡pθ(yt∣x))\mathrm{sg}\big(\log q(y_t\mid x,c)-\log p_\theta(y_t\mid x)\big), as an advantage. This gives an offline OPSD-style objective.
  • Richer privileged information. In OPSD the teacher sees a full solution, not only the final answer. To match this more closely, we condensed each reference solution into a hint. Qwen3-8B summarizes the key idea and main steps in two to three sentences, and we append the gold answer. The teacher thus knows how to solve the problem, while the student never sees the hint.

In all runs the teacher is the same frozen initial model with the same hint, and data and hyperparameters are identical. Only the way the teacher's signal enters the loss changes. In SPFT, the teacher's probability of each reference token becomes that token's weight in the cross-entropy loss.

Objective (same frozen teacher, same hint)Avg@8
SPFT (teacher probability as token weight)31.80±0.77
Full-vocabulary reverse KL30.48±0.17
Full-vocabulary forward KL24.69±0.61
Offline OPSD-style k114.44±0.47

With identical information, the SPFT token-weight form works best. Reverse KL comes closest (−1.3), and forward KL and k1 fall well behind. The gold-conditioned signal alone does not explain the gain; how it enters the loss matters. With only the final answer, SPFT stays at a similar level (32.68).

Part 2/2: Appendix: per-benchmark results

Each cell is the mean ± std over 3 evaluation seeds (in the training-seed table, ± is across the 3 training runs). The Avg@8 and Pass@8 columns are macro averages over the five benchmarks.

Qwen2.5-Math-1.5B

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
SFT56.41±0.4715.50±0.3121.80±0.513.33±1.9130.73±2.0825.55±0.4149.90±1.41
DFT64.46±1.3421.52±0.8224.58±0.405.28±0.6432.40±1.5729.65±0.0146.72±0.55
ASFT58.20±0.9218.72±0.8823.43±0.394.44±1.2731.77±0.9527.31±0.3549.46±1.06
TALR57.19±0.8816.82±1.0522.17±0.383.75±2.0831.87±0.3226.36±0.3149.21±0.93
PriFT57.48±0.2812.07±1.0527.41±0.675.28±1.0537.29±1.4427.90±0.3253.38±1.40
SPFT64.59±0.1420.28±0.5828.93±0.506.39±0.6443.23±1.1032.68±0.3254.76±1.34
SPFT-SR67.70±0.3820.82±0.2330.22±0.606.25±3.4144.58±1.0033.92±0.6155.33±1.06

Qwen2.5-Math-7B

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
SFT68.49±0.8124.97±0.5331.72±0.666.25±1.8141.35±0.7934.56±0.6758.48±1.19
DFT68.15±0.1625.37±0.3833.11±0.166.25±0.4244.79±1.7235.54±0.3450.04±0.51
ASFT70.59±0.2922.32±0.3833.99±0.487.64±1.3448.96±1.9936.70±0.1955.95±0.31
TALR65.93±0.4022.59±0.1829.60±0.446.11±0.8638.02±2.0832.45±0.6355.34±2.01
PriFT71.75±0.3219.95±0.4432.49±0.8516.53±2.1054.69±0.6339.08±0.3560.50±0.62
SPFT74.11±0.2525.71±0.7937.20±0.2614.31±1.4658.86±1.1942.04±0.7262.95±1.62
SPFT-SR72.09±0.0829.47±0.8436.55±0.3614.17±2.3259.38±1.5742.33±0.4562.14±0.84

Qwen2.5-Math-1.5B, 3 training seeds

MethodMATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
SPFT64.64±0.2320.44±0.1728.84±0.146.30±0.1641.73±1.3232.39±0.2654.68±0.13
SPFT-SR68.34±0.6321.20±0.4130.47±0.275.88±0.6445.45±0.7734.27±0.3154.92±0.36

Same frozen teacher and hint, different objectives (Qwen2.5-Math-1.5B)

Objective (privileged info)MATH-500MinervaOlympiadBenchAIME24AMC23Avg@8Pass@8
SPFT (hint)64.34±0.5120.50±0.8028.35±0.366.25±1.1039.58±1.7231.80±0.7754.16±2.83
Reverse KL (hint)60.48±0.5516.36±0.7127.57±0.597.22±0.6440.73±2.0130.48±0.1754.89±1.27
Forward KL (hint)52.18±0.6913.30±0.3723.90±0.496.66±1.9127.40±3.1624.69±0.6153.57±1.17
Offline OPSD-style k1 (hint)28.34±0.497.26±0.2415.92±0.313.89±0.8616.77±2.5314.44±0.4745.87±1.11

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and AC

We sincerely thank the reviewers for their careful reading and constructive suggestions, and the AC for overseeing the discussion. Below we summarize the strengths the reviewers noted and how we addressed the reviewers' concerns. The individual responses give full details.

Strengths noted by the reviewers

  • Theory that guides the method (fGEG). The objective-level analysis separates dynamic from static weighting, and the frozen-vs-online ablation matches the predicted stationary target.
  • A simple, well-motivated use of privileged information (fGEG). Prop. 3 gives the token weight a precise Bayesian interpretation.
  • Easy to adopt (RHPH). SPFT keeps the offline SFT pipeline, with no rollouts, no external teacher, and no extra inference cost.
  • Comprehensive evaluation (RHPH, zPC9). The experiments cover many baselines and report both average accuracy and multi-sample success.
  • Clear writing (fGEG, zPC9).

Main concerns and our responses

1. Are the gains larger than the variance? (fGEG, zPC9) We re-evaluated every Qwen2.5-Math model with 3 evaluation seeds (mean ± std):

Qwen2.5-Math-1.5BQwen2.5-Math-7B
SFT25.55±0.4134.56±0.67
Strongest prior baselineDFT 29.65±0.01PriFT 39.08±0.35
SPFT32.68±0.3242.04±0.72
SPFT-SR33.92±0.6142.33±0.45

We also retrained SPFT and SPFT-SR on 1.5B with 3 training seeds. SPFT-SR scores 34.27±0.31, SPFT scores 32.39±0.26, and SPFT-SR is ahead in every run. Reviewer zPC9 has confirmed that this concern is addressed.

2. Does the gain come from privileged information or from the SR transformation? (RHPH) We added PriFT-SR, which applies the same SR transformation to PriFT, whose weights use no privileged information. We ran all four methods on both model families:

BackbonePriFTSPFTPriFT-SRSPFT-SR
Qwen2.5-Math-1.5B27.90±0.3232.68±0.3232.46±0.4933.92±0.61
Qwen2.5-Math-7B39.08±0.3542.04±0.7240.49±0.3042.33±0.45
Qwen3-8B-Base39.50±0.5240.44±0.5242.10±0.6943.96±0.49
Qwen3-1.7B-Base26.30±0.5226.97±0.6129.00±0.2030.41±0.09

On every backbone, SPFT outperforms PriFT and SPFT-SR outperforms PriFT-SR. Privileged information therefore helps both with and without SR, and SR does not substitute for it.

3. Which variant should be used? (fGEG, zPC9) SR improves SPFT on 1.5B in every training run (+1.23, +2.20, +2.21) and slightly on 7B (42.33 vs 42.04). We will present SPFT-SR as the default and SPFT as an ablation. Reviewer zPC9 has confirmed that this concern is addressed.

4. Comparison with gold-conditioned distillation (fGEG) On Qwen2.5-Math-1.5B, we compared offline objectives that differ only in the loss; teacher, hint and data are identical:

Objective (same frozen teacher, same hint)Avg@8
SPFT (teacher probability as token weight)31.80±0.77
Full-vocabulary reverse KL30.48±0.17
Full-vocabulary forward KL24.69±0.61
Offline OPSD-style k114.44±0.47

The gold-conditioned signal alone does not explain the gain; how it enters the loss matters.

5. Theory (zPC9, RHPH)

  • Prop. 1 holds without the unique-maximizer assumption. DFT concentrates mass on the set of all data modes, and the conclusion is unchanged. Reviewer zPC9 has confirmed that this concern is addressed.
  • Prop. 3 has a process-advantage reading. The privileged weight equals the prior times exp⁡(At)\exp(A_t). Here AtA_t is the change in the frozen model's log-probability of reaching the correct answer caused by token yty_t. This ratio has unit mean under the model, and summing AtA_t over the response gives the total evidence for the answer.
  • Order of cc and yy. Following zPC9's suggestion, we will write the privileged information cc before the solution yy throughout, including Prop. 3 and its proof.

Reviewer zPC9 has indicated that all of their concerns are resolved and has raised their score. We are grateful for this engagement.

We thank the reviewers and the AC again for their time and helpful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.