- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Self-Privileged Fine-Tuning (SPFT), which uses a frozen copy of the initial model conditioned on privileged information as a prior to provide guidance for token-level weighting during fine-tuning.
Critical strengths
S1: This paper is well written and easy to follow.
S2: This paper presents comprehensive experiments to evaluate the proposed method.
Critical weaknesses
W1 (marginal improvement): The empirical results seem to have at most very marginal improvement over baselines. Considering the large variance of the avg@8 and pass@8 metrics, it is unclear whether this method is truly outperforming existing methods.
W2 (unrealistic assumption): It seems unrealistic to assume that language data has a unique maximizer in Proposition 1. The proof of Proposition 1 seems to crucially rely on this unrealistic assumption, so whether Proposition 1 holds in practice is questionable. It is not hard to prove a similar result without this assumption, so it is a bit unclear why this paper decides to use this assumption.
W3 (inconsistent performance): The results show no clear ranking between SPFT and SPFT-SR. Consequently, it is a bit unclear how to decide which variant to use in practice.
Questions for authors
See weaknesses.
8 comments
Part 2/2: Appendix: per-benchmark results
Each cell is the mean ± std over 3 evaluation seeds (in the training-seed table, ± is across the 3 training runs). The Avg@8 and Pass@8 columns are macro averages over the five benchmarks.
Qwen2.5-Math-1.5B
| Method | MATH-500 | Minerva | OlympiadBench | AIME24 | AMC23 | Avg@8 | Pass@8 |
|---|---|---|---|---|---|---|---|
| SFT | 56.41±0.47 | 15.50±0.31 | 21.80±0.51 | 3.33±1.91 | 30.73±2.08 | 25.55±0.41 | 49.90±1.41 |
| DFT | 64.46±1.34 | 21.52±0.82 | 24.58±0.40 | 5.28±0.64 | 32.40±1.57 | 29.65±0.01 | 46.72±0.55 |
| ASFT | 58.20±0.92 | 18.72±0.88 | 23.43±0.39 | 4.44±1.27 | 31.77±0.95 | 27.31±0.35 | 49.46±1.06 |
| TALR | 57.19±0.88 | 16.82±1.05 | 22.17±0.38 | 3.75±2.08 | 31.87±0.32 | 26.36±0.31 | 49.21±0.93 |
| PriFT | 57.48±0.28 | 12.07±1.05 | 27.41±0.67 | 5.28±1.05 | 37.29±1.44 | 27.90±0.32 | 53.38±1.40 |
| SPFT | 64.59±0.14 | 20.28±0.58 | 28.93±0.50 | 6.39±0.64 | 43.23±1.10 | 32.68±0.32 | 54.76±1.34 |
| SPFT-SR | 67.70±0.38 | 20.82±0.23 | 30.22±0.60 | 6.25±3.41 | 44.58±1.00 | 33.92±0.61 | 55.33±1.06 |
Qwen2.5-Math-7B
| Method | MATH-500 | Minerva | OlympiadBench | AIME24 | AMC23 | Avg@8 | Pass@8 |
|---|---|---|---|---|---|---|---|
| SFT | 68.49±0.81 | 24.97±0.53 | 31.72±0.66 | 6.25±1.81 | 41.35±0.79 | 34.56±0.67 | 58.48±1.19 |
| DFT | 68.15±0.16 | 25.37±0.38 | 33.11±0.16 | 6.25±0.42 | 44.79±1.72 | 35.54±0.34 | 50.04±0.51 |
| ASFT | 70.59±0.29 | 22.32±0.38 | 33.99±0.48 | 7.64±1.34 | 48.96±1.99 | 36.70±0.19 | 55.95±0.31 |
| TALR | 65.93±0.40 | 22.59±0.18 | 29.60±0.44 | 6.11±0.86 | 38.02±2.08 | 32.45±0.63 | 55.34±2.01 |
| PriFT | 71.75±0.32 | 19.95±0.44 | 32.49±0.85 | 16.53±2.10 | 54.69±0.63 | 39.08±0.35 | 60.50±0.62 |
| SPFT | 74.11±0.25 | 25.71±0.79 | 37.20±0.26 | 14.31±1.46 | 58.86±1.19 | 42.04±0.72 | 62.95±1.62 |
| SPFT-SR | 72.09±0.08 | 29.47±0.84 | 36.55±0.36 | 14.17±2.32 | 59.38±1.57 | 42.33±0.45 | 62.14±0.84 |
Qwen2.5-Math-1.5B, 3 training seeds
| Method | MATH-500 | Minerva | OlympiadBench | AIME24 | AMC23 | Avg@8 | Pass@8 |
|---|---|---|---|---|---|---|---|
| SPFT | 64.64±0.23 | 20.44±0.17 | 28.84±0.14 | 6.30±0.16 | 41.73±1.32 | 32.39±0.26 | 54.68±0.13 |
| SPFT-SR | 68.34±0.63 | 21.20±0.41 | 30.47±0.27 | 5.88±0.64 | 45.45±0.77 | 34.27±0.31 | 54.92±0.36 |
SPFT vs SPFT-SR with a hint as privileged information (Qwen2.5-Math-1.5B)
| Method | MATH-500 | Minerva | OlympiadBench | AIME24 | AMC23 | Avg@8 | Pass@8 |
|---|---|---|---|---|---|---|---|
| SPFT (hint) | 64.34±0.51 | 20.50±0.80 | 28.35±0.36 | 6.25±1.10 | 39.58±1.72 | 31.80±0.77 | 54.16±2.83 |
| SPFT-SR (hint) | 68.27±0.27 | 20.97±0.10 | 30.22±0.53 | 6.53±1.34 | 44.90±2.96 | 34.18±0.37 | 55.40±0.58 |
Thanks for the response. W2 and W3 have been addressed.
Regarding W1, I am a bit confused by the provided tables. In the first table (3 eval seeds), Qwen2.5-Math-1.5B has mean Avg@8 = 32.68 and 33.92 for SPFT and SPFT-SR, respectively. However, the two numbers are exactly the same as the seed 1 results in the second table, which seems to be different from the description of the first table (i.e., the 3-seed results coincide with the 1-seed results). Could you please clarify the meanings of the two tables? Thanks!
Part 1/2
Thank you for the review and the kind words on the writing and experiments. The concerns led us to run a multi-seed study and to simplify Prop. 1. Full per-benchmark tables, with standard deviations for every metric, are in Part 2.
W1: Are the improvements larger than the variance? With one training run and one evaluation seed, the submission indeed made this hard to judge. We have now re-evaluated every Qwen2.5-Math model with 3 evaluation seeds, keeping all other settings unchanged. Macro Avg@8 over the five benchmarks (mean ± std):
Evaluation-seed stds stay at about 0.7 or below, while SPFT-SR leads the strongest baseline (DFT on 1.5B, PriFT on 7B) by 4.27 and 3.25, and SFT by about 8 points.
Evaluation seeds do not capture training variance, so on 1.5B we also retrained SPFT-SR and SPFT with 3 training seeds. Each run is averaged over its 3 evaluation seeds:
SPFT-SR is ahead of SPFT in every run, by 1.88 on average. This gap is several times the run-to-run std. We will report variance in all tables and base our claims on margins that hold across runs.
W2: The unique-maximizer assumption in Prop. 1. This is correct: uniqueness is not needed, and we will remove it. Let M=argmaxvpdata(v∣h) be the set of all maximizers. Assume pθ0(M∣h)>0 and write p∗=maxvpdata(v∣h). Then
pθk(v∣h)=∑upθ0(u∣h)(pdata(u∣h)/p∗)kpθ0(v∣h)(pdata(v∣h)/p∗)k.
For v∈/M the ratio pdata(v∣h)/p∗ is below 1, so its k-th power vanishes; on M it equals 1. Hence
pθk(v∣h)⟶pθ0(M∣h)pθ0(v∣h)1[v∈M].
DFT therefore still collapses all mass onto the data modes, and when there are ties it keeps the initial model's relative preferences among them. The unique-mode case is just ∣M∣=1. Only one step of the proof changes, and the conclusion of Prop. 1 stays the same. We will update the proposition and its proof.
W3: Which variant should one use? We recommend SPFT-SR and treat SPFT as an ablation. With multiple seeds, the ranking that looked mixed in the submission becomes clearer:
The reason is the design of SR. A hard problem lowers the teacher's probabilities for every token of its solution together, so raw weights partly reflect problem difficulty. Standardizing within each response removes this shared shift and keeps the token-level differences (Section 4). We will state SPFT-SR as the default in the method section.