- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes TBRD, combining pairwise Brier relations and gradient-based position weighting for language model distillation. It reports improvements over existing baselines on downstream tasks.
Critical strengths
The method is clearly presented and addresses a relevant problem in knowledge distillation.
Critical weaknesses
-
The experimental work is incomplete. Section 5.4 explicitly states that the main-budget ablation experiments remain pending. These experiments are essential for validating the paper's core contributions, and their absence makes the submission premature.
-
The available evidence does not support the central mechanism. The existing ablation shows that Uniform Brier outperforms full TBRD in both cross-entropy and accuracy, questioning the effectiveness of the proposed direction-based weighting. Furthermore, several other components lack independent validation.
Questions for authors
Why were the essential ablation experiments left unfinished? How do the authors justify the effectiveness of direction-based weighting when the available results favor removing it?
3 comments
Thank you for the additional experiments. However, I remain concerned that the original submission stated that "Results at the main training budgets remain pending" (Section 5.4). More importantly, the original short-run ablation showed that "uniform position weights achieve better results on both measures" when Brier relations were held fixed. The new 50K-update results show the opposite trend, but the reason for this discrepancy remains unclear, and the new ablation is still based on a single run. I therefore believe the robustness of the proposed weighting mechanism has not been sufficiently established and will maintain my original rating.
Thank you for raising the questions about the main-budget ablation and the early ranking. The original short-run ablation used 512 updates, about 1% of the 50K budget. We have now completed a matched 50K ablation in the same Qwen 200M setting. On the same development split of 512 sequences, Uniform Brier was slightly ahead at 2,048 updates. At 10K, TBRD had lower CE and higher next-token accuracy, with an accuracy gain of 0.938 percentage points. The accuracy advantage persisted at 25K and 50K on that split. At 50K, the final held-out evaluation showed a reduction in CE from 2.9813 to 2.9784 and an increase in next-token accuracy from 43.494% to 43.853%. The nine-task average rose from 34.731% to 35.005%.
| Qwen 200M, seed 1234 | ΔCE ↓ | Δ next-token accuracy (pp)↑ |
|---|---|---|
| 2,048 updates, dev | +0.005796 | −0.010684 |
| 10K updates, dev | −0.028338 | +0.937782 |
| 25K updates, dev | −0.001334 | +0.220003 |
| 50K updates, dev | +0.003236 | +0.224355 |
| 50K updates, final | −0.002885 | +0.359257 |
Differences are TBRD minus Uniform Brier. The final evaluation uses a separate set of 2,048 held-out sequences. The observed reversal on the development split occurred between 2,048 and 10K updates.
We also ran a four-way component ablation in Pythia at 10K updates. With Brier relation supervision held fixed, TBRD reduced held-out CE by 0.11253 and increased next-token accuracy by 1.403 percentage points relative to Uniform Brier.
| Pythia 10K, seed 1234 | Held-out CE ↓ | Next-token accuracy ↑ |
|---|---|---|
| DIST | 4.42387 | 30.183% |
| Uniform Brier | 4.45271 | 29.740% |
| ALIGN | 4.58210 | 28.461% |
| TBRD | 4.34018 | 31.143% |
Thank you for recognizing the relevance of the distillation problem and the clear presentation.
1. Why were the essential ablation experiments left unfinished?
Thank you for raising this concern. We completed the component ablation with the 200M student (Table 2).
200M component ablation
DIST uses distribution matching. Uniform Brier adds Brier relations. ALIGN adds direction-based weighting to DIST. TBRD combines both additions. Full TBRD achieves the lowest cross-entropy and the highest next-token accuracy and nine-task average.
2. How do the authors justify the effectiveness of direction-based weighting when the available results favor removing it?
Thank you for this question. We directly compared Uniform Brier with TBRD. Both include distribution matching and Brier relations. TBRD adds direction-based position weighting.
Uniform Brier and TBRD at 50K updates
Adding direction-based weighting improves both held-out prediction quality and downstream accuracy over Uniform Brier (Table 2).
3. Several other components lack independent validation.
Thank you for this suggestion. We compared DIST with Uniform Brier and ALIGN (Table 2). Brier relations improve the nine-task average by 0.508 percentage points. Direction-based weighting improves it by 0.724 points. Combining both gives the highest average, 35.005%.