acceptodds
Under review as a conference paper at ICLR 2027

TBRD: Task-Aligned LLM Distillation via Brier Relations over Candidate Tokens

Abstract

Knowledge distillation transfers knowledge from a teacher model to a student by matching their output distributions. However, distribution matching supervises candidate preferences implicitly. Its updates can also conflict with the learning signal from the ground-truth token. We propose TBRD to explicitly model candidate relations and adapt the strength of teacher supervision. TBRD includes the teacher's highest-probability token in the student candidate set, models pairwise preferences, and adjusts the weight at each position using the agreement between distillation and task gradients. TBRD improves the average score across nine evaluation tasks by 2.09% over TAD, the strongest baseline in this comparison. These results support jointly modeling candidate relations and adapting supervision to the student's task signal.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Knowledge Distillation, Language Models, Relational Distillation, Gradient Alignment, Adaptive Supervision.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h2zFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TBRD, combining pairwise Brier relations and gradient-based position weighting for language model distillation. It reports improvements over existing baselines on downstream tasks.

Critical strengths

The method is clearly presented and addresses a relevant problem in knowledge distillation.

Critical weaknesses
  1. The experimental work is incomplete. Section 5.4 explicitly states that the main-budget ablation experiments remain pending. These experiments are essential for validating the paper's core contributions, and their absence makes the submission premature.

  2. The available evidence does not support the central mechanism. The existing ablation shows that Uniform Brier outperforms full TBRD in both cross-entropy and accuracy, questioning the effectiveness of the proposed direction-based weighting. Furthermore, several other components lack independent validation.

Questions for authors

Why were the essential ablation experiments left unfinished? How do the authors justify the effectiveness of direction-based weighting when the available results favor removing it?

3 comments

Thank you for recognizing the relevance of the distillation problem and the clear presentation.

1. Why were the essential ablation experiments left unfinished?

Thank you for raising this concern. We completed the component ablation with the 200M student (Table 2).

200M component ablation

VariantHeld-out cross-entropy ↓Next-token accuracy (%) ↑Nine-task average (%) ↑
DIST2.984843.32934.223
Uniform Brier2.981343.49434.731
ALIGN2.981743.62834.947
TBRD2.978443.85335.005

DIST uses distribution matching. Uniform Brier adds Brier relations. ALIGN adds direction-based weighting to DIST. TBRD combines both additions. Full TBRD achieves the lowest cross-entropy and the highest next-token accuracy and nine-task average.

2. How do the authors justify the effectiveness of direction-based weighting when the available results favor removing it?

Thank you for this question. We directly compared Uniform Brier with TBRD. Both include distribution matching and Brier relations. TBRD adds direction-based position weighting.

Uniform Brier and TBRD at 50K updates

MetricUniform BrierTBRDChange
Held-out cross-entropy ↓2.98132.9784−0.0029
Next-token accuracy (%) ↑43.49443.853+0.359 pp
Nine-task average (%) ↑34.73135.005+0.274 pp

Adding direction-based weighting improves both held-out prediction quality and downstream accuracy over Uniform Brier (Table 2).

3. Several other components lack independent validation.

Thank you for this suggestion. We compared DIST with Uniform Brier and ALIGN (Table 2). Brier relations improve the nine-task average by 0.508 percentage points. Direction-based weighting improves it by 0.724 points. Combining both gives the highest average, 35.005%.

Thank you for the additional experiments. However, I remain concerned that the original submission stated that "Results at the main training budgets remain pending" (Section 5.4). More importantly, the original short-run ablation showed that "uniform position weights achieve better results on both measures" when Brier relations were held fixed. The new 50K-update results show the opposite trend, but the reason for this discrepancy remains unclear, and the new ablation is still based on a single run. I therefore believe the robustness of the proposed weighting mechanism has not been sufficiently established and will maintain my original rating.

Thank you for raising the questions about the main-budget ablation and the early ranking. The original short-run ablation used 512 updates, about 1% of the 50K budget. We have now completed a matched 50K ablation in the same Qwen 200M setting. On the same development split of 512 sequences, Uniform Brier was slightly ahead at 2,048 updates. At 10K, TBRD had lower CE and higher next-token accuracy, with an accuracy gain of 0.938 percentage points. The accuracy advantage persisted at 25K and 50K on that split. At 50K, the final held-out evaluation showed a reduction in CE from 2.9813 to 2.9784 and an increase in next-token accuracy from 43.494% to 43.853%. The nine-task average rose from 34.731% to 35.005%.

Qwen 200M, seed 1234ΔCE ↓Δ next-token accuracy (pp)↑
2,048 updates, dev+0.005796−0.010684
10K updates, dev−0.028338+0.937782
25K updates, dev−0.001334+0.220003
50K updates, dev+0.003236+0.224355
50K updates, final−0.002885+0.359257

Differences are TBRD minus Uniform Brier. The final evaluation uses a separate set of 2,048 held-out sequences. The observed reversal on the development split occurred between 2,048 and 10K updates.

We also ran a four-way component ablation in Pythia at 10K updates. With Brier relation supervision held fixed, TBRD reduced held-out CE by 0.11253 and increased next-token accuracy by 1.403 percentage points relative to Uniform Brier.

Pythia 10K, seed 1234Held-out CE ↓Next-token accuracy ↑
DIST4.4238730.183%
Uniform Brier4.4527129.740%
ALIGN4.5821028.461%
TBRD4.3401831.143%
Reviewer_22uDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TBRD for LLM distillation, combining distribution matching with pairwise Brier losses over teacher-anchored candidate tokens. It weights the loss at each prediction position using agreement between the distillation and observed-token cross-entropy gradients in student logit space.

Critical strengths
  • The research problem is clear, and the main idea is intuitive.
Critical weaknesses
  • The method is difficult to follow. Several loss terms are added, but the theoretical and experimental evidence is insufficient to establish the necessity and benefit of each component relative to its computational cost.
  • The experimental evaluation is limited. Only 200M and 500M students from the Qwen family are evaluated. The ablation study is incomplete, hyperparameter settings are insufficiently reported, and no computational cost analysis is provided. The gains over competing baselines are moderate, while TBRD introduces additional computation.
Questions for authors
  • What is the role of each component in TBRD, and what performance benefit does it provide relative to its computational cost?
  • How does TBRD perform in KD involving larger LLMs?
3 comments

Thank you for recognizing the clear research problem and intuitive main idea.

1. What is the role of each component in TBRD?

Thank you for this question. Teacher anchoring keeps the teacher's preferred token in the candidate set. Distribution matching transfers probability mass and conditional distributions. Brier relations match pairwise token preferences. Direction-based weighting assigns more supervision to positions with greater agreement with the observed-token objective.

We added a component ablation with the 200M student (Table 2). DIST uses distribution matching. Uniform Brier adds Brier relations. ALIGN adds direction-based weighting to DIST. TBRD combines both additions.

200M component ablation

VariantHeld-out cross-entropy ↓Next-token accuracy (%) ↑Nine-task average (%) ↑
DIST2.984843.32934.223
Uniform Brier2.981343.49434.731
ALIGN2.981743.62834.947
TBRD2.978443.85335.005

Brier relations improve the nine-task average over DIST by 0.508 percentage points. Adding direction-based weighting to Uniform Brier gives a further 0.274-point gain. Full TBRD achieves the best results across all three metrics.

2. What performance benefit does it provide relative to its computational cost?

Thank you for this question. We measured update times for the 500M and 1B students. The table combines the performance and timing results from Tables 1, 4, and 5.

Downstream scores and update times at 500M and 1B

StudentMethodNine-task average (%) ↑Time per update (s) ↓
500MLDRLD36.345.055
500MTAD35.686.414
500MTBRD37.465.317
1BLDRLD34.415.537
1BTAD34.026.677
1BTBRD36.085.327

At 500M, TBRD gains 1.11 percentage points over LDRLD with 5.2% more time per update. It gains 1.77 points over TAD with 17.1% less time. At 1B, TBRD gains 1.67 points over LDRLD and 2.06 points over TAD. Update time falls by 3.8% and 20.2%, respectively. Score gains use unrounded results.

3. Hyperparameter settings are insufficiently reported.

Thank you for pointing this out. Appendix C now reports the requested hyperparameters and implementation details.

4. How does TBRD perform in KD involving larger LLMs?

Thank you for this question. We added a comparison with an 1B Qwen2 student against LDRLD and TAD (Table 5). TBRD scores higher than LDRLD on all nine tasks and higher than TAD on eight.

1B results at 50K updates

MetricLDRLDTADTBRD
ARC-Challenge (%) ↑18.43019.71018.771
ARC-Easy (%) ↑36.86934.51237.205
HellaSwag (%) ↑27.40527.09627.415
LAMBADA OpenAI (%) ↑6.3464.92913.293
OpenBookQA (%) ↑24.60025.20027.000
PIQA (%) ↑57.99857.01858.977
Social IQA (%) ↑34.54534.74934.903
StoryCloze 2016 (%) ↑53.12752.11153.661
WinoGrande (%) ↑50.35550.82953.512
Nine-task average (%) ↑34.40834.01736.082
Held-out cross-entropy ↓3.02413.11242.8549
Next-token accuracy (%) ↑41.2142.0445.05

TBRD improves the nine-task average by 1.67 percentage points over LDRLD and 2.06 points over TAD. Relative to LDRLD, cross-entropy falls by 0.1692 and next-token accuracy rises by 3.84 percentage points.

Thank you for the additional work, which partially addresses my concerns. However, the benefits of individual components relative to their costs, robustness across seeds, and generalization beyond the Qwen family remain insufficiently established. So, I retain my score.

Thank you for clarifying your concerns about component-level benefits and costs, robustness across training seeds, and generalization across model families.

Component benefits and costs

Brier relation supervision and direction-based weighting play complementary roles. The Brier term teaches the student the teacher's pairwise preferences among candidate tokens at each position. Direction-based weighting gives more weight to positions where the distillation gradient aligns more closely with the cross-entropy gradient for the observed next token. The relation-loss gradient links the two components, so the supervision over candidate pairs contributes to determining the position weights.

In the matched Qwen 200M ablation at 50K updates, adding Brier relations to DIST raised the nine-task average from 34.223% to 34.731%. Adding direction-based weighting to Uniform Brier raised it further to 35.005%. Full TBRD exceeded Uniform Brier and ALIGN by 0.274 and 0.058 percentage points, respectively.

Direction scoring and student backpropagation reuse the same relation-loss gradient with respect to the logits. In the Pythia RTX 4090 benchmark, full TBRD took 62.1% more time per update than Vanilla KD and used 7.0% less peak allocated memory. At the same 50K update budget, it improved next-token accuracy by 6.202 percentage points and the nine-task average by 1.063 percentage points.

Validation across training seeds

We added a paired seed-1235 run at each of the two Qwen student sizes. All comparisons use 50K updates and the nine-task scoring convention in Table 7. TBRD's advantage over LDRLD was observed under the second seed at both sizes.

Qwen student sizeTraining seedLDRLD nine-task averageTBRD nine-task averageTBRD − LDRLD (pp)
500M123434.269%35.444%+1.175
500M123534.436%35.261%+0.826
~1B123434.408%36.082%+1.674
~1B123534.774%35.473%+0.698

Evaluation beyond Qwen

We added a 50K-update experiment with a Pythia teacher and student. All methods shared the student initialization, training-data order, and update budget. Relative to LDRLD, the strongest baseline by nine-task average in this setting, TBRD reduced held-out CE by 0.2081 and improved next-token accuracy and the nine-task average by 4.809 and 0.468 percentage points, respectively.

Pythia method (50K updates)Held-out CE ↓Next-token accuracy ↑Nine-task average ↑
PRETRAIN4.363432.4024%32.2304%
Vanilla KD4.440033.5285%32.2413%
LDRLD3.717134.9216%32.8366%
TAD3.976934.8152%32.2472%
TBRD3.509039.7308%33.3047%
Reviewer_suw1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed TBRD, which addresses potential conflicts between teacher-student distribution matching and ground-truth supervision in knowledge distillation. Although distribution matching transfers information from the teacher's output probabilities, its updates may conflict with the student's supervised learning objective. TBRD explicitly models pairwise token preferences by transforming logit differences into preference probabilities and matching teacher-student preferences using Brier squared error. It constructs a candidate set from high-probability student tokens while ensuring the teacher's most preferred token is included, and combines relational supervision with distribution matching. Importantly, the method measures directional agreement between the distillation gradient and the ground-truth cross-entropy gradient at each token position, using this agreement to adaptively allocate distillation supervision across positions within a sequence.

Critical strengths

The paper is well motivated, identifying potential conflicts between teacher-student distribution matching and ground-truth supervision in knowledge distillation. The proposed TBRD provides an interesting approach to addressing these conflicts. The experimental evaluation covers a reasonable variety of downstream tasks, including commonsense reasoning, question answering, and language modeling, and compares TBRD against several existing methods at multiple training budgets. The paper also reports complementary metrics, including KL divergence, cross-entropy, perplexity, next-token accuracy, and calibration error, providing a broader perspective on distillation quality. The reported results show promising improvements in average downstream performance and several prediction-quality metrics, although their robustness and statistical significance require further validation.

Critical weaknesses
  • One of my main concerns is that the effectiveness of some central component of the proposed method is not convinceingly established. Section 5.4 states that ablation results at the main training budgets remain pending. The available short-run ablations show that uniform position weighting with Brier relations outperforms the proposed direction-based weighting in both recorded comparisons. Since direction-based weighting is a central component of TBRD, these raise concerns about its actual contribution. Section 5.5 compares training-loss trajectories across methods using different objective functions. Such comparisons may not be very conclusive, as the loss values and optimization dynamics are not directly comparable across different objectives. Although the authors acknowledge this limitation, the figure provides limited evidence of comparative effectiveness. Furthermore, the definition and normalization of “relative training loss” are unclear.
  • The current submission does not provide sufficient statistical analysis to establish the reliability of the observed improvements. There are no variance estimates or significance tests across multiple training runs. The authors also acknowledge that different methods use different random seeds and training-data prefixes, further complicating the attribution of performance differences to the proposed method. Given the relatively small improvements, matched multi-seed comparisons are important for supporting the central claims.
  • The performance improvements over the strongest baseline are relatively small; if I am correct, the paper does not provide a direct comparison of training computational costs, GPU-hours, or throughput. Matching training-token budgets is useful but does not convincingly establish computational efficiency, particularly given the additional gradient computations and pairwise operations introduced by TBRD.
  • The evaluation uses eight tasks at the 78M-token budget but nine tasks at the larger budgets. Although the comparisons within each budget use the same task set, changing the evaluation tasks makes it difficult to interpret the reported average performance trends across budgets. Reporting results on a consistent task set would improve clarity and comparability.
  • While Appendix C provides useful implementation details, important questions remain about the comparability of the experimental runs. Were all baseline models trained from scratch using identical configurations, initialization seeds, and training data? If not, how were differences in experimental conditions controlled? Furthermore, Table 1 compares TBRD against eight baselines for the 200M student, whereas Table 2 compares it only against Vanilla KD for the 500M student. The rationale for restricting the larger-model comparison to Vanilla KD should be clarified, particularly because Vanilla KD is not the strongest baseline in Table 1.
Questions for authors

see questions in the weaknesses discussions.

2 comments

Thank you for recognizing the motivation, task coverage, and range of evaluation metrics.

1. The effectiveness of some central component of the proposed method is not convincingly established.

Thank you for raising this concern. We added a component ablation with the 200M student (Table 2).

200M component ablation

VariantHeld-out cross-entropy ↓Next-token accuracy (%) ↑Nine-task average (%) ↑
DIST2.984843.32934.223
Uniform Brier2.981343.49434.731
ALIGN2.981743.62834.947
TBRD2.978443.85335.005

Uniform Brier and TBRD both include distribution matching and Brier relations. TBRD adds direction-based weighting. This addition improves the nine-task average by 0.274 percentage points and next-token accuracy by 0.359 points. Cross-entropy falls from 2.9813 to 2.9784. Full TBRD ranks first across all three metrics.

2. Training-loss trajectories across methods using different objective functions.

Thank you for this suggestion. We replaced the training-loss comparison with nine-task average accuracy (Figure 4). All methods are now compared using the same metric. TBRD ranks first at every evaluated checkpoint from 25K through 126.4K updates. It finishes at 37.46%, exceeding LDRLD by 1.11 percentage points.

3. The definition and normalization of “relative training loss” are unclear.

Thank you for pointing this out. We removed the relative-training-loss plot. Each point in Figure 4 is now the unweighted average accuracy over the same nine tasks.

4. There are no variance estimates or significance tests across multiple training runs.

Recent work on pretraining distillation also reports point estimates for its main downstream results, as in Table 1 of Dasgupta et al. (2026). Given the computational cost of repeating training across multiple seeds, we prioritized matched comparisons in this revision.

5. Training computational costs, GPU-hours, or throughput.

Thank you for this question. We measured update times for the 500M student. The table shows performance alongside the reported timings (Tables 1 and 4).

500M downstream scores and update times

MethodNine-task average (%) ↑Time per update (s) ↓
LDRLD36.345.055
TAD35.686.414
TBRD37.465.317

TBRD exceeds LDRLD by 1.11 percentage points with 5.2% more time per update. It exceeds TAD by 1.77 points with 17.1% less time. Score gains use unrounded results.

Table 4 reports per-update timings for the 500M and 1B students. Table 5 reports the 1B downstream and held-out results.

6. Reporting results on a consistent task set would improve clarity and comparability.

Thank you for this suggestion. We now use a common set of nine tasks and scoring rules in the main comparison and across checkpoints.

Evaluation tasks and metrics

ComparisonEvaluation task setReported metric
Main 500M comparison, Table 1Nine tasks, shared across methodsIndividual task scores and their unweighted average
Cross-budget comparison, Figure 4Nine tasks, shared across checkpointsUnweighted nine-task average

7. Were all baseline models trained from scratch using identical configurations, initialization seeds, and training data?

Thank you for this question. All methods in the revised 500M comparison are trained from scratch. Appendix C reports the shared protocol and method-specific details.

8. If not, how were differences in experimental conditions controlled?

Thank you for raising this point. We use the controlled ablation in Table 2 to assess component contributions. Table 1 reports the expanded 500M baseline comparison. Appendix C states the protocol for each analysis.

9. The rationale for restricting the larger-model comparison to Vanilla KD should be clarified.

Thank you for this suggestion. We expanded the 500M comparison to seven baselines, including LDRLD and TAD (Table 1).

500M downstream comparison

MethodNine-task average (%) ↑
Pretrain33.13
Vanilla KD35.87
PD35.13
ATKD35.72
RLD35.53
LDRLD36.34
TAD35.68
TBRD37.46

TBRD has the highest average, 37.46%. It exceeds LDRLD, the strongest baseline, by 1.11 percentage points. TBRD also ranks first on five tasks and ties for first on ARC-Challenge. Gains and rankings use unrounded scores.

The 1B comparison also includes LDRLD and TAD (Table 5). TBRD improves the average by 1.67 and 2.06 percentage points, respectively.

We added full-update timings for eight methods, paired comparisons across two training seeds on Qwen, and a 50K-update comparison with Pythia.

Training cost, throughput, and memory

Vanilla KD was the fastest distillation baseline in our benchmark. Both Vanilla KD and TBRD compute loss gradients with respect to the student logits at each prediction position. TBRD computes the relation-loss gradient before backpropagating through the student so it can use that gradient to score alignment. It then applies the position weights to the cached gradient and reuses it for student backpropagation. The full-update timings include candidate construction, relation operations, and direction-based weighting. In a same-GPU comparison at a microbatch size of four, increasing the chunk size improved training speed. The two chunk-size configurations produced identical held-out CE and next-token accuracy at 30K, and all 292 model parameter tensors were elementwise identical at the 40K checkpoint.

We timed the eight methods in separate runs on RTX 4090 GPUs of the same model. Each run started from the same Pythia 5K checkpoint and used 30 warm-up updates followed by 100 timed updates, with 8,192 prediction positions per update. GPU-hours refer to the measured 100-update window.

MethodTime per update (ms)↓Throughput (prediction positions/s)↑GPU-hours per 100 updates ↓Peak allocated memory (GiB)↓
PRETRAIN291.9728,0580.008116.364
Vanilla KD548.5314,9340.0152410.054
PD912.318,9790.0253411.332
ATKD721.4411,3550.0200410.822
RLD660.8512,3960.0183610.678
LDRLD1,649.444,9670.0458210.678
TAD1,460.845,6080.0405813.803
TBRD889.409,2110.024719.353

TBRD used 46.1% and 39.1% less time per update than LDRLD and TAD, respectively, and 12.4% and 32.2% less peak allocated memory. Compared with Vanilla KD, TBRD took 62.1% more time per update and used 7.0% less peak allocated memory. At the same Pythia 50K update budget, it achieved lower held-out CE, higher next-token accuracy, and a 1.063 percentage-point gain on the nine-task average. Its next-token accuracy exceeded Vanilla KD's at all six shared evaluation checkpoints from 5K to 50K.

Single-H200 training logs show the same trade-off. The optimized TBRD implementation recorded 0.676 s/update in a separate 10K run. TAD, LDRLD, and Vanilla KD recorded 0.877, 1.330, and 0.460 s/update in their respective training windows. The TBRD run that produced the 50K results below recorded 0.809 s/update. In those windows, that TBRD run was faster than TAD and LDRLD and achieved nine-task averages higher by 1.057 and 0.468 percentage points, respectively. It took longer per update than Vanilla KD and achieved gains of 6.202 percentage points in next-token accuracy and 1.063 percentage points on the nine-task average at 50K.

Validation across training seeds

We added seed 1235 for paired comparisons of TBRD and LDRLD with Qwen 500M and approximately 1B students. All comparisons use 50K updates and the scoring convention in Table 7 of the manuscript. HellaSwag and OpenBookQA use normalized accuracy, and the other seven tasks use accuracy. The nine tasks are averaged with equal weights.

Qwen student sizeTraining seedLDRLD nine-task averageTBRD nine-task averageTBRD − LDRLD (pp)
500M123434.269%35.444%+1.175
500M123534.436%35.261%+0.826
~1B123434.408%36.082%+1.674
~1B123534.774%35.473%+0.698

TBRD's advantage over LDRLD was observed again at both student sizes under the second seed, with a different gain magnitude.

Results with a Pythia teacher and student

We extended the comparison to Pythia. The five methods shared the same student initialization, training-data order, and budget of 50K updates. Held-out metrics were computed on the same 512 sequences containing 262,144 valid target tokens. The methods were evaluated on the same nine downstream tasks under a common scoring protocol.

Pythia method (50K updates)Held-out CE ↓Next-token accuracy ↑Nine-task average ↑
PRETRAIN4.363432.4024%32.2304%
Vanilla KD4.440033.5285%32.2413%
LDRLD3.717134.9216%32.8366%
TAD3.976934.8152%32.2472%
TBRD3.509039.7308%33.3047%

Compared with LDRLD, the strongest baseline by nine-task average in this setting, TBRD reduced held-out CE by 0.2081 and increased next-token accuracy and the nine-task average by 4.809 and 0.468 percentage points, respectively. TBRD achieved the lowest held-out CE and the highest next-token accuracy at all six shared checkpoints (5K, 10K, 20K, 30K, 40K, and 50K). Both metrics were computed on the same observed target tokens across all methods.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for your time and for guiding our submission through the review process. For your convenience, we summarize the evidence added during rebuttal on training cost, performance across student sizes and training seeds, generalization beyond Qwen, and the contribution of individual components.

Summary of Additional Evidence

Training speed and memory. In the Pythia benchmark, we measured 100 complete optimizer updates per method in separate runs on RTX 4090 GPUs of the same model. Each run started from the same checkpoint and followed the same timing protocol. Compared with LDRLD and TAD, TBRD took 46.1% and 39.1% less time per update, respectively, and used 12.4% and 32.2% less peak allocated memory.

Qwen results across student sizes. At 50K updates, TBRD achieved the highest nine-task average among the methods evaluated at both student sizes. With the approximately 1B student at seed 1234, TBRD outperformed LDRLD on all nine tasks. The scores below are percentages. HellaSwag and OpenBookQA use normalized accuracy, the other tasks use accuracy, and the nine tasks are averaged with equal weights.

Qwen 500M, 50K updates, seed 1234

TaskPRETRAINVanilla KDPDATKDRLDLDRLDTADTBRD
ARC-Challenge18.94218.08919.02718.08918.08918.85718.51518.771
ARC-Easy29.67233.20732.15532.87035.01736.23734.30137.584
HellaSwag26.20026.93726.51926.57826.81727.23626.58827.365
LAMBADA OpenAI1.76610.4218.0923.8422.7946.0744.17213.060
OpenBookQA23.00025.00023.80024.20026.20025.20025.40026.200
PIQA53.15657.23656.42055.71356.63858.48757.12759.195
Social IQA32.60033.98235.05633.36733.52134.49333.82833.777
StoryCloze 201650.93551.36350.82852.37852.48552.43251.20352.218
WinoGrande48.14547.27749.09252.24950.35549.40850.67150.829
Nine-task average31.60233.72333.44333.25433.54634.26933.53435.444

Qwen approximately 1B, 50K updates, seed 1234

TaskLDRLDTADTBRD
ARC-Challenge18.43019.71018.771
ARC-Easy36.86934.51237.205
HellaSwag27.40527.09627.415
LAMBADA OpenAI6.3464.92913.293
OpenBookQA24.60025.20027.000
PIQA57.99857.01858.977
Social IQA34.54534.74934.903
StoryCloze 201653.12752.11153.661
WinoGrande50.35550.82953.512
Nine-task average34.40834.01736.082

Validation across training seeds. We added paired experiments with seed 1235 using the same 50K update budget and scoring convention as Table 7. For seeds 1234 and 1235, respectively, TBRD exceeded LDRLD on the nine-task average by 1.175 and 0.826 percentage points with the 500M student, and by 1.674 and 0.698 percentage points with the approximately 1B student.

Evaluation with a Pythia teacher and student. All methods shared the same student initialization, training-data order, and budget of 50K updates. Relative to LDRLD, the strongest baseline by nine-task average in this setting, TBRD reduced held-out CE by 0.2081 and improved next-token accuracy and the nine-task average by 4.809 and 0.468 percentage points, respectively. TBRD achieved the lowest held-out CE and highest next-token accuracy among the five methods at all six shared evaluation checkpoints from 5K to 50K.

Pythia method (50K updates)Held-out CE ↓Next-token accuracy ↑Nine-task average ↑
PRETRAIN4.363432.4024%32.2304%
Vanilla KD4.440033.5285%32.2413%
LDRLD3.717134.9216%32.8366%
TAD3.976934.8152%32.2472%
TBRD3.509039.7308%33.3047%

Component contribution. We completed a matched four-way ablation with Qwen 200M at 50K updates. Adding Brier relation supervision raised the nine-task average from 34.223% to 34.731%. Holding Brier fixed and adding direction-based weighting raised the average to 35.005%, reduced held-out CE from 2.9813 to 2.9784, and increased next-token accuracy from 43.494% to 43.853%. Full TBRD achieved the best result on all three metrics in this ablation.

Thank you again for your time and consideration. We hope this summary makes the evidence added during rebuttal easier to locate and assess.

Warm regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.