acceptodds
Under review as a conference paper at ICLR 2027

Whom Should a Student Trust? Unsupervised Reliability Estimation for Multi-Teacher Distillation

Abstract

Multi-teacher distillation provides diverse supervision for training language models. However, teacher-generated responses vary in quality and suitability for the student, making it difficult to determine which teachers to trust and which responses to learn from. Human annotations and external LLM judges can guide selection, but add cost and may introduce evaluator-specific biases, motivating unsupervised approaches. Existing student-based methods select responses using token probabilities to measure familiarity or loss gradients to characterize their effects on optimization. However, familiarity does not directly indicate learning benefit, and computing parameter gradients requires additional backward passes. Therefore, we propose Student-informed Unsupervised Reliability Estimation (SURE), which jointly estimates teacher reliability and response-selection probabilities without external quality judgments. In particular, SURE measures coherence among student gradients from different teachers’ responses to the same prompt without backpropagating through the frozen student. It then models this evidence over a teacher-prompt bipartite graph to iteratively refine teacher reliability and response-selection probabilities, integrating within-prompt coherence with cross-prompt teacher reliability. Experiments across pretrained students from different model families show consistent improvements over other unsupervised multi-teacher distillation baselines. Further analyses show higher judged quality of the selected responses and support the value of estimating teacher reliability across prompts for curating effective multi-teacher supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unsupervised multi-teacher distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VieBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SURE for unsupervised multi-teacher distillation. SURE computes gradient signals at the final hidden states of a frozen student and combines within-prompt gradient coherence with teacher reliability estimated over a teacher-prompt bipartite graph. It selects one response from four candidates for each of 63,748 UltraFeedback prompts. The selected data are used to fine-tune four student models. Validation includes downstream benchmarks, component ablations, external GPT-4 quality ratings, teacher analyses, case studies, and runtime comparisons.

Critical strengths
  1. The method combines prompt-level response evidence with cross-prompt teacher information in a computationally practical way. SURE derives gradients at the final hidden states without backpropagating through student parameters. It then uses a bipartite graph to jointly refine teacher reliability and response-selection probabilities. This directly addresses a limitation of purely likelihood-based or prompt-local selection.

  2. The empirical results are broad and internally consistent. SURE improves the seven-metric average over every listed baseline for all four students. In the two reported ablation settings, removing reliability estimation reduces the average by 5.98% and 1.87%. The paper also reports higher external quality ratings and modest runtime overhead relative to GRAPE and RSR.

Critical weaknesses
  1. All downstream fine-tuning results are reported with seed 42. Full-parameter fine-tuning can have meaningful training variance. Without independent matched-seed repetitions, it is unclear whether SURE's average advantage over the strongest baseline is stable. This is the main limitation on my confidence in the empirical conclusion. Does SURE's seven-metric average advantage remain stable across at least 2 independent matched fine-tuning seeds for SURE and the strongest baseline for each student?

  2. The estimated teacher reliability is not directly calibrated against independent correctness or human quality. The downstream results and GPT-4 ratings show practical value, but the score may partly reflect teacher consensus or student-relative preference. A direct teacher-level correlation analysis on verifiable tasks, together with high-consensus error cases, would clarify what the reliability quantity measures.

Questions for authors

Construct-validity clarification: How strongly does estimated teacher reliability correlate with verifiable teacher correctness, human quality judgments, or quality across independent judges?

1 comment
AuthorsOpenReview ↗

Official Comments by Authors

W1. All downstream…

We thank the reviewer for highlighting the importance of training variance. To address this concern, we conduct independent matched-seed fine-tuning repetitions with three seeds (42, 43, 44) for SURE and the strongest baseline for each student (GRAPE and Max Token Length) following common experimental practice [1-2]. We report the mean ± standard deviation across seeds below:

Llama-3.2-3BAE2 WRAE2 LC WRIFEval StrictIFEval LooseMT-BenchMMLUMMLU-ProAbs. Δ
SURE3.18±0.663.17±0.6523.91±1.3027.91±1.114.327±0.02854.69±1.0724.99±0.62
GRAPE1.91±0.391.74±0.3618.79±0.7522.55±0.554.048±0.14754.06±0.7923.79±0.652.54
Qwen3-8B-BaseAE2 WRAE2 LC WRIFEval StrictIFEval LooseMT-BenchMMLUMMLU-ProAbs. Δ
SURE11.10±0.8614.80±1.3746.15±1.8550.65±1.446.598±0.14475.78±0.0743.36±0.24
Max Token Length11.08±1.0711.86±1.1632.47±3.2437.71±1.646.079±0.07974.84±0.4238.28±0.875.83

SURE maintains higher mean performance across all metrics for both student models, with seven-metric average advantages (Abs. Δ) of 2.54% and 5.83% over GRAPE and Max Token Length. These matched-seed results support the stability and robustness of SURE's performance advantage.

W2 & Q. The estimated… & Construct-validity…

We thank the reviewer for raising this important question. We first clarify that teacher reliability in SURE is intended to measure how consistently a teacher provides useful supervision for a particular student, rather than the teacher’s standalone response correctness. This distinction is consistent with prior work showing that stronger teachers do not necessarily produce better students, as the effectiveness of distillation also depends on how well the provided supervision fits the student's existing capabilities [3-5]. Specifically, SURE computes student-derived gradients from teacher responses and aggregates their coherence across prompts, assigning higher reliability to teachers whose learning signals are consistently supported by those from other teachers. Accordingly, whether our estimated teacher reliability identifies teachers that provide useful learning signals is best assessed through actual student performance after distillation, rather than teacher response accuracy alone.

To investigate this distinction, we conduct additional experiments on the verifiable MATH training set with a response pool generated by different teacher models [6-7]. We directly evaluate the verified accuracy of the selected teacher responses:

MethodSelections for Llama-3.2-3BSelections for Qwen3-8B Base
SURE76.2676.25
Uniform Sampling78.2378.23
Max Token Length76.4576.40
GRAPE74.4076.00
RSR83.1476.99

Although SURE does not select the responses with the highest verified accuracy, student models trained on SURE-selected responses achieve the highest accuracy on the MATH 5K test set:

MethodLlama-3.2-3BQwen3-8B Base
SURE9.4273.10
Uniform Sampling8.7068.22
Max Token Length8.4872.36
GRAPE8.1071.46
RSR8.3456.70

These results on selected-response accuracy and trained student performance reveal two key observations: (i) SURE achieves the highest student accuracy despite not selecting the most accurate teacher responses; (ii) RSR shows substantially lower student performance despite selecting responses with higher verified accuracy especially on Llama-3.2-3B student. These findings highlight that response correctness alone does not guarantee effective student learning, and support SURE's use of student-relative reliability and gradient coherence to select more effective training data.

[1] Wu, M., et al. (2024). Advancing Parameter Efficiency in Fine-tuning via Representation Editing. ACL.

[2] Tong, T., et al. (2025). BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge. ICLR.

[3] Xu, Z., et al. (2025). Stronger Models Are Not Always Stronger Teachers for Instruction Tuning. NAACL.

[4] Zhu, Y., et al. (2022). Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation. NeurIPS.

[5] Fan, W., et al. (2024). Revisit the Essence of Distilling Knowledge through Calibration. ICML.

[6] Hendrycks, D., et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS.

[7] Somerstep, S., et al. (2025). CARROT: A Cost Aware Rate Optimal Router. ICLR.

Reviewer_npk6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SURE, which selects multi-teacher training responses using student-derived hidden-state gradient coherence and teacher reliability estimated across prompts, without external quality labels.

Critical strengths

I like the combination of local response evidence and cross-prompt teacher information. The downstream evaluation covers four students from two model families and shows substantial instruction-following gains. For Qwen3-8B, IFEval Loose improves by 7.03 points over the strongest baseline. The runtime analysis also supports a relatively small overhead over forward-only selection methods, which I believe is a good adds-one.

Critical weaknesses

However, I do have several concerns towards this work:

First, I think the most discriminating controls are missing. The ablations remove teacher centering and reliability estimation, but do not test selection using the learned teacher prior alone. This would establish whether prompt-specific coherence adds value beyond the global teacher ranking.I believe making comparisons with the discussed gradient-based methods, particularly GRACE, would also strengthen the positioning.

Also, the gradient proxy needs stronger validation. Equation 1computes a final-hidden-state gradient, but averaged hidden-state coherence does not directly establish compatibility between actual parameter updates. Thus, I feel that effects of teacher centering and correlated teacher errors therefore remain empirical assumptions. I believe controlled changes to teacher composition would clarify when these signals remain reliable.

Robustness is evaluated narrowly. All experiments use one UltraFeedback pool and one training seed, which could make the results biased. Multiple seeds and uncertainty estimates would strengthen the smaller downstream differences, while another response pool would test transfer beyond this particular teacher mixture.

Questions for authors

Pls refer to the weakness.

4 comments
AuthorsOpenReview ↗

Official Comments by Authors (1/2)

W1. First, …

We thank the reviewer for this valuable suggestion. We first clarify that local coherence is integral to SURE’s iterative estimation of teacher reliability. At each iteration, response-selection probabilities (rr) combine local coherence with teacher reliability (α\alpha) (Eq. 6), and these probabilities are then aggregated across prompts to update (α\alpha) (Eq. 7). Thus, the two estimates are refined jointly, and the final reliability scores already incorporate local coherence.

We agree that selecting the highest-ranked available teacher using the final (α\alpha) is a reasonable alternative, since these scores already incorporate information from (rr) through iterative optimization. However, using the final (rr) offers a modest additional benefit by considering local evidence for each prompt. In some cases, a lower-ranked teacher’s response may receive stronger support from other teachers’ responses to the same prompt, allowing SURE to select it when this local support outweighs the reliability difference. Consequently, the difference between these selection rules reflects the incremental value of using (rr) for prompt-specific adaptation after joint estimation.

To address the concern, we first compare full SURE with selecting the highest-ranked available teacher based solely on the final reliability scores (α\alpha). As shown below, SURE performs better in six of eight model-metric comparisons. On the remaining two metrics, performance remains comparable. These results support a modest additional benefit from prompt-specific adaptation at selection time.

Llama3.2-3BAE2 WRAE2 LC WRMT-BenchMMLU
SURE3.133.094.30055.77
w/ Reliability Only2.842.854.27555.31
Qwen3-8B-BaseAE2 WRAE2 LC WRMT-BenchMMLU
SURE11.9916.216.74475.71
w/ Reliability Only11.6916.316.51975.96

Following the reviewer’s suggestion, we further compare SURE against GRACE [1], a gradient-based method that uses global teacher ranking for response selection. As shown below, SURE consistently outperforms GRACE across all model-metric comparisons:

Llama3.2-3BAE2 WRAE2 LC WRMT-BenchMMLU
SURE3.133.094.30055.77
GRACE0.620.503.72554.12
Qwen3-8B-BaseAE2 WRAE2 LC WRMT-BenchMMLU
SURE11.9916.216.74475.71
GRACE4.335.055.43874.88

Overall, these results demonstrate that prompt-specific selection provides additional gains beyond the learned global teacher ranking across different student models.

W2. Also…

We thank the reviewer for this insightful comment. As SURE focuses on unsupervised distillation, computing parameter-level gradient alignment would require costly backpropagation during data selection. Therefore, we use hidden-state gradients as an efficient proxy and averaged hidden-state coherence to estimate the compatibility of candidate responses. This design is motivated by prior work showing the effectiveness of hidden-state gradient alignment in knowledge distillation and evidence that representation-level similarity provides valuable signals for knowledge transfer [2-4].

Importantly, directly using parameter-level gradient information does not necessarily lead to better downstream performance. As shown in our response to W1, GRACE relies on student parameter gradients through additional backpropagation, making it 6.95× and 8.74× slower than SURE during the response selection stage, while also consistently underperforming SURE.

AuthorsOpenReview ↗

Official Comments by Authors (2/2)

To assess robustness to teacher composition, we further evaluate SURE on two controlled UltraFeedback subsets: one excluding non-Llama models and the other excluding Llama models. We compare SURE against the strongest baseline for each student (GRAPE and Max Token Length). SURE outperforms baselines in most of benchmark comparisons across the two subsets, which shows that its effectiveness is stable and robust in different teacher compositions:

TeacherLlama-3.2-3BAE2 WRAE2 LC WRIFEval StrictIFEval LooseMT-BenchMMLUMMLU-ProAbs. Δ
All ModelsSURE3.133.0922.5526.804.30055.7725.59
GRAPE1.591.4419.2222.004.03754.7424.532.29
w/o Non-llama ModelsSURE1.150.5619.0420.893.96255.4126.38
GRAPE0.650.2516.0818.483.86953.4624.621.55
w/o Llama ModelsSURE1.010.3217.0118.853.63854.8525.22
GRAPE0.390.1015.3418.302.88853.1722.802.09
TeacherQwen3-8B-BaseAE2 WRAE2 LC WRIFEval StrictIFEval LooseMT-BenchMMLUMMLU-ProAbs. Δ
All modelsSURE11.9916.2145.4749.916.74475.7143.09
Max Token Length11.7812.7435.6739.566.00674.5738.745.24
w/o Non-llama ModelsSURE6.996.0232.1636.046.26375.4944.12
Max Token Length6.585.3528.6533.275.70676.1642.002.05
w/o Llama ModelsSURE3.731.0234.9438.265.67576.8952.48
Max Token Length4.201.0929.3932.165.21976.8340.953.89

W3. Robustness is…

We thank the reviewer for this valuable suggestion and agree that multiple seeds and different datasets should be evaluated for robustness.

To validate our performance gains, we rerun student training and evaluation with three random seeds (42, 43, 44) following common experimental practice [5-6], and compare SURE against the strongest baseline for each student model (GRAPE and Max Token Length). The reported mean ± standard deviation results below show that SURE consistently outperforms both baselines across all metrics and remains robust across seeds.

Llama-3.2-3BAE2 WRAE2 LC WRIFEval StrictIFEval LooseMT-BenchMMLUMMLU-ProAbs. Δ
SURE3.18±0.663.17±0.6523.91±1.3027.91±1.114.327±0.02854.69±1.0724.99±0.62
GRAPE1.91±0.391.74±0.3618.79±0.7522.55±0.554.048±0.14754.06±0.7923.79±0.652.54
Qwen3-8B-BaseAE2 WRAE2 LC WRIFEval StrictIFEval LooseMT-BenchMMLUMMLU-ProAbs. Δ
SURE11.10±0.8614.80±1.3746.15±1.8550.65±1.446.598±0.14475.78±0.0743.36±0.24
Max Token Length11.08±1.0711.86±1.1632.47±3.2437.71±1.646.079±0.07974.84±0.4238.28±0.875.83

To further evaluate SURE’s effectiveness across different response pools, we conduct additional experiments on the verifiable MATH training set using a response pool generated by different teacher models [7-8]. We then evaluate the trained student models on the MATH 5K test set, where SURE achieves the highest accuracy on both student models, showing that its reliability and coherence signals generalize beyond the original UltraFeedback setting:

MethodLlama-3.2-3BQwen3-8B Base
SURE9.4273.10
Uniform Sampling8.7068.22
Max Token Length8.4872.36
GRAPE8.1071.46
RSR8.3456.70

Together with our multi-seed experiments, these results provide additional empirical evidence that SURE's effectiveness is robust across random seeds and generalizes to a different response pool with a different teacher composition.

[1] Panigrahi, A., et al. (2026). In Good GRACEs: Principled Teacher Selection for Knowledge Distillation. ICLR.

[2] Wang, L., Li, L., & Sun, X. (2022). Gradient Knowledge Distillation for Pre-trained Language Models. NeurIPS Workshop on Efficient Natural Language and Speech Processing.

[3] Tung, F., & Mori, G. (2019). Similarity-Preserving Knowledge Distillation. ICCV.

[4] Zhu, J., et al. (2021). Complementary Relation Contrastive Distillation. CVPR.

[5] Wu, M., et al. (2024). Advancing Parameter Efficiency in Fine-tuning via Representation Editing. ACL.

[6] Tong, T., et al. (2025). BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge. ICLR.

[7] Hendrycks, D., et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS.

[8] Somerstep, S., et al. (2025). CARROT: A Cost Aware Rate Optimal Router. ICLR Workshop on Foundation Models in the Wild.

Reviewer_npk6OpenReview ↗

reply to rebuttal

Thanks for the detailed rebuttal. I think most of my concerns are resolved. I will keep my positive vote towards this work.

Thank you for the feedback. We appreciate your time and thoughtful review of our work!

Reviewer_Gg3POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SURE, an unsupervised method for selecting teacher-generated responses for student fine-tuning. It computes loss gradients with respect to the frozen student’s final hidden states, centers the resulting representations by teacher, and measures coherence among responses to the same prompt. A latent-variable model combines this coherence with teacher reliability estimated across prompts to select one response per prompt. Experiments show improvements across four students and several downstream benchmarks, supported by component ablations, external quality analysis, and runtime measurements.

Critical strengths
  1. Consistent downstream improvements from unsupervised response selection. SURE improves performance across students from two model families, with particularly strong gains in instruction following. Improvements extend to rule-based IFEval and length-controlled AlpacaEval, supporting benefits beyond response length or a single evaluator’s preferences. The selected responses also receive higher external quality ratings that were not used for selection.

  2. Efficient use of teacher information across prompts. The method computes hidden-state gradients without backpropagating through the student and adds only modest curation overhead compared with GRAPE and RSR. Removing teacher reliability substantially reduces downstream performance, while removing teacher centering produces smaller but consistent losses.

Critical weaknesses
  1. The benefits of local selection and iterative estimation are not fully isolated. The reliability ablation establishes that teacher information improves over local coherence alone. It does not establish how much local coherence adds once reliable teachers have been identified. Similarly, the initialization already aggregates coherence into a teacher ranking, but its performance before iterative refinement is not reported. Consequently, the experiments leave unclear how much the full procedure improves over simpler teacher-ranking rules.

  2. The comparison with prior teacher-selection methods is incomplete. Section 2 understates the use of information across prompts in prior work. GRACE (Panigrahi et al.) scores teachers using distributional statistics of student gradients, and RSR (Yang et al.) also evaluates teacher selection using aggregated scores. However, the experiments omit GRACE and include only response-level RSR. SURE’s joint use of teacher reliability and prompt-specific coherence remains a distinct contribution, but its advantage over existing unsupervised teacher-ranking methods is not well established.

Questions for authors
  1. How does full SURE compare with selecting the highest-ranked available teacher using only the final reliability scores αk\alpha_k? How does it compare with the initial teacher ranking before iterative refinement?

  2. How does SURE compare with teacher rankings obtained from GRACE or aggregated RSR, selecting the highest-ranked available teacher for each prompt?

3 comments
AuthorsOpenReview ↗

Official Comments by Authors (1/2)

W1 & Q1: The benefits of … & How does full SURE…

We thank the reviewer for this helpful feedback. We first clarify that local coherence is integral to SURE’s iterative estimation of teacher reliability. At each iteration, response-selection probabilities (rr) combine local coherence with teacher reliability (α\alpha) (Eq. 6), and these probabilities are then aggregated across prompts to update (α\alpha) (Eq. 7). Thus, the two estimates are refined jointly, and the final reliability scores already incorporate local coherence.

We agree that selecting the highest-ranked available teacher using the final (α\alpha) is a reasonable alternative, since these scores already incorporate information from (rr) through iterative optimization. However, using the final (rr) offers a modest additional benefit by considering local evidence for each prompt. In some cases, a lower-ranked teacher’s response may receive stronger support from other teachers’ responses to the same prompt, allowing SURE to select it when this local support outweighs the reliability difference. Consequently, the difference between these selection rules reflects the incremental value of using (rr) for prompt-specific adaptation after joint estimation.

Following the reviewer’s suggestion, we compared full SURE with selecting the highest-ranked available teacher based solely on the final reliability scores (α\alpha). SURE performs better in six of eight model-metric comparisons. On the remaining two metrics, performance remains comparable. These results support a modest additional benefit from prompt-specific adaptation at selection time.

Llama3.2-3BAE2 WRAE2 LC WRMT-BenchMMLU
SURE3.133.094.30055.77
w/ Reliability Only2.842.854.27555.31
Qwen3-8B-BaseAE2 WRAE2 LC WRMT-BenchMMLU
SURE11.9916.216.74475.71
w/ Reliability Only11.6916.316.51975.96

Regarding the initial teacher ranking, the reported w/o Reliability Estimation ablation in Table 2 evaluates teacher selections based on coherence-based initialization without iterative refinement. Its substantially lower performance compared to SURE (e.g., AE2 LC WR: 7.41 vs. 16.21 on Qwen3-8B) indicates that initial coherence-based ranking alone is insufficient. Iterative estimation over the bipartite graph further aggregates cross-prompt evidence to refine teacher reliability, and further improves response selection beyond simple teacher-ranking rules.

AuthorsOpenReview ↗

Official Comments by Authors (2/2)

W2 & Q2: The comparison with… & How does SURE…

We thank the reviewer for highlighting the importance of comparing SURE with teacher-level ranking methods. Regarding baseline selection, both GRACE and teacher-level RSR produce global teacher rankings and assume that all teachers are available, which does not hold in UltraFeedback or most real-world settings due to incomplete teacher coverage [1]. Also, GRACE requires backpropagation during data selection, which introduces a substantial computational runtime compared with SURE and other forward-only baselines. Therefore, our original submission focused on response-level RSR. To address the concern, we additionally evaluate both teacher-ranking baselines by selecting the highest-ranked available teacher for each prompt. We fine-tune students with three random seeds (42, 43, 44) following common experimental practice [2-3] and report mean ± standard deviation:

Llama-3.2-3BAE2 WRAE2 LC WRIFEval StrictIFEval LooseMT-BenchMMLUMMLU-ProAbs. Δ
SURE3.18±0.663.17±0.6523.91±1.3027.91±1.114.327±0.02854.69±1.0724.99±0.62
GRACE1.68±0.921.45±0.8216.57±2.7220.27±3.183.881±0.14853.95±1.4624.83±0.483.37
Agg. RSR1.55±0.481.40±0.4316.02±2.7719.29±4.073.835±0.15254.33±0.9424.24±0.843.71
Qwen3-8B-BaseAE2 WRAE2 LC WRIFEval StrictIFEval LooseMT-BenchMMLUMMLU-ProAbs. Δ
SURE11.10±0.8614.80±1.3746.15±1.8550.65±1.446.598±0.14475.78±0.0743.36±0.24
GRACE4.62±0.555.44±0.7727.79±2.0432.53±1.035.429±0.08274.86±0.2338.78±0.979.93
Agg. RSR4.25±0.044.98±0.1026.37±3.1631.42±1.605.173±0.07974.64±0.2038.70±1.8310.82

SURE consistently outperforms both GRACE and aggregated RSR across all benchmark-student pairs. These results highlight the advantage of jointly modeling cross-prompt teacher reliability and prompt-specific gradient coherence. While GRACE and aggregated RSR capture teacher quality through global rankings, they cannot directly account for variations in teacher suitability across individual prompts. In addition, GRACE requires expensive gradient-based computations, making it 6.95× and 8.74× slower than SURE on Llama and Qwen students during the response selection stage. These results show that SURE achieves a favorable balance between selection effectiveness and computational efficiency.

[1] Panigrahi, A., et al. (2026). In Good GRACEs: Principled Teacher Selection for Knowledge Distillation. ICLR.

[2] Wu, M., et al. (2024). Advancing Parameter Efficiency in Fine-tuning via Representation Editing. ACL.

[3] Tong, T., et al. (2025). BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge. ICLR.

Thank authors for the detailed responses. I maintain my positive evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.