- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SURE for unsupervised multi-teacher distillation. SURE computes gradient signals at the final hidden states of a frozen student and combines within-prompt gradient coherence with teacher reliability estimated over a teacher-prompt bipartite graph. It selects one response from four candidates for each of 63,748 UltraFeedback prompts. The selected data are used to fine-tune four student models. Validation includes downstream benchmarks, component ablations, external GPT-4 quality ratings, teacher analyses, case studies, and runtime comparisons.
Critical strengths
-
The method combines prompt-level response evidence with cross-prompt teacher information in a computationally practical way. SURE derives gradients at the final hidden states without backpropagating through student parameters. It then uses a bipartite graph to jointly refine teacher reliability and response-selection probabilities. This directly addresses a limitation of purely likelihood-based or prompt-local selection.
-
The empirical results are broad and internally consistent. SURE improves the seven-metric average over every listed baseline for all four students. In the two reported ablation settings, removing reliability estimation reduces the average by
5.98%and1.87%. The paper also reports higher external quality ratings and modest runtime overhead relative to GRAPE and RSR.
Critical weaknesses
-
All downstream fine-tuning results are reported with seed 42. Full-parameter fine-tuning can have meaningful training variance. Without independent matched-seed repetitions, it is unclear whether SURE's average advantage over the strongest baseline is stable. This is the main limitation on my confidence in the empirical conclusion. Does SURE's seven-metric average advantage remain stable across at least 2 independent matched fine-tuning seeds for SURE and the strongest baseline for each student?
-
The estimated teacher reliability is not directly calibrated against independent correctness or human quality. The downstream results and GPT-4 ratings show practical value, but the score may partly reflect teacher consensus or student-relative preference. A direct teacher-level correlation analysis on verifiable tasks, together with high-consensus error cases, would clarify what the reliability quantity measures.
Questions for authors
Construct-validity clarification: How strongly does estimated teacher reliability correlate with verifiable teacher correctness, human quality judgments, or quality across independent judges?
Official Comments by Authors
W1. All downstream…
We thank the reviewer for highlighting the importance of training variance. To address this concern, we conduct independent matched-seed fine-tuning repetitions with three seeds (42, 43, 44) for SURE and the strongest baseline for each student (GRAPE and Max Token Length) following common experimental practice [1-2]. We report the mean ± standard deviation across seeds below:
SURE maintains higher mean performance across all metrics for both student models, with seven-metric average advantages (Abs. Δ) of 2.54% and 5.83% over GRAPE and Max Token Length. These matched-seed results support the stability and robustness of SURE's performance advantage.
W2 & Q. The estimated… & Construct-validity…
We thank the reviewer for raising this important question. We first clarify that teacher reliability in SURE is intended to measure how consistently a teacher provides useful supervision for a particular student, rather than the teacher’s standalone response correctness. This distinction is consistent with prior work showing that stronger teachers do not necessarily produce better students, as the effectiveness of distillation also depends on how well the provided supervision fits the student's existing capabilities [3-5]. Specifically, SURE computes student-derived gradients from teacher responses and aggregates their coherence across prompts, assigning higher reliability to teachers whose learning signals are consistently supported by those from other teachers. Accordingly, whether our estimated teacher reliability identifies teachers that provide useful learning signals is best assessed through actual student performance after distillation, rather than teacher response accuracy alone.
To investigate this distinction, we conduct additional experiments on the verifiable MATH training set with a response pool generated by different teacher models [6-7]. We directly evaluate the verified accuracy of the selected teacher responses:
Although SURE does not select the responses with the highest verified accuracy, student models trained on SURE-selected responses achieve the highest accuracy on the MATH 5K test set:
These results on selected-response accuracy and trained student performance reveal two key observations: (i) SURE achieves the highest student accuracy despite not selecting the most accurate teacher responses; (ii) RSR shows substantially lower student performance despite selecting responses with higher verified accuracy especially on Llama-3.2-3B student. These findings highlight that response correctness alone does not guarantee effective student learning, and support SURE's use of student-relative reliability and gradient coherence to select more effective training data.
[1] Wu, M., et al. (2024). Advancing Parameter Efficiency in Fine-tuning via Representation Editing. ACL.
[2] Tong, T., et al. (2025). BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge. ICLR.
[3] Xu, Z., et al. (2025). Stronger Models Are Not Always Stronger Teachers for Instruction Tuning. NAACL.
[4] Zhu, Y., et al. (2022). Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation. NeurIPS.
[5] Fan, W., et al. (2024). Revisit the Essence of Distilling Knowledge through Calibration. ICML.
[6] Hendrycks, D., et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS.
[7] Somerstep, S., et al. (2025). CARROT: A Cost Aware Rate Optimal Router. ICLR.