- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TrustMOPD, a label-free method for allocating multi-teacher on-policy distillation supervision at individual student-generated prefixes. Each specialist's centered log-probability displacement from its shared pre-RL reference is measured, divided by a frozen offline calibration mean, and converted into teacher weights through power normalization. Training minimizes a weighted reverse-KL objective, implemented through a geometric mixture of teacher distributions.
Critical strengths
-
TrustMOPD allocates specialist supervision without domain labels, outcome verifiers, or a separately trained router. On SINGLECAP it nearly matches label-routed MOPD, with overall scores of 30.04 versus 30.18 (Table 1), making it a useful option when routing labels are unavailable.
-
Measuring each specialist relative to its pre-RL reference is a useful way to expose specialization rather than relying only on teacher confidence. Offline calibration makes these signals comparable, and improvements persist with another teacher family and independently sourced prompts (Table 11).
Critical weaknesses
-
A teacher that changes more from the reference is not necessarily giving better advice. The score measures the size of that change, not whether the teacher is correct at the current prefix. The theory does not establish this connection, and the domain-level results mainly show that the method recognizes specialization (Section 3.4; Figure 3). Appendix E gives two useful examples, but a larger correctness check is needed to show when the score selects the better teacher.
-
The tested teacher pools are relatively structured, specialists share a reference and are trained for separate domains. Teacher scores are also normalized once using the initial student's rollouts (Sections 3.2 and 4.1). It is unclear how stable the weights remain when the calibration sample is unbalanced, teachers have overlapping skills, or the student's outputs change during training. The paper acknowledges the shared-reference requirement; the concern is how broadly the demonstrated routing gains apply.
Questions for authors
-
When teachers disagree, how often does the higher-weight teacher give the correct guidance? Please check more than the two Appendix E examples.
-
Do the weights and final scores remain stable with an unbalanced calibration sample or overlapping specialists? Do the fixed normalization constants still work on later student rollouts?
Dear Reviewers,
The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.
The discussion period ends on November 18.
Thanks,
Area Chair