- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 3(+2)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CaOPD, which aims to mitigate the miscalibration introduced by OPSD training. CaOPD decouples capability learning from confidence supervision by replacing the confidence signal with a student-grounded estimate of “true” confidence, computed from empirical Monte Carlo rollouts. This revised confidence target is then distilled alongside the original capability objective, allowing the model to preserve task performance while improving calibration. This work applies 4 baselines and two datasets to validate the effectiveness of the proposed CaOPD, along with a series solid ablations and analysis of the proposed algorithm.
Critical strengths
The problem discussed is common and important for recent widely-used OPSD.
The proposed CaOPD framework is simple and well motivated, and its effectiveness is supported by both theoretical analysis and extensive empirical results across models, domains, and training settings.
Critical weaknesses
The paper appears to overgeneralize a failure mode specific to privileged-context OPSD as a general limitation of OPD.
The theoretical analysis relies critically on the teacher having additional privileged information unavailable to the student, which is not a defining property of OPD itself. For example, OPD can be used as strong-to-weak models, or Multi-teacher OPD, the supervision signal can from multiple sources while they dont see additional information.
As a result, claims that OPD inherently induces overconfidence seem broader than what the theory and experiments directly support.
Questions for authors
whether the identified miscalibration is intended to apply to OPD broadly or specifically to privileged-context OPSD methods such as SDPO/SDFT? If it is for OPD, I believe another set of experiments which using stronger model as teacher should be discussed.
3 comments
Thank you for the reply and the latest editing, my main concern about the scope of OPD vs. OPSD is addressed. I will raise my confidence and informative score accrodingly.
thanks for your response
Thank you for reading our response and the revision, and for raising your confidence and informativeness scores. We are glad that the OPD vs. OPSD scope concern is resolved. We will keep this scope throughout the final version, including changing the title to "... in On-Policy Self-Distillation". Thank you again for your constructive review.
Response to Reviewer AbWc
Thank you for the positive assessment and for the precise question; it identified the main weakness in our framing. Changes in the revised PDF are in blue.
Q: Does the miscalibration apply to OPD broadly, or to privileged-context OPSD such as SDPO/SDFT? Our claims and experiments concern privileged-context OPSD. We agree that the original framing overgeneralized, and we have revised the abstract, Sec. 1, 2.1 and 3 to state the scope as OPSD and to define OPD and OPSD explicitly. The title will become "... in On-Policy Self-Distillation" in the final version (title changes are not permitted during the discussion period).
What carries over to OPD with a stronger teacher or multiple teachers. Following your question, a new paragraph in Sec. 3.2 states what each result requires:
A stronger teacher could instead judge the student's own trajectory (acting partly as a verifier), so the size of this effect is an empirical question. We therefore state the strong-teacher case as an analytical extension, together with the corresponding limitation, rather than claim it empirically; testing it is a natural next step.
We hope the clarified scope addresses your concern. If so, we would be grateful if you would consider raising your score. Thank you again for the constructive review.