- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MAD-OPD, an on-policy distillation framework that uses multiple teachers to debate and mitigate the biases and shared errors of a single teacher. The teachers debate at the states visited by the student, reach more reliable conclusions, and then provide token-level supervision for distillation. For multi-turn tool-use tasks, the paper further introduces a step-wise supervision mechanism, OPAD, which applies distillation at each tool-call decision. It also analyzes and compares different divergence objectives, including JSD and reverse KL, and assigns them to different task settings. Experiments show that the proposed method improves performance across multiple agentic and code-generation benchmarks.
Critical strengths
The paper is well motivated, as a single teacher may introduce systematic biases or share errors with the student, while multi-teacher debate can provide more reliable supervision. The experimental evaluation is also solid, covering multiple teacher–student configurations, model families, and agentic and code-generation benchmarks. Moreover, MAD-OPD achieves reasonably convincing and consistent improvements over the compared baselines.
Critical weaknesses
The baseline comparisons are not sufficiently comprehensive. It would be helpful to compare against stronger methods, such as reward-filtered SFT, rejection-sampling SFT, verifier-guided training, or multi-agent inference/distillation under the same computational budget; without these comparisons, the practical advantages of the proposed method remain unclear. In addition, each student rollout state requires multiple teachers to conduct several rounds of debate followed by force decoding. Although the paper reports performance improvements, it does not provide a detailed comparison with single-teacher OPD in terms of GPU hours, token usage, and wall-clock time, making it difficult to assess whether the gains justify the additional computational cost.
Questions for authors
Could the authors provide SFT results under the same multi-teacher debate setting? In particular, I am curious whether rejection-sampling SFT using the outputs selected or verified by multiple debating teachers could achieve better performance than the proposed OPD approach. More broadly, since the method mainly combines several existing components in an engineering-oriented pipeline, the algorithmic novelty appears limited. Could the authors further clarify whether this particular OPD formulation is truly necessary, and what advantages it offers over a simpler multi-teacher rejection-sampling SFT approach?
4 comments
Response to Reviewer WbTz (Part 2 / 2)
Q2. How do the training costs and gains compare with simpler baselines?
Each 100-update student job used eight H20 GPUs. R3-Table2 gives elapsed training time in each domain.
R3-Table2: Elapsed student training time on eight H20 GPUs
| Student training job | Tool use | Code |
|---|---|---|
| MAD-OPD (representative runs, B.2) | ~16 h | ~32 h |
| MAD-SFT | 20 h | 32 h |
MAD-SFT reused archived candidates. Rebuilding it from scratch would also require collecting the debate trajectories and running GPT-5.1 selection. The approximately 8,000 H20 GPU-hours in B.2 cover the whole project.
In MAD-OPD, Qwen3-14B and Qwen3-8B teach a Qwen3-4B student. OPD-Long and ST-Indep-4 train the same student model with Qwen3-14B alone, the stronger of the two teachers. OPD-Long runs ordinary OPD for the full time budget, so the student completes more updates. In ST-Indep-4, the 14B teacher draws four independent drafts at each student state. We concatenate the drafts in a fixed order. The teacher then scores the student's sampled continuation in one forward pass, and the student trains with the same distillation loss as MAD-OPD.
R3-Table3: Performance and teacher token use at matched training time with a Qwen3-4B student
| Domain (training time) | Method | Student updates | Generated tokens/state | Scored tokens/state | Domain avg. |
|---|---|---|---|---|---|
| Tool use (20 h) | OPD-Long | 205 | 0 | 4,464 | 23.86 |
| Tool use (20 h) | ST-Indep-4 | 136 | 5,133 | 6,432 | 24.60 |
| Tool use (20 h) | MAD-OPD | 117 | 4,742 | 6,174 | 25.78 |
| Code (30 h) | OPD-Long | 167 | 0 | 13,635 | 45.12 |
| Code (30 h) | ST-Indep-4 | 120 | 12,523 | 14,538 | 45.92 |
| Code (30 h) | MAD-OPD | 108 | 10,263 | 12,479 | 48.01 |
Generated counts teacher draft tokens. Scored counts tokens processed when the teacher scores the student's sampled continuation. Both are averages per student state. The MAD-OPD token counts are method-level averages shared with the main runs.
Four independent drafts improved on OPD-Long in both domains. Although the single 14B teacher generated and scored more tokens per state with four drafts than the debating pair did, MAD-OPD scored 1.18 points higher on tool use and 2.09 higher on code. Its gains over OPD-Long were 1.92 and 2.89 points, respectively. We added the four-sample results on all five benchmarks to Appendix C.1, Table 3, and the update and token counts to Table 4.
Our additional baseline covers rejection-sampling SFT. Verifier-guided training and test-time multi-agent inference remain separate comparisons.
We sincerely appreciate your suggestion to test SFT on the same debate outputs. It gave us a direct comparison between imitating the selected teacher answer and learning from feedback on the student’s own actions.
Thanks for your response
I will maintain my score.
Dear Reviewer WbTz,
Thank you for your time and positive feedback.
Response to Reviewer WbTz (Part 1 / 2)
Thank you for your valuable feedback to help us improve our paper. Following your suggestion, we have conducted additional experiments to further strengthen our work. We detail our response below.
Q1. What does on-policy distillation add over rejection-sampling SFT with the same debating teachers?
Thank you for asking how much the student gains from on-policy distillation after both teachers have debated. For both MAD-SFT and MAD-OPD, Qwen3-14B and Qwen3-8B provided the debate outputs and Qwen3-4B was the student.
For MAD-SFT we reused the teachers' archived second-round outputs. GPT-5.1 scored both candidates offline. We kept the highest-scoring eligible answer and dropped the example when neither qualified. On tool use, we retained 4,784 of 10,462 decision states and trained on the selected action at its recorded state. On code, we retained 4,291 of 11,143 problems and trained on the selected program. Each run used an effective batch size of 128, 100 optimizer steps and a learning rate of 1e-5. This corresponds to about 2.7 passes over the retained tool data and 3.0 over the code data.
At 100 updates, MAD-SFT scored 46.32 on code, above strong-teacher OPD (44.41). MAD-OPD reached 48.65 on code and 27.63 on tool use, compared with 46.32 and 21.59 for MAD-SFT. Both deploy the 4B student alone. MAD-SFT imitates the selected teacher answer. In MAD-OPD the teachers score the action the student took at each state or the program it sampled. Appendix C.11 gives the selection setup and all five benchmark scores (Table 12).