acceptodds
Under review as a conference paper at ICLR 2027

MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate

17 views

Abstract

On-policy distillation (OPD) trains a student on its own trajectories with token-level teacher supervision, but existing methods face a single-teacher capability ceiling: when the teacher and student make the same mistake, the teacher can reinforce it. This problem is sharper in agentic tasks, where per-step errors compound across long trajectories. We introduce MAD-OPD (Multi-Agent Debate-driven On-Policy Distillation), which uses debate among teachers to improve supervision at states visited by the student. The teachers use the debate transcript to evaluate the student's tokens and report confidence scores that determine their loss weights. For agentic tool use, we introduce On-Policy Agentic Distillation (OPAD), which performs step-wise OPD by sampling each tool action from the student and applying token-level supervision at that decision state. Our analysis of gradients and teacher aggregation motivates Jensen–Shannon divergence for agentic tasks and reverse Kullback–Leibler divergence for code generation. The divergence ablation supports these choices in the tested configuration. Across six teacher–student configurations spanning Qwen3 and Qwen3.5, 1.7B–14B students, and 8B–32B teachers on five agentic and code benchmarks, MAD-OPD ranks first in overall average. On Qwen3 14B+8B→4B, it improves the agentic average by 4.39 points and the code average by 4.24 points over the stronger single-teacher OPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, multi-agent debate, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WbTzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAD-OPD, an on-policy distillation framework that uses multiple teachers to debate and mitigate the biases and shared errors of a single teacher. The teachers debate at the states visited by the student, reach more reliable conclusions, and then provide token-level supervision for distillation. For multi-turn tool-use tasks, the paper further introduces a step-wise supervision mechanism, OPAD, which applies distillation at each tool-call decision. It also analyzes and compares different divergence objectives, including JSD and reverse KL, and assigns them to different task settings. Experiments show that the proposed method improves performance across multiple agentic and code-generation benchmarks.

Critical strengths

The paper is well motivated, as a single teacher may introduce systematic biases or share errors with the student, while multi-teacher debate can provide more reliable supervision. The experimental evaluation is also solid, covering multiple teacher–student configurations, model families, and agentic and code-generation benchmarks. Moreover, MAD-OPD achieves reasonably convincing and consistent improvements over the compared baselines.

Critical weaknesses

The baseline comparisons are not sufficiently comprehensive. It would be helpful to compare against stronger methods, such as reward-filtered SFT, rejection-sampling SFT, verifier-guided training, or multi-agent inference/distillation under the same computational budget; without these comparisons, the practical advantages of the proposed method remain unclear. In addition, each student rollout state requires multiple teachers to conduct several rounds of debate followed by force decoding. Although the paper reports performance improvements, it does not provide a detailed comparison with single-teacher OPD in terms of GPU hours, token usage, and wall-clock time, making it difficult to assess whether the gains justify the additional computational cost.

Questions for authors

Could the authors provide SFT results under the same multi-teacher debate setting? In particular, I am curious whether rejection-sampling SFT using the outputs selected or verified by multiple debating teachers could achieve better performance than the proposed OPD approach. More broadly, since the method mainly combines several existing components in an engineering-oriented pipeline, the algorithmic novelty appears limited. Could the authors further clarify whether this particular OPD formulation is truly necessary, and what advantages it offers over a simpler multi-teacher rejection-sampling SFT approach?

4 comments
AuthorsOpenReview ↗

Response to Reviewer WbTz (Part 1 / 2)

Thank you for your valuable feedback to help us improve our paper. Following your suggestion, we have conducted additional experiments to further strengthen our work. We detail our response below.

Q1. What does on-policy distillation add over rejection-sampling SFT with the same debating teachers?

Thank you for asking how much the student gains from on-policy distillation after both teachers have debated. For both MAD-SFT and MAD-OPD, Qwen3-14B and Qwen3-8B provided the debate outputs and Qwen3-4B was the student.

For MAD-SFT we reused the teachers' archived second-round outputs. GPT-5.1 scored both candidates offline. We kept the highest-scoring eligible answer and dropped the example when neither qualified. On tool use, we retained 4,784 of 10,462 decision states and trained on the selected action at its recorded state. On code, we retained 4,291 of 11,143 problems and trained on the selected program. Each run used an effective batch size of 128, 100 optimizer steps and a learning rate of 1e-5. This corresponds to about 2.7 passes over the retained tool data and 3.0 over the code data.

R3-Table1: MAD-SFT and MAD-OPD on five benchmarks after 100 updates

Method, 100 updatesBFCL-v4τ²VitaLCB-v6MBPP+Tool avg.Code avg.
MAD-SFT22.4526.3316.0027.4265.2121.5946.32
MAD-OPD28.3933.7420.7529.8767.4227.6348.65

At 100 updates, MAD-SFT scored 46.32 on code, above strong-teacher OPD (44.41). MAD-OPD reached 48.65 on code and 27.63 on tool use, compared with 46.32 and 21.59 for MAD-SFT. Both deploy the 4B student alone. MAD-SFT imitates the selected teacher answer. In MAD-OPD the teachers score the action the student took at each state or the program it sampled. Appendix C.11 gives the selection setup and all five benchmark scores (Table 12).

AuthorsOpenReview ↗

Response to Reviewer WbTz (Part 2 / 2)

Q2. How do the training costs and gains compare with simpler baselines?

Each 100-update student job used eight H20 GPUs. R3-Table2 gives elapsed training time in each domain.

R3-Table2: Elapsed student training time on eight H20 GPUs

Student training jobTool useCode
MAD-OPD (representative runs, B.2)~16 h~32 h
MAD-SFT20 h32 h

MAD-SFT reused archived candidates. Rebuilding it from scratch would also require collecting the debate trajectories and running GPT-5.1 selection. The approximately 8,000 H20 GPU-hours in B.2 cover the whole project.

In MAD-OPD, Qwen3-14B and Qwen3-8B teach a Qwen3-4B student. OPD-Long and ST-Indep-4 train the same student model with Qwen3-14B alone, the stronger of the two teachers. OPD-Long runs ordinary OPD for the full time budget, so the student completes more updates. In ST-Indep-4, the 14B teacher draws four independent drafts at each student state. We concatenate the drafts in a fixed order. The teacher then scores the student's sampled continuation in one forward pass, and the student trains with the same distillation loss as MAD-OPD.

R3-Table3: Performance and teacher token use at matched training time with a Qwen3-4B student

Domain (training time)MethodStudent updatesGenerated tokens/stateScored tokens/stateDomain avg.
Tool use (20 h)OPD-Long20504,46423.86
Tool use (20 h)ST-Indep-41365,1336,43224.60
Tool use (20 h)MAD-OPD1174,7426,17425.78
Code (30 h)OPD-Long167013,63545.12
Code (30 h)ST-Indep-412012,52314,53845.92
Code (30 h)MAD-OPD10810,26312,47948.01

Generated counts teacher draft tokens. Scored counts tokens processed when the teacher scores the student's sampled continuation. Both are averages per student state. The MAD-OPD token counts are method-level averages shared with the main runs.

Four independent drafts improved on OPD-Long in both domains. Although the single 14B teacher generated and scored more tokens per state with four drafts than the debating pair did, MAD-OPD scored 1.18 points higher on tool use and 2.09 higher on code. Its gains over OPD-Long were 1.92 and 2.89 points, respectively. We added the four-sample results on all five benchmarks to Appendix C.1, Table 3, and the update and token counts to Table 4.

Our additional baseline covers rejection-sampling SFT. Verifier-guided training and test-time multi-agent inference remain separate comparisons.

We sincerely appreciate your suggestion to test SFT on the same debate outputs. It gave us a direct comparison between imitating the selected teacher answer and learning from feedback on the student’s own actions.

Reviewer_WbTzOpenReview ↗

Thanks for your response

I will maintain my score.

Dear Reviewer WbTz,

Thank you for your time and positive feedback.

Reviewer_MkStOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agentic on-policy distillation trains a student agent on its own trajectories while teachers score every token. The paper argues that the standard diagnosis — compounding error, i.e. the student drifts outside the region where teachers are reliable — is incomplete. From the states where the student deviates most, teacher takeovers still complete the task, yet teachers scoring the student's token in isolation approve premature or wrong tool calls, because the scorer cannot see that the episode is about to end. The authors call this the rollout–scorer gap and attribute it to the myopia of next-token scoring.

Their remedy, MAD-OPD, repairs the scorer before distilling. Two teachers debate for R=2 rounds to produce a transcript H; each teacher then force-decodes the student's sampled tokens conditioned on H; the per-token distributions are aggregated across teachers with confidence-derived weights; the student is trained on the aggregated token distribution (JSD by default). OPAD embeds this in a per-decision-step normalization over an environment-interleaved rollout. Validation: 6 configurations (Qwen3 students from 1.7B to 14B, teachers 8B–32B, plus one Qwen3.5 teacher pair), 5 benchmarks (BFCL-v4, τ²-Bench, VitaBench, LiveCodeBench-v6, MBPP+), 100 optimizer updates per run, roughly 8k H20 GPU-hours. MAD-OPD is reported best in all 6 configurations, with +4.39 agentic and +4.24 code over the stronger-teacher OPD baseline, and the 4B student is reported to exceed its 14B teacher on LCB-v6. The stated contributions are: (1) the rollout-scorer gap diagnosis, (2) MAD-OPD, which repairs the teacher's scoring before distilling, and (3) OPAD, the step-wise scheme that applies both to an environment-interleaved rollout.

Critical strengths
  1. The problem diagnosis is sharp and testable, and the control ladder actually tests it. The paper does not merely assert that teachers are myopic; Table 3 (p.17) supplies exactly the controls a sceptic would ask for — peer visibility removed (MT-NoComm-2R), extra teacher compute redistributed to the strong teacher (ST-SelfRefine-2x2, four 14B generations; OPD-Long at the same 20 h wall-clock), and a longer teacher-only privileged trace (OPD-Teacher-CoT at 1.34×/1.65× the privileged tokens) — and all four single-teacher/extra-compute controls land in a 0.30-wide band (23.86–24.16), well below MAD-OPD's 25.78. This is why my recommendation is not a rejection: the paper runs the control that could have falsified its own headline (equal wall-clock, Table 4) and reports it. That is what makes the reframing plausible rather than rhetorical, and it is rare enough to be worth crediting.

  2. The empirical base is broad and directionally consistent, and the reported arithmetic is faithful. Six configurations, five benchmarks, 18/18 agentic and 12/12 code cells favour MAD-OPD over OPD, and the abstract's +4.39/+4.24 reproduce exactly from Table 1 (27.63−23.24, 48.65−44.41), with the baseline set to the stronger teacher of each pair as claimed. The code numbers carry 16-seed standard deviations throughout. Whatever one concludes about the novelty framing, this is a real system run at real scale (six teacher–student–environment cells, ~8k H20 hours), not a two-run demo — and I found no number I could not reproduce from the tables.

Critical weaknesses
  1. The headline row is a composite of two different checkpoints, and the paper's own divergence ablation shows it. Agentic and code checkpoints are trained separately on different corpora (§B.3; "Code checkpoints are trained separately", §C.1). Table 7 (p.18) then reports, for the same configuration, agentic 27.63 / code 44.71 under JSD and agentic 22.21 / code 48.65 under reverse KL. The Table 1 "MAD-OPD" row for 14B+8B→4B is Ag-Avg 27.63 with Co-Avg 48.65 — the two figures that Table 7 reports for the JSD run on agentic and for the reverse-KL run on code respectively. If that reading is correct, the headline row combines two checkpoints trained with different divergences. That pooled row then drives Table 6 and the "+1.78/+2.05 from confidence weighting" attribution. So the method as reported does not have a single hyperparameter setting: each domain reports its own best divergence, and both choices were made on the same five test benchmarks that subsequently score them (there is no validation split anywhere). The honest single-divergence overalls are 34.46 (JSD) and 32.78 (Rev KL), not 36.03. Why it is consequential: the divergence ablation is simultaneously the evidence for JSD and the reason JSD is not used for half the headline, so RQ4 and the §C.4 attribution do not survive as stated. This is fixable by reporting the single-divergence numbers and stating the selection rule — but not by the current framing.

  2. The central quantitative claim is equal-update rather than compute-matched, the paper's own matched control shrinks it by more than half, and the same configuration varies between tables by nearly that much. MAD-OPD spends K·R=4 teacher generations (≤4,096 new tokens each) plus 2 privileged force-decodes per decision step; OPD spends one force-decode and no teacher generation. Table 1 equalises updates, and the equal-time control shows what that costs: one MAD-OPD update takes ≈1.75× an OPD update (205 vs 117 updates in 20 h). Table 4 (p.17, equal 20 h) therefore gives the matched number — 23.86 (OPD, 205 updates) → 25.78 (MAD-OPD, 117 updates), i.e. +1.92, not +4.39. There is no compute-matched control at all for the code half ("Code checkpoints are trained separately"), so +4.24 is entirely unmatched. Compounding this, the same configuration and method appear as 27.63 in Table 1 and 25.78 in Table 3 — a 1.85-point spread, never explained, arising (per §B.2) from more updates. Since every ablation is a single run and the only measurable run-to-run spread is 1.85, none of the component attributions (+1.78 for confidence weighting, +3.11 for peer visibility) is resolvable against it. The one knob that is swept behaves sharply non-monotonically: R=2 gives 27.63, R=3 collapses to 23.40 — below the no-debate MT-OPD control (24.15) and at single-teacher OPD's level (23.24) — with no variance to say whether R=2 is a plateau or a lucky point, and the paper disposes of this in one clause ("favoring a single exchange of teacher feedback in this setting", §5.4). Why it is consequential: if the matched and unmatched numbers differ by 2.5×, the headline sentence in the abstract and TL;DR is not supported at the stated magnitude by the paper's own protocol.

Questions for authors

The Reproducibility Statement promises only appendix-level detail; are code and checkpoints released? Concretely: the "supervised-token mask" is referred to but never defined, no numerical floor is given for the reverse-KL term over a ~150k-token vocabulary in BF16, and the force-decode target distribution (raw T=1 softmax vs. the T=0.7/top-p 0.8 sampler used for generation) is unstated. Also, τ²-Bench was run with a fork containing "75+ airline and retail task fixes" and a Claude-opus-4-6 user simulator, and the training environment uses Claude-opus-4-6 to simulate both tool responses and user replies (§B.4). Do the reported levels survive the unmodified official harness, and does the advantage persist under a different model simulator?

4 comments
AuthorsOpenReview ↗

Response to Reviewer MkSt (Part 1 / 2)

Thank you for checking the training budgets and asking how we chose the loss in each domain. Your questions prompted us to add the code training times and update counts to Appendix C.1 and to report all five benchmarks for each training seed in Appendix C.4.

Q1. How do training time and run-to-run variation affect the reported gains?

As you pointed out, the gain depends on the training budget. In R2-Table1, MAD-OPD trains Qwen3-4B with Qwen3-14B and Qwen3-8B. The single-teacher baseline trains Qwen3-4B with the stronger Qwen3-14B teacher.

R2-Table1: MAD-OPD gains over strong-teacher OPD at equal updates and equal time

Comparison with strong-teacher OPDTool-use gainCode gain
Equal updates (100 per run, Table 1)+4.39+4.24
Equal time (20 h tool use, 30 h code, Tables 3–4)+1.92+2.89

At 30 hours on code, MAD-OPD reached 48.01 after 108 updates. OPD-Long reached 45.12 after 167.

The two agentic scores you identified, 27.63 and 25.78, come from 100- and 117-update runs. Their difference includes a change in training duration. Also, the 3.11-point peer-visibility gain in Table 3 is the five-benchmark average. The corresponding tool-use gain is 1.69. As you observed, tool use falls from 27.63 at two rounds to 23.40 at three. Two rounds gave the highest score in this sweep. Across three training seeds for full MAD-OPD, the tool average was 27.49 ± 0.12 and the code average was 48.54 ± 0.09 (mean ± sample SD). The 16 code decoding seeds in B.7 measure evaluation variability.

Q2. How were the domain-specific losses chosen, and what happens with a single divergence?

Sections 3.2–3.3 motivated the losses we tested. We derive bounds on JSD and its logit gradients (Appendix D.2) and show how reverse KL combines the teachers’ token distributions through a weighted geometric mean (Appendix D.4). We then trained with forward KL, JSD and reverse KL in both domains and tracked the loss over 100 updates. JSD gave a smoother loss curve on tool use than reverse KL, while reverse KL converged smoothly on code (Appendix C.8, Figure 7). We used these observations together with the results on the five reported benchmarks to choose JSD for tool use and reverse KL for code (§5.4 and Appendix C.5, Table 9). We used no separate validation set for this choice. As you noted, Table 1 combines the two domain-specific checkpoints.

R2-Table2: Tool, code, and overall scores by distillation loss

Divergence choiceTool avg.Code avg.Five-benchmark avg.
JSD throughout27.6344.7134.46
Reverse KL throughout22.2148.6532.78
JSD for tools, reverse KL for code27.6348.6536.03

Within each domain, Table 6 keeps the divergence fixed when comparing uniform and confidence weighting.

AuthorsOpenReview ↗

Response to Reviewer MkSt (Part 2 / 2)

Q3. Which training and evaluation settings are needed to reproduce these results?

We appreciate the chance to clarify the setup. For the Qwen3-4B results in our first reply, Qwen3-14B and Qwen3-8B provided the MAD-OPD supervision. Figure 1 and Appendix E illustrate a student and teacher making a shared error. The student takes each action in our experiments, including at states where the teacher disagrees.

Both domains use full-vocabulary BF16 teacher logits. Top-k 20, top-p 0.8 and temperature 0.7 apply to student sampling (Section 4, Table 2 and B.7).

The MAD loss applies log-softmax to aligned student and teacher logits at temperature 1. For reverse KL, the kernel computes KL(student || teacher) with log targets and an FP32 accumulator. It sets negative values of the mean divergence to zero after reduction and applies no per-token probability floor (Appendix B.6).

ToolACE trajectories are split into examples at assistant decisions. Prior tool and user replies supply context. The trainer aligns the sampled response with each teacher's forced-decoding positions and scores the covered response span, including invalid student actions (B.4). The MAD kernel receives this span without a second special-token mask. The main recipe disables thinking.

During ToolACE training, Claude simulates both tool and user responses (B.4). Both τ²-Bench and VitaBench let users configure the simulator separately from the agent. On these benchmarks, Claude Opus 4.6 played the user across all methods, and each benchmark scored task success. To keep API costs manageable, we used one simulator for the full comparison. The reported gains are for that simulator and the harness versions described in B.10–B.11.

We hope the revised tables and the implementation details above answer your questions. If a setting is still unclear, we would be glad to clarify it.

Dear Reviewer MkSt,

Thank you again for your thoughtful review. With two days remaining in the discussion period, we wanted to follow up on our rebuttal. We ran the requested equal-time comparison, MAD-OPD remains ahead in each case. Meanwhile, we clarified the setup about training and evaluation.

Could you please let us know whether these results address your two main concerns? If anything remains unclear, we would be glad to respond before the discussion closes.

Best regards,

The Authors

Dear Reviewer MkSt,

We hope this message finds you well. As the discussion period is nearing its end, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Best regards,

The Authors

Reviewer_phbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MAD-OPD, which integrates multi-teacher debate into on-policy distillation. Teachers exchange and refine their answers, leveraging the debate transcripts to provide token-level supervision on student-generated trajectories while weighting the distillation loss by self-reported confidence. The authors further extend this method to multi-step tool interactions and report performance gains over single-teacher and naive multi-teacher distillation baselines on tool-use and code generation tasks.

Critical strengths

The paper is well-written and well-organized. The motivation, methodological pipeline, and training objectives are clearly presented and easy to follow, and the appendix provides detailed experimental setups and supplementary analyses.

Critical weaknesses

Weakness 1

While the main experiments align the number of student training updates, MAD-OPD employs two teachers with four total teacher generations per decision state, requiring both teachers to compute token distributions over long contexts containing full debate transcripts. In contrast, single-teacher OPD operates with a substantially smaller teacher-side compute budget. Consequently, the main results demonstrate that scaling teacher-side compute improves distillation, but do not sufficiently establish that MAD-OPD outperforms single-teacher approaches under an equal teacher supervision budget.

Although the appendix includes valuable controls—such as matched wall-clock training time, single-teacher self-correction, disabled inter-teacher communication, and extended teacher reasoning traces—that partially mitigate this concern, these experiments only isolate wall-clock time, query counts, or partial reasoning lengths individually. They do not clearly demonstrate comparability under a unified total teacher token budget. In particular, both the number of teacher-generated tokens and the context length processed during force-decoding dictate the actual computational cost; merely aligning the number of teacher calls is insufficient. Even though the longer teacher reasoning baseline uses more privileged tokens, this does not directly guarantee that its overall compute budget matches that of MAD-OPD.

To address this, I recommend a more direct compute-matched baseline: allowing a strong single teacher to perform multiple independent samplings under the exact same teacher token budget as MAD-OPD, and supervising the student using the resulting candidate responses via the identical context construction and distillation pipeline. Distinct from the existing single-teacher self-correction baseline, this setup directly tests whether multi-teacher debate surpasses compute-matched single-teacher diverse sampling. Furthermore, the evaluation should separately report the number of teacher-generated tokens, the tokens processed for distribution computation (force-decoding), and actual GPU hours, while comparing performance across different budget scales. Without such a control, it remains difficult to fairly disentangle whether the gains of MAD-OPD stem from effective multi-teacher interaction or simply from increased teacher-side compute.

Weakness 2

The proposed method assumes that teachers can correct each other's errors through debate and that post-debate self-reported confidence accurately reflects the reliability of each teacher's supervision signal. However, disagreements between teachers may arise not only from errors, but also from distinct yet equally valid solution paths. Especially in code generation, different algorithms and implementations can yield substantially divergent yet equally valid token distributions. Debate does not necessarily resolve such conflicts effectively; instead, it may cause a correct teacher to capitulate to an erroneous peer, or lead multiple teachers into a more confident yet incorrect consensus. The paper currently lacks a systematic analysis of these failure modes, leaving it unclear under what conditions multi-teacher communication reliably improves supervision quality.

Moreover, confidence weighting relies on an unverified assumption: that a teacher's global confidence in its own final answer accurately reflects the token-level accuracy of its supervision conditioned on the student's actual rollout prefixes. These two quantities are not equivalent, and different teacher models may exhibit distinct scoring calibrations and biases. Directly applying uncalibrated scores for weighting risks favoring overconfident teachers rather than genuinely reliable ones.

Furthermore, the weight transformation severely compresses differences between teachers. By dividing scores ci∈[0,100]c_i \in [0, 100] by 100100 before applying a softmax with temperature τ=1\tau = 1:

wi=exp⁡(ci/100)∑jexp⁡(cj/100)w_i = \frac{\exp(c_i / 100)}{\sum_{j} \exp(c_j / 100)}

scores of c1=95c_1 = 95 and c2=90c_2 = 90 yield weights of:

w1≈0.513,w2≈0.487w_1 \approx 0.513, \quad w_2 \approx 0.487

which is virtually indistinguishable from uniform weighting (0.5000.500). Despite this minimal differentiation, Table 6 reports average gains of 1.781.78 percentage points on tool-use tasks and 2.052.05 percentage points on code generation from confidence weighting. While this does not invalidate the results, it necessitates rigorously ruling out training variance. The paper does not specify whether this ablation is evaluated across multiple independent training seeds; notably, the 1616 random seeds in code generation only account for evaluation-time decoding variance, failing to capture training stochasticity arising from student rollouts, teacher debate sampling, and optimization dynamics. Thus, it remains unclear whether the gains in Table 6 can be reliably attributed to confidence weighting rather than run-to-run training variance.

To substantiate these claims, the authors should conduct paired comparisons between the full method and the uniform-weighting variant across multiple independent training seeds, reporting the mean improvements, standard deviations, and confidence intervals. Additionally, holding the debate transcripts and teacher distributions fixed, the authors should compare self-reported confidence against fixed weights and shuffled confidence scores, while reporting the empirical distribution of the resulting weights and the calibration between confidence and accuracy. These experiments would directly verify whether the mechanism genuinely exploits teacher reliability signals and whether the gains in Table 6 are statistically reproducible.

Questions for authors

See Critical Weaknesses for details.

6 comments
AuthorsOpenReview ↗

Response to Reviewer phbi (Part 1 / 3)

Thank you for your constructive comments and suggestions. We have revised our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.

Q1. Does debate outperform four independent samples from the stronger teacher at a comparable training budget?

Thank you for suggesting this control. In MAD-OPD, Qwen3-14B and Qwen3-8B teach a Qwen3-4B student. OPD-Long and ST-Indep-4 train the same student model with Qwen3-14B alone, the stronger of the two teachers. For ST-Indep-4, the 14B teacher generated four drafts independently at each student state. We concatenated them in a fixed order, scored the student continuation in one teacher pass and used the same distillation loss as MAD-OPD.

R1-Table1: Tool and code scores at matched training time

MethodTool score (20 h)UpdatesCode score (30 h)Updates
Strong-teacher OPD-Long23.8620545.12167
Strong teacher, four independent samples24.6013645.92120
MAD-OPD, 14B + 8B25.7811748.01108

Four independent samples improved on OPD-Long in both domains. At the same training time, MAD-OPD scored 1.18 points higher on tool use and 2.09 higher on code than the four-sample baseline.

We counted teacher draft tokens under “Generated” and tokens processed during the teacher forward pass on the fixed student continuation under “Scored.” Both are averages per student state. We use the same method-level averages for the main and equal-time MAD-OPD runs.

R1-Table2: Teacher tokens generated and scored per student state

Domain and methodGenerated tokensScored tokens (forced decoding)
Tool use, ST-Indep-45,1336,432
Tool use, MAD-OPD4,7426,174
Code, ST-Indep-412,52314,538
Code, MAD-OPD10,26312,479

The independent teacher used more tokens per state in both domains. The matched budget here is elapsed training time at one budget per domain. Matching cumulative teacher tokens or teacher GPU-hours would require a separate control. Paper Table 3 tests peer visibility directly. With the same two teachers and two rounds, sharing drafts raised the code average from 42.78 to 48.01. We have added ST-Indep-4 and its costs to Appendix C.1 (Tables 3–4).

AuthorsOpenReview ↗

Response to Reviewer phbi (Part 2 / 3)

Q2. With weights so close to 50:50, does confidence reliably improve supervision?

You are right about the example. The two teachers receive a 51.25:48.75 split (original Table 10, revised Table 14). We checked how often the more confident of the Qwen3-14B and Qwen3-8B teachers was correct, then compared confidence and uniform weights when distilling the pair into Qwen3-4B.

For the teacher-selection rows in R1-Table3, we used cases with exactly one correct final answer and gave half credit for tied confidence scores.

R1-Table3: Confidence weights and teacher accuracy after debate

DiagnosticTool useCode
States with valid weights9699
Median absolute departure from equal weighting (pp)0.870.75
90th percentile of absolute departure (pp)1.871.30
Exactly one correct final answer3034
Accuracy selecting 14B on those cases53.33%55.88%
Accuracy selecting by confidence76.67%70.59%
Accuracy at confidence ≥90, 14B43/4646/50
Accuracy at confidence ≥90, 8B34/3736/40

Across the twelve teacher/domain bins in revised Table 8, observed accuracy was within two percentage points of mean reported confidence. These labels describe the teachers' own answers. During training, each teacher scores the student continuation, and its confidence sets the weight on its loss.

Paper Table 6 compares uniform and confidence weights with the same debate protocol. Confidence weighting improved all five benchmarks in the main run (R1-Table4). The fixed-cache comparison with shuffled confidence remains open.

R1-Table4: Five-benchmark gains from confidence weighting over uniform weighting

Gain over uniform debateBFCL-v4τ²VitaLCB-v6MBPP+
MAD-OPD, Table 1 run+0.84+2.49+2.00+2.79+1.30

R1-Table5: Domain scores across three MAD-OPD training runs

Method and training runTool avg.Code avg.
Strong-teacher OPD23.2444.41
Uniform-weight debate25.8546.60
MAD-OPD, Table 127.6348.65
MAD-OPD, repeat 127.4448.50
MAD-OPD, repeat 227.4048.48
MAD-OPD, three-run mean ± sample SD27.49 ± 0.1248.54 ± 0.09

Across two additional MAD-OPD training runs, the gains over the uniform baseline were 1.55–1.59 points on tool use and 1.88–1.90 on code. Revised Table 7 gives the five benchmark scores for each run.

AuthorsOpenReview ↗

Response to Reviewer phbi (Part 3 / 3)

Q3. Does sharing drafts correct errors while preserving different valid solutions?

A teacher can give up a correct solution after seeing its peer’s draft. We tested this directly on drafts from the Qwen3-14B and Qwen3-8B teachers. Each teacher revised its original program independently in one condition and with access to its peer's draft in the other. We ran both revisions against the same tests (Appendix C.12).

R1-Table6: Code revisions on 50 problems with one correct draft and 50 with two passing algorithms

OutcomeIndependent revisionShared revision
Initially one correct: wrong answer became correct17/5030/50
Same 50 problems: correct answer became wrong5/506/50
Initially two correct, different algorithms: both still passed46/5045/50
Same 50 problems: both still passed with different algorithms44/5043/50

Seeing the peer's draft produced 13 additional corrections and one additional regression on the first set. Of the six regressions after sharing, three matched the peer's original error. On the second set, 43 of 50 pairs retained two distinct passing algorithms after sharing. Separately, in the naturally sampled code set, both teachers reported confidence of at least 90 and made the same identifiable error on 0/99 evaluable problems after sharing (1/98 after independent revision).

The two teachers scored the same programs sampled from the Qwen3-4B student under both revision conditions on 36 problems, each with two passing approaches and a failing implementation. We averaged NLL over the code tokens in each program. For each row, we then averaged the two teachers equally within each problem and gave all 36 problems equal weight. Each teacher's initial algorithm defines its own approach.

R1-Table7: Token NLL on fixed student programs for 36 code problems

Student implementationIndependent NLLShared NLLChange (nats/token)
Passing, teacher's own approach0.580.57−0.01
Passing, alternative approach0.830.75−0.08
Failing1.191.16−0.03

Sharing drafts reduced NLL most for the alternative passing algorithm (0.08 nats/token). Figure 9 in Appendix C.12 shows the execution outcomes, and Table 13 gives the token scores.

Thank you again for the careful review and for suggesting these controls. We hope the additional experiments address your concerns and would welcome any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive comments. Reviewers recognized the motivation and consistent gains, the breadth of evaluation and controlled comparisons, and the clear presentation of the method. Their main questions concerned the cost of debate, SFT as a simpler alternative to OPD, the reliability of confidence weighting and peer revision, and training stability and loss selection. We have added comparisons and teacher-level analyses, repeated training with two further seeds, and clarified the implementation and evaluation settings.

The additional comparisons use Qwen3-14B + Qwen3-8B → Qwen3-4B. Single-teacher controls use Qwen3-14B, the stronger teacher.

1. Training cost (phbi, WbTz). For ST-Indep-4 (Appendix C.1), the 14B teacher generates four independent drafts, which we concatenate before scoring the student's continuation. OPD-Long spends the same training time on more student updates.

G-Table1 — Equal-time scores and updates (Appendix C.1, Tables 3–4)

MethodTool avg. (20 h)UpdatesCode avg. (30 h)Updates
OPD-Long23.8620545.12167
ST-Indep-424.6013645.92120
MAD-OPD25.7811748.01108

MAD-OPD gains 1.18 / 2.09 points over four-sample OPD and 1.92 / 2.89 over OPD-Long on tool use / code. The four-sample baseline also uses more generated and scored teacher tokens per state in both domains (Table 4).

2. SFT on debate outputs (WbTz). For MAD-SFT, GPT-5.1 selected eligible answers from the same teachers' archived second-round outputs. All methods below trained for 100 updates.

G-Table2 — Scores after 100 updates (Appendix C.11, Table 12; C.4, Table 6)

MethodTool avg.Code avg.
Strong-teacher OPD23.2444.41
MAD-SFT21.5946.32
MAD-OPD, uniform weights25.8546.60
MAD-OPD27.6348.65

MAD-SFT improves on OPD on code. MAD-OPD scores 6.04 points higher on tool use and 2.33 higher on code than SFT. Its teachers score the student's own sampled actions and programs.

3. Confidence weights (phbi). We measured the weights and checked teacher correctness in Appendix C.4. Selection accuracy uses the 30 tool states and 34 code problems with exactly one correct teacher answer, with half credit for ties.

G-Table3 — Confidence diagnostics and student gains (Appendix C.4, Figure 5 and Table 6)

MeasureTool useCode
States with valid weights9699
Median absolute weight deviation from 50% (pp)0.870.75
Always select 14B: accuracy53.33%55.88%
Select by confidence: accuracy76.67%70.59%
Student gain over uniform debate weights (points)+1.78+2.05

All five benchmark scores improve over uniform weights in Table 7.

4. Code corrections and valid alternatives (phbi). Appendix C.12 compares independent and shared revision of the same 14B and 8B drafts, using identical execution tests.

G-Table4 — Teacher revisions on two sets of 50 problems (Appendix C.12, Figure 9)

OutcomeIndependentShared
One correct draft: failing draft corrected17/5030/50
Same set: passing draft became wrong5/506/50
Two correct algorithms: both still pass46/5045/50
Same set: two different passing algorithms remain44/5043/50

Sharing produces 13 more corrections and one more regression, while retaining different passing algorithms on 43/50 problems. Both teachers also scored three fixed 4B student programs per problem on 36 problems. We averaged NLL across the teachers and problems.

G-Table5 — Mean code NLL, nats/token (Appendix C.12, Table 13)

Student programIndependentShared
Passing, teacher's original approach0.580.57
Passing, alternative approach0.830.75
Failing1.191.16

5. Training seeds and loss choice (MkSt, phbi). Two further training runs give similar domain scores.

G-Table6 — MAD-OPD training seeds, 100 updates (Appendix C.4, Table 7)

RunTool avg.Code avg.
Original27.6348.65
Repeat 127.4448.50
Repeat 227.4048.48
Mean ± sample SD27.49 ± 0.1248.54 ± 0.09

Code evaluation uses 16 decoding seeds per checkpoint. We train separate tool-use and code checkpoints, chose JSD for tools and reverse KL for code using the theoretical analysis (§§3.2–3.3, Appendix D), the loss curves (C.8, Figure 7) and the five benchmark results (C.5, Table 9). Table 9 also reports each loss on both domains. Appendix B gives the logits, temperature, response mask and evaluation settings.

The revised §5.1 points to these experiments. We look forward to the reviewers' favourable consideration and remain grateful for their valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.