acceptodds
Under review as a conference paper at ICLR 2027

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student on its own reasoning trajectories using feedback from a stronger teacher. Teacher interventions can improve these trajectories, but also change the distribution on which the student learns. Our controlled studies show that rollout quality alone is an incomplete criterion for allocating teacher guidance. Deeper intervention yields diminishing gains in rollout accuracy while increasing off-policy load. In a training probe with a restricted rollout horizon, peak student accuracy and performance retention favor different intervention strengths. The preferred intervention depth and placement also vary across benchmarks. These findings motivate MAESTRO, which uses local policy disagreement to jointly adapt when the teacher takes over and how long it generates. Its policy disagreement score combines teacher-weighted candidate coverage with local distribution similarity and is aggregated within reasoning paragraphs. Across eight mathematical reasoning benchmarks, MAESTRO achieves the highest macro-average accuracy among the compared methods for both 0.6B and 1.7B Qwen3 students, with the 1.7B student leading on every benchmark. MAESTRO also reduces average training response length by 67.3% relative to standard OPD.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
On-Policy Distillation, Knowledge Distillation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y5dnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAESTRO, an adaptive teacher intervention framework for on-policy distillation that uses a Policy Disagreement Score (PDS) to determine intervention timing and depth. The method is evaluated on eight mathematical reasoning benchmarks using two Qwen3 student models and compared against existing baselines.

Critical strengths
  1. The paper discusses an interesting question of when and how teachers should intervene in on-policy distillation.
  2. The method is evaluated against multiple existing baselines across several mathematical reasoning benchmarks.
Critical weaknesses
  1. Figures 2–4 present observations about intervention depth, position, and training performance, but do not directly justify using policy disagreement to guide intervention timing and depth.
  2. Policy disagreement does not necessarily indicate reasoning errors or greater benefits from teacher intervention. The ablation studies show empirical improvements but do not establish whether PDS can reliably predict intervention benefits.
  3. The definition of PDS as 1−CtBt1-C_tB_t lacks sufficient justification. It remains unclear why this combination is appropriate or preferable for measuring useful intervention signals.
  4. The weighting scheme assumes that earlier tokens are more important for reasoning, but this assumption is not directly validated, and the rationale for the specific weighting function is insufficient.
  5. MAESTRO combines PDS-based triggering, adaptive intervention depth, weighted aggregation, and early rollout termination. Although ablations are provided, the independent benefit of PDS-based adaptation over simpler strategies under the same termination conditions remains unclear.
  6. Experiments are conducted only on relatively small student models (0.6B and 1.7B), leaving the effectiveness of the method on larger models (e.g., 7B+) unverified.
  7. Although relevant studies are cited, the paper does not clearly distinguish its technical contributions from existing methods, particularly Relay-OPD[1], SKD[2], and FastOPD[3].

[1] Pass the Baton: Trajectory-Relayed On-Policy Distillation

[2] Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling

[3] Fast and Effective On-Policy Distillation from Reasoning Prefixes

Questions for authors

See Weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Y5dn (1/6)

We thank the reviewer for the detailed comments. They help us make the link between our analysis and our method clearer.

Concern 1: Figures 2–4 do not directly justify using policy disagreement to guide intervention timing and depth.

Response 1. Our analysis motivates PDS in two ways. First, it identifies the policy gap as the key quantity. Second, each finding maps to a design choice, and the trained model behaves as predicted.

1. Intervene briefly where the gap is large.

  • The cost is a policy gap. Section 2.1 measures off-policy load as the teacher–student log-probability gap (Eq. 1). PDS measures a closely related gap locally, on the student's own text.
  • A large gap grows if left alone. A high PDS means the student has drifted from the teacher. Without correction, the student keeps generating from this drifted state.
  • Short takeovers are enough. Finding 1 shows that shallow intervention captures most of the benefit, while off-policy cost keeps growing with depth.
  • Design. MAESTRO places short takeovers where the gap is largest. This closes the most gap per teacher token. Limits on takeover length and count keep the off-policy cost low.

2. Each finding maps to a design choice.

Finding in Section 2Design choiceObserved behavior (Figures 5, 6, 8)
Finding 1: gains saturate, off-policy load keeps growingIntervene only at high disagreement, with short takeoversTeacher usage comparable to Relay-OPD (1.05×), with higher accuracy
Finding 2: short continuation after intervention is enoughEnd the rollout after the final takeoverTraining responses 67.3% shorter than OPD, with higher accuracy (47.12 vs. 41.23)
Finding 3: harder problems need more interventionMore frequent and longer takeovers where PDS is highTrigger rate rises with difficulty: 21% on AMC23, 60% on AIME24, 70% on HMMT
Finding 3: the best position differs across problemsRe-evaluate PDS at every paragraphTakeover tracks the student: teacher-token fraction drops from ~16% to ~5% during training

3. Each choice is tested. Removing any component lowers accuracy at 1.7B (47.1 for the full method):

  • Coverage only: 43.3.
  • Similarity only: 44.5.
  • No weighted aggregation: 43.6.
  • No adaptive depth: 45.1.
  • No termination: 44.53.

(Continued in part 2/6.)

AuthorsOpenReview ↗

Response to Reviewer Y5dn (2/6)

Concern 2: Policy disagreement does not necessarily indicate reasoning errors or greater benefits from teacher intervention.

Response 2. We agree that PDS is not an error detector for the final answer. PDS is a signal for allocating teacher guidance. Its goal is to improve the quality of the whole training trajectory and to shrink the policy gap. We support its value with three levels of evidence.

1. What PDS is designed to do.

  • The student does most of the reasoning. The teacher writes only a small part of each rollout.
  • A takeover pulls the student back. When the gap is large, a few teacher tokens move the student back toward the teacher's policy.
  • The benefit is on the trajectory. The rest of the rollout, and the distillation signal on it, then come from a state closer to the teacher. This holds even when the final answer does not change.

2. Why closing the gap matters.

  • Late tokens in long OPD rollouts help little. Supervision on reasoning prefixes keeps most of the benefit of full OPD [1]. Training only on the first N rollout tokens can beat full rollouts [2].
  • Our results agree. Removing termination makes rollouts much longer and lowers accuracy (Response 5, part 5/6).
  • MAESTRO shrinks this low-gain region. It corrects the student where the gap is largest and ends the rollout after the final takeover.

3. Evidence 1: downstream performance.

  • MAESTRO has the best macro-average among the evaluated methods at both sizes.
  • At 1.7B, it reaches 47.12%, vs. 45.18% for Relay-OPD and 41.23% for OPD.
  • The gains hold across three seeds and are significant (p < 0.05).

4. Evidence 2: behavior consistent with our findings.

  • Difficulty. Finding 3 shows harder problems need more intervention. PDS triggers more often on harder benchmarks (21% on AMC23, 60% on AIME24, 70% on HMMT).
  • Training progress. Takeover falls as the student improves (teacher-token fraction from ~16% to ~5%).
  • Budget. Finding 1 shows that the intervention budget should be spent where it helps most. MAESTRO uses a teacher budget comparable to Relay-OPD (1.05×) yet reaches higher accuracy.

5. Evidence 3: trajectory quality under matched prefixes.

  • Setup. We sample 1,000 rollouts from the training set. From the same student prefix, we build two continuations: one with a PDS takeover, and one where the student continues alone.
  • Judging. An LLM judge (GPT-6.1-sol) sees the two continuations as anonymous A and B, with ties allowed. It scores each on a fixed rubric.
  • Rubric. Mathematical validity, reasoning completeness, error correction, reasoning depth, and reasoning efficiency. The judge is told not to reward length.
  • Result. PDS takeovers are preferred in 68.3% of pairs. On average, they lead by 42.3 points on a 100-point scale.

6. Case studies. We compare rollouts from the same prefix, with and without a PDS takeover.

CaseWithout takeoverWith PDS takeover
Polynomial remainderThe student applies both remainder conditions at the same root and derives a false contradiction (α = 1).The teacher proposes a degree bound for P(x). The student then builds the correct cubic, checks both remainders, and gets the answer 23.
Ice-cream assignmentThe student adds the three counts wrongly (N = 1916, answer 916).The teacher fixes the counting formula for a given flavor split. The student then gets N = 2016 and the correct answer 16.

Revision. We will add the judge protocol and these cases to the appendix. We will also clarify in Section 3 that PDS targets trajectory quality and the policy gap.

[1] Fast and Effective On-Policy Distillation from Reasoning Prefixes, Findings of ACL 2026.

[2] Less is More: Early Stopping Rollout for On-Policy Distillation, arXiv 2026.

(Continued in part 3/6.)

AuthorsOpenReview ↗

Response to Reviewer Y5dn (3/6)

Concern 3: It remains unclear why this combination is appropriate for measuring useful intervention signals.

Response 3. The two terms catch different failures. The ablation and case studies both show that each term alone is not enough.

1. Design (Section 3.1).

  • Coverage (C) checks whether the student keeps the teacher's preferred tokens.
  • Similarity (B) compares their normalized distributions over the shared candidates.
  • PDS = 1 − C·B rises when either term is low.

2. Ablation (Figure 5(a), 1.7B).

VariantAvg. accuracyPass@3
Full PDS47.159.2
Coverage only43.355.8
Similarity only44.555.1

3. Case studies: one term alone mistimes the takeover. Teacher text is in bold.

Coverage only: the takeover comes too early.

  • The teacher hands back right after a lead-in sentence.
  • The error appears in the student's next line and is never corrected.
Takeover with coverage onlyWell-timed takeover
Infinite product of cube roots"Let a₁ = 10^{1/3}, a₂ = a₁^{1/3}, a₃ = a₁^{1/9}. So the general term is:" → Student: a_n = a₁^{1/3^n} (off by one)Student: a_n = a₁^{1/3^n}. "The exponent is off by one: at n = 1 this gives a₁^{1/3}, not a₁. The correct term is a_n = a₁^{1/3^{n−1}} = 10^{1/3^n}." → Student: P = 10^{1/2}, so m = 10

Similarity only: the takeover comes too late.

  • The student has already entered an enumeration loop.
  • It ignores the same correct hint.
Takeover with similarity only (late)Earlier takeover
Fourth square triangular numberStudent: "Let's test n = 50, 51, 52, … None works. We should continue with n = 100, 101, …" "Do not enumerate n one by one. Rewrite the condition as (2n+1)² − 8k² = 1. Its solutions give n = 1, 8, 49, 288." → Student: "I should verify all intermediate values first. Let's return to n = 100, 101, …"Student: "We need the next n after 49 with n(n+1)/2 a square. Direct search may be expensive." Same teacher text. → Student: "The next index is 288. Then t₂₈₈ = 41616 = 204², whose digit sum is 18."

4. The two terms are complementary.

  • Coverage catches drift early, but it can fire before the error is visible.
  • Similarity confirms the drift, but it can fire too late.
  • Together, they give the well-timed takeovers shown on the right.

(Continued in part 4/6.)

Reviewer_hEjKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies adaptive teacher intervention in on-policy distillation and highlights the trade-off between improving reasoning trajectories and introducing off-policy deviation. It proposes MAESTRO, which uses a Policy Disagreement Score to adapt both the timing and depth of teacher intervention during student rollouts. The method is evaluated on eight mathematical reasoning benchmarks with two Qwen3 student scales and consistently outperforms standard OPD and related teacher-intervention baselines.

Critical strengths
  1. The paper provides a well-motivated analysis of teacher intervention in OPD, showing the trade-off between trajectory improvement and off-policy deviation.
  2. MAESTRO is evaluated on eight mathematical reasoning benchmarks and consistently improves over strong OPD-based baselines across two student scales.
Critical weaknesses
  1. The efficiency analysis mainly relies on response length, but MAESTRO requires online teacher–student distribution comparison to compute PDS. The resulting teacher inference overhead is not included in the reported efficiency comparison.
  2. Some key implementation details are insufficiently specified, especially the exact mapping from paragraph-level PDS to intervention depth and the protocol used to select important hyperparameters.
Questions for authors

Questions For Authors:

  1. How is the paragraph-level PDS converted into the number of teacher-generated paragraphs during intervention?
  2. How were the main hyperparameters, including KK, the PDS threshold, α\alpha, and τ\tau, selected?
  3. What is the end-to-end computational cost of MAESTRO when the additional teacher inference required for PDS computation is included?
  4. Are the reported improvements stable across multiple independent training runs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hEjK (1/2)

We thank the reviewer for the constructive assessment and the detailed questions.

Concern 1: The efficiency analysis mainly relies on response length, but MAESTRO requires online teacher–student distribution comparison to compute PDS. The resulting teacher inference overhead is not included.

Response 1. We now report the full end-to-end training time, with all teacher and PDS costs included. PDS adds little cost because it shares the teacher forward pass used during rollout. MAESTRO is faster in two ways.

  • Faster per step. Over the same 35 steps, MAESTRO takes 27.3% less time than OPD (1.38× speed-up).
  • Fewer steps. MAESTRO finishes in 35 steps. OPD reaches its reported result only at step 50.
  • Overall. MAESTRO reaches its result in about half the training time of OPD.

1. Why PDS adds little cost. PDS does not need a separate teacher inference pass.

  • One teacher forward serves PDS and distillation. During rollout, the teacher verifies the student's draft tokens, as in speculative decoding. PDS uses the teacher and student distributions from this same forward pass. The teacher log-probabilities for the distillation loss come from the same teacher outputs.
  • Teacher KV cache. Takeover decoding continues the same teacher request and extends its KV cache. It does not prefill the prefix again.
  • Student KV cache. The student keeps its own KV cache. It continues from both accepted student tokens and teacher-written tokens without replaying the prefix.
  • What is still paid for. The teacher forward, the short takeovers (at most two per rollout), the PDS tensor operations, and the actor update. PDS itself generates no tokens.

2. Measurement. Both methods use the same data, batch size, models, and hardware. The setting is thinking mode (Qwen3-1.7B student, Qwen3-4B teacher). Both are timed over 35 steps, which is the full training length of MAESTRO. Teacher log-probability caching is on, as described above. Times exclude checkpoint saving.

MetricOPDMAESTROChange
End-to-end training time (35 steps)65.6 min47.7 min27.3% less
Generation + teacher log-prob time (s/step)72.3670.892.0% faster
Actor update (s/step)38.428.844.35× faster
Average response length (tokens)7,0881,15083.8% shorter
  • No extra cost on the generation side. The MAESTRO generation stage includes PDS scoring and teacher takeovers. It is still 2.0% faster than OPD's generation and teacher log-prob stages combined.
  • Reuse keeps PDS cheap. Teacher log-probabilities come from the rollout forward pass, so little separate teacher scoring is needed.
  • Shorter rollouts save the most. Responses are 83.8% shorter, so the actor update is 4.35× faster.
  • The time adds up. The stages above cover nearly all of the total time. The remaining ~1 min for each method is fixed per-step overhead, mainly weight synchronization from actor to rollout and advantage computation.
  • Fewer steps save more. OPD reaches its reported result at step 50. Its later steps are also slower, since its responses keep growing. A linear estimate gives at least ~94 min for 50 OPD steps. MAESTRO therefore needs about 49% less training time, with higher accuracy (55.69 vs. 48.70).
  • Same trend in the main setting. MAESTRO also shortens training responses by 67.3% relative to OPD (Figure 5(c)), with higher accuracy (47.12 vs. 41.23 at 1.7B; 31.36 vs. 27.16 at 0.6B).

Concern 2: How is the paragraph-level PDS converted into the number of teacher-generated paragraphs during intervention?

Response 2. When PDS exceeds the threshold δ, the number of teacher paragraphs n grows linearly with the gap to the threshold, up to a maximum:

n = min(n_max, n_0 + ⌊k · (PDS − δ)⌋), applied only when PDS > δ.

  • Values (0.6B). δ = 0.14, n_0 = 4, k = 50, n_max = 12.
  • Examples. PDS 0.16 gives 5 teacher paragraphs. PDS 0.20 gives 7. PDS ≥ 0.30 gives the cap of 12.
  • Revision. We will add this rule and the values for every setting to Section 3.2.

(Continued in part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer hEjK (2/2)

Concern 3: Parameter selection and experimental setup.

Response 3. We select hyperparameters on a validation set. Sensitivity analyses and ablations support the choices.

1. Selection.

  • We use the same validation set and protocol as the baselines.
  • Defaults are chosen on this validation set and fixed before test evaluation.
  • The cap of two takeovers is an efficiency constraint, not a tuned value (Appendix C).

2. Sensitivity. Every tested setting stays 2.2 to 5.9 pp above OPD at 1.7B. With the validation-selected defaults, the gain over Relay-OPD holds across three seeds (Response 4).

HyperparameterValues testedAccuracy rangeGain over OPD (41.23)Source
PDS threshold δ0.06, 0.08, 0.10, 0.13, 0.1643.8–47.1+2.6 to +5.9Figure 5(b)
Weight strength α0, 0.25, 0.5, 1.043.4–47.1+2.2 to +5.9Figure 7(a)
Decay scale τ6, 12, 24, 4843.9–47.1+2.7 to +5.9Figure 7(b)

3. Ablations: the components are complementary. Removing any component lowers accuracy (1.7B).

VariantAvg. accuracy
Full method47.1
Coverage only43.3
Similarity only44.5
Without weighted aggregation43.6
Without adaptive depth45.1
Without rollout termination44.5
Fixed-position trigger41.4

Each component fixes a different failure:

  • Coverage alone fires too early.
  • Similarity alone fires too late.
  • Without weighting, the takeover fires before a concrete error appears.
  • With a fixed depth, the teacher hands back before naming the error.
  • Without termination, long student continuations dilute the corrected trajectory.

Our responses to Reviewer Y5dn (parts 3/6 to 5/6) give a case study for each failure.

Concern 4: Are the reported improvements stable across multiple independent training runs?

Response 4. Yes. We repeated training with three seeds for MAESTRO and Relay-OPD, using the same budget.

StudentMethodRun 1 (paper)Run 2Run 3Mean ± SD
0.6BRelay-OPD29.7129.6929.8029.73 ± 0.06
0.6BMAESTRO31.3631.4831.7931.54 ± 0.22
1.7BRelay-OPD45.1844.9445.6245.25 ± 0.34
1.7BMAESTRO47.1247.5846.9047.20 ± 0.35
  • Higher in every run at both sizes.
  • No overlap. The lowest MAESTRO run beats the highest Relay-OPD run (31.36 vs. 29.80 at 0.6B; 46.90 vs. 45.62 at 1.7B).
  • Consistent with Table 1. The mean gains are +1.81 and +1.95 pp.
  • Significant. Paired tests over problems give p < 0.05 for every comparison, e.g., +1.65 pp [+0.13, +3.16] at 0.6B and +1.94 pp [+0.58, +3.32] at 1.7B over Relay-OPD.
  • Evaluation. Each problem is scored with 32 sampled responses.
Reviewer_nadwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MAESTRO uses teacher–student policy disagreement to decide when the teacher intervenes during distillation and how long it generates. Experiments on eight mathematical reasoning benchmarks show improved accuracy and shorter training responses for 0.6B and 1.7B students.

Critical strengths
  • The 1.7B results are encouraging: MAESTRO leads on all eight benchmarks and improves average accuracy from 45.18% to 47.12% over Relay-OPD.
  • Component ablations support the intervention design, and training responses are substantially shorter than standard OPD.
Critical weaknesses
  • Evaluation is limited to small, non-thinking students. Both students are below 2B parameters and use non-thinking configurations. It remains unclear whether the method benefits larger students or remains competitive with thinking-mode baselines. The 0.6B model achieves the best average among compared methods, but its absolute performance on difficult benchmarks remains limited.

  • The robustness of the gains needs clarification. Average improvements over Relay-OPD are modest, and the aggregation hyperparameter sweeps show substantial performance variation. Without repeated runs or uncertainty estimates, it is difficult to assess how reliably the gains hold.

Questions for authors

How does MAESTRO compare with thinking-mode students and corresponding distillation baselines under comparable budgets?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nadw (1/2)

We thank the reviewer for the positive comments on our 1.7B results and ablations.

Concern 1: Evaluation is limited to small, non-thinking students. It remains unclear whether the method benefits larger students or remains competitive with thinking-mode baselines.

Response 1. MAESTRO keeps a clear lead in thinking mode. For model scale, we are limited by compute, and we follow common practice in distillation research.

1. Thinking mode: higher accuracy with fewer training steps.

  • Setup. Qwen3-1.7B thinking student, Qwen3-4B thinking teacher, same setup as the baselines.
  • Training length. MAESTRO is trained for 35 steps. OPD is trained for 50 steps, and we report its best checkpoint (step 50).
  • Threshold. We use a PDS threshold of 0.15.
BenchmarkOPDRelay-OPDMAESTRO
AIME2442.8136.9854.69
AIME2533.6530.7346.46
AIME2631.1526.8848.65
AMC2370.6769.2477.75
HMMT Feb 202628.3122.5428.44
HMMT Nov 202525.9420.8329.79
MATH50092.4889.7993.21
OlympiadBench64.6063.7266.54
Macro average48.7045.0955.69
  • Large gains. MAESTRO improves the macro-average by +6.99 pp over OPD and +10.60 pp over Relay-OPD.
  • Consistent gains. It beats both baselines on all eight benchmarks.
  • Fewer steps. It uses 15 fewer training steps than OPD.
  • Reliable baselines. On AIME24 and AIME25, our OPD results (42.81 and 33.65) are within 1.4 pp of those reported by [1] under the same student and teacher (41.5 and 34.2).

2. Model scale: limited by academic compute.

  • We agree that 7B+ students would make the study stronger.
  • Compute limit. As an academic lab, we cannot train a 7B+ student with a 30B+ teacher in an OPD pipeline within the rebuttal period.
  • Common practice. Strong published work also studies distillation on small students:
    • GKD [2]: T5 students of 77M to 800M.
    • DistiLLM [3]: students up to 3B.
    • ATKD [4]: students of 68M to 410M.
    • Entropy-Aware OPD [5]: Qwen3 students of 0.6B to 4B.
  • Revision. Our evidence supports the studied scales. We will discuss 7B+ students as future work.

[1] On-Policy Delta Distillation, arXiv 2026.

[2] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024.

[3] DistiLLM: Towards Streamlined Distillation for Large Language Models, ICML 2024.

[4] Revisiting Knowledge Distillation for Autoregressive Language Models, ACL 2024.

[5] Entropy-Aware On-Policy Distillation of Language Models, ICML 2026.

Concern 2: The 0.6B model achieves the best average, but its absolute performance on difficult benchmarks remains limited.

Response 2. We agree that 0.6B scores on AIME and HMMT stay low in absolute terms. This reflects model capacity. Within this limit, MAESTRO gives the largest gains.

  • Large gains over the initial student. The macro-average rises from 11.80% to 31.36%. AIME25 rises from 2.40% to 21.98%.
  • Best among all methods. MAESTRO has the best macro-average, 1.65 pp above Relay-OPD. The largest gains are on hard benchmarks: AIME25 (+5.31 pp) and HMMT Nov 2025 (+4.06 pp).
  • Higher scores with a stronger student. At 1.7B, MAESTRO leads on all eight benchmarks.

(Continued in part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer nadw (2/2)

Concern 3: The robustness of the gains needs clarification. Without repeated runs or uncertainty estimates, it is difficult to assess how reliably the gains hold.

Response 3. The gains are reliable from three angles: statistical tests, training seeds, and hyperparameters.

1. Statistical tests: all gains are significant.

  • 95% CI. Paired bootstrap over problems, stratified by benchmark (10,000 resamples).
  • p-value. Two-sided paired permutation test (50,000 permutations).
  • Scoring. Each problem is scored by its avg@32 accuracy. The statistic is the macro-average gain.
StudentMAESTRO vs.Gain (pp)95% CIp
0.6BOPD+4.20[+2.86, +5.59]<0.05
0.6BFastOPD+1.71[+0.16, +3.30]<0.05
0.6BRelay-OPD+1.65[+0.13, +3.16]<0.05
1.7BOPD+5.89[+4.58, +7.17]<0.05
1.7BFastOPD+3.68[+2.36, +5.02]<0.05
1.7BRelay-OPD+1.94[+0.58, +3.32]<0.05
  • No 95% CI includes zero.
  • These tests cover problem-sampling uncertainty. Training-run variation is reported below.

2. Training seeds: MAESTRO wins in every run. We repeated training with three seeds for MAESTRO and Relay-OPD, using the same budget.

StudentMethodRun 1 (paper)Run 2Run 3Mean ± SD
0.6BRelay-OPD29.7129.6929.8029.73 ± 0.06
0.6BMAESTRO31.3631.4831.7931.54 ± 0.22
1.7BRelay-OPD45.1844.9445.6245.25 ± 0.34
1.7BMAESTRO47.1247.5846.9047.20 ± 0.35
  • The lowest MAESTRO run beats the highest Relay-OPD run at both sizes.

3. Hyperparameters: every setting beats OPD.

  • Selection. All defaults are selected on the validation set and fixed before test evaluation.
  • Sensitivity. Every tested setting stays above OPD at 1.7B.
  • Against the strongest baseline. With the validation-selected defaults, the gain over Relay-OPD holds across three seeds and is significant across eight benchmarks.
HyperparameterValues testedAccuracy rangeGain over OPD (41.23)Source
PDS threshold δ0.06, 0.08, 0.10, 0.13, 0.1643.8–47.1+2.6 to +5.9Figure 5(b)
Weight strength α0, 0.25, 0.5, 1.043.4–47.1+2.2 to +5.9Figure 7(a)
Decay scale τ6, 12, 24, 4843.9–47.1+2.7 to +5.9Figure 7(b)

Concern 4: How does MAESTRO compare with thinking-mode students and corresponding distillation baselines under comparable budgets?

Response 4. Please see Response 1 (part 1/2).

  • MAESTRO reaches 55.69% macro-average in thinking mode.
  • This is +6.99 pp over OPD and +10.60 pp over Relay-OPD.
  • It beats both baselines on all eight benchmarks, with 35 training steps vs. 50 for OPD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.