acceptodds
Under review as a conference paper at ICLR 2027

Less Is More: Selective Expert Supervision for Multi-Teacher On-Policy Distillation

Abstract

Domain-specialist language-model agents can perform well individually, but integrating their capabilities into one student remains difficult. Multi-Teacher On-Policy Distillation (MOPD) addresses sequential-training interference by supervising student rollouts with domain specialists, yet our four-domain study reveals uneven transfer: WebShop success is 57.29% for its teacher and only 40.53% for the Vanilla MOPD student. A directed gradient analysis of Vanilla MOPD further finds time-varying opposition among domain learning signals, raising the question of whether every student trajectory needs full teacher supervision. We propose SEAM (Selective Expert Supervision and Advantage Matching), which preserves rollout quotas and fixed teacher routing but scores every failed trajectory and only a sampled subset of successful ones. To limit changes in domain influence when retained batches shrink, SEAM calibrates each domain's distillation signal to its original rollout budget before a shared student update. Across ALFWorld, WebShop, SciWorld, and Search, SEAM improves WebShop success by 9.35 percentage points and four-domain macro success by 2.20 points over Vanilla MOPD across two training seeds. It uses 5.75% fewer teacher-scored trajectories and reduces measured end-to-end training time by 5.48%. Selective supervision thus improves integration without reducing exploration, although specialist performance is not inherited losslessly.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Agentic RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RaTzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SEAM for multi-teacher on-policy distillation of language-model agents. SEAM keeps all failed student trajectories, subsamples successful ones before teacher scoring, rescales each retained domain loss, and averages losses equally across domains. On four agent benchmarks with a Qwen2.5-3B student, the method improves WebShop and macro success over a Vanilla MOPD implementation while reducing the number of teacher-scored trajectories and measured training time.

Critical strengths

(i) The paper is candid about where the gain occurs. WebShop improves substantially in both reported runs, while ALFWorld and Search regress slightly; the authors do not present this as a Pareto improvement.

(ii) The experimental setup is documented in useful detail, including rollout quotas, evaluation panels, teacher training, and timing components. Separating evaluation-pass variability from training-run variability is also the right statistical distinction, even though the latter is not estimated adequately.

Critical weaknesses

(i) The experiment does not isolate selective supervision, which is the paper's title claim. Relative to Vanilla MOPD, SEAM changes at least three things at once: trajectory admission, global-token averaging to equal per-domain averaging, and a domain-level calibration factor. This matters greatly because Search contributes 256 trajectories per update while each other domain contributes only 16. The paper itself acknowledges that the no-selection arm retains both aggregation and calibration and that there is no shared-update-only arm. In the sole ablation seed, this no-selection arm already raises WebShop from 40.43 to 45.35, so a large part of the result is obtained without selecting fewer trajectories. The gradient-conflict analysis does not repair the gap: it samples seven updates, finds effects that change sign, and never shows that successful trajectories are the conflicting or redundant ones. The current evidence supports a bundled optimizer change, not the stated conclusion that selective expert supervision drives better integration.

(ii) The empirical basis is too narrow for an RL/post-training claim. The main comparison has only two training seeds, and all component ablations use one seed. The reported 95% intervals come from repeated stochastic evaluations of fixed checkpoints; they do not measure training variance. This is particularly problematic because the claimed macro improvement is only 1.68 and 2.74 points across the two runs, several domains move in opposite directions, and the selection-specific increment is reported for only one run. The 5.48% wall-clock saving is likewise measured on one run, while the combined rollout-and-scoring phase is actually slower for SEAM in every interval. Results from one 3B model, one set of four teachers, and one domain mixture are not enough to establish robustness or generality.

Questions for authors

(i) Can the authors run a factorial ablation that separately changes domain-wise averaging, calibration, and trajectory selection, including a Vanilla+domain-wise-mean arm and a selection-only arm?

(ii) Can all main and ablation arms be evaluated over enough independent training seeds to report confidence intervals across training runs rather than across repeated evaluations of the same checkpoint?

(iii) Does the method retain its benefit under other domain mixtures, student sizes, and teacher qualities, and against an equal-compute baseline that simply performs fewer Vanilla updates or rollouts?

Reviewer_QPAyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies capability integration in MOPD problem. Motivated by the uneven capability transfer across four tasks, the authors propose SEAM, which retains all failed student trajectories while sampling part of the successful trajectories before teacher scoring and act optimization. Experiments on ALFWorld, WebShop, SciWorld, and Search show improved WebShop performance and gains in aggregate success, together with reduced teacher-scoring and training costs.

Critical strengths

1: The paper is motivated by a real empirical case and addresses a relevant problem in multi-teacher distillation. The observation that Webshop suffers considerably more capability loss than others provides a concrete motivation for studying interference during multi-domain distillation.

2: The proposed method is simple and practical, requiring no change to both the teachers or student rollout budget. I believe its introduced trajectory selection rule, if with proper adaptation, can be easily integrated to current OPD pipelines.

Critical weaknesses

1: Using Figure 1 as motivation is a good try, but it also raises some concerns. In Figure 1 (right), we can observe that SciWorld actually experiences more severe gradient conflicts than WebShop at several steps, yet it does not suffer from a comparable performance degradation. The paper lacks an in-depth analysis of this phenomenon. Meanwhile, the gradient analysis itself is still quite preliminary. It is conducted at only seven optimizer updates of Vanilla MOPD, and the paper does not demonstrate whether SEAM actually reduces these conflicts. Moreover, the proposed selection rule is based on trajectory success or failure, rather than whether a trajectory produces conflicting gradients. The connection between the motivation and the proposed method is therefore not sufficiently clear.

2: The ablation study doesn't fully isolate the effect of proposed trajectory selection and the combination of gradient from different domains. Vanilla MOPD uses a global token-averaged loss, whereas SEAM first computes a token-averaged loss for each domain and then combines the four domains with equal nominal weights and additional calibration factors. This is a substantial change because the rollout quotas are highly imbalanced: 16 trajectories each for ALFWorld, WebShop, and SciWorld, compared with 256 for Search. If trajectory lengths were similar, Search would contribute approximately 84% of the tokens under Vanilla MOPD, while WebShop would contribute only 5%. Equal domain averaging changes their nominal weights to 25% each before calibration. There are also plenty of papers talking about the gradient allocation issue, for example:
[1]Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation (arxiv 2608.19098) [2] ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Models (arxiv 2509.21826) [3] Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy (arxiv 2605.14558) [4] D3-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation (arxiv 2608.24987)

3: Lack of stronger baselines. The current experiments compare SEAM primarily against Vanilla MOPD and its component ablations. Comparisons with stronger recent methods, as well as simpler alternatives such as domain balancing and random trajectory filtering, would help better establish the effectiveness of the proposed method.

4: Limited experimental scope and too few training seeds. I fully understand that running RL experiments is expensive. However, given the randomness of RL training, the limited experimental settings, and the relatively small model scale (3B), I believe the paper would benefit from more independent training runs. It would also be helpful to include at least some experiments with larger models (Even just one on 7B or smaller models) to demonstrate that the proposed method generalizes beyond the current setting.

Questions for authors

1: Could the authors provide a similar gradient analysis across all four domains using SEAM? In particular, does SEAM actually reduce the gradient conflicts observed in Figure 1?

2: Could the authors add a baseline that applies equal domain averaging without trajectory selection or calibration? This would help clarify how much of the improvement comes from changing the domain weights.

3: What would the results look like if the authors simply combined domain balancing with downsampling successful trajectories, without the proposed advantage-matching calibration?

Reviewer_GbMxOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents SEAM (Selective Expert Supervision and Advantage Matching), which selectively scores failures and sub-sampled successes in multi-teacher on -policy distillantion, calibrates domain signales to original budgets, and improves webshop/maro success over Vanilla MOPD with faewer socred trajectories.

Critical strengths
  • Clear diagnosis of uneven multi-domain transfer plus time-varying gradient opposition that moticates slective supervision.

  • Practical low overhead method e.g., keep failures, sub-sample successes, budget-aware calibration, improves the weakest domain and overall macro while cutting teacher work.

Critical weaknesses
  • Gains concentrate on Webshop, ALFWorld/Search show small regressions and the method does not achieve Pareto improvement.

  • Ablations and analysis do not fully disentagle the independent contributions of selection versus calibration, and generality beyond the four-domain Qwen2.5-3B setting is untested

Questions for authors

can you privide additional analysis or ablation studies that isolate how much of the WebShop gain is attributable to selection versus the calibration factor c_d, ideally with gradient-conflict meausurment under each ablation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.