- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes SEAM for multi-teacher on-policy distillation of language-model agents. SEAM keeps all failed student trajectories, subsamples successful ones before teacher scoring, rescales each retained domain loss, and averages losses equally across domains. On four agent benchmarks with a Qwen2.5-3B student, the method improves WebShop and macro success over a Vanilla MOPD implementation while reducing the number of teacher-scored trajectories and measured training time.
Critical strengths
(i) The paper is candid about where the gain occurs. WebShop improves substantially in both reported runs, while ALFWorld and Search regress slightly; the authors do not present this as a Pareto improvement.
(ii) The experimental setup is documented in useful detail, including rollout quotas, evaluation panels, teacher training, and timing components. Separating evaluation-pass variability from training-run variability is also the right statistical distinction, even though the latter is not estimated adequately.
Critical weaknesses
(i) The experiment does not isolate selective supervision, which is the paper's title claim. Relative to Vanilla MOPD, SEAM changes at least three things at once: trajectory admission, global-token averaging to equal per-domain averaging, and a domain-level calibration factor. This matters greatly because Search contributes 256 trajectories per update while each other domain contributes only 16. The paper itself acknowledges that the no-selection arm retains both aggregation and calibration and that there is no shared-update-only arm. In the sole ablation seed, this no-selection arm already raises WebShop from 40.43 to 45.35, so a large part of the result is obtained without selecting fewer trajectories. The gradient-conflict analysis does not repair the gap: it samples seven updates, finds effects that change sign, and never shows that successful trajectories are the conflicting or redundant ones. The current evidence supports a bundled optimizer change, not the stated conclusion that selective expert supervision drives better integration.
(ii) The empirical basis is too narrow for an RL/post-training claim. The main comparison has only two training seeds, and all component ablations use one seed. The reported 95% intervals come from repeated stochastic evaluations of fixed checkpoints; they do not measure training variance. This is particularly problematic because the claimed macro improvement is only 1.68 and 2.74 points across the two runs, several domains move in opposite directions, and the selection-specific increment is reported for only one run. The 5.48% wall-clock saving is likewise measured on one run, while the combined rollout-and-scoring phase is actually slower for SEAM in every interval. Results from one 3B model, one set of four teachers, and one domain mixture are not enough to establish robustness or generality.
Questions for authors
(i) Can the authors run a factorial ablation that separately changes domain-wise averaging, calibration, and trajectory selection, including a Vanilla+domain-wise-mean arm and a selection-only arm?
(ii) Can all main and ablation arms be evaluated over enough independent training seeds to report confidence intervals across training runs rather than across repeated evaluations of the same checkpoint?
(iii) Does the method retain its benefit under other domain mixtures, student sizes, and teacher qualities, and against an equal-compute baseline that simply performs fewer Vanilla updates or rollouts?