acceptodds
Under review as a conference paper at ICLR 2027

MAS-OPD: On-Policy Distillation for Multi-agent Systems

Abstract

Multi-agent systems (MAS) split a task across specialized roles and are promising on complex tasks, yet a prevailing approach relies on inference-time orchestration alone. General-purpose APIs are costly and hard to customize, while small models with role prompts rarely develop stable role competence or reliable collaboration, so post-training a MAS jointly is central. Most attempts use reinforcement learning, whose team-level reward leaves undetermined which step of which agent brought about the outcome, while local rewards need redesigning per task. On-policy distillation (OPD) gives token-level teacher supervision on trajectories the student samples, a denser signal needing no local reward, yet is underexplored for the interdependent agents of a MAS. Two difficulties arise: building complementary specialization from a judgement of which role a behavior belongs to while preserving the knowledge all roles need, and turning cross-agent collaborative information into supervision OPD can exploit. We present MAS-OPD, where Role-Advantage Specialization defines the role advantage as the difference between the teacher signals under target and non-target role conditions, and Privileged Attribution for Coordination attributes an interaction conflict to its source and supplies it to the teacher alone as privileged information. Extensive experiments on code and mathematics benchmarks show that MAS-OPD attains the highest mean score at both student scales and leads the agents to develop clearer role specialization and more effective collaborative behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Agent Post-training, Multi-agent Systems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_moT9OpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the issues of insufficient role specialization and the difficulty in effectively converting cross-agent collaboration information into training supervision in existing multi-agent systems by proposing the MAS-OPD framework. MAS-OPD uses Role-Advantage Specialization to compare the differences in teacher scores for the same token under target and non-target role conditions, thereby enhancing role-specific capabilities while preserving shared capabilities. At the same time, it employs Privileged Attribution for Coordination to attribute conflicts between agents using reference solutions and validation results from the training phase, and provides this attribution information exclusively to the teacher, thereby generating more coordinated token-level supervision. Experiments were conducted on six benchmarks using Qwen models. The results show that this method achieved the highest average scores across benchmarks.

Critical strengths

S1: The paper break down the challenges of training multi-agent systems into two problems, which are role specialization and cross-agent collaboration, and provides targeted solutions for each. S2: Role-Advantage Specialization is well-designed, by comparing differences in teacher ratings for the same token under different role conditions, MAS-OPD construct signals of role-specific strengths to help agents develop complementary professional competencies while retaining shared capabilities.

Critical weaknesses

W1: MAS-OPD treats differences in token probabilities across different roles as role advantages, but these differences may reflect the model's sensitivity to prompts instead of representing true functional complementarity. W2: PAC uses conflict attribution information, which is invisible to students, to guide teacher in generating supervision signals. However, students may not be able to reproduce the instructors' judgments based solely on their own visible interaction history.

Questions for authors

Q1: How do the authors verify that role-conditioned token probability differences reflect genuine functional complementarity? Q2: How does PAC ensure that students can learn reliable collaboration strategies from the supervisory signals generated by teachers based on privileged attribution information? Q3: Do the performance gains of MAS-OPD persist under a matched teacher-inference and computational budget Q4: When multiple agents have partially correct judgments, or when the source of the conflict is ambiguous, is PAC’s attribution mechanism still reliable?

Reviewer_ZgEWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies on-policy distillation (OPD) to multi-agent systems with independent role-specific policies. The students jointly generate interaction trajectories, a frozen teacher scores their tokens through forced decoding, and each role's policy is updated separately. Role-Advantage Specialization (RAS) compares teacher log probabilities under target and non-target roles with the same context and output prefix, using their difference to strengthen role differentiation. Privileged Attribution for Coordination (PAC) uses training-time reference answers, reference programs, and execution results to identify the source of a previous turn's conflict, and provides this attribution only to the teacher.

The experiments use a Qwen3-14B teacher and 1.7B/4B students to train two-role systems for code and mathematics, comparing single-agent, multi-agent, and several RL methods across six benchmarks. In Table 1, MAS-OPD achieves the highest mean score on all six benchmarks at both student scales. The authors also analyze role exchange, standalone Coder ability, revision behavior after conflicts, and interaction turns. The appendix expands the system to eight agents by replicating the same two role types.

Critical strengths
  1. The training interface is simple, and the scope of privileged information is clear. RAS and PAC share one token-level training signal. RAS adds one non-target-role scoring pass, while PAC changes only the teacher's context. Section 3.3 and Appendix F explain that attribution comes only from completed turns, rather than from the response currently being scored, and students never see it during training or inference. Distilling training-time facts about who should revise into student policies is a reasonable and implementable design.
  2. The results cover multiple benchmarks and two student scales, with relevant behavioral analyses. All twelve student-benchmark combinations in Table 1 attain the highest mean score. Removing either module in Table 2 lowers the mean on all six benchmarks, supporting an individual contribution from each. Figure 4's conflict-decision analysis and the appendix's cases additionally examine the specific behavior of revising when wrong and retaining an output when correct, beyond final accuracy alone. Five independent training runs also provide stronger evidence than repeated sampling from one trained model.
Critical weaknesses
  1. Missing ordinary multi-agent OPD and single-agent OPD baselines prevent a complete assessment of the contribution. Table 1 mainly compares MAS-OPD with a 14B teacher against RL methods without equivalent teacher token supervision. Table 2 already includes w/o RAS and w/o PAC, but each retains the other module. There is no baseline removing both while retaining the role-conditioned OPD of Section 3.1. Consequently, the results do not quantify how much of the overall improvement comes from existing OPD, the joint gain from the two modules, or their complementary relationship. The single-agent baselines likewise use inference or GRPO, without single-agent OPD under the same teacher and training resources. The comparisons support the effectiveness of the full system, but do not sufficiently establish that the proposed multi-agent distillation mechanisms outperform simpler teacher-distillation alternatives. This is my principal reservation about acceptance.
  2. The interpretations of specialization and collaboration do not sufficiently rule out role-format, feedback, and computation differences. The teacher difference in RAS still depends on context and student-generated tokens; cancellation of the student log-probability term does not automatically make it a calibrated measure of responsibility attribution. In Appendix F.3, the non-target role still faces the original target role's response-format requirements, so the difference may mix responsibility content with format compatibility. The role-exchange performance drop in Figure 3 shows sensitivity to role assignments, but may also reflect formatting, local competence, and co-adaptation. It does not directly establish reduced knowledge redundancy or saved representation capacity; similar standalone Coder accuracy tests only one type of task ability. In addition, Appendices D.3/D.6 indicate that single-agent multi-turn refinement lacks execution feedback from a complementary role, whereas the multi-agent system receives test or program outputs. This limits the attribution of gains specifically to cross-role collaboration rather than additional turns. Scaling to eight agents also replicates only two role types while increasing resources, and does not establish scalability to general workflows with heterogeneous roles. These issues constrain the interpretation of the gains and the method's scope.
Questions for authors
  1. Essential distillation baselines: Please provide role-conditioned multi-agent OPD without either RAS or PAC, completing a 2 x 2 ablation of the two modules. Please also include single-agent OPD with the same teacher, data, and total generation/token or training-compute budget. If the modules and multi-role structure still provide stable improvements under these controls, I would substantially raise my assessment.
  2. Meaning of the role advantage: Does RAS retain its effect under semantically equivalent role-prompt rewrites, removal of formatting tokens, or a common output format? Can you separately measure the Coder's code correctness and the Tester's test validity/error-detection ability on fixed inputs, rather than only measuring system performance after exchanging policies? These results would affect my assessment of responsibility specialization.
  3. Collaboration gains and cost: How much advantage remains when the single agent receives the same test/program-execution feedback and the total generation budget is matched? Please report actual generation volumes and end-to-end training time for ordinary OPD, MAS-OPD, and the main RL baselines, specifying whether both teacher scoring passes are included. Reporting only MAS-OPD's per-step cost does not establish its relative efficiency.
Reviewer_wrfJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes MAS-OPD, an on-policy distillation framework for jointly training collaborating language-model agents. Role-Advantage Specialization (RAS) contrasts teacher likelihoods under different role conditions to encourage specialization. Privileged Attribution for Coordination (PAC) provides the teacher with training-only verification results about previous interactions. Experiments on code generation and mathematical reasoning show improvements across six benchmarks.

Critical strengths
  1. The paper addresses a relevant problem and presents a straightforward framework for training specialized LLM agents with dense teacher supervision.

  2. The experiments cover six benchmarks, with five independent training runs for the main results. The consistent mean improvements and individual component ablations present encouraging evidence for the approach.

Critical weaknesses
  1. The approach imo doesn't obviously generalize beyond two agent role types. Although appendix E.1 increases the agent count to eight, it only replicates the same two roles. It remains unclear how contrasting roles should be selected when three or more distinct roles are involved, and whether the proposed pairwise training signal would support specialization across the team. Additionally, in settings where more than 2 agent types are involved, the existence of a decent verifier cannot be safely assumed.

  2. The baseline comparisons are incomplete. The paper does not include a standard role-conditioned multi-agent OPD setting with both RAS and PAC turned off. The ablations show that the proposed components can help under certain conditions, but without the basic OPD baseline, it is difficult to tell how much the full method adds beyond ordinary distillation. Strong RL baselines specifically designed for multi-agent systems, e.g., Stronger-MAS’s AT-GRPO, should also be included.

  3. The RAS signal may reflect both instruction compatibility and functional role preference. For example, when a Coder response is evaluated under the Tester condition, the teacher is instructed not to implement the final program, while the original task instructions still ask for an implementation. The likelihood difference could therefore be caused by this instruction conflict or by differences in expected output style, rather than by meaningful role specialization. Results in Figure 3 and Appendix G do not fully separate these possibilities.

Questions for authors
  1. What is PAC’s specific advance over existing on-policy context and privileged-information distillation, beyond supplying historical correctness information to the teacher?

  2. How accurate are the attribution labels produced by the verifier? Could the authors independently audit these labels and evaluate sensitivity to attribution errors?

  3. Does Table 4 include teacher scoring, reference verification, and attribution generation? Could the authors report comparable end-to-end training costs for the strongest baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.