acceptodds
Under review as a conference paper at ICLR 2027

SRPO: Setwise Relative Policy Optimization for Multi-Agent Systems

Abstract

Multi-agent systems enable complex reasoning and tool use by coordinating agents that divide roles and refine candidate solutions. Existing methods typically update individual agent responses or treat a complete trajectory as one training example. However, these methods may produce misleading policy updates because they assign the same final outcome to responses or trajectory segments that may play different roles in different team decisions. This is because treating each response as an independent update may separate outputs that jointly determine the next action, while treating an entire trajectory as one update may combine decisions made after different observations. These limitations call for a policy update defined at the level of a team decision, outputs that lead to the same state transition are optimized under a shared objective. In this paper, we propose Setwise Relative Policy Optimization (SRPO) for multi-agent systems. We represent the outputs used together to produce one state transition as an active set. A singleton set covers division of labor, while a larger set covers joint co-evolution, and the set composition can change across decisions. SRPO assigns a shared advantage to each active set, clips the combined policy change, and normalizes its scale according to the set size. This ties each update to the decision that produced the next state. Experiments on mathematical reasoning and multi-turn search demonstrate the effectiveness of SRPO across both tasks. Additional ablation studies analyze the normalization choice and training behavior under changing active-set sizes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, Reinforment learning, Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X1LHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SRPO for training LLM-based multi-agent systems. An active set consists of newly sampled outputs generated from the same logical pre-action state and consumed together by one transition, including a transition in shared orchestration memory. Sequential routing, verification, and aggregation are separate events. SRPO sums the sequence log-ratios within an event, divides by the square root of the active-set cardinality, exponentiates this quantity, and applies one clipped group-relative objective with a shared event advantage. It averages the loss over events. The theoretical analysis explains the variance and local score scale of square-root normalization under stated independence or covariance conditions. Evaluation covers mathematical reasoning with Qwen3-4B/8B and multi-turn search with Qwen2.5-3B/7B, reporting Avg@16 and Pass@16.

Critical strengths

The optimization unit is defined precisely enough to be useful in an implementation. Sections 3.1 and 4.3 distinguish simultaneous outputs from later decisions that observe them, include shared memory in the state, and require complete event records in each batch. This matters because joint updates can otherwise accidentally combine dependent outputs or omit a member. The explicit treatment of singleton router and aggregator events also makes the formulation applicable to mixed workflows.

Critical weaknesses
  1. The headline comparisons do not isolate an algorithmic improvement. Sections 5.1–5.2 and Tables 1–2 compare SRPO runs with published baseline values. Appendix A explicitly states that the baselines are cross-paper references rather than paired reruns. Matching metric definitions does not establish matching orchestration, training data processing, decoding, retrieval resources, or computation. As can be seen from the result, the 0.2-point macro gains are smaller than the reported SRPO seed standard deviations, while comparable baseline uncertainty is unavailable. Appendix A.1 measures changes within SRPO traces and tool calls, rather than a matched budget against competing methods.
  2. Table 3 compares three jointly clipped reductions and K = 1 versus K = 5, but contains no K = 5 agent-wise clipping control using the same advantage estimator and event weighting.
  3. No comparison for joint update (JointPPO) vs agent-wise update.
Questions for authors
  1. Which baseline results in Tables 1–2 are compatible with SRPO in training split, checkpoint selection, prompts, decoding settings, and retrieval configuration? Can you identify the differences that remain after harmonizing the metric definitions?
  2. Under the same orchestration and matched training and inference budgets, what are the results and paired uncertainty for SRPO versus the strongest baseline.
  3. At K = 5, how does SRPO compare with independent member clipping when the event advantage, event weighting, optimizer, and generated trajectories are held fixed? Does the benefit persist after tuning the clipping scale for each variant?
  4. hich exact event-reduction implementation produced each headline result, and how do those runs relate to the token-uniform and event-uniform diagnostics in Sections 5.4–5.5 and Appendix F?
Reviewer_mJpeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the granularity of policy updates in multi-agent large language model systems. The authors define an active set as responses generated from the same pre-action state that jointly produce one state transition. Singleton sets cover serial division of labor, while multi-member sets cover parallel generation. SRPO first computes the sequence log-ratio for each member's complete response, divides the sum within the set by the square root of the set size, and exponentiates the result to obtain a set score. It applies a single PPO-style clipping operation to this score, uses a shared event-level advantage, and averages the training objective over events. The authors provide variance and local Fisher-scale interpretations of square-root normalization and evaluate four model configurations on mathematical reasoning and multi-turn search. The main results report statistics over three seeds. Search ablations compare different normalization schemes at a fixed set size, with additional diagnostics of varying set sizes and training dynamics.

Critical strengths
  1. The paper provides a unified decision representation for different forms of multi-agent collaboration. Sections 3.1–3.2 define an active set as the agent outputs used together for one state transition. Singleton sets represent individual decisions in serial division of labor, while multi-member sets represent joint decisions in parallel collaboration, with set size allowed to vary over the course of interaction. This representation explicitly distinguishes joint outputs for the same decision from new decisions made at subsequent states, providing a basis for a unified policy optimization objective across collaboration structures. Section 4 further connects this representation to policy ratios over complete responses and clipping operations. The method is relatively clearly defined and straightforward to implement within existing PPO/GRPO training pipelines.

  2. Square-root normalization has an explicit mathematical basis and is supported by targeted ablation experiments. The two propositions in Section 3.3 analyze, respectively, the variance scale of the setwise log-ratio score and the relationship between the normalized local score and member Fisher information. Under the corresponding independence and variance assumptions, these results provide a theoretical basis for 1/K1/\sqrt{K} normalization. Table 3 further compares sum, mean, and square-root normalization in the same K=5 Search setting. Square-root normalization achieves 40.1% Avg@16, outperforming the sum at 38.6% and the mean at 34.7%. The theoretical analysis and this controlled comparison jointly support the choice of normalization.

Critical weaknesses
  1. The method design does not sufficiently address the misleading policy updates identified by the paper. The Introduction points out that an early incorrect response in serial multi-agent collaboration may be corrected by subsequent agents but still be positively reinforced because the team ultimately succeeds. The paper uses this example to explain how existing response-level updates may produce misleading learning signals. However, SRPO represents serial division of labor as singleton events with Ke=1K_e=1 and still computes event-level advantages from downstream team returns; it cannot identify the early response's actual contribution to the final success. For parallel events with Ke>1K_e>1, all responses in the set likewise share an event-level advantage, and a single clipping operation cannot distinguish different members' contributions to the team outcome. Thus, a clear mechanistic explanation is still missing for how setwise updates alleviate the reinforcement of incorrect responses illustrated by the paper. The two propositions in Section 3.3 establish the variance scale and local Fisher properties of square-root normalization, respectively, providing a mathematical basis for the normalization design. However, they do not establish that this design reduces misleading updates or improves policy optimization outcomes. The existing method analysis is not yet sufficient to support the paper's central motivation of addressing misleading policy updates.

  2. The core algorithm lacks a controlled performance comparison, and the existing experiments cannot attribute the reported gains to setwise updates. The GRPO and Dr. MAS baselines in Tables 1–2 are directly taken from other papers. Appendix A explicitly states that these results are not paired reruns under this paper's workflow. The methods may differ in agent organization, the number of parallel outputs, and training configurations, so the main tables cannot isolate the contribution of setwise optimization to final performance. In particular, on Math, SRPO's macro-average advantage over the strongest Dr. MAS baseline is only 0.2 percentage points, without comparable statistics across seeds to establish the stability of this advantage. Although Table 3 validates the advantage of square-root normalization over alternative normalization schemes at a fixed K=5, it lacks a control with independent response-level clipping under the same conditions. The K=1 versus K=5 comparison changes both the number of parallel outputs and the update unit, so the results in Table 3 also cannot validate the performance contribution of setwise clipping. Paired experiments using the same multi-agent workflow, model, advantage estimation, and training and inference budgets are needed to support the conclusion that the core algorithm outperforms existing response-level update methods.

Questions for authors
  1. The specific mechanism for addressing misleading policy updates: The Introduction illustrates the problem with existing response-level updates through a serial collaboration scenario in which an early incorrect response is corrected by subsequent agents but is nevertheless reinforced because the team ultimately succeeds. However, Section 3.2 represents serial division of labor as singleton events with Ke=1K_e=1, and Section 4.2 still computes event-level advantages from downstream team returns. In this scenario, through what mechanism does SRPO reduce the positive reinforcement of early incorrect responses? For events with Ke>1K_e>1, how do a shared advantage and a single clipping operation improve the learning signals received by different members? Please clarify whether "misleading updates" refers to inaccurate assignment of credit for rewards or to the lack of a unified constraint on policy changes for a joint decision, and provide a corresponding mechanistic analysis or evidence from training updates.

  2. The independent contribution of setwise clipping relative to response-level clipping: Could you compare independent per-response clipping with SRPO's setwise clipping within the same K=5 multi-agent workflow, keeping the model, routing and aggregation, rewards, advantage estimation, training budget, and inference budget fixed? Please report the corresponding task performance and results from repeated training runs, and explain the specific effects of setwise clipping on update behavior, such as the clipping fraction and changes in policy probabilities, to distinguish the contributions of setwise updates and square-root normalization.

  3. Comparability of the baselines in the main experiments: Appendix A explicitly states that the GRPO and Dr. MAS results in Tables 1–2 are taken from other papers and were not rerun as paired comparisons under SRPO's workflow. Please specify how these baselines differ from SRPO in the number of agents, serial or parallel collaboration structure, routing and aggregation, model parameter sharing, training data, and inference compute budget. Could you retrain at least one representative response-level optimization baseline under the same workflow and report comparable results across seeds? In particular, given the macro-average advantage of only approximately 0.2 percentage points over the strongest Dr. MAS baseline on Math, can the existing data support a conclusion of stable performance improvement?

Reviewer_ALinOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes Setwise Relative Policy Optimization (SRPO) for multi-agent systems. Outputs jointly causing one state transition are treated as an active set; they share one advantage, and their combined log-ratio is normalized and clipped once. Experiments on mathematical reasoning and multi-turn search report the highest displayed macro averages among the compared methods. The proposed solution does not solve the proposed problem fully. For example, Equation (7) derives each event advantage from the downstream team return. Thus, an early incorrect event followed by successful correction may still receive a positive advantage.

Critical strengths

The optimization unit is aligned with an environment transition. The active-set formulation gives a unified treatment of singleton and parallel multi-agent decisions and is clearly defined.

The normalization ablation is informative. On Search with K=5, square-root normalization obtains 40.1/56.1 Avg@16/Pass@16, compared with 38.6/54.3 without normalization and 34.7/48.6 with mean normalization.

Critical weaknesses

The proposed objective does not fully resolve the temporal credit-assignment problem used as motivation. Equation (7) still derives each event advantage from the downstream team return. Thus, an early incorrect event followed by successful correction may still receive a positive advantage.

The main comparisons do not isolate the contribution of the setwise objective. The appendix states that baseline values are cross-paper references rather than paired reruns, and there is no controlled comparison against response-level or trajectory-level optimization using the same workflow and compute. On Math, the macro improvement over the strongest displayed Dr. MAS result is only 0.2 points for both metrics at both model scales, while SRPO’s reported standard deviations range from 0.5 to 0.8 points and baseline uncertainty is unavailable.

Questions for authors

Clarification: For a trajectory where an early event is incorrect, a later event corrects it, and the final outcome is successful, what are the respective, grouping and advantages? How does Equation (7) prevent reinforcement of the early error?

Assessment-critical experiment: Can the authors compare SRPO with response-level clipping, an unnormalized joint likelihood-ratio objective, and trajectory-level optimization under identical orchestration, active-set sizes, training data, and compute? Without this control, it is difficult to attribute the reported gains specifically to the SRPO objective.

Comments on the paper

2 comments

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

AuthorsOpenReview ↗

General Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and reviewers ALin, mJpe, and X1LH for their careful assessments and constructive feedback. Our responses clarify SRPO's unified formulation and provide additional controlled experiments. We summarize the central contribution and the supporting evidence below.

1. One Policy Objective for Division of Labor and Joint Co-evolution · ALin, mJpe, X1LH

Core contribution. SRPO brings two major forms of multi-agent collaboration into a common decision-level optimization framework.

  • Unified representation. An active set contains outputs generated from the same pre-action state that jointly produce one state transition. Singleton sets represent serial division of labor, while multi-member sets represent joint co-evolution. The same objective therefore applies as a workflow moves between individual and joint decisions, with routing, verification, and aggregation represented as separate events when they observe a new state.
  • Unified optimization. SRPO combines the active members' log-ratios, normalizes the sum by the square root of the set size, and applies one clipped surrogate to the resulting setwise score. It retains the team-return advantage estimator and changes how the participating outputs enter the policy objective. The framework connects the structure of a team decision to the structure of its policy update.

This gives subsequent work a common foundation for improving coordination or advantage estimation across both collaboration regimes. The reviews recognized the precision and usefulness of the active-set formulation.

Supporting evidence: Sections 3–4 and responses to ALin (1/2), mJpe (1/3), and X1LH (3/4–4/4).

2. Additional Experiments Support the Objective Under Matched Conditions · ALin, mJpe, X1LH

Concern. Does SRPO improve performance when workflow, collaboration size, and computational budgets are held fixed?

We reported an additional comparison at K = 5, holding the model, workflow, training data, routing and aggregation, reward and advantage estimation, and training and inference budgets fixed.

MethodAvg@16Pass@16
Independent response-level clipping39.0 ± 0.554.7 ± 0.7
Trajectory-level clipping38.2 ± 0.853.5 ± 1.1
JointPPO sum38.7 ± 0.454.4 ± 0.5
JointPPO mean34.7 ± 0.748.7 ± 1.1
SRPO40.1 ± 0.556.0 ± 0.8

SRPO achieves the highest mean performance on both metrics. It improves over independent response-level clipping by 1.1/1.3 percentage points in Avg@16/Pass@16 and over trajectory-level clipping by 1.9/2.5 points. Because all variants use the same K = 5 workflow, these improvements are obtained without adding parallel responses or increasing the allocated training and inference budgets.

The new comparison directly addresses the missing same-workflow controls identified in the reviews. It provides evidence for the complete SRPO objective under matched conditions, complementing the published reference results in the original main tables.

Supporting evidence: Responses to ALin (2/2), mJpe (2/3), and X1LH (1/4–2/4).

3. The Joint-Update Controls Support Square-Root Normalization · mJpe, X1LH

Concern. What does normalization contribute within the joint-update formulation?

  • Controlled evidence. Among the jointly clipped variants, SRPO improves over JointPPO sum by 1.4/1.6 points and JointPPO mean by 5.4/7.3 points in Avg@16/Pass@16. These variants share the joint-update structure, making their comparison informative about the normalization choice.
  • Connection to the analysis. Section 3.3 studies the variance and local Fisher scale of the normalized score under stated assumptions. The additional results complement this analysis by showing that square-root normalization performs best among the tested joint reductions in the fixed K = 5 setting.

The controls answer two complementary questions. Independent clipping versus SRPO evaluates the full proposed objective. Comparisons among joint sum, mean, and square-root variants evaluate normalization within joint updating. Together, they support the combination of event-level joint clipping and square-root normalization.

Supporting evidence: Section 3.3 and responses to mJpe (2/3) and X1LH (2/4–3/4).


SRPO's central contribution is a unified optimization framework for division of labor and joint co-evolution, supported by controlled evidence for its complete objective and normalization design. We hope this summary helps the Area Chair and reviewers assess the contribution alongside the additional results.

We would be glad to clarify any remaining questions. Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.