acceptodds
Under review as a conference paper at ICLR 2027

UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems

Abstract

Large language model (LLM) multi-agent systems are usually assembled from prompts, tools, and hand-written control logic, while reinforcement learning (RL) is applied to one policy at a time. This mismatch makes it difficult to optimize intermediate roles, compare parameter-sharing choices, or reuse training infrastructure across workflows. We present UnityMAS-O, a general RL optimization framework that treats a user-defined multi-agent workflow as the unit of optimization. The framework exposes four first-class objects: logical roles, graph-structured trajectories, explicit role-to-model mappings, and role-, turn-, or trajectory-level rewards. A Ray-based controller executes arbitrary workflows and routes invocations to model-local worker groups built on vLLM rollout and PPO training. The same interface supports fully shared, partially shared, and fully separated parameters. We train and evaluate multiple workflows on retrieval-augmented QA and executable code tasks, demonstrating the effectiveness of UnityMAS-O across diverse multi-agent settings. We further explore multi-agent optimization under shared and independent parameter settings, examining their differences in task performance and training dynamics. UnityMAS-O provides a reusable framework for turning manually designed LLM multi-agent systems into trainable multi-agent policies.

Primary area:
reinforcement learning
Keywords:
multi-agent RL, LLM-based multi-agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_avKtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work focuses on reinforcement learning (RL) for multi-agent systems, aiming to improve generalizability across different workflows. To achieve this, the authors propose UnityMAS-O, a framework that decouples workflow execution logic from the underlying RL training process. Specifically, the framework represents multi-agent workflows as directed acyclic graphs (DAGs), capturing execution dependencies and intermediate trajectories. Based on the constructed graphs, the framework performs role-aware credit assignment using predefined reward functions or verification mechanisms, supporting node-level, turn-level, and trajectory-level rewards. Experimental results on question-answering and code-generation benchmarks demonstrate the effectiveness of the proposed framework across different multi-agent workflows.

Critical strengths
  1. Reasonable graph-based workflow representation. Using a graph structure to capture the execution logic and dependencies within multi-agent workflows is a sound and reasonable design choice. Such a representation provides a structured basis for credit assignment across different agents and execution steps, potentially improving the effectiveness of RL training.
Critical weaknesses
  1. Heavy reliance on manually designed intermediate rewards. The proposed framework relies heavily on task-specific intermediate reward functions or verification mechanisms, which limits its general applicability. One of the key motivations for applying RL to multi-agent systems is the difficulty of evaluating intermediate decisions and assigning credit to individual steps. If human experts can already design accurate intermediate rewards or reliable verifiers, existing RL approaches could potentially leverage these signals directly for effective optimization. Therefore, the reliance on manually specified reward functions raises concerns about the contribution and generalizability of the proposed credit assignment framework.

  2. Insufficient experimental validation to isolate the framework’s contribution. The current experiments are insufficient to fully establish the effectiveness and generalizability of the proposed framework. In particular, some evaluated datasets, such as Natural Questions (NQ), are relatively simple for modern LLMs and may not adequately reflect the complexity of real-world agentic tasks. Evaluation on more challenging agent benchmarks, such as ALFWorld, would provide stronger evidence. Furthermore, the experiments rely on task-specific intermediate reward designs, making it difficult to distinguish whether the observed improvements primarily result from the proposed framework or from the effectiveness of these manually designed reward functions and verifiers.

Questions for authors
  1. How does the proposed framework perform on more realistic and challenging agent benchmarks, such as ALFWorld, where agents must interact with environments and make sequential decisions?

  2. Would the proposed framework still demonstrate similar improvements when using more generic intermediate reward functions or verification mechanisms, rather than benchmark-specific reward designs? How sensitive is its performance to the quality of these intermediate rewards?

Reviewer_bUniOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • UnityMAS-O is an RL framework where a user-defined multi-agent workflow is the unit of optimization. The workflow is a DAG of trainable LLM roles and tool nodes. Users give the roles, the graph, a role-to-model map phi (full / partial / no parameter sharing), and node-, turn- and trajectory-level reward terms.
  • A Ray-based star runtime has a central controller that runs the graph, calls tools and commits rewards, and model-local worker groups that only get the invocations mapped to their model.
  • Experiments: retrieval QA on NQ and HotpotQA with three workflow templates and Qwen2.5 from 0.5B to 14B, and a reflective planner-coder-verifier-reflector code workflow with 3x Qwen3-4B / 3x Qwen3-8B. Headline numbers: NQ F1 62.1 / HotpotQA 57.3 for Qwen2.5-7B, code "strict all-passed" 0.255 -> 0.686 and 0.290 -> 0.738, and a shared-vs-independent parameter study.
  • The paper positions itself as a general alternative to GRPO-style grouped methods: PPO with model-local buffers needs no same-state candidate groups.
Critical strengths
  • The abstraction is clean and useful. Separating logical roles from physical models, making the role-to-model map an experimental variable, and routing each reward back to the exact invocation is a sensible way to build MAS-RL infrastructure. Appendix D and E are concrete about configs and the launch path.
  • The parameter-sharing study asks a good question, and the finding that sharing is fine for homogeneous short QA roles but hurts heterogeneous code roles is plausible and matches what STRONGER-MAS reported for shared vs per-role policies.
  • The before/after sweeps across five model sizes and three QA templates at least show the pipeline runs end to end at some scale.
Critical weaknesses
  • The evidence does not support a method claim. Almost every result is "untrained workflow vs the same workflow after RL", which any RL algorithm produces, and the starting points are very low, so gains are inflated. There is no controlled comparison of UnityMAS-O's PPO/GAE against AT-GRPO, MHGPO, Dr. MAS or MMOA-RAG on the same workflow, same data and same backbone, which is exactly what STRONGER-MAS did against MAPoRL / MARFT / CURE on matched base models. Appendix C admits "the public baseline protocols are not fully matched".
  • Table 3 compares gains across different tasks, metrics and models and draws conclusions from it. Saying +43.1 pp "all-passed" on a pooled LCB v5 + Codeforces set with 3x Qwen3-4B "exceeds STRONGER-MAS's largest source gain (5.1 pass@1 points)" on LiveCodeBench with one Qwen3-8B is not a meaningful comparison; same for the NQ / HotpotQA rows. This kind of table should not be in the main text.
  • The code metric may leak the tests. In the reflective workflow the verifier tool runs the candidate and returns pass count, pass rate and error text, the loop repeats up to 3 times and stops when all tests pass, and the reported metric is "all available tests pass". If the verifier at evaluation time runs the same tests used for the final score, the agent gets oracle test feedback and three tries, and 0.686 / 0.738 all-passed for 4B / 8B models would be an artifact. The paper must say which tests the verifier sees at evaluation.
  • Reproducibility and data-integrity issues. The reproducibility statement says the anonymous repo "provides the ... reward components, training configurations, and scripts", but Appendix C says "configuration and reward code are not included in this submission snapshot". Table 6 has what look like copy-paste errors: the QD-Evidence-Answer 3B row is identical for NQ and HotpotQA (0.107 -> 0.473, +0.366, +342%), and "Before" = 0.279 appears three times (NQ 7B, NQ 14B, HotpotQA 14B). Also, recent MAS credit-assignment work (MATPO, M-GRPO, SHARP, GiGPO) is not discussed, and STRONGER-MAS / Dr. MAS / MHGPO appear only through copied numbers.
Questions for authors
  • At evaluation time, which tests does the verifier execute inside the reflective loop, and are they the same tests used to compute "all-passed"? Please also report standard LCB v5 pass@1 or the trained coder so it can be compared to the DeepCoder model card and the Qwen3 report.
  • Can you train AT-GRPO (or GRPO with agent / turn grouping) and your PPO on the same QD-Answer or code workflow inside UnityMAS-O, same data and backbone, and report both? Without this the "graph-first alternative to grouping" argument in Appendix G is only a design argument.
  • Please check Table 6: the QD-Evidence-Answer 3B row is identical for NQ and HotpotQA, and "Before" = 0.279 repeats three times. Typos or real values?
  • Which statement about the released code is correct, the reproducibility statement (reward components and configs included) or Appendix C (not included)?
  • Please ablate the reward contract (node only, trajectory only, node + trajectory, full three-scale) on at least one QA workflow and the code workflow. Does the turn-level term actually matter?
Reviewer_HSKKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UnityMAS-O, a RL optimization framework for LLM multi-agent workflows. It represents workflows as directed acyclic graphs, explicitly maps logical roles to physical models, and supports node-, turn-, and trajectory-level rewards. A centralized Ray-based controller coordinates asynchronous workflow execution, while model-local workers perform rollout and PPO-style optimization. The framework supports shared, partially shared, and independent policies. Experiments on retrieval QA and executable code workflows show substantial improvements over untrained workflow initializations.

Critical strengths
  1. The paper addresses a practical problem: existing multi-agent workflows combine manually designed orchestration with RL implementations that are often tied to specific training setups. A configurable workflow-level interface could reduce engineering overhead and facilitate systematic experiments.
  2. The separation between logical roles, workflow structure, model assignments, and reward contracts is conceptually clean. Supporting shared and independent policies under the same workflow definition is useful for studying parameter efficiency and role specialization.
  3. The experiments cover retrieval QA and executable code workflows, and the results suggest that RL can substantially improve multi-agent systems.
Critical weaknesses
  1. The algorithmic novelty is limited or insufficiently established. The framework combines DAG-based workflow execution, configurable reward assignment, model routing, and PPO-style updates. But the paper does not convincingly demonstrate which aspects are essential beyond providing a unified software interface. A more direct comparison against existing general-purpose LLM-MARL frameworks would strengthen the contribution.
  2. The evaluation covers two QA datasets and one executable-code workflow. Although these settings are useful, evidence for the framework’s generality across more diverse workflows, tools, and model configurations remains limited.
  3. Reproducibility needs clarification. The paper notes that configuration and reward code are absent from the submission snapshot. More complete training configurations, reward definitions, and multi-seed results would improve confidence in the reported gains.
Questions for authors
  1. Can the authors provide a controlled comparison with an existing general-purpose LLM-MARL framework using the same workflow, model, reward function, and training budget?
  2. What are the individual contributions of node-level, turn-level, and trajectory-level rewards? Does role-aware reward attribution outperform assigning the same final reward to all trainable invocations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.