- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work focuses on reinforcement learning (RL) for multi-agent systems, aiming to improve generalizability across different workflows. To achieve this, the authors propose UnityMAS-O, a framework that decouples workflow execution logic from the underlying RL training process. Specifically, the framework represents multi-agent workflows as directed acyclic graphs (DAGs), capturing execution dependencies and intermediate trajectories. Based on the constructed graphs, the framework performs role-aware credit assignment using predefined reward functions or verification mechanisms, supporting node-level, turn-level, and trajectory-level rewards. Experimental results on question-answering and code-generation benchmarks demonstrate the effectiveness of the proposed framework across different multi-agent workflows.
Critical strengths
- Reasonable graph-based workflow representation. Using a graph structure to capture the execution logic and dependencies within multi-agent workflows is a sound and reasonable design choice. Such a representation provides a structured basis for credit assignment across different agents and execution steps, potentially improving the effectiveness of RL training.
Critical weaknesses
-
Heavy reliance on manually designed intermediate rewards. The proposed framework relies heavily on task-specific intermediate reward functions or verification mechanisms, which limits its general applicability. One of the key motivations for applying RL to multi-agent systems is the difficulty of evaluating intermediate decisions and assigning credit to individual steps. If human experts can already design accurate intermediate rewards or reliable verifiers, existing RL approaches could potentially leverage these signals directly for effective optimization. Therefore, the reliance on manually specified reward functions raises concerns about the contribution and generalizability of the proposed credit assignment framework.
-
Insufficient experimental validation to isolate the framework’s contribution. The current experiments are insufficient to fully establish the effectiveness and generalizability of the proposed framework. In particular, some evaluated datasets, such as Natural Questions (NQ), are relatively simple for modern LLMs and may not adequately reflect the complexity of real-world agentic tasks. Evaluation on more challenging agent benchmarks, such as ALFWorld, would provide stronger evidence. Furthermore, the experiments rely on task-specific intermediate reward designs, making it difficult to distinguish whether the observed improvements primarily result from the proposed framework or from the effectiveness of these manually designed reward functions and verifiers.
Questions for authors
-
How does the proposed framework perform on more realistic and challenging agent benchmarks, such as ALFWorld, where agents must interact with environments and make sequential decisions?
-
Would the proposed framework still demonstrate similar improvements when using more generic intermediate reward functions or verification mechanisms, rather than benchmark-specific reward designs? How sensitive is its performance to the quality of these intermediate rewards?