- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RMAP, a lightweight framework for runtime multi-agent topology planning. Instead of determining the complete communication graph before execution, RMAP progressively constructs a DAG conditioned on intermediate execution results. It first predicts edges from the current layer to candidate agent roles, thereby determining the next layer, and then adds optional connections from earlier layers. Agent representations progressively combine role descriptions, structural context, and execution outputs. The planner is trained through supervised learning on successful workflows, followed by reinforcement learning to reduce execution costs. Experiments cover seven benchmarks and four executor backbones, with additional ablation, transferability, and overhead analyses.
Critical strengths
- A practical formulation with a lightweight implementation. Adapting agent composition and information flow to intermediate results is practically useful, particularly for tasks involving verification and revision. RMAP provides a clear decomposition into next-layer instantiation, cross-layer dependency selection, and execution. Its measured planning overhead suggests that runtime adaptation can be implemented without an additional expensive LLM-based planning call at every step.
- Broad empirical coverage with encouraging results. The evaluation includes multiple task domains and executor backbones, as well as component ablations, cross-dataset and cross-backbone transfer, and post-training efficiency analyses. RMAP achieves the highest reported mean performance in 26 of 28 main-table settings, with substantial improvements in some cases. These results provide encouraging evidence of practical utility, although they do not fully isolate the source of the gains.
Critical weaknesses
- The specified decoder substantially restricts feedback-conditioned role selection. Equation 6 defines candidate embeddings using role descriptions alone. Combined with the linear concatenation decoder in Equation 20, the score can be written as logitp(u,v)=a^⊤ h_u+b^⊤ h_v. Consequently, the relative ranking of two available candidate roles is independent of the query and execution feedback. Feedback can change the number of selected roles or trigger termination, but cannot reverse the relative preference between two candidates while both remain available. This restriction is consequential because state-dependent agent selection is central to the paper’s motivation. The concern applies to the architecture as specified; if the implementation includes additional source–candidate interactions or context-conditioned candidate embeddings, these need to be documented.
- The contribution over existing dynamic orchestration methods and its practical benefits are insufficiently isolated. Feedback-conditioned agent and context selection already appear in AnyMAC, while DyLAN already uses intermediate responses to select agents for subsequent layers. AgentConductor further combines execution-feedback-driven topology adaptation with layered DAGs and parallel execution. RMAP therefore needs a more precise account of the contribution of its incremental layerwise construction and lightweight parameterization. The current comparisons do not adequately separate these benefits from differences in realized computation: a shared per-role invocation cap does not match actual agent calls or token usage. Moreover, low planner overhead does not establish an end-to-end parallelism advantage, especially since Section 4.1 waits for the entire current layer to finish before planning the next one.
Questions for authors
- Decoder clarification. Is Equation 20 the complete implemented decoder, and are candidate embeddings independent of the current execution state as specified in Equation 6? If so, how does the policy express different relative preferences between two available roles across execution states?
- Contribution and efficiency clarification. What evidence isolates the benefit of layerwise planning relative to sequential runtime routing under comparable realized computation? Please clarify the actual agent-call and token costs of the main baselines, whether the reported execution times represent individual query latency or amortized batch time, and whether successors can begin before all agents in the preceding layer finish.
- Evaluation protocol clarification. Please specify the train/test partition and filtering procedure for each benchmark, including the number of successful training trajectories retained. For code-generation tasks, which tests are available to CodeTester during inference, and how are these separated from the tests used for final scoring?
- Code Implementation Could the authors clarify how the role selector changes its preference between candidate roles based on execution feedback? In the released code, fixed role-description embeddings and two linear layers without an intervening nonlinearity appear to make the relative ranking independent of feedback.
3 comments
Official Comment by Authors
Dear AC and Reviewers,
We sincerely appreciate the time and effort you have devoted to carefully evaluating our paper.
As Author- Reviewer Discussion will conclude soon, we would be grateful if you could take a moment to review our responses. We would also greatly appreciate any additional feedback or further discussion regarding any remaining concerns.
Thank you again for your valuable time and consideration.
Best regards, The Authors
Request for Further Feedback
Dear Reviewer iZxZ,
The author–reviewer discussion period is approaching its end (with two days remaining). We would greatly appreciate any further feedback or comments you may have.
Best regards, Authors
Rebutall from authors
Q1 Q4 W1: Decoder clarification
(1) Eq. 20 specifies the complete decoder.
(2) Although the concatenated embeddings of an executed agent and a candidate agent are linearly mapped in Eq. 20, the executed agent's embedding is nonlinearly updated to incorporate system information, including its description, execution outputs, and topological context (Eq. 8, 10, 11). Thus, the overall mapping from execution context to edge probabilities is nonlinear.
In other words, the decoder considers not only the static candidate semantics, but also the dynamic context progressively accumulated during layer-wise execution.
For example, in Fig. 3, the embedding of query initially contains only input semantics, whereas the embedding of ProgrammingExpert incorporates its execution-related information. Therefore, the decoder predicts different agents for layer1 and layer3.
(3) Following prior works [1–3] and stated in line 205-207 of the paper, RMAP adopts an inductive bias that candidate connections are independent without direct pairwise comparison. The inductive bias is practical because a real-world MAS selects agents based on task requirements rather than relative rankings. Accordingly, the decoder estimates the probability of each candidate edge.
Q2 W2:
(1) DAG layer-wise execution. DAG-structured MASs [1, 2, 3] typically require layer-wise execution, which is not specific to RMAP. In this way, they can benefit from an asynchronous environment with time efficiency improved greatly (Anymac is a counterexample).
(2) Novelty compared to mentioned baselines. As stated in the paper, a desirable DAG-based MAS designer should not only support layer-wise execution but also adapt to evolving execution states and inter-agent semantic interactions throughout runtime execution.
However, Anymac adopts sequential agent selection, which does not support layer-wise execution. AgentConductor focuses on LLM-based topology adaptation across turns, which still generates a fixed graph before execution. DyLAN focuses on agent selection within predefined layers and adopts fully connected connections between adjacent layers, rather than optimizing a DAG-formed MAS. Essentially, they do not consider the runtime requirements of layer-wise execution like us.
(3) Comparison protocol. In MAS design domain, strictly controlling token and time cost is challenging because compared workflows involve different workflows with multiple agents and edges. Therefore, following traditional works [1, 2, 3], the comparison is conducted over a fixed scale of the graph. We acknowledge that the setting does not strictly equalize the budget, and thus additionally report the actual time and token costs. LLM is GPT-4o-mini and RMAP is only pre-trained.
Token cost per execution:
Time cost per execution:
The results demonstrate: (i) Even before RL post-training, RMAP achieves comparable inference costs to baselines. Thus, our performance gains are not driven by substantially increased overhead. (ii) Compared with Anymac's one-by-one generation, RMAP's layer-wise generation is much more time-efficient.
(4) In Section 5.5, we report execution times represent individual query latency.
(5) In DAG-formed MASs [1,2,3], successors must wait before pre-layer execution.
Q3:
(1) Train/test split. Please refer to Appendix B. This keeps the same for fairness.
(2) The number of training trajectories. ARG-Designer and Anymac also require supervised training. For fairness, they also use the supervised training data generated by RMAP. The number of successful trajectories is min(300, len(train_dataset)). Moreover, Baselines follow their default setting to ensure sufficient training. For example, following G-Designer, RL-based designers have 200 epochs, each with 4 samples.
(3) CodeTester cannot access ground-truth tests, but generate tests with assert itself. Baselines can also employ CodeTester.
REFERENCES
[1] GPTSwarm: Language Agents as Optimizable Graphs
[2] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks
[3] Cut the Crap: An Economical Communication Pipeline For LLM-Based Multi-Agent Systems