acceptodds
Under review as a conference paper at ICLR 2027

From Static to Dynamic: Towards Runtime Multi-Agent Topology Planning via Progressive Next Layer Generation

29 views

Abstract

Large language model (LLM)-driven agents are designed to handle a wide range of tasks autonomously. As tasks become increasingly complex, the multi-agent system (MAS) becomes a promising solution, where the underlying communications are typically represented as a directed acyclic graph. To autonomously construct a desired MAS topology, various design strategies are proposed based on graph-informed techniques. However, these frameworks primarily follow the "design-then-deploy" paradigm. As a result, the generated graph-structure MAS cannot satisfy runtime requirements and adapt to the dynamic environment. To break the constraints of static designs, our key insight is that the dynamic planning of interaction topology for MAS is inherently an autoregressive next layer generation problem, where nodes are agents and connections are information dependencies. Building upon this motivation, we propose a Runtime Multi-Agent topology Planning scheme named RMAP. Specifically, to capture the evolving semantics of agents during runtime topology construction, RMAP progressively integrates three agentic semantics, including role descriptions, topological context, and execution outputs, into unified task-oriented representations. To enable layer-wise execution, RMAP decomposes topology prediction into two stages: first generating the leaf agents of the current layer, followed by supplementing optional connections between these leaf agents and their ancestor agents. Experimental results show that our scheme improves task performance in 92.8% of scenarios, and can be easily implemented on personal computers with less than 1 second of additional latency. The code is available at https://anonymous.4open.science/r/RMAP-1884.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph-Structured Multi-Agent Systems, Runtime Topology Planning, Autoregressive Next Layer Generation, Progressive Agent Representation, Asynchronous Workflow Execution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iZxZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RMAP, a lightweight framework for runtime multi-agent topology planning. Instead of determining the complete communication graph before execution, RMAP progressively constructs a DAG conditioned on intermediate execution results. It first predicts edges from the current layer to candidate agent roles, thereby determining the next layer, and then adds optional connections from earlier layers. Agent representations progressively combine role descriptions, structural context, and execution outputs. The planner is trained through supervised learning on successful workflows, followed by reinforcement learning to reduce execution costs. Experiments cover seven benchmarks and four executor backbones, with additional ablation, transferability, and overhead analyses.

Critical strengths
  1. A practical formulation with a lightweight implementation. Adapting agent composition and information flow to intermediate results is practically useful, particularly for tasks involving verification and revision. RMAP provides a clear decomposition into next-layer instantiation, cross-layer dependency selection, and execution. Its measured planning overhead suggests that runtime adaptation can be implemented without an additional expensive LLM-based planning call at every step.
  2. Broad empirical coverage with encouraging results. The evaluation includes multiple task domains and executor backbones, as well as component ablations, cross-dataset and cross-backbone transfer, and post-training efficiency analyses. RMAP achieves the highest reported mean performance in 26 of 28 main-table settings, with substantial improvements in some cases. These results provide encouraging evidence of practical utility, although they do not fully isolate the source of the gains.
Critical weaknesses
  1. The specified decoder substantially restricts feedback-conditioned role selection. Equation 6 defines candidate embeddings using role descriptions alone. Combined with the linear concatenation decoder in Equation 20, the score can be written as logitp(u,v)=a^⊤ h_u+b^⊤ h_v. Consequently, the relative ranking of two available candidate roles is independent of the query and execution feedback. Feedback can change the number of selected roles or trigger termination, but cannot reverse the relative preference between two candidates while both remain available. This restriction is consequential because state-dependent agent selection is central to the paper’s motivation. The concern applies to the architecture as specified; if the implementation includes additional source–candidate interactions or context-conditioned candidate embeddings, these need to be documented.
  2. The contribution over existing dynamic orchestration methods and its practical benefits are insufficiently isolated. Feedback-conditioned agent and context selection already appear in AnyMAC, while DyLAN already uses intermediate responses to select agents for subsequent layers. AgentConductor further combines execution-feedback-driven topology adaptation with layered DAGs and parallel execution. RMAP therefore needs a more precise account of the contribution of its incremental layerwise construction and lightweight parameterization. The current comparisons do not adequately separate these benefits from differences in realized computation: a shared per-role invocation cap does not match actual agent calls or token usage. Moreover, low planner overhead does not establish an end-to-end parallelism advantage, especially since Section 4.1 waits for the entire current layer to finish before planning the next one.
Questions for authors
  1. Decoder clarification. Is Equation 20 the complete implemented decoder, and are candidate embeddings independent of the current execution state as specified in Equation 6? If so, how does the policy express different relative preferences between two available roles across execution states?
  2. Contribution and efficiency clarification. What evidence isolates the benefit of layerwise planning relative to sequential runtime routing under comparable realized computation? Please clarify the actual agent-call and token costs of the main baselines, whether the reported execution times represent individual query latency or amortized batch time, and whether successors can begin before all agents in the preceding layer finish.
  3. Evaluation protocol clarification. Please specify the train/test partition and filtering procedure for each benchmark, including the number of successful training trajectories retained. For code-generation tasks, which tests are available to CodeTester during inference, and how are these separated from the tests used for final scoring?
  4. Code Implementation Could the authors clarify how the role selector changes its preference between candidate roles based on execution feedback? In the released code, fixed role-description embeddings and two linear layers without an intervening nonlinearity appear to make the relative ranking independent of feedback.
3 comments
AuthorsOpenReview ↗

Rebutall from authors

Q1 Q4 W1: Decoder clarification

(1) Eq. 20 specifies the complete decoder.

(2) Although the concatenated embeddings of an executed agent and a candidate agent are linearly mapped in Eq. 20, the executed agent's embedding is nonlinearly updated to incorporate system information, including its description, execution outputs, and topological context (Eq. 8, 10, 11). Thus, the overall mapping from execution context to edge probabilities is nonlinear.

In other words, the decoder considers not only the static candidate semantics, but also the dynamic context progressively accumulated during layer-wise execution.

For example, in Fig. 3, the embedding of query initially contains only input semantics, whereas the embedding of ProgrammingExpert incorporates its execution-related information. Therefore, the decoder predicts different agents for layer1 and layer3.

(3) Following prior works [1–3] and stated in line 205-207 of the paper, RMAP adopts an inductive bias that candidate connections are independent without direct pairwise comparison. The inductive bias is practical because a real-world MAS selects agents based on task requirements rather than relative rankings. Accordingly, the decoder estimates the probability of each candidate edge.

Q2 W2:

(1) DAG layer-wise execution. DAG-structured MASs [1, 2, 3] typically require layer-wise execution, which is not specific to RMAP. In this way, they can benefit from an asynchronous environment with time efficiency improved greatly (Anymac is a counterexample).

(2) Novelty compared to mentioned baselines. As stated in the paper, a desirable DAG-based MAS designer should not only support layer-wise execution but also adapt to evolving execution states and inter-agent semantic interactions throughout runtime execution.

However, Anymac adopts sequential agent selection, which does not support layer-wise execution. AgentConductor focuses on LLM-based topology adaptation across turns, which still generates a fixed graph before execution. DyLAN focuses on agent selection within predefined layers and adopts fully connected connections between adjacent layers, rather than optimizing a DAG-formed MAS. Essentially, they do not consider the runtime requirements of layer-wise execution like us.

(3) Comparison protocol. In MAS design domain, strictly controlling token and time cost is challenging because compared workflows involve different workflows with multiple agents and edges. Therefore, following traditional works [1, 2, 3], the comparison is conducted over a fixed scale of the graph. We acknowledge that the setting does not strictly equalize the budget, and thus additionally report the actual time and token costs. LLM is GPT-4o-mini and RMAP is only pre-trained.

Token cost per execution:

MethodMMLUSVAMPAQuAHumanEvalMBPPDS1000AsyncHow
G-Designer3442674181436963523246485641
ARG-Designer2488536257614783470056015892
Anymac2602544360594946447453766243
RMAP2643487361514643411051566083

Time cost per execution:

MethodMMLUSVAMPAQuAHumanEvalMBPPDS1000AsyncHow
G-Designer11.923.427.025.423.320.220.8
ARG-Designer9.318.421.619.821.624.321.6
Anymac13.228.628.423.625.535.333.9
RMAP9.717.023.120.419.422.722.3

The results demonstrate: (i) Even before RL post-training, RMAP achieves comparable inference costs to baselines. Thus, our performance gains are not driven by substantially increased overhead. (ii) Compared with Anymac's one-by-one generation, RMAP's layer-wise generation is much more time-efficient.

(4) In Section 5.5, we report execution times represent individual query latency.

(5) In DAG-formed MASs [1,2,3], successors must wait before pre-layer execution.

Q3:

(1) Train/test split. Please refer to Appendix B. This keeps the same for fairness.

(2) The number of training trajectories. ARG-Designer and Anymac also require supervised training. For fairness, they also use the supervised training data generated by RMAP. The number of successful trajectories is min(300, len(train_dataset)). Moreover, Baselines follow their default setting to ensure sufficient training. For example, following G-Designer, RL-based designers have 200 epochs, each with 4 samples.

(3) CodeTester cannot access ground-truth tests, but generate tests with assert itself. Baselines can also employ CodeTester.

REFERENCES

[1] GPTSwarm: Language Agents as Optimizable Graphs

[2] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks

[3] Cut the Crap: An Economical Communication Pipeline For LLM-Based Multi-Agent Systems

AuthorsOpenReview ↗

Official Comment by Authors

Dear AC and Reviewers,

We sincerely appreciate the time and effort you have devoted to carefully evaluating our paper.

As Author- Reviewer Discussion will conclude soon, we would be grateful if you could take a moment to review our responses. We would also greatly appreciate any additional feedback or further discussion regarding any remaining concerns.

Thank you again for your valuable time and consideration.

Best regards, The Authors

AuthorsOpenReview ↗

Request for Further Feedback

Dear Reviewer iZxZ,

The author–reviewer discussion period is approaching its end (with two days remaining). We would greatly appreciate any further feedback or comments you may have.

Best regards, Authors

Reviewer_94g8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RMAP builds a multi-agent communication graph at runtime, one layer at a time: after each layer executes, a lightweight encoder combines role descriptions, the topology so far, and agent outputs, and two predictors choose the next layer's agents and their links to earlier layers. It is pre-trained on LLM-generated graphs that solved their queries, can be post-trained with reinforcement learning, and reports the best mean in 26 of 28 settings across seven benchmarks and four executor LLMs, with under one second of planning overhead.

Critical strengths
  1. Layer-wise runtime planning is a clean formulation. Building the communication graph one layer at a time, with each later layer chosen after the previous one executes, separates choosing the next layer's agents from adding optional cross-layer links; this preserves topological order by construction, lets the agents within a layer execute asynchronously, and lets the topology respond to intermediate outputs rather than being fixed before execution.

  2. The evaluation is broad, and removing the execution outputs consistently lowers performance. RMAP is evaluated across a broad set of benchmarks, executor LLMs, and topology designers and achieves the best mean in most settings, and removing the execution outputs lowers performance in every setting, evidence that the output information contributes.

Critical weaknesses
  1. Neither the benefit of runtime adaptation over a matched static design nor its comparative cost is established. The paper's central idea is that the topology should adapt at runtime to intermediate execution outputs. The no-output ablation shows that these outputs help, but no matched static control or same-query intervention isolates the benefit of changing the topology in response to them, and much of RMAP's advantage persists without them: the no-output variant still beats the best baseline in 10 settings plus one tie, including all four DS1000 settings, compared with 26 for RMAP (assuming comparable runs across Tables 2 and 3). The training graphs are also proposed in full before execution, with only successful graphs kept (Appendix E), so it is unclear whether the training data contain topologies that respond to intermediate outcomes. Comparative cost is unmeasured as well: Table 2 reports no tokens, LLM calls, graph sizes, or latency for any method, and the claimed asynchrony advantage over Anymac (Section 2) is not tested.

  2. Prior adaptive designers narrow the novelty claim. Section 1 says previous designers mostly fix a template before execution, but SelfOrg (Tastan et al., 2026, "Stochastic Self-Organization in Multi-Agent Systems") already builds and updates an agent DAG from agents' responses during collaboration. CARD (Wu et al., 2026, "CARD: Towards Conditional Design of Multi-agent Topological Structures") conditions topologies on environment changes such as model or tool updates, and neither work is cited. The paper itself also presents Anymac as runtime planning that generates agents one by one (Section 2). The clearest distinct contribution therefore appears to be the feedback-conditioned, layer-wise parameterization.

Questions for authors
  1. Decoder: Equation 20 scores each parent–candidate pair linearly over their concatenated embeddings, and Equation 6 makes candidate embeddings depend only on role descriptions, so, under the 0.5 threshold, each step selects from one fixed ranking of eligible roles; yet Figure 3 selects ProjectManager over CodeTester at the first layer and CodeTester over ProjectManager at the third. Does the implementation add any candidate-dependent mask, nonlinear interaction, or other rule beyond Equation 20 and the 0.5 threshold to produce these choices?
  2. Runtime contribution: How does RMAP compare with a fully matched static variant and with SelfOrg, and does the topology change when intermediate outcomes differ for the same query?
  3. Fair comparison: Beyond the common train/test split and execution settings, what method-specific supervision, search budget, and tuning did each baseline receive, and what are each method's tokens, LLM calls, and wall-clock times, including for the claimed asynchrony advantage over Anymac (Section 2)?
  4. Statistics: Several of the 26 wins fall within one standard deviation, for example 93.0 ± 1.0 against 92.6 ± 1.3 on SVAMP with GPT-4o-mini. What varies across the five runs, and do paired tests support the close wins?
  5. Code evaluation: Which tests can CodeTester access at inference, and how are they kept apart from grading tests? The paper does not say, and access to grading tests could inflate the code results.
4 comments
AuthorsOpenReview ↗

Rebuttal from authors

Q1: Decoder clarification.

Although the concatenated embeddings of an executed agent and a candidate agent are linearly mapped, the executed agent's embedding is nonlinearly updated to incorporate system information, including its description, execution outputs, and topological context (Eq. 8, 10, 11). Thus, the overall mapping from execution context to edge probabilities is nonlinear.

That is, the decoder considers not only static candidate semantics, but also dynamic context progressively accumulated during layer-wise execution.

For example, in Fig. 3, the embedding of query contains only input semantics, whereas the embedding of ProgrammingExpert incorporates execution-related information. Therefore, the decoder predicts different agents for layer1 and layer3.

W2 Q2:

(1) Novelty compared to mentioned works. SelfOrg considers pre-round outputs and CARD considers additional conditions, they still generate a static graph before execution. Anymac is a runtime designer, but its one-by-one planning still not practical for a DAG-formed MAS (please refer to time report). Essentially, they do not consider the runtime requirements of layer-wise execution like us.

Thus, our claim of “previous designers mostly fix a template before execution” is reasonable.

(2) The w/o output RMAP variant is a static version without runtime feedback.

(3) Compare to SelfOrg with GPT-4o-mini:

MethodMMLUSVAMPAQUAHumanEvalMBPPDS100AsyncHow
SelfOrg78.892.079.086.276.036.460.7
RMAP80.193.081.389.480.347.763.8

In summary, SelfOrg is not a competitive baseline.

(4) If topology is conditioned on outputs, it could change even for same query due to stochasticity of LLM-generated answers. However, SelfOrg has coarse-grained adaptation in system level, while RMAP enables fine-grained adaptation in runtime layer level.

W1 Q3:

(1) Comparison protocol. In the MAS design domain, strictly controlling token and time cost is challenging because different workflows involve multiple agents and edges. Therefore, following traditional settings in prior works [1, 2, 3], comparison is conducted over a fixed scale of the graph.

(2) Cost report. We acknowledge that the setting does not strictly equalize the inference budget, and thus report the actual time and token costs. The LLM is GPT-4o-mini.

Token cost per execution:

MethodMMLUSVAMPAQuAHumanEvalMBPPDS1000AsyncHow
G-Designer3442674181436963523246485641
ARG-Designer2488536257614783470056015892
Anymac2602544360594946447453766243
RMAP2643487361514643411051566083

Time cost per execution:

MethodMMLUSVAMPAQuAHumanEvalMBPPDS1000AsyncHow
G-Designer11.923.427.025.423.320.220.8
ARG-Designer9.318.421.619.821.624.321.6
Anymac13.228.628.423.625.535.333.9
RMAP9.717.023.120.419.422.722.3

The results demonstrate: (i) Even before RL post-training, RMAP achieves comparable inference costs to baselines. Adaptation to immediate feedback prevents unbounded workflow expansion. Thus, our performance gains are not driven by substantially increased overhead. (ii) Compared with Anymac's one-by-one generation, RMAP's layer-wise generation is much more time-efficient.

(3) Baseline settings. Baselines follow their default setting to ensure sufficient training. Specifically, for RL-based designers, following G-Designer, there are 200 epochs and 4 trajectories per epoch. For LLM-search-based AFlow, it has 50 search iterations, each with 8 samples. ARG-Designer and Anymac require supervised training. Therefore, for fairness, supervised training data generated by RMAP is used for them.

W1 Q4:

(1) Randomness. There are two sources: (i) stochastic LLM reasoning via APIs and (ii) randomness in parameterized training.

(2) Statistics. For easy tasks, RMAP achieves competitive performance compared with baselines. However, for more complex tasks, such as DS1000 and Asynchow, the improvements are statistically significant. For example, when using GPT-4o-mini, RMAP outperforms baseline with holm-adjusted p = 0.0006 on DS1000 and with holm-adjusted p = 0.0436 on AsyncHow.

Q5: CodeTester cannot access ground-truth tests, but generate tests with assert itself. All baselines can employ CodeTester.

REFERENCES

[1] GPTSwarm: Language Agents as Optimizable Graphs

[2] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks

[3] Cut the Crap: An Economical Communication Pipeline For LLM-Based Multi-Agent Systems

AuthorsOpenReview ↗

Official Comment by Authors

Dear AC and Reviewers,

We sincerely appreciate the time and effort you have devoted to carefully evaluating our paper.

As Author- Reviewer Discussion will conclude soon, we would be grateful if you could take a moment to review our responses. We would also greatly appreciate any additional feedback or further discussion regarding any remaining concerns.

Thank you again for your valuable time and consideration.

Best regards, The Authors

Thank you for the detailed rebuttal and additional experiments. The new token and latency comparisons, clarification of baseline settings, and explanation of CodeTester address several of my original concerns. I also appreciate the additional discussion of related work and the comparisons with SelfOrg.

That said, I believe some aspects of the evaluation could still benefit from more comprehensive analysis, particularly regarding comparison protocols, cost reporting, and statistical significance. For example, the additional token and latency comparisons are currently limited to one LLM backbone, and the statistical significance analysis covers only selected benchmark settings. I encourage the authors to expand these analyses and clarify the remaining limitations in the revised manuscript.

Overall, considering the additional evidence and the limited time available during the rebuttal period, I believe the authors have addressed most of my concerns to a reasonable extent. The added clarification and experimental results in the rebuttal improved my assessment of the paper and thus raised my score from 2 to 3.

Thank you for your valuable insights and for raising the score! Just like RMAP, we will keep improving based on your feedback!

Reviewer_U4yGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RMAP generates and executes a workflow one layer at a time, allowing subsequent topology decisions to use intermediate execution outputs. This is a useful contribution, and the experimental coverage is relatively strong. 24512_From_Static_to_Dynamic_T.pdf

Critical strengths
  1. Broad evaluation with some substantial gains. Table 2 covers seven benchmarks and four LLM backbones, with means and standard deviations across five runs. RMAP has the highest reported mean in 26 of 28 settings. On DS1000 with Gemini-3.1-flash, it achieves 76.4%, compared with 66.5% for the strongest competing baseline.
  2. Ablations support using execution information. Removing agent outputs consistently hurts performance in Table 3. For example, GPT-4o-mini’s MBPP result falls from 80.3% to 74.3%. This directly supports an important part of the proposed mechanism.
  3. Meaningful efficiency and transfer experiments. RL post-training reduces RMAP’s own token and latency costs substantially. Appendix F also includes cross-dataset and cross-backbone transfer experiments, which strengthen the generalization evidence.
Critical weaknesses
  • I am not convinced that the accuracy gains hold up under matched inference budgets. In my view, capping how many times each role can be instantiated doesn't actually equalize the total number of LLM calls, token usage, or overall runtime. Right now, the efficiency analysis mostly just compares RMAP to itself before and after post-training. I would find a direct cost–accuracy comparison against Anymac, ARG-Designer, and a strong iterative single-agent baseline much more helpful for isolating the actual benefit of the method.
  • I feel the claims about adapting to changing environments go beyond what the experiments directly show. While I agree the paper demonstrates that execution outputs are useful, I didn't see much direct evidence of the system adapting to unavailable tools, corrupted intermediate outputs, shifting agent capabilities, or unexpected execution failures. To really test this, I'd love to see an experiment that holds the original query fixed while deliberately perturbing the intermediate feedback.
  • I think the deployment and win-rate claims need to be interpreted with a bit more caution. As I understand it, the sub-second latency number only reflects the planner's added overhead under the specific hardware and API setup tested, while the algorithm still has to wait for the entire current layer to finish executing before it can plan the next one. Also, achieving the highest mean in 26 out of 28 settings doesn't necessarily mean the method is statistically superior across all 26, in several of those cases, the margins look pretty small to me relative to the reported variance.
Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Rebuttal from authors

Q1:

(1) Comparison protocol. In the MAS design domain, strictly controlling token consumption and runtime is challenging because different generated workflows involve multiple agents and edges. Therefore, following traditional settings in prior works [1, 2, 3], the comparison is conducted over a fixed scale of the graph.

(2) Cost comparison. We acknowledge that the setting does not strictly equalize the total inference budget, and thus additionally report the actual time and token costs. The LLM is GPT-4o-mini and RL-based post-training is not used.

Token cost per execution:

MethodMMLUSVAMPAQuAHumanEvalMBPPDS1000AsyncHow
G-Designer3442674181436963523246485641
ARG-Designer2488536257614783470056015892
Anymac2602544360594946447453766243
RMAP2643487361514643411051566083

Time cost per execution:

MethodMMLUSVAMPAQuAHumanEvalMBPPDS1000AsyncHow
G-Designer11.923.427.025.423.320.220.8
ARG-Designer9.318.421.619.821.624.321.6
Anymac13.228.628.423.625.535.333.9
RMAP9.717.023.120.419.422.722.3

The results demonstrate: (i) Even before RL post-training, RMAP achieves comparable inference costs to baselines. Adaptation to immediate feedback prevents unbounded workflow expansion. Thus, our performance gains are not driven by substantially increased overhead. (ii) Compared with Anymac's one-by-one generation, RMAP's layer-wise generation is much more time-efficient.

Q2:

(1) Impact of feedback quality. RMAP relies on the quality of execution outputs. To further demonstrate the importance of output quality, we conduct an ablation study by removing CodeTester's test results (pass/fail) while retaining its generated test code. The LLM backbone is GPT-4o-mini, results:

SettingHumanEvalMBPPDS1000
w/o result87.075.846.4
w/ result89.480.347.7

This result demonstrates the importance of high-quality feedback for informed topology generation. Moreover, RMAP is easily extensible to incorporate more feedback signals, which can be integrated into additional semantic channels based on the requirements.

(2) Integration with defenses against perturbations. Like other learning-based MAS designers, RMAP does not inherently guarantee adaptation to unseen adversarial attacks [4, 5]. A practical solution is to employ MAS anomaly detections [4, 5] to identify and isolate agents with malicious outputs. Specifically, if an executed agent is detected, it will be pruned from the current graph. Following [4], we conduct prompt injection attacks against randomly selected agents, with GPT-4o-mini as the LLM backbone.

Without defense:

MethodMMLUAQUADS1000AsyncHow
ARG-Designer75.276.239.057.3
Anymac73.876.437.655.4
RMAP76.778.942.557.0

With defense:

MethodMMLUAQUADS1000AsyncHow
ARG-Designer77.277.840.660.0
Anymac75.478.339.157.5
RMAP78.980.446.563.0

This experiment demonstrates that RMAP can be integrated with existing detection-based defenses to support topology planning under runtime attacks.

Q3:

(1) DAG layer-wise execution. Prevailing DAG-structured MASs [1, 2, 3] naturally require layer-wise parallel execution in real employment, which is not specific to RMAP. In this way, they can benefit from an asynchronous environment with time efficiency improved greatly (Anymac is a counterexample). For an ordered execution, successors must wait before the preceding layer finish. But as stated in Section 5.5, the additional overhead is limited across platforms because introduced modules of RMAP are much more light-weighted compared to underlying LLMs for agent reasoning.

(2) Statistics. For easy tasks, RMAP achieves competitive performance compared with baselines. For more complex tasks, such as DS1000 and Asynchow, the performance improvements are statistically significant. For example, when using GPT-4o-mini, RMAP outperforms baseline with holm-adjusted p = 0.0006 on DS1000 and with holm-adjusted p = 0.0436 on AsyncHow.

REFERENCES

[1] GPTSwarm: Language Agents as Optimizable Graphs

[2] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks

[3] Cut the Crap: An Economical Communication Pipeline For LLM-Based Multi-Agent Systems

[4] G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systems

[5] BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks

Thanks for the response. I would like to keep my score.

Thank you for your valuable insights and positive score!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

Dear PCs, SACs, and ACs,

We sincerely thank reviewers U4yG, 94g8, and iZxZ for their careful reviews and for recognizing our rebuttal. Below, we summarize the key concerns raised by the reviewers and our corresponding responses.

How to conduct a fair comparison with baselines? (U4yG, 94g8, iZxZ)

(1) In the MAS design domain, strictly controlling token consumption and runtime is challenging because different generated workflows involve multiple agents and edges. Our setting is to compare RMAP with baselines under a fixed scale of the graph, which follows prior works [1, 2, 3].

(2) We acknowledge that the setting does not strictly equalize the total inference budget, and thus additionally report the actual time and token costs to reviewers. From the report, (i) our performance gains are not driven by substantially increased overhead. (ii) Compared with Anymac's one-by-one generation, RMAP's layer-wise generation is much more time-efficient.

Why can the decoder select different roles across layers? (94g8, iZxZ)

(1) Reviewers think concatenated embeddings of an executed agent and a candidate agent are linearly mapped, where candidate embeddings depend only on role descriptions in Eq.6. In this way, the embedding of a potential edge is fixed across layers.

(2) However, the executed agent's embedding is nonlinearly updated to incorporate system information, including its description, execution outputs, and topological context (Eq. 8, 10, 11). Thus, the overall mapping from execution context to edge probabilities is nonlinear. That is, the decoder considers not only static candidate semantics, but also dynamic context progressively accumulated during layer-wise execution.

Has RMAP's low latency been overstated given its layer-wise execution? (U4yG, iZxZ)

Prevailing DAG-structured MASs [1, 2, 3] typically require layer-wise parallel execution in real employment, which is not specific to RMAP. In this way, they can benefit from an asynchronous environment with time efficiency improved greatly (Anymac is a counterexample). For an ordered execution, successors must wait before the preceding layer finishes. But as stated in Section 5.5, the additional overhead is limited across platforms because introduced modules of RMAP are much more light-weighted compared to underlying LLMs for agent reasoning.

What is the underlying novelty compared with previous works? (94g8, iZxZ)

(1) As stated in the paper, a desirable DAG-based MAS designer should not only support layer-wise execution for asynchronous parallelization but also adapt to evolving execution states and inter-agent semantic interactions throughout runtime execution.

(2) To the best of our knowledge, RMAP is the first to fully support runtime layer-wise planning based on the evolved systematic execution semantics. In contrast, previous works only output a static template or only support agent-wise execution.

Is CodeTester involved in a test data leak? (94g8, iZxZ)

CodeTester cannot access ground-truth tests, but generate tests with assert itself. Baselines can also employ CodeTester.

We believe that the rebuttal has helped strengthen our submission. We have carefully addressed the reviewers' concerns and provided additional experimental results to further support our claims. We sincerely hope that our work meets the high standards of ICLR 2027 and will be of interest to its readership.

Best regards,

The Authors.

REFERENCES

[1] GPTSwarm: Language Agents as Optimizable Graphs

[2] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks

[3] Cut the Crap: An Economical Communication Pipeline For LLM-Based Multi-Agent Systems

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.