acceptodds
Under review as a conference paper at ICLR 2027

TopoRanker: From Successful Graphs to Utility-Aware Topology Ranking for Multi-Agent Reasoning

Abstract

Learning communication topologies for multi-agent systems requires not only identifying structures that can successfully solve a task, but also distinguishing their collaborative utility. Successful-graph reconstruction treats observed feasible structures as supervision targets, yet does not explicitly compare the cost differences among multiple successful graphs or identify which roles and communication links contribute meaningfully to task completion. We propose TopoRanker, which shifts communication topology learning from successful-graph imitation to verifier-driven structural utility learning. During training, TopoRanker retains the complete execution outcomes of multiple candidate graphs for the same task, captures differences in overall utility through graph-level preferences, and derives local structural credit by removing nodes and edges. A task-conditioned graph utility model jointly learns from these complementary signals. At test time, the model ranks candidate graphs before execution and runs only the selected communication topology. Across eight benchmarks, compared with the fixed multi-agent baseline achieving the highest accuracy on each benchmark, TopoRanker improves accuracy on six tasks and matches it on one, yielding a macro-average accuracy gain of percentage points while reducing token consumption by on average across benchmarks. On HumanEval, a -percentage-point decrease in accuracy is accompanied by a reduction in token usage. Further selector analyses and supervision ablations demonstrate the benefits of jointly learning from global preferences and local structural feedback, while revealing task-dependent trade-offs between performance and communication cost.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Multi-Agent Systems, Communication Topology, Topology Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2HPnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TopoRanker, a task-adaptive communication topology selection framework for LLM-based multi-agent collaboration. It addresses the limitations of existing methods that rely solely on successful graph executions, which provide limited information about the task-specific utility of different topologies and the contributions of individual structural components. Specifically, the method first constructs a set of candidate directed acyclic graphs (DAGs) and evaluates their utility based on execution outcomes and communication costs. It then derives global preference supervision from utility differences among candidate graphs for the same task and obtains local structural intervention supervision by removing individual nodes or communication edges. Using these supervision signals, a task-conditioned Edge-aware Graph Transformer is trained by jointly optimizing utility regression, topology ranking, and intervention consistency losses to predict the utility of candidate topologies. During inference, only the graph with the highest predicted utility is executed. Experimental results demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper proposes a task-specific utility-based topology selection method, providing a more principled approach to selecting multi-agent collaboration structures.

  2. The paper introduces a topology learning method that integrates global comparisons with local interventions. By assessing the contributions of different structural components to collaborative outcomes, it provides finer-grained supervision for topology selection.

Critical weaknesses
  1. The proposed method shares some similarities with prior work, and its novelty and key distinctions require further clarification.

  2. The construction of candidate topologies and the associated experimental settings are not sufficiently justified and require further explanation.

  3. The paper provides insufficient analysis of certain experimental settings and results, which would benefit from further explanation and discussion.

Questions for authors
  1. The paper proposes a task-adaptive method for selecting multi-agent communication topologies based on predicted utility, demonstrating improvements in both performance and efficiency. However, prior studies, such as AMAS and Codebook Agent, have also explored topology selection based on task performance or utility. The authors should further clarify how their approach differs from these methods, particularly the additional benefits of incorporating node- and edge-level structural supervision compared with existing graph-level topology ranking methods. Direct experimental comparisons with these closely related methods would also help establish the effectiveness and novelty of the proposed approach.

  2. The paper primarily constructs candidate DAGs from predefined topology templates, such as Chain, Star, Tree, and Dense, and their randomly pruned variants. Although this strategy reduces topology search and evaluation costs, the model is restricted to selecting from a finite candidate pool, potentially limiting its performance due to insufficient coverage and structural diversity. The authors should further discuss this limitation and consider additional experiments with different candidate generation strategies, candidate pool sizes, and levels of topological diversity.

  3. In the experiments, the number of executions per task–graph pair is set to (K=1). Given the stochastic nature of LLM inference, a single execution may not reliably estimate the true utility of a candidate graph, potentially affecting the reliability of graph-level preference labels and node- and edge-level intervention labels. The authors are encouraged to evaluate different values of (K) (e.g., (K=1,3,5)) to examine their impact on final performance.

  4. During training, the method requires executing multiple candidate communication topologies and re-executing modified graphs after removing individual nodes or edges to construct global and local supervision signals. Although only the selected topology is executed during inference, offline data collection and intervention evaluation may incur substantial computational costs. The authors should provide a more detailed analysis of the training overhead, including the number of LLM calls, token consumption, and wall-clock time, while distinguishing the costs of supervision data construction from those of training the utility prediction model.

Reviewer_TQHWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TopoRanker builds candidate communication DAGs from a fixed pool of five agent roles and, at test time, executes only the graph ranked highest by a lightweight model. The model is trained on verifier-derived utilities through utility regression, within-task preferences, and utility differences induced by deleting one node or one edge. Across eight benchmarks in math, code, distributed information integration, and scientific reasoning, against fixed topologies and adaptive methods adapted to the same interface, it reports the best accuracy on six benchmarks, a macro-average gain of 4.91 percentage points, and a 51.95% average token reduction, both relative to the most accurate fixed topology on each benchmark. The three-loss ablation is the paper’s evidence that these signals are complementary; the candidate-pool analysis is its evidence that, on the two hardest benchmarks, the primary bottleneck is ranking rather than candidate availability.

Critical strengths

The deployment protocol is clear: candidate graphs are executed during training, while only the selected graph is executed at test time. Tables 1 and 2 report both task accuracy and token consumption under a shared experimental setup, making the performance–cost trade-offs explicit. Table 3 goes beyond end-task accuracy to examine selection quality. It includes unfavorable results on AQuA, where Top-1 Match is 37.50%, below the reported random expectation of 42.50%.

Critical weaknesses

W1. With K=1K=1, Oracle Pass@1 counts a task as solved if any candidate succeeds once. The paper reports OlympiadBench 85.34% versus 40.50% and FrontierScience-Olympiad 81.67% versus 51.25%, with normalized regrets of 21.31% and 12.40%, respectively, and calls ranking the primary bottleneck. That oracle is the union of one-shot successes, not the best candidate’s expected success rate. The gap therefore does not isolate ranking error from execution randomness. AQuA Top-1 Match is 37.50%, below the reported random expectation of 42.50%, and pairwise accuracy is 47.37%.

W2. DfitD_{\mathrm{fit}} already contains the original candidate graphs and the executed node- or edge-deletion variants, whose observed utility differences Δ^a\hat{\Delta}{a} provide the targets for Eq. (13). Table 4 removes Lint\mathcal{L}{\mathrm{int}} without stating whether those intervention graphs remain in DfitD_{\mathrm{fit}}. The full model also uses about 26% more tokens than the variant without intervention supervision, so the accuracy comparison is not made at the same token cost.

Questions for authors

Q1 Section 4.1 specifies a fixed pool of five agents, but Table 4 reports an average of 5.1 nodes for both Utility Only and w/o Preference on OlympiadBench. How are nodes counted? Can multiple nodes use the same role?

Q2 Section 3.4 states that utility regression uses both the original candidate graphs and the executed intervention graphs. In the “w/o Intervention” ablation in Table 4, are the intervention graphs still used for utility regression, with only the intervention loss removed?

Q3 How many candidate graphs are evaluated per question, including the randomly pruned variants? The Offline Pass@1 results in Table 3 match the TopoRanker accuracies in Table 1 on all eight benchmarks. Was Table 3 computed by executing every candidate once on the same test split used for Table 1?

Q4 How is the tie-aware random expectation in Table 3 computed? In particular, what criterion is used to treat candidate utilities as tied?

Reviewer_9RZmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TopoRanker, a framework for selecting task-specific communication topologies for LLM-based multi-agent reasoning. During training, candidate graphs are executed to obtain performance–cost utilities, which provide within-task ranking preferences and local supervision through node- and edge-removal interventions. A lightweight, task-conditioned graph model learns from these signals. At inference, it scores candidate topologies without executing them and runs only the highest-ranked graph. Experiments evaluate accuracy, token consumption, topology-selection latency, and selector quality across eight benchmarks.

Critical strengths

1.A clear approach to learning topology utility beyond task success.

The framework combines comparisons between complete graphs with utility changes induced by local structural interventions. This provides richer supervision than reconstructing successful graphs alone, and the ablations show benefits from both supervision signals.

2.Potentially Useful evaluation of performance, cost, and selection quality.

The experiments include challenging information-integration and scientific-reasoning tasks alongside conventional benchmarks. The candidate-pool oracle analysis helps distinguish limitations of the available topologies from errors made by the learned selector, while token and topology-selection latency measurements characterize the deployment tradeoffs.

Critical weaknesses
  1. The detailed experimental setup (prompts and hyperparameters, ...) is missing. It will be very difficult to reproduce the results.

  2. Main results don't support the method well. From Table 1, we can see that GSM8K, SVAMP, MultiArith, and Human-Eval are already saturated. I think the only meaningful results come from 3 of the total benchmarks. Take a closer look at these benchmarks: compared to the second-best ARG-Designer baseline, the improvement in Olympiad-Bench and Frontier-Science is 1.09% and 1%, respectively, which is nearly noise. I don't see the advantage of the proposed method in Table 1.

  3. No qualitative examples are provided. I don't know how this pipeline is actually working.

  4. The paper does not report repeated runs or uncertainty estimates, and its structural supervision is derived from a single execution per task–graph pair without examining how generation stochasticity affects preference labels and intervention-based utility differences.

I would suggest that the author substantially revise the paper.

Questions for authors

What motivates the choice of topology templates and their pruned variants? How sensitive are the results to the composition and size of the candidate pool?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.