acceptodds
Withdrawn from ICLR 2027

ECHOROUTER: Forecast-Guided Evolution of Runtime-Adaptive Multi-Agent Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent systems face a recurring decision: after each intermediate result, is another model call worth its cost? The right amount of collaboration varies dramatically: hardly needs an agent, while an attempt at a Navier–Stokes proof may call for many specialists. Making this decision requires choosing how agents collaborate, which models to assign, what context each call receives, and when to stop under a budget. Code-based workflow optimizers such as AFlow can express conditional execution, but searching through repeated real rollouts is costly, especially on multi-turn tasks. We introduce EchoRouter, which uses experience from past execution trajectories as an “echo” to understand where the current policy stands and guide subsequent search. During optimization, it proposes diverse candidate policies, each decomposed into three decisions: how to collaborate during task execution, which models to assign, and what each call receives and returns. Guided by the echo, EchoRouter treats LLM as value model to forecast candidate quality and cost, executes promising policies, reflects on their trajectories, and updates the echo. It also compares models and prompts on saved task states while hiding model names to improve credit assignment. During execution, a lightweight controller uses the selected policy and current task state to construct subgraphs, assign models, and specify each call's prompt and output format. Across various model-pool scales and mathematical, coding, and general reasoning benchmarks, EchoRouter achieves quality–cost trade-offs. Results on multi-turn tasks and open-ended artifact generation further show competitive quality under an LLM as Judge evaluation. Its decoupled, model-anonymous design allows new LLMs to be assessed without rebuilding the collaboration policy. Together, these results demonstrate the effectiveness and potential of EchoRouter.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
self-adaptive system, multi agent system, model routing

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_pdZ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

EchoRouter optimizes multi-agent systems that can change their collaboration structure during a task. It uses execution experience and LLM forecasts to select policies offline, then a trained controller uses the selected policy and intermediate results to decide which agents to call next or when to stop. Experiments across reasoning, coding, and tool-use tasks report favorable quality–cost trade-offs.

Critical strengths
  1. Runtime adaptation is useful and well motivated. The controller can respond to intermediate evidence, allowing the system to spend additional calls where verification or repair is needed.
  2. The optimization method has a clear empirical basis. Matched comparisons help attribute failures to models or prompts, while forecasts prioritize candidate policies for expensive evaluation. The ablations support the contribution of these mechanisms.
Critical weaknesses
  1. The distinct contribution relative to prior work is not yet clear enough. AgentConductor also changes multi-agent topology in response to execution feedback [1], while AgentSwift uses value-guided search to reduce agent-design cost [2]. The paper needs to establish what its combination of offline experience and runtime control achieves beyond these approaches.
  2. The evidence for the quality–cost claim has limited scope. The forecast analysis does not show how reliably candidates are selected across all task domains and optimization rounds. Online cost excludes the compute cost of the local controller, so the reported trade-off does not fully characterize deployment cost.

[1] AgentConductor: Topology Evolution for Multi-Agent Competition-Level Code Generation. [2] AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search.

Questions for authors
  1. Across task domains and optimization rounds, how often does forecasting select the best measured candidate under the cost constraint?
  2. How does the quality–cost comparison change when local controller inference and training costs are included?
Reviewer_qUyGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed EchoRou, which uses prior experience from the past to generate candidate policies that guide adaptive collaboration during execution. The experimental results show that it outperforms most of the multi-agent workflow optimization and model routing methods and shows a more efficiently evolving policy than EvoMAS.

Critical strengths
  1. The problem that the authors try to solve is timely and motivating. The paper deals with problems in recursively improving a multi-agent system, where there is a fixed budget and a set of models. Even though the method relies on heuristics and LLM as a value function, the problem is worth noting.
  2. It shows strong performance compared to the baselines.
Critical weaknesses

Lack of novelty and unclear contribution claims. The three contribution claims are not novel. (1) State-adaptive collaboration: the line of work has been proposed in several papers like this one [1]. Even though EchoRouter (2) Forecast before spending: works like AgentSquare [2] have proposed LLM as a predictor for candidate agent search. Also, iterating over all possible configurations is infeasible, which naturally leads to forecasting before spending. (3) Trajectory-based credit assignment: this is a fancy way of saying controlled study. This is the core idea of many prior works like Reflexion, ADAS, AFlow, which try to optimize the workflow or prompt using an LLM-driven or heuristic-driven approach.

[1] Dang et al, Multi-Agent Collaboration via Evolving Orchestration, NeurIPS 2025
[2] Shang et al, AgentSquare: Automatic LLM Agent Search in Modular Design Space, ICLR 2025

Questions for authors

Please clarify the contribution claims and novelty of the work addressed above.

Reviewer_KiWvOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

EchoRouter optimizes structured policies that separate collaboration, model assignment and context configuration. An Echo memory of trajectories, model profiles and measured policy outcomes supports candidate generation and LLM forecasts of quality and execution cost. Selected candidates undergo real evaluation before policy admission. Matched-state model and prompt comparisons update the evidence used in later optimization rounds. At deployment, a local controller constructs subsequent collaboration subgraphs from intermediate observations while the policy and profiles remain frozen. The paper evaluates two model pools on eight benchmarks, extends the evaluation to multi-turn and artifact-generation tasks, and studies component ablations, frozen-pool candidate selection and new-model integration.

Critical strengths
  1. The integrated system has encouraging results across substantially different settings. Tables 1–2 report the highest average score among the displayed methods in each model pool. In the larger pool, Medium reaches 90.7 versus EvoMAS's 87.6, with mean Agent API cost of $0.0115 versus $0.0316 per query and similar reported optimization cost ($1.04 versus $1.03). Tables 3–4 also provide three-run summary standard deviations. These are useful aggregate quality–cost results, although they do not establish superiority on every benchmark or statistical significance of every difference.
  2. The paper provides evidence about components and explicitly limits several interpretations. Figure 5 includes static-policy, random-selection, public-name and single-factor-comparison ablations. Table 5 evaluates selection on frozen candidate pools against an exactly enumerated random-pair baseline, supporting the usefulness of forecasting beyond an illustrative trajectory. Appendix E states that forecasts only allocate evaluation slots, admission uses measured outcomes, and shadow outputs do not replace the ongoing trajectory. Appendix C.3 reports cases where additional collaboration is unhelpful. Together these make the system more informative than an aggregate performance comparison alone.
Critical weaknesses
  1. The evaluation population and protocol are insufficiently specified to assess the strength of the broad empirical claims. Appendix B and Table 11 state that optimization and test splits are disjoint, but do not give per-benchmark train/validation/test counts or a concrete split-reconstruction procedure for Tables 1–4. Counts supplied for individual validation studies do not identify the main test populations. Likewise, Appendix B.3 and Table 12 identify 14 generation categories and three LLM judges, and Figure 4 reports three repetitions, but do not provide the number of distinct prompts per category or the judge prompts and operational scoring rubric. The available three-run standard deviations are welcome, but cannot resolve uncertainty about sample coverage or how the generation scores were produced. This is a reporting concern, not evidence that test leakage occurred.
  2. The central search-efficiency claim needs a clearer connection to cumulative optimization expenditure. Figure 3 plots quality against optimization steps; its cost bars are deployment Agent cost per query, not accumulated search cost. Figure 5 includes the relevant random-selection control, but similarly plots steps and normalized test cost. Table 5 establishes Best-of-2 utility on eight frozen pools with 16 candidates each; its cost column explicitly excludes candidate execution. These results support effective candidate selection, but do not establish its advantage at a matched total search budget. Tables 1–2 do report optimization costs, including higher small-pool expenditure than all displayed MAS baselines, so my concern is not that costs are wholly absent. A quality-versus-cumulative-spend analysis including proposals, forecasts, shadow comparisons and validation would directly test the claimed acceleration. Appendix H separately reports local controller time; the API-cost advantage should also remain clearly distinguished from total computational cost.
Questions for authors
  1. Please provide the train/validation/test counts and reproducible split definitions for the main benchmarks, and the distinct generation-prompt counts, judging instructions and scoring rubric. How are the reported runs paired across methods? These details would materially affect my assessment of coverage and reliability.
  2. From the recorded cost ledgers, can you compare forecast-guided and random selection, and the main competing optimizer, at matched cumulative optimization expenditure or matched achieved quality? Please include all optimization-side calls, identify the price schedule, and keep local controller resources separate. If this comparison is unavailable, please narrow the acceleration claim to the step-based and frozen-pool evidence actually demonstrated.
  3. Table 13 lists D for LiveCodeBench, whereas the following text says it retains its original incumbent. Is that incumbent D, or is one description incorrect? What held-out quality and deployment cost result from profile-only integration versus optimization from zero? Lower access cost alone does not establish a comparable-quality transfer advantage.
Reviewer_EdbsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EchoRouter, which adaptively constructs, adjusts, and executes multi-agent systems. The authors divide the collaboration policy into three components: meta, model, and context. The evolution stage consists of Propose Candidates, Forecast Quality, and Allocate & Rollout, drawing on Anonymous Profiles, Trajectory Reflection, and Adopted Policies. During execution, a lightweight local controller adaptively decides whether to construct the next subgraph based on the current task state and remaining budget. The authors conduct experiments on eight benchmarks covering mathematics, coding, and general reasoning, as well as multi-turn tasks and open-ended generation tasks. They report that EchoRouter achieves favorable quality–cost trade-offs.

Critical strengths

The problem is important and well motivated. Building on existing work, the paper integrates these ideas into a unified framework with a cleanly factorized formulation of cost-aware, state-adaptive collaboration. Several design choices are interesting, including the decomposition into meta-policy, model-policy, and context-policy, as well as the model-anonymous design. Overall, the design is reasonable.

Critical weaknesses

The description of the experimental design and the analysis of the results lack clarity. For example, the Random, Ensemble, and Greatest baselines are not clearly defined, cascade baselines are missing, and the sizes of the dataset splits used in the main experiments are not clearly reported. Please see the questions below for details. If the authors clarify the experimental setup and satisfactorily address the concerns raised, I would consider raising my rating.

Questions for authors
  1. How are the Direct Model Baselines Random, Ensemble, and Greatest in Table 2 defined?
  2. Some baseline scores in Table 2 appear unusually low. For example, GraphRouter performs poorly on MATH. All three “Standalone Model Routing” baselines score substantially lower than “Random,” “Ensemble,” and “Greatest” on AIME. MLPRouter and GraphRouter also perform poorly on LCB. Could you explain the reasons for these results?
  3. Could you report the sizes of the optimization split and the held-out test split for each dataset?
  4. Could you include a cascade baseline, such as FrugalGPT, to demonstrate the effectiveness of EchoRouter?
  5. In Table 1 and 2, GPTSwarm and AFlow are restricted to using GPT-OSS-120B and GPT-4o, whereas EchoRouter appears to have access to more models. Is there a way to quantify how much of the overall improvement comes from workflow optimization and how much comes from execution-model selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.