- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MeterRouter, a budget-aware dynamic routing framework for LLM-based multi-agent systems. The method jointly determines agent roles, backbone LLMs, communication edges, and termination decisions under a user-specified monetary budget.
Unlike approaches that select from predefined multi-agent configurations, MeterRouter incrementally constructs a directed acyclic collaboration graph while tracking the remaining budget and previously observed execution costs. Its training procedure consists of two stages: supervised imitation from successful execution graphs, followed by group-relative reinforcement learning to improve the accuracy–cost trade-off.
Experiments are conducted on five benchmarks covering general reasoning, mathematical reasoning, and code generation, using a heterogeneous pool of three LLMs. The reported results show that MeterRouter achieves 66.72% overall accuracy at an average cost of $0.00256 per query, compared with 66.80% accuracy and $0.01415 per query for MasRouter. The paper also reports high budget-compliance rates and investigates the effects of reinforcement learning, budget constraints, and maximum agent count.
Critical strengths
- A practically relevant formulation of budget-aware multi-agent orchestration.
The paper addresses an important limitation of existing multi-agent routing methods: the need to jointly optimize communication structure and heterogeneous model allocation under explicit per-query budget constraints.
Allowing different queries to receive different computational budgets is practically meaningful, and the incremental graph-construction procedure provides a flexible alternative to selecting from fixed collaboration templates.
The combination of role selection, backbone assignment, edge generation, and termination decisions is a potentially useful direction for resource-efficient multi-agent systems.
- Strong reported accuracy–cost trade-offs across multiple benchmarks.
MeterRouter achieves competitive accuracy while substantially reducing the estimated inference cost compared with several multi-agent baselines. In particular, the approximately 82% cost reduction relative to MasRouter, with nearly identical aggregate accuracy, is practically appealing.
The experiments cover five benchmarks and include comparisons with single-agent methods, fixed multi-agent architectures, dynamic multi-agent systems, and model routers. The budget sweeps also demonstrate that the generated system adapts its model allocation and agent count according to the available budget.
The results suggest that joint routing can improve the practical efficiency of multi-agent inference.
Critical weaknesses
- The independent contributions of the proposed routing components are not sufficiently established, and stronger budget-matched comparisons are needed.
Although the paper emphasizes the joint optimization of roles, backbone LLMs, communication topology, and termination, the current ablation studies do not adequately isolate these components.
Table 2 primarily examines the effect of removing the reinforcement learning stage, varying the budget, and changing the maximum agent count. Removing RL reduces overall accuracy from 66.72% to 57.34%, but this comparison does not reveal whether the improvement arises from better backbone selection, topology generation, role assignment, or termination decisions.
The lack of component-level ablations makes it difficult to assess which parts of MeterRouter are essential to its performance.
In addition, the main comparison with MasRouter and other multi-agent baselines is conducted at substantially different realized inference costs. Although this supports the reported cost savings, it does not establish how the baselines would perform under the same monetary constraints.
A stronger evaluation would compare methods under matched realized cost budgets, including simpler query-conditioned backbone routing combined with fixed or pruned multi-agent topologies.
This is particularly important because the heterogeneous model pool exhibits considerable differences in accuracy across tasks, and selecting an appropriate backbone may already account for a substantial fraction of the reported improvement.
Without these controlled comparisons, the necessity of the full joint optimization framework remains insufficiently demonstrated.
- The claims regarding strict budget guarantees and execution-aware routing require further qualification and validation.
MeterRouter relies on conservative byte-based estimates of token consumption to construct affordable model sets and determine completion limits. However, the paper does not establish that these estimates are guaranteed upper bounds under all tokenization patterns and supported model families.
The reported budget-compliance rates are high, but are not equivalent to a formally established per-query cost guarantee. The use of fixed accounting rates derived from an external pricing source also requires clarification, since these may not exactly correspond to actual provider billing.
Furthermore, the routing policy's observations of previous agents mainly consist of token counts, message lengths, and realized costs. The state representation does not explicitly include the semantic content, correctness, or reasoning quality of intermediate agent outputs.
Consequently, the method appears to adapt primarily to resource consumption and previously constructed graph structure, rather than directly adapting to the substantive reasoning outcomes of earlier agents. This limits the strength of the claim that routing decisions are informed by execution feedback.
The evaluation also provides insufficient information about run-to-run variability. The main performance table reports point estimates without confidence intervals, making it difficult to determine the statistical reliability of small accuracy differences between the strongest methods.
Finally, several numerical statements in the results section do not agree with Tables 1 and 2, including the claimed BBEH accuracy gap and the reported accuracy change when reducing the maximum agent count. These inconsistencies should be corrected.
Questions for authors
Q1. Component-level contribution of joint routing.
Can the authors provide separate ablations for role assignment, backbone selection, communication-edge generation, and learned termination?
In particular, how does MeterRouter compare against a simpler approach that uses a fixed sparse topology and optimizes only the backbone assigned to each agent?
Similarly, what is the contribution of allowing topology adaptation when model selection and agent count are held fixed?
These experiments would clarify whether the observed improvement genuinely requires joint optimization.
Q2. Budget-matched comparisons with existing routers.
Can the authors compare MeterRouter with MasRouter, BAMAS, and other relevant routing baselines at matched realized monetary costs?
For example, could the baselines be evaluated across comparable budget limits or tuned cost preferences, so that their accuracy–cost frontiers can be compared under similar resource constraints?
The current comparison demonstrates substantial cost savings, but does not fully establish superiority over competing methods at equivalent spending levels.
Q3. Validity of the budget guarantee.
Can the authors explain whether the byte-based token estimator is a mathematically guaranteed upper bound for all three model tokenizers, or an empirically conservative approximation?
How frequently do budget overruns, budget-blocked queries, or truncated completions occur at each budget level, and how are these cases handled in the reported accuracy?
Additionally, do the accounting prices match the actual billing rates of the LLM provider used in the experiments?
These details are important for evaluating the strength and practical scope of the claimed budget guarantee.
Q4. Adaptation to intermediate reasoning outcomes.
The routing state includes observations of previous agents' resource usage and output lengths, but does not explicitly incorporate their semantic outputs.
How can the router distinguish between a successful intermediate solution and an unsuccessful one if their token statistics are similar?
Could the authors compare the current resource-aware state representation with a content-aware routing variant, or clarify that the intended execution feedback refers primarily to realized cost and message length?
This would help establish the specific type of adaptivity provided by MeterRouter.
Q5. Statistical reliability and computational overhead.
Can the authors report results across multiple random seeds, together with paired confidence intervals for the main accuracy comparisons?
It would also be valuable to report the training-stage computational cost and end-to-end latency, rather than only the monetary cost of the executed agent calls.
Since the method requires supervised demonstration generation followed by reinforcement learning, how many inference queries are needed to amortize the training cost relative to simpler routing alternatives?
Such analyses would provide a more complete assessment of the practical efficiency of the proposed framework.
6 comments
Q2&W1
Thank you for the question. We understand your concern about "matched realized monetary costs" and "comparable budget limit".
A comparable budget and a comparable cost mean different things. A comparable budget gives every method the same budget. A comparable cost means every method actually spends the same amount, which depends on how each router runs and cannot be fixed in advance. Even with a shared budget, methods with soft constraints (e.g., MasRouter) or no constraints cannot keep their cost within it. BAMAS is the baseline that most closely resembles a hard budget constraint, but it applies the constraint only once, before execution: an integer linear program selects the strongest LLMs whose estimated costs fit within , and nothing tracks the actual cost during execution. As a result, BAMAS can exceed the budget, and its paper reports out-of-budget queries. In Table 1, we ran BAMAS at , the same budget as MeterRouter. Over all test questions, BAMAS reaches 56.15% at USD per query, while MeterRouter reaches 66.72% at USD, so MeterRouter is both more accurate and cheaper under the same budget.
MasRouter and BAMAS also compare methods on the accuracy–cost trade-off using a Pareto front (Figure 3 of the MasRouter paper and Figure 2 of the BAMAS paper). We will explain this point more clearly in the revised paper.
Q3&W2
Thank you for this question.
Input side: an estimate. The provider does not return token counts before a call, so input tokens are estimated from bytes, . Here is the UTF-8 length of the full request of agent (system and role prompts, query, and the messages it reads from its predecessors), and 768 tokens cover the chat-template overhead. Typical tokenizers use about four bytes per token on English text, so assuming two bytes per token roughly doubles the count. The input cost is therefore also an estimate. The byte-based estimate is an empirically conservative approximation, not a guaranteed upper bound: densely tokenized text such as long digit strings can be underestimated. It is the only source of overruns, hence the in-budget rate of at least 99.98% rather than 100%.
Output side: a hard cap. Before each call, the remaining budget is converted into a token cap passed to the provider as max_tokens:
where reserves the cost of agents still needed to close the graph. No agent, including the last, can exceed the budget through a long output. Backbones that cannot fund 256 output tokens are masked, and the graph terminates when none remains.
In deployment, the input estimator, a single function, can be replaced by exact counts from the model's tokenizer or the provider, without changing the framework. Then for every call, so holds for every query.
Handling in the reported accuracy. No query is dropped. Each query ends by a learned stop, the agent cap, or a budget stop, and is graded normally. A query counts as incorrect if its final answer is truncated at its token cap or if no agent is affordable at all. An overrun is graded normally and counted as a violation.
How often each mechanism acts. To show how often the input estimate and the output cap matter in practice, we recorded for every test query how its graph ended, whether its final answer was cut at its token cap, and whether its cost exceeded . Table R1 reports these shares at each test budget.
Table R1. How queries end, and how often truncation and overruns occur, at each budget : share of queries (%), weighted by the number of test questions over the five benchmarks, mean std over 3 seeds. As in Table 2, means for Omni-MATH-2 and BBEH and for the other three benchmarks. Learned, agent-cap and budget stops are the three ways a query ends and sum to 100%.
| Budget | Learned stop | Agent-cap stop | Budget stop | Truncation | Overrun |
|---|---|---|---|---|---|
| 86.77 0.57 | 0.53 0.20 | 12.70 0.43 | 16.93 0.34 | 0.02 0.02 | |
| 91.06 0.24 | 3.20 0.12 | 5.74 0.12 | 7.71 0.24 | 0.01 0.02 | |
| 88.25 0.48 | 11.72 0.46 | 0.03 0.03 | 0.08 0.12 | 0.00 0.00 |
- Learned stop (from learned policy): the policy's stop decision ends the graph. A stop counts as learned only if at least one backbone could still afford another agent.
- Budget stop (from hard affordability filter): no backbone can afford another agent, meaning its estimated input plus a minimum response of 256 output tokens exceeds the remaining budget, so the affordability filter ends the graph.
- Agent-cap stop (neither; it's the fixed limit on the number of agents): the graph reaches the maximum number of agents and ends.
- Truncation: the final answer is cut at its token cap.
- Overrun: the query's actual cost exceeds .
Pricing. We use the llm prices from https://artificialanalysis.ai/ as accounting rates. Different providers charge different prices for the same model and their bills are not directly comparable, especially for open-source llms. Each call's cost is the provider-reported input and output token counts times these rates, so costs reflect actual usage, and all methods in Table 1 use the same rates.
We will update to our revised paper.
Thanks for your questions. Here are our replies to your questions and concerns.
Q1&W1
To address your concerns, we ran two representative experiments: (1) fixed topology and (2) single LLM. Below are the results.
Fixed topology
In this experiment, MeterRouter is trained with its topology fixed to a chain or a complete graph, while the rest of the policy is unchanged, so it no longer constructs the graph but still assigns heterogeneous backbones under the same budget. We compare it with the Table 1 baselines of the same topology under the budget B=5×10−2, as shown in Fig. R1.
https://github.com/Anon-Submission-1024/Figure/blob/main/3/Fig.%20R1.png
Fig. R1. Accuracy versus real average cost per query for the fixed-topology ablation and the Table 1 baselines of the same topology, Chain (top) and Complete Graph (bottom).
We can conclude that with a fixed topology, part of the gain remains: heterogeneous model selection alone places the fixed-topology variant on the Pareto front of its same-topology baselines in all 12 panels. The full MeterRouter is still more accurate than the fixed-topology variant in every panel, and over all test questions it is 7.03 points higher than the chain and 7.51 points higher than the complete graph, at a lower average cost.
Single LLM
In this experiment, MeterRouter is trained with a single shared backbone, gpt-oss-120b, while the rest of the policy is unchanged, so it still constructs the graph but can no longer choose models. We compare it with the full MeterRouter and with all Table 1 baselines that use the same backbone under the budget B=5×10−2, as shown in Fig. R2.
https://github.com/Anon-Submission-1024/Figure/blob/main/3/Fig.%20R2.png
Fig. R2. Accuracy versus real average cost per query for MeterRouter with a single LLM, the full MeterRouter, and the Table 1 baselines with gpt-oss-120b.
With one backbone, MeterRouter is only 3.19 points below the full model, so most of the gain survives. Learning the topology alone makes the single-LLM variant more accurate and cheaper than the Table 1 Chain, Tree, and Complete Graph baselines with the same backbone on every benchmark; over all test questions it scores 3.92 to 5.31 points higher at half their cost or less. It also lies on the Pareto front of the gpt-oss-120b methods in all 6 panels. Four multi-agent baselines with the same backbone are more accurate overall (AutoGen 64.18%, GPTSwarm 64.56%, AgentPrune 65.90%, LLM-Debate 66.70%), but they cost 2.3 to 6.3 times more per query. The full MeterRouter is more accurate than the single-LLM variant on every benchmark. Comparing the two ablations, fixing the topology lowers overall accuracy by 7.03 to 7.51 points, while using a single backbone lowers it by only 3.19 points. Graph construction therefore accounts for more of the gain, and model selection adds to it.
We will update this in our revised paper.