acceptodds
Withdrawn from ICLR 2027

Quality-Aware Online Budget Control for LLM-based Multi-Agent Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cost-efficient language-model multi-agent systems (MAS) often target average cost reduction, whereas deployment under user quotas requires maximizing quality within a hard per-request budget. We propose Quality-Aware Online Budget Control (QOBC), a unified controller for single-round MAS with explicit dependencies and enforceable output limits. QOBC learns agent budget–quality curves through single-agent interventions, selects a feasible execution graph, and reallocates output budgets during execution. A shared solver accounts for generation and downstream input costs. Actual usage drives replanning; joint batch reservations enforce safety under explicit accounting assumptions. On 50 held-out MATH questions per controlled workflow, QOBC improves normalized budget–accuracy AUC over online equal allocation by 0.220 and 0.230 in sequential and parallel structures. On 100 new held-out inputs per native application, MoA AUC rises from 0.538 with static quality allocation to 0.618. On Screenplay, AUC rises from at most 0.292 with online equal allocation to at least 0.469, with the best quality at the 100% reference-budget level. No request-budget violation is observed across 6,000 native conditions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-agent systems, budget control, token allocation, online optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GYZLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QOBC, a quality-aware online budget control framework for LLM multi-agent systems (MAS). Instead of merely reducing average costs with unbounded budgets, QOBC explicitly formulates the objective as maximizing expected task scores under strict per-request token constraints. The framework consists of three stages: offline calibration (fitting agent-specific budget-quality response curves via single-agent interventions), pre-execution (selecting the optimal graph from candidate topologies using a geometric mean surrogate), and online reallocation (using a KKT solver to dynamically redistribute output token caps for remaining agents after each call batch, factoring in both generation and downstream input costs).

  The method's theoretical guarantee against budget overruns under explicit accounting assumptions is empirically validated by a zero violation rate across 9,000 conditions. Extensive experiments on two controlled structures (Sequential Pipeline and Parallel on MATH) and three real-world applications (MoA, CrewAI Screenplay Writer, Corex Review) demonstrate its effectiveness. QOBC improves the normalized budget-quality AUC by 0.220 and 0.230 over online equal allocation on controlled structures, and beats static quality allocation by up to 0.169 on real-world tasks, notably boosting the AUC of Corex Review from 0.570 to 0.739.
Critical strengths
  1. Rigorous budget safety guarantee.Theorem 1 proves that cumulative cost never exceeds B under explicit accounting assumptions (valid input bounds, enforced output caps, complete billing records), and this guarantee is verified empirically with zero violations across 9,000 test conditions. The separation between quality prediction and budget enforcement is technically sound and practically important.

  2. Downstream input cost modeling.By pricing each output token together with its induced downstream reading cost (the π_i term), the planner captures the true marginal cost of allocating tokens to upstream agents—a dimension consistently overlooked in prior MAS efficiency work and directly relevant to real deployment economics.

Critical weaknesses
  1. Absence of empirical comparisons against direct competitors. BAMAS (Yang et al., 2026) and FutureWeaver (Jung et al., 2026) share highly overlapping problem formulations with this work. While cited, they are conspicuously absent from the experimental baselines. Merely outlining technical differences in the text is insufficient to substantiate your novelty claims. Without head-to-head quantitative comparisons, the exact incremental contribution of QOBC cannot be rigorously assessed.

  2. Inadequate analysis of the Screenplay failure case undermines the framework's core assumption. The Historical Proportional baseline noticeably outperforms QOBC on the overall AUC for Screenplay ([0.511, 0.522] vs. [0.469, 0.474]), despite relying entirely on heuristics rather than learned curves. Correlating this with Table 4, the κ values for the three Screenplay content agents are near zero. This implies nearly flat budget-quality curves—the precise regime where quality-guided optimization loses its edge. The authors gloss over this finding in a single sentence. There is no parameter-level post-mortem, no discussion on when practitioners should anticipate such cases, and no proposed detection or fallback mechanisms. This oversight threatens the central premise of QOBC: that agents exhibit meaningful, differential responses to varying budgets.

Questions for authors
  1. On the Screenplay failure case and the flat κ response The Historical Proportional baseline outperforms QOBC on the overall AUC for Screenplay, despite operating completely independent of any learned curves. According to Table 4, the κ values for all three Screenplay content agents are near zero (reaching a maximum of only 0.02081), indicating that their budget-quality curves are essentially flat across the experimental range. Do you agree that this flat response is the root cause of QOBC losing its advantage in this scenario? If so, would it be possible to introduce an a priori diagnostic mechanism based on the fitted parameters (e.g., a minimum κ threshold) during the calibration phase? Such a mechanism could predict whether QOBC will actually yield benefits over simple heuristics, thereby guiding when to apply or bypass your framework.

    1. Direct comparison with overlapping literature Both BAMAS (Yang et al., 2026) and FutureWeaver (Jung et al., 2026) align very closely with your problem formulation, yet they are only differentiated textually in the Related Work. Are there specific technical or experimental hurdles preventing a head-to-head empirical comparison? If their official code is unavailable, would it be feasible to implement a proxy baseline (e.g., a BAMAS-style model selection strategy) in your controlled settings during the rebuttal? Providing some form of quantitative comparison—even an approximated one—would help significantly in clearly delineating QOBC's exact contribution boundary.

    If you can effectively resolve my issue, I will consider increasing the points.

Reviewer_6UQqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents QOBC (Quality-Aware Online Budget Control), a framework for allocating inference budgets in multi-agent systems (MAS) while respecting a hard per-request budget limit.

The approach consists of three main stages:

  1. Offline calibration: The framework learns agent-specific budget-quality curves through single-agent interventions across different execution graphs and budget settings. During each intervention, one agent's output budget is varied while the others are held fixed. The resulting curves are combined into a quality surrogate used for budget allocation and execution-graph se-lection.

  2. Pre-execution planning: Based on the learned curves, the framework selects an execution graph, when alternatives are available, and determines an initial per-agent budget allocation.

  3. Online reallocation: During execution, the framework updates its cost model based on actual budget consumption and reallocates the remaining budget among agents that have not yet executed.

QOBC is evaluated on two controlled workflows and three adapted multi-agent applications. The results show substantial improvements over several budget-allocation baselines in most experimental settings.

The paper presents a practically relevant framework with a clear methodology, coherent budget accounting, and promising empirical results. In particular, the fixed-graph experiments provide useful evidence that online budget reallocation can improve task quality.

However, my main concern is that the central learned allocation mechanism is not sufficiently validated. The fitted budget-quality curves and their geometric-mean composition are used to optimize budget allocations across constrained agents, yet the evaluation does not sufficiently establish how reliably the surrogate ranks joint allocations. Furthermore, the experiments do not clearly demonstrate the addition-al benefits of the proposed approach over simpler calibrated allocation policies.

The evidence supporting graph selection is also limited, and calibration overhead is not sufficiently investigated, although the authors acknowledge some of these limitations.

Overall, I consider the approach promising, but stronger validation of the quality surrogate and comparisons with simpler learned allocation policies are needed. I therefore recommend a weak reject. Additional ablation studies or evidence addressing the concerns outlined above could change my assessment.

Critical strengths
  • The paper addresses a relevant practical problem: maximizing task quality under constrained inference budgets.
  • The methodology is clear and well-motivated:
    • Accounting for downstream input costs is a useful aspect of the approach.
    • Separating quality-guided allocation from hard budget enforcement is a sensible design choice.
  • The evaluation shows promising results:
    • Experiments cover both controlled workflows and adapted multi-agent applications.
    • Several budget-allocation baselines are evaluated, together with ablation studies examining the contributions of different system components.
  • The authors explicitly acknowledge experimental limitations, including missing evaluation labels and weaker results in certain settings.
Critical weaknesses
  1. Is the quality model reliable when budgets are shared?

My main concern is whether the learned quality model remains reliable when multiple agents share a common budget. Section 4.1 estimates each agent's budget–quality curve by varying its budget while holding the others fixed. However, the actual optimization adjusts multiple agents' budgets simultaneously, potentially introducing interactions between them. Equation (10) combines the independently fit-ted curves using a geometric mean, but the validity of this composition for ranking joint allocations is not sufficiently evaluated.

  • Appendix B.1 evaluates the individual response curves in isolation. Some curves perform only marginally better than a constant predictor.
  • The curves also remain fixed across tasks and throughout execution, preventing the quality model from adapting to the specific requirements of individual tasks.
  1. The benefit of the learned model is not clearly demonstrated.

The baselines demonstrate improvements over equal and historical allocations, but they do not establish whether a simpler quality-aware allocation policy could achieve comparable performance. For example, a controller could use agent priorities tuned during calibration while retaining the same budget ac-counting and online replanning mechanisms. Such an approach would not require fitting budget–quality curves or solving the proposed allocation optimization.

  • Calibration requires numerous interventions and downstream executions. The paper should quantify this overhead and evaluate how well the calibration generalizes when tasks or model backends change.
  1. The value of graph selection is not clearly demonstrated.

All three native applications use fixed execution graphs. In the controlled experiments, graph selection provides almost no improvement on Pipeline (−0.0025 AUC). On Parallel, it improves AUC by +0.075, but QOBC consistently selects the one-expert configuration. Consequently, it is unclear whether the improvement comes from effective graph selection or simply from using a smaller fixed workflow.

  1. The results are not consistent across experiments.

On Screenplay, Historical Proportional achieves a higher AUC than QOBC (0.517 vs. 0.471). Similarly, in the graph-matched Parallel ablation, Static Quality marginally outperforms QOBC (0.6267 vs. 0.6167), although the difference is small. These results suggest that the benefits of quality guidance and online replanning are not consistent across workflows.

Questions for authors
  1. How accurately does the quality surrogate rank held-out allocations in which multiple agents are simultaneously constrained? How sensitive are these rankings to calibration-set size and extrapolation beyond the observed budget range?

  2. Could QOBC be compared against a simpler calibration-tuned online policy that uses the same execution graph, budget accounting, and replanning mechanism? What is the calibration cost for each workflow, and how robust is the resulting policy to task heterogeneity?

  3. How does graph-selecting QOBC compare against a fixed one-expert configuration or a simple budget-dependent graph-selection policy? Could the authors clarify how the reported ranking accuracy relates to the observed performance improvements?

Reviewer_fRo1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents Quality-Aware Online Budget Control (QOBC), a controller for single-round LLM-based multi-agent workflows operating under hard per-request budgets. It estimates agent-specific budget–quality responses through offline interventions, incorporates both generation costs and downstream input costs into allocation, and reallocates the remaining budget after each execution batch. When permitted, it also selects an execution graph from a predefined family of admissible alternatives. A separate reservation-and-settlement mechanism enforces budget constraints under explicit accounting assumptions. Experiments cover two controlled structures and three adapted native workflows.

Critical strengths

Accounting for the downstream input costs induced by upstream outputs is an insightful design choice. Integrating this dependency with budget–quality curves provides a useful basis for allocating resources according to end-to-end benefit rather than local generation cost.

Critical weaknesses
  1. The presentation is insufficiently self-contained, with unexplained acronyms, delayed attribution, and important definitions deferred until much later (only 22 references in total).

Several concrete examples increase the effort required to follow the paper:

  • KKT: The introduction refers to a “KKT-based solver” in line 63, and Section 4.3 later invokes KKT stationarity. A brief explanation at first use would improve accessibility.

  • MoA: MoA appears in the abstract and introduction without being expanded to Mixture-of-Agents or accompanied by a citation at those initial mentions. Earlier attribution would help readers distinguish the established framework.

  • VPA: Section 5 invokes “Kubernetes VPA” to motivate Historical Proportional allocation, but does not expand it to Vertical Pod Autoscaler or briefly explain the connection between that mechanism.

  • Content-F1: Figure 3 already uses Content-F1 to present calibration results, but its operational definition is deferred to Appendix C.2. A concise definition should accompany its first use.

These are not all cases of information being entirely absent, whose accumulation makes the main text harder to understand.

  1. Substantial overlap with an omitted prior work, together with insufficient comparisons against established methods, weakens both the novelty claim and the empirical case.

The paper substantially overlaps with ZEBRA: Zero-shot Budgeted Resource Allocation for LLM Orchestration, yet does not cite or compare against it. ZEBRA already addresses fixed-budget allocation across LLM workflow stages using budget–utility curves, saturating exponential responses, multiplicative utility aggregation, and water-filling optimization. Its Appendix F also implements and evaluates online budget reallocation. This overlap substantially weakens QOBC’s methodological novelty.

The evaluation relies on simple allocation heuristics and ablations of QOBC itself, rather than comparisons with the closest prior approach. ZEBRA and its online reallocation variant are essential baselines. Without these comparisons, the reported gains do not establish an advantage over existing quality-aware allocation methods. Moreover, the absence of a dedicated ablation of downstream input-cost modeling leaves the value of QOBC’s most appealing distinction unverified

Questions for authors

Why is ZEBRA neither cited nor compared against? Was it overlooked during the literature review, considered concurrent work, or excluded because of reproduction costs? Please clarify the technical differences between QOBC and ZEBRA and explain why a direct comparison was not conducted. If reproduction was a barrier, what specific difficulties prevented the comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.