- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes QOBC, a quality-aware online budget control framework for LLM multi-agent systems (MAS). Instead of merely reducing average costs with unbounded budgets, QOBC explicitly formulates the objective as maximizing expected task scores under strict per-request token constraints. The framework consists of three stages: offline calibration (fitting agent-specific budget-quality response curves via single-agent interventions), pre-execution (selecting the optimal graph from candidate topologies using a geometric mean surrogate), and online reallocation (using a KKT solver to dynamically redistribute output token caps for remaining agents after each call batch, factoring in both generation and downstream input costs).
The method's theoretical guarantee against budget overruns under explicit accounting assumptions is empirically validated by a zero violation rate across 9,000 conditions. Extensive experiments on two controlled structures (Sequential Pipeline and Parallel on MATH) and three real-world applications (MoA, CrewAI Screenplay Writer, Corex Review) demonstrate its effectiveness. QOBC improves the normalized budget-quality AUC by 0.220 and 0.230 over online equal allocation on controlled structures, and beats static quality allocation by up to 0.169 on real-world tasks, notably boosting the AUC of Corex Review from 0.570 to 0.739.
Critical strengths
-
Rigorous budget safety guarantee.Theorem 1 proves that cumulative cost never exceeds B under explicit accounting assumptions (valid input bounds, enforced output caps, complete billing records), and this guarantee is verified empirically with zero violations across 9,000 test conditions. The separation between quality prediction and budget enforcement is technically sound and practically important.
-
Downstream input cost modeling.By pricing each output token together with its induced downstream reading cost (the π_i term), the planner captures the true marginal cost of allocating tokens to upstream agents—a dimension consistently overlooked in prior MAS efficiency work and directly relevant to real deployment economics.
Critical weaknesses
-
Absence of empirical comparisons against direct competitors. BAMAS (Yang et al., 2026) and FutureWeaver (Jung et al., 2026) share highly overlapping problem formulations with this work. While cited, they are conspicuously absent from the experimental baselines. Merely outlining technical differences in the text is insufficient to substantiate your novelty claims. Without head-to-head quantitative comparisons, the exact incremental contribution of QOBC cannot be rigorously assessed.
-
Inadequate analysis of the Screenplay failure case undermines the framework's core assumption. The Historical Proportional baseline noticeably outperforms QOBC on the overall AUC for Screenplay ([0.511, 0.522] vs. [0.469, 0.474]), despite relying entirely on heuristics rather than learned curves. Correlating this with Table 4, the κ values for the three Screenplay content agents are near zero. This implies nearly flat budget-quality curves—the precise regime where quality-guided optimization loses its edge. The authors gloss over this finding in a single sentence. There is no parameter-level post-mortem, no discussion on when practitioners should anticipate such cases, and no proposed detection or fallback mechanisms. This oversight threatens the central premise of QOBC: that agents exhibit meaningful, differential responses to varying budgets.
Questions for authors
-
On the Screenplay failure case and the flat κ response The Historical Proportional baseline outperforms QOBC on the overall AUC for Screenplay, despite operating completely independent of any learned curves. According to Table 4, the κ values for all three Screenplay content agents are near zero (reaching a maximum of only 0.02081), indicating that their budget-quality curves are essentially flat across the experimental range. Do you agree that this flat response is the root cause of QOBC losing its advantage in this scenario? If so, would it be possible to introduce an a priori diagnostic mechanism based on the fitted parameters (e.g., a minimum κ threshold) during the calibration phase? Such a mechanism could predict whether QOBC will actually yield benefits over simple heuristics, thereby guiding when to apply or bypass your framework.
- Direct comparison with overlapping literature Both BAMAS (Yang et al., 2026) and FutureWeaver (Jung et al., 2026) align very closely with your problem formulation, yet they are only differentiated textually in the Related Work. Are there specific technical or experimental hurdles preventing a head-to-head empirical comparison? If their official code is unavailable, would it be feasible to implement a proxy baseline (e.g., a BAMAS-style model selection strategy) in your controlled settings during the rebuttal? Providing some form of quantitative comparison—even an approximated one—would help significantly in clearly delineating QOBC's exact contribution boundary.
If you can effectively resolve my issue, I will consider increasing the points.