- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies cost-or-coverage orchestration: an LLM must infer which sources a query needs and engage a cheap set of access-scoped agents that covers them. It builds CoC-Bench (178 two-source questions, eight agents with overlapping scopes, two scope-based pricing schemes, oracle retrieval) and finds that models either route narrowly and miss sources or route broadly and overspend.
Critical strengths
- The setup isolates routing cleanly. Oracle retrieval, the corpus-wide uniqueness audit, and the gold_route condition separate source inference from agent allocation.
- The diagnostic conditions are well chosen: unit vs. agent pricing, repricing with known sources, re-granted teams, and paired bill-disclosure contrasts in revision.
- Uncertainty reporting is careful. Contrasts are paired by question and draw. An effect is flagged only when it exceeds both bootstrap and draw-level noise.
Critical weaknesses
- The central claim is not identified. Gold_route shows that allocation is easy once sources are known. Low first-attempt coverage may therefore only mean that source inference is hard, or even impossible for questions stripped of platform names. The paper never measures how inferable the sources are.
- There is no reference point for good performance. Costs are compared with the hindsight optimum c*, so every model falls short by construction, and no simple baselines are reported. Under access-scope pricing, a fixed product_mgmt + sre_oncall route covers all ten units at about 39% of full-team cost. IDENT also lacks a size-matched chance level: Scout’s 60% under agent_pricing is close to what random picks of 5.5 agents give.
- The cost model penalizes broad routing. Prices scale linearly with scope size, yet Appendix D shows prompt tokens growing only about 2.2x over an 80x larger scope. Oracle retrieval also removes the recall gain from redundant agents. The reported 7x to 12x over-engagement depends on these choices.
Questions for authors
- When asked only to name the required source units, how accurate are the models? How does a predict-then-solve pipeline (predicted sources passed to an exact set-cover solver) compare with direct routing?
- The round-one bill can be computed from prices the model already has. Would feedback that carries new information, such as the cheapest covering cost, change the conclusion that cost feedback does not produce cost-effective correction?
- With one top-up round, Qwen3.5-122B reaches 83% coverage at EFF-OPT 3.31. Llama-4-Scout, in a single round, scores 6.98. If both rounds are scored on total spend c(S1∪S2), is narrow commitment a failure mode or a reasonable exploration strategy?
2 comments
Author Response (Part 2/2): Questions
We address the three questions below.
Q1. Source inference baselines
To further investigate this distinction, we implemented the Source-inference baseline where an LLM predicts candidate sources and an exact minimum-cost set-cover algorithm selects the agents. The results are shown in the table below.
| Model | Condition | COVER | #AGENTS | COST | COST-R |
|---|---|---|---|---|---|
| GPT-5.6-sol | Agent-pricing | 28% | 1.24 | 124.2 | 1.36 |
| Gold-route | 100% | 1.48 | 219.8 | 1.00 | |
| Source-inference (k=2) | 44% | 1.07 | 222.8 | 1.87 | |
| Source-inference (k=4) | 62% | 1.44 | 322.8 | 2.31 | |
| Source-inference (k=6) | 82% | 1.70 | 421.8 | 2.98 | |
| Source-inference (k=8) | 96% | 2.00 | 511.7 | 3.71 | |
| Fixed-agents (Full-coverage) | 100% | 2.00 | 534.0 | 3.85 | |
| Qwen3.5-122B | Agent-pricing | 38% | 1.86 | 207.4 | 2.35 |
| Gold-route | 100% | 1.48 | 219.8 | 1.00 | |
| Source-inference (k=2) | 35% | 1.09 | 224.6 | 2.05 | |
| Source-inference (k=4) | 58% | 1.38 | 326.9 | 2.53 | |
| Source-inference (k=6) | 83% | 1.74 | 412.4 | 3.15 | |
| Source-inference (k=8) | 97% | 2.01 | 501.0 | 3.67 | |
| Fixed-agents (Full-coverage) | 100% | 2.00 | 534.0 | 3.85 | |
| Llama-4-Scout | Agent-pricing | 90% | 5.53 | 934.2 | 7.28 |
| Gold-route | 99% | 4.39 | 830.2 | 5.82 | |
| Source-inference (k=2) | 17% | 1.01 | 118.8 | 1.46 | |
| Source-inference (k=4) | 43% | 1.61 | 262.9 | 2.57 | |
| Source-inference (k=6) | 63% | 2.01 | 367.6 | 2.90 | |
| Source-inference (k=8) | 90% | 2.19 | 483.5 | 3.35 | |
| Fixed-agents (Full-coverage) | 100% | 2.00 | 534.0 | 3.85 |
Table: Access-scope pricing. Source-inference uses LLM-ranked top-k sources and an exact solver; Fixed-agents covers all sources. COST averages all queries; COST-R is the mean ratio of cost to the query-specific minimum cost over covered queries.
For GPT and Qwen3.5-122B, Gold-route achieves near-optimal agent selection, while direct routing suffers from low coverage. The predict-then-solve approach achieves substantially higher coverage as the predicted source set expands, although at additional cost. In contrast, Llama-4-Scout exhibits weaker source inference performance and continues to over-engage agents even when gold sources are provided. Notably, the predict-then-solve approach achieves comparable coverage to Scout's direct routing at substantially lower cost.
The fixed full-coverage baseline is also included in the table for comparison. As clarified in W2, it guarantees coverage but does not provide query-specific cost-efficient routing.
Q2. Cost feedback in revision
We tested a third feedback condition, bill + c*, to both revision settings (blind do-over and outcome-shown top-up). Specifically, we added the following information to the prompt: "The minimum possible total cost for this question is X." We evaluated this condition using Qwen3.5-122B under access-scope pricing, following the same three-draw protocol.
| Condition | COVER | IDENT | #AGENTS | COST-R |
|---|---|---|---|---|
| Round one | 38% | 20% | 1.86 | 2.35 |
| Do-over + bill | 31% | 20% | 1.63 | 1.79 |
| Do-over + bill + c* | 90% | 89% | 1.59 | 1.00 |
| Top-up + bill | 78% | 40% | 2.40 | 2.35 |
| Top-up + bill + c* | 87% | 84% | 1.81 | 1.09 |
Table: Revision results for Qwen3.5-122B under access-scope pricing. c* denotes the query-specific minimum covering cost. Metrics are computed on the complete second-turn assignment.
The results show substantial improvements, but also reveal an important issue: providing c* can indirectly reveal the optimal agent combination. For example, if support_eng is the only agent combination costing 296, telling the model that the minimum cost is 296 effectively identifies the agent to select.
This explains the large improvement from do-over + bill to do-over + bill + c*, even without retrieval outcome feedback. Therefore, the improvement cannot be attributed solely to better cost-aware reasoning, since the additional information may effectively reveal the answer.
Q3. Narrow commitment and multi-round exploration
We agree that narrow initial routing may be a reasonable exploration strategy in a multi-round setting. However, our primary task evaluates a single agent-selection decision before retrieval feedback is available. The revision experiments in Section 5 are designed to examine whether models can repair earlier routing failures after receiving feedback, rather than to evaluate an optimized multi-round routing strategy. We agree that studying how models balance initial exploration, subsequent correction, and cumulative costs in multi-round orchestration is an interesting direction for future work.
Author Response (Part 1/2): Critical Weaknesses
We thank the reviewer for the detailed feedback. We address the concerns below.
W1. Central claim
We would like to clarify that Gold-route is designed to isolate agent selection from source uncertainty, rather than suggest that agent allocation is universally easy. While some models achieve near-optimal selection when sources are known, others continue to over-engage agents even under Gold-route. Our central claim is that LLMs struggle to assign tasks to agents cost-efficiently while satisfying source coverage requirements under uncertainty. This joint task cannot be reduced to source identification alone or to agent selection when the required sources are given, as the orchestrator must make allocation decisions without knowing exactly where the required evidence is located. We further investigate this distinction through the new decomposition baseline reported in our response to Q1.
W2. Evaluation metrics
We would like to clarify that c* is the query-specific minimum covering cost, providing an exact reference for evaluating routing efficiency rather than an arbitrary fixed baseline. Models can achieve COST-R = 1, as demonstrated under Gold-route.
The suggested fixed product_mgmt + sre_oncall route guarantees coverage by accessing all ten sources, but does not perform query-specific cost optimization. While this solution may appear straightforward to a human observer, some models engage even more agents than this fixed pair. This illustrates that models do not necessarily succeed at reasoning about source requirements and selecting appropriate agents, even when a simple full-coverage solution exists. Identifying where and why models struggle with these decisions is precisely the motivation for our evaluation.
Importantly, our five metrics are designed to capture complementary aspects of routing performance and should be interpreted jointly. IDENT measures whether the selected agents include an optimal covering combination, not whether the entire assignment is cost-efficient. For example, Scout's relatively high IDENT comes with substantial redundant engagement, reflected in its high COST-R and EFF-OPT. Therefore, our conclusions do not rely on IDENT in isolation or require it to serve as a size-adjusted measure of routing quality.
W3. Cost assumptions
We agree that excess-cost magnitudes depend on pricing, but making broader engagement more expensive does not itself establish an unfair penalty: our benchmark asks whether models choose economical routes under explicitly stated costs. Crucially, under gold_route, source uncertainty is removed, yet Scout still spends 5.82× and 7.97× the minimum covering cost under the two schemes, while several other models achieve essentially complete coverage near the optimum. This demonstrates avoidable expenditure under the specified costs. Appendix D motivates breadth-based proxies rather than proving linear scaling; moreover, the reviewer’s 80× scope range and 2.2× aggregate token comparison are not matched scaling measurements. Oracle retrieval deliberately isolates routing from retrieval reliability, so redundancy’s potential recall benefits fall outside this controlled setting. We will clarify that the quantitative ratios are conditional on our pricing assumptions.