- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper studies the selection of models for different roles in an LLM workflow, with the aim of returning several quality-cost trade-offs under a limited search budget. The proposed CC-Gittins method alternates between quality- and cost-oriented scalar controllers. Both controllers use the same observations, but adjust their evaluation-cost penalties separately. A conservative readout then returns a Pareto set using the current quality and cost estimates. Experiments replay eight response matrices, including two complete two-role assignment spaces and six retained configuration sets from SCOPE. The reported results recover useful portions of the frontier at about 10% of the exhaustive API evaluation cost, with coverage improving as the budget increases.
Critical strengths
-
The problem formulation makes a useful distinction between the cost of deploying an assignment and the cost of finding it. Evaluating a complicated agent workflow can itself be expensive, so measuring search spend is appropriate. Returning multiple trade-offs is also useful when the user's preferred balance between quality and cost is not known in advance.
-
The experiments cover several workflow settings and include both multi-objective search methods and random-search controls. The reported hypervolume and precision results suggest that partial evaluation can identify useful assignments. Using the same recorded outcomes across methods makes the search-cost comparisons easier to interpret.
Critical weaknesses
-
I find the contribution fairly incremental. CC-Gittins combines cost-aware scalar controllers through shared observations and alternating visits, followed by a conservative Pareto readout. The construction is reasonable, but I am not convinced that the two-controller design adds enough over simpler cost-aware search with the same readout. Complete role assignments are still treated as independent configurations, so the method does not use the workflow structure to reduce the combinatorial search space.
-
The distance plots give an incomplete view of recommendation quality. Appendix G states that truly dominated points are removed before GD and IGD are computed. These metrics therefore leave out dominated configurations that a user would also receive. Hypervolume is unaffected, and the precision results provide useful evidence, but the complete returned sets still need to be assessed under the stated distance metrics. The larger SCOPE instances also use retained configuration subsets, so their budget savings are relative to those subsets rather than exhaustive evaluation of all role assignments.
The writing repeats the search objective at length, while the evaluation conventions are harder to follow.
Questions for authors
-
How do the comparative GD and IGD results change when computed on the full returned sets, without first removing truly dominated points?
-
How much of the improvement comes from selecting evaluations, and how much comes from the conservative readout? What happens if the random-search controls or another acquisition method use the same readout as CC-Gittins?
-
How does the cost of warm-starting all candidate assignments scale as more roles and model choices are added? For larger workflows, does the method require a preselected subset of assignments?
-
How well do assignments recommended from one query set perform on held-out queries from the same task?
2 comments
Thank you for the clarification. I will maintain my original rating.
Thank you for the feedback on the presentation and on the evidence for the two-controller design. We clarified the scoring convention and added frontier plots for the original allocation control.
1. Recommendation scoring. Yes, the main conclusions remain qualitatively similar under full-set evaluation. Sec. 2 of the submission defined GD and IGD on the complete returned set. The revised curves apply that definition to every method (Appendix A, Figs. 8–9). The earlier filter gave several baselines lower GD, and the overall comparison remains similar under full-set scoring. Fig. 18 reports precision and recall under a shared recommendation rule.
2. Allocation vs. recommendation. Both components contribute, and the advantage persists under a common readout. Comparing G0 against quality-only G1 with identical posterior models, acquisition prices, and recommendation rules, adding the cost controller improves mean precision on all eight benchmarks and recall on seven at 10% evaluation cost (Fig. 18). For example, recall increases from 0.58 to 0.89 on BIRD Dev and from 0.07 to 0.42 on Bing Query Logs.
Moreover, Appendix K.6 compares CC-Gittins and random-question sampling under both conservative and posterior-mean readouts, using the same posterior modeling approach. Under either readout, CC-Gittins achieves lower mean HV regret on all eight benchmarks at both 10% and 30% evaluation cost (Fig. 25). Holding acquisition fixed, conservative readout lowers GD on seven benchmarks at 30% cost, while increasing IGD on seven and HV regret on six. These experiments distinguish the benefit of adaptive acquisition from the trade-off introduced by conservative recommendation.
3. Warm-start scaling and candidate pools. Warm-start cost scales linearly with the candidate pool size, requiring 4K evaluations for K configurations. For example, Bing Query Logs has 1,681 configurations and requires 6,724 initial evaluations. Since we treat complete assignments as independent arms, the candidate space can grow exponentially with the number of roles. For larger workflows, a preselected candidate pool can be used. Our reported savings are relative to exhaustive evaluation within these pools, as clarified in Sec. 4.1 and Appendix E.1.
4. Held-out questions. Yes, the recommendations retain high hypervolume on the two evaluated held-out tasks. Appendix M evaluates frozen recommendations from Restaurant Valid and BIRD Mini Dev on disjoint test queries, using fixed shared candidate pools. At 30% search cost, the recommendations retain 99.93% of test-frontier hypervolume on Restaurant and 99.999% on BIRD. Mean precision is 0.48 on Restaurant and 1.00 on BIRD, with mean recall of 2/3 on both tasks. These results show that the selected configurations retain nearly all test-frontier hypervolume on both held-out tasks.