acceptodds
Under review as a conference paper at ICLR 2027

Which Model for Which Role? Cost-Efficient Pareto Recommendation for LLM Agentic Workflows

Abstract

LLM agentic workflows increasingly employ different models for specialized roles. Choosing among these assignments requires balancing task quality against per-query API cost. Finding good assignments incurs a separate search cost from evaluating multiple candidates. We introduce Cost-Coupled Gittins (CC-Gittins), which exploits a shared cost structure: assignments that cost more to run also cost more to evaluate. Gittins controllers for quality and cost share observations and independently relax their evaluation-cost penalties to allocate further evaluations. We evaluate CC-Gittins by offline replay on eight quality-cost matrices from workflows with two to five roles and up to 1,681 candidate assignments. Across the eight quality-cost benchmarks, CC-Gittins recovers a substantial fraction of the true Pareto frontier with few false positives at roughly 10% of the exhaustive-evaluation cost, and recovers most of the frontier by 30%.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM agents, multi-role model selection, multi-objective optimization, cost-aware search, Pareto frontier, Gittins index

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CMYGOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies the selection of models for different roles in an LLM workflow, with the aim of returning several quality-cost trade-offs under a limited search budget. The proposed CC-Gittins method alternates between quality- and cost-oriented scalar controllers. Both controllers use the same observations, but adjust their evaluation-cost penalties separately. A conservative readout then returns a Pareto set using the current quality and cost estimates. Experiments replay eight response matrices, including two complete two-role assignment spaces and six retained configuration sets from SCOPE. The reported results recover useful portions of the frontier at about 10% of the exhaustive API evaluation cost, with coverage improving as the budget increases.

Critical strengths
  1. The problem formulation makes a useful distinction between the cost of deploying an assignment and the cost of finding it. Evaluating a complicated agent workflow can itself be expensive, so measuring search spend is appropriate. Returning multiple trade-offs is also useful when the user's preferred balance between quality and cost is not known in advance.

  2. The experiments cover several workflow settings and include both multi-objective search methods and random-search controls. The reported hypervolume and precision results suggest that partial evaluation can identify useful assignments. Using the same recorded outcomes across methods makes the search-cost comparisons easier to interpret.

Critical weaknesses
  1. I find the contribution fairly incremental. CC-Gittins combines cost-aware scalar controllers through shared observations and alternating visits, followed by a conservative Pareto readout. The construction is reasonable, but I am not convinced that the two-controller design adds enough over simpler cost-aware search with the same readout. Complete role assignments are still treated as independent configurations, so the method does not use the workflow structure to reduce the combinatorial search space.

  2. The distance plots give an incomplete view of recommendation quality. Appendix G states that truly dominated points are removed before GD and IGD are computed. These metrics therefore leave out dominated configurations that a user would also receive. Hypervolume is unaffected, and the precision results provide useful evidence, but the complete returned sets still need to be assessed under the stated distance metrics. The larger SCOPE instances also use retained configuration subsets, so their budget savings are relative to those subsets rather than exhaustive evaluation of all role assignments.

The writing repeats the search objective at length, while the evaluation conventions are harder to follow.

Questions for authors
  1. How do the comparative GD and IGD results change when computed on the full returned sets, without first removing truly dominated points?

  2. How much of the improvement comes from selecting evaluations, and how much comes from the conservative readout? What happens if the random-search controls or another acquisition method use the same readout as CC-Gittins?

  3. How does the cost of warm-starting all candidate assignments scale as more roles and model choices are added? For larger workflows, does the method require a preselected subset of assignments?

  4. How well do assignments recommended from one query set perform on held-out queries from the same task?

2 comments

Thank you for the feedback on the presentation and on the evidence for the two-controller design. We clarified the scoring convention and added frontier plots for the original allocation control.

1. Recommendation scoring. Yes, the main conclusions remain qualitatively similar under full-set evaluation. Sec. 2 of the submission defined GD and IGD on the complete returned set. The revised curves apply that definition to every method (Appendix A, Figs. 8–9). The earlier filter gave several baselines lower GD, and the overall comparison remains similar under full-set scoring. Fig. 18 reports precision and recall under a shared recommendation rule.

2. Allocation vs. recommendation. Both components contribute, and the advantage persists under a common readout. Comparing G0 against quality-only G1 with identical posterior models, acquisition prices, and recommendation rules, adding the cost controller improves mean precision on all eight benchmarks and recall on seven at 10% evaluation cost (Fig. 18). For example, recall increases from 0.58 to 0.89 on BIRD Dev and from 0.07 to 0.42 on Bing Query Logs.

Moreover, Appendix K.6 compares CC-Gittins and random-question sampling under both conservative and posterior-mean readouts, using the same posterior modeling approach. Under either readout, CC-Gittins achieves lower mean HV regret on all eight benchmarks at both 10% and 30% evaluation cost (Fig. 25). Holding acquisition fixed, conservative readout lowers GD on seven benchmarks at 30% cost, while increasing IGD on seven and HV regret on six. These experiments distinguish the benefit of adaptive acquisition from the trade-off introduced by conservative recommendation.

3. Warm-start scaling and candidate pools. Warm-start cost scales linearly with the candidate pool size, requiring 4K4K evaluations for KK configurations. For example, Bing Query Logs has 1,681 configurations and requires 6,724 initial evaluations. Since we treat complete assignments as independent arms, the candidate space can grow exponentially with the number of roles. For larger workflows, a preselected candidate pool can be used. Our reported savings are relative to exhaustive evaluation within these pools, as clarified in Sec. 4.1 and Appendix E.1.

4. Held-out questions. Yes, the recommendations retain high hypervolume on the two evaluated held-out tasks. Appendix M evaluates frozen recommendations from Restaurant Valid and BIRD Mini Dev on disjoint test queries, using fixed shared candidate pools. At 30% search cost, the recommendations retain 99.93% of test-frontier hypervolume on Restaurant and 99.999% on BIRD. Mean precision is 0.48 on Restaurant and 1.00 on BIRD, with mean recall of 2/3 on both tasks. These results show that the selected configurations retain nearly all test-frontier hypervolume on both held-out tasks.

Thank you for the clarification. I will maintain my original rating.

Reviewer_cnPuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Pareto recommendation of model assignments for multi-role LLM workflows under limited API evaluation budgets. CC-GITTINS alternates quality- and cost-oriented Gittins controllers, shares observations, and independently relaxes evaluation-cost penalties. It returns a Pareto set based on uncertainty-adjusted estimates. Evaluation uses offline replay on eight quality–cost matrices, with comparisons against five baselines and component ablations.

Critical strengths
  1. The distinction between deployment cost and evaluation expenditure is practically relevant. Accounting for their coupling directly addresses the expense of searching over workflow assignments.

  2. The evaluation covers eight instances and includes substantial component ablations. Matched-budget comparisons and repeated replay seeds provide useful empirical evidence.

Critical weaknesses
  1. The reported distance metrics do not evaluate the complete recommendation set. Appendix G states that GD and IGD are computed after filtering recommendations using their exhaustive, true-coordinate dominance relationships, whereas Equations (3)–(4) define these metrics on the full output. This filtering requires information unavailable during search. In particular, filtered GD can conceal poor additional recommendations, while hypervolume also does not penalize dominated extras. The reported precision/recall diagnostics are useful, but do not resolve the missing full-output distance comparisons. This matters because the central deliverable is the recommendation set actually returned to the user.

  2. The additional methodological contribution is insufficiently established. Cost-aware scalar Gittins allocation and uncertainty-adjusted recommendation already appear in Xie et al. (2026). The main extensions are interleaving two controllers, independently relaxing their prices, and constructing a Pareto readout. Proposition 1 explains a restrictive deterministic case but provides limited justification for the stochastic Pareto-search policy. The ablations support these implementation choices, but comparisons against simpler cost-aware Pareto strategies are missing. Consequently, the evidence does not convincingly establish a substantial advantage attributable to the proposed controller design.

Reference: Qian Xie, Yueli He, and Nairen Cao. Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices. 2026. https://arxiv.org/abs/2609.25645

Questions for authors
  1. After recomputing GD and IGD on the complete, unfiltered outputs of every method, do the main comparisons and ablation conclusions remain unchanged? Please also provide full-output precision, recall, and set sizes across all eight instances.

  2. How does CC-GITTINS compare with cost-normalized hypervolume acquisition or a portfolio of scalarized cost-aware Gittins policies under matched budgets? Can a common recommendation rule isolate acquisition quality from differences in readout?

1 comment

Thank you for the constructive feedback. We have updated the experimental evaluation, added cost-aware baselines, and clarified the contribution of the two-controller design.

1. Recommendation scoring. Yes, the main comparisons and ablation conclusions remain qualitatively similar. We recomputed all results using complete, unfiltered recommendation sets (Appendix A, Figs. 7–9). The correction changes several baselines' GD scores, but CC-Gittins remains competitive. Fig. 18 additionally reports full-output precision and recall for the matched-readout G0–G1 comparison.

2. Cost-aware baselines and common readout. Yes, we compare against cost-normalized hypervolume acquisition and evaluate acquisition separately from the readout. The revision includes qNEHVI-PC, which normalizes expected hypervolume improvement by estimated evaluation cost, as well as CoPSI and CoHV-UCB (Sec. 4.1 and Appendix E.4). At 30% cost, CC-Gittins achieves lower GD than qNEHVI-PC on HotpotQA, MathQA, and both Restaurant splits, while qNEHVI-PC has lower IGD on MathQA and Restaurant Test. The larger instances also reveal substantial computational overhead for qNEHVI-PC (Appendix H). The qNEHVI-PC results on BIRD Mini Dev and Bing Query Logs remain incomplete due to its long runtime;

Appendix K.6 isolates acquisition and recommendation effects by comparing CC-Gittins and random-question sampling under common posterior-based readouts. Under either conservative or posterior-mean readout, CC-Gittins achieves lower mean HV regret on all eight benchmarks at both 10% and 30% evaluation cost (Fig. 25). Holding acquisition fixed, conservative readout reduces mean GD on seven benchmarks at 30% cost, at the expense of increased IGD on seven. Thus, the acquisition advantage cannot be attributed solely to the conservative readout.

3. Contribution of the controller design. Extending scalar Gittins to anytime Pareto recommendation is not simply duplicating a controller: it requires choosing search directions and coordinating their evaluations. We build on the scalar allocation and uncertainty adjustment of Xie et al. (2026); our central contribution is the insight that cost coupling enables effective Pareto recommendation with a compact set of complementary controllers—just two in the quality–cost setting—together with the design choices that make their coordination effective. Compared with quality-only G1, adding the cost controller improves mean precision on all eight benchmarks and recall on seven at 10% evaluation cost, with posteriors, acquisition prices, and readout fixed (Fig. 18).

The remaining question is how many directions are needed. A natural hypervolume-oriented extension would search many weight vectors using Chebyshev-style hypervolume scalarizations (Zhang and Golovin, 2020). Proposition 1 illustrates the mechanism under deterministic assumptions: relaxing the quality controller's price changes its implicit deployment-cost weight. The cost controller targets inexpensive configurations, and the shared readout can return interior trade-offs. Offline, supported points retain over 99.9% of frontier HV across all benchmarks—six of 15 Pareto points on HotpotQA (Appendix F).

In the direction ablation, two controllers generally match or improve HV regret and IGD over three or five with less total policy computation time, although extra directions sometimes improve GD (Figs. 17 and 19). Independent relaxation also lowers mean HV regret versus synchronized decay on six of seven benchmarks at 30% cost, with one tie (Appendix K.3). These findings support the design empirically, without establishing complete-frontier recovery.

Reference

Richard Zhang and Daniel Golovin. Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization. ICML, PMLR 119:11096–11105, 2020.

Reviewer_2giLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CC-Gittins for cost-efficient Pareto recommendation of model assignments in multi-role LLM workflows, using quality and cost controllers that share observations to allocate a limited API evaluation budget.

Critical strengths
  1. The paper studies a practical problem: reducing the evaluation cost of finding useful quality–cost trade-offs in LLM workflows.
  2. The formulation clearly distinguishes evaluation cost from deployment cost, and the overall method is easy to follow.
Critical weaknesses
  1. The claim of few false positives is not systematically established across all eight benchmarks, as the main GD/IGD results assess ground-truth-filtered subsets rather than complete outputs (Sec. 2 and Figs. 3–4).
  2. The specific benefit of independent price relaxation in Sec. 3.3 remains unclear because the continuation ablation changes both the price schedule and stopping behavior (Appendix K.3).
  3. The comparisons in Sec. 4 leave EGE-SH’s budget configuration unclear and do not separate sampling gains from differences in recommendation rules (Sec. 3.5 and Appendix E.4).
  4. The cost scale RR, residual variances, and index-computation settings in Secs. 3.1–3.2 are not fully specified, limiting reproducibility from the manuscript alone (Section 3 and Appendix C).
Questions for authors
  1. Do the main conclusions hold when the complete returned sets are evaluated without ground-truth filtering, with GD/IGD, precision, recall, and set size reported across all benchmarks and main baselines?
  2. Does independent price relaxation outperform a shared or synchronized price schedule when the other components, including the stopping logic, are matched as closely as possible?
  3. Do the gains from configuration-specific evaluation costs persist against a constant-cost control with a matched initial mean evaluation penalty?
  4. Are EGE-SH’s 10% and 30% results obtained from budget-specific runs or prefixes of a longer run, and how are heterogeneous API costs incorporated into its budget allocation?
  5. The discussion in Sec. 1 would benefit from relating CC-Gittins to work onc ost-aware Pareto identification (CoPSI [1]) and budget-constrained workflow execution (MCPP [2]) .
  6. How much of the advantage over random-question sampling remains when both methods use the same posterior model and conservative recommendation rule?
  7. Can the authors specify how the cost scale RR and residual variances are determined, together with the numerical settings used to compute the Gittins indices?

[1] Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluation. arXiv:2608.04333, 2026.

[2] On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows, NeurIPS, 2026.

2 comments

Thank you for the detailed feedback. We added cost-aware baselines and clarified the replay protocol.

1. Recommendation scoring. Yes, the main conclusions remain qualitatively similar under full-set evaluation. Sec. 2 of the submission defined GD and IGD on the complete returned set. The revised curves apply that definition to every method (Appendix A, Figs. 8–9). The earlier filter gave several baselines lower GD, and the overall comparison remains similar under full-set scoring. Fig. 18 reports precision and recall under a shared recommendation rule.

2. Independent versus synchronized relaxation. Yes, independent relaxation generally outperforms synchronized relaxation under matched stopping logic. Appendix K.3 (Fig. 21) introduces a synchronized control that halves both multipliers whenever either controller triggers an admissible price decrease. Across seven benchmarks and 20 paired seeds, independent relaxation achieves lower mean HV regret on 7/7 benchmarks at 10% cost and 6/7 at 30% (one tie). At 30%, it also achieves lower GD on all seven and lower IGD on five, with one tie and one regression. These results support independent relaxation while acknowledging metric-dependent differences.

3. Configuration-specific versus mean-matched prices. Yes, configuration-specific pricing generally improves HV regret over a mean-matched constant-price control. Appendix K.7 (Fig. 28) compares our acquisition prices κ^i=BDˉiwarm\hat\kappa_i=B\bar D_i^{\mathrm{warm}} against a constant κˉ=K−1∑iκ^i\bar\kappa=K^{-1}\sum_i\hat\kappa_i, keeping other components fixed. Across seven benchmarks and 20 paired seeds, configuration-specific pricing achieves lower mean HV regret on 6/7 benchmarks at 10% cost and 5/7 at 30% (one tie). On MathQA, constant pricing improves HV regret and IGD but reduces precision from 0.94 to 0.45 at 30% cost. Thus, heterogeneous pricing generally helps, with benchmark-dependent trade-offs.

4. EGE-SH budgets. Yes, the revised results use independent, budget-specific EGE-SH runs. The submitted version took the 10% and 30% checkpoints from a single longer EGE-SH run. In the revision, we run EGE-SH separately with 10% and 30% of the exhaustive observation count as input to its successive-halving schedule. The newly added CoPSI baseline also uses separate 10% and 30% input budgets, expressed in evaluation cost (Sec. 4.1 and Appendix E.4). This gives both baselines advance knowledge of the target budget, whereas CC-Gittins uses a single anytime trajectory. Despite this advantage, CC-Gittins typically achieves lower three-objective HV regret and GD at 30% cost (Sec. 4.3).

5. Related work. Yes, both suggested papers are now discussed in Sec. 1.1. We additionally include CoPSI and CoHV-UCB in the experiments. Your suggestion of MCPP also inspired us to visualize the three-objective Pareto frontier directly. The submission showed only metric curves for this setting; the revised main text includes a 3D quality–cost–latency frontier plot (Fig. 6) alongside those curves (Fig. 5).

6. Separating allocation and recommendation. Yes, the advantage persists under a common readout. Appendix K.6 compares CC-Gittins against random-question sampling using common posterior-mean and conservative readouts. Under either readout, CC-Gittins achieves lower mean HV regret on all eight benchmarks at both 10% and 30% evaluation cost (Fig. 24). It also achieves lower mean IGD on all eight benchmarks under posterior-mean readout and seven under conservative readout at both checkpoints (Fig. 25). Moreover, adding the cost controller improves recall on seven of eight benchmarks at 10% cost under the same readout (Fig. 18).

7. Calibration and index computation. The submission already specified prior calibration, warm-start and batch sizes, and the numerical stopping rule. We have expanded Appendix C to specify how RR is computed from warm-start costs, how the quality and cost observation-noise variances are set, and how the Gittins indices are computed using a 129-point state grid, Brent's root-finding method, and explicit error tolerances.

Thank you for your clarification.I share the concern that the methodological advance over existing cost-aware scalar Gittins search remains modest. Therefore, I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal: Key Revisions and Additional Evidence

We sincerely appreciate the reviewers' feedback. During rebuttal, we substantially revised the paper and conducted additional experiments to address their concerns.

1. Revised evaluation and strengthened baselines. In response to the reviewers' concerns, we recomputed GD and IGD on complete, unfiltered recommendation sets, aligning the evaluation with the actual outputs. Our main conclusions remain intact; notably, the original filtering favored several baselines by lowering their reported GD. We also added three cost-aware baselines (qNEHVI-PC, CoPSI, and CoHV-UCB) and evaluated EGE-SH (and CoPSI) using budget-specific runs (a stronger setting for EGE-SH, which knows the target budget in advance, unlike our anytime method).

2. Additional experimental evidence. In direct response to the reviewers' suggestions, we conducted additional controlled experiments disentangling the effects of adaptive acquisition and conservative recommendation, further examined the two-controller design, and evaluated generalization to held-out queries. The results demonstrate that CC-Gittins retains its advantage over random-question sampling under a matched posterior and recommendation rule, while its recommended configurations maintain strong hypervolume performance on unseen queries (Appendix M).

3. Methodological contribution. We clarified that our work extends the single-objective cost-aware Gittins framework of Xie et al. (2026) to anytime multi-objective Pareto recommendation, which requires modeling uncertainty in both quality and deployment cost, as well as coordinating multiple search directions. Our key insight is that cost coupling enables a compact set of complementary controllers to efficiently explore Pareto trade-offs. In the quality–cost setting, adding the cost controller improves mean precision on all eight benchmarks and recall on seven at 10% evaluation cost (Fig. 18). The approach also extends to three objectives with four controllers. Proposition 1 provides theoretical intuition, while controlled experiments support the effectiveness of the design.

We believe the revised paper demonstrates a meaningful methodological contribution, supported by extensive controlled experiments that directly address the reviewers' concerns.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.