acceptodds
Under review as a conference paper at ICLR 2027

Cut My Pool Into Pieces: Rethinking Model Pool Construction for LLM Routing

Abstract

The enormous popularity of large language models (LLMs) has led to a vast and highly diverse landscape of models, varying in their strengths, weaknesses, and inference costs. As different models are suitable for different queries, LLM routing is increasingly applied to select an appropriate model for each query while balancing response quality and cost. Nonetheless, a router can only select among the models available in its pool, so the composition of that pool directly affects the routing system's quality-cost trade-off. Despite this, prior work has primarily focused on the router itself, paying considerably less attention to how the model pool should be constructed. In this work, we study model pool composition through pruning, systematically evaluating 10 pruning strategies across 27 benchmark tasks, 39 distinct models, and six router families. We find that well-pruned pools can preserve nearly all of the full pool's routing performance, while pruning strategies vary widely in their ability to serve low-cost budgets. Pool composition is particularly important for small pools, where its impact on routing performance exceeds that of router choice itself. Finally, pruning transfers well to unseen tasks, with performance close to that of in-task pruning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model routing, model pool pruning, llm routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BUsROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies model-pool pruning for LLM routing across 27 tasks, 39 models, ten pruning strategies, and six router families. It evaluates oracle utility retention, preservation of the quality–cost frontier, realized routing performance, and cross-task transfer. The main finding is that small, appropriately selected pools can retain much of the full pool’s performance.

Critical strengths
  1. The distinction between utility retention and budget coverage is useful: a pool can preserve performance at one cost preference while losing affordable operating points. Evaluating both exposes a limitation that a single utility score would miss.

  2. The study goes beyond oracle pool quality by retraining routers on the retained pools. This helps establish whether the apparent redundancy survives actual routing.

Critical weaknesses
  1. The “11–18×” result does not establish the claimed relative importance of pool selection and router choice. Appendix A.6 compares variation across pruning configurations after averaging over routers with variation across routers after averaging over pruning configurations. This statistic depends heavily on which configurations are included. The pruning set includes random selection, quality-only selection under cost-sensitive objectives, and multiple overlap variants, whereas random routing is excluded from the router comparison. Figure 3 illustrates the issue: at t=1,k=3t=1, k=3, highest-avg-quality has retention of 0.325, versus roughly 0.95 for utility-aware selection. Such failures can drive the ratio without showing that choosing among reasonable pruning methods matters more than choosing a router. Averaging also hides pool–router interactions. This weakens one of the paper’s headline conclusions.

  2. The transfer results do not separate transferable model capabilities from a shared preference for cheap models. In Table 2, the end-to-end transfer shortfall at k=3k=3 falls from 13.1% at t=0t=0 to 2.0% at t=1t=1 and 1.6% at t=4t=4. This is consistent with source and target tasks increasingly favoring inexpensive models, even without learning useful cross-task complementarity. Yet the transfer evaluation reports only three utility-based strategies, without a cost-only control. Moreover, the shortfalls use utility normalized against full-pool random routing; small gaps in this metric do not directly establish small quality losses at matched expenditure. The paper therefore leaves unclear how much its transfer finding adds beyond the observation that cheap models remain cheap across tasks.

Questions for authors
  1. Does pool selection still dominate router choice when the comparison uses one configuration per cost-aware pruning family and reports router variation within each selected pool?

  2. How do the transferred pools compare with cheapest-first selection on the target tasks, and does the small transfer gap persist when comparing achieved quality at matched mean cost?

Reviewer_q4mYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper evaluates model pool pruning for routing. It finds that a carefully selected small pool can largely preserve routing performance of a larger pool.

Critical strengths
  • Good coverage of models, routers, and pruning strategies.
Critical weaknesses
  • My main concern is that most of the paper's findings are either already established by prior work or largely expected from the experimental setup. Although I appreciate seeing more empirical evidence, I struggle to identify sufficiently interesting insights. I will list a few of the paper's main claims and my assessment of them:

    • Small but carefully selected model pools can preserve routing performance: This has already been demonstrated by prior work, such as LLMRouterBench, Who Routes the Router, and LOCUS. Kudos to the authors for discussing these works, though.
    • Pruning affects the quality–cost trade-off: Prior work such as Select-then-Route and Who Routes the Router has already explored the relationship between model pool selection and the quality–cost trade-off. Moreover, this finding is largely expected: removing a cheap but sufficiently capable model would naturally affect the quality–cost trade-off.
    • Pool selection matters more than router choice: This claim is somewhat trivial under certain conditions. For example, if we prune all the good models, no router can compensate for their absence. More on this below.
    • Pool selection transfers across tasks: This seems potentially novel, but I have some concerns about the experimental setup, as discussed below.
  • Some experimental designs are too rough to support the conclusions. Two examples:

    • Figure 9: The authors compare performance variation across all pruning methods, including weak baselines, against variation across non-random routers. I don't find this comparison particularly meaningful, as weak pruning methods can naturally inflate the variation.
    • Cross-task transfer: The model pools are selected using up to 26 of the 27 tasks, while the routers are specifically trained on the remaining 27th task. I feel this is a relatively forgiving setup, where reasonably good transfer performance is largely expected, especially if a few generally strong models dominate across tasks.

I don't think there is anything wrong with providing more empirical evidence to support common knowledge in the field. One could even argue that this is part of the purpose of benchmarking: to put numbers on observations that were previously only intuitive. However, several prior work have already provided such evidence, and I struggle to identify what new insights this paper offers. If the authors could provide a concrete comparison with prior work on model pool pruning and draw out some interesting, previously unknown insights, I could see this becoming a decent accept. As it stands, however, the work feels rather routine and low on the informativeness axis.

Questions for authors

None

Reviewer_msvjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines how pruning the model pool affects LLM routing, comparing ten strategies across 27 tasks, 39 models, and six router families. It evaluates utility retention, budget coverage, and realized routing performance separately, and tests whether pool selection transfers across tasks.

Critical strengths

S1. Router performance depends on which models are available, yet pool construction often receives less attention than router design. Studying this choice directly makes the paper relevant to deployment.

S2. Table 1 shows why utility retention and budget coverage need separate measures: quality-based pruning can remove much of the feasible budget range. The budget-gap metric captures this limitation.

S3. The comparison covers a broad range of pruning strategies and routers across five data partitions. Pruning uses only training data, and the transfer experiments exclude source tasks that share queries with the target.

Critical weaknesses

W1. The claimed 11 to 18× difference depends on which configurations enter the comparison. Appendix A.6 compares variation across pruning configurations with variation across routers, averaging over the other factor in each case. Some pruning strategies are poorly suited to particular objectives, such as quality-only selection under strong cost penalties, and several strategies contribute multiple configurations. Both choices can increase the measured variation across pruning configurations. I would be cautious about interpreting the ratio as a general measure of how much more pool choice matters than router choice.

W2. The conclusions about cheap models and preserved budget ranges depend directly on the assumed prices. Appendix A.1 assigns prices to all local 7 to 9B models using one reference price pair, scaled by parameter count. The approximation is transparent, but it cannot account for differences in serving efficiency. The paper provides no alternative pricing analysis to show how sensitive the conclusions are to this assumption.

W3. Averaging over tasks, splits, and sometimes routers makes it hard to see where pruning performs poorly. The main claims about preserving performance lack uncertainty estimates, and the paper does not identify tasks with large losses. ArenaHard and its category subsets also receive separate task weights even though they share queries. Per-task results, together with aggregation by dataset family, would make clear how broadly the conclusions apply.

Questions for authors

Q1. How much of the 11 to 18× ratio remains if each pruning strategy contributes one configuration and the comparison includes only strategies suited to the evaluated objective?

Q2. How stable are pruning rankings and removal-cost shares under alternative local-model prices?

Q3. Which tasks suffer the largest losses from pruning? Please include paired uncertainty estimates and results grouped by dataset family.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.