- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies model-pool pruning for LLM routing across 27 tasks, 39 models, ten pruning strategies, and six router families. It evaluates oracle utility retention, preservation of the quality–cost frontier, realized routing performance, and cross-task transfer. The main finding is that small, appropriately selected pools can retain much of the full pool’s performance.
Critical strengths
-
The distinction between utility retention and budget coverage is useful: a pool can preserve performance at one cost preference while losing affordable operating points. Evaluating both exposes a limitation that a single utility score would miss.
-
The study goes beyond oracle pool quality by retraining routers on the retained pools. This helps establish whether the apparent redundancy survives actual routing.
Critical weaknesses
-
The “11–18×” result does not establish the claimed relative importance of pool selection and router choice. Appendix A.6 compares variation across pruning configurations after averaging over routers with variation across routers after averaging over pruning configurations. This statistic depends heavily on which configurations are included. The pruning set includes random selection, quality-only selection under cost-sensitive objectives, and multiple overlap variants, whereas random routing is excluded from the router comparison. Figure 3 illustrates the issue: at , highest-avg-quality has retention of 0.325, versus roughly 0.95 for utility-aware selection. Such failures can drive the ratio without showing that choosing among reasonable pruning methods matters more than choosing a router. Averaging also hides pool–router interactions. This weakens one of the paper’s headline conclusions.
-
The transfer results do not separate transferable model capabilities from a shared preference for cheap models. In Table 2, the end-to-end transfer shortfall at falls from 13.1% at to 2.0% at and 1.6% at . This is consistent with source and target tasks increasingly favoring inexpensive models, even without learning useful cross-task complementarity. Yet the transfer evaluation reports only three utility-based strategies, without a cost-only control. Moreover, the shortfalls use utility normalized against full-pool random routing; small gaps in this metric do not directly establish small quality losses at matched expenditure. The paper therefore leaves unclear how much its transfer finding adds beyond the observation that cheap models remain cheap across tasks.
Questions for authors
-
Does pool selection still dominate router choice when the comparison uses one configuration per cost-aware pruning family and reports router variation within each selected pool?
-
How do the transferred pools compare with cheapest-first selection on the target tasks, and does the small transfer gap persist when comparing achieved quality at matched mean cost?