- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on how long it takes to recover the cost of collecting data for training a cost-effective model router. The author only selects some problems and model combinations to obtain feedback, then estimates the model's ability by leveraging the commonalities among similar problems, and fills in the differences of each problem itself. The experiment combines the upfront supervision cost and the subsequent invocation cost to compare when different methods break even.
Critical strengths
S1. Taking into account the cost of collecting supervision data, this perspective has practical significance. Even if a router is more cost-effective after deployment, it does not necessarily mean that the upfront investment is worthwhile.
S2. Grouping by problems and supplementing the individual differences of each problem, this design is relatively clear. The corresponding ablation experiments also show that only looking at the average ability within the group is not enough.
S3. The author reports a situation where the option that breaks even earlier does not save the most in the long term. This allows one to see the trade-offs between different goals rather than just looking at a cost ranking.
Critical weaknesses
W1. Figure 1 already shows that EmbedLLM and kNN can retain most of the answer quality with less supervision. However, in the main experiment, these two baselines use full supervision, and comparisons of the costs and break-even results for their low-supervision versions are missing. Therefore, it is not clear how much of the break-even advantage in this paper can be retained.
W2. Currently, the cheapest solution that meets the target quality is selected from the test results. This can demonstrate the cost curve obtained after the fact, but it cannot indicate that choosing a route plan in advance and deploying it can achieve the same quality and break-even speed.
Questions for authors
Q1. Can you report the call costs and break-even results for the low-supervision versions of EmbedLLM and kNN in Figure 1 and compare them with SAVEROUTER under the same quality target?
Q2. Can you select the routing scheme on the validation set first, then keep it fixed and evaluate its quality and break-even point, including the additional feedback acquisition cost?
2 comments
Response to Reviewer FEoE — Part (2/2)
R2: Offline Frontier Evaluation and Validation-Based Policy Selection
We first clarify that our original evaluation does not use test data for router training or deployment-time policy selection. All candidate routing policies are constructed using training data only, while test outcomes are used solely to evaluate these candidates and characterize the offline quality–cost frontier. This protocol aims to characterize the achievable quality–cost trade-offs across different operating points and is widely adopted in LLM routing [1–3].
We agree that offline frontier characterization and deployment-oriented policy selection serve different purposes. While the former provides an overview of the available operating points, the latter evaluates whether a policy selected on validation data can maintain its quality and cost efficiency on unseen test data. To provide this complementary evaluation, we conduct two additional experiments on LLMRouterBench using a 16%/4%/80% train/validation/test split, with the validation set separated from the original training partition and the test set unchanged. Each method selects its routing policy exclusively on the validation set and evaluates the fixed policy on the independent test set. The additional feedback-acquisition cost incurred during validation is included in the upfront supervision cost .
(1) Minimum-cost policy meeting the best single-model quality target.
We first select the lowest-cost routing policy that meets the best single-model quality target on the validation set. The selected policy is then fixed and evaluated on the independent test set.
| Method | ↑ | CR ↓ | SA-BEP ↓ |
|---|---|---|---|
| SAVERouter | 0.5947 | 0.3505 | 6.6K |
| EmbedLLM | 0.5641 | ∞ | ∞ |
| kNN | 0.5693 | ∞ | ∞ |
| OmniRouter | 0.5606 | ∞ | ∞ |
| RMSoftmax | 0.5679 | ∞ | ∞ |
| TRouter | 0.5863 | ∞ | ∞ |
| UniRoute | 0.5848 | ∞ | ∞ |
| InferenceDyn. | 0.4257 | ∞ | ∞ |
| WISERouter | 0.4642 | ∞ | ∞ |
| BaRP | 0.5647 | ∞ | ∞ |
| SemiRouter | 0.5787 | ∞ | ∞ |
SAVERouter is the only method that maintains the target quality on the test set, achieving a CR of 0.3505 and breaking even after 6.6K queries. In contrast, the validation-selected policies of all competing methods fail to meet the quality target on the test set. These results demonstrate that SAVERouter can maintain both quality and economic efficiency when its routing policy is selected before deployment.
(2) Maximum-quality policy selected on validation data.
We further consider an alternative setting where each method selects its highest-quality policy on the validation set and evaluates it without modification on the independent test set.
| Method | ↑ | CR ↓ | SA-BEP ↓ |
|---|---|---|---|
| SAVERouter | 0.6281 | 0.4447 | 7.8K |
| kNN | 0.6091 | 0.5206 | 15.5K |
| EmbedLLM | 0.5867 | ∞ | ∞ |
| OmniRouter | 0.6040 | 0.4565 | 13.7K |
| RMSoftmax | 0.6092 | 0.5541 | 16.7K |
| TRouter | 0.5863 | ∞ | ∞ |
| UniRoute | 0.6000 | 0.5221 | 15.6K |
| InferenceDyn. | 0.5783 | ∞ | ∞ |
| WISERouter | 0.5771 | ∞ | ∞ |
| BaRP | 0.6064 | 0.4911 | 14.6K |
| SemiRouter | 0.5806 | ∞ | ∞ |
SAVERouter achieves the highest test quality (0.6281), lowest CR (0.4447), and earliest SA-BEP (7.8K) among all methods. These results demonstrate that SAVERouter maintains its advantages in both routing quality and economic efficiency when policies are selected on validation data and evaluated without modification on an independent test set.
Together, these two experiments complement our original offline frontier evaluation and show that SAVERouter's quality and economic advantages persist under validation-selected, fixed-policy evaluation, with additional validation feedback costs included.
References
[1] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. TMLR 2024.
[2] RouteLLM: Learning to Route LLMs from Preference Data. ICLR 2025.
[3] MixLLM: Dynamic Routing in Mixed Large Language Models. NAACL 2025.
Response to Reviewer FEoE — Part (1/2)
We thank the reviewer for recognizing the practical significance of our work and for the constructive suggestions. We conduct additional experiments on budget-matched supervision and validation-selected routing policies to address both concerns.
R1: Comparison under Matched Supervision Expenditure
We thank the reviewer for this constructive suggestion. We agree that the economic advantage of SAVERouter should also be evaluated against low-supervision versions of conventional routers.
To address this, we train EmbedLLM, kNN, and UniRoute using exactly the same sparse query–model feedback acquired by SAVERouter (K = 4). This ensures identical supervision acquisition costs and enables a direct comparison of routing quality, serving cost, and break-even points under the same quality target.
The results yield three key observations:
Informative sparse feedback. Some conventional routers achieve higher peak quality with SAVERouter's sparse observations than with full supervision, demonstrating the usefulness of the acquired feedback across routing architectures.
Effective learning from sparse feedback. Given identical acquired observations, SAVERouter consistently achieves better routing quality and cost efficiency, demonstrating that its hierarchical capability estimation and query-level refinement make more effective use of limited supervision.
Earlier break-even under matched expenditure. Despite identical acquisition costs, SAVERouter achieves earlier break-even points than the compared baselines, confirming its economic advantage under matched supervision.
Together, these results demonstrate the complementary benefits of informative feedback acquisition and effective learning from sparse supervision.