- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the labeling cost issue in large language model routing. The authors consider a model-onboarding setting in which a new model is introduced into an existing routing system and its performance is learned using as few labeled queries as possible. To this end, the authors propose an adaptive decision-centric query-by-committee (ADC-QBC) method, which combines predictive disagreement sampling from standard query-by-committee (S-QBC) with routing-decision disagreement sampling from decision-centric query-by-committee (DC-QBC), and uses an inverse-time decay schedule to gradually shift the labeling budget from S-QBC to DC-QBC. The proposed DC-QBC uses vote entropy weighted by the utility margin to prioritize decision-relevant queries and aggregates acquisition scores across different cost-sensitivity levels. The authors evaluate the method on the SPROUT benchmark, with GPT-4o mini and Claude 3.5 Sonnet each serving as the newly onboarded model in separate experiments. The experimental results indicate that, compared with S-QBC, ADC-QBC achieves higher routing accuracy in most experimental settings and better label efficiency in some settings.
Critical strengths
-
This paper studies the labeling cost problem of model onboarding in large language model routing, which has practical application value.
-
The authors combine active learning with routing decisions so that the sampling process considers not only predictive uncertainty but also decision-level disagreement from the committee and utility margins when constructing the acquisition criterion. This idea offers an interesting perspective.
Critical weaknesses
-
The experimental comparisons and ablation studies are insufficient, and the performance gains are limited. This paper only uses S-QBC as the active learning baseline and lacks baselines such as random sampling and standalone DC-QBC. It also does not conduct ablation studies on the utility margin weighting or the decay schedule. In addition, as can be seen from Tables 1 and 2, the routing accuracy improvements of ADC-QBC are generally small, and some differences are comparable to the standard deviations across seeds. This makes it difficult to adequately assess the practical advantages of the method and the contribution of each component.
-
The method design lacks sufficient theoretical and experimental support. The authors argue that S-QBC may waste the labeling budget on queries irrelevant to routing decisions, while DC-QBC may suffer from self-confirming bias, and propose corresponding improvements on this basis. However, the paper does not provide targeted theoretical analysis of these claimed failure modes and their mitigation, nor targeted experiments to verify the existence of these issues and the effectiveness of the proposed method. Therefore, the design rationale from S-QBC to DC-QBC and then to ADC-QBC remains insufficiently justified.
Questions for authors
-
Could the authors add comparisons with random sampling and standalone DC-QBC, as well as ablation studies on the utility-margin weighting and the decay schedule, to clarify the actual contribution of each component?
-
Is there any theoretical analysis or empirical evidence showing that S-QBC indeed suffers from the claimed inefficient allocation of the labeling budget, that DC-QBC indeed exhibits self-confirming bias, and that ADC-QBC effectively mitigates these issues?
2 comments
Thanks for your response. I appreciate the authors' efforts in addressing the concerns raised in my review.
W1: I appreciate the additional experiments and ablations. However, the results (Table 4) show that pure DC-QBC outperforms ADC-QBC in mean accuracy across all five label budgets. Removing the utility-margin weighting produces no significant difference either. These findings cast doubt on the necessity of the additional components introduced in ADC-QBC.
W2: The new analysis of acquisition-induced bias on RouterBench is informative. However, the absence of self-confirming bias on SPROUT weakens the motivation for adaptive exploration, and its effectiveness isn't sufficiently supported by the current results.
I also share Reviewer EhnH's concern: the improvements over random sampling at matched inference cost are limited and inconsistent across labeling budgets. This suggests that the label-efficiency claims should be more carefully qualified.
A minor point: The layout of the revised appendix seems broken. I suggest the authors to fix it in future revisoins.
Overall, I acknowledge the authors' substantial efforts in revising the paper. However, my concerns regarding the justification and empirical benefits of the full ADC-QBC design remain unresolved. Therefore, I'll maintain my score.
We thank the reviewer for the feedback. Table numbers refer to the revised PDF.
W1 / Q1. Baselines, ablations, gains vs. variance. We added random sampling, pure DC-QBC, vote entropy without the margin weight, and a fixed mixture in place of the decaying schedule (Tables 1–4; details in our reply to Reviewer EhnH).
On variance: the standard deviations in the original tables include variation that both methods share, since they start from the same random batch in each seed. The paired 95% intervals remove that shared part. On SPROUT, ADC-QBC is ahead of random sampling at every budget, significantly at 1,000, 4,000 and 10,000 labels. On RouterBench, it is ahead of S-QBC at every budget, significantly at 1,000, 2,000 and 12,000 labels.
W2 / Q2. Evidence for the failure modes. Your question led us to test each claim directly: