acceptodds
Under review as a conference paper at ICLR 2027

Decision Centric Active Learning for LLM Routing

Abstract

LLM routing selects, for each query, the model from an available pool that best trades off answer quality and cost. Predictive routers estimate these criteria for every model and pick the best one, which presumes training data in which queries have been evaluated on all models—data that is costly to collect and must be collected again whenever a new model is added. We study this model-onboarding setting: a router over a set of incumbent models is already trained, and a new LLM must be integrated using as few labeled queries as possible. Classical active learning reduces labeling cost by querying where a predictor is most uncertain, but it ignores the routing decision and therefore spends much of its budget on queries the new model would never win. We propose Adaptive Decision-Centric Query-by-Committee (ADC-QBC), which mixes, within every batch, standard query-by-committee samples that reduce global uncertainty about the new model with decision-centric samples on which the committee disagrees about whether the new model should replace the best incumbent, shifting the budget from the former to the latter through a decaying schedule. On the SPROUT benchmark, onboarding GPT-4o mini and Claude 3.5 Sonnet, ADC-QBC attains higher routing accuracy than standard query-by-committee in 54 of 72 combinations of label budget and cost sensitivity; when onboarding GPT-4o mini, with 4,000 labels it outperforms standard query-by-committee with 8,000 at intermediate costs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, Active Learning, Query-by-Committee, Decision-Centric Learning, Label Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bcbcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the labeling cost issue in large language model routing. The authors consider a model-onboarding setting in which a new model is introduced into an existing routing system and its performance is learned using as few labeled queries as possible. To this end, the authors propose an adaptive decision-centric query-by-committee (ADC-QBC) method, which combines predictive disagreement sampling from standard query-by-committee (S-QBC) with routing-decision disagreement sampling from decision-centric query-by-committee (DC-QBC), and uses an inverse-time decay schedule to gradually shift the labeling budget from S-QBC to DC-QBC. The proposed DC-QBC uses vote entropy weighted by the utility margin to prioritize decision-relevant queries and aggregates acquisition scores across different cost-sensitivity levels. The authors evaluate the method on the SPROUT benchmark, with GPT-4o mini and Claude 3.5 Sonnet each serving as the newly onboarded model in separate experiments. The experimental results indicate that, compared with S-QBC, ADC-QBC achieves higher routing accuracy in most experimental settings and better label efficiency in some settings.

Critical strengths
  1. This paper studies the labeling cost problem of model onboarding in large language model routing, which has practical application value.

  2. The authors combine active learning with routing decisions so that the sampling process considers not only predictive uncertainty but also decision-level disagreement from the committee and utility margins when constructing the acquisition criterion. This idea offers an interesting perspective.

Critical weaknesses
  1. The experimental comparisons and ablation studies are insufficient, and the performance gains are limited. This paper only uses S-QBC as the active learning baseline and lacks baselines such as random sampling and standalone DC-QBC. It also does not conduct ablation studies on the utility margin weighting or the decay schedule. In addition, as can be seen from Tables 1 and 2, the routing accuracy improvements of ADC-QBC are generally small, and some differences are comparable to the standard deviations across seeds. This makes it difficult to adequately assess the practical advantages of the method and the contribution of each component.

  2. The method design lacks sufficient theoretical and experimental support. The authors argue that S-QBC may waste the labeling budget on queries irrelevant to routing decisions, while DC-QBC may suffer from self-confirming bias, and propose corresponding improvements on this basis. However, the paper does not provide targeted theoretical analysis of these claimed failure modes and their mitigation, nor targeted experiments to verify the existence of these issues and the effectiveness of the proposed method. Therefore, the design rationale from S-QBC to DC-QBC and then to ADC-QBC remains insufficiently justified.

Questions for authors
  1. Could the authors add comparisons with random sampling and standalone DC-QBC, as well as ablation studies on the utility-margin weighting and the decay schedule, to clarify the actual contribution of each component?

  2. Is there any theoretical analysis or empirical evidence showing that S-QBC indeed suffers from the claimed inefficient allocation of the labeling budget, that DC-QBC indeed exhibits self-confirming bias, and that ADC-QBC effectively mitigates these issues?

2 comments

We thank the reviewer for the feedback. Table numbers refer to the revised PDF.

W1 / Q1. Baselines, ablations, gains vs. variance. We added random sampling, pure DC-QBC, vote entropy without the margin weight, and a fixed mixture in place of the decaying schedule (Tables 1–4; details in our reply to Reviewer EhnH).

On variance: the standard deviations in the original tables include variation that both methods share, since they start from the same random batch in each seed. The paired 95% intervals remove that shared part. On SPROUT, ADC-QBC is ahead of random sampling at every budget, significantly at 1,000, 4,000 and 10,000 labels. On RouterBench, it is ahead of S-QBC at every budget, significantly at 1,000, 2,000 and 12,000 labels.

W2 / Q2. Evidence for the failure modes. Your question led us to test each claim directly:

  • S-QBC spends labels on decisions that are already clear. Its score ignores the incumbents and λ by construction (Eq. 4). In outcome terms, ADC-QBC is ahead of S-QBC at every budget on both benchmarks, and S-QBC needs 5,000 labels to match ADC-QBC at 1,000 on RouterBench.
  • Self-confirming bias of pure DC-QBC. We tested it with pure DC-QBC (Table 4). After the initial random batch of 250 labels, it does not arise on SPROUT: pure DC-QBC is on par with or ahead of ADC-QBC. We revised Section 3.3 to present it as a potential failure mode, and we no longer claim it as observed.
  • A different, measured failure mode (Table 3, Section 4.4). Decision-centric labels are not representative of the pool. On RouterBench, with 1,000–4,000 labels, they lead the regressor to underestimate Yi-34B by 3.6–5.2 points, so the router under-uses it and spends more. This explains why random sampling leads at matched cost early on. The bias vanishes by 12,000 labels, where ADC-QBC leads.

Thanks for your response. I appreciate the authors' efforts in addressing the concerns raised in my review.

W1: I appreciate the additional experiments and ablations. However, the results (Table 4) show that pure DC-QBC outperforms ADC-QBC in mean accuracy across all five label budgets. Removing the utility-margin weighting produces no significant difference either. These findings cast doubt on the necessity of the additional components introduced in ADC-QBC.

W2: The new analysis of acquisition-induced bias on RouterBench is informative. However, the absence of self-confirming bias on SPROUT weakens the motivation for adaptive exploration, and its effectiveness isn't sufficiently supported by the current results.

I also share Reviewer EhnH's concern: the improvements over random sampling at matched inference cost are limited and inconsistent across labeling budgets. This suggests that the label-efficiency claims should be more carefully qualified.

A minor point: The layout of the revised appendix seems broken. I suggest the authors to fix it in future revisoins.

Overall, I acknowledge the authors' substantial efforts in revising the paper. However, my concerns regarding the justification and empirical benefits of the full ADC-QBC design remain unresolved. Therefore, I'll maintain my score.

Reviewer_EhnHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to add a new LLM to an existing router using a limited labeling budget. It proposes ADC-QBC, which combines prediction uncertainty with committee disagreement about whether the new model should replace the best incumbent, gradually shifting toward decision-focused selection. Experiments on SPROUT show improvements over standard query-by-committee when onboarding GPT-4o mini and Claude 3.5 Sonnet.

Critical strengths
  1. The paper addresses a practical problem: selecting evaluation queries that help integrate a new model into an existing router. Conditioning acquisition on competition with incumbent models provides a clear connection between label selection and downstream routing decisions.

  2. The comparison with standard QBC uses the same committee, predictor, and batch size, helping isolate the effect of acquisition. Results across five seeds, several label budgets, and accuracy–cost curves provide useful initial evidence of improved label efficiency.

Critical weaknesses
  1. The evaluation omits essential baselines and component ablations. There is no comparison with random sampling, pure DC-QBC, a fixed exploration mixture, or vote entropy without margin weighting. Consequently, the experiments do not establish whether the proposed schedule and weighting are necessary, or whether a simpler acquisition strategy would perform equally well.

  2. The utility-margin weighting is insufficiently justified. At a given cost sensitivity, Equation 10 assigns zero score when the committee mean equals the incumbent utility, even if committee members predict substantially different outcomes on either side of that value. Such a query can involve consequential decision uncertainty. The paper does not examine whether this weighting suppresses useful queries or compare it with a criterion based on expected decision regret.

  3. The empirical scope is narrow, covering one benchmark, two onboarded models, and one predictor configuration. Several tabulated improvements are small relative to the reported variation across seeds, and the accuracy–cost curves lack uncertainty bands. Counting wins across 72 related budget–cost settings does not establish how reliable or general the improvements are.

Questions for authors
  1. How does ADC-QBC compare with random sampling, pure DC-QBC, and a fixed mixture of standard and decision-centric QBC under the same labeling budgets?

  2. What happens when the margin weighting is removed or replaced with committee-based expected regret, particularly when the committee mean is close to the incumbent utility?

  3. Do the improvements persist for other onboarded models and less accurate incumbent predictors? Can you report paired confidence intervals for accuracy differences at matched inference costs?

2 comments

We thank the reviewer for the precise suggestions; we ran most of them. Table numbers refer to the revised PDF.

W1 / Q1. Baselines and ablations. We added random sampling (Tables 1–2) and the three ablations (Table 4, SPROUT), all under the same committee, regressor, budgets and initial batches. The fixed mixture's constant exploration share equals the average share of our decaying schedule, so it spends the same budget on exploration and differs only in timing.

At matched cost:

  • The margin weight is neutral: vote entropy alone is within 0.13 pp of ADC-QBC at every budget, and no interval excludes zero.
  • The decaying schedule matches the fixed mixture up to 8,000 labels and is better at 10,000 (+0.17 pp, interval excludes zero).
  • Pure DC-QBC is on par with ADC-QBC up to 4,000 labels and ahead at 8,000 (+0.20 pp).

So with an initial random batch of 250 labels, SPROUT needs no extra exploration, and the decision-centric score is the main source of the gains. On its own, it exceeds random sampling's mean accuracy at matched cost by 0.24–0.49 pp from 4,000 labels on. We revised Sections 3.3 and 4.3 accordingly.

W2 / Q2. Margin weighting.

  1. The zero affects single cost levels, not queries. The score is averaged over the cost grid Λ (Eq. 11). Each member's margin varies continuously with λ, so a committee split at one λ stays split at neighboring values. There the mean margin is non-zero unless tnewt_{\text{new}} and the best incumbent also have the same cost on that query. Such a query therefore keeps a positive aggregate score (new paragraph in Section 3.3).
  2. At a fixed λ, we agree. A committee split around the incumbent, with mean margin near 0, carries decision uncertainty that our weighting discounts. The committee-based expected regret you suggest, Ek[(Δk)+]−(Δˉ)+\mathbb{E}_k[(\Delta_k)^+] - (\bar\Delta)^+, captures it, and the same committee can compute it at no extra cost. We could not run it within the discussion period and will add it as an ablation in the final version.
  3. Empirically, removing the weight makes no significant difference at matched cost (Table 4). At fixed λ, entropy only scores slightly higher, but the difference shrinks at matched cost, so part of it comes from routing to more expensive models.

W3 / Q3. Scope and statistics.

  • Second benchmark and more onboarded models: RouterBench, onboarding Yi-34B (Table 2), and Claude 3.5 Sonnet rerun with all baselines (Appendix G).
  • Less accurate incumbent predictors: RouterBench is a natural instance. The cross-validated R2R^2 of its incumbent predictors is 0.03–0.12, against 0.12–0.28 on SPROUT, and ADC-QBC still leads S-QBC at every budget. A controlled degradation on SPROUT will follow in the final version.
  • Uncertainty bands: every curve now shows ± one std across seeds (Figure 1, Appendix C).
  • Paired CIs at matched inference cost: Tables 1b and 2b (General response 2/3 and 3/3; all intervals in Appendix D). They replace the win counts over the 72 settings.
    • On SPROUT, ADC-QBC is ahead of S-QBC at every budget and on par with or ahead of random sampling.
    • On RouterBench, random sampling is ahead at small budgets. We traced this to a bias (Table 3, Section 4.4). Decision-centric labels concentrate on queries where Yi-34B's routing is contested, which appear harder than average, so with 1,000–2,000 labels the regressor underestimates Yi-34B by about 5 points (random sampling: 0.5). The router therefore sends Yi-34B too few queries (42% vs. 55% at λ = 1) and pays for more expensive models. The bias vanishes as labels accumulate (0.2 points at 12,000), where ADC-QBC leads at matched cost. Correcting it with the randomly drawn labels is a natural extension.

Thank you for the detailed response and the many new experiments.

The new baselines, ablations, second benchmark and confidence intervals address my questions, but they don’t change my view much. In Table 4, pure DC-QBC does as well as or better than ADC-QBC at every budget. The margin weight makes no clear difference, and the decaying schedule only helps at 10,000 labels. So I still don’t see a benefit over the simpler method. At matched cost (Table 1b), ADC-QBC is clearly better than random sampling on SPROUT only at 10,000 labels. On RoutrBench (Table 2b), random sampling does bettter at 1,000 and 2,000 labels. The claim that random sampling need 7,000 labels to match ADC-QBC at 2,000 comes from accuracy averaged over λ (Table 8). At matched cost, random sampling with 2,000 labels gets 83.84, which is already the same as ADC-QBC at 83.83. For the margin weight, I appreciate that you acknowledge the issue at a fixed λ, but the expected regret comparrison is still missing.

Reviewer_63GnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trying to solve LLM routing problem in a query-intensive setting, which is selecting the model from an available pool (with limited data) that best trades off answer quality and cost. Based on traditional S-QBC, this paper further propose DC-QBC and ADC-QBC, which improves compared to traditional S-QBC.

Critical strengths
  1. The problem setting is important, especially considering current LLMs are huge and costly in performance estimation.
  2. The writing is easy to follow, and the method proposed is reasonable considering the setting.
Critical weaknesses
  1. The main concern is, it just compares with S-QBC, which is a pretty old method. It should compare with current SOTA baselines in current fields, also an ablation study (in appendix maybe) would help others better understand the contribution of each part of the method.
  2. Figure 1 is interesting, however, i think the gain is marginal and sometimes not statistically significant based on the quantitative results in the tables.
  3. I think doing experiment on GPT-4O-mini and Claude 3.5 Sonnet is fine, but it would be great to see more results on different models, such as qwen and llama families, to assess the usage of the model better.
Questions for authors
  1. Is there a specific reason that you didn't compare your method with other SOTA method in LLM routing?
2 comments

We thank the reviewer for the feedback. Table numbers refer to the revised PDF.

W1 / Q1. Comparison with SOTA routing methods. Our method is not a router, and we realize the paper did not make this clear enough. We study model onboarding: a router over incumbent models exists, and a new model must be added with as few labeled queries as possible. Routing methods such as RouteLLM or CARROT assume labels for every model, including the new one. ADC-QBC decides which queries to label, and any of these routers can then be trained on the result. The two are complementary, so the right comparison is against other label-acquisition strategies. The revised introduction says this upfront (General response, "Setting").

Within that class, we added random sampling, the standard active-learning reference, and an ablation of each component (Tables 1–4). Recent deep active-learning methods such as BADGE need gradients of a neural predictor and do not apply to our tree-based regressors. Core-Set works directly on the query embeddings, and we will add it in the final version.

W2. Marginal gains, significance. We now report paired 95% confidence intervals across seeds (Tables 1–2, Appendix D). Pairing matters here: all methods share the initial random batch of each seed, so part of the seed-to-seed variation in the original tables is shared by both methods and cancels in their difference. On SPROUT, ADC-QBC is ahead of random sampling at every budget, significantly at 1,000, 4,000 and 10,000 labels.

The absolute differences look small because the whole range is narrow. On SPROUT, the full-information router is only 0.88 pp above a router trained on 1,000 random labels, and ADC-QBC closes 38% of that gap at 1,000 labels. Measured in labels, the gain is large: random sampling needs 7,000 labels and S-QBC 5,000 to match ADC-QBC at 2,000 (Appendix E).

W3. Other model families. We agree this matters. The revision onboards models from three families: GPT-4o mini (OpenAI) and Claude 3.5 Sonnet (Anthropic) on SPROUT, the cheapest and the most expensive models of our pool, and the open-weight Yi-34B (01.AI) on RouterBench, with a different pool and task distribution (Table 2, Appendix G). SPROUT contains no Qwen models.

Reviewer_63GnOpenReview ↗

Concerns solved

I thank the authors for their updates. These generally solved my questions and concerns, thus, i raise my score from 2 to 3.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3): setting, new baselines and ablations, second benchmark, statistics

We thank the reviewers. Below, R1, R2 and R3 refer to Reviewers 63Gn, EhnH and bcbc. Changes are marked in blue in the revised PDF, and table numbers refer to it.

Setting. ADC-QBC is not a router. A router over incumbent models is already trained, and a new model tnewt_{\text{new}} must be added to it. Any predictive router (e.g., RouteLLM, CARROT) needs per-query quality labels for tnewt_{\text{new}}, and each label costs an LLM call plus grading. ADC-QBC decides which queries to label; the router is then trained on those labels. Its competitors are therefore label-acquisition strategies, not routing methods. The revised introduction states this upfront.

What we added. All methods share the committee, regressor, batch size and each seed's initial random batch; 5 seeds, compared seed by seed.

  1. Random sampling as a baseline (R1, R2, R3).
  2. Two versions of ADC-QBC: exploration by S-QBC, as in the paper, or by random sampling.
  3. A second benchmark, RouterBench (R1, R2): five models on the pool's accuracy–cost frontier, seven task families, onboarding the open-weight Yi-34B.
  4. Ablations (R2, R3): pure DC-QBC, vote entropy without the margin weight, and a fixed mixture with the same total exploration as our schedule.
  5. Statistics (R1, R2, R3): paired 95% confidence intervals in place of win counts, ± std bands on all curves, and accuracy at matched inference cost (R2): each router's accuracy–cost curve read at the cost the full-information router incurs at each λ ∈ [0, 1].
  6. Analysis of where decision-centric labels help and where they do not (R3, Section 4.4).
  7. Claude 3.5 Sonnet rerun with all baselines (Appendix G).

Results.

  • SPROUT, GPT-4o mini (Table 1). ADC-QBC (S-QBC exploration, as in the paper) has a higher mean accuracy than both baselines at every budget from 1,000 to 10,000 labels; against random sampling the paired interval excludes zero at 1,000, 4,000 and 10,000, and every interval that excludes zero favors ADC-QBC. At matched cost, it is ahead of S-QBC at every budget and on par with or ahead of random sampling (significantly at 10,000). Random sampling needs 7,000 labels to match ADC-QBC at 2,000.
  • RouterBench, Yi-34B (Table 2). A version of ADC-QBC has the highest mean accuracy at every budget, both versions are ahead of S-QBC at every budget, and random sampling needs 12,000 labels to match ADC-QBC at 4,000. At matched cost, random sampling beats ADC-QBC (S-QBC exploration) at 1,000–2,000 labels (by up to 0.85 pp), the two are on par from 4,000 to 10,000, and ADC-QBC is ahead at 12,000 (+0.23), where it exceeds the full-information router. Random sampling is a known strong baseline at low budgets (Lowell et al., 2019; Hacohen et al., 2022).
  • Why random sampling leads early on RouterBench (Table 3). Decision-centric labels concentrate on contested queries, which appear harder for Yi-34B, so with 1,000–4,000 labels the regressor underestimates it by 3.6–5.2 points (random sampling: at most 0.6). The router under-uses the cheap Yi-34B (42% vs. 55% of queries at λ = 1) and pays for more expensive models. The bias vanishes by 12,000 labels, and so does the gap.
  • Sonnet (Appendix G). Sonnet is worth its cost only when cost matters little, so all methods stay within 0.4 pp. At matched cost, ADC-QBC with random exploration is significantly ahead of random sampling at 8,000 and 10,000 labels; with S-QBC exploration, no difference is significant.
  • Ablations (Table 4). At matched cost, the margin weight is neutral, the decaying schedule matches the fixed mixture up to 8,000 labels and beats it at 10,000, and pure DC-QBC is on par with or ahead of ADC-QBC: after the initial random batch, the decision-centric score is the main source of the gains.

Also changed. The full-information reference now uses the same regressor configuration as the evaluated routers (previously a larger multi-output regressor), so labeling the entire pool reproduces it exactly. The self-confirming bias is now presented as a potential failure mode (Section 3.3). In the final version, we will add Core-Set, a committee-based expected-regret criterion and a controlled degradation of the incumbent predictors (R2), and ablations on RouterBench.

AuthorsOpenReview ↗

General response (2/3): SPROUT, onboarding GPT-4o mini

Rows: mean ± std over 5 seeds (best mean in bold). Italic rows: paired difference in pp, mean [95% CI] (seeds where the first method is ahead); bold = interval excludes 0.

Table 1a. Routed answer accuracy (%) averaged over λ ∈ [0, 1].

Methodn = 1,000n = 2,000n = 4,000n = 8,000n = 10,000
Random83.53 ± 0.1983.88 ± 0.2383.94 ± 0.1484.13 ± 0.1984.04 ± 0.11
S-QBC83.60 ± 0.1983.86 ± 0.1583.96 ± 0.1884.27 ± 0.0784.17 ± 0.15
ADC-QBC (ours), S-QBC exploration83.86 ± 0.1384.05 ± 0.2584.17 ± 0.1084.29 ± 0.1084.36 ± 0.12
ADC-QBC (ours), random exploration83.75 ± 0.2384.11 ± 0.1384.23 ± 0.1784.25 ± 0.1484.36 ± 0.11
ADC-QBC (S-QBC expl.) − Random+0.33 [+0.12, +0.53] (5/5)+0.16 [-0.16, +0.49] (4/5)+0.23 [+0.03, +0.42] (5/5)+0.16 [-0.10, +0.41] (4/5)+0.32 [+0.08, +0.56] (5/5)
ADC-QBC (S-QBC expl.) − S-QBC+0.26 [+0.04, +0.47] (4/5)+0.19 [-0.01, +0.38] (5/5)+0.21 [-0.04, +0.47] (4/5)+0.01 [-0.12, +0.14] (2/5)+0.19 [-0.07, +0.45] (4/5)
ADC-QBC (random expl.) − Random+0.22 [-0.19, +0.63] (4/5)+0.23 [-0.14, +0.60] (3/5)+0.29 [-0.01, +0.59] (4/5)+0.12 [-0.17, +0.41] (2/5)+0.32 [+0.21, +0.43] (5/5)
ADC-QBC (random expl.) − S-QBC+0.15 [-0.18, +0.48] (4/5)+0.25 [-0.08, +0.58] (4/5)+0.28 [+0.13, +0.42] (5/5)-0.02 [-0.24, +0.19] (3/5)+0.19 [+0.00, +0.37] (5/5)
Full information (all labels)84.4184.4184.4184.4184.41

Table 1b. Accuracy (%) at matched inference cost.

Methodn = 1,000n = 2,000n = 4,000n = 8,000n = 10,000
Random83.50 ± 0.1683.84 ± 0.2183.91 ± 0.1384.11 ± 0.1884.03 ± 0.10
S-QBC83.35 ± 0.2083.67 ± 0.1283.78 ± 0.1784.13 ± 0.0984.00 ± 0.14
ADC-QBC (ours), S-QBC exploration83.56 ± 0.1283.83 ± 0.2684.08 ± 0.0784.28 ± 0.0984.38 ± 0.11
ADC-QBC (ours), random exploration83.60 ± 0.2283.97 ± 0.1284.10 ± 0.1784.23 ± 0.1384.37 ± 0.11
ADC-QBC (S-QBC expl.) − Random+0.06 [-0.13, +0.26] (2/5)-0.01 [-0.30, +0.27] (4/5)+0.16 [-0.04, +0.37] (4/5)+0.17 [-0.09, +0.44] (4/5)+0.35 [+0.12, +0.59] (5/5)
ADC-QBC (S-QBC expl.) − S-QBC+0.20 [-0.07, +0.47] (4/5)+0.16 [-0.12, +0.44] (4/5)
AuthorsOpenReview ↗

General response (3/3): RouterBench, onboarding Yi-34B

Same format as Tables 1a–1b in General response (2/3): mean ± std over 5 seeds (best mean in bold); italic rows give paired differences in pp, mean [95% CI] (seeds where the first method is ahead), with bold intervals excluding 0.

Table 2a. Routed answer accuracy (%) averaged over λ ∈ [0, 1].

Methodn = 1,000n = 2,000n = 4,000n = 8,000n = 10,000n = 12,000
Random77.04 ± 0.2477.27 ± 0.1877.28 ± 0.1577.50 ± 0.1577.51 ± 0.2077.64 ± 0.12
S-QBC76.69 ± 0.2676.77 ± 0.1077.01 ± 0.3177.41 ± 0.2977.57 ± 0.1277.54 ± 0.16
ADC-QBC (ours), S-QBC exploration77.02 ± 0.2377.21 ± 0.1577.52 ± 0.1777.64 ± 0.2077.70 ± 0.1577.80 ± 0.13
ADC-QBC (ours), random exploration77.20 ± 0.3477.39 ± 0.1777.58 ± 0.0977.64 ± 0.1277.66 ± 0.1077.77 ± 0.14
ADC-QBC (S-QBC expl.) − Random-0.01 [-0.46, +0.43] (3/5)-0.06 [-0.37, +0.24] (1/5)+0.24 [-0.01, +0.49] (4/5)+0.14 [-0.21, +0.50] (3/5)+0.18 [-0.17, +0.53] (3/5)+0.15 [-0.03, +0.34] (4/5)
ADC-QBC (S-QBC expl.) − S-QBC+0.33 [+0.08, +0.57] (5/5)+0.44 [+0.22, +0.65] (5/5)+0.51 [-0.04, +1.05] (4/5)+0.23 [-0.30, +0.75] (3/5)+0.13 [-0.12, +0.37] (3/5)+0.26 [+0.09, +0.42] (5/5)
ADC-QBC (random expl.) − Random+0.16 [-0.51, +0.84] (4/5)+0.11 [-0.29, +0.52] (3/5)+0.30 [+0.04, +0.56] (5/5)+0.15 [-0.15, +0.44] (4/5)+0.14 [-0.22, +0.51] (3/5)+0.13 [-0.17, +0.43] (2/5)
ADC-QBC (random expl.) − S-QBC+0.50 [+0.01, +0.99] (4/5)+0.62 [+0.48, +0.76] (5/5)+0.57 [+0.19, +0.96] (5/5)+0.23 [-0.06, +0.52] (4/5)+0.09 [-0.01, +0.19] (4/5)+0.23 [-0.04, +0.50] (4/5)
Full information (all labels)77.6977.6977.6977.6977.6977.69

Table 2b. Accuracy (%) at matched inference cost.

Methodn = 1,000n = 2,000n = 4,000n = 8,000n = 10,000n = 12,000
Random77.27 ± 0.1277.46 ± 0.1677.42 ± 0.2277.60 ± 0.1177.60 ± 0.2177.73 ± 0.15
S-QBC76.48 ± 0.3476.59 ± 0.1876.93 ± 0.3677.47 ± 0.3077.63 ± 0.1577.63 ± 0.14
ADC-QBC (ours), S-QBC exploration76.42 ± 0.3676.63 ± 0.2077.19 ± 0.1777.61 ± 0.2277.79 ± 0.1577.96 ± 0.09
ADC-QBC (ours), random exploration76.89 ± 0.4776.90 ± 0.2377.23 ± 0.0977.54 ± 0.1177.67 ± 0.0677.83 ± 0.11
ADC-QBC (S-QBC expl.) − Random-0.85 [-1.40, -0.30] (0/5)-0.83 [-1.21, -0.46] (0/5)-0.23 [-0.64, +0.18] (1/5)+0.01 [-0.37, +0.39] (2/5)+0.18 [-0.19, +0.56] (4/5)+0.23 [+0.04, +0.43] (5/5)
ADC-QBC (S-QBC expl.) − S-QBC-0.06 [-0.31, +0.19] (2/5)+0.04 [-0.33, +0.40] (3/5)+0.26 [-0.33, +0.85] (4/5)+0.14 [-0.37, +0.66] (3/5)+0.16 [-0.08, +0.39] (3/5)+0.33 [+0.13, +0.53] (5/5)
ADC-QBC (random expl.) − Random-0.38 [-1.11, +0.35] (1/5)-0.56 [-0.91, -0.21] (0/5)-0.19 [-0.49, +0.10] (1/5)-0.06 [-0.26, +0.13] (2/5)+0.07 [-0.24, +0.37] (3/5)+0.10 [-0.20, +0.41] (3/5)
ADC-QBC (random expl.) − S-QBC+0.41 [-0.14, +0.96] (4/5)+0.31 [-0.06, +0.68] (5/5)+0.30 [-0.22, +0.81] (3/5)+0.07 [-0.16, +0.31] (3/5)+0.04 [-0.11, +0.19] (4/5)+0.20 [-0.09, +0.48] (4/5)
Full information (all labels)77.6977.6977.6977.6977.6977.69

Table 3. Why random sampling leads at small budgets at matched cost (mean over 5 seeds).

Predicted minus true accuracy of Yi-34B (points)

n = 1,000n = 2,000n = 4,000n = 12,000
Random-0.4-0.5-0.6-0.4
S-QBC-2.2-2.0-1.5-0.3
ADC-QBC, S-QBC expl.-4.9-5.2-3.6-0.2
ADC-QBC, random expl.-3.8-4.6-3.8-0.7
Full information-0.3-0.3-0.3-0.3

Share of test queries routed to Yi-34B at λ = 1 (%)

n = 1,000n = 2,000n = 4,000n = 12,000
Random55.355.354.755.3
S-QBC49.750.451.554.8
ADC-QBC, S-QBC expl.41.941.745.956.8
ADC-QBC, random expl.46.243.345.855.0
Full information54.754.754.754.7

Normalized cost per query at λ = 1

n = 1,000n = 2,000n = 4,000n = 12,000
Random0.0330.0330.0330.033
S-QBC0.0370.0370.0350.034
ADC-QBC, S-QBC expl.0.0420.0420.0400.034
ADC-QBC, random expl.0.0390.0420.0400.035
Full information0.0330.0330.0330.033

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.