acceptodds
Under review as a conference paper at ICLR 2027

Compare or Cover? Feedback Geometry for Budgeted LLM Router Construction

Abstract

Learned LLM routers are trained on request–model quality scores, but a finite annotation budget reveals only part of the score matrix. This creates a choice: cover more distinct requests, or compare models on shared requests. We formalize this choice as support design at equal annotation cost, varying training-request overlap while holding labels per model, predictors and tuning data fixed. Deferral curves at matched switching rates separate routing quality from switching frequency. An exact coupling identity links the reduction in gain-prediction mean squared error to exactly twice the covariance between separately fitted predictors. For linear smoothers, shared supports turn gain prediction into regression on observed paired gains. Across 10,080 router fits on five archives, full sharing achieves the best mean rank among nine equal-cost designs, including D-optimal, k-center, uncertainty sampling and covariance-aware adaptive allocation. Sharing improves deployed routing quality over independent sampling in all ten archive–predictor pairs and reduces the build-label budget needed to reach zero area under the deferral curve by up to a factor of 1.8. Direct measurements show lower gain-prediction error and selection optimism, with routing gains increasing as model pools grow from 2 to 32 models. The benefit persists across representations and predictor families and replicates on a sixth archive under a frozen protocol. These results establish full sharing as a simple default for budgeted router construction at no additional annotation cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model routing, data acquisition, label efficiency, variance reduction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BFB6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies support design for LLM router construction under a fixed annotation budget, comparing models scored on independently sampled requests with models scored on shared requests. It derives a coupling identity explaining how shared supports can reduce gain-prediction error and evaluates routing quality at matched switching rates.

Critical strengths
  1. The paper addresses a practical and useful question in router construction. It shows that support design can affect routing quality at equal annotation cost. Shared supports offer a simple way to improve routing without changing the predictor family or purchasing additional labels.
  2. The theory and experiments jointly support the benefits of shared supports. The analysis explains how sharing can reduce gain-prediction error, and the experiments measure both this error and routing quality at matched switching rates. This makes the evidence more convincing than reporting downstream performance gains alone.
Critical weaknesses
  1. All six archives were accessed during method development, and exploratory experiments on the sixth archive preceded the frozen-protocol replication. The paper therefore lacks a confirmatory evaluation on an archive entirely untouched during method development.
  2. The theory and main experiments assume a separately fitted predictor for each model. With a joint predictor that can learn from labels across models, broader request coverage may offer additional benefits, but the paper does not test whether shared supports remain preferable in this setting. The recommendation of full sharing as a default therefore lacks sufficient support beyond separately fitted predictors.
Questions for authors
  1. Do you have results comparing shared and independent supports with a joint predictor that learns across models? If not, should the recommendation of full sharing as a default be limited to separately fitted predictors?
  2. What did the four exploratory runs on the sixth archive examine, and did their results influence the method or the frozen evaluation protocol?
3 comments
AuthorsOpenReview ↗

Response to Reviewer BFB6 [Comment 1/1]

We thank the reviewer for the clear assessment and the two concrete questions.

Joint predictors and the scope of the recommendation (W2, Q1). The submitted version noted that overlap leaves each separate fit's marginal distribution unchanged, whereas pooling labels across models introduces a trade-off between pairing and coverage (Remark 1; Appendix C). The abstract and conclusion did not state this scope clearly enough. The revision restricts the recommendation to separately fitted predictors and develops the joint-learning case through an explicit decomposition and a new proposition. Eq. (6) in Section 4 shows that, under joint learning, sharing must win the comparison between the paired cross term and the single-model error that broader coverage can lower. For joint residual smoothers, Prop. 4 shows that learning the common difficulty function can remove its contribution to gain error under independent supports. With a fixed estimate of the common function and the stated noise assumptions, pairing retains a noise-variance advantage. The revised manuscript distinguishes this theoretical result from the empirical comparisons of joint learners fitted to the purchased labels.

We examine this trade-off using three learner settings (Section 6.9; Appendices P, Q.4).

  • Real archives. A network trained on all purchased labels compares seven designs on SPROUT and MixInstruct (3 budgets × 20 seeds, two implementations). Independent supports cover 2.5–5× more distinct requests, but neither layout consistently wins. The sign at b ≥ 256 differs between implementations, and the pooled differences are +0.045 [−0.078, 0.190] and +0.115 [−0.054, 0.295] pp. Under this fixed recipe, all joint routers remain below the separately fitted shared router (SPROUT b = 1024: at most +0.35 vs +1.44/+1.97 pp).
  • Joint residual learner on all five archives (Prop. 4). Learning the common function from all pooled labels yields no clear improvement over separate independent fits (−0.009 pp [−0.028, 0.011]), while sharing still leads by +0.112 pp [0.069, 0.156].
  • Controlled joint low-rank learner. Sharing raises single-model error in 17 of 18 cells. Under correlated score noise (ρ_ε ≥ 0.3), the paired cross term more than offsets this cost in all 12 cells. Without coupling, no consistent winner emerges.

The covariance that governs the separate-predictor case is positive in 97.7% of 216 challenger cases across the five archives, and significantly negative in none (Section 6.4, Appendix O). CASA's pilot allocation tracks it (Spearman 0.85).

The sixth archive and the four exploratory runs (W1, Q2). Appendix K now describes the four runs. They came from an earlier phase of the project and tested other allocation strategies: a cardinality-adaptive rule, an external comparison and two hierarchical variants. Their design differs from the replication throughout:

Earlier runsReplication
Pool sizeK = 2–4K = 5 and 12
Labels per model1–3232–1024
MetricCurve-level AUC reductionAUDC at q̄ = 0.5

None compared shared with independent supports or evaluated CASA. The replication protocol fixed the pool (the alphabetically first five of twelve models), incumbent rule, CASA's original parameters, predictors, grids, budgets, seeds, metric and decision rules, and was hashed before the replication opened any score. No element was taken from the earlier runs. Because the same score matrix had been accessed before, we describe the result as an external replication under a frozen protocol, not a confirmatory test on a never-opened archive. Sections 5 and 6.7 state the corresponding status of all real-archive results.

The response addresses some of my concerns, they do not change my overall assessment. I am keeping my original rating.

Thank you for reviewing our paper and for following up. We’re pleased that our response helped address some of the concerns you raised.

Reviewer_YSoAOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper studies how to allocate a limited annotation budget when building LLM routers: whether to evaluate different models on the same requests or cover more distinct requests. The authors find that shared requests generally improve routing performance when a separate quality predictor is trained for each model.

The problem is relevant, but I have substantial concerns about the restrictive experimental setting and the limited novelty of the contribution. In particular, I do not think the current evidence is sufficient to support the paper's broader recommendation of full sharing as a general default for router construction.

Critical strengths

The paper addresses an important practical problem. Building LLM routers under limited annotation budgets has received increasing attention, and understanding how to allocate request–model evaluations is a meaningful research question. I also appreciate the authors' transparency in reporting negative results, inconsistent effects with nonlinear predictors, and prior exploratory use of the sixth dataset.

Critical weaknesses

1. The restrictive problem setting substantially limits the validity and generality of the main conclusion.

My main concern is that the paper studies the compare-or-cover trade-off under a setting that largely removes the potential benefit of coverage. Specifically, each model has its own independently trained quality predictor, which only uses labels collected for that model. Consequently, although independent sampling covers substantially more distinct requests, the additional coverage cannot be exploited through information sharing across models. Under this design, both sampling strategies provide the same marginal training distribution for each predictor, leaving cross-model prediction covariance as the key difference.

This restriction is also central to Proposition 1, which assumes identical marginal prediction distributions under shared and independent supports. While the covariance identity is mathematically valid under this assumption, it does not characterize the more general trade-off between paired comparisons and broader request coverage. In particular, joint predictors that learn from all purchased request–model labels may benefit from covering more distinct requests, potentially changing the relative advantages of the two strategies.

This is a fundamental limitation rather than a minor missing experiment. The paper aims to answer whether a limited budget should be spent on comparing models or covering requests, but its modeling assumptions effectively prevent the broader coverage from benefiting the learning process. The supplementary MLP and tree experiments do not address this concern, as they still train separate predictors for each model.

As a result, the experiments establish that shared supports are beneficial for a particular class of independently trained predictors, rather than demonstrating that full sharing is a generally preferable strategy for budgeted LLM router construction. I find the recommendation of full sharing as a general default insufficiently justified.

2. The paper offers limited methodological and theoretical novelty.

The main theoretical insights largely follow from classical paired experimental design and standard variance decomposition. The proposed approach does not introduce a new routing architecture, learning objective, or substantially new annotation strategy; it primarily evaluates shared random sampling with existing predictors. The additional adaptive method, CASA, does not meaningfully outperform full sharing. While the empirical analysis is useful, I find the methodological contribution rather limited for ICLR, especially given the restrictive setting discussed above.

Questions for authors
  1. Why does the paper focus primarily on separately trained per-model quality predictors? How representative is this setting of practical LLM routers, particularly those that share representations or parameters across models?

  2. Proposition 1 relies on identical marginal prediction distributions under the two support designs. Given that this assumption may not hold for joint predictors, what evidence supports recommending full sharing as a general default rather than specifically for independently trained predictors?

  3. What happens if a single predictor is trained on all purchased request–model labels? In this setting, independent sampling could expose the model to substantially more distinct requests. Would full sharing still be preferable?

  4. Can the authors identify meaningful conditions under which broader coverage would outperform shared sampling? Understanding these boundaries seems essential to answering the compare-or-cover question rather than only demonstrating the benefits of comparison.

  5. Beyond applying paired experimental design to LLM routing, what is the main new methodological insight? Given that CASA does not meaningfully outperform random full sharing, what new capability or practical decision rule does the paper provide beyond existing statistical intuition?

  6. Several evaluated routers still underperform the best single model, even with shared supports. How should the practical value of the proposed recommendation be interpreted in settings where the resulting router does not provide positive routing gains?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YSoA [Comment 1/2]

We thank the reviewer for focusing on the central question of the paper.

Separately fitted predictors and the joint-learning boundary (W1, Q1–Q3). We chose per-model predictors as the starting point because they isolate the channel through which overlap acts. With equal marginals, any difference between layouts must come from the dependence between predictors (Prop. 1). They are also the design used by similarity routers, the strong baselines in recent routing analyses. Section 6.6 extends the result to separately trained networks and trees. We agree that this is the central boundary of the analysis. The submitted version noted that separate fits cannot use the coverage of the support union, whereas pooling labels across models introduces a pairing–coverage trade-off (Remark 1; Appendix C).

The abstract and conclusion nevertheless stated the recommendation too broadly. The revision restricts the recommendation to separately fitted predictors and develops the joint-learning analysis:

  • Section 4 now gives the joint-learning decomposition, Eq. (6): sharing must win the comparison between the paired cross term and the single-model error that coverage can lower, so positive coupling alone no longer settles the choice. For joint residual smoothers, Prop. 4 shows how learning common difficulty can remove its contribution to gain error under independent supports. Its noise-variance comparison assumes a fixed estimate of the common function and the stated noise conditions, rather than applying to arbitrary joint predictors.
  • Section 6.9 and Appendices P and Q.4 test Eq. (6).
  • The abstract and conclusion state the recommendation for separately fitted predictors.

On Q3, a single network trained on all purchased labels compares seven designs on SPROUT and MixInstruct (3 budgets × 20 seeds, two implementations). Independent supports cover 2.5–5× more distinct requests, yet neither layout consistently wins. The sign at b ≥ 256 differs between implementations; pooled differences are +0.045 [−0.078, 0.190] and +0.115 [−0.054, 0.295] pp. Of twelve cell estimates, only two are significant, one in each direction (about 0.05 pp each). Uncertainty sampling exceeds sharing on SPROUT at b = 1024 (+0.51 pp). Under this fixed recipe, all joint routers remain below the separately fitted shared router (SPROUT b = 1024: at most +0.35 vs +1.44/+1.97 pp).

When coverage can win (Q4). Prop. 4 characterizes this trade-off for joint residual smoothers: under its assumptions, learning the common function can replace the cancellation of feature-predictable common difficulty, while positively correlated score noise still favours pairing. On the five real archives, this learner shows no clear improvement over separate independent fits (−0.009 pp [−0.028, 0.011]), while sharing still leads by +0.112 pp [0.069, 0.156]. A controlled joint low-rank learner measures both terms of Eq. (6). Sharing raises single-model error in 17 of 18 cells, showing that coverage does help joint fitting, as you anticipated. Under correlated score noise (ρ_ε ≥ 0.3), the paired cross term more than offsets this cost in all 12 cells. At ρ_ε = 0.6 and b = 128, the corresponding AUDC gain is +0.62 pp. In a post-hoc control with no coupling at all, four of six budgets show no significant difference; b = 128 favours sharing and b = 512 favours independent sampling (−0.034 pp [−0.047, −0.018]). In this generator, coverage competes when coupling is weak, but no setting is consistently dominated by coverage.

Whether coupling is weak in practice is measurable. Across the five real archives, the covariance of fitted predictions is positive in 97.7% of 216 challenger cases and significantly negative in none (Section 6.4, Appendix O).

AuthorsOpenReview ↗

Response to Reviewer YSoA [Comment 2/2]

Novelty and the decision rule (W2, Q5). The contribution is to characterize how support overlap affects router construction at a fixed budget. The classical covariance identity provides the starting point for this analysis. A router compares learned functions at new requests, then takes a maximum over challengers and applies a threshold. Whether the covariance is positive at that point is a property of the archive and the predictor; Appendix Q.3 shows complementary challengers for which it turns negative. The revised paper brings together the original study design and covariance analysis with the added joint-learning results:

  1. makes overlap a controlled variable at equal cost;
  2. reduces compare-or-cover for per-model predictors to the sign of this covariance, and extends the analysis to joint learning through Eq. (6);
  3. derives and tests consequences that the pairwise identity alone does not give: growth of the benefit with the number of challengers through reduced selection optimism (Prop. 5; 6.97 → 4.78 pp, and the benefit rises with pool size, Section 6.7), and, for joint residual smoothers under the stated assumptions, how learning common difficulty and pairing correlated score noise contribute differently to gain error (Prop. 4);
  4. turns the condition into a budgeted allocation rule.

What CASA provides. CASA turns the coupling condition into an allocation rule using a residual-correlation estimate from an in-budget pilot. We test whether this proxy tracks the covariance in Prop. 1 and whether adapting overlap improves performance:

  • Real archives. The post-pilot allocation tracks each challenger's measured coupling (Spearman 0.85 across 19 challengers, p = 3×10⁻⁶), and the two weakest-coupled challengers receive the least overlap. Since coupling is almost uniformly positive there, CASA converges to sharing and matches it (+0.004 pp [−0.021, 0.030]).
  • Controlled archives with mixed coupling signs. CASA has the smallest worst-case kNN shortfall among deployable designs (0.004 pp vs 0.077–0.207 pp for every fixed overlap, intervals above zero). It also beats a fixed 62.5% overlap design in 83 of 84 cells. Coverage varies between the two designs across configurations. At κ = κ_ε = 1 and b = 128, both cover approximately 316 distinct requests, supporting the value of allocating overlap according to challenger coupling at comparable coverage.
  • Ridge and same-sign coupling. For ridge, CASA and full sharing are within ±0.02 pp. In the controlled synthetic archives, when coupling has the same sign for all challengers, CASA costs 0.002–0.033 pp relative to sharing. On the sixth archive’s twelve-model pool, CASA trails sharing by 0.077 pp (95% CI [0.008, 0.147]), failing the registered H3 criterion (Appendix K).

The resulting decision rule makes the condition underlying CASA explicit: for separately fitted predictors, use shared supports when coupling is positive, and use the in-budget pilot to guide the overlap allocation for challengers whose coupling is weak, uncertain, or negative.

Routers below the best single model (Q6). We agree that these two questions must be kept apart: our claim concerns which support layout performs better at a given budget, not whether routing beats the incumbent at every budget. Sharing improves deployed gain in all ten archive–predictor pairs; where both layouts remain below the incumbent, it narrows the shortfall (Appendix H, Table 19). Sharing also reaches break-even first in all four pairs whose mean curves cross zero. On the sixth archive, both layouts remain below the incumbent in the registered five-model pool (shared: −0.206 pp; independent: −0.554 pp). With all twelve models, both achieve positive gains (shared: +0.278 pp; independent: +0.049 pp). The threshold rule includes the option not to switch, although a threshold selected on the panel does not guarantee a non-negative gain on the audit set.

Thank you for the detailed response and revisions. I appreciate that the authors acknowledge the overly broad original claim and have conducted the suggested joint-learning experiments, whose results are consistent with my concerns.

While the main claim is now better justified for separately trained per-model predictors, its generality remains limited. Moreover, CASA shows little practical benefit over full sharing on real-world benchmarks.

Overall, my main concern has been substantially addressed, but I still have reservations about the novelty and practical significance of the contribution. I therefore raise my rating to 2.

Thank you for the follow-up. We are glad the revision addressed your main concern. We see the theoretical contribution as identifying when training two model predictors on the same requests improves their comparison on a new request, and explaining how that condition changes when the predictors learn jointly. We appreciate your careful reading and respect your assessment of the novelty and practical significance of this contribution.

Reviewer_crDoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a fixed annotation budget is better spent evaluating models on shared or independently sampled requests when training separate quality predictors for LLM routing. It explains the benefit of sharing through predictor covariance and reports a +0.115 pp average AUDC improvement over independent sampling across five archives. The main contribution is an empirical analysis of training-data layout rather than a new routing architecture.

Critical strengths
  • The paper addresses a practical data-collection question. Keeping per-model label counts, predictor families, and selection panels fixed makes the shared-versus-independent comparison easy to interpret.
  • Comparing routers at matched switching rates is useful. It separates routing quality from differences in how often a router switches, while deployment results provide a complementary evaluation.
  • The covariance identity and paired-label interpretation give a clear explanation for the observed effect under their stated assumptions.
  • The fixed-hyperparameter controls and direct measurements of gain error and prediction variance strengthen the mechanism analysis beyond aggregate performance comparisons.
  • The experiments cover several archives, budgets, representations, and predictor families. The authors also openly report negative results and prior exploratory exposure.
Critical weaknesses
  • The contribution is limited by the use of separate predictors. Common-query supervision already appears in RouteLLM and CARROT, and the theoretical results largely follow from standard covariance decomposition and linearity. A joint predictor that can use labels - across models would better test whether sharing remains preferable when broader request coverage can actually help.
  • The explanation of the acquisition baselines seems unconvincing. D-optimal and k-center already use fully shared supports in Algorithm 1, so their underperformance cannot be attributed to missing cross-model alignment. These comparisons test input selection, not overlap.
  • On RouterBench with ridge, shared AUDC is +0.034 pp at 512 labels but returns to −0.004 pp at 1024, making the first zero crossing an uncertain efficiency measure. Including selection labels also reduces the reported 1.84x ratio to about 1.58x; crossing uncertainty and total-budget curves are needed.
  • Mean deployed gains remain negative on three of the five main archives. Switching frequency also matters: RouterBench ridge’s switch rate falls from 25.3% to 15.9%, while BiGGen kNN improves overall despite worse per-switch quality. These results do not support attributing deployment gains solely to better switch selection.
  • All six archives had prior exposure, and additional RouterBench pools were sampled after the initial slope interval included zero. Although the additional batch supports the trend, the pooled interval needs qualification. The pool-size scan also increases total annotation budget with K, so it does not establish the same trend under a fixed total budget.
Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer crDo [Comment 1/2]

We thank the reviewer for the precise reading of our manuscript.

Separate predictors, common queries and the theory (W1). RouteLLM and CARROT use common queries so that comparisons are made on comparable data. Neither asks whether this is the best use of a fixed budget compared with scoring more distinct requests. We compare these layouts at equal cost. The covariance identity is classical, with its basis in paired designs and common random numbers. The revised contribution list distinguishes this identity from its role in routing: for separately fitted per-model predictors with unchanged marginal training distributions, the compare-or-cover question reduces to the sign of the covariance between their predictions at new requests. This sign is an empirical property; Appendix Q.3 constructs challengers for which it is negative and sharing hurts them.

The analysis yields predictions supported by the following results:

  • the covariance is positive in all 56 settings;
  • fixed-hyperparameter variance ratios match the covariance prediction to a median of 0.014;
  • selection optimism falls from 6.97 to 4.78 pp;
  • the benefit grows with the number of challengers.

The challenger-level analysis further shows positive covariance in 97.7% of 216 cases and no significantly negative cases. It also directly tests the link between this condition and CASA's allocation, finding a Spearman correlation of 0.85 between measured coupling and post-pilot allocation.

The submitted version noted that the union of supports does not enter any separate fit (Remark 1), whereas pooling labels across models introduces a trade-off between pairing and coverage (Appendix C). The revision develops this point through Eq. (6), Prop. 4, and the following experiments. Prop. 4 analyzes joint residual smoothers with common smoothing weights under shared supports. Their predicted gains are algebraically identical to those of separate fitting. Under independent supports, learning the common difficulty function can remove its contribution to gain error. The noise-variance result assumes a fixed estimate of the common function and the stated noise conditions; the experiments separately examine learners fitted to the purchased labels.

  • Real archives. A network trained on all purchased labels, seven designs, SPROUT and MixInstruct, 3 budgets × 20 seeds, two implementations: independent sampling does not consistently beat sharing. The sign at b ≥ 256 differs between implementations; pooled differences are +0.045 [−0.078, 0.190] and +0.115 [−0.054, 0.295] pp. At SPROUT b = 1024, the best joint design reaches +0.35 pp, against +1.44/+1.97 pp for separately fitted shared kNN/ridge, under the single fixed training recipe we used.
  • Joint residual learner (Prop. 4). On all 56 real settings, learning the common function from up to Kb distinct requests changes independent-support AUDC by −0.009 pp [−0.028, 0.011]; sharing still leads by +0.112 pp [0.069, 0.156] (Appendix P).
  • Controlled low-rank learner. Sharing raises single-model error in 17 of 18 cells, so coverage does help joint fitting. Under correlated score noise (ρ_ε ≥ 0.3), the paired cross term more than offsets this cost in all 12 cells (Section 6.9, Appendix Q.4).

Acquisition baselines (W2). You are right that our description of these baselines was inaccurate. Our D-optimal and k-center implementations use one common support (Algorithm 1). The submitted Appendix A sentence saying that these criteria do not reward alignment misdescribed them. The revision corrects that sentence and reorganizes Section 6.1 and the Figure 4 caption around two axes:

  • alignment at random selection: shared minus independent, +0.115 pp;
  • selection at full alignment: random sharing beats D-optimal by 0.161 pp and k-center by 0.260 pp.

On the real archives, the benefit therefore comes from alignment, and informative selection does not add to it. In the synthetic archives, alignment again produces the larger contrast, while informative selection adds a small further gain (Appendix Q.2).

AuthorsOpenReview ↗

Response to Reviewer crDo [Comment 2/2]

Break-even ratio (W3). We agree that the single factor was too sensitive to report as a headline. The revision removes it from the abstract and reports seed-bootstrap crossings (Section 6.3; Appendix F.2, Tables 11–12, Figure 12). The direction is consistent: sharing crosses zero first in all four archive–predictor pairs whose mean curves cross within the measured range, with bootstrap probabilities of 95% (SPROUT kNN), 66% (SPROUT ridge), 83% (MixInstruct ridge) and 66% (RouterBench ridge). The magnitude is uncertain where curves are flat: the RouterBench ridge build-label ratio is 1.84 [0.22, 3.58]. The selection panel is identical for every design. The build-label ratio therefore measures support efficiency, and the total-label ratio (1.58) measures project cost; both are reported.

Deployed gains and switching frequency (W4). You are right that our sentence "which requests are switched rather than how many" was inaccurate for deployment; we have replaced it. AUDC compares routers at matched switching rates (Section 3.1), with gains of +0.053 / +0.100 / +0.144 pp at fixed rates of 5 / 10 / 20% (Appendix J.4). At a panel-selected threshold, however, switching frequency is an output of the router, so both the requests switched and their number can change. Sharing lowers the optimism of the selected challenger (6.97 → 4.78 pp), and the revised Section 6.5 and Appendix H describe the consequence for deployment:

  • RouterBench ridge switches less (15.9% vs 25.3%) and loses less per switch (−0.342 vs −0.479 pp);
  • on BiGGen kNN, per-switch quality is slightly lower under sharing (-5.55 vs -5.41 pp on 179 audit requests), and the gain comes from switching less often (10.8% vs 12.8%).

Sharing improves the deployed gain in all ten archive–predictor pairs. Where both designs remain below the incumbent, sharing narrows the shortfall.

Prior exposure and pool size (W5). Section 5 states the development use of the five archives and labels all real-archive results as development results. Section 6.7 now reports the RouterBench batches next to the pooled slope: the initial pools give +0.118 [−0.006, 0.250] and the additional pools alone give +0.153 [0.025, 0.275]; EmbedLLM independently gives +0.173. The original pool-size scan was designed to test Prop. 5, which concerns increasing the number of challengers at a fixed per-model budget. It therefore addressed the proposition's stated setting; it did not test whether the same growth persists when the total budget is fixed. The revision makes this distinction explicit and adds a complementary scan at fixed total build budget T = Kb ∈ {512, 2048}, using new pools (Appendix J.5, Table 30). Sharing improves mean AUDC at every pool size on both archives. Growth with K persists on RouterBench (+0.113 pp per doubling, [0.024, 0.213]) and is flat on EmbedLLM (−0.007, [−0.088, 0.065]).

Reviewer_crDoOpenReview ↗

Official comments by Reviewer crDo

Thank you for the responses to my questions. After carefully reading the responses, I would like to keep my original scores, as the paper may need substantial modification with comprehensive re-evaluation on the new setups.

Thank you for reading our responses so carefully. We appreciate the time you devoted to reviewing the paper and the questions you raised during the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading. Our central question is when comparing models on shared requests is better than covering more distinct requests. The revision makes the connection between the theoretical condition, empirical evidence, and allocation rule clearer, states the recommendation within its scope, and strengthens the analysis through finer-grained measurements and controlled experiments.

The condition (Section 4). For separately fitted predictors whose marginal training distributions do not change, sharing lowers gain-prediction error by twice the covariance between the fitted challenger and incumbent predictions at new requests (Prop. 1). The algebra is the classical paired-variance identity. What it adds for routing is that the condition concerns learned functions evaluated outside their supports, so its sign must be measured, and it can be negative. The submitted version noted that pooling labels across models introduces a trade-off between pairing and coverage (Remark 1; Appendix C). The revision formalizes this trade-off in Eq. (6). Eq. (6) shows that under joint learning, sharing must win a comparison between the paired cross term and the single-model error that broader coverage can lower. A new proposition (Prop. 4) analyzes joint residual smoothers. For this class of learners, learning the common difficulty function can remove the corresponding error term under independent supports. With a fixed estimate of the common function and the stated noise assumptions, pairing retains a variance advantage against positively correlated score noise. This variance comparison does not directly characterize a common function estimated from the same noisy labels.

Measuring the condition (Section 6.4, Appendix O). We extend the pool-level coupling analysis to individual challengers, measuring the covariance of fitted predictions across 20 support draws in 216 challenger–budget–predictor cases across five archives. It is positive in 97.7%, significantly positive in 91.2% and significantly negative in none. The revised text separates this quantity from raw score correlation and from the residual correlation used by CASA's pilot.

Varying the condition (Sections 6.8–6.9, Appendix Q). In synthetic archives frozen before evaluation (10 instances × 12 seeds, 203,040 fits), the sharing gain rises with noise correlation (kNN +0.13 → +0.79 pp) and disappears when all coupling is removed (+0.000 pp [−0.027, 0.028]). When challengers differ in coupling sign, the best fixed overlap changes from one configuration to the next.

The allocation rule. CASA allocates sharing according to the coupling estimated from an in-budget pilot. We test whether this pilot identifies the relevant condition and whether adaptive allocation improves performance when coupling signs differ across challengers. On real archives, its post-pilot allocation tracks measured coupling (Spearman 0.85 across 19 challengers). In the controlled archives with heterogeneous coupling signs, it has the smallest worst-case kNN shortfall among deployable designs (0.004 pp versus 0.077 to 0.207 pp for fixed overlaps) and beats a fixed 62.5% overlap design in 83 of 84 cells. For ridge, CASA and full sharing are within ±0.02 pp.

Joint learning (Section 6.9, Appendices P, Q.4). A network trained on all purchased labels shows no consistent shared–independent ordering across two implementations (pooled +0.045 [−0.078, 0.190] and +0.115 [−0.054, 0.295] pp). A direct real-archive test of Prop. 4 fits a common kNN function to all pooled labels before per-model residual smoothing. Over all 56 settings, this changes AUDC under independent supports by −0.009 pp [−0.028, 0.011], and sharing still leads by +0.112 pp [0.069, 0.156]. Thus, the tested common-function learner does not eliminate the sharing advantage on these archives. A controlled joint low-rank learner measures both terms of Eq. (6): sharing raises single-model error in 17 of 18 cells, and under correlated score noise (ρ_ε ≥ 0.3), the paired cross term more than offsets this cost in all 12 cells.

Scope of the recommendation. The abstract and conclusion now give the recommendation in conditional form: for separately fitted predictors, share when predictions are positively coupled. For joint predictors, the choice depends on the two terms of Eq. (6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.