acceptodds
Under review as a conference paper at ICLR 2027

CopRoute: Dependence-Aware Conformal Routing for Language Models

Abstract

Set-based language model routing aims to increase the coverage of the event where at least one model answers a query correctly while controlling the number of inference calls made. Existing risk-controlled routing policies bound the all-miss risk (the probability that every model in the selected set fails) by calibrating a single conformal threshold and applying it to a nested family of model sets. Because each model is scored in isolation, these policies ignore the dependence structure of model failures. Models that share training data or architecture tend to fail on the same queries, so two individually strong models can be jointly redundant while a weaker but complementary model covers the queries that the others miss. To account for this dependence, which existing risk-controlled policies ignore, we propose CopRoute, which orders models by how many new queries each one answers correctly beyond those already selected, decides per query how far down that order to go, and then picks the smallest cutoff that still meets the target risk. We show that CopRoute's ordering attains, on the design data, the best guarantee any efficient fixed-ordering algorithm can give for the average number of calls to the first correct model, and that a separate calibration step certifies the all-miss risk of the deployed sets. On RouterBench and RouterEval, CopRoute attains the fewest certified calls on nine of ten datasets at a target risk of 0.05 and on eight of ten at 0.10, making up to 79 percent fewer calls (median 24 percent) than the state-of-the-art risk-controlled policy at the same certified all-miss risk.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM routing, dependence-aware, conformal prediction

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_QRSuOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a method to learn an order from joint model-correctness labels, size a set for each query using router scores, and calibrate one cutoff on held-out data. It uses an independence procedure for sizing and a co-failure tally for ordering. The paper applies the minimum-sum set-cover approximation to its design objective and gives a frozen-family split-calibration guarantee. It evaluates certified call counts on RouterBench and RouterEval, with cost, transfer, numerical, and estimator analyses.

Critical strengths
  1. A simple way to use joint model errors. The work adds co-failure information to a marginal router through a fixed order. The method is easy to understand, and the controls help separate the benefit of the order from that of dependence-corrected stopping.
  2. Theory with clear targets and assumptions. The paper connects the design ordering to the known minimum-sum set-coverage and gives a split-calibration guarantee for a frozen nested family. The calibration result does not require a correct dependence estimator.
Critical weaknesses
  1. Mismatch between theoretical guarantees and deployment-level efficiency. Theorem 1 provides a 4-approximation for the average first-success position on the design set, but this objective does not directly characterize the expected number of certified calls at a fixed risk level. Moreover, Theorem 2 controls the joint certification-and-miss event, whereas the reported call efficiency is evaluated only over certified redraws. Although the paper acknowledges these distinctions, the theoretical results do not yet provide a unified characterization of deployment-level risk and efficiency, particularly when certification is infrequent.

  2. Limited exploitation of query-specific dependence. Although CopRoute is presented as a dependence-aware routing method, its default implementation uses a fixed global ordering learned from historical co-failure patterns, while query-adaptive sizing still relies on an independence-based approximation. Moreover, the explicit dependence correction (CopRoute-Dep) does not consistently improve performance, raising questions about the extent to which the method effectively captures conditional dependence.

  3. Concerns about baseline fairness and data efficiency. CopRoute uses an additional design split to learn model dependencies, whereas RACER does not fully exploit these labels for router training. Although the shared-router comparison isolates the contribution of the proposed ordering, it does not establish superiority under equal total labeled-data budgets. Furthermore, the evaluation modifies RACER's original abstention mechanism, making the comparison less representative of the complete original framework.

Questions for authors
  1. Can you align the main risk and efficiency claims with the joint-event guarantee, report the certification denominator, and state the action and cost for non-certification? Please also keep the price and answer-selector conditions next to the cost claim.
  2. Why does the proposed method rely on a fixed global ordering rather than query-specific dependence-aware selection? Could the authors provide a systematic comparison against query-local greedy routing on the real benchmarks and clarify when global co-failure information is preferable to conditional dependence estimates?
  3. What is the main theoretical contribution beyond RACER's existing conformal risk-control framework? In addition, does CopRoute retain its efficiency advantage when both methods are given the same total labeled-data budget? A budget-matched comparison would help distinguish improvements attributable to dependence-aware routing from those resulting from additional training information.
Reviewer_SHW5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors build on and extend a recently proposed set-based routing paradigm. The authors' contribution is to build nested subsets of candidate models, based on minimizing the joint probability that all chosen models answer the query incorrectly. Compared to competing methodologies, the method needs fewer models to certify that at least one model within the set will answer the query correctly. The authors provide theoretical results to support their methodology.

Critical strengths
  • The paper's methodology attains correctness certificates with smaller model sets compared to competing methodologies
Critical weaknesses
  • The paper deals entirely with correctness of models within LLM pools and disregards model cost/latency, ignoring the most important practical motivations for LLM routing
  • There is no clear way to apply the paper's certificate: we know that at least one model out of a candidate set will answer the query correctly, but we do not know which model's answer is the correct one
Questions for authors
  • Explain how your set-based routing methodology can be employed in practice. Let's say your method stops at step k and returns a certificate for a pool of k models. How do we know which of these models will yield the correct answer? In practice, which answer would we pick?
  • How does your method result in inference cost reductions, especially the quoted 46% reduction (see question above)?
  • Your all-error metric is sensitive to evaluation noise, especially when the pool of models is large: only one LLM's correctness value has to flip from "incorrect" to "correct" for an entire row in the design matrix to be corrupted. This also applies to published benchmarks, which often have label noise around 5%. Have you analyzed the impact of label noise on your methodology?
  • You do not explicitly state the models on which you evaluate. Please describe the range of model sizes and capabilities present within the pool under consideration.
  • In your contribution statement, you start with "We show that the all-miss probability of a model set depends jointly on the individual model accuracies and on the dependence among model failures. Under conditional independence, scoring models in isolation is optimal at every set size, while under dependence the best set need not contain the individually best models." Please explain this specific contribution further: where does your differential insight lie relative to the existing literature?
  • You mention a "correctness subcopula" in the body of the manuscript but do not define it. Please explain.
11 comments
AuthorsOpenReview ↗

Response to Reviewer SHW5 - Part 1

We thank you the reviewer for their constructive feedback. Here we would like to clarify a few questions and concerns raised by them.

1. Explain how your set-based routing methodology can be employed in practice. Let's say your method stops at step k and returns a certificate for a pool of k models. How do we know which of these models will yield the correct answer? In practice, which answer would we pick?

In practice, we call the selected k models in parallel and return the answer from the called model with the highest router score. This is already the deployment rule used in submitted Table 3 (lines 451 to 455) and described again in Appendix L (lines 1878 to 1885). It reuses the router scores computed before model selection, and the submitted appendix measures its overhead at 2.3 microseconds per query.

This separation between model selection and answer selection is also used by prior set-based routing methods. RACER selects a model set and then applies either majority voting or weighted aggregation to its outputs (RACER Section 3.3 and Algorithm 1). A certificate over a selected set does not reveal which member is correct under either method. It ensures that the downstream answer selector receives a set that is unlikely to contain only wrong answers.

During the rebuttal, we conducted an additional analysis of three answer-selection rules: highest-router-score selection; plurality voting (return the answer produced by the largest number of called models, breaking ties by their average router score); and router-score-weighted voting (return the answer receiving the largest sum of softmax-normalized router weights). We evaluated CopRoute, RACER-G, and RACER-P over 100 random data splits of all four RouterBench datasets at target all-miss risks 0.05 and 0.10. The table presents the target-risk 0.10 results. All 100 splits produced certificates. “At least one correct” is the percentage of questions for which one or more called models answered correctly.

DatasetMethodMean models calledAt least one called model is correctHighest-router-score answer is correctMost-common answer is correctRouter-score-weighted answer is correct
ARC-ChallengeCopRoute1.0096.1%96.1%96.1%96.1%
RACER-G1.0690.2%89.7%89.7%89.7%
RACER-P1.6589.8%88.6%88.4%88.4%
HellaSwagCopRoute1.8390.5%84.2%84.2%84.2%
RACER-G1.8190.0%83.6%83.9%83.9%
RACER-P2.2989.9%82.1%82.0%82.0%
MMLUCopRoute3.0290.2%81.0%77.0%77.1%
RACER-G5.6990.2%80.3%75.0%75.0%
RACER-P6.5190.2%80.3%74.3%74.3%
WinoGrandeCopRoute1.8290.9%81.7%81.6%81.6%
RACER-G3.2190.3%80.5%76.0%76.1%
RACER-P3.1690.3%80.4%75.3%75.3%

The highest-router-score rule matches or exceeds both voting rules for CopRoute on all four datasets and is substantially better on MMLU, where it reaches 81.0 percent accuracy compared with 77.0 and 77.1 percent. It also gives CopRoute higher final-answer accuracy than both RACER variants on every dataset shown, while requiring substantially fewer calls on MMLU and WinoGrande.

This also answers the concern that the certificate has no clear practical use. If every called model is wrong, no answer-selection rule can succeed. CopRoute controls this failure before answer selection and reduces the number of candidates that must be considered. The highest-router-score rule then supplies one concrete final answer. When an external check is available, such as unit tests for code, a mathematical verifier, or human review, it can replace the router-score rule and inspect the same smaller candidate set. The revised paper will report both the frequency with which at least one called model is correct and the accuracy of the final selected answer.

AuthorsOpenReview ↗

Response to Reviewer SHW5 - Part 2

2. Have you analyzed the impact of label noise on your methodology?

We agree with the reviewer’s observation: a false-positive label can make an all-miss query appear covered. If each of m incorrect model labels independently flips to correct with probability η, the probability that an all-miss row receives at least one false positive is 1 − (1 − η)^m. At η = 0.01 and m = 500, this probability is 99.3 percent. We therefore evaluated false-positive and false-negative errors separately and varied the model-pool size explicitly.

During the rebuttal, we corrupted separate copies of the correctness labels while retaining the released benchmark labels for final evaluation. The experiment covers ten datasets, model pools from 11 to 500, 100 random data splits, and ten random corruption patterns per split. We test false-positive flips, false-negative flips, symmetric flips, and corruption of entire rows at rates from 0.1 to 20 percent. We corrupt the design, router-training, validation, and final-calibration labels separately and evaluate CopRoute, CopRoute-Dep, two ordering controls (a fixed order based only on individual design accuracy and a query-specific order based only on router scores), RACER-G, and RACER-P at target all-miss risks 0.05 and 0.10.

The main finding is that the effect depends strongly on where the error occurs. False positives in the design labels can distort CopRoute's model order and increase the number of calls, but final calibration with uncorrupted labels continues to control the all-miss risk. False positives in the final-calibration labels are more serious: they make failures appear successful, so calibration selects fewer models and the all-miss risk measured against the original labels increases. This vulnerability also affects RACER because both methods rely on the same final-calibration labels.

The table reports representative large-pool results at a 5 percent false-positive rate and target risk 0.10. Each entry is mean model calls / all-miss risk measured using the original labels. Bold highlights CopRoute’s lower call count relative to RACER-G under the same label condition. Under false-positive calibration noise, CopRoute also has lower all-miss risk than RACER-G in every setting shown.

Dataset and poolMethodNo label corruptionFalse positives in design labelsFalse positives in final-calibration labels
IFEval, 500 modelsCopRoute3.07 / 9.2%3.29 / 9.7%2.39 / 10.5%
RACER-G4.77 / 9.5%4.77 / 9.5%3.67 / 11.0%
GPQA Diamond, 500 modelsCopRoute14.35 / 9.4%15.33 / 9.6%9.18 / 13.3%
RACER-G18.42 / 10.3%18.42 / 10.3%10.95 / 17.3%
MuSR, 200 modelsCopRoute13.33 / 9.8%20.09 / 9.9%6.48 / 14.2%
RACER-G56.72 / 9.3%56.72 / 9.3%29.27 / 15.4%

These comparisons separate two effects. Design-label false positives primarily reduce efficiency: on MuSR, CopRoute increases from 13.33 to 20.09 calls while the all-miss risk remains 9.9 percent. RACER-G is unchanged because it does not use the design labels to construct an order. By contrast, false-positive calibration noise makes both methods overconfident. On GPQA Diamond, the all-miss risk rises from 9.4 to 13.3 percent for CopRoute and from 10.3 to 17.3 percent for RACER-G.

False-negative calibration errors have the opposite effect. They make correct answers appear wrong during calibration, so both methods call more models and the all-miss risk measured against the original labels decreases. The table reports a 5 percent false-negative rate in the final-calibration labels at target risk 0.10. Each entry is mean model calls / all-miss risk measured using the original labels. Bold highlights the reduction in all-miss risk under false-negative noise.

Dataset and poolMethodNo label corruptionFalse negatives in final-calibration labels
IFEval, 100 modelsCopRoute3.35 / 9.2%3.76 / 8.8%
RACER-G4.43 / 9.5%4.82 / 8.8%
GPQA Diamond, 100 modelsCopRoute11.03 / 9.2%11.98 / 8.2%
RACER-G14.10 / 9.9%14.85 / 9.2%
MuSR, 100 modelsCopRoute14.32 / 9.8%15.99 / 9.3%
RACER-G45.61 / 9.6%47.41 / 9.2%

Thus, CopRoute is reasonably tolerant to noise in the labels used to learn its order because an independent final-calibration set can correct the resulting efficiency error. The certificate is sensitive to false positives in the calibration labels themselves, as is any method calibrated against those labels. In practice, annotation quality should therefore be concentrated on the final-calibration set, especially on questions recorded as having at least one correct model. The revised paper will report the complete results by corruption type, corrupted data split, noise rate, method, and model-pool size.

AuthorsOpenReview ↗

Response to Reviewer SHW5 - Part 3

3. How does your method result in inference cost reductions, especially the quoted 46% reduction?

CopRoute selects the model set before generating any answers and calls only those selected models. On MMLU at target all-miss risk 0.10, CopRoute calls 2.99 models per query on average, compared with 5.64 for RACER-G. The reduction therefore comes from avoiding model invocations that add little new coverage.

The monetary cost includes every selected model call and uses the public per-model prices recorded by RouterBench, rather than treating all models as equally expensive. CopRoute costs USD 1.90 per 1,000 queries, compared with USD 3.55 for RACER-G at the same target all-miss risk. Using the unrounded costs, this is a 46.6 percent reduction.

The final-answer procedure does not remove this saving. As described in Table 3 (lines 451 to 455) and Appendix L (lines 1878 to 1885), we return the answer from the called model with the highest router score. This rule reuses scores already computed before model selection, adds 2.3 microseconds per query, and requires no additional model invocation.

Thus, the quoted 46.6 percent is an end-to-end model-inference cost comparison against RACER-G on MMLU at matched certified risk. It is not a comparison against the best single model or an estimate obtained from call counts alone.

Reviewer_3JcHOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CopRoute, a dependence-aware conformal routing method for selecting a small set of language models while controlling the probability that all selected models fail. The key idea is to exploit model complementarity by greedily ordering models according to how many previously uncovered queries they can answer, and then use query-specific stopping together with a conformal calibration step to satisfy a target all-miss risk. Experiments on RouterBench and RouterEval show that CopRoute can substantially reduce the number of model calls compared with existing risk-controlled routing baselines at the same certified risk level.

Critical strengths
  1. The motivation is convincing, and the paper is clearly written and easy to follow.
  2. The core insight is simple and intuitive: model selection should account not only for individual model quality, but also for whether a newly selected model can cover queries missed by the previously selected models. This provides a reasonable and easy-to-understand way to exploit model complementarity.
  3. The experimental results demonstrate the efficiency of the proposed method. Across RouterBench and RouterEval, CopRoute achieves the fewest certified calls on most evaluated datasets, with substantial reductions compared with RACER and other baselines while maintaining the same target all-miss risk.
Critical weaknesses
  1. The main results in Table 1 and Figure 1 report certified mean calls at fixed target risk levels, but do not systematically report the realized test all-miss risk. I suggest reporting the realized risk together with the call count for each method. Ideally, the authors could also provide risk–calls curves, which would make it easier to assess whether the efficiency gains remain consistent across different empirical risk levels.
  2. The theory does not directly establish the main efficiency claim. Theorem 1 guarantees a good ordering, while Theorem 2 guarantees risk control, but neither shows that CopRoute achieves fewer calls than RACER at a fixed target risk. Thus, the call-efficiency advantage is mainly supported empirically. Moreover, Theorem 2 controls the joint event of issuing a certificate and making an all-miss error, rather than the all-miss risk conditional on certification. Thus, the call-efficiency advantage is mainly supported empirically, and the risk guarantee is weaker when certification does not occur almost surely.
Questions for authors

N/A

Reviewer_jSqxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CopRoute, a method for selecting a small set of LLMs while keeping the risk of all models failing low. Unlike existing methods, it accounts for shared failures between models and uses conformal calibration to control risk. It performs better than several baselines on most datasets, using fewer model calls. The approach is practical and well tested, though its contribution is mainly empirical rather than theoretical.

Critical strengths
  • The method is simple and practical. It works with existing routers without retraining, and the experiments show that most of the improvement comes from how models are ordered based on their shared failures.

  • The theoretical guarantee is clearly explained. The authors make it clear that the guarantee applies to the joint event of certification and failure, rather than failure given certification, and that model selection must be fixed before calibration.

  • The evaluation is thorough, covering different model pool sizes, risk levels, and routers. The authors also consider releasing their code (which I believe will be very useful) and openly discuss cases where their method does not improve performance.

Critical weaknesses
  • The paper somewhat overstates its dependence-aware approach, since the main method still assumes independence when deciding when to stop. Dependence is only considered when ordering models, and the theoretical result is based on an existing method that does not directly apply at test time.

  • The experimental improvements are not as convincing as they seem. Some results have overlapping error bars, and certain averages are based on very few successful runs. The paper also lacks a simple fixed-depth baseline for comparison.

  • The practical benefits are not fully clear. On MMLU, the best single model is both cheaper and more accurate, while the reported 46.6% cost reduction comes from just one dataset. The paper also overlooks the cost of collecting calibration labels from every model.

Questions for authors
  • If the stopping rule that actually uses dependence does not help, what makes the default policy “dependence-aware” beyond the static ordering? Can you add a baseline with the tally order and one global depth, and one with the tally order and RACER-style scores, to show what the per-query part adds?

  • One of my main concern is the practical value of the framework. Users care about accuracy per unit cost, not just coverage, and Table 3 shows the best single model beating all certified policies. When is paying for a coverage guarantee worth it?

2 comments
AuthorsOpenReview ↗

Gentle Reminder to Reviewer jSqx

Dear Reviewer jSqx,

Thank you for the constructive reviews. We have shared our responses to your points and updated the draft as well to incorporate these points. If our rebuttal has addressed the concerns of the reviewer, we kindly request them to reconsider their rating. Since the discussion period is ending in a couple of days, if there any further questions or concerns, we remain available to clarify them.

Thanks,

Dear Reviewer jSqx,

Thank you for reading our responses and increasing our score.

Best,

Comments on the paper

2 comments
AuthorsOpenReview ↗

Updated manuscript: summary of all the changes

Thank you to the reviewers and Area Chair for the feedback. We remain available to clarify any further questions as well. We have uploaded a revised manuscript incorporating the analyses from the rebuttal.

Risk and uncertainty. Table 1 now reports observed test all-miss risk and certification counts alongside model calls. Appendix H adds paired bootstrap comparisons and reports both the risk among certified splits and the frequency of the joint certificate-and-miss event where certification is incomplete.

Ordering and sizing. Table 2 and Appendix I.1 compare query-specific sizing with a single global depth and RACER-style scores on the same tally order. They also compare fixed tally with query-local greedy ordering while holding sizing fixed. On RouterEval, query-specific sizing used fewer calls than a single global depth in 138 of 164 comparable settings.

Equal label budgets. Table 27 gives RACER access to the union of the design and router-training labels available to CopRoute. Across four RouterBench datasets and two risk targets, CopRoute used fewer calls than RACER-G in seven of eight comparisons and fewer than RACER-P in all eight.

Costs, answer selection, and label noise. We expanded the cost table to show observed risk and certification counts, compared three answer selectors, and evaluated the cost of calling all models when calibration does not certify. Appendix M separates errors in design labels from errors in final-calibration labels and shows how each affects calls and risk.

We also define the correctness subcopula at first use, list the evaluated RouterBench models, and clarify how the design-set ordering guarantee, the deployment risk certificate, and the empirical call comparisons relate to one another.

Dear Reviewer 3JcH,

As the discussion period ends in a few hours, we would appreciate your acknowledgement of our responses and revised manuscript. Could you please let us know whether this evidence addresses your concerns, and which issues, if any, remain unresolved? If our rebuttal has addressed the concerns, we kindly request to reconsider their rating. We remain available to provide any further clarification

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.