- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed a method to learn an order from joint model-correctness labels, size a set for each query using router scores, and calibrate one cutoff on held-out data. It uses an independence procedure for sizing and a co-failure tally for ordering. The paper applies the minimum-sum set-cover approximation to its design objective and gives a frozen-family split-calibration guarantee. It evaluates certified call counts on RouterBench and RouterEval, with cost, transfer, numerical, and estimator analyses.
Critical strengths
- A simple way to use joint model errors. The work adds co-failure information to a marginal router through a fixed order. The method is easy to understand, and the controls help separate the benefit of the order from that of dependence-corrected stopping.
- Theory with clear targets and assumptions. The paper connects the design ordering to the known minimum-sum set-coverage and gives a split-calibration guarantee for a frozen nested family. The calibration result does not require a correct dependence estimator.
Critical weaknesses
-
Mismatch between theoretical guarantees and deployment-level efficiency. Theorem 1 provides a 4-approximation for the average first-success position on the design set, but this objective does not directly characterize the expected number of certified calls at a fixed risk level. Moreover, Theorem 2 controls the joint certification-and-miss event, whereas the reported call efficiency is evaluated only over certified redraws. Although the paper acknowledges these distinctions, the theoretical results do not yet provide a unified characterization of deployment-level risk and efficiency, particularly when certification is infrequent.
-
Limited exploitation of query-specific dependence. Although CopRoute is presented as a dependence-aware routing method, its default implementation uses a fixed global ordering learned from historical co-failure patterns, while query-adaptive sizing still relies on an independence-based approximation. Moreover, the explicit dependence correction (CopRoute-Dep) does not consistently improve performance, raising questions about the extent to which the method effectively captures conditional dependence.
-
Concerns about baseline fairness and data efficiency. CopRoute uses an additional design split to learn model dependencies, whereas RACER does not fully exploit these labels for router training. Although the shared-router comparison isolates the contribution of the proposed ordering, it does not establish superiority under equal total labeled-data budgets. Furthermore, the evaluation modifies RACER's original abstention mechanism, making the comparison less representative of the complete original framework.
Questions for authors
- Can you align the main risk and efficiency claims with the joint-event guarantee, report the certification denominator, and state the action and cost for non-certification? Please also keep the price and answer-selector conditions next to the cost claim.
- Why does the proposed method rely on a fixed global ordering rather than query-specific dependence-aware selection? Could the authors provide a systematic comparison against query-local greedy routing on the real benchmarks and clarify when global co-failure information is preferable to conditional dependence estimates?
- What is the main theoretical contribution beyond RACER's existing conformal risk-control framework? In addition, does CopRoute retain its efficiency advantage when both methods are given the same total labeled-data budget? A budget-matched comparison would help distinguish improvements attributable to dependence-aware routing from those resulting from additional training information.
Response to Reviewer SHW5 - Part 1
We thank you the reviewer for their constructive feedback. Here we would like to clarify a few questions and concerns raised by them.
1. Explain how your set-based routing methodology can be employed in practice. Let's say your method stops at step k and returns a certificate for a pool of k models. How do we know which of these models will yield the correct answer? In practice, which answer would we pick?
In practice, we call the selected
kmodels in parallel and return the answer from the called model with the highest router score. This is already the deployment rule used in submitted Table 3 (lines 451 to 455) and described again in Appendix L (lines 1878 to 1885). It reuses the router scores computed before model selection, and the submitted appendix measures its overhead at 2.3 microseconds per query.This separation between model selection and answer selection is also used by prior set-based routing methods. RACER selects a model set and then applies either majority voting or weighted aggregation to its outputs (RACER Section 3.3 and Algorithm 1). A certificate over a selected set does not reveal which member is correct under either method. It ensures that the downstream answer selector receives a set that is unlikely to contain only wrong answers.
During the rebuttal, we conducted an additional analysis of three answer-selection rules: highest-router-score selection; plurality voting (return the answer produced by the largest number of called models, breaking ties by their average router score); and router-score-weighted voting (return the answer receiving the largest sum of softmax-normalized router weights). We evaluated CopRoute, RACER-G, and RACER-P over 100 random data splits of all four RouterBench datasets at target all-miss risks 0.05 and 0.10. The table presents the target-risk 0.10 results. All 100 splits produced certificates. “At least one correct” is the percentage of questions for which one or more called models answered correctly.
The highest-router-score rule matches or exceeds both voting rules for CopRoute on all four datasets and is substantially better on MMLU, where it reaches 81.0 percent accuracy compared with 77.0 and 77.1 percent. It also gives CopRoute higher final-answer accuracy than both RACER variants on every dataset shown, while requiring substantially fewer calls on MMLU and WinoGrande.
This also answers the concern that the certificate has no clear practical use. If every called model is wrong, no answer-selection rule can succeed. CopRoute controls this failure before answer selection and reduces the number of candidates that must be considered. The highest-router-score rule then supplies one concrete final answer. When an external check is available, such as unit tests for code, a mathematical verifier, or human review, it can replace the router-score rule and inspect the same smaller candidate set. The revised paper will report both the frequency with which at least one called model is correct and the accuracy of the final selected answer.