acceptodds
Withdrawn from ICLR 2027

Where Does Harmful Replacement Come From? Coverage and Switching in Frozen Model Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adding a frozen expert gives a router another possible answer, but also another way to replace a correct incumbent answer with an incorrect one. We introduce a diagnostic protocol for this trade-off when two new candidates are available. Holding a fixed support budget of 64 candidate evaluations per intervention, a pair-fixed policy uses one candidate on every eligible request, whereas a per-request policy chooses between the two candidates separately for each request. Across two development assets, per-request choice increases net accuracy gain relative to the incumbent by 0.182 percentage points (pp) compared with pair-fixed choice, while increasing harmful-replacement mass by 0.201 pp. An exact request-wise decomposition attributes the difference to coverage, where one policy returns a candidate and the other retains the incumbent, and switching, where both return candidates but return different ones. A controlled experiment shows that these decisions interact: expanding candidate use can help or hurt depending on which candidate is returned on the added requests. A held-out text-model asset and an independently constructed score reproduce the interaction and the sign reversal of the coverage effect, whereas switching-harm reduction is not stable across assets or score constructions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model routing, frozen model routing, model onboarding, coverage-switch decomposition, request-level routing, risk control, selective prediction, evaluation protocol

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_a3JdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines how adding two frozen models to a routing system can correct existing mistakes while introducing new ones. It compares choosing one new model globally with choosing separately for each request, and separates changes in which requests use a new model from changes in which model is selected.

Critical strengths

The manuscript reports unfavorable findings, including the failure to consistently reproduce switching-harm reduction and the instability of support-based estimates. It appropriately acknowledges that the analysis does not establish a reliable admission rule.

Critical weaknesses
  • The paper is unacceptably difficult to follow. In the abstract and main text, dense terminology and compressed sentences make it difficult to work out what is being tested, what the results show, and what is new. I found the paper extremely frustrating to read. The abstract contains: "An exact request-wise decomposition attributes the difference to coverage, where one policy returns a candidate, and the other retains the incumbent, and switching, where both return candidates but return different ones". Almost none of the aspects required to understand this sentence are introduced in the abstract, making it impossible for a human reader to even understand what the abstract is trying to say unless they read the entire paper and appendix! In Appendix B, the adoption policies use a "task-profile threshold", but neither the quantity being thresholded nor the complete fitting procedure is clearly defined. This leaves the decisions that determine the main results unclear.

  • The analysis is not an analysis providing new explanations but rather just repeating results. In Sections 3.1-3.3, the propositions mainly regroup observed correctness changes. The interaction being zero wherever the adoption decisions agree follows directly from its definition. Yan et al.'s 2021 work already establishes why newly introduced errors should be reported separately from average accuracy. In Appendix D, the crossed comparison can apply an acceptance decision associated with one candidate while returning another, potentially measuring the damage caused by that mismatch. The same appendix shows essentially no advantage from group-specific gates over a single margin threshold at the reported matched harm level. It remains unclear what additional explanation or practical benefit the decomposition provides.

  • Inconsistent evaluations. In Appendix B, the fixed candidate is chosen using only 32 support examples. In Appendix G, however, the Matrix Z experiment already uses thousands of labeled items to train candidate-specific correctness predictors and tune thresholds. Without a fixed baseline that selects its candidate and threshold using those same available tuning data, the reported advantage may partly reflect errors in selecting the fixed candidate. Appendix G also shows that the "new" candidates have already been extensively evaluated: the 64 evaluations cover only an additional support profile. The experiment therefore does not establish applicability to genuinely unfamiliar candidates.

Questions for authors
  1. Exact adoption rule: What is the "task-profile" statistic in Appendix B, and what exact condition causes each policy to replace the incumbent? How are the thresholds fitted, and do they vary by task, candidate, or intervention?

  2. Fixed-candidate baseline: Why is the fixed candidate selected using only 32 support examples when trained correctness scores and larger labeled tuning splits are already available? Does an existing comparison select the fixed candidate and its threshold using those same tuning data? This would help distinguish the value of per-request selection from errors in choosing the fixed candidate.

  3. Meaning of the interaction: The crossed policy can accept a request because one candidate appears suitable, then answer it using another candidate. Which reported result distinguishes the proposed explanation from the damage caused by this mismatch? This would clarify what the interaction establishes beyond the behavior of these particular policy combinations.

Reviewer_7iKZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a frozen-pool model-routing onboarding scenario in which two new candidates are added to an existing router. A pair-fixed policy commits to one candidate for the whole pair, while a per-request policy chooses between the two candidates independently at every request. Under a fixed support budget of 64 candidate evaluations per intervention, the authors propose an exact request-wise decomposition of the policy difference into a coverage event (one policy returns a candidate, the other retains the incumbent) and a switching event (both return candidates but different ones). On two development assets S/U, per-request improves net accuracy by +0.182 pp but also increases harmful-replacement mass by +0.201 pp. A 2x2 controlled experiment shows adoption and identity interact; a held-out asset Z and an independently constructed score R1 reproduce the interaction and the coverage sign reversal, but switching-harm reduction is not stable (its interval covers zero on U and Z).

Critical strengths

The problem framing (separating adoption from identity in onboarding decisions) is genuinely useful for routing practitioners. The experimental protocol is careful: frozen cached outputs, disjoint support/tuning/evaluation splits, cluster-aware bootstrap, and a pre-registered held-out asset Z whose protocol was fixed before outcomes were inspected. The authors are honest about the boundary of their claims, explicitly stating the protocol does not turn the observed interaction into a generally safe deployment rule.

Critical weaknesses
  1. The methodological contribution is very thin. Proposition 1 partitions the sample space into the disjoint events E_cov and E_sw and sums indicator-weighted differences; Proposition 2 expands a 2x2 interaction contrast into scale-times-selection-advantage terms. Both are standard accounting identities - the paper itself calls Proposition 1 "an accounting identity." As headline technical contributions for ICLR, they do not clear the bar.

  2. Effect sizes are minuscule and half of the findings do not replicate. The main numbers are +0.182 pp net gain against +0.201 pp additional harmful replacement, both at the two-per-thousand level. Switching-harm reduction is supported only on asset S; on U and Z the interval covers zero. The paper's own conclusion acknowledges the protocol does not justify a deployment rule, leaving the practical value of the diagnostic unclear.

  3. External validity is severely limited and the writing is over-notated. All experiments run on three cached correctness matrices (VL-RouterBench, a VLM exec-routerbench, and one held-out text asset); there are no new prompts, no new model families, and no live routing experiments - the authors state that neither the perturbation ladder nor R1 "projects onto new prompts, new pairs or new model families." Meanwhile the main text introduces a dense vocabulary (pair-fixed / per-request / coverage / switching / ledger / C_add / C_drop / I_G / I_H / mu_add / mu_drop / R0 / R1 / S / U / Z) that obscures rather than clarifies the argument.

Questions for authors
  1. What is the technical novelty of Propositions 1 and 2 relative to standard 2x2 interaction decomposition and negative-flip accounting (Yan et al. 2021)?

  2. Given that switching-harm reduction fails to replicate on U and Z, how should we interpret the diagnostic value of the coverage/switching separation - does the instability undermine the main claim?

  3. Can you provide any validation beyond cached correctness matrices, e.g., a new model family or a live routing setting?

Reviewer_SWn5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares two ways of adding two new models to an existing routing system. One policy chooses a single new model in advance; the other chooses between them for each request. Both can also keep the original system's answer. The authors vary which requests receive a replacement and which model provides it. Changing which requests receive a replacement lowers accuracy with a fixed candidate model, but improves it when the candidate is chosen for each request.

Critical strengths

The comparison in Section 3.2 is useful. Changing which requests receive a replacement lowers accuracy substantially when the replacement model is fixed, but raises it when the model is chosen for each request.

Critical weaknesses
  1. The main weakness, in my opinion, is the limited new insight. Deciding whether to use another model and which model to use is already explicit in Verma et al. (AISTATS 2023). Proposition 1 considers two types of change. Either only one policy replaces the original answer, or both replace it but choose different models. Adding the accuracy changes in these two groups gives the total change, since all other answers are unchanged. This is useful for reporting results, but I do not see a substantial theoretical contribution here.

  2. The empirical result does not provide enough new understanding to justify the paper. Even with random routing scores, changing which requests receive a replacement has different effects under the two model-choice rules. This alone therefore does not show that the router makes useful choices for individual requests. The authors' separate shuffle test provides some evidence for useful choices. It randomly reassigns requests between candidates while keeping the number assigned to each candidate fixed. The authors report that the original assignment marginally improves accuracy over these shuffled assignments. The proposed explanation of differences between model pairs also has an error. The authors claim that two contributions usually cancel, but their count includes cases where one or both contributions are zero. The claimed majority disappears when these cases are counted correctly.

Questions for authors
  1. What do the experiments establish beyond showing that the effect of replacing an answer depends on which model provides the replacement? What additional conclusion is supported by the learned scores compared with the random-score control?

  2. Please position the contribution against these relevant references:

  3. Please correct the cancellation claim in Section 3 and Figure 2. The two contributions in Proposition 2 come from requests newly assigned to a candidate and requests returned to the original system. For them to cancel, both must be nonzero and have opposite signs. Using the authors' supplied data and counting, the results are 48/128, 16/47 and 144/1723 such model pairs on datasets S, U and Z. None is a majority. Please also provide `SCORE_ASSET_AUDIT.json which was used to generate the original R0 scores and the exact versions of the source datasets for reproduction purposes.

  4. The references contain hallucinated author details. The MMLU entry inserts Kadavath, Arora and Tang and omits Zou and Mazeika. The third author of Model soups is Samir Ya Gadre, not “Shuang Gadre.”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.