- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper examines how adding two frozen models to a routing system can correct existing mistakes while introducing new ones. It compares choosing one new model globally with choosing separately for each request, and separates changes in which requests use a new model from changes in which model is selected.
Critical strengths
The manuscript reports unfavorable findings, including the failure to consistently reproduce switching-harm reduction and the instability of support-based estimates. It appropriately acknowledges that the analysis does not establish a reliable admission rule.
Critical weaknesses
-
The paper is unacceptably difficult to follow. In the abstract and main text, dense terminology and compressed sentences make it difficult to work out what is being tested, what the results show, and what is new. I found the paper extremely frustrating to read. The abstract contains: "An exact request-wise decomposition attributes the difference to coverage, where one policy returns a candidate, and the other retains the incumbent, and switching, where both return candidates but return different ones". Almost none of the aspects required to understand this sentence are introduced in the abstract, making it impossible for a human reader to even understand what the abstract is trying to say unless they read the entire paper and appendix! In Appendix B, the adoption policies use a "task-profile threshold", but neither the quantity being thresholded nor the complete fitting procedure is clearly defined. This leaves the decisions that determine the main results unclear.
-
The analysis is not an analysis providing new explanations but rather just repeating results. In Sections 3.1-3.3, the propositions mainly regroup observed correctness changes. The interaction being zero wherever the adoption decisions agree follows directly from its definition. Yan et al.'s 2021 work already establishes why newly introduced errors should be reported separately from average accuracy. In Appendix D, the crossed comparison can apply an acceptance decision associated with one candidate while returning another, potentially measuring the damage caused by that mismatch. The same appendix shows essentially no advantage from group-specific gates over a single margin threshold at the reported matched harm level. It remains unclear what additional explanation or practical benefit the decomposition provides.
-
Inconsistent evaluations. In Appendix B, the fixed candidate is chosen using only 32 support examples. In Appendix G, however, the Matrix Z experiment already uses thousands of labeled items to train candidate-specific correctness predictors and tune thresholds. Without a fixed baseline that selects its candidate and threshold using those same available tuning data, the reported advantage may partly reflect errors in selecting the fixed candidate. Appendix G also shows that the "new" candidates have already been extensively evaluated: the 64 evaluations cover only an additional support profile. The experiment therefore does not establish applicability to genuinely unfamiliar candidates.
Questions for authors
-
Exact adoption rule: What is the "task-profile" statistic in Appendix B, and what exact condition causes each policy to replace the incumbent? How are the thresholds fitted, and do they vary by task, candidate, or intervention?
-
Fixed-candidate baseline: Why is the fixed candidate selected using only 32 support examples when trained correctness scores and larger labeled tuning splits are already available? Does an existing comparison select the fixed candidate and its threshold using those same tuning data? This would help distinguish the value of per-request selection from errors in choosing the fixed candidate.
-
Meaning of the interaction: The crossed policy can accept a request because one candidate appears suitable, then answer it using another candidate. Which reported result distinguishes the proposed explanation from the damage caused by this mismatch? This would clarify what the interaction establishes beyond the behavior of these particular policy combinations.