- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies support design for LLM router construction under a fixed annotation budget, comparing models scored on independently sampled requests with models scored on shared requests. It derives a coupling identity explaining how shared supports can reduce gain-prediction error and evaluates routing quality at matched switching rates.
Critical strengths
- The paper addresses a practical and useful question in router construction. It shows that support design can affect routing quality at equal annotation cost. Shared supports offer a simple way to improve routing without changing the predictor family or purchasing additional labels.
- The theory and experiments jointly support the benefits of shared supports. The analysis explains how sharing can reduce gain-prediction error, and the experiments measure both this error and routing quality at matched switching rates. This makes the evidence more convincing than reporting downstream performance gains alone.
Critical weaknesses
- All six archives were accessed during method development, and exploratory experiments on the sixth archive preceded the frozen-protocol replication. The paper therefore lacks a confirmatory evaluation on an archive entirely untouched during method development.
- The theory and main experiments assume a separately fitted predictor for each model. With a joint predictor that can learn from labels across models, broader request coverage may offer additional benefits, but the paper does not test whether shared supports remain preferable in this setting. The recommendation of full sharing as a default therefore lacks sufficient support beyond separately fitted predictors.
Questions for authors
- Do you have results comparing shared and independent supports with a joint predictor that learns across models? If not, should the recommendation of full sharing as a default be limited to separately fitted predictors?
- What did the four exploratory runs on the sixth archive examine, and did their results influence the method or the frozen evaluation protocol?
3 comments
The response addresses some of my concerns, they do not change my overall assessment. I am keeping my original rating.
Thank you for reviewing our paper and for following up. We’re pleased that our response helped address some of the concerns you raised.
Response to Reviewer BFB6 [Comment 1/1]
We thank the reviewer for the clear assessment and the two concrete questions.
Joint predictors and the scope of the recommendation (W2, Q1). The submitted version noted that overlap leaves each separate fit's marginal distribution unchanged, whereas pooling labels across models introduces a trade-off between pairing and coverage (Remark 1; Appendix C). The abstract and conclusion did not state this scope clearly enough. The revision restricts the recommendation to separately fitted predictors and develops the joint-learning case through an explicit decomposition and a new proposition. Eq. (6) in Section 4 shows that, under joint learning, sharing must win the comparison between the paired cross term and the single-model error that broader coverage can lower. For joint residual smoothers, Prop. 4 shows that learning the common difficulty function can remove its contribution to gain error under independent supports. With a fixed estimate of the common function and the stated noise assumptions, pairing retains a noise-variance advantage. The revised manuscript distinguishes this theoretical result from the empirical comparisons of joint learners fitted to the purchased labels.
We examine this trade-off using three learner settings (Section 6.9; Appendices P, Q.4).
The covariance that governs the separate-predictor case is positive in 97.7% of 216 challenger cases across the five archives, and significantly negative in none (Section 6.4, Appendix O). CASA's pilot allocation tracks it (Spearman 0.85).
The sixth archive and the four exploratory runs (W1, Q2). Appendix K now describes the four runs. They came from an earlier phase of the project and tested other allocation strategies: a cardinality-adaptive rule, an external comparison and two hierarchical variants. Their design differs from the replication throughout:
None compared shared with independent supports or evaluated CASA. The replication protocol fixed the pool (the alphabetically first five of twelve models), incumbent rule, CASA's original parameters, predictors, grids, budgets, seeds, metric and decision rules, and was hashed before the replication opened any score. No element was taken from the earlier runs. Because the same score matrix had been accessed before, we describe the result as an external replication under a frozen protocol, not a confirmatory test on a never-opened archive. Sections 5 and 6.7 state the corresponding status of all real-archive results.