- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper conprehensively studies most the existing routing methods including the 21 routing methods across five common benchmarks: routerbench, nineby30, embedllm, r2bench and carrot. It finds that the simple KKN can achieve comparable performance compared with some SOTA methods. It further studies if finetuning on a larger dataset can resolve the issue that the classifier struggles on domain-specific queries.
Critical strengths
- The writing of this work is clear and readers can easily understand all the findings.
- This paper offers a systematic empirical study. It considers most SOTA routers and uses five common benchmarks.
- The authors also conduct boostrap comparisons and fine tuning experiments which make the work complete.
Critical weaknesses
- I feel the scope of the final conclusion might be limited. In the new 5k testing dataset, most of them are reading and QA queries, so I feel the plateau finding may not be easily generalized to wider domains.
- I may not understand the oracle definition very well, but I feel there is some gap between oracle and the best single model that we cant close. For example, if both model A and B can solve a question with probability 0.7, then the P(correct)=0.7 whatever the router model is. However, if the oracle knows the result in the hindsight, then its accuracy is 1-(0.3)^2=0.91, which means there are 0.21 gap we can never resolve. I think Appendix B.5 partially addresses this with the repeated sampling, but Table 22 computes the majority-vote oracle while still using the original router accuracy (0.684) as the reference. I feel the router should also be evaluated under the same majority-vote labels. Therefore, I feel the claim that around 93% of the oracle gap remains does not fully rule out generation randomness. This is my major concern. Can you provide a matched evaluation?
Questions for authors
Please see the above weakness section.
2 comments
A quick summary for your final assessment
Dear Reviewer x4YH,
For your final assessment, we summarize your two concerns and our responses.
-
Matched evaluation of router and oracle (your major concern): Your reasoning about oracle inflation is correct, and we tested it on real data, scoring the router and the oracle on the same labels (each model's expected score over 10 samples). The top routers stay at 68.2–68.4%, while the oracle drops from 82.7% to 80.5%. However, the oracle inflation is only 2.1 pp of the 14.3 pp gap, so 85% of the gap remains. Generation randomness is therefore not the main cause of the gap below the oracle; the correctness-prediction bottleneck is, as routers cannot predict from the query whether a given model will succeed.
-
Generalization beyond reading and QA: QA-type questions are very common in routing benchmarks because their answers are easy to verify. To test a wider domain, we ran routers on Chatbot Arena, using real open-ended requests ranging from creative writing to coding, with human preference as the label. Routers trained on 2.6k–82.1k votes plateau again: at every training size, the top five lie within 0.41 pp, and doubling the data from 41.0k to 82.1k votes changes each router by at most 0.6 pp. The plateau, therefore, extends to a broader domain and is not limited to QA.
We hope this summary helps your final assessment.
Remindar to participate in discussion
Thank you again for your review. As the discussion period comes to an end, we wonder whether our responses have addressed your concerns. If any remain, we would be glad to clarify or provide any further results :)