- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how to combine answers from many small LMs when each model samples answers separately and a router picks the final answer from the cached samples (no talking b/w models). Paper proposes OrchSLM, one scoring rule with 3 knobs... k for how much to trust stronger models, Sum vs Max for combining models, and alpha to mix each model's top answer with its full answer distribution. Majority vote, SLM-MUX style self-consistency, weighted voting and single-best come out as special cases. Using 7 open models on MATH-500, GSM8K, GPQA and MMLU-Pro they scan 800 settings and all 127 model subsets, and report that expert weighting matters most, that more models increase oracle coverage but not routed accuracy, and that the best setting depends on the task. They also test transfer of settings b/w similar tasks.
Critical strengths
1- Practical setting... combining cached answers from small models is cheap, and putting majority vote, SLM-MUX, weighted voting and single-best in one formula makes it easy to see they only differ in few choices. The fix/break counts (Table 20) and the Atr vs Ctr split (Tables 30, 31) are more useful than just one accuracy number, e.g. Table 31 shows on GPQA even Qwen's accepted support is more often wrong than right. These are some interesting findings..
2- Authors report a lot of checks (prior transforms, tie-breaks, K=50, sample budget, plateau width). This is good and helps reader to check the claims.
Critical weaknesses
1- Many findings are already known in literature.. so its very hard for me to find the new knowledge after reading the paper. With Sum, Eq. 3 is just soft voting with a small bonus on each model's top answer, weighted by accuracy^k. This is standard weighted voting, and the best weights for independent voters are already known to be log-odds of accuracy (Nitzan & Paroush 1982), not cited. SLM-MUX (Wang et al. 2025, Sec 4.3, 4.4) already says union accuracy grows with more models but contradictions also grow, and GPQA accuracy peaks at 2 models. Large Language Monkeys (Brown et al. 2024) already shows coverage keeps increasing while majority vote stays flat, and Chen et al. 2024 ("Are More LLM Calls All You Need?") already show more calls/models can hurt. Already known findings should be clearly declared (something like " as shown in X paper") so the reader can see what is new. Also the "unifies existing methods" claim is too strong... Please clarify.
2- The best setting is chosen out of 800 settings (and out of 127 pools) on the same test questions, I could not find any held out split. And the improvements over single-best are very small (Table 10)... 0.14% on MATH (less than 1 question of 500), 0.77% on GSM8K, 3.13% on GPQA (around 6 of 198 questions) and 2.44% on MMLU-Pro. Taking the max over 800 similar settings can give this much even if no setting is really better. The std reported is only from cache subsampling, not from questions. Also its not said where acc_i in Eq. 2 comes from... MATH-500 and GPQA dont have a standard validation split, so if it is computed on test questions then single-best and every k> 0 setting use test labels.
Questions for authors
-
Please clarify all the novelties/new findings compared to these papers - a clear distinguish b/w what was already known and what your paper has.
-
What is "oracle" exactly in Table 7? If it is any model's top answer being correct, 7 models guessing randomly on 4-option GPQA already cover about 87%, more than the reported 78.5%. Can you report it next to the random guessing level?
-
How and on which questions is acc_i computed (main results and transfer)? If test labels are used, can you redo it with a separate split?
-
The full pool uses 7x20 = 140 samples per question, but Qwen with 50 samples gets 79.8% on MATH (Table 26), higher than OrchSLM full pool at K=20 (79.4%). Can you compare with best model self-consistency at same number of samples and one bigger model (14B/32B) at same cost?
We thank the reviewer for the constructive review and for recognising that the framework exposes aggregation rules as explicit design choices. Tables G1–G8 are in the General Response.
W1/Q — Matched inference compute. We agree that this is an important practical question. Our answer is mixed rather than uniformly favourable, and we report it as such.
We did not evaluate extended-reasoning variants of larger models within the rebuttal period. The paper does not claim compute efficiency; the revision will add Table G6 and state this limitation explicitly.
W2 — Unavailable or inaccurate quality estimates.