acceptodds
Under review as a conference paper at ICLR 2027

OrchSLM: Probing the Dynamics of Small Language Model Orchestration

Abstract

Although large language models (LLMs) have demonstrated remarkable capabilities, their reliance on cloud-scale infrastructure poses fundamental challenges for deployment in agentic pipelines, including latency, privacy, connectivity, and substantial computational cost. Small language models (SLMs) offer a compelling alternative: recent studies suggest that many repetitive and narrowly scoped subtasks in agentic workloads may be better served by specialized SLMs than by monolithic LLMs. However, the limited capacity and context windows of SLMs can constrain long-horizon reasoning and interaction-heavy orchestration strategies such as iterative verification and debate. This motivates a complementary, non-interactive paradigm in which heterogeneous SLMs independently generate candidate solutions and a router orchestrates their cached samples without further model interaction. To further understand the mechanisms of such orchestration, we introduce OrchSLM, a routing framework that unifies existing non-interactive orchestration methods and exposes their underlying design choices as controllable parameters. Using OrchSLM as a systematic probe, we reveal how orchestration behavior emerges from diverse knobs, including the task structure, model-pool composition, and multi-agent consensus.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Small language model, Multi-agent framework

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yUwCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to combine answers from many small LMs when each model samples answers separately and a router picks the final answer from the cached samples (no talking b/w models). Paper proposes OrchSLM, one scoring rule with 3 knobs... k for how much to trust stronger models, Sum vs Max for combining models, and alpha to mix each model's top answer with its full answer distribution. Majority vote, SLM-MUX style self-consistency, weighted voting and single-best come out as special cases. Using 7 open models on MATH-500, GSM8K, GPQA and MMLU-Pro they scan 800 settings and all 127 model subsets, and report that expert weighting matters most, that more models increase oracle coverage but not routed accuracy, and that the best setting depends on the task. They also test transfer of settings b/w similar tasks.

Critical strengths

1- Practical setting... combining cached answers from small models is cheap, and putting majority vote, SLM-MUX, weighted voting and single-best in one formula makes it easy to see they only differ in few choices. The fix/break counts (Table 20) and the Atr vs Ctr split (Tables 30, 31) are more useful than just one accuracy number, e.g. Table 31 shows on GPQA even Qwen's accepted support is more often wrong than right. These are some interesting findings..

2- Authors report a lot of checks (prior transforms, tie-breaks, K=50, sample budget, plateau width). This is good and helps reader to check the claims.

Critical weaknesses

1- Many findings are already known in literature.. so its very hard for me to find the new knowledge after reading the paper. With Sum, Eq. 3 is just soft voting with a small bonus on each model's top answer, weighted by accuracy^k. This is standard weighted voting, and the best weights for independent voters are already known to be log-odds of accuracy (Nitzan & Paroush 1982), not cited. SLM-MUX (Wang et al. 2025, Sec 4.3, 4.4) already says union accuracy grows with more models but contradictions also grow, and GPQA accuracy peaks at 2 models. Large Language Monkeys (Brown et al. 2024) already shows coverage keeps increasing while majority vote stays flat, and Chen et al. 2024 ("Are More LLM Calls All You Need?") already show more calls/models can hurt. Already known findings should be clearly declared (something like " as shown in X paper") so the reader can see what is new. Also the "unifies existing methods" claim is too strong... Please clarify.

2- The best setting is chosen out of 800 settings (and out of 127 pools) on the same test questions, I could not find any held out split. And the improvements over single-best are very small (Table 10)... 0.14% on MATH (less than 1 question of 500), 0.77% on GSM8K, 3.13% on GPQA (around 6 of 198 questions) and 2.44% on MMLU-Pro. Taking the max over 800 similar settings can give this much even if no setting is really better. The std reported is only from cache subsampling, not from questions. Also its not said where acc_i in Eq. 2 comes from... MATH-500 and GPQA dont have a standard validation split, so if it is computed on test questions then single-best and every k> 0 setting use test labels.

Questions for authors
  1. Please clarify all the novelties/new findings compared to these papers - a clear distinguish b/w what was already known and what your paper has.

  2. What is "oracle" exactly in Table 7? If it is any model's top answer being correct, 7 models guessing randomly on 4-option GPQA already cover about 87%, more than the reported 78.5%. Can you report it next to the random guessing level?

  3. How and on which questions is acc_i computed (main results and transfer)? If test labels are used, can you redo it with a separate split?

  4. The full pool uses 7x20 = 140 samples per question, but Qwen with 50 samples gets 79.8% on MATH (Table 26), higher than OrchSLM full pool at K=20 (79.4%). Can you compare with best model self-consistency at same number of samples and one bigger model (14B/32B) at same cost?

Reviewer_hRpxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the decision mechanisms underlying non-interactive orchestration of small language models. It introduces OrchSLM, a framework that unifies representative strategies, including majority voting and self-consistency routing, through three controllable dimensions: expert-weight concentration (κ\kappa), evidence aggregation (Agg\mathrm{Agg}), and the balance between top-answer and pooled support (α\alpha). Across multiple tasks and model subsets, the experiments show that effective routing depends on task characteristics, expert quality, and pool composition. Adding models does not necessarily improve final accuracy, and having a correct candidate answer in the pool does not ensure that the router selects it.

Critical strengths
  • The framework expresses several representative non-interactive aggregation rules through explicit choices of expert weighting, evidence aggregation, and answer support. This provides a controlled way to study which design choices affect routing performance.
  • The experiments distinguish answer availability from routed accuracy, and model agreement from counterfactual contribution. These analyses help explain why adding models may fail to improve performance and why model selection and answer aggregation should be considered jointly.
Critical weaknesses
  • The evaluation does not establish how multi-SLM orchestration compares with spending the same total inference compute on a stronger single model or additional samples from the best SLM. This limits its guidance on practical resource allocation.
  • The transfer experiments still use estimates of model quality on the target task. It remains unclear whether the observed gains persist when these estimates are unavailable or inaccurate.
Questions for authors

Under a matched inference-compute budget, how does OrchSLM compare with allocating more samples to the strongest single SLM or using a larger model with extended reasoning?

3 comments

We thank the reviewer for the constructive review and for recognising that the framework exposes aggregation rules as explicit design choices. Tables G1–G8 are in the General Response.

W1/Q — Matched inference compute. We agree that this is an important practical question. Our answer is mixed rather than uniformly favourable, and we report it as such.

  1. More samples from the strongest SLM. Single-best in the paper is self-consistency of the strongest SLM (Qwen2.5-7B) at the same number of samples per model. OrchSLM is at or above it on all four benchmarks, but significantly so only on MMLU-Pro (Table G3, first column).
  2. A larger model. In Table G6, the best 7–9B pool matches Qwen2.5-32B and exceeds the other larger models on MATH-500. On the knowledge-heavy GPQA and MMLU-Pro, however, it is clearly below Phi-4, Mistral-Small-24B and Qwen2.5-32B, even though one call of these models costs far less than drawing 20 samples from each pool member. Routing cannot supply knowledge that no pool member has, so on knowledge-bound tasks a stronger model is the better use of compute.
  3. Routing is complementary to scale. Applying the same routing family to a pool of 14B/32B-class models gives the best result in Table G6 on every benchmark (e.g., 0.6414 vs. 0.5697 for the best single model on GPQA).

We did not evaluate extended-reasoning variants of larger models within the rebuttal period. The paper does not claim compute efficiency; the revision will add Table G6 and state this limitation explicitly.

W2 — Unavailable or inaccurate quality estimates.

  • Transfer experiments. No target-task quality estimate is used: both the configuration and the expert prior come from the source benchmark. The Figure 8 caption ("target expert prior") is misleading and will be fixed. With source priors, OrchSLM gains +1.1 to +3.2 pp over the strongest fixed baseline; a target-label prior, reported as a sensitivity check, gives +0.7 to +1.8 pp (Table G7). Knowing target-task quality is therefore not needed for the gains.
  • Inaccurate priors on the main benchmarks. We froze the router at the paper's configuration and estimated the prior from only 50 or 100 held-out calibration questions (Table G8). A noisy prior costs up to 4.1 pp on GPQA, but OrchSLM stays above single-best under the same prior on GSM8K, GPQA and MMLU-Pro (+1.1 to +3.0 pp), and within 0.2 pp of it on MATH-500. A prior estimated on few questions can misrank the strongest model; this hurts hard selection of one expert more than soft expert weighting.
  • No prior at all. The uniform-prior (κ=0\kappa=0) results are the equal-weight column of Table 4: they are up to 6 pp below expert weighting, so a reasonable prior helps, but routing remains well defined without one.

Below are all the tables referenced in our response above (Tables G3, G6, G7 and G8).

Table G3. Full pool, OrchSLM minus baseline (pp).

Benchmarkvs Single-bestvs Majorityvs SLM-MUXvs Cross-support
MATH-500+0.14 [+0.00, +0.28]+6.92 [+5.70, +8.12]+6.96 [+5.92, +8.04]+6.36 [+5.30, +7.42]
GSM8K+0.69 [+0.32, +1.05]+0.64 [+0.30, +0.96]+0.59 [+0.27, +0.91]+0.49 [+0.17, +0.81]
GPQA+3.13 [+1.16, +5.05]+4.65 [+2.37, +6.87]+9.60 [+6.62, +12.47]+6.01 [+3.33, +8.59]
MMLU-Pro+2.44 [+1.34, +3.52]+1.20 [+0.22, +2.16]+6.64 [+5.22, +8.10]+2.72 [+1.54, +3.90]

Table G6. Accuracy of OrchSLM vs. larger models.

ModelMATH-500GSM8KGPQAMMLU-Pro
OrchSLM (SLM pool)0.79520.94770.44290.6526
Qwen2.5-14B0.76760.94660.43280.6370
Phi-4 (14B)0.76440.94870.56970.7022
OLMo-2-13B0.42560.88870.27020.3982
Mistral-Small-24B0.68840.94500.47530.6718
Qwen2.5-32B0.79260.95820.46920.6770
OLMo-2-32B0.51900.92210.30250.4436
OrchSLM over 14B/32B-class pool0.83960.97070.64140.7574

Table G7. Transfer accuracy; Δ = gain over the strongest fixed baseline (pp).

Source → TargetSource priorΔTarget-label priorΔ
SocialIQA → CommonsenseQA0.7968+1.060.7935+0.73
LogiQA → ReClor0.7268+2.080.7242+1.82
MedQA-USMLE → MedMCQA0.5986+3.160.5870+1.40

Table G8. Single-best (SB) and OrchSLM with the prior estimated in-sample or on 50/100 held-out calibration questions (router frozen).

BenchmarkIn-sample: SBOrchSLM50 q.: SBOrchSLM100 q.: SBOrchSLM
MATH-5000.79240.79380.79290.79130.79640.7966
GSM8K0.93850.94540.92160.93820.93120.9425
GPQA0.40150.43280.36490.39190.37220.3956
MMLU-Pro0.61580.64020.60490.63490.61460.6386
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you so much for your time and feedback. We have responded to each of your questions and would greatly appreciate it if you could take a look at our rebuttal. We hope our responses have addressed your main concerns.

Best regards,
The Authors

Reviewer_axR8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies non-interactive orchestration of small language models, where heterogeneous models independently generate candidate answers and a router aggregates their cached outputs. The proposed OrchSLM framework organizes routing strategies through an expert-trust concentration parameter κ an answer-support interpolation parameter α, and a Sum/Max aggregation operator. It additionally distinguishes support for the selected answer, measured by Atr, from a model’s leave-one-out accuracy contribution, measured by Ctr. Experiments cover seven models and four primary benchmarks, supplemented by three within-domain cross-task transfer experiments. The main observations are that expert weighting often helps, enlarging the model pool can improve oracle coverage without improving routed accuracy, and support for the final answer does not necessarily imply a positive contribution. The principal contribution is a unified analytical framework and a systematic empirical investigation of how expert reliability, consistency signals, and model-pool composition jointly influence non-interactive model orchestration.

Critical strengths

1.The research question of this work is clear, and the controlled experimental setup is appropriate. Evaluating routers on identical cached generations helps isolate the effect of aggregation rules. Separating expert reliability, within-model consistency, and cross-model support provides a more informative perspective than comparing final accuracy alone.

2.This paper includes substantial ablation and robustness analyses. Beyond the main parameter and model-pool sweeps, the appendix investigates repeated subsampling, prior transformations, tie-breaking, sampling budgets, and fix/break decompositions. The explicit statement that K20R10 standard deviations measure cache-subsampling variability is helpful for interpreting the reported error bars.

3.Several observations are practically useful for model ensembling. Distinguishing answer availability from answer selection helps explain why adding models need not improve performance. The Atr/Ctr comparison also illustrates that agreement with a decision does not establish a positive accuracy contribution. The three within-domain transfer experiments are a useful attempt to examine whether the observations extend beyond the initial benchmarks.

Critical weaknesses

1.The data splits and selection protocol are insufficiently specified. The paper does not clearly explain how expert reliability scores are obtained or whether parameter and model-pool selection are separated from final evaluation. The target-task priors in the transfer experiments raise the same question, making generalization beyond the selection data difficult to assess.

2.The statistical analysis does not clearly address repeated sampling and selection bias. Repeated evaluations of the same question cannot simply be treated as independent questions, yet the significance tests do not specify how this dependence is handled. Correction across multiple baselines also does not automatically account for optimism introduced by searching parameters and model combinations.

3.The claim of exact unification overstates the framework’s coverage. Pooling all sampled answers differs from giving each model one vote for its most frequent answer. Similarly, selecting a fixed best expert does not recover query-dependent routing. The paper should distinguish exact equivalences from conceptual similarities.

4.The stability of some task-level conclusions is insufficiently established. Performance differences between several aggregation strategies are small, and the preferred configuration can change substantially with the sampling budget. These results do not yet establish stable domain-specific preferences and require independent validation or more qualified interpretation.

Questions for authors

1.What are the exact expert-prior values and their provenance? Please distinguish the data used for prior estimation, parameter/pool selection, and final evaluation. How are the target priors in Figure 8 obtained?

2.What is the randomization or resampling unit in Table 21? Can you report paired confidence intervals that preserve all replicates of each question together, and clarify how parameter and pool selection are handled?

3.Does the implemented Majority baseline vote over model-modal answers or pool all sampled answers? Which mappings in Table 2 admit an exact equivalence derivation?

4.Under a fixed selection protocol, which task-level conclusions remain stable on held-out questions and across sampling budgets? In particular, how should the MMLU-Pro configurations in Tables 8 and 26 be reconciled?

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you so much for your time and feedback. We have responded to each of your questions and would greatly appreciate it if you could take a look at our rebuttal. We hope our responses have addressed your main concerns.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.