acceptodds
Under review as a conference paper at ICLR 2027

Inference-Time Model Selection via Capability Synthesis

Abstract

We consider the problem of inference-time model selection in language models, where a selector has access to a finite pool of models and aims to select the model that performs best on a given task. In this setup, querying every model on each task is computationally expensive; therefore, the goal is to design contextual model selection strategies that can predict the best-performing model for each context. We propose a simple framework named *Capability Synthesis* that discovers directions along which the models' capabilities are most distinguishable. Capability Synthesis learns to jointly represent models and tasks in a shared capability space, and uses these representations to predict the performance of each model on any given task at inference time. This framework is particularly suited for model selection over large pools of models, as larger pools provide more observations for learning the capability structure, while the cost of the selection strategy remains negligible at inference time. Finally, our experiments on multiple routing benchmarks show that this approach, which admits closed-form solutions, competes with more complex trainable routers and outperforms single-model deployments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Selection, Routing, Large Language Models, Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sQQiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Capability Synthesis, a framework for inference-time LLM routing. It uses PCA to learn shared capability representations of models and tasks, followed by ridge regression to predict task representations for unseen queries. Experiments on two routing benchmarks demonstrate competitive performance.

Critical strengths
  1. The proposed method is straightforward, combining PCA and ridge regression while achieving competitive performance against existing routing methods.
  2. The method achieves competitive results against existing routing approaches, particularly when the model pool is large.
Critical weaknesses
  1. This method primarily combines standard PCA and ridge regression. Given that prior work has already explored shared model-task representations and low-rank factorization for LLM routing, the novelty appears limited. In addition, the performance gains are relatively small on EmbedLLM.
  2. The experiments are insufficient. More ablation studies should be included, such as the impact of PCA rank, regression design, and so on. Besides, the claimed efficiency is not supported by quantitative experiments.
  3. For presentation, the related work section could better clarify the differences from existing approaches. Moreover, the experimental figure/tables could also be improved by highlighting the proposed method and including key numerical results in the main text. This can make the paper more intuitive and easier to follow.
Questions for authors

None

1 comment

We thank the reviewer for their feedback. Below, we provide a detailed response to your concerns and the changes we made to improve the paper.

Weakness 1: Prior work on shared model task representations, particularly EmbedLLM, trains an encoder-decoder using gradient-descent-based optimizers to learn capability profiles and task representations, and learns a separate classification head for model selection. In Capability Synthesis, we directly derive models capability profiles from PCA, and learn task specifications using closed-form ridge regression. Therefore, capability synthesis is a simpler, training-free alternative that is scalable to larger model pools. Additionally, the EmbedLLM paper mentions model pool extension as one of the limitations of their method, since adding a new model requires training the router from scratch. As discussed in Section 3.3, Capability Synthesis has a pool extension mechanism that doesn't require refitting as long as the newly added model can addequately be represented in the existing capability space.

For efficiency, we added a quantitative comparison of the number of fitted router parameters between Capability Synthesis and EmbedLLM.

BenchmarkMCapability SynthesisEmbedLLMRatio (EmbedLLM / CS)
EmbedLLM53,092180,03458.2
107,002181,19425.9
2519,032184,6749.7
5040,082190,4744.8
11297,680204,8582.1
RouterEval (Averaged)52,018180,03489.2
103,242181,19455.9
10031,682202,0746.4
1000324,133410,8741.3

The results indicate that Capability Synthesis has significantly fewer fitted parameters across different settings, while matching or outperforming EmbedLLM on these benchmarks. More details on these results are available in Appendix A5, Table 7.

Weakness 2: We've added both theoretical and empirical results to clarify the role of PCA in Capability Synthesis and the connection to direct ridge regression. In Theorem 1-Appendix A1, we showed that Capability Synthesis recovers direct ridge regression at full rank; therefore, the solution of direct ridge regression is always within the candidate solutions of Capability Synthesis. To investigate the role of PCA, we added experimental results on a pool of eight models and conducted an ablation on three specialized domains (Coding, Mathematical Reasoning, and Scientific Reasoning), as well as their combined dataset. In Section 4.2 of the paper, we provide a side-by-side comparison of direct ridge regression and Capability Synthesis at different ranks for these experiments and study the effect of truncation on downstream model selection performance.

Additionally, we extended the rank ablation and comparison to direct ridge to all individual datasets of RouterEval and EmbeddLLM, available in Appendix A4, Figure 5-6. We've included both settings where truncation improves performance over direct ridge and settings such as the combined dataset of EmbedLLM, where capability synthesis achieves its best performance by retaining all principal components, recovering the solution of direct ridge regression.

Weakness 3: Following your feedback, we improved the related work section to better distinguish Capability Synthesis from existing routers, particularly EmbeddLLM. We also improved the presentation of experiments by reporting detailed numerical results with Bootstrap intervals across different pool compositions (Table 1), highlighting the key results in the main text, and adding plots for regret curves of baselines on routing benchmarks (Figure 4) and the shared capability space across different ranks (Figure 1).


We once again thank the reviewer for their feedback. Please let us know if you have any follow-up questions or concerns that we can answer.

Reviewer_oRvCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies inference-time LLM routing and proposes a framework called Capability Synthesis. The method applies PCA/SVD to a model–task performance matrix to obtain low-dimensional representations of models and tasks in a shared capability space. It then uses Ridge Regression to map new queries into this space, predict the relative performance of candidate models, and select the most suitable one. The authors also introduce a low-cost model pool extension mechanism and analyze the performance gap between routing and oracle selection. Experiments on EmbedLLM and RouterEval demonstrate competitive performance, particularly with larger model pools.

Critical strengths
  1. Simple and efficient approach. The method uses standard PCA/SVD and Ridge Regression with closed-form solutions, avoiding gradient-based training and offering a lightweight alternative to existing routing methods.

  2. Competitive empirical performance. The method achieves competitive results on two established routing benchmarks, with more noticeable improvements on RouterEval, particularly for larger model pools.

Critical weaknesses
  1. Limited novelty and insufficient motivation. This paper builds on the existing LLM routing framework, mainly replacing the trainable low-rank representation learning with PCA/SVD and Ridge Regression. The methodological modification is relatively simple, and the Introduction does not sufficiently explain the limitations of existing methods or justify the necessity of the proposed design.

  2. Insufficient presentation of the main experimental results. The paper mainly reports highly aggregated average results, without detailed results across different tasks, model pool compositions, and experimental settings. This makes it difficult to assess the consistency and general effectiveness of the method.

  3. Lack of necessary method analysis and ablation studies. The paper lacks simple baselines such as Direct Ridge Regression, as well as analyses of PCA/SVD, low-rank choices, and their relationship with routing performance. The necessity of the core design choices remains insufficiently validated.

Questions for authors
  1. Completeness of Experimental Results. I would appreciate more detailed experimental results across different tasks, model pool compositions, and pool sizes, rather than only aggregated averages, to better assess the consistency and general effectiveness of the method.

  2. Performance Differences Across Benchmarks. Why does the proposed method show only marginal improvements over EmbedLLM MF on the EmbedLLM benchmark, while achieving more noticeable gains on RouterEval? It would be helpful to further analyze the reasons behind this discrepancy.

  3. Method Analysis and Ablation Studies. I would appreciate additional comparisons with simple baselines, such as Direct Ridge Regression, as well as analyses of PCA/SVD, low-rank choices, reconstruction quality, and their relationship with routing performance to better justify the core design choices.

  4. Motivation and Method Design. What specific limitations of existing low-rank representation learning methods, such as EmbedLLM, does this work aim to address? Why is PCA/SVD + Ridge Regression a more appropriate choice rather than simply a simpler alternative? Further clarification of the motivation and design rationale would be helpful.

1 comment

We thank the reviewer for providing a detailed review and helpful feedback. Below, we address the questions and weaknesses that you pointed out in your review and the improvements we made to improve the paper accordingly.

Novelty, Motivation, and Method Design: Our motivation was to design a simple, scalable router that is practical to deploy and maintain at inference, and can compete with more complex routing algorithms and architectures. In comparison to prior works, particularly the EmbedLLM router, our method directly derives the model's capability profiles from PCA and learns task specifications using Ridge Regression. EmbedLLM trains an encoder-decoder using a gradient-descent optimizer and learns a classification head for model selection. Therfore, our method is a simpler gradient-free alternative, with a more straightforward training procedure, and a fraction fitted parameters. We provide a side-by-side comparison of the number of fitted parameters for both Capability Synthesis and EmbedLLM, which shows that Capability Synthesis has notably fewer parameters than EmbedLLM while matching and outperforming EmbedLLM across different pool sizes and benchmarks. More details on these results are available in Table 7 of Appendix A5.

BenchmarkMCapability SynthesisEmbedLLMRatio (EmbedLLM / CS)
EmbedLLM53,092180,03458.2
107,002181,19425.9
2519,032184,6749.7
5040,082190,4744.8
11297,680204,8582.1
RouterEval (Averaged)52,018180,03489.2
103,242181,19455.9
10031,682202,0746.4
1000324,133410,8741.3

Additionally, EmbedLLM mentions model pool extension as one of the limitations of their method, since adding a new model requires training the router from scratch. As discussed in Section 3.3, Capability Synthesis has a pool extension mechanism that doesn't require refitting as long as the newly added model can addequately be represented in the existing capability space.

Completeness of Experiment Results and Ablations: Following your feedback, We've improved our experimental results in the following ways:

  • The role of PCA and Rank Ablations: We added an eight-model experiment on three specialized domains (coding, mathematical reasoning, and scientific reasoning), as well as their combined dataset. We study the effect of rank truncation and compare Capability Synthesis with direct ridge regression, and the results and discussions are available in Section 4.2 of our revised manuscript. Additionally, in theorem 1-Appendix A1, we formally show that full-rank capability synthesis recovers the solution of direct ridge regression. Therefore the solution of direct ridge is always withing the candidate solutions of Capability Synthesis and the role of PCA is mainly spectral regularization. We extended the rank ablation to individual datasets of EmbedLLM and RouterEval in Appendix A4 (Figures 5 and 6).
  • Different Model Pool Compositions: We report the performance of Capability Synthesis and other baselines on different model pool compositions of RouterEval including All-weak, All-strong, and Strong-to-Weak model pools. The results are summarized in Table 1, Section 4, of the paper, where Capability Synthesis can outperform other baselines in eight out of twelve settings and leads across all three pool compositions at the largest pool. To answer your second question on performance differences across benchmarks, we summarize the results from Table 1 below, and show that model pool compositions is one of the main factors that is contributing to the larger gain on RouterEval versus EmbedLLM. We report the test pass@1 of Capability Synthesis and the gain over best single model, EmbedLLm and Avengers at pool of size M=1000M=1000.
Pool CompostionCSCS - Best Single ModelCS - EmbedLLMCS - Avengers
All Strong78.1+2.0+2.4+2.2
All Weak45.0+8.5+6.4+4.9
Strong-to-Weak77.8+1.7+6.3+1.0
Averaged66.9+4.0+5.0+ 2.6

The results indicate that the larger gain of Capability Synthesis in RouterEval mainly comes from all-weak pools, and partly Strong-to-Weak pools.


We once again thank the reviewer for their constructive feedback. Please let us know if there are any follow-up comments and questions.

Reviewer_7W8COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates inference-time model selection (routing) among M candidate LLMs to optimize accuracy. The proposed method utilizes matrix decomposition to extract model and query features, and employs ridge regression to integrate new LLMs. Experimental results on the EmbedLLM and RouterEval benchmarks were provided.

Critical strengths

The proposed method is training-free and demonstrates empirical scalability to large model pools. This characteristic facilitates its seamless application to newly introduced LLMs.

Critical weaknesses
  1. Limited novelty: The proposed method theoretically collapses into linear operations on the LM and query embeddings. Furthermore, the mathematical derivations are largely trivial, and the empirical results show a significant performance gap between the proposed method and the oracle.
  2. Missing ablation studies: Ridge regression serves as a crucial baseline for comparison, yet it is not explored.
Questions for authors

What are the specific advantages of the proposed method compared to simple ridge regression? Please provide a theoretical analysis and corresponding ablation studies to clarify this.

1 comment

We thank the reviewer for their comments. Below, we address the concerns and highlight the revisions we made to improve the paper accordingly.

  • Novelty: Our proposed method consists of two well-studied statistical procedures, PCA and Ridge Regression. Our main contribution is obtaining a simple gradient-free router that is scalable to large model pools and showing that it can compete with or outperform more complex routing algorithms and architectures.
  • The Gap to oracle: As explained in the paper, the oracle in the plots is the realized reward oracle, which observes the query and every model's verified generation before model selection. In Section 5, we formally distinguished this oracle from the expected reward oracle and the model selection comparator. We explained why part of this gap is the gain of verification at inference, and is inaccessible to routers that only observe the query before selection. Our method and all baselines share this gap, as they do not observe the realized rewards at inference.
  • Ablation and Comparison to Direct Ridge Regression: In our revised manuscript, we've provided a theoretical analysis to establish the connection to direct ridge regression, and conducted an extensive rank ablation. Theorem 1 (Appendix A1) shows that Capability Synthesis recovers direct ridge regression at full rank; therefore, the solution of direct ridge regression is always within the candidate solutions of Capability Synthesis. We added new experiments on three domains (coding, mathematical reasoning, scientific reasoning), and studied the effect of rank on the performance of Capability Synthesis, and provided a side-by-side comparison to direct ridge regression. The results are available in Section 4.2 of the paper. We've also extended the ablation and comparison to direct ridge to individual datasets of EmbedLLM, and RouterEval benchmark, available in Figures 5 and 6 of Appendix A4. We've included both settings where truncation improves performance over direct ridge and settings where capability synthesis achieves its best performance by retaining all principal components, recovering the solution of direct ridge regression.

We once again thank the reviewer for their feedback. Please let us know if there are any other questions or concerns that we can address.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.