- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Capability Synthesis, a framework for inference-time LLM routing. It uses PCA to learn shared capability representations of models and tasks, followed by ridge regression to predict task representations for unseen queries. Experiments on two routing benchmarks demonstrate competitive performance.
Critical strengths
- The proposed method is straightforward, combining PCA and ridge regression while achieving competitive performance against existing routing methods.
- The method achieves competitive results against existing routing approaches, particularly when the model pool is large.
Critical weaknesses
- This method primarily combines standard PCA and ridge regression. Given that prior work has already explored shared model-task representations and low-rank factorization for LLM routing, the novelty appears limited. In addition, the performance gains are relatively small on EmbedLLM.
- The experiments are insufficient. More ablation studies should be included, such as the impact of PCA rank, regression design, and so on. Besides, the claimed efficiency is not supported by quantitative experiments.
- For presentation, the related work section could better clarify the differences from existing approaches. Moreover, the experimental figure/tables could also be improved by highlighting the proposed method and including key numerical results in the main text. This can make the paper more intuitive and easier to follow.
Questions for authors
None
We thank the reviewer for their feedback. Below, we provide a detailed response to your concerns and the changes we made to improve the paper.
Weakness 1: Prior work on shared model task representations, particularly EmbedLLM, trains an encoder-decoder using gradient-descent-based optimizers to learn capability profiles and task representations, and learns a separate classification head for model selection. In Capability Synthesis, we directly derive models capability profiles from PCA, and learn task specifications using closed-form ridge regression. Therefore, capability synthesis is a simpler, training-free alternative that is scalable to larger model pools. Additionally, the EmbedLLM paper mentions model pool extension as one of the limitations of their method, since adding a new model requires training the router from scratch. As discussed in Section 3.3, Capability Synthesis has a pool extension mechanism that doesn't require refitting as long as the newly added model can addequately be represented in the existing capability space.
For efficiency, we added a quantitative comparison of the number of fitted router parameters between Capability Synthesis and EmbedLLM.
The results indicate that Capability Synthesis has significantly fewer fitted parameters across different settings, while matching or outperforming EmbedLLM on these benchmarks. More details on these results are available in Appendix A5, Table 7.
Weakness 2: We've added both theoretical and empirical results to clarify the role of PCA in Capability Synthesis and the connection to direct ridge regression. In Theorem 1-Appendix A1, we showed that Capability Synthesis recovers direct ridge regression at full rank; therefore, the solution of direct ridge regression is always within the candidate solutions of Capability Synthesis. To investigate the role of PCA, we added experimental results on a pool of eight models and conducted an ablation on three specialized domains (Coding, Mathematical Reasoning, and Scientific Reasoning), as well as their combined dataset. In Section 4.2 of the paper, we provide a side-by-side comparison of direct ridge regression and Capability Synthesis at different ranks for these experiments and study the effect of truncation on downstream model selection performance.
Additionally, we extended the rank ablation and comparison to direct ridge to all individual datasets of RouterEval and EmbeddLLM, available in Appendix A4, Figure 5-6. We've included both settings where truncation improves performance over direct ridge and settings such as the combined dataset of EmbedLLM, where capability synthesis achieves its best performance by retaining all principal components, recovering the solution of direct ridge regression.
Weakness 3: Following your feedback, we improved the related work section to better distinguish Capability Synthesis from existing routers, particularly EmbeddLLM. We also improved the presentation of experiments by reporting detailed numerical results with Bootstrap intervals across different pool compositions (Table 1), highlighting the key results in the main text, and adding plots for regret curves of baselines on routing benchmarks (Figure 4) and the shared capability space across different ranks (Figure 1).
We once again thank the reviewer for their feedback. Please let us know if you have any follow-up questions or concerns that we can answer.