- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
StarMap represents LLMs as fixed sized vectors built from evaluation results. Given a (query_embedding, score) tuple, this is tokenised and a permutation invariant attention encoder is used to pool a model’s tokens into a single embedding. An MLP decoder takes this embedding as input to predict whether the model will answer a new query correctly. Since the encoder is trained once, a new model is simply embedded with a single forward pass.
Across 112 relatively small open models across 10 benchmarks, it routes slightly better than the tested baselines with fewer labelled evaluation queries required. Distances between embeddings correlate with model agreement and smaller selected model pools (m=15) nearly matches the full 112 models in terms of routing accuracy.
Critical strengths
This is a well motivated problem, solved using a simple, sound architecture.The desiderata frame the problem well and demonstrate the positives of having no retraining requirements for newly added models, in stark comparison to parametric approaches.
A large strength for real-world routing is that models do not need to be evaluated on the same set of queries in order to route between them - extremely helpful for large and evolving model pools. On top of this, the authors demonstrate a broad set of downstream capabilities by utilising the embedding space to cluster by specialisation or select a set of models to route between.
Critical weaknesses
The encoder feels very close to a conditional neural process, but all of this very relevant literature is not cited at all. Therefore the novelty is more in the application than in the method.
There are no seeds or uncertainty reported and therefore small differences between methods are difficult to measure.
A kNN is very close in terms of routing accuracy (yet is only in the appendix, Table 4), suggesting that a simple non-learned method gets most of the way in terms of routing accuracy. Have you tried simply predicting the correct model based on a 10 dimension benchmark accuracy vector? This idea is also supported by the clustering of the query encodings shown in Figure 10 which forms clusters by benchmarks, so the router may be learning to largely associate benchmarks with the best model.
Some key baselines are missing which make it difficult to contextualise the routing accuracy. Could you add an oracle router (i.e. best model per query) and a single best model across all queries to see the benefits that can be gained from routing here - and what perfect routing looks like?
Changing the embedding dimensions and the number of layers (Appendix D) barely seems to affect the routing accuracy or correct predictive accuracy - although, again these differences are not properly meaningful until multiple seeds have been run. A mean pooling encoder would isolate the effect of attention here. Additionally, routing accuracy is highly sensitive to the number of heads, dropping to 43.46% when 8 heads are used (versus ~62% with 1 head). Could the authors explain this drop?
Questions for authors
Can you clarify the architectural novelty against a deterministic conditional neural process?
Can you report results with multiple seeds to demonstrate training stability and quantify whether smaller changes between methods are statistically significant?
Have you tried a simpler baseline of simply predicting the correct model based on the 10 dimension benchmark accuracy vector?
Can you add a single best model and an oracle router baseline and contextualise the router improvements to these baselines?
Can you ablate the main components (attention vs. mean pooling, the latent bottleneck, learned query pooling) to show that each is necessary?
2 comments
Rebuttal to Review 3EvX (2/2)
"Have you tried simply predicting the correct model based on a 10 dimension benchmark accuracy vector? … the router may be learning to largely associate benchmarks with the best model."
We thank the reviewer for their comment on the benchmark accuracy vector. The benchmark accuracy vector for routing requires perfect knowledge of the query task at test time, which is hard to obtain, and existing approaches like kNN and KMeans are approximations for that. We find that such a clairvoyant baseline of using benchmark accuracy scores is surprisingly strong, although it depends on the granularity of benchmark/task-label classification (refer to the rebuttal to reviewer eM4T). We find that routing performance varies significantly based on the choice of such tasks/benchmark taxonomy and granularity. We also underscore that STARMAP is a broad approach which allows better utilization of a large pool of evolving language models, rather than being just a query routing approach like kNN/KMeans, etc. Existing routing approaches like kNN and KMeans fail to provide the desirable use cases which STARMAP permits, such as requiring full query evaluation data, lack of geometrical interpretation and fallback routing, model portfolio selection, etc.
"Changing the embedding dimensions and the number of layers (Appendix D) barely seems to affect the routing accuracy."
We attribute this behaviour partly to the relatively small number of models (112 models) and evaluation queries in our examined scenarios. For larger and more diverse model ecosystems and query distributions, we expect to see a more pronounced effect of hyperparameter choice on the downstream performance of STARMAP.
"A mean pooling encoder would isolate the effect of attention here." / "Can you ablate the main components (attention vs. mean pooling, the latent bottleneck, learned query pooling) to show that each is necessary?"
| Variant | Routing | Correctness prediction |
|---|---|---|
| STARMAP (latent-bottleneck attention ×2 + learned-query pooling) | 62.70 | 68.27 |
| Full self-attention instead of latent bottleneck | 61.43 | 68.02 |
| Mean pooling instead of learned-query pooling | 62.67 | 68.32 |
| Deep Sets / vanilla CNP (token MLP → mean) | 59.10 | 67.10 |
| No attention layers | 60.50 | 59.42 |
We observe that the attention mechanism over evaluation tokens is the most critical aspect of our approach, followed by the efficient latent bottleneck attention.
"… routing accuracy is highly sensitive to the number of heads, dropping to 43.46% when 8 heads are used (versus ~62% with 1 head). Could the authors explain this drop?"
We apologize for the error in the routing accuracy value for 8 heads. The correct value is 59.42%. We re-evaluated the empirical results and found that the original value corresponded to a failed training run.
We once again thank the reviewer for their constructive feedback and careful evaluation of our work. We have presented additional evaluation and clarifications, answered all your questions, and addressed your concerns. We would be happy to see our additional results help in your positive evaluation through an increased overall score. We would be more than happy to clarify further details and will update the draft to reflect the changes and factor in your suggestions.
Rebuttal to Review 3EvX (1/2)
We thank the reviewer for their detailed feedback and are encouraged by their consideration of our work as "well-motivated" and "extremely helpful for large and evolving model pools".
Below, we address your key concerns and present additional evidence to better convey the strengths of STARMAP.
"The encoder feels very close to a conditional neural process, but all of this very relevant literature is not cited at all." / "Can you clarify the architectural novelty against a deterministic conditional neural process?"We thank you for drawing our attention to the related work on conditional neural processes. STARMAP uses latent bottleneck attention (Set Transformer (Lee et al., 2019), Slot Attention (Locatello et al., 2020) and Perceiver (Jaegle et al., 2021)) and attention pooling (Lin et al. (2017), Ilse et al. (2018) and Lee et al. (2019)) as its core architecture. We will discuss the important and related direction of Conditional Neural Processes, DeepSets, Attentive Neural Processes, etc., in our updated version. To position our work in comparison to neural processes, we note that STARMAP is a deterministic CNP in which the context set is the scored evaluations of language models, the target function is the correctness function of LLMs and the meta-learning task is the LLM itself. Our contribution is the identification and formulation of the problem and the examination and analysis of embedding space geometry and subsequent downstream applications. We also empirically observe that DeepSets and vanilla conditional neural processes lose up to 3.6% in routing accuracy as compared to STARMAP.
"Can you report results with multiple seeds … and quantify whether smaller changes between methods are statistically significant?"We describe the average performance and standard deviation for the approaches evaluated over 5 seeds.
Table A. Routing accuracy (%), mean (standard deviation) over 5 seeds, at K = 256 and 1,024 evaluations per model.
The finer-label vector uses 80 sub-categories on EmbedLLM, MMLU subjects on ProxRouter and 6 source sub-datasets on SPROUT (MMLU-Pro / MATH / GPQA / MuSR).
Across datasets and numbers of training queries, STARMAP outperforms the baselines and even closely matches the clairvoyant baselines (such as the benchmark accuracy vector), with the performance gap improving with more evaluation data. We highlight that in all of the examined query-model evaluation scenarios and pools (EmbedLLM/ProxRouter/SPROUT), the model strengths and capabilities are diverse, as the models range from small models (1B-7B) to large (70B) and frontier API-based models, hence making the best model a particularly strong baseline. A focused pool with a narrow range of model sizes would demonstrate a larger gap between the routing accuracy of STARMAP and that of the single best model, substantiating our claims. Below, we evaluate our approach on a narrower yet specialized range of model capabilities and parameters (6-8B) to substantiate the improvement in STARMAP's performance.