acceptodds
Under review as a conference paper at ICLR 2027

STARMAP: Representing Language Model Capabilities through Embedding Space Geometry

Abstract

The rapidly growing ecosystem of Large Language Models (LLMs) makes it increasingly challenging to manage and utilize the vast and dynamic pool of models effectively. We propose STARMAP, a method that produces low-dimensional vector embeddings that compactly represent a language model’s capabilities across queries. STARMAP is an attention-based approach that generates embeddings by a deterministic forward pass over query encodings and evaluation scores via an encoder model, enabling seamless incorporation of new models to the pool and refinement of existing model embeddings without having to perform any retraining. We additionally train a correctness predictor that uses model embeddings and query encodings to achieve state-of-the-art routing accuracy on unseen queries. Experiments show that STARMAP needs up to 4.8× fewer query evaluation samples than baselines to produce informative and robust embeddings. Moreover, the learned embedding space is geometrically meaningful: proximity reflects model similarity, enabling a range of downstream applications including model comparison and clustering, model portfolio selection, and resilient proxies of unavailable models. We additionally provide theoretical results on efficiency and stability of STARMAP, validating our empirical observations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM Model Embeddings, Query Routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3EvXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StarMap represents LLMs as fixed sized vectors built from evaluation results. Given a (query_embedding, score) tuple, this is tokenised and a permutation invariant attention encoder is used to pool a model’s tokens into a single embedding. An MLP decoder takes this embedding as input to predict whether the model will answer a new query correctly. Since the encoder is trained once, a new model is simply embedded with a single forward pass.

Across 112 relatively small open models across 10 benchmarks, it routes slightly better than the tested baselines with fewer labelled evaluation queries required. Distances between embeddings correlate with model agreement and smaller selected model pools (m=15) nearly matches the full 112 models in terms of routing accuracy.

Critical strengths

This is a well motivated problem, solved using a simple, sound architecture.The desiderata frame the problem well and demonstrate the positives of having no retraining requirements for newly added models, in stark comparison to parametric approaches.

A large strength for real-world routing is that models do not need to be evaluated on the same set of queries in order to route between them - extremely helpful for large and evolving model pools. On top of this, the authors demonstrate a broad set of downstream capabilities by utilising the embedding space to cluster by specialisation or select a set of models to route between.

Critical weaknesses

The encoder feels very close to a conditional neural process, but all of this very relevant literature is not cited at all. Therefore the novelty is more in the application than in the method.

There are no seeds or uncertainty reported and therefore small differences between methods are difficult to measure.

A kNN is very close in terms of routing accuracy (yet is only in the appendix, Table 4), suggesting that a simple non-learned method gets most of the way in terms of routing accuracy. Have you tried simply predicting the correct model based on a 10 dimension benchmark accuracy vector? This idea is also supported by the clustering of the query encodings shown in Figure 10 which forms clusters by benchmarks, so the router may be learning to largely associate benchmarks with the best model.

Some key baselines are missing which make it difficult to contextualise the routing accuracy. Could you add an oracle router (i.e. best model per query) and a single best model across all queries to see the benefits that can be gained from routing here - and what perfect routing looks like?

Changing the embedding dimensions and the number of layers (Appendix D) barely seems to affect the routing accuracy or correct predictive accuracy - although, again these differences are not properly meaningful until multiple seeds have been run. A mean pooling encoder would isolate the effect of attention here. Additionally, routing accuracy is highly sensitive to the number of heads, dropping to 43.46% when 8 heads are used (versus ~62% with 1 head). Could the authors explain this drop?

Questions for authors

Can you clarify the architectural novelty against a deterministic conditional neural process?

Can you report results with multiple seeds to demonstrate training stability and quantify whether smaller changes between methods are statistically significant?

Have you tried a simpler baseline of simply predicting the correct model based on the 10 dimension benchmark accuracy vector?

Can you add a single best model and an oracle router baseline and contextualise the router improvements to these baselines?

Can you ablate the main components (attention vs. mean pooling, the latent bottleneck, learned query pooling) to show that each is necessary?

2 comments
AuthorsOpenReview ↗

Rebuttal to Review 3EvX (1/2)

We thank the reviewer for their detailed feedback and are encouraged by their consideration of our work as "well-motivated" and "extremely helpful for large and evolving model pools".

Below, we address your key concerns and present additional evidence to better convey the strengths of STARMAP.

"The encoder feels very close to a conditional neural process, but all of this very relevant literature is not cited at all." / "Can you clarify the architectural novelty against a deterministic conditional neural process?"

We thank you for drawing our attention to the related work on conditional neural processes. STARMAP uses latent bottleneck attention (Set Transformer (Lee et al., 2019), Slot Attention (Locatello et al., 2020) and Perceiver (Jaegle et al., 2021)) and attention pooling (Lin et al. (2017), Ilse et al. (2018) and Lee et al. (2019)) as its core architecture. We will discuss the important and related direction of Conditional Neural Processes, DeepSets, Attentive Neural Processes, etc., in our updated version. To position our work in comparison to neural processes, we note that STARMAP is a deterministic CNP in which the context set is the scored evaluations of language models, the target function is the correctness function of LLMs and the meta-learning task is the LLM itself. Our contribution is the identification and formulation of the problem and the examination and analysis of embedding space geometry and subsequent downstream applications. We also empirically observe that DeepSets and vanilla conditional neural processes lose up to 3.6% in routing accuracy as compared to STARMAP.

"Can you report results with multiple seeds … and quantify whether smaller changes between methods are statistically significant?"

We describe the average performance and standard deviation for the approaches evaluated over 5 seeds.

Table A. Routing accuracy (%), mean (standard deviation) over 5 seeds, at K = 256 and 1,024 evaluations per model.

MethodEmbedLLM K=256EmbedLLM K=1,024ProxRouter K=256ProxRouter K=1,024SPROUT K=256SPROUT K=1,024
Oracle (best model per query)98.098.097.897.897.897.8
Single best model60.0 (1.2)60.0 (1.2)77.9 (0.0)77.9 (0.0)78.9 (1.0)79.3 (0.4)
Benchmark-accuracy vector (10 / 12 / 4 benchmarks)65.0 (2.1)66.9 (0.5)78.6 (0.7)81.5 (0.8)81.4 (1.6)82.6 (0.3)
Finer-label vector (≈80 sub-categories / MMLU subjects / 6 sources)58.2 (1.5)62.2 (0.9)76.2 (0.3)79.4 (0.6)81.4 (1.5)82.5 (0.3)
k-means, C = #benchmarks59.2 (2.1)62.5 (0.5)77.5 (1.6)80.5 (0.9)81.1 (1.3)82.4 (0.6)
kNN, k tuned on val.60.9 (0.5)62.6 (1.2)79.6 (1.2)80.9 (0.3)81.1 (0.9)82.3 (0.5)
IRT-Net59.3 (1.1)62.4 (1.2)78.5 (1.5)81.0 (0.3)78.8 (1.4)82.3 (0.9)
STARMAP61.0 (1.2)63.4 (0.5)81.0 (0.2)81.3 (0.3)81.5 (0.7)82.4 (0.2)

The finer-label vector uses 80 sub-categories on EmbedLLM, MMLU subjects on ProxRouter and 6 source sub-datasets on SPROUT (MMLU-Pro / MATH / GPQA / MuSR).

Across datasets and numbers of training queries, STARMAP outperforms the baselines and even closely matches the clairvoyant baselines (such as the benchmark accuracy vector), with the performance gap improving with more evaluation data. We highlight that in all of the examined query-model evaluation scenarios and pools (EmbedLLM/ProxRouter/SPROUT), the model strengths and capabilities are diverse, as the models range from small models (1B-7B) to large (70B) and frontier API-based models, hence making the best model a particularly strong baseline. A focused pool with a narrow range of model sizes would demonstrate a larger gap between the routing accuracy of STARMAP and that of the single best model, substantiating our claims. Below, we evaluate our approach on a narrower yet specialized range of model capabilities and parameters (6-8B) to substantiate the improvement in STARMAP's performance.

Routing accuracy, mean (sd), 5 seedsK = 256K = 1,024
Oracle93.893.8
Single best model67.5 (1.4)69.0 (0.0)
Benchmark-accuracy vector73.0 (1.6)75.0 (0.7)
Sub-category vector71.6 (1.2)74.1 (0.3)
STARMAP73.7 (1.1)75.1 (0.3)
AuthorsOpenReview ↗

Rebuttal to Review 3EvX (2/2)

"Have you tried simply predicting the correct model based on a 10 dimension benchmark accuracy vector? … the router may be learning to largely associate benchmarks with the best model."

We thank the reviewer for their comment on the benchmark accuracy vector. The benchmark accuracy vector for routing requires perfect knowledge of the query task at test time, which is hard to obtain, and existing approaches like kNN and KMeans are approximations for that. We find that such a clairvoyant baseline of using benchmark accuracy scores is surprisingly strong, although it depends on the granularity of benchmark/task-label classification (refer to the rebuttal to reviewer eM4T). We find that routing performance varies significantly based on the choice of such tasks/benchmark taxonomy and granularity. We also underscore that STARMAP is a broad approach which allows better utilization of a large pool of evolving language models, rather than being just a query routing approach like kNN/KMeans, etc. Existing routing approaches like kNN and KMeans fail to provide the desirable use cases which STARMAP permits, such as requiring full query evaluation data, lack of geometrical interpretation and fallback routing, model portfolio selection, etc.

"Changing the embedding dimensions and the number of layers (Appendix D) barely seems to affect the routing accuracy."

We attribute this behaviour partly to the relatively small number of models (112 models) and evaluation queries in our examined scenarios. For larger and more diverse model ecosystems and query distributions, we expect to see a more pronounced effect of hyperparameter choice on the downstream performance of STARMAP.

"A mean pooling encoder would isolate the effect of attention here." / "Can you ablate the main components (attention vs. mean pooling, the latent bottleneck, learned query pooling) to show that each is necessary?"

VariantRoutingCorrectness prediction
STARMAP (latent-bottleneck attention ×2 + learned-query pooling)62.7068.27
Full self-attention instead of latent bottleneck61.4368.02
Mean pooling instead of learned-query pooling62.6768.32
Deep Sets / vanilla CNP (token MLP → mean)59.1067.10
No attention layers60.5059.42

We observe that the attention mechanism over evaluation tokens is the most critical aspect of our approach, followed by the efficient latent bottleneck attention.

"… routing accuracy is highly sensitive to the number of heads, dropping to 43.46% when 8 heads are used (versus ~62% with 1 head). Could the authors explain this drop?"

We apologize for the error in the routing accuracy value for 8 heads. The correct value is 59.42%. We re-evaluated the empirical results and found that the original value corresponded to a failed training run.

We once again thank the reviewer for their constructive feedback and careful evaluation of our work. We have presented additional evaluation and clarifications, answered all your questions, and addressed your concerns. We would be happy to see our additional results help in your positive evaluation through an increased overall score. We would be more than happy to clarify further details and will update the draft to reflect the changes and factor in your suggestions.

Reviewer_9PHmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Starmap, an embedding model that maps an LLM to an embedding for applications such as model comparison and clustering, model portfolio selection, and correctness prediction on a given test query.

Starmap takes as input a set of (query input embedding, evaluation score) tuples, and maps them to a single &d&-dimensional embedding through a stack of MLP, latent bottleneck attention blocks, and finally learned-query attention pooling, where the set of queries need not be the same for each model. The authors also additionally train a query correctness predictor on top of Starmap embeddings for model selection.

Experiments demonstrate that Starmap achieves better model routing accuracy and matching correctness prediction accuracy compared with two baseline methods (EmbedLLM and IRT-Net), with up to 4.8 times training sample efficiency.

Critical strengths
  • Theoretical analyses are provided on embedding stability, routing robustness, and generalization.

  • 112 models are used for evaluation.

Critical weaknesses
  • Citation format is used incorrectly throughout the paper. Use citep instead of citet when the citation is not a part of the sentence's structure.

  • When generating model embeddings, a single response is sampled for each query: for a text query x and model m, we observe a text response and compute a score... (L066). However, a single response does not necessarily reflect model capabilities. Sampling multiple responses from a single query could lead to different answers.

  • Most of the 10 datasets used for evaluation are outdated. I'd suggest more modern benchmarks such as MATH, AIME, HLE, and SWE-Bench.

  • At the beginning, the paper claims "black-box compatible" as a desideratum (L071), yet the experiments do not cover any black-box model such as GPT, Claude, or Gemini.

Questions for authors
2 comments
AuthorsOpenReview ↗

Rebuttal to Review 9PHm (1/2)

We thank the reviewer for their detailed feedback and review of our work. We address all your key concerns and present additional evaluation results to substantiate our claims. We will update the final version to reflect the changes and add new results.

"Citation format is used incorrectly" Thank you for bringing this to our attention. We will update all incorrect citation format instances in the paper.

"When generating model embeddings, a single response is sampled for each query … a single response does not necessarily reflect model capabilities. Sampling multiple responses from a single query could lead to different answers."

We thank the reviewer for this observation, and we address varying aspects of this below.

  1. STARMAP accepts multi-sample scores by considering an evaluation for a query-model pair as y ∈ [0, 1], which can be an average over available responses for the same query. Our evaluation already considers the continuous LLM-as-judge scores in the appendix. We have also presented a robustness analysis of STARMAP in response to reviewer eM4T, which extends the idea of multiple responses and variance and correctness scores.
  2. We generate multiple responses for 3960 queries from 11 benchmarks on 9 open models (Llama-3.2 1B/3B, Llama-3.1-8B, Llama-2-7B, Gemma 2B/7B, Qwen2 1.5B/7B, Mistral-7B). We obtain 1 greedy response and 10 sampled responses with T=0.7. We find that 31% of query-model pairs have at least one disagreeing sample, with the majority of responses being the same. We calculate the average routing and correctness accuracy of STARMAP under varying scenarios. Overall, the performance of STARMAP is relatively stable under different sampling and evaluation scenarios.

(A) STARMAP trained and embedded with varying sampling strategies for each query (routing and correctness prediction accuracy)

Evaluation labelsRouting, K = 1,024Corr., K = 1,024Routing, K = 256Corr., K = 256
1 sampled response (mean of 3 independent draws)68.869.268.967.8
Greedy response68.869.068.967.6
Majority of 10 samples68.470.068.669.0
Mean of 10 samples (soft labels)69.070.168.867.9

(B) One STARMAP model trained on single sampled responses, with LLMs embedded from different labels (no retraining) (routing and correctness prediction accuracy)

Labels used to embed each LLMRouting, K = 1,024Corr., K = 1,024Routing, K = 256Corr., K = 256
1 sampled response (10 independent draws)68.969.468.968.0
Greedy response68.969.668.968.0
Majority of 10 samples68.969.568.967.9
Mean of 10 samples (soft labels)68.969.568.967.9

"Most of the 10 datasets used for evaluation are outdated. I'd suggest more modern benchmarks such as MATH, AIME, HLE, and SWE-Bench."

We now evaluate STARMAP on 4 advanced tasks (MATH, GPQA, MMLU-Pro, MuSR) using 13 models (including open-source and closed-source models: GPT-4o, Llama-3.3-70B-Instruct, Claude-3.5-Sonnet, Llama-3.1-405B-Instruct, Llama-3.1-70B-Instruct, GPT-4o-mini, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Granite-3-8B-Instruct, Mixtral-8x7B-Instruct, Amazon Titan Text Premier, Granite-3-2B-Instruct, Llama-3.2-1B-Instruct). On the four advanced benchmarks (with K = 256 / 1,024):

MethodRouting (%)
STARMAP81.5 / 82.4
Single best model78.9 / 79.3
Benchmark-accuracy vector81.4 / 82.6
k-means81.4 / 82.4
kNN81.1 / 82.3
EmbedLLM81.6 / 82.3
IRT-Net78.8 / 82.3

SWE-Bench is focused more on agentic execution harnesses and requires larger and more capable models (similarly for HLE and AIME). Our models in the current analysis usually perform poorly on these benchmarks due to task difficulty. We hence leave analysis on frontier models as a future extension of this work (all presented approaches and the evaluation scheme in STARMAP are directly applicable to models and tasks across the spectrum of capability and difficulty).

AuthorsOpenReview ↗

Rebuttal to Review 9PHm (2/2)

"… the paper claims 'black-box compatible' as a desideratum (L071), yet the experiments do not cover any black-box model such as GPT, Claude, or Gemini."

  • Our App. B.1 uses RouterBench, whose pool includes GPT-4-1106-preview, GPT-3.5-Turbo-1106 and Claude-v1/v2/Instant. We will explicitly highlight this in the revised draft.
  • We train STARMAP on its 9 open-weight models and later embed GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet and Amazon Titan with one forward pass over their (query, score) pairs. We evaluate routing over all 13 models:
Router256 evals / model1,024 evals / model
Oracle97.897.8
GPT-4o alone (single best model)79.779.7
Benchmark-accuracy vector, true / predicted labels78.7 / 78.982.5 / 82.4
kNN, k tuned79.882.0
IRT-Net, retrained on all 13 models80.582.4
EmbedLLM, retrained on all 13 models80.282.3
STARMAP, trained on all 13 models81.782.6
STARMAP, 9 open models; API models onboarded without retraining81.581.9

"Attention-based pooling is commonly used in the embedding literature (e.g. https://arxiv.org/abs/2405.17428). The authors should discuss them in Section 2.1 or in the related work section."

We thank the reviewer for bringing to our attention the relevant literature. We will describe and cite https://arxiv.org/abs/2405.17428 and surrounding work in our draft. Our Section 2.1 currently cites attention pooling with a learned query (Yang et al., 2016; Lin et al., 2017; Ilse et al., 2018; Lee et al., 2019) and latent bottlenecks (Locatello et al., 2020; Jaegle et al., 2021), which are closest to our work.

"What's the number of parameters of Starmap?"

The main setting examined in our evaluation has 1,012,865 trainable parameters (1.01M) (embedding dim. d = 128, 2 latent-bottleneck blocks, 64 latents, 4 heads). The breakdown across the components in the main version of STARMAP is:

ComponentParameters
Evaluation tokenizer (query encoding + score → token)196,864
2 latent-bottleneck attention blocks (272,128 each)544,256
Learned-query pooling132,096
Output layer16,512
Correctness predictor123,137

The table below shows the number of parameters as we vary key dimensions and hyperparameters for STARMAP.

ConfigurationEncoderPredictorTotal
Main setting889,728123,1371.01M
d = 16 / 32 / 6437,744 / 96,480 / 276,92815,505 / 30,881 / 61,6330.05M / 0.13M / 0.34M
1 / 3 / 4 blocks617,600 / 1,161,856 / 1,433,984123,1370.74M / 1.28M / 1.56M
16 / 256 latents877,440 / 938,880123,1371.00M / 1.06M
1 / 2 / 8 heads889,728123,1371.01M

"Can the authors visualize the 112 models using Starmap embedding (like these: …2502.16173, …2503.10633)?"

We will now include additional representations for model embeddings generated by STARMAP in accordance with the mentioned papers. Our current version of the submission already visualizes model embeddings in Figures 5 and 15 and query embeddings in Figure 10.

To summarize, we address your key concerns and questions regarding our work:

  • corrected citation format
  • multiple response generation for queries and STARMAP performance
  • newer and more difficult benchmarks
  • black-box model evaluation
  • a broader discussion of literature
  • discussion on the number of parameters for STARMAP
  • visualizing model embeddings

We once again thank the reviewer for their detailed and constructive feedback, and we hope that our additional evidence and clarification will help with your reevaluation. We would be glad to see our evidence support your positive evaluation through an increased overall score. We would be happy to further clarify any questions. We will update the draft to reflect the changes.

Reviewer_eM4TOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed an attention-based method called STARMAP that learns low-dimensional embedding to represent the LLM's abilities across different fields, and this method also allows new models or results to be incorporated without retraining the model again. The empirical studies suggest that the resulting embedding space captures meaningful model similarities and support multiple applications.

Critical strengths
  1. I think this paper studies an interesting and popular topic and it is a great idea to learn a shared mapping from evaluation results to capability embeddings. I think this paper has lots of potential for real applications.
  2. I think the empirical results are promising. The routing accuracy is higher than other modern methods.
  3. The paper is overal easy to read and understand.
Critical weaknesses
  1. If I understand correctly this paper mainly uses an existing set-attention architecture rather than a new attention method. And the theory actually relies on a stronger assumption as shown in Appendix I.2 instead of the general architecture.
  2. I have a mild concern: I am curious about the benefit of STARMAP’s learned representation over simpler alternatives. In particular, Appendix E.1 suggests that task-level accuracy profiles alone may capture much of the useful information, even when instance-level correctness is randomized. Comparisons with simple baselines such as task-level accuracy vectors would help establish what additional information STARMAP’s learned geometry gives.
Questions for authors

Please refer to the above Critical Weaknesses section

3 comments
AuthorsOpenReview ↗

Rebuttal to Review eM4T (1/2)

We thank reviewer eM4T for their encouraging and constructive feedback. We are glad that you describe our work as "studying an interesting and popular topic" and note its "potential for real applications", along with "promising" empirical results. Below we address all your questions and also present additional evaluation and results to further help convey the strengths of our work.

"This paper mainly uses an existing set-attention architecture rather than a new attention method."
We agree with the reviewer and acknowledge that the core architectural contributions in the model embedding generator (encoder) are standard building blocks as discussed in the literature we cite (Set Transformers (Lee et al., 2019), Slot Attention (Locatello et al., 2020), Perceiver (Jaegle et al., 2021), learned query pooling (Lin et al., 2017, Ilse et al., 2018, Lee et al., 2019)). Abstractly, STARMAP is a deterministic conditional neural process over the scored evaluations for language models, and we will extend our discussion of the literature to cover this (reviewer 3EvX). We highlight the formulation and desiderata we identify and the applications enabled by such a framework as our core contributions. We propose joint training for the encoder and decoder, which respectively encode model behavior and use the resulting embeddings for downstream purposes within a common framework. We examine and highlight the core geometric meaning of such embeddings and how they co-exist with semantic information that can be used by the decoder for correctness prediction and routing decisions. Furthermore, our design choices for the architecture of the encoder are guided by empirical evaluation. In these evaluations, a DeepSets encoder (Zaheer et al., 2017) or a vanilla conditional neural process (Garnelo et al., 2018) loses up to 3.6% in routing accuracy as compared to our approach (over 5 seeds).

"the theory actually relies on a stronger assumption as shown in Appendix I.2 instead of the general architecture."
Here we highlight the flavour of the key theoretical results presented and discuss how the empirically observed rates match the theoretical results. First, our theoretical examination is motivated by demonstrating the robustness of model embeddings to varying query evaluation sets, the subsequent robustness of the correctness predictor to these model embeddings, and the generalization of model embeddings to newer and previously unseen models at test time. Overall, we make the standard assumptions of boundedness, encoder stability (general form), and the Lipschitz stability of the decoder. Empirically, we subsample evaluations, generate model embeddings, and measure the log–log slope of the error ‖ẑ_k − z‖ vs. √(ρ_{n,k}/k) as provided in Theorem 3.1. We observe a slope of 0.99 against the two terms, justifying our key theoretical results. Moreover, for k=1024 queries, 84% of the routing decisions across 112 models match those made by embeddings generated from k=4096 queries.

"... the benefit of STARMAP's learned representation over simpler alternatives. In particular, Appendix E.1 suggests that task-level accuracy profiles alone may capture much of the useful information … Comparisons with simple baselines such as task-level accuracy vectors would help establish what additional information STARMAP's learned geometry gives."

We appreciate the reviewer's concern regarding the feasibility of using benchmark scores as a vector for capturing useful information. Firstly, we highlight that using a vector of benchmark scores to perform routing is infeasible at test time as the exact mapping from a query to a benchmark label is unknown and imperfect, and existing approaches like kNN and KMeans are proxy approaches for that observation. Consequently, a clairvoyant yet simple approach using a benchmark accuracy vector is a very strong baseline.

Additionally, the amount of information captured in the benchmark score vector depends on the coarseness of benchmark labels and how informative/discriminative they are for queries under examination. Below we compare the model embeddings from STARMAP with an average benchmark score vector for encapsulating model behaviour (for general benchmark and finer sub-category labels as well as for MMLU subjects).

AuthorsOpenReview ↗

Rebuttal to Review eM4T (2/2)

"... the benefit of STARMAP's learned representation over simpler alternatives. In particular, Appendix E.1 suggests that task-level accuracy profiles alone may capture much of the useful information … Comparisons with simple baselines such as task-level accuracy vectors would help establish what additional information STARMAP's learned geometry gives."

We appreciate the reviewer's concern regarding the feasibility of using benchmark scores as a vector for capturing useful information. Firstly, we highlight that using a vector of benchmark scores to perform routing is infeasible at test time as the exact mapping from a query to a benchmark label is unknown and imperfect, and existing approaches like kNN and KMeans are proxy approaches for that observation. Consequently, a clairvoyant yet simple approach using a benchmark accuracy vector is a very strong baseline.

Additionally, the amount of information captured in the benchmark score vector depends on the coarseness of benchmark labels and how informative/discriminative they are for queries under examination. Below we compare the model embeddings from STARMAP with an average benchmark score vector for encapsulating model behaviour (for general benchmark and finer sub-category labels as well as for MMLU subjects).

STARMAP embeddings capture pairwise correctness disagreement between models better than a 10-dimensional benchmark profile does (Spearman correlation of 0.87 vs. 0.75) and generally demonstrate higher routing accuracy and robustness than benchmark scores (except the 10-benchmark labels in EmbedLLM). The table below shows the routing accuracy of the presented approaches (over 3 seeds), along with that of the Benchmark Vector (10 dimensions), the sub-category vector (80 dimensions), and the single best model on the entire query distribution. Each cell shows the base routing accuracy and the accuracies when 10/20/30% of query-model evaluations are randomly flipped to evaluate robustness to noise (0/10/20/30% of labels flipped). STARMAP beats all practical baselines and closely matches or even outperforms the clairvoyant "benchmark vector" approach, while demonstrating high robustness to evaluation noise.

MethodK = 128K = 256K = 512K = 1,024
STARMAP61.2 / 58.4 / 59.4 / 57.361.7 / 60.3 / 59.9 / 59.663.2 / 62.5 / 61.3 / 60.063.4 / 62.3 / 60.8 / 60.3
EmbedLLM59.4 / 57.1 / 55.1 / 52.061.6 / 60.0 / 55.3 / 51.963.2 / 61.0 / 57.8 / 52.963.7 / 61.9 / 58.0 / 51.8
IRT-Net57.4 / 56.1 / 55.3 / 53.659.6 / 58.8 / 56.1 / 54.461.5 / 59.8 / 59.0 / 55.762.8 / 62.7 / 61.2 / 56.8
Benchmark vector (10)64.5 / 60.9 / 59.4 / 53.264.6 / 64.1 / 60.0 / 52.566.5 / 63.7 / 62.7 / 58.067.0 / 66.2 / 64.7 / 60.8
Sub-category vector (80)57.6 / 55.0 / 56.4 / 49.258.1 / 57.8 / 51.8 / 47.560.5 / 58.2 / 56.4 / 51.362.1 / 61.7 / 57.6 / 54.4
Single best59.6 / 55.7 / 56.2 / 54.159.6 / 58.8 / 53.5 / 58.260.5 / 60.5 / 58.3 / 56.760.5 / 59.6 / 56.5 / 56.6

Above, we have addressed your concerns and presented additional results. We hope our clarifications will help with your reevaluation of this work. We would be glad to see your positive reevaluation reflected in an increase in our overall score.

Thanks for your response. I have no more question and will maintain my current positive rating on this work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Requesting Reviewers to Engage with Authors

We thank the reviewers for their detailed and constructive feedback. We have provided all requested and additional evidence to clarify your questions and convey the performance and applicability of our approach, STARMAP. Particularly, we have highlighted the novelty and wide-ranging applications of our presented idea, demonstrated robustness of our approach, compared with oracle and clairvoyant (benchmark performance vector) baselines, and contextualized our gains across more recent benchmarks and large models.

We request reviewers to engage in discussion with us, and would be glad to see our new evidence contribute to your positive revaluation of our work. Once again, we would be more than happy to make additional clarifications.

We request the Area Chairs to facilitate reviewer-author discussion in this critical phase.

We once again kindly request reviewers and area chairs to engage with us and consider our new evidence and clarifications for your final evaluation of this work.

Thank you,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.