acceptodds
Under review as a conference paper at ICLR 2027

COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use

13 views

Abstract

LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change. We propose COMPASS, an agentic router in which an LLM follows a routing recipe, gathers performance and cost evidence through tools, and optionally inspects candidate answers before selecting a model. Our evaluations on LLMRouterBench and DeepSWE show that agentic routing can improve accuracy and lower model-call cost relative to specialized and trained routing baselines, without routing-specific training. On LLMRouterBench, COMPASS reaches 69.85% accuracy at $139.57, compared with RLCascadeRouter's 69.14% at $180.22. On DeepSWE under fixed five-fold cross-validation, it reaches 78.1% at $215, compared with 74.1% at $500 for the best single model. COMPASS also supports routing across benchmarks and adapts to an expanded model pool by reading new evaluation records, without changing the router's weights or prompt.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, LLM routing, model selection, agentic routing, tool-using agents, adaptive inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XbFhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes COMPASS, a training-free LLM routing framework that uses an LLM agent with tool calling to select suitable models based on historical performance and cost. The method supports adaptive evidence gathering and adding new models without retraining the router. Experiments on LLMRouterBench and DeepSWE show good accuracy and cost trade-offs compared with existing routing methods.

Critical strengths
  1. No retraining is needed when adding new models, which makes the router more flexible.
  2. Adaptive tool use allows the router to gather more evidence before making decisions.
Critical weaknesses
  1. Limited novelty. The method mainly combines existing LLM agents, tool calling and benchmark-based routing, without a new routing algorithm.
  2. Router overhead is not fully considered. Additional LLM calls may increase the actual cost and latency.
Questions for authors

What's the router overhead?

1 comment

Thank you for highlighting the flexibility of COMPASS and the value of adaptive tool use. We address your two concerns below.

🟦 W1

“Limited novelty. The method mainly combines existing LLM agents, tool calling and benchmark-based routing, without a new routing algorithm.”

Response. The contribution of COMPASS is how the agentic layer uses evidence to make a routing decision for each task. Appendices D–F show three end-to-end routing trajectories where the agent overrides the fixed kNN recommendation. In the beta spectrometer example, the agent finds that the neighbouring tasks provide weak evidence and rejects both previewed answers. It then checks broader historical scores and selects a third model. These examples show how the agentic layer can assess the quality of the neighbours, choose another tool when needed, and revise the initial recommendation.

🟦 W2/Q1

“What's the router overhead?”

Response. Figure 2 now includes the router inference bill for each COMPASS point. In the headline LLMRouterBench run, DeepSeek-V4.1-Flash incurred USD 7.95 in router inference costs, or approximately USD 0.0022 per scored task. This was about 6% of the task-model spend. We estimate the bill using OpenRouter rates, including discounted rates for cached input tokens.

We cap the router’s token budget at 32,000 tokens per task to make inference costs easier to budget. GPT-5.6 Luna, GPT-5.6 Sol, and DeepSeek-V4.1-Flash achieved similar accuracy on LLMRouterBench, suggesting that smaller models can serve as routing agents in this setting. The tools in Table 1 handle queries in real time at minimal cost, so their runtime and memory overhead are negligible.

Reviewer_MpVDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the challenges of model selection and cost optimization in large language model (LLM) routing. We propose the COMPASS framework, which employs an existing LLM as an "Agentic Router." Guided by a predefined "Routing Recipe," this router dynamically queries information—such as the historical performance of candidate models, their execution costs, and data on similar tasks—via external tools. It can also preview model responses to determine the most suitable model for a given task. This approach eliminates the need for additional training specifically for the routing task and allows for adaptation to new candidate models by simply updating external evaluation data. Experiments conducted on benchmarks such as LLMRouterBench and DeepSWE demonstrate that COMPASS achieves a superior balance between accuracy and cost compared to various existing routing methods, while also exhibiting strong adaptability across different tasks and to newly introduced models.

Critical strengths

The research motivation is clear: existing LLM routers often rely on routing strategies trained for a specific set of candidate models, which may necessitate retraining or updates whenever new models emerge or the task distribution shifts. This paper explores the use of existing LLMs' tool-use capabilities—leveraging external evaluation data—to achieve dynamic routing, a concept that holds practical significance.

Critical weaknesses
  1. Insufficient methodological innovation. The proposed COMPASS framework essentially leverages the existing tool-calling and prompt-based routing capabilities of Large Language Models (LLMs) to select an appropriate model by querying historical performance, costs, and data on similar tasks. The framework is primarily a combination of existing LLM agent, RAG (Retrieval-Augmented Generation), and traditional model-routing techniques, lacking novel model architectures, training algorithms, or distinct technical innovations. In particular, the paper fails to adequately demonstrate that "agentic routing" offers any unique, irreplaceable advantages over simpler routing strategies—such as those based on historical performance statistics, k-nearest neighbors (kNN), or predefined rules.

  2. Inadequate experimental comparisons and lack of robust support for certain conclusions. While the paper compares COMPASS against methods like FrugalGPT, GraphRouter, and Avengers-Pro, it omits systematic comparisons with more recent LLM routing approaches, such as Router-R1, GraphPlanner, and Universal Model Routing. Furthermore, the results for RLCascadeRouter in the LLMRouterBench evaluation are cited directly from existing literature rather than being reproduced under identical experimental conditions. The DeepSWE dataset consists of only 113 tasks; some reported performance gains exhibit wide confidence intervals, and the performance advantage over the best-performing single model across benchmarks fails to reach statistical significance. Consequently, the current experimental evidence is insufficient to conclusively demonstrate that COMPASS provides stable and significant performance advantages across diverse task distributions.

  3. Lack of comprehensive evaluation regarding routing overhead and practical deployment efficiency. COMPASS employs an additional LLM agent to perform multi-turn tool calls, query historical performance, and preview responses from candidate models—a process likely to incur significant token costs and inference latency. However, the paper focuses primarily on the "model-call cost" of the candidate models, failing to fully account for the inference costs of the "router" LLM itself within the primary cost-performance analysis. In low-cost routing scenarios, in particular, the overhead introduced by the router could negate any savings gained from selecting a less expensive model. Additionally, the method requires the prior collection of performance records for candidate models across a large number of tasks; the integration of any new model would necessitate further evaluation efforts. It is recommended that the authors provide a complete analysis of end-to-end costs, latency, and data-collection overhead to validate the framework's practical deployment value.

  4. The presentation of the paper leaves significant room for improvement. The paper lacks a sufficiently clear and intuitive explanation of the specific implementation and decision-making process of COMPASS. For instance, the rationale behind the selection of different "Routing Recipes," the sequence of tool invocations, the termination criteria, and the final model selection process are described primarily in text, without concrete end-to-end examples to illustrate how the agent makes sequential decisions based on evidence returned by various tools. Furthermore, certain aspects of the experimental setup and the presentation of results lack clarity; for example, the scope of cost statistics for different methods in Figure 2 is not entirely consistent, some COMPASS results incorporate additional ad-hoc adjustments, and there are discrepancies in model configurations and evaluation methodologies across different experiments. It is recommended that the authors provide detailed examples of "Routing Trajectory" and further standardize and clarify the experimental setup, cost calculation methods, and result presentation to enhance the paper's readability and the credibility of its conclusions.

Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer MpVD

Thank you for the careful review and for highlighting the practical motivation for updating the model pool without retraining. We address each concern below.

🟦 W1

“[…] the paper fails to adequately demonstrate that "agentic routing" offers any unique, irreplaceable advantages over simpler routing strategies […]”

Response. To show what the agentic layer adds, we evaluated fixed statistical routing strategies on the same held-out tasks. The first three rows show results for the best single model selected on the reference split, kNN routing, and reliability-only routing, respectively. The fourth row adds cascading to the reliability-only recipe, allowing us to compare the same recipe with and without cascading.

RecipeDeepSWEFrontierSWEProgramBenchSWE-ProTerminalBench
Best single model on reference split77.1451.5070.2547.0080.95
kNN77.1451.5068.9441.4780.95
Reliability-only65.7151.5068.8147.4780.95
Reliability + cascade65.7151.5069.9948.8580.95
kNN cascade80.0051.2072.2048.8585.71

On the 35 held-out DeepSWE tasks, kNN cascade improved by 2.86 points over direct kNN and the reference-best single model. It led on DeepSWE, ProgramBench, and TerminalBench, tied reliability + cascade on SWE-Pro, and trailed the best FrontierSWE score.

🟦 W2

“[…] it omits systematic comparisons with more recent LLM routing approaches, such as Router-R1, GraphPlanner, and Universal Model Routing.”

Response. We added four UniRoute settings to Figure 2, with costs that include agent inference cost. We removed the imported RLCascadeRouter point and the three adjusted pilot points. To broaden the evaluation beyond DeepSWE, we added a five-benchmark SWE cohort of 1,137 tasks, split into 795 reference tasks and 342 held-out test tasks. We report results for the agentic router and the individual routing strategies described in the table above. Three additional LLMRouterBench cascade runs at α = 1 characterize the robustness of the agent across routing rounds.

Routing roundaccuracy (%)
169.06
269.64
369.28
Mean ± sample SD69.33 ± 0.29

🟦 W3

“[…] provide a complete analysis of end-to-end costs, latency, and data-collection overhead […]”

Response. The agent token budget is capped at 32,000 tokens per task to make inference costs easier to budget. Using OpenRouter rates available for our estimate, including discounted rates for cached input tokens, the reported DeepSeek-V4.1-Flash inference bill averaged approximately USD 0.0022 per scored task, which is negligible even for low-cost tasks.

🟦 W4

“[…] without concrete end-to-end examples to illustrate how the agent makes sequential decisions based on evidence returned by various tools.”

Response. Appendix B gives the reliability and cascade recipes, and Appendix D walks through three routing trajectories from the kNN recommendation to final submission. In the first example, the router previews the default answer and requests neighbouring outcomes, then checks the answer and previews an alternative before committing. The cascade permits up to two paid previews and stops when it submits a configuration or exhausts its budget. The prompts make the evidence and escalation rules explicit.

The revised evaluation section distinguishes the five-benchmark SWE split from the 113-task DeepSWE five-fold evaluation. Figure 2 uses unadjusted COMPASS results on the seed-42 LLMRouterBench test split (3,674 scored tasks per setting), and its caption states which methods include router inference.

Reviewer_iVe2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

COMPASS uses an existing LLM agent over external model/effort outcome records to select a configuration. Routing recipes govern statistical-tool calls, optional paid answer previews, and stopping. The evaluation covers accuracy–cost tradeoffs, cross-benchmark transfer, model-pool expansion, and disjoint-trial estimates of routing headroom without training router-specific weights.

Critical strengths

Keeping capability records separate from router weights is practically useful for updating candidate pools, and the tool interface brings multiple kinds of evidence into one decision procedure. The paper includes rerun baselines, task-level five-fold DeepSWE cross-validation, and a careful disjoint-trial analysis that distinguishes optimistic oracle results from headroom supported by independent outcomes. It explicitly qualifies uncertain transfer, reports a negative FrontierSWE result, identifies pilot-adjusted plotted points, and acknowledges omitted agent-inference expense. These candid evaluations make the design easier to assess.

Critical weaknesses

The novelty and added value of the agent layer are not yet isolated from established record-based routing and tool-assisted configuration selection. Table 1 and §4.1 already provide nearest-neighbor recommendations, accuracy–cost frontiers, and held-out policy backtests. Section 4.2 specifies four recipe families, including “choose a policy,” in which the agent compares predefined policies but cannot invent them. Record-based routing that consumes new model evidence is also established in cited work such as UniRoute. Using an existing LLM to interpret this evidence may be useful engineering, but the paper does not show how much it adds beyond directly executing a selected statistical policy. An agent-free execution of the best backtested policy under the same evidence mask, objective, and preview budget—on both the 18- and 28-configuration DeepSWE pools—would clarify the contribution. Pool expansion in §5.6 already fixes the prompt and policy; the missing test is attribution to the agent rather than the underlying evidence and preset rules. Distinguishing the agent's own contribution from the benefit of a stronger pool is essential to the novelty claim.

The reported results do not yet establish a favorable end-to-end production tradeoff or robust transfer. Recipe design and selection are setup activities, while §4 says the LLM agent gathers evidence and submits a configuration for each routed query; §4.1 reports two or three tool calls per task after precomputation. The plotted USD 139.57 versus USD 180.22 LLMRouterBench model-call comparison leaves a USD 40.65 margin, but §5.2, §5.7.2, and §6 exclude the online agent's own inference bill. The larger DeepSWE USD 215 versus USD 500 model-call gap may be more forgiving, but also needs all-in measurement. Corpus reads count as free model calls, and previews are already charged; the missing quantities are point-level agent inference, tool/runtime overhead, latency, previews, and unresolved/retry rates, with offline corpus and recipe-development costs reported separately. Rejected previews should remain charged and accepted cached answers counted once, as the stated protocol requires. The three lowest-cost Fig. 2 points are already disclosed as +0.5-point-adjusted pilots with scaled costs (§5.2), and RLCascadeRouter is an identified imported published point (§5.2.1); unadjusted, cohort-labeled results, uncertainty, and matched split, pool, metric, and price conditions would make the frontier easier to assess. This is a comparability question, not a fabrication allegation. Exact prompts, tool schemas, and recipe selection matter because §4.2 reports 5.5–9.5-point sensitivity to recipe wording. The cross-benchmark result in §5.4 is a useful transfer test, but its paired advantage over the best single model includes zero; §5.6 adds models using records on the same DeepSWE tasks. Given the paper's own coverage and distribution caveats (§6), performance under sparse, stale, or shifting production evidence remains insufficiently demonstrated.

The stated evidence masks and fold-specific table rebuilding are useful safeguards, but reproducibility still requires a point-level manifest and an example preview payload showing what the agent can see and confirming hidden grading metadata is excluded from traces. Corpus-only settings average a configuration's recorded trials, whereas cascades score an accepted attempt (§4.4); each plotted point should identify its recipe and estimand. The existing task-bootstrap intervals are useful, while paired headline differences and routing repeats would characterize uncertainty from agent behavior. Finally, the completion-first objective's per-input success target is not shown to be implemented by the alpha-based, uncalibrated recipes. Reporting its heuristic status, coverage, and budget-exhausted cases would make this claim more precise. The retrospective behavioral-neighbor bound in Appendix A should remain distinct from a deployable retrieval result.

Questions for authors
  1. What is the incremental gain over directly executing the best Table 1/backtested policy without an LLM agent, using the same mask, objective, and preview budget on the 18- and 28-configuration pools?

  2. At the headline LLMRouterBench and DeepSWE points, what are the online agent-inference bills, point-level preview and tool-call counts, unresolved/retry rates, and end-to-end latency, with symmetric accounting for comparators? Please report offline corpus and recipe-development costs separately.

  3. Can the three disclosed pilot-adjusted Fig. 2 points be shown unadjusted and by cohort, with the recipe/prompt and uncertainty for each point? What split, model pool, and prices make the imported RLCascadeRouter point comparable to the rerun curves? Can a point-level manifest also show the protected evidence/scoring boundary?

1 comment
AuthorsOpenReview ↗

Response to Reviewer iVe2

Thank you for the careful review. We address each concern below.

🟦 W1/Q1

“What is the incremental gain over directly executing the best Table 1/backtested policy without an LLM agent […]?”

Response. We assessed the contribution of the agentic layer by comparing it with fixed routing strategies across all test samples. The first three rows report results for the best single model on the reference split, kNN, and reliability-only routing, respectively. To assess the effect of cascading, we also evaluated reliability with cascading in the fourth row. These comparisons cover the main tools available to the agent in our experiments. Table 2 compares five benchmarks on a frozen split of 1,137 SWE tasks (795 reference and 342 held out).

RecipeDeepSWEFrontierSWEProgramBenchSWE-ProTerminalBench
best single model77.1451.5070.2547.0080.95
kNN77.1451.5068.9441.4780.95
Reliability-only65.7151.5068.8147.4780.95
Reliability + cascade65.7151.5069.9948.8580.95
kNN cascade80.0051.2072.2048.8585.71

On the 35 DeepSWE test tasks, kNN cascade improved by 2.86 points over direct kNN and the best single model. It led on DeepSWE, ProgramBench, and TerminalBench, tied reliability + cascade on SWE-Pro, and trailed the best FrontierSWE score.

🟦 W2/Q2

“[…] what are the online agent-inference bills, point-level preview and tool-call counts, unresolved/retry rates, and end-to-end latency […]?”

Response. Figure 2 now includes the router inference bill for each COMPASS point. On LLMRouterBench, COMPASS at α = 1 achieved 69.85% accuracy at a total cost of USD 147.53, including USD 7.95 for routing. Direct GPT-5 achieved 65.12% accuracy at USD 124.82. In the five-fold DeepSWE evaluation on 113 tasks, COMPASS achieved 78.10% accuracy at USD 215.74, including USD 0.30 for routing. The best single configuration achieved 74.12% accuracy at USD 500.49. The reported costs include rejected previews and charge for each accepted cached answer only once.

We also compared routing performance on LLMRouterBench across GPT-5.6 Luna, GPT-5.6 Sol, and DeepSeek-V4.1-Flash. The three models achieved similar accuracy, suggesting that smaller models can also serve as routing agents. We capped the router’s budget at 32,000 tokens per task. In the headline run, DeepSeek-V4.1-Flash’s reported inference bill averaged approximately USD 0.0022 per scored task.

🟦 W2/Q3

“Can the three disclosed pilot-adjusted Fig. 2 points be shown unadjusted and by cohort, with the recipe/prompt and uncertainty for each point?”

Response. We removed the three adjusted pilot points and the imported RLCascadeRouter point from Figure 2. The eight COMPASS points now report unadjusted results. All the points in the figure shares the same setup as follows:

AttributeValue
BenchmarkLLMRouterBench, covering 10 benchmarks
Split70% reference, 30% test
Split seed42
Scored tasks per setting3,674
Accuracy metricaccuracy, computed as the average of accuracy across the 10 benchmarks
Model pool13 models: claude-sonnet-4, deepseek-r1-0528, deepseek-v3-0324, deepseek-v3.1-terminus, gemini-2.5-flash, gemini-2.5-pro, glm-4.6, gpt-5, gpt-5-chat, intern-s1, kimi-k2-0905, qwen3-235b-a22b-2507, and qwen3-235b-a22b-thinking-2507
Routing LLMDeepSeek-V4.1-Flash
Evidence toolsShared tool schemas as described in table 1
COMPASS pointsEight unadjusted test-split results
Cost accountingCOMPASS and UniRoute totals include task-model calls and router inference. Rejected COMPASS previews are charged, and each accepted preview is counted once.
Avengers-ProAll 33 α settings
FrugalGPTBudgets of 0.005, 0.01, 0.02, 0.03, 0.04, and 0.06, plus an unconstrained setting
GraphRouterCost, balanced, and performance settings
UniRouteScaled λ values of 0, 0.01, 0.1, and 1
Single-model frontierDeepSeek-V3, Qwen3-235B, Kimi-K2, Qwen3-235B-Thinking, and GPT-5
kNN cascadeUses the 48 nearest visible reference tasks and allows up to two paid previews
Routing promptsReliability and cascade prompts are provided in Appendix B

Appendix B includes the reliability and cascade prompts. The kNN cascade calls statistic_recommendation to select a default configuration based on the 48 most similar visible reference tasks. The agent previews the default answer and accepts it if it satisfies the task. Otherwise, the agent uses the kNN statistics to choose an alternative and may preview its answer before submitting a final recommendation. The recipe allows up to two paid previews. Rejected previews remain charged, and each accepted preview is counted once.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.