acceptodds
Under review as a conference paper at ICLR 2027

Route by think, not just semantics: LLM routing via query cognition state

Abstract

Large Language Models (LLMs) exhibit highly heterogeneous capabilities across domains and tasks, making it increasingly important to route each query to the most cost-effective model that can answer it reliably. Existing LLM routers typically rely on textual or semantic representations of queries. However, semantic similarity does not necessarily imply routing similarity. Specifically, semantically similar queries may require substantially different capabilities, while semantically different queries may be best handled by the same model. In fact, humans typically form an internal understanding of a question before producing an answer. Likewise, an LLM progressively transforms a query into contextualized internal representations before generating its response, motivating us to explore whether such representations provide richer signals for routing. Based on this insight, we present CogRouter, an LLM routing framework that routes queries according to their internal representations from a generative model and matches them against candidate model capabilities. Each candidate is represented by combining its capability description with a learnable embedding derived from historical performance, capturing complementary semantic and empirical capability signals. To avoid discarding equally best routing choices, we further propose a tie-aware pairwise training strategy that preserves all tied-best candidates and avoids artificial ordering among them. Extensive experiments on xRouteBench and RouterBench demonstrate that CogRouter outperforms state-of-the-art LLM routing methods by up to 13.81% and 20.09% in routing quality. Our code is available at https://anonymous.4open.science/r/CogRouter-F45C/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Routing, Contextualized Internal Representations, Query–Model Matching, Tie-Aware Pairwise Learning, Quality–Cost Trade-offs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xhHDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents the idea of cognitive embeddings and how they can help in routing of language queries across models. Authors identify that sentence embedding based, tag based or LLM based routing approaches have some undesirable aspects, and how cognitive embeddings can be used for overcoming them. A motivating prestudy is presented, and the method is described after this motivating example. Experiments are then discussed where Cogrouter shows improvements in performance.

Critical strengths
  1. The presentation of cognitive embeddings gives new insight into understanding routing.
  2. Experimental results are reasonably detailed.
  3. The motivating prestudy gives insights into failure modes of existing routing approaches.
Critical weaknesses

This paper has multiple and significant weaknesses.

  1. This paper's approach uses arguments and ideas presented in bits and pieces across literature, and it is hard to find novelty after accounting for surrounding literature. For example, the notion of "cognitive embeddings" for routing is not new, and many papers have examined very similar ideas across literature. Model Embedding Literature (EmbedLLM, IRT-Net, LLM-DNA, IRT-Router,) already try to build model embeddings ("cognitive embedding") to use them across multiple domains and not just routing. Similarly internal model representations have been used across routing and uncertainty quantification literature to identify query-model mappings.

  2. The idea of tie-aware pairwise ranking is not new, even in routing literature. Ranking loss and contrastive learning literature has examined this in 2000s. Furthermore, even independent correctness BCE loss (instead of pairwise) identically treats equal scoring models instead of enforcing and order over them. The contrastive loss in RouterDC also suffices for this purpose.

  3. "CogRouter is the first to combine query-gated model capability representations with tied-best pairwise supervision for LLM routing.": From 1 and 2, this is not novel. model capability representations are already existing in literature. query-gating has been studied across multiple domains. tied-best pairwise supervision is oversold as a contribution.

  4. the methodology section can be significantly improved in its exposition, and currently it is very hard to parse; partly due to the complicated motivation of cognitive embeddings and the cumbersome notation introduced.

Learning to Rank using Gradient Descent, Burges et al, 2005.

LLM Router: Rethinking Routing with Prefill Activations Tanay Varshney et al 2026.

Query-Level Uncertainty in Large Language Models — Chen et al., 2025.

LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models — Ratnakar et al., 2026.

ETC and other papers in literature mentioned above.

Questions for authors

Current form of the paper needs major restructuring and thought before being considered independently for main conference track.

All questions and concerns mentioned in the weaknesses. Thank you.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xhHD—W1. Novelty, distinctions from existing model embedding and internal representation-based methods (1/4)

W1. This paper's approach uses arguments and ideas presented in bits and pieces across literature, and it is hard to find novelty after accounting for surrounding literature. For example, the notion of "cognitive embeddings" for routing is not new, and many papers have examined very similar ideas across literature. Model Embedding Literature (EmbedLLM, IRT-Net, LLM-DNA, IRT-Router,) already try to build model embeddings ("cognitive embedding") to use them across multiple domains and not just routing. Similarly internal model representations have been used across routing and uncertainty quantification literature to identify query-model mappings.

We sincerely thank the reviewer for highlighting these closely related studies. We apologize that our original presentation did not sufficiently clarify the methodological distinctions between CogRouter and existing model embedding and internal representation-based routing approaches.

Regarding model embedding methods, we recognize that EmbedLLM and LLM-DNA have investigated learned model representations, while IRT-Net and IRT-Router characterize model capabilities and query characteristics through ability–difficulty relationships. CogRouter differs in how query–model suitability is estimated. Rather than explicitly parameterizing model abilities and query difficulty, CogRouter learns suitability scores through quality-supervised query projections and candidate representations that combine learned performance identities with semantic capability descriptions. A query-dependent fusion mechanism further adjusts the contribution of candidate descriptions according to the incoming query.

Regarding internal representation-based methods, the previous studies have explored hidden states and model activations for routing and uncertainty estimation. Our contribution does not lie in introducing internal representations themselves. CogRouter instead uses the standard output of a frozen embedding encoder and learns routing-specific suitability through a trainable projection and candidate-matching mechanism supervised by observed response quality. The distinction therefore lies in how query representations interact with candidate-specific capability information to support routing decisions.

To further substantiate the effectiveness of this suitability-learning framework, we provide the following experimental evidence. The main benchmark results demonstrate that CogRouter achieves higher routing quality than the strongest reported baselines under the corresponding evaluation settings.

BenchmarkStrongest reported baselineCogRouterImprovement
xRouteBench (five-domain macro)RouterDC: 47.43%53.98%+6.55 pp
RouterBenchGraphRouter: 65.12%78.20%+13.08 pp

As shown above, CogRouter improves routing quality by 6.55 and 13.08 percentage points over the strongest reported baselines on xRouteBench and RouterBench, respectively. These results demonstrate the effectiveness of the complete query–candidate suitability-learning framework across two routing benchmarks.

Furthermore, to examine the contribution of candidate capability representations, we have conducted controlled experiments with matched training conditions. On Generic, incorporating query-gated capability descriptions improves routing quality from 77.370% (identity-only) to 78.021%, corresponding to a gain of 0.651 percentage points, with a conditional 95% confidence interval of [0.151, 1.141]. This result supports the value of incorporating candidate description information alongside learned performance identities. The comparison does not, by itself, establish an independent advantage of dynamic gating over a global gate.

Together, these results provide empirical support for CogRouter's suitability-learning design. While the benchmark comparisons demonstrate its overall effectiveness, the controlled experiment further examines the contribution of integrating candidate capability information. We emphasize that these results support our specific methodological design rather than establishing novelty for model embeddings or hidden-state extraction in isolation.

We have clarified the methodological distinctions and refined the contribution statements in Section 1 (Lines 73–94) and Section 2.2 (Lines 115–140). The query representation, candidate fusion, and suitability-matching mechanisms are detailed in Sections 4.2–4.4 (Lines 206–273). The benchmark results are reported in Section 5.2 (Table 1 and Figure 3, Lines 333–376), while the controlled candidate-representation analysis is presented in Section 5.5 (Lines 502–508) and Appendix I (Lines 1869–1878).

AuthorsOpenReview ↗

Response to Reviewer xhHD—W2. Novelty and effectiveness of tie-aware pairwise supervision (2/4)

W2. The idea of tie-aware pairwise ranking is not new, even in routing literature. Ranking loss and contrastive learning literature has examined this in 2000s. Furthermore, even independent correctness BCE loss (instead of pairwise) identically treats equal scoring models instead of enforcing and order over them. The contrastive loss in RouterDC also suffices for this purpose.

We thank the reviewer for highlighting the connections between our training objective and established ranking, binary classification, and contrastive learning approaches. We apologize that the original manuscript did not clearly distinguish the established ranking formulation from the routing-specific supervision construction employed by CogRouter.

We recognize that logistic pairwise ranking is a well-established technique and that BCE and contrastive objectives can also accommodate multiple candidates with equal quality labels. The contribution of our supervision design lies in how observed query-level quality feedback is converted into preference comparisons and balanced during optimization, rather than in introducing pairwise ranking or tie handling as new principles.

Specifically, for each query, CogRouter identifies all candidates achieving the highest recorded quality and compares each of them against every strictly lower-quality candidate, without enforcing an ordering among equally best candidates. For example, given candidate qualities (1, 1, 0), the resulting comparisons are (1, 3) and (2, 3), leaving candidates 1 and 2 unordered. Furthermore, pairwise losses are normalized within each query, ensuring that queries with different numbers of valid comparisons contribute equally to the training objective.

To directly address the reviewer's concern regarding alternative supervision objectives, we have conducted additional controlled experiments comparing Tied-best supervision with Quality BCE and Multi-positive supervision. Within each encoder condition, these experiments use identical query and candidate representations, matching architectures, initialization, optimization settings, training budgets, and checkpoint-selection procedures. Only the supervision objective is changed.

The results are summarized below.

Encoder / DatasetTied-bestQuality BCEMulti-positive
Qwen / LoCoMo27.374%26.168%27.293%
Qwen / Geometry3K43.716%40.437%43.716%
BGE / Geometry3K45.902%42.077%45.355%

As shown above, Tied-best supervision consistently achieves higher mean routing quality than Quality BCE across the three evaluated settings, with improvements ranging from 1.206 to 3.825 percentage points. It also matches or outperforms Multi-positive supervision in all three cases. These results show that, even when the representations and training configurations remain fixed, the construction and aggregation of preference comparisons can affect routing performance. Moreover, the controlled fixed-winner experiment further supports the benefit of retaining all valid best candidates: replacing a single fixed winner with Tied-best supervision improves five-domain macro quality from 53.03% to 53.72% under matched training conditions.

We have expanded the explanation of preference construction, query-level normalization, and the corresponding training objective in Section 4.5 (Lines 275–299). The fixed-winner comparison is reported in Section 5.3 and Table 2 (Lines 340–349 and 378–383). The additional matched-objective experiments, numerical results, and analysis are presented in Section 5.5 and Table 6 (Lines 489–508), with detailed experimental configurations and uncertainty estimates in Appendix I (Lines 1836–1867, Table 18).

AuthorsOpenReview ↗

Response to Reviewer xhHD—W3. Clarification of the methodological contribution and novelty claims (3/4)

W3. "CogRouter is the first to combine query-gated model capability representations with tied-best pairwise supervision for LLM routing.": From 1 and 2, this is not novel. model capability representations are already existing in literature. query-gating has been studied across multiple domains. tied-best pairwise supervision is oversold as a contribution.

We would like to clarify that CogRouter does not claim novelty for model capability embeddings, query-dependent gating, or pairwise ranking as independent techniques. Instead, our work addresses two interconnected characteristics of LLM routing: semantic similarity does not necessarily determine model suitability, and a query may have multiple equally optimal candidate models.

CogRouter addresses these characteristics through a unified suitability-learning formulation. Candidate representations combine empirical performance identities with semantic descriptions, while query-conditioned matching estimates candidate suitability. The training objective then uses observed response quality to construct best-versus-strictly-worse preferences, preserving multiple valid routing choices instead of reducing each query to a single target.

The methodological contribution is therefore centered on the formulation of query–candidate suitability learning and the supervision strategy used to optimize it, rather than the novelty of the constituent techniques in isolation.

To provide empirical evidence for these design choices, we have strengthened the controlled analysis of supervision and candidate representations. The five-domain ablation results are summarized below.

ConfigurationFive-domain macro quality
Identity-only + fixed winner53.03%
Identity-only + tied-best53.72%
Tie + gated descriptions53.88%

These results show that preserving all tied-best candidates improves macro routing quality by 0.69 percentage points under the matched fixed-winner control. Adding query-gated descriptions further improves the mean from 53.72% to 53.88% in this experiment.

We have additionally examined description fusion under controlled training conditions. On Generic, the query-gated description model achieves 78.021%, compared with 77.370% for the identity-only configuration, yielding a gain of 0.651 percentage points, with a conditional 95% confidence interval of [0.151, 1.141]. The comparison against a learned global gate yields a smaller increment; therefore, we position query-dependent gating as a component of the suitability-matching design rather than claiming it as an independently established innovation.

The effectiveness of the complete framework is further supported by the main benchmark comparisons. CogRouter achieves 53.98% five-domain macro quality on xRouteBench, compared with 47.43% for the strongest reported baseline, RouterDC. On RouterBench, CogRouter reaches 78.20%, compared with 65.12% for GraphRouter, the strongest reported baseline in that evaluation.

Together, these results provide evidence for the effectiveness of the proposed suitability-learning framework and its supervision design, while the controlled analyses clarify the contributions of the individual components.

To address the reviewer's concern about overstated novelty, we have refined the contribution statements in Section 1 (Lines 68–94) and clarified the methodological positioning in Section 2.2 (Lines 122–140). The model representation and matching mechanism are detailed in Sections 4.3–4.4 (Lines 244–273). The relevant experimental evidence is presented in Section 5.2 (Table 1 and Figure 3, Lines 333–376), Section 5.3 (Tables 2–3, Lines 340–400), and Section 5.5 (Lines 489–508), with the additional candidate-fusion controls reported in Appendix I (Lines 1869–1878).

AuthorsOpenReview ↗

Response to Reviewer xhHD— W4. Methodology presentation and notation clarity & Q. Overall concerns (4/4)

W4. the methodology section can be significantly improved in its exposition, and currently it is very hard to parse; partly due to the complicated motivation of cognitive embeddings and the cumbersome notation introduced.

To address this concern, we have revised the methodology to explain CogRouter through its complete routing and training process, introducing the main operations and their notation in a more progressive manner.

The revised methodology first provides an overview of how a query is represented, matched with candidate models, and converted into a routing decision. It then explains the frozen query representation and its learned routing projection, followed by the construction of candidate representations from learned identities and encoded descriptions. The query-dependent fusion mechanism and matching score are subsequently introduced to explain how model suitability is estimated. Finally, the Tied-best training procedure describes how recorded response quality determines the preference comparisons used for optimization.

We have also clarified several technical details that were insufficiently explained in the original version. In particular, the query representation extracted from the frozen encoder is explicitly distinguished from the learned projection used for routing, and the candidate identity, description, and query gate are defined alongside their corresponding equations. For the training objective, we now introduce the concrete example of candidate qualities (1, 1, 0) before presenting the general preference sets and pairwise loss, making the handling of equally best candidates more intuitive.

Furthermore, Figure 2 provides an overview connecting query representation, candidate representation and fusion, matching scores, and Tied-best supervision. This enables readers to follow the relationship between the architectural components and the routing objective without relying solely on the mathematical definitions.

These improvements are reflected in Section 4.1 (Lines 194–203) for the overall process, Section 4.2 (Lines 206–215 and 241–243) for query representations, Section 4.3 (Lines 246–259) for candidate representations and fusion, Section 4.4 (Lines 263–273) for suitability scoring and routing inference, and Section 4.5 (Lines 275–299) for Tied-best supervision. The complete architectural overview is presented in Figure 2 (Page 5, Lines 216–239). We believe these revisions provide a clearer and more accessible explanation of the relationship between the motivation, model architecture, supervision objective, and routing inference process.

Q. Current form of the paper needs major restructuring and thought before being considered independently for main conference track. All questions and concerns mentioned in the weaknesses. Thank you.

We sincerely thank the reviewer for the comprehensive assessment and constructive suggestions. We recognize that the concerns regarding novelty, methodological positioning, and presentation are closely related, and we have carefully revised the manuscript to address them. The revision more clearly distinguishes CogRouter from prior model embedding, internal representation, and routing approaches, while narrowing our contribution claims to the specific suitability-learning formulation and supervision design. We have also strengthened the empirical evidence through controlled comparisons of query representations, supervision objectives, and candidate-fusion strategies. These additional experiments help clarify which aspects of the proposed framework contribute to routing quality and provide a more precise basis for evaluating its effectiveness.

Furthermore, we have improved the methodology exposition by clarifying the distinction between frozen representations and learned routing components, explaining the preference construction with a concrete example, and presenting the complete routing process more systematically.

These changes are reflected in Sections 1–2 (Lines 68–140), Section 4 (Lines 191–299), Section 5.5 (Lines 427–524), and Appendices G–I (Lines 1728–1878).

Finally, we sincerely appreciate the reviewer's detailed feedback, which has helped us clarify the methodological contributions, strengthen the supporting experiments, and improve the manuscript's presentation. We hope that these revisions and additional results provide a clearer basis for assessing CogRouter's technical contributions and effectiveness, and we respectfully ask the reviewer to reconsider the overall evaluation in light of the revision.

Reviewer_bKxVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CogRouter to address the limitations of conventional semantic embeddings in capturing the cognitive capabilities required by a query. By extracting LLM hidden states to identify the required capabilities and leveraging the historical performance of candidate models, CogRouter selects the most suitable model, improving response quality while balancing inference costs. Additionally, the paper introduces a tie-aware pairwise training strategy to handle multiple equally optimal candidate models. Extensive experiments on two LLM routing benchmarks demonstrate its effectiveness.

Critical strengths

S1. CogRouter adopts a relatively simple and easy-to-follow architecture, making the overall methodology clear and straightforward to understand.

S2. CogRouter demonstrates strong performance on two LLM routing benchmarks, outperforming or achieving competitive results against various existing routing methods.

Critical weaknesses

W1. Insufficient alignment between motivation and methodology, with limited novelty. The paper defines the final-layer hidden states of the frozen Qwen3-Embedding-0.6B as Cognitive Embeddings, but does not clearly distinguish them from conventional Semantic Embeddings. Moreover, the model identity representations, description representations, and dynamic gated fusion largely rely on existing techniques, without an explicit mechanism for modeling cognitive capabilities. Consequently, the paper does not sufficiently explain how its approach achieves Cognitive Routing or what distinguishes it from prior methods.

W2. Lack of a controlled comparison between Semantic and Cognitive Embeddings. The paper lacks a critical controlled experiment directly comparing Semantic and Cognitive Embeddings. Since existing comparisons involve simultaneous differences in encoders, matching architectures, and training objectives, it remains unclear whether the performance gains can be attributed to the proposed Cognitive Embeddings.

Questions for authors

Q1. How does CogRouter capture the cognitive capabilities required by a query, and what unique capabilities does it offer compared to existing routing methods?

Q2. The prestudy visualizes (fig 1) the original frozen Semantic Embeddings against the supervised CogRouter Matching Score Vectors rather than directly comparing the two input representations. Could the authors provide a direct visualization comparison between Semantic and Cognitive Embeddings?

Q3. Could the authors conduct a controlled experiment that replaces only the Cognitive Embeddings (i.e., the final-layer hidden states of the frozen Qwen3-Embedding-0.6B) with conventional Semantic Embeddings, while keeping all other components unchanged?

Q4. CogRouter's cross-domain generalization remains insufficiently validated. Could the authors conduct experiments in which CogRouter is trained on tasks from one domain and evaluated on tasks from different domains?

3 comments
AuthorsOpenReview ↗

Response to Reviewer bKxV—W1 & Q1: Motivation, Cognitive Routing, and Methodological Contributions (1/3)

W1. Insufficient alignment between motivation and methodology, with limited novelty. The paper defines the final-layer hidden states of the frozen Qwen3-Embedding-0.6B as Cognitive Embeddings, but does not clearly distinguish them from conventional Semantic Embeddings. Moreover, the model identity representations, description representations, and dynamic gated fusion largely rely on existing techniques, without an explicit mechanism for modeling cognitive capabilities. Consequently, the paper does not sufficiently explain how its approach achieves Cognitive Routing or what distinguishes it from prior methods.

Q1. How does CogRouter capture the cognitive capabilities required by a query, and what unique capabilities does it offer compared to existing routing methods?

We thank the reviewer for highlighting the need to better align our motivation with the proposed methodology and clarify the contributions of CogRouter.

Cognitive-Capability-Aware Routing. Semantic similarity does not necessarily reflect the capabilities required to solve a query. For instance, a financial definition question and a multistep financial calculation may be semantically related but demand different reasoning capabilities. CogRouter addresses this discrepancy by learning query-dependent model suitability from historical answer-quality feedback, rather than relying solely on semantic proximity.

Specifically, the frozen Qwen3-Embedding-0.6B provides an initial 1024-dimensional semantic representation, which is transformed into a 128-dimensional routing representation through a quality-supervised projection. Historical performance feedback guides the query projection, candidate representations, and matching mechanism to estimate the suitability of each candidate model. Candidate identity representations encode learned model-specific information, while description representations provide complementary information through query-dependent fusion. Importantly, we clarify that the frozen encoder does not inherently produce a distinct cognitive embedding; rather, capability-relevant routing information is learned through quality-supervised representation adaptation and matching.

Methodological Contributions. While identity representations and gated fusion build upon established techniques, CogRouter integrates them into a quality-supervised routing framework that learns query–model suitability without predefined capability labels. An additional contribution is Tied-best supervision, which explicitly handles multiple equally optimal candidates by comparing each observed best candidate against strictly worse candidates without imposing arbitrary preferences among tied-best models. For example, given candidate qualities (1, 1, 0), the supervision constructs comparisons (1, 3) and (2, 3), while leaving candidates 1 and 2 unordered.

Accordingly, the methodological distinction lies in learning performance-oriented query–model relationships through routing-specific supervision, rather than introducing a fundamentally new frozen embedding architecture or relying on manually defined cognitive capability dimensions. We have clarified this distinction and refine the corresponding terminology in the revision.

AuthorsOpenReview ↗

Response to Reviewer bKxV—W2 & Q3: Controlled Representation Comparisons (2/3)

W2. Lack of a controlled comparison between Semantic and Cognitive Embeddings. The paper lacks a critical controlled experiment directly comparing Semantic and Cognitive Embeddings. Since existing comparisons involve simultaneous differences in encoders, matching architectures, and training objectives, it remains unclear whether the performance gains can be attributed to the proposed Cognitive Embeddings.

Q3. Could the authors conduct a controlled experiment that replaces only the Cognitive Embeddings (i.e., the final-layer hidden states of the frozen Qwen3-Embedding-0.6B) with conventional Semantic Embeddings, while keeping all other components unchanged?

We thank the reviewer for emphasizing the importance of controlled comparisons to isolate the contribution of query representations.

Semantic Embedding Equivalence. We first verified that CogRouter's original frozen query representation follows the standard Qwen3-Embedding-0.6B extraction procedure, using final-layer last-valid-token pooling and L2 normalization. Both extraction paths produce identical representations for all 7,299 RouterBench queries. Therefore, directly replacing the original frozen input with the corresponding standard semantic embedding, as suggested in Q3, leaves the input features unchanged. The distinction lies in the subsequent quality-supervised representation learning rather than in an inherently different frozen embedding.

Controlled Representation Comparisons. To further examine the influence of frozen query representations, we have conducted controlled experiments varying only the extracted query features while keeping the encoder weights, candidate representations, matching architecture, supervision objective, initialization, and training budget fixed.

Query Representation (Generic)Quality (%)
Standard Semantic Embedding (Final Layer)78.021 ± 0.124
Intermediate-Layer Embedding (Layer 14)79.983 ± 0.982
Final Block Before RMSNorm77.848

The intermediate-layer representation improves routing quality by 1.962 percentage points over the standard final-layer embedding under identical downstream settings, while removing the final RMSNorm does not reproduce this improvement. These controlled results demonstrate that the choice of frozen query features influences the routing-relevant information available to the matching model.

Clarification of Routing-Specific Representations. CogRouter's frozen input is a standard semantic embedding, while the learned routing representation is obtained through quality-supervised projection and candidate matching. Accordingly, the representation comparisons isolate the effect of frozen feature selection, whereas routing-specific adaptation arises from learning with observed candidate-model quality. We have clarified this distinction and refined the corresponding terminology in the revision.

AuthorsOpenReview ↗

Response to Reviewer bKxV— Q2: Direct visualization comparison & Q4: Cross-Domain Generalization (3/3)

Q2. The prestudy visualizes (fig 1) the original frozen Semantic Embeddings against the supervised CogRouter Matching Score Vectors rather than directly comparing the two input representations. Could the authors provide a direct visualization comparison between Semantic and Cognitive Embeddings?

We thank the reviewer for this constructive suggestion. To address this concern, we include a direct visualization comparison between the original semantic embeddings and the learned routing representations in Figure 10 of this revision providing a clearer illustration of how quality-supervised learning affects the representation space .

Q4. CogRouter's cross-domain generalization remains insufficiently validated. Could the authors conduct experiments in which CogRouter is trained on tasks from one domain and evaluated on tasks from different domains?

We thank the reviewer for the valuable suggestion regarding cross-domain generalization. To further evaluate CogRouter's transferability, we have conducted additional source-to-target experiments covering all 20 directed transfer settings among five xRouteBench datasets, using the same 18 candidate models.

Training and checkpoint selection are performed exclusively on the source dataset, without using target-domain training or validation data. We compare CogRouter with MFRouter, RouterDC, and GraphRouter under a shared candidate pool and aligned source-data splits. Representative results are summarized below.

Source → TargetCogRouterMFRouterRouterDCGraphRouter
LoCoMo → Generic71.7159.2271.3066.18
TimeSeries → Generic72.0455.1270.6566.54
Geometry3k → MathVista45.6735.0042.0039.67

CogRouter achieves the highest routing quality among the compared methods in these three representative transfer settings, outperforming RouterDC by 0.41, 1.39, and 3.67 percentage points, respectively. Since both training and checkpoint selection rely exclusively on source-domain data, these results support the transferability of learned query–model suitability relationships to unseen target datasets under a shared candidate pool. We have supplemented the complete results for all 20 transfer directions in the revision.

Reviewer_QDPAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work focus on LLM routing task, propose a new routing method named CogRouter. Motivated by the gap between semantic similarity and routing requirements, CogRouter maps LLM-derived cognitive embedding of queries with hybrid candidate representations, which are combinations of capability descriptions and embeddings learned from historical performance. A tie-aware pairwise objective preserves all tied-best candidates without imposing artificial preferences among these candidates. Experiments on xRouteBench and RouterBench show relative routing-quality improvements of 13.81% and 20.09%, obtain the best performance comparing with 11 baselines. Besides, well-designed ablation studies further support the benefit of retaining tied-best candidates.

Critical strengths
  1. This work propose a simple yet effective method and a reasonable training objective that preserves multiple equally valid model choices. Experiments on two routing benchmarks show the effectness and robustness of this method.
  2. This work highlight an undiscovered issue in routing problem. Previous works collapse equally best candidates into a single target, which leads to a loss of valid routing alternatives. Well-designed ablation studies verify this issue and provide empirical support for addressing this issue.
Critical weaknesses
  1. The distinction between semantic embeddings and cognitive embeddings is insufficiently established. In experimental settings, the method uses a frozen text-embedding model without controlled evidence isolating a distinct cognitive signal, which cannot support the effectiveness of cognitive signals.

  2. The discussion of performance gains is lack. Differences in training protocols between CogRouter and the existing baselines make it difficult to isolate the contribution of the proposed components.

  3. The cost of routing models are lacked in reported result, which makes the end-to-end cost of proposed method unclear.

Questions for authors
  1. What is the difference between proposed cognitive embeddings and standard semantic embeddings? Can authors provide controlled comparisons and experiments to clarify such difference?
  2. What are the online routing latency and total serving cost of the end-to-end inference, considering the query encoding and matching?
  3. Under a matched training setup, does tied-best supervision improve on existing multi-positive objectives? Are the modest gains from description gating statistically significant?
3 comments
AuthorsOpenReview ↗

Response to Reviewer QDPA — W1 & Q1: Cognitive Representations and Controlled Comparisons(1/3)

W1. The distinction between semantic embeddings and cognitive embeddings is insufficiently established. In experimental settings, the method uses a frozen text-embedding model without controlled evidence isolating a distinct cognitive signal, which cannot support the effectiveness of cognitive signals.

Q1. What is the difference between proposed cognitive embeddings and standard semantic embeddings? Can authors provide controlled comparisons and experiments to clarify such difference?

We thank the reviewer for highlighting the need to clarify the distinction between semantic and cognitive representations and to provide controlled evidence of their effectiveness.

Representation Mechanism. Conventional semantic embeddings primarily capture semantic relationships between queries, whereas model routing requires identifying which candidate models can effectively satisfy the capabilities demanded by each query. CogRouter uses a frozen Qwen3-Embedding-0.6B to obtain an initial query representation xqx_q, followed by a quality-supervised projection zq=Wqxqz_q=W_qx_q. Historical answer-quality feedback guides the learned representation and matching mechanism toward query–model suitability. Thus, routing-specific information arises from quality-supervised learning rather than assuming that the frozen encoder inherently produces a distinct cognitive representation.

Controlled Representation Comparisons. We have conducted controlled comparisons by varying only the extracted query features while keeping the encoder weights, candidate representations, matching architecture, objective, and training budget fixed.

Query Representation (Generic)Quality (%)
Standard final layer78.021 ± 0.124
Layer 1479.983 ± 0.982
Before final RMSNorm77.848

The intermediate layer-14 representation improves routing quality by 1.962 percentage points over the standard final-layer output, whereas removing the final RMSNorm does not yield a similar improvement.

We have further conducted equal-capacity fusion experiments to examine whether internal representations provide additional routing-relevant information.

DatasetStandard + StandardStandard + InternalStandard + Shuffled
Generic77.65980.53776.911
RouterBench78.24278.33178.204

On Generic, incorporating real internal features improves quality by 2.878 percentage points over duplicated standard features, while shuffled features do not reproduce this gain.

To further isolate representation quality from learned matching, we have evaluated frozen representations using the same cosine kNN routing rule (k=10k=10), without a learned matching head. Internal layers are selected using validation data.

DatasetStandard EmbeddingInternal LayerImprovement
Generic73.55176.118+2.567
RouterBench72.76773.383+0.616

These controlled comparisons indicate that selected internal representations contain additional routing-relevant information beyond standard final-layer embeddings. Together with quality-supervised projection, they support CogRouter's ability to learn representations suited to candidate-model selection rather than relying solely on generic semantic similarity.

AuthorsOpenReview ↗

Response to Reviewer QDPA —W2 & Q3: Training Fairness, Supervision Effectiveness, and Statistical Significance(2/3)

W2. The discussion of performance gains is lack. Differences in training protocols between CogRouter and the existing baselines make it difficult to isolate the contribution of the proposed components.

Q3. Under a matched training setup, does tied-best supervision improve on existing multi-positive objectives? Are the modest gains from description gating statistically significant?

We thank the reviewer for raising concerns regarding the sources of performance improvements, training fairness, and statistical reliability. We have clarified the baseline evaluation protocols and provided controlled comparisons and statistical analyses.

Training Fairness. We follow the released method-specific configurations from LLMRouter (Appendix C). Since kNN, RouterDC, and GraphRouter employ different learning mechanisms, enforcing identical training procedures could disadvantage particular methods. We therefore preserve their respective training protocols while maintaining consistent evaluation queries, candidate pools, and metrics.

Tied-best Supervision. To isolate the contribution of supervision, we fix the query and candidate representations, matching architecture, initialization, query order, optimizer, batch size, training budget, and checkpoint selection, varying only the supervision objective. We compare Tied-best against Quality BCE and Multi-positive objectives. Quality BCE predicts recorded candidate quality, whereas Multi-positive uses the same positive and negative candidate sets as Tied-best but adopts a different loss aggregation strategy.

Encoder / DatasetTied-bestQuality BCEMulti-positive
Qwen / LoCoMo27.37426.16827.293
Qwen / Geometry3k43.71640.43743.716
BGE / Geometry3k45.90242.07745.355

Tied-best consistently improves over Quality BCE by 1.206–3.825 percentage points and matches or outperforms Multi-positive across all three reported configurations. Unlike Multi-positive loss aggregation, Tied-best explicitly compares each observed best candidate against strictly worse candidates without imposing preferences among equally optimal models. These controlled results support the effectiveness of the proposed supervision formulation under matched training conditions.

Description Gating Significance. We have additionally conducted 10,000 paired resamples of evaluation queries, conditional on three fitted models, to assess the statistical reliability of description gating. Results on Generic are summarized below.

ComparisonImprovement (pp)Conditional 95% CI
Dynamic vs. identity-only+0.651[0.151, 1.141]
Dynamic vs. global gate+0.158[-0.189, 0.504]

The complete description/gating module improves quality over identity-only routing by 0.651 percentage points, with a conditional confidence interval excluding zero. However, the additional improvement of dynamic gating over global gating is not statistically established, as its confidence interval includes zero. This distinction supports the contribution of description information while acknowledging that the incremental advantage of dynamic gating remains uncertain.

The accompanying revision now clarifies the evaluation protocols and incorporates these controlled comparisons and statistical analyses in the revision.

AuthorsOpenReview ↗

Response to Reviewer QDPA — W3 & Q2: Online Routing Latency and Serving Cost(3/3)

W3. The cost of routing models are lacked in reported result, which makes the end-to-end cost of proposed method unclear.

Q2. What are the online routing latency and total serving cost of the end-to-end inference, considering the query encoding and matching?

We thank the reviewer for emphasizing the importance of routing overhead in practical deployment. We have conducted additional measurements of the online routing process and estimate the combined serving cost, accounting for both router computation and candidate-model invocation.

Online Routing Latency. We evaluated CogRouter on a single RTX 4090 using FP32 precision, batch size one, and three timing rounds. Results on RouterBench are summarized below.

MeasurementLatency (ms)
Query encoding (mean)27.183
Candidate matching (mean)0.350
Full routing path (mean)28.248
Full routing path (P95)30.993

CogRouter completes an online routing decision in 28.248 ms on average, including query encoding, tokenization, data transfers, and candidate matching. Candidate matching contributes only 0.350 ms, while the average routing overhead across six evaluation settings ranges from 27.108 to 31.049 ms.

Serving Cost. We have further estimated the combined serving cost for 1,000 RouterBench requests as: Cost1000=2.541887+0.0078466(P/u) USD,\mathrm{Cost}_{1000}=2.541887+0.0078466(P/u)\ \mathrm{USD},

where the first term represents historical candidate-model invocation costs, PP denotes the hourly device-and-host price, and uu denotes effective utilization. The second term explicitly accounts for router computation.

These results quantify the additional computational overhead introduced by CogRouter and its contribution to total serving cost. The reported latency measures the routing process rather than complete answer generation and excludes candidate-model generation, network delays, and queueing. The cost estimate incorporates historical candidate-model invocation costs together with router computation. We have included these measurements and clarified their scope in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.