- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents the idea of cognitive embeddings and how they can help in routing of language queries across models. Authors identify that sentence embedding based, tag based or LLM based routing approaches have some undesirable aspects, and how cognitive embeddings can be used for overcoming them. A motivating prestudy is presented, and the method is described after this motivating example. Experiments are then discussed where Cogrouter shows improvements in performance.
Critical strengths
- The presentation of cognitive embeddings gives new insight into understanding routing.
- Experimental results are reasonably detailed.
- The motivating prestudy gives insights into failure modes of existing routing approaches.
Critical weaknesses
This paper has multiple and significant weaknesses.
-
This paper's approach uses arguments and ideas presented in bits and pieces across literature, and it is hard to find novelty after accounting for surrounding literature. For example, the notion of "cognitive embeddings" for routing is not new, and many papers have examined very similar ideas across literature. Model Embedding Literature (EmbedLLM, IRT-Net, LLM-DNA, IRT-Router,) already try to build model embeddings ("cognitive embedding") to use them across multiple domains and not just routing. Similarly internal model representations have been used across routing and uncertainty quantification literature to identify query-model mappings.
-
The idea of tie-aware pairwise ranking is not new, even in routing literature. Ranking loss and contrastive learning literature has examined this in 2000s. Furthermore, even independent correctness BCE loss (instead of pairwise) identically treats equal scoring models instead of enforcing and order over them. The contrastive loss in RouterDC also suffices for this purpose.
-
"CogRouter is the first to combine query-gated model capability representations with tied-best pairwise supervision for LLM routing.": From 1 and 2, this is not novel. model capability representations are already existing in literature. query-gating has been studied across multiple domains. tied-best pairwise supervision is oversold as a contribution.
-
the methodology section can be significantly improved in its exposition, and currently it is very hard to parse; partly due to the complicated motivation of cognitive embeddings and the cumbersome notation introduced.
Learning to Rank using Gradient Descent, Burges et al, 2005.
LLM Router: Rethinking Routing with Prefill Activations Tanay Varshney et al 2026.
Query-Level Uncertainty in Large Language Models — Chen et al., 2025.
LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models — Ratnakar et al., 2026.
ETC and other papers in literature mentioned above.
Questions for authors
Current form of the paper needs major restructuring and thought before being considered independently for main conference track.
All questions and concerns mentioned in the weaknesses. Thank you.
4 comments
Response to Reviewer xhHD—W2. Novelty and effectiveness of tie-aware pairwise supervision (2/4)
W2. The idea of tie-aware pairwise ranking is not new, even in routing literature. Ranking loss and contrastive learning literature has examined this in 2000s. Furthermore, even independent correctness BCE loss (instead of pairwise) identically treats equal scoring models instead of enforcing and order over them. The contrastive loss in RouterDC also suffices for this purpose.
We thank the reviewer for highlighting the connections between our training objective and established ranking, binary classification, and contrastive learning approaches. We apologize that the original manuscript did not clearly distinguish the established ranking formulation from the routing-specific supervision construction employed by CogRouter.
We recognize that logistic pairwise ranking is a well-established technique and that BCE and contrastive objectives can also accommodate multiple candidates with equal quality labels. The contribution of our supervision design lies in how observed query-level quality feedback is converted into preference comparisons and balanced during optimization, rather than in introducing pairwise ranking or tie handling as new principles.
Specifically, for each query, CogRouter identifies all candidates achieving the highest recorded quality and compares each of them against every strictly lower-quality candidate, without enforcing an ordering among equally best candidates. For example, given candidate qualities (1, 1, 0), the resulting comparisons are (1, 3) and (2, 3), leaving candidates 1 and 2 unordered. Furthermore, pairwise losses are normalized within each query, ensuring that queries with different numbers of valid comparisons contribute equally to the training objective.
To directly address the reviewer's concern regarding alternative supervision objectives, we have conducted additional controlled experiments comparing Tied-best supervision with Quality BCE and Multi-positive supervision. Within each encoder condition, these experiments use identical query and candidate representations, matching architectures, initialization, optimization settings, training budgets, and checkpoint-selection procedures. Only the supervision objective is changed.
The results are summarized below.
| Encoder / Dataset | Tied-best | Quality BCE | Multi-positive |
|---|---|---|---|
| Qwen / LoCoMo | 27.374% | 26.168% | 27.293% |
| Qwen / Geometry3K | 43.716% | 40.437% | 43.716% |
| BGE / Geometry3K | 45.902% | 42.077% | 45.355% |
As shown above, Tied-best supervision consistently achieves higher mean routing quality than Quality BCE across the three evaluated settings, with improvements ranging from 1.206 to 3.825 percentage points. It also matches or outperforms Multi-positive supervision in all three cases. These results show that, even when the representations and training configurations remain fixed, the construction and aggregation of preference comparisons can affect routing performance. Moreover, the controlled fixed-winner experiment further supports the benefit of retaining all valid best candidates: replacing a single fixed winner with Tied-best supervision improves five-domain macro quality from 53.03% to 53.72% under matched training conditions.
We have expanded the explanation of preference construction, query-level normalization, and the corresponding training objective in Section 4.5 (Lines 275–299). The fixed-winner comparison is reported in Section 5.3 and Table 2 (Lines 340–349 and 378–383). The additional matched-objective experiments, numerical results, and analysis are presented in Section 5.5 and Table 6 (Lines 489–508), with detailed experimental configurations and uncertainty estimates in Appendix I (Lines 1836–1867, Table 18).
Response to Reviewer xhHD—W3. Clarification of the methodological contribution and novelty claims (3/4)
W3. "CogRouter is the first to combine query-gated model capability representations with tied-best pairwise supervision for LLM routing.": From 1 and 2, this is not novel. model capability representations are already existing in literature. query-gating has been studied across multiple domains. tied-best pairwise supervision is oversold as a contribution.
We would like to clarify that CogRouter does not claim novelty for model capability embeddings, query-dependent gating, or pairwise ranking as independent techniques. Instead, our work addresses two interconnected characteristics of LLM routing: semantic similarity does not necessarily determine model suitability, and a query may have multiple equally optimal candidate models.
CogRouter addresses these characteristics through a unified suitability-learning formulation. Candidate representations combine empirical performance identities with semantic descriptions, while query-conditioned matching estimates candidate suitability. The training objective then uses observed response quality to construct best-versus-strictly-worse preferences, preserving multiple valid routing choices instead of reducing each query to a single target.
The methodological contribution is therefore centered on the formulation of query–candidate suitability learning and the supervision strategy used to optimize it, rather than the novelty of the constituent techniques in isolation.
To provide empirical evidence for these design choices, we have strengthened the controlled analysis of supervision and candidate representations. The five-domain ablation results are summarized below.
| Configuration | Five-domain macro quality |
|---|---|
| Identity-only + fixed winner | 53.03% |
| Identity-only + tied-best | 53.72% |
| Tie + gated descriptions | 53.88% |
These results show that preserving all tied-best candidates improves macro routing quality by 0.69 percentage points under the matched fixed-winner control. Adding query-gated descriptions further improves the mean from 53.72% to 53.88% in this experiment.
We have additionally examined description fusion under controlled training conditions. On Generic, the query-gated description model achieves 78.021%, compared with 77.370% for the identity-only configuration, yielding a gain of 0.651 percentage points, with a conditional 95% confidence interval of [0.151, 1.141]. The comparison against a learned global gate yields a smaller increment; therefore, we position query-dependent gating as a component of the suitability-matching design rather than claiming it as an independently established innovation.
The effectiveness of the complete framework is further supported by the main benchmark comparisons. CogRouter achieves 53.98% five-domain macro quality on xRouteBench, compared with 47.43% for the strongest reported baseline, RouterDC. On RouterBench, CogRouter reaches 78.20%, compared with 65.12% for GraphRouter, the strongest reported baseline in that evaluation.
Together, these results provide evidence for the effectiveness of the proposed suitability-learning framework and its supervision design, while the controlled analyses clarify the contributions of the individual components.
To address the reviewer's concern about overstated novelty, we have refined the contribution statements in Section 1 (Lines 68–94) and clarified the methodological positioning in Section 2.2 (Lines 122–140). The model representation and matching mechanism are detailed in Sections 4.3–4.4 (Lines 244–273). The relevant experimental evidence is presented in Section 5.2 (Table 1 and Figure 3, Lines 333–376), Section 5.3 (Tables 2–3, Lines 340–400), and Section 5.5 (Lines 489–508), with the additional candidate-fusion controls reported in Appendix I (Lines 1869–1878).
Response to Reviewer xhHD— W4. Methodology presentation and notation clarity & Q. Overall concerns (4/4)
W4. the methodology section can be significantly improved in its exposition, and currently it is very hard to parse; partly due to the complicated motivation of cognitive embeddings and the cumbersome notation introduced.
To address this concern, we have revised the methodology to explain CogRouter through its complete routing and training process, introducing the main operations and their notation in a more progressive manner.
The revised methodology first provides an overview of how a query is represented, matched with candidate models, and converted into a routing decision. It then explains the frozen query representation and its learned routing projection, followed by the construction of candidate representations from learned identities and encoded descriptions. The query-dependent fusion mechanism and matching score are subsequently introduced to explain how model suitability is estimated. Finally, the Tied-best training procedure describes how recorded response quality determines the preference comparisons used for optimization.
We have also clarified several technical details that were insufficiently explained in the original version. In particular, the query representation extracted from the frozen encoder is explicitly distinguished from the learned projection used for routing, and the candidate identity, description, and query gate are defined alongside their corresponding equations. For the training objective, we now introduce the concrete example of candidate qualities (1, 1, 0) before presenting the general preference sets and pairwise loss, making the handling of equally best candidates more intuitive.
Furthermore, Figure 2 provides an overview connecting query representation, candidate representation and fusion, matching scores, and Tied-best supervision. This enables readers to follow the relationship between the architectural components and the routing objective without relying solely on the mathematical definitions.
These improvements are reflected in Section 4.1 (Lines 194–203) for the overall process, Section 4.2 (Lines 206–215 and 241–243) for query representations, Section 4.3 (Lines 246–259) for candidate representations and fusion, Section 4.4 (Lines 263–273) for suitability scoring and routing inference, and Section 4.5 (Lines 275–299) for Tied-best supervision. The complete architectural overview is presented in Figure 2 (Page 5, Lines 216–239). We believe these revisions provide a clearer and more accessible explanation of the relationship between the motivation, model architecture, supervision objective, and routing inference process.
Q. Current form of the paper needs major restructuring and thought before being considered independently for main conference track. All questions and concerns mentioned in the weaknesses. Thank you.
We sincerely thank the reviewer for the comprehensive assessment and constructive suggestions. We recognize that the concerns regarding novelty, methodological positioning, and presentation are closely related, and we have carefully revised the manuscript to address them. The revision more clearly distinguishes CogRouter from prior model embedding, internal representation, and routing approaches, while narrowing our contribution claims to the specific suitability-learning formulation and supervision design. We have also strengthened the empirical evidence through controlled comparisons of query representations, supervision objectives, and candidate-fusion strategies. These additional experiments help clarify which aspects of the proposed framework contribute to routing quality and provide a more precise basis for evaluating its effectiveness.
Furthermore, we have improved the methodology exposition by clarifying the distinction between frozen representations and learned routing components, explaining the preference construction with a concrete example, and presenting the complete routing process more systematically.
These changes are reflected in Sections 1–2 (Lines 68–140), Section 4 (Lines 191–299), Section 5.5 (Lines 427–524), and Appendices G–I (Lines 1728–1878).
Finally, we sincerely appreciate the reviewer's detailed feedback, which has helped us clarify the methodological contributions, strengthen the supporting experiments, and improve the manuscript's presentation. We hope that these revisions and additional results provide a clearer basis for assessing CogRouter's technical contributions and effectiveness, and we respectfully ask the reviewer to reconsider the overall evaluation in light of the revision.
Response to Reviewer xhHD—W1. Novelty, distinctions from existing model embedding and internal representation-based methods (1/4)
We sincerely thank the reviewer for highlighting these closely related studies. We apologize that our original presentation did not sufficiently clarify the methodological distinctions between CogRouter and existing model embedding and internal representation-based routing approaches.
Regarding model embedding methods, we recognize that EmbedLLM and LLM-DNA have investigated learned model representations, while IRT-Net and IRT-Router characterize model capabilities and query characteristics through ability–difficulty relationships. CogRouter differs in how query–model suitability is estimated. Rather than explicitly parameterizing model abilities and query difficulty, CogRouter learns suitability scores through quality-supervised query projections and candidate representations that combine learned performance identities with semantic capability descriptions. A query-dependent fusion mechanism further adjusts the contribution of candidate descriptions according to the incoming query.
Regarding internal representation-based methods, the previous studies have explored hidden states and model activations for routing and uncertainty estimation. Our contribution does not lie in introducing internal representations themselves. CogRouter instead uses the standard output of a frozen embedding encoder and learns routing-specific suitability through a trainable projection and candidate-matching mechanism supervised by observed response quality. The distinction therefore lies in how query representations interact with candidate-specific capability information to support routing decisions.
To further substantiate the effectiveness of this suitability-learning framework, we provide the following experimental evidence. The main benchmark results demonstrate that CogRouter achieves higher routing quality than the strongest reported baselines under the corresponding evaluation settings.
As shown above, CogRouter improves routing quality by 6.55 and 13.08 percentage points over the strongest reported baselines on xRouteBench and RouterBench, respectively. These results demonstrate the effectiveness of the complete query–candidate suitability-learning framework across two routing benchmarks.
Furthermore, to examine the contribution of candidate capability representations, we have conducted controlled experiments with matched training conditions. On Generic, incorporating query-gated capability descriptions improves routing quality from 77.370% (identity-only) to 78.021%, corresponding to a gain of 0.651 percentage points, with a conditional 95% confidence interval of [0.151, 1.141]. This result supports the value of incorporating candidate description information alongside learned performance identities. The comparison does not, by itself, establish an independent advantage of dynamic gating over a global gate.
Together, these results provide empirical support for CogRouter's suitability-learning design. While the benchmark comparisons demonstrate its overall effectiveness, the controlled experiment further examines the contribution of integrating candidate capability information. We emphasize that these results support our specific methodological design rather than establishing novelty for model embeddings or hidden-state extraction in isolation.
We have clarified the methodological distinctions and refined the contribution statements in Section 1 (Lines 73–94) and Section 2.2 (Lines 115–140). The query representation, candidate fusion, and suitability-matching mechanisms are detailed in Sections 4.2–4.4 (Lines 206–273). The benchmark results are reported in Section 5.2 (Table 1 and Figure 3, Lines 333–376), while the controlled candidate-representation analysis is presented in Section 5.5 (Lines 502–508) and Appendix I (Lines 1869–1878).