- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Lens-Router learns a query representation through resource-demand supervision, filters candidate models by capability clusters, and then predicts pairwise costs to select a model. The encoder is trained using annotations from three LLMs, while model-specific routing uses 700 queries. Experiments across nine benchmarks and two model pools report improved quality–cost trade-offs over existing routers.
Critical strengths
The distinction between token price and query-dependent realized cost is useful and well motivated. The cost-ranking component also has concrete empirical support: in Table 5, replacing pairwise ranking with direct ranking increases cost from 0.13 to 0.21 at the same reported accuracy. Evaluating a pool of similarly priced models makes this contribution particularly relevant.
Critical weaknesses
-
The justification for capability clustering does not establish that models within a cluster are interchangeable for a given query. Appendix A.1 reports overlapping accuracy intervals and frequent ranking reversals across benchmarks. However, ranking reversals can reflect model specialization, rather than unreliable capability differences. These are precisely the differences a router could exploit. Once capability filtering is complete, Lens-Router selects solely by cost, so this distinction matters directly to answer quality. The definition of cluster correctness is also missing: whether one, most, or all members must answer correctly leads to very different supervision. Without measuring correctness disagreement within clusters and the quality lost through cost-only selection, the central decomposition remains insufficiently justified.
-
The experiments do not isolate the benefit of the proposed resource-demand supervision. The 700-query adaptation budget follows encoder training on 59,577 queries with responses and annotations from three strong LLMs. Amortizing this investment is reasonable, but Table 4 mainly compares demand-supervised encoders with pretrained encoders; it does not distinguish the proposed 24-dimensional representation from the benefit of additional task-relevant supervision. Likewise, the quoted supervision counts for competing routers do not establish their requirements at a comparable quality–cost target. A comparison using the same source responses to train a simpler auxiliary representation, followed by equal-budget adaptation, would directly test the paper’s main claim.
Questions for authors
-
How exactly are and the inference-time capability cutoff defined? Among queries passed to a cluster, how often do its members disagree on correctness, and how often does cost ranking select an incorrect model when a correct candidate remains?
-
With the same ModernBERT backbone and source-response data, how does demand supervision compare with directly predicting response length and model correctness before 700-query adaptation? Does the advantage persist for target models excluded from encoder construction?
Thank you for the response.