- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies why static LLM routers can predict model performance without improving on a strong default model. It defines the oracle improvement over the default and decomposes the routing gains into switch frequency and the average gain per switch. Experiments on RouterBench compare query-only readers, prefill probes, and two cascade evaluation protocols. The main finding is that predictive improvements often fail to produce better routing decisions.
Critical strengths
The focus on actual gains over a stated deployment baseline is useful. Reporting switch frequency and gain per switch makes router behavior easier to interpret, and the authors explicitly report negative results and limitations of their oracle-aware experiments.
Critical weaknesses
-
Several central conclusions do not follow from the experiments. In Table 2, with the stated shared Ridge setup, predicting gives , so the selected model is unchanged by construction. This does not establish that changing the training target fails to improve decisions. Likewise, Section 6.2 uses a break-even precision derived for random candidate selection to explain a probe-selected cascade. The actual switch-conditioned gains and losses are needed, and selecting the trigger threshold based on balanced accuracy does not optimize the routing value.
-
The main cascade comparison and supporting results need substantial clarification. Table 3 attributes the difference to trigger information, but compares a nonrepresentative 5,000-query subset with the full pool. Their prior qualities and detection difficulty differ substantially. There are also unresolved numerical inconsistencies: Table 12 reports an aligned-subset prior near 0.828, whereas elsewhere it is approximately 0.717; and Table 19's displayed rows average approximately 0.711/0.912 for its prior/oracle proxies, rather than the stated 0.500/0.747.
Questions for authors
- What aspect of Table 2's implementation breaks Ridge's linearity in the targets? If none does, what evidence supports the claim that decision-oriented supervision was tested meaningfully?
- On identical held-out queries, does the oracle-aware versus leakage-free difference persist? What happens when thresholds optimize routed quality within the training folds, and what break-even precision follows from the actual probe-selected switches?
- Can you reconcile the prior values in Tables 10 and 12 and the aggregates in Table 19? Please also specify the tie convention required by Eq. (3): a prior can be tied for best while differing from the chosen argmax, and switching to a tied model causes no quality loss.
- What new insight does this work establish beyond the cited The Routing Plateau, which already discusses limited query-specific prediction and routing changes whose gains and losses cancel? Please clarify what the decision-gap framework adds and which result demonstrates that contribution.