- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies zero-shot, cost-aware model routing for multi-turn spreadsheet agents, with the routing decision made at the user-turn boundary. It varies four things:
- the decision rule: zero-shot LLM router, escalation, de-escalation, or random;
- what the router sees: the current request only, or the session history;
- the candidate pair: GPT-5.6 Sol/Luna, Sol/Haiku 4.5, or Opus 4.8 at high/low effort;
- prompt-cache behavior.
Experiments use multi-turn versions of SheetCopilot (112 tasks, 2.88 turns on average) and SpreadsheetBench 2 (221 tasks, 4.76 turns on average). The main findings are:
- At a matched offloading rate, giving the router session history raises pass rate by 3.3 points on SheetCopilot.
- Price per token is a poor proxy for cost per task. Haiku’s cost advantage over Sol shrinks from 6.8× to 3.2× on longer sessions.
- A closed-form cache analysis shows that switching down to the cheaper model recovers its cache cost within about 1.25 calls, while switching up never breaks even.
Critical strengths
- I like the fact that this paper is addressing the very practical constraints of cache hits in model routing in multi-turn trajectories, and I also like that they are building a very systematic approach to routing where they consider several axes of routing choices.
- The motivation for zero-shot routers in agentic settings is convincing. Labels require running full tool loops under each candidate, the number of trajectories grows as 2^T, and candidate models change every few months.
- The asymmetric cost of switching models is a good observation / result to make. The break-even expression k* (Eq. 3) is simple and gives actionable guidance: switch down early and switch up rarely.
- The observation that a model cheap per token is not necessarily cheap per task (Fig. 3) is practically useful. Single-turn routing evaluations cannot reveal it.
Critical weaknesses
- Are the authors citing enough prior literature on model routing? Relevant missing works include xRouter, Router-R1, Adaptive LLM Routing under Budget Constraints, Online Multi-LLM Selection via Contextual Bandits, Federate the Router, ProxRouter, Beyond Accuracy and Cost (latency-aware routing), and Not All Turns Are Equally Hard (adaptive thinking budgets). These prior works bring me to question if the contributions of formulating a cost-aware router at multi-turn boundaries and routing being a decision making problem listed in the introduction are genuinely novel contributions.
- Only two evaluation benchmarks seems a little too small, and both being on spreadsheet agents seems a little weird – the paper definitely needs broader evaluation. Results are also single runs with no confidence intervals, and run-to-run noise is comparable to the reported gains.
- I want to understand if the evaluations rigorously study the cache misses due to model changes mid-trajectory phenomenon. From my reading, they do not as the cache analysis is a closed-form calculation (Eq. 3), with no measured cache hit rates.
- I am not sure all the axes (including routing rule, access to session history, candidate models that differ in size, reasoning effort, or model provider, and prompt-cache behavior) are evaluated effectively. For example, the appendix prompts contradict the definitions of escalation and de-escalation in Sec. 2. Also, the “history-aware” router sees only earlier user requests.
- The main router does not reduce cost - it costs more than always using the expensive model in all six pair × benchmark settings (Tables 3, 4).
- [Minor] It is not professional that the appendix is not formatted properly. Appendices B and C are empty headings, and numbers disagree between the text, figures and tables.
Questions for authors
- What exactly was run under “escalation” and “de-escalation”? Please reconcile Sec. 2 and Fig. 1b with Figs. 7–8.
- What differs in the “separate harness” used for escalation, de-escalation and random routing?
- How were the routed costs for Sol/Haiku and Sol/Luna computed? Is the router call included?
- Why does the history-aware router cost more than expensive-only in every cell?
- The motivation for zero-shot routers is convincing, but I wonder if continually learnt routers also make sense in this case?