acceptodds
Desk rejected at ICLR 2027

Attempt-Matched Evaluation of Model-Routing Cascades in LLM Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Routing between language models inside an agent is motivated by an oracle: send each task to the right model and quality rises ten points or more. Operationalised as a cheap-then-strong reactive fallback cascade, the form cost-aware cascade work uses, that oracle rests on an unequal comparison: the cascade spends two attempts, the always-strong baseline one. The control that isolates routing is retrying that same strong model, and on two independent benchmark families it gives opposite answers. On -bench the routing-specific effect is indistinguishable from zero on most datasets and significantly negative on one; on BFCL's multi-turn split it is significantly positive. An exact identity reconciles them, regrouping the same sum into a first-slot cost plus a rescue term. What differs across families is the strong model's self-rescue rate, how often it succeeds when retried on its own failure (rank correlation over cells): where retrying already recovers failures, switching adds nothing. Where the capability gap is large the first-slot cost dominates instead, and that is why our one negative dataset is negative. A one-sided screen follows: every significantly positive effect we measure sits at a self-rescue rate at or below , which one extra rollout per task estimates. A cascade oracle needs the same-model-retry control, since retry's share of the naive gain reaches times it. And on what survives is price, not quality, at matched budget and without needing a positive effect; where the effect is positive, as on BFCL, the cascade wins on both axes. Two scope limits. The confound comes from spending two attempts, so nothing here bounds a one-call pre-execution dispatch router; its selection-corrected oracle excludes zero on of datasets. And our retry is blind, so a failure-conditioned one could only push these effects down.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, model routing, model cascades, retry baselines, evaluation methodology, measurement confound, off-policy evaluation, tool use, agent benchmarks, reproducibility

Related papers

Discussion (0)

Sign in to comment.