- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work studies cost-aware model routing in multi-step agent trajectories, where a router chooses between a cheap and an expensive model at each step and an error at one step can corrupt later steps. It asks how much of the benefit of reacting to failures actually comes from knowing the execution state, rather than from calling the strong model more often or at random positions. The paper answers this in two separate parts. First, in a simplified clean-or-corrupted model, it shows that a router ignoring the state should place strong calls at the end of the trajectory. Second, it proposes StopRoute that checks whether each step’s output runs and is well-formed. On a synthetic pandas data-processing workflow, StopRoute outperforms a cost-matched control that triggers the same fallback at random positions. The overall message is that an adaptive router’s gains should be credited to state information only after the state-blind baseline, including randomization, has been optimized.
Critical strengths
-
The paper points out that gains from adaptive routing are easily confused with gains from spending more or from randomization. This is a useful reminder for anyone evaluating routers.
-
The random-trigger control has its cost calibrated in advance, the test tasks are held out, and every policy is run with fresh model calls and repetitions.
-
The paper reports earlier cohorts that did not meet its own cost criterion and states the limits of its theory openly.
Critical weaknesses
-
The introduction never names StopRoute or explains what it does. After reading the abstract and introduction, it is hard to tell whether this is an analysis paper or a method paper, and how the two parts relate.
-
The theory assumes the router sees the hidden error state for free before each call and always makes a fixed number of calls. StopRoute instead checks outputs after paying for a call, retries, and may stop early. The authors acknowledge this, and their own check finds that the theory’s conditions do not hold on any of the real tasks they examine. The theory therefore does not explain or support the experimental result..
-
A policy that can see the state can always copy a policy that cannot, so it can never be worse. The “certificate” is just the cost of the best state-blind policy minus the cost of some cheaper state-aware policy. The only non-obvious result is that strong calls should go last, which holds only when reward is given at the end. The main numerical example is a single hand-picked three-step case.
-
All results come from one synthetic three-step pandas task with three models. Retrying with another model when an output fails a format or execution check is common practice.
Questions for authors
-
The reproducibility statement mentions a SWE-bench diagnostic, but no SWE-bench result appears in the paper. Was this experiment run and what did it show?
-
The theory assumes the router sees the hidden state for free before each call and makes a fixed number of calls, while StopRoute checks outputs after each call, retries, and may stop early. Section 4.2 also finds that the theory’s conditions do not hold on any of the real tasks examined. Which design choices in StopRoute follow from the theoretical results?
4 comments
Comment 2: The paid-output mechanism and the missing SWE-bench result
How does the post-call policy connect to the analysis? In Section 5, we ask whether the detector sends a fallback to prefixes where that same fallback has greater incremental benefit. This question applies directly to StopRoute's paid-output decision.
For a common paid primary response, let indicate detector rejection, , and be the conditional effect of the configured fallback on final success. With fixed downstream rules and an independent random trigger of probability , Equation (3) gives
Retries and changes in downstream stopping are included in the identity. Appendix O.6 evaluates its terms on the existing 300-task v4 recorded tree:
| Initial detector decision | Tasks | Failures rescued | Successes lost | Mean fallback effect |
|---|---|---|---|---|
| Reject | 39 | 22 | 0 | +56.41 pp |
| Accept | 261 | 34 | 68 | −13.03 pp |
Placement contributes 7.853 points and trigger frequency 0.5948 points, reconstructing the 8.448-point replay difference. Fallback replaces the initial output under the recorded rule. Rejected outputs otherwise stop, accounting for the zero losses in that group. Accepted outputs can be harmed by replacement. Avoiding those harmful replacements accounts for part of the comparison. At equal trigger frequency, placement adds USD 0.0001758 per task through downstream execution, so trigger matching alone leaves a cost difference. These are retrospective quantities conditional on recorded completions.
We test placement independently with fresh calls in Table 2. StopRoute achieves 79.50% success versus 74.50% for random triggering over 200 held-out tasks with three repetitions per policy. The paired gain is 5.00 points, with 95% CI . The random trigger was cost-calibrated on a separate cohort. Its primary assignments and detector stopping match StopRoute's. The 2.67% mean-cost gap meets the prespecified point-estimate tolerance, while statistical cost equivalence remains unresolved. The online comparison estimates conditioned allocation in the frozen configuration, while the recorded-tree decomposition examines placement under its recorded branch rules.
Was SWE-bench run? Yes. The SWE-bench diagnostic was run before submission. We have now added its results to Section 7 and Appendix E, Table 11. The diagnostic pairs one DeepSeek and one Kimi continuation from each of 11 reconstructed development states at round 16. Both models preserved the two initially passing states. Among the nine initially failing states, Kimi rescued one that DeepSeek did not, giving 3/11 successes for Kimi and 2/11 for DeepSeek, with exact two-sided McNemar . Kimi's nominal continuation cost was 9.54 times DeepSeek's. This small paired diagnostic records one Kimi-only rescue. Whether execution signals can identify such cases before continuation and improve success at comparable cost on SWE-bench remains unresolved.
The separate Online-200 experiment estimates feedback-dependent fallback allocation in the frozen DataFlow configuration, with primary model assignments and stopping rules held fixed. It provides the controlled allocation evidence for the three-step case study.
Thank you for the detailed response and the revision. The revised abstract and introduction are much clearer. Reporting the SWE-bench and ALFWorld results in the main text is also helpful, and I appreciate how openly the limitations are stated. My main concern about scope remains. The evidence comes from one synthetic three-step task, the external probes on ALFWorld and SWE-bench are inconclusive, and the comparison with same-model retry is still missing. Given the improved presentation and the added analyses, I will raise my score from 1 to 2.
Thank you for reading the revised paper and for updating your assessment. We appreciate your recognition of the clearer presentation and added analyses. Your suggestions on broader task coverage and additional comparisons are valuable directions.
We followed up on your question about retrying the same model. On the 300-task DataFlow cohort, the first DeepSeek answer triggered fallback on 39 tasks. We could reconstruct 36 paired comparisons using separate DeepSeek and Kimi responses to identical first-step prompts from an earlier cohort. We executed both responses on the target table, shared the recorded second-step response, and obtained fresh Qwen responses for the final step.
Retrying DeepSeek solved 22 of the 36 tasks. Switching to Kimi solved those same 22 plus six others. Mean episode cost on these pairs was 2.19 times as high with Kimi (USD 0.003804 versus 0.001734). Three pairs remain unresolved because the replayed detector decision differed from the log. This helps answer what a change of model adds after a rejection: a simple retry already recovered many cases, and all six cases where the two choices differed favored Kimi.
Comment 1: What the paper contributes and what guides StopRoute
Thanks for the review. We have rewritten the introduction to explain StopRoute and the comparison it implements. With primary model assignments and stopping rules held fixed, does execution feedback help choose when to spend a fallback call? The theory examines the value of state information against an optimized randomized baseline. The DataFlow experiment tests feedback-dependent allocation in an executable workflow. Both the abstract and introduction now describe StopRoute and these roles.
Which design choices follow from the theory? The theory guides how we compare policies. It motivates optimizing the randomized open-loop benchmark, fixing the other decision components in the empirical control, and accounting for expenditure separately from information access. The execution checks and stopping rules are implementation choices, and development data select the cascade configuration. We make these roles explicit in Sections 1 and 5.
What goes beyond policy-class inclusion? We agree that a richer information class can copy a state-blind policy and that subtracting a feasible witness cost is elementary. The result that needs the transmission model is the suffix exchange inequality. Every schedule with a fixed number of strong calls is dominated by a strong suffix. Proposition 1 then reduces optimization over arbitrary randomized schedules to the lower convex envelope of K+1 points.
Optimizing that baseline changes the comparison in Table 1:
The deterministic optimum overshoots the target because schedules are discrete. Randomization alone saves 6.679 cost units. A residual 1.760-unit gap survives after both classes allow randomization. The certificate quantifies that residual gap against the optimized baseline.
How dependent is this on the single example? Appendix O.3 adds 125 parameter combinations under two target rules. At fixed target 0.85, 52 have a positive gap and 27 save at least 10% of randomized open-loop cost. The attainable-midpoint target gives 80 and 45. The grid varies horizon, propagation and prices, keeps the conditional success probabilities fixed, and includes zero-gap cases.
The terminal-reward and homogeneity conditions remain essential. Appendix B.5 examines intermediate rewards. Appendix O.1 gives a two-step counterexample with stepwise strong-model dominance: strong–weak succeeds with probability 0.836 versus 0.374 for weak–strong at equal cost. Section 4.1 now states this boundary. Proposition 2 concerns the specified noisy-signal certificate, so its zero value leaves other state-aware policies open.