acceptodds
Under review as a conference paper at ICLR 2027

When State Information Pays: Cost Separation in Sequential Multi-Model Routing

Abstract

Errors in agent trajectories can change the value of later model calls. We isolate this coupling in a two-state transmission model and compare state-conditioned policies with both deterministic and randomized open-loop schedules. Under capability ordering, the best schedule with a fixed number of strong calls has a strong suffix. This yields an exact deterministic cost in an explicit target window and reduces the randomized optimum to the lower convex envelope of K+1 suffix points. A feasible state policy then gives a matched-class cost-separation certificate. In a controlled instance, randomization reduces the open-loop cost from 12 to 5.32. The exact randomized state-history optimum costs 3.56, leaving a matched information gap of 1.76. A noisy-signal certificate vanishes under an uninformative signal, separating information from exogenous randomization. We evaluate a gold-free detector cascade on a real-execution data workflow. After calibrating a state-independent random trigger on costs alone, we freeze it before testing on 200 disjoint tasks with three fresh repetitions per policy (1,200 episodes). StopRoute reaches 79.50% success versus 74.50%, a 5.00-point paired gain on average, while the nominal-cost gap is 2.67%, inside the prespecified 5% point-estimate tolerance. Together, the exact model analysis and controlled online comparison isolate the value of where additional model calls are allocated, beyond randomization or heterogeneous model assignment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Routing, Agentic Workflows, Value of Information, Model Cascades, State-Conditioned Policies

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aWVeOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work studies cost-aware model routing in multi-step agent trajectories, where a router chooses between a cheap and an expensive model at each step and an error at one step can corrupt later steps. It asks how much of the benefit of reacting to failures actually comes from knowing the execution state, rather than from calling the strong model more often or at random positions. The paper answers this in two separate parts. First, in a simplified clean-or-corrupted model, it shows that a router ignoring the state should place strong calls at the end of the trajectory. Second, it proposes StopRoute that checks whether each step’s output runs and is well-formed. On a synthetic pandas data-processing workflow, StopRoute outperforms a cost-matched control that triggers the same fallback at random positions. The overall message is that an adaptive router’s gains should be credited to state information only after the state-blind baseline, including randomization, has been optimized.

Critical strengths
  • The paper points out that gains from adaptive routing are easily confused with gains from spending more or from randomization. This is a useful reminder for anyone evaluating routers.

  • The random-trigger control has its cost calibrated in advance, the test tasks are held out, and every policy is run with fresh model calls and repetitions.

  • The paper reports earlier cohorts that did not meet its own cost criterion and states the limits of its theory openly.

Critical weaknesses
  • The introduction never names StopRoute or explains what it does. After reading the abstract and introduction, it is hard to tell whether this is an analysis paper or a method paper, and how the two parts relate.

  • The theory assumes the router sees the hidden error state for free before each call and always makes a fixed number of calls. StopRoute instead checks outputs after paying for a call, retries, and may stop early. The authors acknowledge this, and their own check finds that the theory’s conditions do not hold on any of the real tasks they examine. The theory therefore does not explain or support the experimental result..

  • A policy that can see the state can always copy a policy that cannot, so it can never be worse. The “certificate” is just the cost of the best state-blind policy minus the cost of some cheaper state-aware policy. The only non-obvious result is that strong calls should go last, which holds only when reward is given at the end. The main numerical example is a single hand-picked three-step case.

  • All results come from one synthetic three-step pandas task with three models. Retrying with another model when an output fails a format or execution check is common practice.

Questions for authors
  • The reproducibility statement mentions a SWE-bench diagnostic, but no SWE-bench result appears in the paper. Was this experiment run and what did it show?

  • The theory assumes the router sees the hidden state for free before each call and makes a fixed number of calls, while StopRoute checks outputs after each call, retries, and may stop early. Section 4.2 also finds that the theory’s conditions do not hold on any of the real tasks examined. Which design choices in StopRoute follow from the theoretical results?

4 comments
AuthorsOpenReview ↗

Comment 1: What the paper contributes and what guides StopRoute

Thanks for the review. We have rewritten the introduction to explain StopRoute and the comparison it implements. With primary model assignments and stopping rules held fixed, does execution feedback help choose when to spend a fallback call? The theory examines the value of state information against an optimized randomized baseline. The DataFlow experiment tests feedback-dependent allocation in an executable workflow. Both the abstract and introduction now describe StopRoute and these roles.

Which design choices follow from the theory? The theory guides how we compare policies. It motivates optimizing the randomized open-loop benchmark, fixing the other decision components in the empirical control, and accounting for expenditure separately from information access. The execution checks and stopping rules are implementation choices, and development data select the cascade configuration. We make these roles explicit in Sections 1 and 5.

What goes beyond policy-class inclusion? We agree that a richer information class can copy a state-blind policy and that subtracting a feasible witness cost is elementary. The result that needs the transmission model is the suffix exchange inequality. Every schedule with a fixed number of strong calls is dominated by a strong suffix. Proposition 1 then reduces optimization over arbitrary randomized schedules to the lower convex envelope of K+1K+1 points.

Optimizing that baseline changes the comparison in Table 1:

Policy classSuccessMinimum cost to meet target 0.85
Deterministic open loop95.75%12.00
Randomized open loop85.00%5.321
Randomized state history85.00%3.560

The deterministic optimum overshoots the target because schedules are discrete. Randomization alone saves 6.679 cost units. A residual 1.760-unit gap survives after both classes allow randomization. The certificate quantifies that residual gap against the optimized baseline.

How dependent is this on the single example? Appendix O.3 adds 125 parameter combinations under two target rules. At fixed target 0.85, 52 have a positive gap and 27 save at least 10% of randomized open-loop cost. The attainable-midpoint target gives 80 and 45. The grid varies horizon, propagation and prices, keeps the conditional success probabilities fixed, and includes zero-gap cases.

The terminal-reward and homogeneity conditions remain essential. Appendix B.5 examines intermediate rewards. Appendix O.1 gives a two-step counterexample with stepwise strong-model dominance: strong–weak succeeds with probability 0.836 versus 0.374 for weak–strong at equal cost. Section 4.1 now states this boundary. Proposition 2 concerns the specified noisy-signal certificate, so its zero value leaves other state-aware policies open.

AuthorsOpenReview ↗

Comment 2: The paid-output mechanism and the missing SWE-bench result

How does the post-call policy connect to the analysis? In Section 5, we ask whether the detector sends a fallback to prefixes where that same fallback has greater incremental benefit. This question applies directly to StopRoute's paid-output decision.

For a common paid primary response, let DD indicate detector rejection, p=E[D]p=\mathbb E[D], and τ\tau be the conditional effect of the configured fallback on final success. With fixed downstream rules and an independent random trigger of probability qq, Equation (3) gives

ΔS=Cov⁡(D,τ)+(p−q)E[τ].\Delta S=\operatorname{Cov}(D,\tau)+(p-q)\mathbb E[\tau].

Retries and changes in downstream stopping are included in the identity. Appendix O.6 evaluates its terms on the existing 300-task v4 recorded tree:

Initial detector decisionTasksFailures rescuedSuccesses lostMean fallback effect
Reject39220+56.41 pp
Accept2613468−13.03 pp

Placement contributes 7.853 points and trigger frequency 0.5948 points, reconstructing the 8.448-point replay difference. Fallback replaces the initial output under the recorded rule. Rejected outputs otherwise stop, accounting for the zero losses in that group. Accepted outputs can be harmed by replacement. Avoiding those harmful replacements accounts for part of the comparison. At equal trigger frequency, placement adds USD 0.0001758 per task through downstream execution, so trigger matching alone leaves a cost difference. These are retrospective quantities conditional on recorded completions.

We test placement independently with fresh calls in Table 2. StopRoute achieves 79.50% success versus 74.50% for random triggering over 200 held-out tasks with three repetitions per policy. The paired gain is 5.00 points, with 95% CI [0.50,9.33][0.50,9.33]. The random trigger was cost-calibrated on a separate cohort. Its primary assignments and detector stopping match StopRoute's. The 2.67% mean-cost gap meets the prespecified point-estimate tolerance, while statistical cost equivalence remains unresolved. The online comparison estimates conditioned allocation in the frozen configuration, while the recorded-tree decomposition examines placement under its recorded branch rules.

Was SWE-bench run? Yes. The SWE-bench diagnostic was run before submission. We have now added its results to Section 7 and Appendix E, Table 11. The diagnostic pairs one DeepSeek and one Kimi continuation from each of 11 reconstructed development states at round 16. Both models preserved the two initially passing states. Among the nine initially failing states, Kimi rescued one that DeepSeek did not, giving 3/11 successes for Kimi and 2/11 for DeepSeek, with exact two-sided McNemar p=1.0p=1.0. Kimi's nominal continuation cost was 9.54 times DeepSeek's. This small paired diagnostic records one Kimi-only rescue. Whether execution signals can identify such cases before continuation and improve success at comparable cost on SWE-bench remains unresolved.

The separate Online-200 experiment estimates feedback-dependent fallback allocation in the frozen DataFlow configuration, with primary model assignments and stopping rules held fixed. It provides the controlled allocation evidence for the three-step case study.

Thank you for the detailed response and the revision. The revised abstract and introduction are much clearer. Reporting the SWE-bench and ALFWorld results in the main text is also helpful, and I appreciate how openly the limitations are stated. My main concern about scope remains. The evidence comes from one synthetic three-step task, the external probes on ALFWorld and SWE-bench are inconclusive, and the comparison with same-model retry is still missing. Given the improved presentation and the added analyses, I will raise my score from 1 to 2.

Thank you for reading the revised paper and for updating your assessment. We appreciate your recognition of the clearer presentation and added analyses. Your suggestions on broader task coverage and additional comparisons are valuable directions.

We followed up on your question about retrying the same model. On the 300-task DataFlow cohort, the first DeepSeek answer triggered fallback on 39 tasks. We could reconstruct 36 paired comparisons using separate DeepSeek and Kimi responses to identical first-step prompts from an earlier cohort. We executed both responses on the target table, shared the recorded second-step response, and obtained fresh Qwen responses for the final step.

Retrying DeepSeek solved 22 of the 36 tasks. Switching to Kimi solved those same 22 plus six others. Mean episode cost on these pairs was 2.19 times as high with Kimi (USD 0.003804 versus 0.001734). Three pairs remain unresolved because the replayed detector decision differed from the log. This helps answer what a change of model adds after a rejection: a simple retry already recovered many cases, and all six cases where the two choices differed favored Kimi.

Reviewer_ELBcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when observing execution state actually saves cost in step-level multi-model routing. It models a trajectory as a two-state (clean/corrupted) chain and shows that, under capability ordering and terminal reward, the best open-loop schedule puts its strong calls at the end. That makes the randomized open-loop optimum exactly computable, so the authors can separate the gain from state information from the gain that plain randomization already gives. In a toy K=3 instance, randomization alone cuts cost from 12 to 5.32, and state information takes it further to 3.56. On the empirical side, a detector-triggered fallback cascade (StopRoute) beats a cost-calibrated random-trigger control by 5 points (79.5% vs 74.5%) on 200 held-out tasks of a synthetic 3-step pandas workflow.

Critical strengths

I like the core methodological message. Step-level routers are usually compared against deterministic baselines, and Table 1 shows that randomization alone accounts for most of the apparent gap (6.68 of 8.44 cost units). Any paper claiming that a router "uses state" should have to beat this baseline, and that is a useful standard for the field. The suffix-optimality lemma reads clean and correct.

Critical weaknesses

It is hard to connect the theory and the experiments The authors' own proxy-based check finds the target window empty in all four real domains (Table 18). Appendix K shows that the reset assumption fails in DataFlow-v2, because a correct step 2 cannot repair a corrupted step-1 table. The theory assumes a free signal before the call, while the detector checks an output after the call has been paid for. The theory also measures the gap in cost at matched success, while the experiment measures success at roughly matched cost. Section 7 calls the link "mechanistic, not a fitted theorem test", so the theorems neither predict nor explain the empirical numbers.

The scope seems narrow and the evidence is marginal. All main-text results come from one synthetic 3-step task family, which is far from the "agent trajectories" framing in the introduction. The primary 95% CI is [0.5, 9.3] points, and 98 of the 200 tasks are ties. The 5% cost criterion applies to the point estimate only. The paired cost-difference CI runs from about -4% to +9% of StopRoute's cost, so the costs are not shown to be matched (the authors do say this is not an equivalence test). Policy selection is unstable: the selected policy is chosen in only 34.6% of development resamples, and on the same test records the next most frequent policy scores 45/48 vs. 42/48 at about 28% of the cost.

Questions for authors
  1. How does the randomized information gap vary across γ, c_s/c_w, and K, and in what region is it a meaningful fraction of the cost?
  2. What did the SWE-bench diagnostic show?
  3. How does StopRoute compare against same-model retry on rejection, and against the best cost-matched randomized policy over all assignments?
  4. Is there any real domain where the target window is non-empty, or where the reset and time-homogeneity assumptions roughly hold?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ELBc (1/3)

Thank you for pointing out what a correct second aggregation can and cannot repair. That example helped us make the connection between the model and the workflow more precise. Your two suggested controls also let us address two distinct decisions about where to place a fallback and which model to call after a rejection. Below we provide a point-by-point response.

W1. How does the chain relate to a workflow where an early error persists?

A correct second aggregation leaves an incorrect first table intact. In the chain, a successful call resets the state, the policy can observe that state before a call, and the horizon is fixed. In DataFlow, the detector first sees an output that has already been paid for, and rejecting it can lead to another call or early termination.

Section 5 therefore starts at the point where the workflow actually makes its decision. From the same paid first response, what happens to final success if we replace that response or keep it, with the downstream rules held fixed? On the 300-task v4 recorded tree, replacing rejected outputs rescued 22 of the 39 tasks that would otherwise have stopped. Among the 261 accepted outputs, replacement rescued 34 cases and lost 68. The mean effect of replacement was +56.41 percentage points after rejection and −13.03 after acceptance (Appendix O.6). Preserving a usable first answer accounts for part of the detector's benefit.

Equation (3) separates the effect of fallback placement from the effect of trigger frequency. On that tree, placement contributed 7.853 points and frequency contributed 0.5948, for a total of 8.448. For the chain, Table 1 and the paragraph below it now give both readings of the same theoretical frontiers. The saving at target success 0.85 is 1.760 cost units. At the randomized open-loop budget, the success gain is 4.868 percentage points. The corresponding workflow estimate comes from fresh calls.

W2. How much evidence is there for the online gain, and how stable was policy selection?

The primary test covers one synthetic three-step pandas task family. We calibrated the random trigger using only costs on 40 separate tasks and froze its rate before evaluating the following 200 held-out tasks (Section 6.1, Table 2).

PolicySuccessMean nominal cost per run (USD)Mean wall time (s)
StopRoute79.50%0.00204435.04
Random trigger74.50%0.00198935.41

We averaged three fresh runs per policy within each task. All 200 tasks, including the 98 ties, entered the paired analysis. The gain was 5.00 percentage points (task-bootstrap 95% CI [0.50, 9.33], sign-flip p = 0.0342). The observed mean costs differed by 2.67%, within the prespecified 5% point-estimate tolerance. The paired 95% cost interval [−3.91%, 9.36%] extends beyond that tolerance (Appendix I). These data support a success gain at nearby observed costs. Mean wall times were similar.

Your selection concern is borne out by the original 24-task development study. The success-first rule preferred a cascade that solved 24/24 tasks to a cheaper one that solved 23/24. The selected cascade recurred in 34.6% of development resamples. On the original 48 test tasks, the cheaper cascade solved 45 cases at about 28% of the selected cascade's cost. The selected cascade solved 42 (Appendix M.1). Online-200 freezes the same primary assignments and stopping rules for both policies, so the comparison estimates the effect of fallback placement within that configuration. The reliability of selecting a cascade from a new development sample remains open.

AuthorsOpenReview ↗

Response to Reviewer ELBc (2/3)

Q1. When is the randomized information gap substantial?

Appendix O.3, Table 27 varies the horizon, failure-propagation probability, and strong-to-weak price ratio across 125 configurations while holding conditional model capabilities fixed.

Success targetPositive cost saving from stateSaving at least 10% of randomized open-loop cost
Fixed at 0.8552/12527/125
Midpoint of all-weak and all-strong success80/12545/125

Whenever the all-weak schedule already reaches 0.85, observing the state cannot lower the cost of meeting that target. The midpoint measures savings at a target set relative to what the models can achieve in each configuration. At horizon 3 and propagation probability 0.8, increasing the price ratio from 2 to 20 raises relative savings from 6.003% to 47.04%. At price ratio 10 and the same propagation probability, savings are 14.49%, 33.08%, 29.97%, 20.14%, and 11.78% for horizons 2, 3, 5, 10, and 20. Longer horizons do not consistently increase the gap. Savings vanish at zero propagation and at the equal-price boundary.

Q2. What did SWE-bench show?

We ran one DeepSeek and one Kimi continuation from each of 11 reconstructed development states at round 16. Both models preserved the two initially passing repairs. Of the nine failing repairs, one passed only with Kimi and eight failed with either model. That one discordant pair accounts for the full difference between Kimi (3/11) and DeepSeek (2/11), with exact two-sided McNemar p = 1.0. Kimi's nominal continuation cost was 9.54 times DeepSeek's. We report this in Section 7 and Appendix E, Table 11. The diagnostic identifies one checkpoint where switching helped in hindsight. Whether an execution signal could select such checkpoints in advance at comparable cost remains open.

Q4. Has a real workflow been shown to meet the theorem's conditions?

We have yet to verify one. All four empirical proxy windows were empty (Appendix K, Table 22). The window gives a sufficient condition for Theorem 1's deterministic one-strong-call formula. Proposition 1 gives the randomized open-loop frontier at every feasible target within the homogeneous chain.

For a separate test of longer-horizon routing, StopRoute won 80 of 110 ALFWorld games and random placement won 77. The paired outcomes were 12 wins unique to StopRoute and nine unique to random placement (p = 0.664). StopRoute also used 5.95 more calls per game on average. The extension leaves the success advantage uncertain at this higher call count (Appendix D). Our evidence for improved placement at nearby observed cost comes from the DataFlow comparison in W2.

AuthorsOpenReview ↗

Response to Reviewer ELBc (3/3)

Q3a. How much does a same-model retry recover after rejection?

In our earlier reply to Q3, we left this control open. Each original tree node held only one completion. We have since reconstructed a paired comparison by drawing on separate, previously recorded answers to identical first-step prompts.

The first DeepSeek output triggered fallback on 39 of the 300 DataFlow v4 tasks. For each trigger, we took one separate DeepSeek answer and one Kimi answer from an earlier cohort and executed both expressions on the target task's table. Both branches used the same recorded second-step Kimi answer and stopping rule. We obtained fresh final-step Qwen responses, sharing a response when both branches produced the same final prompt.

Fallback on the same rejected outputFinal successes on 36 reconstructed pairsMean nominal episode cost on those pairs (USD)
Retry DeepSeek22/360.001734
Switch to Kimi28/360.003804

On the 36 reconstructed pairs, Kimi solved the same 22 tasks as DeepSeek and six more, at 2.19 times the mean episode cost. Three trigger pairs remain unresolved because the reconstructed detector decision differed from the log. The other 261 tasks followed identical paths under both policies. Thus a second attempt with DeepSeek already recovered many rejected cases. The six additional successes measure what switching models added on these reconstructed pairs, with the rejection decision held fixed.

Q3b. How does StopRoute compare with the best randomized mixture of fixed assignments at its budget?

Appendix O.7, Table 28 considers all 27 three-step assignments with shared detector stopping. Fitting the mixture on 24 development tasks selected DeepSeek–Kimi–Qwen, which then spent less than StopRoute on the 300-task v4 tree. To answer your equal-budget question, our response-only analysis lets the comparator use the evaluated outcomes to choose its best mixture at StopRoute's observed mean cost.

Policy on the same 300 recorded tasksSuccessMean nominal cost (thousandths of USD)
Development-selected mixture72.33%1.549
Best mixture fitted to the evaluated tree74.31%1.962
StopRoute79.67%1.962

The last two costs agree before rounding. Enumeration of the candidate optima and a supporting-line upper bound give the same optimum for this fixed-assignment family. StopRoute's observed advantage is 5.355 percentage points. In 20,000 paired whole-task bootstrap resamples, we refitted the mixture and recomputed StopRoute's budget each time. The exploratory 95% interval [−1.021, 8.326] includes zero and conditions on the stored completions. The comparator chooses one model per stage and shares the stopping rule. Q3a separately evaluates the additional call that a same-model retry requires.

Thanks for the detailed follow-up and analyses. It partially addresses my concerns. One concern remains: After a rejection, simply retrying DeepSeek fixes 22 of the 36 cases. Switching to Kimi fixes the same 22 plus 6 more but at 2.19 times the cost. So most of the benefit seems from trying again than calling a stronger model. Over the full 300 tasks, switching models adds only about 2 points. It seems indicating "retry when the detector flags a failure" rather than multi-model routing. Therefore, I maintain my rating of 2.

Reviewer_oAdLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the value of execution-state information in multi-step LLM agent routing, namely whether observing state reduces inference cost subject to a target success rate. The paper formulates a two-state transmission model with K steps, strong and weak models, capability ordering, and monotonicity assumptions. The paper proves that, for a fixed number of strong calls, the optimal open-loop schedule is a contiguous strong suffix; that the randomized open-loop optimum is the lower convex envelope of K+1 suffix points; and that within the target window the open-loop cost admits a closed form, yielding a lower bound on the value of state information through a feasible state witness. The paper further proves that this certificate vanishes when the observation signal carries no information. The paper proposes StopRoute, a gold-free detector cascade consisting of development selection, early stopping, and an escalation cascade. The theoretical component verifies the direction of each proposition by exhaustive enumeration on random instances, and a controlled instance verifies the existence of an information gap by exact enumeration. Experiments on DataFlow-v2 show that, with primary assignments, detector stopping, and fallback frequency held fixed, placing fallbacks according to the detector improves task success.

Critical strengths
  1. The matched-comparison design eliminates two standard confounds. The paper's central empirical claim is that conditioned placement has value, and this claim is most vulnerable to two objections: StopRoute simply spends more, or StopRoute simply uses better models. The paper rules out both confounds through a matched theoretical and experimental design.

  2. The zero certificate under uninformative signals gives the claim a falsification condition. Proposition 2 proves that when the observation signal is uninformative (r = f), the matched certificate is exactly zero. Most papers claiming that state helps only show a positive gap and do not specify when the gap should vanish. Proposition 2 supplies that condition: if the signal carries no information, the value must be zero. This separates "information" from "exogenous randomization" at the theoretical level and gives the experiment a clear interpretive frame — the detector either carries information (positive gap) or does not (zero gap).

Critical weaknesses
  1. The theory's time-homogeneity and step-independent capability-ordering assumptions do not hold for the paper's experiment. Definition 1 and Assumption 1 jointly require the success probability to be time-homogeneous (p(m,x) does not depend on t) and each tier to admit a step-independent capability ordering (b_x ≥ a_x). Yet the paper's own Appendix K (Table 19) suggests that the DataFlow workflow is not well described by either condition: Qwen succeeds on 8/93 clean-proxy step-2 prefixes but 169/170 clean-proxy step-3 prefixes; Kimi on 80/93 and 170/170; DeepSeek on 82/93 and 129/170. Thus, for the same model under the same clean-proxy label, observed success varies sharply by step, and the relative ordering of models is step-dependent. The paper may still use the theory as motivation, but it does not show that the exact applicability conditions of Lemma 1 and Theorem 1 hold for the experimental workflow, leaving a meaningful theory–experiment gap.

  2. The paper does not currently provide a testable bridge between Theorem 1's fixed-target cost result and the main experiment, which measures a nearby-cost success gap. The main theorem of Section 4 (Theorem 1) applies only when the target success rate falls in the window s* ∈ (g0, min(g1, P†')]. Appendix J (Table 18) reports that the paper's empirical proxy analogue of this window is empty on all four tested real domains (HotpotQA, ToolChain, DataFlow-v1, DataFlow-v2); correspondingly, the only direct demonstration of Theorem 1 is the synthetic controlled instance of Section 4.5, whose parameters are chosen for analysis. The paper therefore does not exhibit any real workflow for which the exact conditions of Theorem 1 are shown to hold. Meanwhile, the empirical claim of Section 6 (+5.00 pp) measures a different quantity: a success-rate gap at approximately fixed cost, rather than a cost gap at fixed success.

Questions for authors
  1. How essential are time homogeneity and a global capability ordering to Lemma 1? If success probabilities are step-dependent, and capability ordering holds only within each step, does the exchange argument still imply suffix optimality?

  2. What role is the theory meant to play in this paper: prediction or motivation? If it is meant to explain the real experiments, please provide a concrete, testable bridge to the +5.00 pp result. If it is meant only as motivation / an existence proof, please state that explicitly and soften the current “unified contribution” framing. Please also put 1.76 and +5.00 pp on a common scale.

  3. Could the introduction state the scope more explicitly? In particular, it would help to say up front that the paper measures the residual value of state information after the state-blind policy class has already been optimized.

  4. The paper’s main claim is about the marginal value of state information, but it does not currently position itself relative to the value-of-information / EVPI literature or to scheduling / ordering-structure results, so the novelty of the matched-class certificate and suffix-optimality claim is hard to locate. It would also help to expand the “Failure state” column in the main text.

3 comments
AuthorsOpenReview ↗

Response to Reviewer oAdL (1/2)

Thank you for pressing us on the link between the theorem and the experiment. Your example of a model changing rank across workflow stages makes the issue concrete. It prompted us to spell out which ordering result needs homogeneity and which allocation decision we can test directly on a paid output.

W1. Do the ordering assumptions describe DataFlow?

The stage-specific data support your concern. On clean-proxy inputs, Qwen succeeded on 8/93 second-stage aggregations and 169/170 final-stage answers. DeepSeek succeeded on 82/93 and 129/170, respectively (revised Appendix L, Table 23). Even the relative ordering of these two models reverses between stages. StopRoute allows the model order to vary by stage and selects it on development data. Lemma 1 instead concerns a homogeneous chain with ordered capabilities and terminal reward. Section 4.1 states these assumptions explicitly. The counterexample in Q1 shows what can go wrong when the stage changes.

W2. What testable claim connects the theory to the online experiment?

The chain asks how much state information can save after optimizing a randomized open-loop schedule. In DataFlow the decision comes later. Both policies have already paid for a first answer. Does the detector select the outputs for which replacing that answer most improves final success?

At a common paid prefix, let D indicate rejection and let p = E[D] be the detector’s trigger rate. Write τ for the conditional difference in final success probability between executing and skipping the configured fallback. Keep the downstream rules fixed and let the independent random trigger have rate q. Equation (3) in Section 5 gives the difference in success as

ΔS = Cov(D, τ) + (p − q) E[τ].

At equal trigger rates the second term vanishes. The covariance then measures what is gained by placing the same fallback where it is more useful. Appendix O.6 evaluates both branches on each of the 300 tasks in the v4 recorded tree.

First outputTasksFailures rescued by replacementSuccesses lost by replacementMean effect on final success
Rejected39220+56.41 pp
Accepted2613468−13.03 pp

Rejected outputs would otherwise terminate, which explains their zero losses. Among accepted outputs, replacement lost twice as many successes as it rescued. Preserving usable answers therefore contributes to the placement gain. At the frozen v4 rates, placement accounts for 7.853 points and the rate difference for 0.5948, totaling 8.448 points on the recorded tree. Even at equal trigger rates, placement adds USD 0.0001758 per task through downstream execution. This is why the independent Online-200 comparison measures realized costs as well as success. Q2 gives the common-scale comparison you requested.

AuthorsOpenReview ↗

Response to Reviewer oAdL (2/2)

Q1. Does strong-model dominance within each step suffice?

No. Appendix O.1 gives a two-step counterexample with propagation probability 0.8. Each entry below gives the probabilities of success in the clean and corrupted states, in that order.

StepWeak modelStrong model
1(0.10, 0)(0.90, 0.80)
2(0.90, 0.10)(0.95, 0.15)

Both steps satisfy strong-model dominance and clean-state monotonicity. At identical cost, strong–weak succeeds with probability 0.836 and weak–strong with 0.374. Here the incremental benefit of the strong model is much larger at the first step. Once capabilities can change across steps, moving that call to the end can reduce terminal success.

Q2. Is the theory predictive, and how do 1.760 and 5.00 compare?

The theory gives an exact benchmark in the chain and motivates the allocation comparison in DataFlow. Table 1 and the paragraph below it now give both readings of the controlled frontiers. At target success 0.85, the randomized open-loop optimum costs 5.321 and the randomized state-aware optimum costs 3.560, a saving of 1.760 before rounding. Holding the open-loop budget fixed instead, optimal success rises from 85.00% to 89.87%, a gain of 4.868 percentage points.

For DataFlow, the independent Online-200 experiment measures a 5.00-point gain in success (79.50% versus 74.50%) at nearby observed mean costs (2.67% apart, Table 2). The two gains are now expressed in the same units. The chain parameters determine the 4.868 points. Fresh calls on held-out workflows estimate the 5.00 points. The evidence linking feedback to useful fallback locations comes from following the two branches in W2.

Q3. Which state-blind class has been optimized?

Section 1 now identifies the comparator in each setting. In the homogeneous chain, Proposition 1 optimizes all open-loop schedules and their episode-level mixtures at every feasible success target. Theorem 1's target window gives the deterministic one-strong-call formula. All four empirical proxy windows were empty (revised Appendix K, Table 22), so we have yet to verify a real workflow satisfying the chain assumptions and that window. In Online-200, the policies share primary model assignments and detector stopping. The control randomizes first-step fallbacks at a rate calibrated on separate tasks using costs alone. This tests the value of choosing fallback locations within the frozen configuration.

Q4. What goes beyond standard value-of-information and scheduling arguments?

A state-aware policy can always imitate an open-loop schedule. The step that uses the transmission model is Lemma 1's exchange inequality. Among schedules with a fixed number of strong calls, placing them in a suffix weakly improves terminal success. This reduces the exact randomized open-loop frontier to the lower convex envelope of K+1 suffix points.

The distinction matters numerically. At target success 0.85, the cheapest deterministic schedule costs 12.00 and overshoots the target, achieving 95.75%, because the available schedules are discrete. Randomization saves 6.679 cost units by allowing the comparator to meet the target exactly. A further 1.760 units are saved by access to state at the same target. Proposition 2 gives a zero certificate for its specified witness under an uninformative signal. The value of other state-aware policies remains open in that comparison.

Section 2 now discusses Howard (1966) on information value and Smith (1956) on exchange arguments. We also expanded the closest-work comparison to identify both the signal and its timing. A router may observe latent corruption before a call, check execution validity after paying for the output, or infer earlier failures from the trajectory history. Those distinctions determine what information the comparator must share or withhold.

R1: The authors' verification confirms that Lemma 1's structural claim does not extend to the experimental workflow, and the authors now state that StopRoute uses step-specific model orderings selected on development data. The theoretical contribution is therefore narrower than the paper's framing suggests.

R2: This concern is resolved.

R3: The revision to the introduction improves readability. However, the response states that no real workflow has yet satisfied the homogeneous-reset assumptions and the required window. The paper's claimed contribution therefore shrinks to a model plus an independent case study.

R4: This concern is resolved.

Therefore, I maintain my rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.