- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a framework for learning per-step model routers for multi-turn LLM agents. It uses counterfactual rollout trees to identify pivotal states, where candidate models differ substantially in their best recorded downstream returns. These comparisons guide data collection and training with ranking, performance, and cost objectives. At inference time, predicted model differences adjust the cost penalty. The framework is evaluated on six interactive benchmarks, including three OOD settings.
Critical strengths
- A useful contribution to the design of routing supervision.
The central insight is that a difficult or important step is not necessarily informative for learning model selection: if all candidate models preserve similar outcomes, there is little performance preference to learn. This shifts attention from imitating a selected model at every state to identifying where comparative supervision is useful. I find this a meaningful refinement of search-based routing, with a concrete consequence for how training data should be collected and selected.
- Evidence that selective supervision can improve routing rather than merely reduce training data.
The filtering results provide a useful empirical finding: training on 32.7% of the available examples improves mean online performance over the plain all-state recipe on all three source benchmarks. The fifteen additional all-state alternatives make this more informative than a single full-data comparison, although small differences should not be interpreted as statistically established superiority. Together with the online cost-quality results, this supports the practical value of concentrating supervision on model-discriminative states (Table 3 and Appendix D.4).
Critical weaknesses
- Comparison with the closest sequential routers could be more direct.
The paper cites DialRouter, MTRouter, and Hera, but does not evaluate against these closely related approaches. Since search-derived supervision and outcome-based multi-turn routing already appear in prior work, a more direct comparison would help establish the added value of pivotal-state selection. Appendix A.5 also identifies Router-R1 as an adaptation rather than a faithful reproduction. The internal ablations are useful, but leave the relative advantage over the closest existing frameworks less clear.
- The performance and cost targets do not necessarily describe the same continuation.
Equation 1 backs up the highest discovered return, whereas Equation 5 targets the cheapest continuation, which need not achieve that return. Combining these estimates in Equation 9 can therefore associate a favorable outcome with a cost that belongs to a different, potentially unsuccessful path. This limits the interpretation of the cost-aware score and merits discussion or validation. The concern primarily affects the cost-aware extension; it does not directly invalidate Table 2's performance-only results at lambda = 0.
Questions for authors
-
Could the authors provide a matched comparison with an appropriate closest sequential router, particularly DialRouter, to clarify the additional benefit of pivotal-state selection over search-derived routing supervision?
-
Does pivotal-only training retain its advantage over a random subset of the same size when optimizer-update counts and learning-rate schedules are matched? This would help distinguish informative selection from generic data reduction and optimization effects.
-
How should the combined performance-cost score be interpreted when its two targets may correspond to different continuations? Is there evidence that the cost predictions track the costs incurred by the deployed router under different cost preferences?
3 comments
Rebuttal Part 2
Weakness 2 / Question 3: Performance–cost scoring and its online evaluation
Equation 9 is a routing decision rule that combines separately learned performance and cost signals. The performance target favors the highest backed return, while Eq. 5 uses the cheapest expanded continuation. Although both heads train on the same state–model transitions, their future choices can differ. The combined score therefore expresses a preference over models using these two signals, rather than an exact expected net return under a single continuation policy. Appendix A.2 specifies the cost signal as a discounted, dimensionless proxy based on model-price ratios.
We evaluate the resulting decisions through online task outcomes and realized model-call costs. Both quantities are recorded on the same executed trajectory. The evaluation harness computes dollar cost from the actual input/output token usage of each selected model. Thus the reported end-to-end results assess the performance and expenditure of the policy that actually runs, rather than combining the success of one path with the cost of another.
The λ/r analyses examine how this decision rule responds to cost preferences: Figure 3 varies λ, Figure 4 compares state-dependent and fixed cost penalties, and Appendix E.4, Figure 11 jointly varies λ and r. Together, these analyses examine the operating points obtained by changing the strength and state dependence of the cost penalty. Tables 10–11 separately provide the full benchmark and aggregate quality–cost results at λ = 0. The inference sweeps evaluate the practical use of the cost signal in model selection.
At λ = 0, Table 2's action selection uses only the performance score. Cost supervision still contributes to learning the shared representation: Appendix D.3 shows that retaining the cost loss under Anchored CE raises SQL performance from 0.348 to 0.422 and WebShop performance from 0.499 to 0.573, with λ = 0 throughout. This provides additional evidence for the usefulness of cost supervision alongside the evaluation of cost-aware inference.
Comment on statistical significance: Small differences from the best all-state variants
The main finding is the substantial improvement over the plain all-state recipe with much less supervision. The complete set of comparisons is:
| Training recipe | Data share | SQL | WebShop | SWE-bench | Equal-benchmark mean |
|---|---|---|---|---|---|
| Plain all-state | 100% | 0.373 | 0.481 | 0.653 | 0.5023 |
| Benchmark-wise best of 15 all-state alternatives | 100% | 0.402 | 0.566 | 0.703 | 0.5570 |
| Pivotal-only | 32.7% | 0.422 | 0.573 | 0.710 | 0.5683 |
The improvement over the plain recipe is 6.60 percentage points in the equal-benchmark mean. Against the benchmark-wise best full-data alternatives, the mean difference is 1.13 points. The latter comparator selects a different best setting for each benchmark from the fifteen tested mechanisms; pivotal-only uses the common reported recipe. This makes it a useful comparison with a broad set of full-data approaches, but not a statistical demonstration of superiority over every alternative.
All learned rows report three independent training seeds, and Appendix D.4 presents their variation. We therefore interpret the small gaps as showing that the compact supervision set remains competitive with the tested full-data alternatives.
Thanks for the response. I accept the clarification that Eq. 9 is a preference rule and that the reported costs come from the same executed trajectories, which addresses my second concern. However, my concern about comparison with more relevant baselines remains, and the matched-size random-subset control was not run, so the advantage of pivotal selection over generic data reduction is still unestablished. I am keeping my score at 3.
Rebuttal Part 1
We thank the reviewer for recognizing the value of selective routing supervision and for the careful assessment of the all-state comparisons. We address the closest-method comparisons, the random control, and the performance–cost interpretation below.
Weakness 1 / Question 1: Closest sequential routers and the scope of Router-R1
We compare simplified forms of the learning strategies underlying these routers within a common five-model, terminal-reward setting. Our MSE, CE/Anchored CE, and all-state controls serve this purpose: they compare outcome regression, return-derived model preferences, and the use of every collected decision state. The relationship to the closest methods is therefore more specific than a general comparison with unrelated routers:
We prioritized these common-protocol controls because importing the full frameworks would change several factors at once. For MTRouter, preserving its outcome estimator would also introduce its model representation, error adjustments, and trajectory-collection scheme. For Hera, moving from two device–cloud candidates to five models requires changing its binary labels and cloud-call cost definition, as well as reproducing its two-stage training. DialRouter additionally requires adapting dialogue feedback and future-state retrieval to environment-terminal rewards. The simplified controls retain the shared learning ideas while allowing us to test the training-state selection under a fixed encoder, feedback source, and collected dataset. They are strategy-level counterparts, not measurements of the full published systems.
The most direct comparison is Table 3: with the same encoder and Anchored CE objective, the all-state recipe scores 0.373/0.481/0.653, while pivotal-only training scores 0.422/0.573/0.710 on SQL/WebShop/SWE-bench using 32.7% of the examples. Appendix D.4 adds fifteen full-data variants. These results establish the incremental value of the selection rule in the common pipeline; a ranking against complete DialRouter, MTRouter, or Hera implementations would require their full evaluation.
Question 2: Pivotal selection versus a matched-size random training subset
The existing ablations establish the effectiveness and data efficiency of pivotal-only training. Table 3 compares the same encoder and Anchored CE objective with all decision states versus the pivotal subset. Retaining 32.7% of the examples improves the equal-benchmark mean from 0.502 to 0.568, with gains on all three source benchmarks.
The evidence extends beyond a single full-data baseline. Appendix D.4 evaluates fifteen alternatives that retain all states while incorporating pivotality through reweighting, gating, adaptive temperatures, or objective selection. Pivotal-only training remains competitive with their strongest configurations while using substantially fewer examples. These comparisons support filtering as an effective way to use comparative supervision in our pipeline.
A matched-size random subset would further distinguish informative selection from generic data reduction and optimization effects. Our existing training comparisons do not isolate these factors, since dataset sizes and optimizer-update counts differ. We therefore interpret the results as evidence for an effective, data-efficient training recipe, without attributing the entire improvement exclusively to the quality of the selected states. The Random baseline in Table 2 is an inference-time control and serves a separate purpose from this training-subset comparison.