- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RSI-router jointly evolves subtask definitions, large/small model assignments, and model-specific execution skills. A proposal agent analyzes training trajectories, proposes candidate routers, improves their skills, and maintains a validation-based Pareto population. At inference time, the small model identifies the current subtask and selects its assigned executor. Across five benchmarks, the paper reports an average 51.7% online cost reduction while the selected configurations match or exceed the large-model baseline. The approach is promising, but the selection protocol, limited test samples, and incomplete attribution of the gains leave the claimed advance insufficiently established.
Critical strengths
-
Jointly optimizing model assignment and execution guidance is well motivated: a skill can change what a small model can reliably execute, so routing against fixed capabilities can be unnecessarily restrictive. The training/validation separation for trajectory access and the use of a small model for runtime routing are sensible design choices.
-
The paper evaluates several distinct agent environments and includes routing calls and skill context in online token costs. The large reductions on ALFWorld, ScienceWorld, and WebShop are encouraging. It also reports the unfavorable side of the skill ablation: skills can reduce coding-task success while saving cost. This is more informative than assuming that every skill necessarily improves accuracy.
Critical weaknesses
-
The headline comparison appears to involve selection using test outcomes. Section 3.1 evaluates every final Pareto router on the test set, and Section 3.2 reports the cheapest strategy whose measured performance reaches the large-model baseline. Unless that final choice was fixed using validation data alone, the reported operating point is selected with test feedback, even if test trajectories never guide evolution. This is especially consequential with only 32 SWE-bench and 25 Terminal-Bench test tasks. The SWE-bench difference, 84.38% versus 83.33% over three repeats, corresponds to approximately one additional successful trial among 96. Table 1 lacks paired uncertainty for these selected performance/cost comparisons. Repeating inference on the same tasks does not address variability from task sampling and router evolution.
-
The experiments do not adequately isolate the contribution of recursive subtask routing from execution-skill or harness optimization. SkillOrchestra, which already learns skill-conditioned routing from execution experience, is discussed but not evaluated. More generally, the paper lacks large-only and small-only controls receiving comparably evolved execution skills, and a budget-matched fixed-subtask search. Masking skills after joint optimization is useful but does not substitute for these comparisons; indeed, it raises SWE-bench and Terminal-Bench performance by 9.7 and 10.5 points while increasing cost (Figure 6(c)). Thus the evidence establishes a promising combined system more clearly than an advance attributable to the proposed routing/evolution mechanism.
Questions for authors
-
Selection clarification: Were the exact Table 1 configurations fixed before observing any test results? Please report validation-selected operating points with task-paired uncertainty, and distinguish inference repeats from independent router-evolution runs.
-
Contribution evidence: Is a comparison with SkillOrchestra, or with the same skill-evolution procedure applied to fixed large-only/small-only executors, available? Which gain remains when proposal/search compute and execution guidance are matched?
-
Cost clarification: What are the offline proposal and candidate-evaluation costs, and the resulting deployment break-even workload? How sensitive is the online frontier to the assumed 1:30 small/large price ratio, given that Appendix C's six measured GPU-hour ratios range from 25.32 to 99.15?
5 comments
Response to Reviewer ByGF Part II
R-W2: Attribution of routing versus skills and harness optimization
We added the four comparisons you ask for. None of them reaches RSI-router's performance at a lower cost. Each cell is score / cost (USD over the test set):
| Method | ALFWorld | ScienceWorld | WebShop | SWE-bench Verified | Terminal-Bench 2.0 |
|---|---|---|---|---|---|
| Large-only | 90.6 / \$1.50 | 34.4 / \$1.91 | 59.2 / \$0.75 | 83.3 / \$3.46 | 40.0 / \$17.10 |
| Small-only | 65.6 / \$0.13 | 14.6 / \$0.06 | 47.5 / \$0.02 | 62.5 / \$0.69 | 5.3 / \$0.17 |
| SkillOrchestra | 87.5 / \$0.77 | 31.3 / \$1.26 | 47.5 / \$0.40 | 71.9 / \$3.25 | 20.0 / \$6.47 |
| RSI-router (validation-selected) | 98.4 / \$0.37 | 37.0 / \$0.47 | 62.3 / \$0.25 | 84.4 / \$3.18 | 34.7 / \$13.88 |
Comparison with SkillOrchestra. We use the official implementation with the same training trajectories as RSI-router. SkillOrchestra extracts its skills from pre-collected trajectories and writes them for the router, to decide which model executes a step; the executing model receives no guidance. RSI-router instead re-executes every candidate router and writes skills for the executing model. In the results, SkillOrchestra saves 6–62% of the large-only cost but loses 3–20 points on every benchmark. RSI-router scores 5.7–14.8 points higher than SkillOrchestra on all five benchmarks, at lower or equal cost on four. On Terminal-Bench, SkillOrchestra is cheaper (\13.88), but its success rate is half that of the large model (20.0 vs. 40.0; RSI-router: 34.7).
The three controls are on ALFWorld, ScienceWorld, and WebShop, where the validation-selected router uses skills:
| Method | ALFWorld | ScienceWorld | WebShop |
|---|---|---|---|
| Large-only | 90.6 / \$1.50 | 34.4 / \$1.91 | 59.2 / \$0.75 |
| Large-only + evolved skills | 100.0 / \$1.08 | 57.8 / \$2.49 | 63.7 / \$1.09 |
| Small-only + evolved skills | 75.0 / \$0.13 | 21.9 / \$0.12 | 57.1 / \$0.04 |
| Fixed subtasks | 96.9 / \$0.51 | 39.1 / \$0.75 | 56.8 / \$0.12 |
| RSI-router (validation-selected) | 98.4 / \$0.37 | 37.0 / \$0.47 | 62.3 / \$0.25 |
With evolved skills but no routing, neither model reaches RSI-router's trade-off. In these two controls, one model executes every step and gets its own six-iteration skill evolution with the same proposal agent. RSI-router costs 66–81% less than the large model with skills and scores 5–23 points higher than the small model with skills. The large model with skills is on par with RSI-router on ALFWorld and WebShop (within 1.6 points); on ScienceWorld it is 20.8 points higher but costs 5.3× as much. Skills do not make it cheap: on ScienceWorld and WebShop it costs more with skills than without. The low cost therefore requires routing.
Fixed-subtask search. We added this control: a new six-iteration search with the same number of candidates, in which the subtask definitions stay at their initial version and only model assignments and skills evolve. The router is selected with the validation rule of R-W1. With fixed subtasks, the search already saves more than 60% of the large-only cost. Evolving the subtasks as well lowers cost further, by more than a quarter on ALFWorld and more than a third on ScienceWorld, at similar performance (within 2.1 points).
Skill masking (Fig. 6(c)). The gain from masking skills that you cite is the other side of a cost trade: on SWE-bench and Terminal-Bench, skills give up about 10 points of success and save about a third of the large-only cost, and the Pareto population keeps routers both with and without skills. The validation-selected routers there carry no skills.
Response to Reviewer ByGF Part III
R-Q1: Selection protocol and task-paired uncertainty
The Table 1 configurations were not fixed before testing (see R-W1). The operating points fixed on validation alone are in R-W1, with task-paired 95% intervals: they reduce cost by 48.9% on average (51.7% in Table 1) and match or exceed the large model on four of the five benchmarks.
The three repeats are inference repeats of one evolution run per benchmark. We added a second, independent evolution with a different small model (Qwen3.5-4B), with the router selected by the same validation-only rule:
| Benchmark | Large-only | 4B small-only | RSI-router (4B) | Δ (± 95% CI) | Cost saving (± 95% CI) |
|---|---|---|---|---|---|
| ALFWorld | 90.6 | 44.8 | 95.3 | +4.7 ± 6.6 | 67.1% ± 13.4 |
| ScienceWorld | 34.4 | 9.4 | 31.8 | −2.6 ± 10.6 | 67.2% ± 6.9 |
| WebShop | 59.2 | 51.5 | 58.7 | −0.5 ± 4.8 | 65.3% ± 8.4 |
| SWE-bench Verified | 83.3 | 42.7 | 76.0 | −7.3 ± 9.0 | 53.1% ± 13.7 |
| Terminal-Bench 2.0 | 40.0 | 1.3 | 22.7 | −17.3 ± 13.4 | 80.1% ± 9.9 |
On ALFWorld, ScienceWorld, and WebShop the 4B router is within −2.6 to +4.7 points of the large model at 65–67% lower cost. On SWE-bench it trades 7.3 points for 53% lower cost. The only large loss is on Terminal-Bench (17.3 points), because the 4B model is too weak for these tasks: alone it solves 1 of 75 runs.
R-Q2: SkillOrchestra and single-model controls with evolved skills
Yes; both are in R-W2: SkillOrchestra in the first table and the single-model controls with evolved skills in the second. The controls use the same proposal agent and six iterations, and each fixed executor gets skills evolved for itself. What remains is the cost: RSI-router costs 66–81% less than large-only + skills, at comparable performance on two of the three benchmarks.
R-Q3: Offline cost and price sensitivity
Offline cost. One complete evolution run per benchmark (6 iterations × 4 candidates, each evaluated before and after its skill update, two repeats on training and validation):
| ALFWorld | ScienceWorld | WebShop | SWE-bench | Terminal-Bench | |
|---|---|---|---|---|---|
| Candidate evaluation | $154 | $276 | $66 | $965 | $1,103 |
| Single-model trajectories | $8 | $10 | $5 | $45 | $61 |
| Proposal agent (DeepSeek-V4.1-Flash) | $60 | $65 | $47 | $67 | $61 |
| Total | $222 | $351 | $118 | $1,077 | $1,225 |
| Wall-clock | 13.6 h | 13.1 h | 6.5 h | 16 h | 41 h |
This cost is paid once per benchmark. The evolved router is then used for every task, at 67.0–75.5% lower cost per task on ALFWorld, ScienceWorld, and WebShop, 8.1% on SWE-bench, and 18.8% on Terminal-Bench (R-W1).
Price ratio. Repricing every call of the same routers gives the following cost savings relative to large-only:
| Small:large price ratio | 1:25.32 | 1:30 | 1:35.96 | 1:50 | 1:99.15 |
|---|---|---|---|---|---|
| ALFWorld | 74.7% | 75.3% | 75.9% | 76.6% | 77.6% |
| ScienceWorld | 74.6% | 75.5% | 76.3% | 77.5% | 78.9% |
| WebShop | 65.5% | 67.0% | 68.4% | 70.3% | 72.7% |
| SWE-bench Verified | 7.9% | 8.1% | 8.3% | 8.5% | 8.9% |
| Terminal-Bench 2.0 | 18.7% | 18.8% | 19.0% | 19.2% | 19.4% |
The small model accounts for only 1–25% of RSI-router's cost, so the savings move by −1.5 to +5.7 points over the measured range (25.32–99.15). The assumed 1:30 is also below the aggregate measured ratio (35.96), so the reported savings are conservative. At no ratio in this range does any method in Table 1 reach RSI-router's performance at a lower cost.
Response to Reviewer ByGF Part I
We appreciate your careful and constructive feedback! Below we provide our responses, and we are happy to follow up on any additional questions.
Several methods, including ours, produce multiple operating points. Table 1 therefore summarizes every method with the same reporting rule: the cheapest of its configurations that reaches large-only performance on the test tasks. To make the reported point independent of test results, we repeated the selection with the same rule on validation data only: for each benchmark, we take the cheapest router on the validation Pareto front whose validation performance is at least that of the large model, and evaluate it on the test set afterwards. The validation-selected routers still reduce cost by 48.9% on average (51.7% in Table 1) and match or exceed the large model on four of the five benchmarks.
The table reports these routers on the test sets of Table 1 (three repeats per task; ± is the half-width of a 95% interval from a task-level paired bootstrap):
On SWE-bench, the rule picks the same router as Table 1; as you point out, its gap to the large model (84.38 vs. 83.33) is one trial out of 96, so we do not claim a performance gain there. On this benchmark, the router's benefit is its lower cost. Terminal-Bench is the exception: the rule picks a cheaper router that is 5.3 points below the large model at 18.8% lower cost. The revision will report the validation-selected routers for all benchmarks.