- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies query-level selection of both an LLM and its execution harness. HM-Router learns separate model and harness representations from identities, outcome-derived behavioral summaries, and harness descriptions, then combines component scores with a query-dependent multiplicative interaction. HM-ROUTER-BENCH aggregates 12 public benchmarks with 293 routes. Experiments examine routing accuracy, cost, withheld routes, limited observations, and benchmark transfer. The results support joint routing and the usefulness of shared representations under the evaluated conditions. My remaining concerns are the operational interpretation of the cost envelope, the breakdown of generalization by component observability, and whether the specific composition rule offers an advantage over alternatives with matched inputs.
Critical strengths
- Treating the harness as part of the routing decision is well motivated. The formulation distinguishes feasible routes from observed routes, and sharing component representations provides a natural way to use sparse execution histories.
- Mean oracle-normalized accuracy increases from 66.65 for IRT, the strongest learned baseline by overall mean, to 73.93 (Table 1, p. 6). HM-Router has the highest point estimate among the compared methods on 10 of the 12 benchmarks. These are normalized accuracy gains, rather than raw task-success gains, and the method does not lead on every benchmark.
- The component-only ablation supports the interaction term: normalized accuracy rises from 68.51 to 73.93 (Table 3, p. 9). Additional predictive-agreement analyses support its ability to model compatibility. The authors explicitly distinguish predictive agreement from causality and state that the implemented layer-normalized scorer does not retain the CP-rank bound of the unnormalized construction (Appendix B.2). Table 4 also reports low routing overhead relative to agent execution.
- The withholding protocol excludes hidden outcomes from both behavioral summaries and the loss. Table 2 includes a component-average baseline with access to the same full candidate pool. At 90% removal, HM-Router reaches 49.31 versus 37.43 for this baseline and 42.22 for the oracle restricted to observed routes. This supports useful recovery of hidden options beyond merely showing that a larger portfolio can contain better routes. Limited-sample and held-out-benchmark experiments provide further evidence of transfer, within their stated evaluation settings.
Critical weaknesses
- The cost envelope does not establish how to choose a deployable policy before testing. Table 6 (p. 17) describes each entry as the best setting affordable by realized test spend on each cost benchmark, then averaged. This supports an attainable accuracy-cost comparison, and HM-Router leads the reported non-oracle methods at all seven budgets under this protocol. However, the manuscript does not specify a validation-only rule for selecting the penalty coefficient or other policy settings before observing test performance and spend. A validation-selected policy, frozen before testing, with test accuracy, realized mean cost, and uncertainty relative to the target budget would substantiate the operational interpretation. This concerns policy selection, not the training-based route-cost estimator in Eq. (8), and is not evidence of scorer-training leakage.
- The generalization results need a breakdown by component observability. Section 5.3 explicitly studies a degraded observed portfolio, removing train-best routes and allowing component cold start. Its comparison between the same scorer with and without hidden candidates appropriately measures the value of retaining those options; the different candidate pools are intentional. The full-pool component-average control further supports learned scoring, although it slightly exceeds HM-Router at 50% removal (52.89 versus 52.12). What remains unresolved is performance on unseen pairs when both components retain observations elsewhere, separately from cases with an unseen model or harness. Reporting these groups and a random-removal control would clarify the extent of pairwise compositional generalization beyond the strong-route-removal setting. The current results remain useful evidence for that stated setting.
- The specific composition rule is not isolated by sufficiently matched comparisons. All methods receive the same frozen query features and observed training outcomes (Section 5.1). HM-Router additionally organizes outcomes into component summaries and uses harness descriptions. The manuscript does not report alternative interaction architectures with these same inputs. The component-only ablation supports adding an interaction within HM-Router, but does not distinguish its shared-factor rule from, for example, a parameter-matched concatenation MLP or a more general bilinear interaction with the same features. A train-best fixed route would also be useful in the main unconstrained Table 1; fixed-route controls already appear in Table 2 and the cost evaluation. These requests concern attribution and practical value, not a demonstrated unfairness of the existing comparisons.
- Main-result uncertainty and some evaluation details remain underreported. Table 1 averages five fixed query splits without dispersion or paired intervals for the method differences. This matters particularly for small differences such as 88.22 versus 88.06 on SWE-bench Verified, whose significance cannot be determined from the reported means. Table 5 enumerates route identities and therefore permits route-graph statistics to be derived; the missing summaries are retained query/outcome counts and absolute oracle success rates, together with the re-derived cost-pool sizes. Section 5.5 explicitly excludes target-benchmark training samples. However, it does not detail how source benchmarks are combined or how much target-route component overlap remains. Those details would help interpret the reported transfer gains. Appendix B.2's per-benchmark fitting description is not, by itself, evidence that target data were used in that separate experiment.
Questions for authors
- Can the cost comparison be repeated with policy settings chosen only on validation data and frozen before testing, reporting test accuracy, realized mean dollars per task, and uncertainty relative to the target mean budget?
- How does unseen-route performance differ between pairs whose two components remain observed and cases with component cold start? Does the advantage persist with random route removal?
- Do the gains persist against interaction architectures with matched inputs and comparable parameter counts, and against a train-best fixed route in the main unconstrained setting?
- Can the authors provide retained query/outcome counts, absolute oracle rates and cost-pool sizes, paired uncertainty for the main comparisons, and the source-training and component-overlap details for held-out-benchmark transfer?
Thank you and here is our rebuttal
W1 / Q1 — Validation-selected, frozen cost policy.
We re-ran the cost evaluation so that every knob is chosen on training-split folds under the budget, then frozen and evaluated once on test. Under this protocol HM-Router still leads at all seven budgets, ahead of Av-Pro by 6.6 to 14.1 points.
W2 / Q2 — Generalization by component observability; random removal.
HM-Router ranks never-executed pairs above randome baseline at every removal level, whether both components are observed in other routes or one is cold (Table A). Under random removal it leads both alternatives when few routes are removed, and the methods converge as removal grows (Table B). Randomly removed routes are never better than the retained ones, so the access gain in Table 2 is specific to the setting it targets, where the strong routes lack execution histories.
Table A. Picks among the withheld pairs only, under random removal (accuracy relative to the best withheld pair per query).
Table B. Random removal, full candidate pool.
W3 / Q3 — Matched-input interaction architectures; train-best fixed route.
On the interaction architecture, GfWJ Q1 shows that HM-Router's scorer is related to factorization machine. We have added the train-best fixed route to the main setting and will include it in Table 1; see HZVY Q1.
W4 / Q4 — Uncertainty, dataset statistics, transfer details.
Paired uncertainty for Table 1 (HM-Router minus baseline with 95% CIs):
We reported CI of gains across benchmarks for each baseline.
Dataset statistics. There are 39,626 retained outcomes in total. Absolute oracle accuracy ranges from 15.3% (SciCode) to 100% (CORE-Bench 1.1). Cost-pool sizes are 15 routes (TerminalBench 2.1), 16 (TauBench), 31 (CORE-Bench Hard), 17 (ScienceAgentBench), 21 (GAIA) and 22 (SciCode). We will add a per-benchmark table with these and the query, route, model and harness counts.
Held-out transfer One router is trained on the union of the other benchmarks, over a shared model and harness vocabulary, with a per-benchmark bias that is constant across routes. CORE-Bench Hard and CORE-Bench 1.1 share task text, so each is excluded from the other's sources for every method. On average over the 12 targets, the model of a target route is seen in the sources for 77% of routes and the harness for 54%, but the exact pair for only 25%. Pair-level baselines therefore treat about three quarters of target routes as cold, while HM-Router can still use component evidence. Over all 12 targets, HM-Router scores 59.6, against 52.2 for EmbedLLM, 51.6 for IRT, 51.8 for Avengers and 49.8 for Random.