acceptodds
Under review as a conference paper at ICLR 2027

HM-ROUTER: Joint Model and Harness Routing For Agentic Systems

Abstract

Agent performance depends on both the underlying model and the harness that manages its tool use and execution. Selecting a suitable pair requires accounting for their compatibility, yet training samples may cover only a subset of the growing combination space. We introduce HM-Router, a routing method that jointly selects a model and harness for each query. It learns separate model and harness representations shared across routes, with an interaction term inspired by canonical polyadic (CP) tensor decomposition to capture how their compatibility varies with the query. This sharing allows training samples from observed pairs to inform predictions for unobserved combinations. We curate HM-ROUTER-BENCH from 12 public agent benchmarks, covering 293 routes, 73 models, and 25 harnesses. HM-Router exceeds the strongest evaluated learned baseline by 7.3 percentage points in mean routing accuracy and leads at all seven evaluated cost budgets on the six-benchmark HM-ROUTE-COST subset. When 90% of routes have their training outcomes withheld, allowing unobserved combinations improves normalized accuracy by 15.8 points over restricting the same router to observed routes. HM-Router has also demonstrated training sample efficiency for new routes and components and generalization to unseen benchmarks.

Primary area:
generative models
Keywords:
agent harness, model routing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qkXbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies query-level selection of both an LLM and its execution harness. HM-Router learns separate model and harness representations from identities, outcome-derived behavioral summaries, and harness descriptions, then combines component scores with a query-dependent multiplicative interaction. HM-ROUTER-BENCH aggregates 12 public benchmarks with 293 routes. Experiments examine routing accuracy, cost, withheld routes, limited observations, and benchmark transfer. The results support joint routing and the usefulness of shared representations under the evaluated conditions. My remaining concerns are the operational interpretation of the cost envelope, the breakdown of generalization by component observability, and whether the specific composition rule offers an advantage over alternatives with matched inputs.

Critical strengths
  1. Treating the harness as part of the routing decision is well motivated. The formulation distinguishes feasible routes from observed routes, and sharing component representations provides a natural way to use sparse execution histories.
  2. Mean oracle-normalized accuracy increases from 66.65 for IRT, the strongest learned baseline by overall mean, to 73.93 (Table 1, p. 6). HM-Router has the highest point estimate among the compared methods on 10 of the 12 benchmarks. These are normalized accuracy gains, rather than raw task-success gains, and the method does not lead on every benchmark.
  3. The component-only ablation supports the interaction term: normalized accuracy rises from 68.51 to 73.93 (Table 3, p. 9). Additional predictive-agreement analyses support its ability to model compatibility. The authors explicitly distinguish predictive agreement from causality and state that the implemented layer-normalized scorer does not retain the CP-rank bound of the unnormalized construction (Appendix B.2). Table 4 also reports low routing overhead relative to agent execution.
  4. The withholding protocol excludes hidden outcomes from both behavioral summaries and the loss. Table 2 includes a component-average baseline with access to the same full candidate pool. At 90% removal, HM-Router reaches 49.31 versus 37.43 for this baseline and 42.22 for the oracle restricted to observed routes. This supports useful recovery of hidden options beyond merely showing that a larger portfolio can contain better routes. Limited-sample and held-out-benchmark experiments provide further evidence of transfer, within their stated evaluation settings.
Critical weaknesses
  1. The cost envelope does not establish how to choose a deployable policy before testing. Table 6 (p. 17) describes each entry as the best setting affordable by realized test spend on each cost benchmark, then averaged. This supports an attainable accuracy-cost comparison, and HM-Router leads the reported non-oracle methods at all seven budgets under this protocol. However, the manuscript does not specify a validation-only rule for selecting the penalty coefficient or other policy settings before observing test performance and spend. A validation-selected policy, frozen before testing, with test accuracy, realized mean cost, and uncertainty relative to the target budget would substantiate the operational interpretation. This concerns policy selection, not the training-based route-cost estimator in Eq. (8), and is not evidence of scorer-training leakage.
  2. The generalization results need a breakdown by component observability. Section 5.3 explicitly studies a degraded observed portfolio, removing train-best routes and allowing component cold start. Its comparison between the same scorer with and without hidden candidates appropriately measures the value of retaining those options; the different candidate pools are intentional. The full-pool component-average control further supports learned scoring, although it slightly exceeds HM-Router at 50% removal (52.89 versus 52.12). What remains unresolved is performance on unseen pairs when both components retain observations elsewhere, separately from cases with an unseen model or harness. Reporting these groups and a random-removal control would clarify the extent of pairwise compositional generalization beyond the strong-route-removal setting. The current results remain useful evidence for that stated setting.
  3. The specific composition rule is not isolated by sufficiently matched comparisons. All methods receive the same frozen query features and observed training outcomes (Section 5.1). HM-Router additionally organizes outcomes into component summaries and uses harness descriptions. The manuscript does not report alternative interaction architectures with these same inputs. The component-only ablation supports adding an interaction within HM-Router, but does not distinguish its shared-factor rule from, for example, a parameter-matched concatenation MLP or a more general bilinear interaction with the same features. A train-best fixed route would also be useful in the main unconstrained Table 1; fixed-route controls already appear in Table 2 and the cost evaluation. These requests concern attribution and practical value, not a demonstrated unfairness of the existing comparisons.
  4. Main-result uncertainty and some evaluation details remain underreported. Table 1 averages five fixed query splits without dispersion or paired intervals for the method differences. This matters particularly for small differences such as 88.22 versus 88.06 on SWE-bench Verified, whose significance cannot be determined from the reported means. Table 5 enumerates route identities and therefore permits route-graph statistics to be derived; the missing summaries are retained query/outcome counts and absolute oracle success rates, together with the re-derived cost-pool sizes. Section 5.5 explicitly excludes target-benchmark training samples. However, it does not detail how source benchmarks are combined or how much target-route component overlap remains. Those details would help interpret the reported transfer gains. Appendix B.2's per-benchmark fitting description is not, by itself, evidence that target data were used in that separate experiment.
Questions for authors
  1. Can the cost comparison be repeated with policy settings chosen only on validation data and frozen before testing, reporting test accuracy, realized mean dollars per task, and uncertainty relative to the target mean budget?
  2. How does unseen-route performance differ between pairs whose two components remain observed and cases with component cold start? Does the advantage persist with random route removal?
  3. Do the gains persist against interaction architectures with matched inputs and comparable parameter counts, and against a train-best fixed route in the main unconstrained setting?
  4. Can the authors provide retained query/outcome counts, absolute oracle rates and cost-pool sizes, paired uncertainty for the main comparisons, and the source-training and component-overlap details for held-out-benchmark transfer?
1 comment
AuthorsOpenReview ↗

Thank you and here is our rebuttal

W1 / Q1 — Validation-selected, frozen cost policy.

We re-ran the cost evaluation so that every knob is chosen on training-split folds under the budget, then frozen and evaluated once on test. Under this protocol HM-Router still leads at all seven budgets, ahead of Av-Pro by 6.6 to 14.1 points.

≤ $/task0.7511.523510
Av-Pro51.351.952.954.455.456.556.5
IRT51.952.552.954.254.354.654.6
EmbedLLM51.552.153.457.257.257.258.4
HM-Router57.959.962.768.568.568.268.2

W2 / Q2 — Generalization by component observability; random removal.

HM-Router ranks never-executed pairs above randome baseline at every removal level, whether both components are observed in other routes or one is cold (Table A). Under random removal it leads both alternatives when few routes are removed, and the methods converge as removal grows (Table B). Randomly removed routes are never better than the retained ones, so the access gain in Table 2 is specific to the setting it targets, where the strong routes lack execution histories.

Table A. Picks among the withheld pairs only, under random removal (accuracy relative to the best withheld pair per query).

Routes removed10%25%50%75%
Both components observed: HM-Router74.471.267.067.5
Both components observed: random pick68.863.960.064.0
A component cold: HM-Router72.568.460.557.1
A component cold: random pick70.763.757.451.8

Table B. Random removal, full candidate pool.

Routes removed10%25%50%75%
HM-Router71.270.867.364.1
Component-only67.166.765.863.5
Comp. Avg. (all)66.866.266.463.1

W3 / Q3 — Matched-input interaction architectures; train-best fixed route.

On the interaction architecture, GfWJ Q1 shows that HM-Router's scorer is related to factorization machine. We have added the train-best fixed route to the main setting and will include it in Table 1; see HZVY Q1.

W4 / Q4 — Uncertainty, dataset statistics, transfer details.

Paired uncertainty for Table 1 (HM-Router minus baseline with 95% CIs):

BaselineNormalized [95% CI]
IRT+7.3 [+0.5, +14.3]
Avengers+7.3 [+3.9, +11.7]
EmbedLLM+8.4 [+3.1, +16.4]

We reported CI of gains across benchmarks for each baseline.

Dataset statistics. There are 39,626 retained outcomes in total. Absolute oracle accuracy ranges from 15.3% (SciCode) to 100% (CORE-Bench 1.1). Cost-pool sizes are 15 routes (TerminalBench 2.1), 16 (TauBench), 31 (CORE-Bench Hard), 17 (ScienceAgentBench), 21 (GAIA) and 22 (SciCode). We will add a per-benchmark table with these and the query, route, model and harness counts.

Held-out transfer One router is trained on the union of the other benchmarks, over a shared model and harness vocabulary, with a per-benchmark bias that is constant across routes. CORE-Bench Hard and CORE-Bench 1.1 share task text, so each is excluded from the other's sources for every method. On average over the 12 targets, the model of a target route is seen in the sources for 77% of routes and the harness for 54%, but the exact pair for only 25%. Pair-level baselines therefore treat about three quarters of target routes as cold, while HM-Router can still use component evidence. Over all 12 targets, HM-Router scores 59.6, against 52.2 for EmbedLLM, 51.6 for IRT, 51.8 for Avengers and 49.8 for Random.

Reviewer_GfWJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HM-Router, a method that selects both model and harness for each task. It learns separate representations for models and harnesses and also combines the task features to predict execution success. Particularly an interaction term is used to capture how models and harnesses work together.

The method is evaluated on 12 benchmarks and report improvements in routing accuracy, cost efficiency, and generalization to unseen combinations.

Critical strengths
  1. The paper studies a practical problem. Select models and harnesses together makes sense for agent systems that support multiple configurations. The method is simple, and the paper is clear and easy to follow.

  2. The evaluation is extensive, covering 12 benchmarks. HM-Router improves average routing accuracy over the evaluated baselines. The ablation study also shows that the interaction term helps, and the results on unseen combinations are promising.

Critical weaknesses
  1. The technical novelty needs to be analyzed in more details. The method mainly learns representations for tasks, models, and harnesses, then uses their interactions to predict success. This is closely related to existing feature-based prediction and factorization methods. The paper shows that adding an interaction term helps, but does not compare with other simple learned interaction models, such as an MLP or a factorization machine. It is not very clear how much the specific design contributes. The authors are encouraged to show if the current design is better than existing technicals?

  2. The experiments has not yet well proved the advantage of the method on unseen combinations. In Table 2, the reported 15.8-point improvement compares routing over all feasible combinations with routing over observed combinations only. Part of this gain is due to more choices. The comparison with component averages is useful, but a stronger learned baseline with access to the same candidate pool is needed to better support the main contributions.

Questions for authors
  1. How does HM-Router compare with a simple MLP or another learned factorized model using the same input features and candidate pool? This would help clarify the contribution of the proposed architecture.

  2. In Table 2, could you report the performance specifically on withheld model–harness pairs? This would help assess how well the method actually predicts unseen combinations.

  3. Could you compare joint routing with a strong model-only router using a fixed harness on benchmarks with multiple feasible harnesses? This would help to understand how much benefit comes from selecting the harness itself.

4 comments
AuthorsOpenReview ↗

Thank you and here is our rebuttal

W1 / Q1 — Novelty and relation to factorization machines.

We agree that factorization machines (FMs) are closely related; a higher-order FM is a plausible alternative. In fact, HM-Router is a restricted third-order FM. Setting LayerNorm aside, with ⟨a,b⟩w=∑fwfafbf\langle a,b\rangle_w = \sum_f w_f a_f b_f:

S(q,m,h)=b+⟨q~,um⟩w1+⟨q~,uh⟩w2+∑fw3,f q~f um,f uh,f.S(q,m,h) = b + \langle \tilde q, u_m\rangle_{w_1} + \langle \tilde q, u_h\rangle_{w_2} + \textstyle\sum_f w_{3,f}\, \tilde q_f\, u_{m,f}\, u_{h,f}.

The first two terms are the FM's second-order query–model and query–harness terms, with a weighted inner product. The last term is the FM's third-order term, a CP decomposition of the query × model × harness tensor. Because q~\tilde q has a bias, it also contains the FM's model–harness term. So our contribution is not the scorer's form. It is the problem (joint model–harness routing with a feasible pool larger than the observed one), the benchmark, and factorizing over components rather than atomic routes. Prior routers (EmbedLLM, IRT, Avengers) learn one embedding per route and cannot score an unseen pair. We will present HM-Router as an FM-family instance and cite relevant work.

Q2 — Accuracy on the withheld pairs themselves.

Here we restrict the argmax to the withheld pairs only, under random removal, and report accuracy relative to the best withheld pair for each query.

Routes removed10%25%50%75%
HM-Router, both components observed elsewhere74.471.267.067.5
Random pick among those pairs68.863.960.064.0

HM-Router consistently ranks never-executed pairs above random baseline.

Q3 — Joint routing versus a strong model-only router with a fixed harness.

For each benchmark we fix one harness at random and route over models only, using HM-Router's scorer restricted to that harness.

Joint (HM-Router)Model-only, random fixed harness
Mean73.9369.22

Joint routing outperforms a strong model-only router by 4.7 points on average, so choosing the harness adds value beyond choosing the model.

Thanks! Two main concerns still remain.

  1. You said that the scorer is a restricted third-order FM, can you compare with an MLP or a standard FM using the same inputs and candidate pool?
  2. For Q3, the model-only router uses a random harness, so it is not the strong model-only router I asked about, and the 4.7-point gap may partly come from a weak harness choice. Can you fix the train-best harness instead and report per-benchmark results?

I keep the rating now and would like to see if the comparisons have some interesting results.

Thank you for the follow-up. We ran both comparisons. Numbers are oracle-normalized accuracy (%), averaged over the 12 benchmarks.

1. MLP and standard FM with the same inputs and candidate pool.

All variants share HM-Router's inputs, encoders, training recipe and candidate pool. MLP is a 2-layer MLP on the concatenated [query; model; harness] representations. FM is a standard second-order FM.

MLPFMHM-Router
Mean69.772.773.9

HM-Router is better than MLP, and it has closer gap as compared to FM. The gain comes from factorizing routes into shared model and harness representations and letting them interact. We will state the claim this way.

2. Model-only router with the train-best harness.

We fix the harness of the train-best route (chosen per split) and route over models only, with the same scorer.

Joint (HM-Router)Model-only, train-best harness
Mean73.972.7

As expected, against the train-best harness the average gap shrinks from 4.7 to 1.3 points. The gain is concentrated rather than uniform:

  • Complementary harnesses: joint routing gains on TauBench (+11.6), SciCode (+5.3), Online Mind2Web (+4.5) and TerminalBench 2.1 (+2.9), where it picks a different harness on 44–72% of queries.
  • One dominant harness: the two are within 1 point on CORE-Bench Hard, GAIA, ScienceAgentBench, SWE-bench Verified and HarnessBench.

Thank you for running both comparisons. HM-Router is 1.2 points better than a standard FM, and joint routing is 1.3 points better than a model-only router with the train-best harness. The joint-routing gain comes mainly from four benchmarks. Results for CORE-Bench 1.1, SkillsBench and WorkspaceBench-Lite are not reported, and the averages suggest that joint routing is worse on these three combined. So the specific scorer and per-query harness selection add little on average, and I will keep my rating.

Reviewer_HZVYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HM-Router, a routing method that jointly selects a model and harness for each query in agentic systems. To address model–harness compatibility and incomplete execution histories, it learns separate component representations with a query-dependent interaction term inspired by canonical polyadic tensor decomposition. Experiments across 12 agent benchmarks show improved routing accuracy over existing learned baselines, with additional evidence of favorable cost–accuracy tradeoffs on a six-benchmark subset.

Critical strengths

S1. The paper addresses the practical problem of joint model–harness routing through a compositional approach that captures their compatibility while sharing information across combinations. S2. The paper provides extensive experiments across diverse agent benchmarks, with additional evaluations of cost efficiency and generalization to unseen combinations.

Critical weaknesses

W1. Section 5.3 withholds the training outcomes of the strongest routes while keeping them available at inference. Comparisons with observed-only variants measure the value of retaining hidden candidates, but do not isolate the interaction term’s contribution. For example, in TerminalBench 2.1, some Claude Code models also appear under Terminus 2, while others are only paired with Claude Code, so withholding pairs can mix compositional generalization with model cold starts. Although Table 2 includes Comp. Avg. (all), the learned component-only variant is evaluated only on observed routes in Table 3, leaving the interaction term’s contribution to unseen-pair generalization unclear.

W2. Table 1 lacks a baseline that selects the best fixed model–harness pair based on training accuracy, making the actual benefit of query-level routing difficult to assess. Although HM-Router outperforms Fixed in Table 6, Fixed itself exceeds the other learned baselines at higher budgets. Moreover, this comparison covers only six benchmarks with different candidate pools. Table 6 already suggests cost savings, but it uses realized test spend to identify affordable settings. Clarifying how operating points would be selected before test evaluation, together with a direct matched-accuracy cost comparison, would strengthen the evidence.

Questions for authors
  1. Could you compare HM-Router with the training-selected best fixed pair on all 12 benchmarks under Table 1’s protocol?
  2. Can HM-Router match the success rate of the best fixed pair at lower realized cost when both the fixed pair and routing settings are selected using only training or validation data?
  3. How does HM-Router compare with its learned component-only variant under the same withholding protocol when both components of each held-out pair remain observed in other training combinations?
  4. How is execution cost estimated for completely unseen model–harness pairs, given that Equation (8) uses a route’s mean training cost?
  5. Which model–harness pairs missing from Table 5 are actually infeasible, rather than simply untested?
1 comment
AuthorsOpenReview ↗

Thank you and here is our rebuttal

Q1 / W2 — Best fixed pair under Table 1's protocol.

We added Fixed, which is the pair with the highest training accuracy, chosen separately on each split.

RandomIRTAvengersEmbedLLMFixedHM-Router
Mean (12 benchmarks)49.5866.6566.6165.4970.9673.93

HM-Router beats Fixed by +3.0 points on average. It wins on 8 of 12 benchmarks and on 28 of the 41 splits where the two differ. Fixed is a strong reference on agent benchmarks because a single dominant route often exists. The largest gains come where routes are complementary: ScienceAgentBench (+17.5), TauBench (+9.9), Mind2Web (+7.6) and SciCode (+7.5).

Q2 / W2 — Operating points chosen without test data, and matched-accuracy cost.

We re-ran the cost evaluation with every knob selected on training data only. λ for HM-Router, the analogous knobs for the baselines, and the cheapest qualifying route for Fixed are all chosen inside the training split under the budget constraint. Each choice is then frozen, the model is refit on the full training split, and test is evaluated once.

≤ $/task0.7511.523510
Fixed48.850.251.559.659.259.065.7
Av-Pro51.351.952.954.455.456.556.5
IRT51.952.552.954.254.354.654.6
EmbedLLM51.552.153.457.257.257.258.4
HM-Router57.959.962.768.568.568.268.2

HM-Router leads at every budget. Its realized test spend is $0.39–$1.03 per task. In the matched comparison below, Fixed is the train-selected pair, which reaches 65.7% at $1.79/task. HM-Router, with λ chosen on validation to stay within Fixed's training cost, reaches 66.7% at $0.90/task. That is the same accuracy at 50% of the cost.

Q3 / W1 — Component-only variant under withholding; held-out pairs whose components stay observed.

We ran two withholding settings. All are rerun under one pipeline.

(a) Random removal, where both components of almost every hidden pair stay observed in other routes.

Routes removed10%25%50%75%
HM-Router71.270.867.364.1
Component-only67.166.765.863.5
Comp. Avg. (all)66.866.266.463.1

HM-Router beats component-only by +4.1 points at 10% and at 25%. It beats Comp. Avg. by +4.5 and +4.6 points at the same levels.

(b) The withheld pairs themselves. See GfWJ Q2.

The interaction term mainly sharpens selection when most of the pool is observed. Transfer to never-executed pairs is carried by the shared component representations, which both variants use. We will state this division explicitly in Section 5.3.

Q4 — Cost of an unseen pair. In Eq. 8, ĉ(r) comes from training runs, so it applies to observed routes. In the cost experiments every candidate route is observed. For an unseen pair we use the same factorization: log ĉ(m,h) = a + α_m + β_h, fitted on the observed route costs. If we treat every route as unseen and replace all costs with these estimates, HM-Router's accuracy at ≥ $2/task changes by less than 1 point. It stays above every baseline that uses the true costs.

Q5 — Infeasible versus untested pairs. Genuinely infeasible cells come from interface requirements. SeeAct acts on screenshots, so text-only models (DeepSeek-R1/V3) cannot run under it on Online Mind2Web. In Section 5.3, every hidden pair is feasible by construction, since it was executed. Other pairs are untested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.