acceptodds
Under review as a conference paper at ICLR 2027

MRG-Router: Pre-Rendering Routing via Capability Probes for Multi-Reference Image Generation

Abstract

Multi-reference image generation (MRIG) requires preserving visual entities across multiple exemplars while following textual instructions, with applications in personalized advertising, e-commerce display, and creative editing. However, diverse generative models exhibit query-dependent complementarity alongside differences in inference costs, meaning that statically deploying a single model is either expensive or sub-optimal in quality. To address this issue, we study pre-rendering routing, selecting the optimal generator prior to rendering to maximize capability gains without candidate rendering overhead. Distinct from existing routers relying on static model identities or coarse global rankings, we propose MRG-Router to address the multimodal evidence mismatch prior to rendering by representing generators through empirical behavior rather than identity. Specifically, MRG-Router profiles candidate generators on a compact Capability Probe Bank and uses Query-Conditioned Probe Reweighting (QPR) to align probe-level evidence with incoming multimodal queries. For systematic evaluation, we construct MR-RouteBench, a dense benchmark systematically covering twelve representative compositional scenarios with associated compute traces. Evaluations across MR-RouteBench and two external benchmarks demonstrate that under quality-driven routing, MRG-Router outperforms the single strongest fixed generator. Under practical deployment constraints, it preserves visual quality comparable to the best fixed model while reducing generation costs by up to 96.0%, achieving superior quality-cost trade-offs over competing routing baselines. Furthermore, MRG-Router onboards unseen generators without parameter updates simply by appending probe profiles, demonstrating the potential of our routing framework to scale alongside advancements in generative modeling.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Model Selection, Multimodal Generation, Model Profiling, Model Onboarding, Cost-Aware Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_95xrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Before rendering, the paper routes each multi-reference generation query to one of several available generators. The authors constructed MR-RouteBench (comprising 9,000 queries and 10 generators, complete with VLM-based evaluation scores and cost data). Each generator’s capabilities are profiled using 512 probe queries. For a new query, the router retrieves similar probe queries and employs a lightweight re-weighting network to predict the quality of each generator, factoring in a cost penalty. New generators can be added simply by running them on the probe set, without requiring retraining.

Critical strengths
  1. The dense query-by-generator result matrix itself is quite useful. I can reproduce the anchor rows in Table 2 exactly using public data.
  2. The design is simple and suitable for open pools; adding a generator does not require retraining.
Critical weaknesses
  1. The performance gains are modest, and a simple baseline accounts for most of them. I verified a few points using public data:
  • On the test set, a quality-only router outperforms "Best Fixed" by 1.43 points, closing about 16% of the gap to the oracle. The 33.8% figure in Fig. 1c relies on an unspecified data split (Best Fixed: 48.4, Oracle: 58.2); these numbers match neither the validation set nor the test set.
  • Simply selecting a generator via a lookup table based on the number of reference images (K) yields a score of 49.08 (at a cost of $80/1K), whereas MRG-Router-Q achieves 49.30 (at $57/1K).
  • In cost-aware settings, consistently using SenseNova-8step yields 46.26 at $6.05/1K, while the router yields 46.93 at $6.03/1K. The claim of "up to 96.0% savings" (MICoBench: $5.33 vs. $134/1K) is derived by billing "NanoBanana" at "Pro" rates—a method Appendix A.6 terms a "conservative proxy." A cheap open-source model can already achieve most of these savings.
  • On the MultiBanana dataset, the cost-aware router performs slightly worse than "Best Fixed" in terms of quality (75.91 vs. 75.94); the Q–C variant also lags slightly (75.30 vs. 75.41) while being more expensive ($6.35 vs. $6.10/1K).
  1. The gains from integrating new models stem primarily from the models themselves rather than the routing mechanism. On a shared batch of 200 queries, using Wan2.7 alone yields a score of 48.87 (Table 10). Adding Wan to the routing pool raises the score to 49.06, but including both new models results in a score of exactly 48.87.

  2. All scores are derived from the author's own evaluator, and the router was trained using the same evaluation system. Its alignment with human judgment is moderate (τb = 0.528), and the gain of approximately 1 point has not been validated against human assessment.

Questions for authors
  1. How does MRG-Router-Q compare to the lookup-table method (based on reference map counts) and KNN? Please provide pairwise confidence intervals.
  2. On the dataset used to evaluate the new model, how often does the router actually outperform the "always use Wan2.7" strategy?
  3. There are three discrepancies between the published code and the appendix: the probe-ranking loss is missing; profile estimation uses uniform weights rather than the post-stratification weights from Eq. 6; and costs are normalized per query rather than following Eq. 16. Which version produced the results in Table 2? I cannot reproduce the router-related rows myself because the features, checkpoints, and prediction results are not included in the release package.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 95xr

Thank you for these questions. Below we compare the quality-only policies, report the query-level onboarding results, and clarify the implementation used in the experiments.

1. Quality-only routing versus the lookup and KNN

On the same 1,000 MR-RouteBench queries, MRG-Router-Q scores 49.30. The reference-count lookup is fitted on the designated training rows and selects NanoBanana for K=2/3 and SenseNova-8step for K=4/5. KNN below also uses a quality-only objective.

PolicyAvg.QOurs minus policyPaired 95% CI
Best Fixed47.87+1.43[0.65, 2.20]
Reference-count lookup49.08+0.22[-0.40, 0.84]
KNN48.80+0.50[0.07, 0.96]
MRG-Router-Q49.30N/AN/A

Intervals use 10,000 paired query bootstrap resamples after averaging selected quality over three training seeds. The intervals quantify query-sampling uncertainty for the evaluated models and split; they do not account for model or split selection. The difference from KNN has a positive interval; the lookup remains competitive.

2. How often does routing beat always-Wan?

The mean per-run win rate is 16.3% with Wan onboarding and 27.3% with both newcomers, each evaluated on the same 200 queries. A win means strictly higher automatic quality than Wan's output, not merely selecting a different generator.

Expanded poolRouted QWin rateDifference vs. WanPaired 95% CI
Base + Wan49.059916.3%+0.1917[-0.91, 1.31]
Base + Seedream + Wan48.865727.3%-0.0024[-1.59, 1.59]

Always-Wan scores 48.8681. The per-seed win/tie/loss counts are 33/139/28, 31/139/30, and 34/138/28 for Wan onboarding; 55/95/50, 54/95/51, and 55/94/51 for dual onboarding. Each triplet sums to 200. Intervals resample queries after averaging seed-level differences, rather than treating repeated runs as 600 independent queries.

The aggregate quality is close to always-Wan. The onboarding result demonstrates that the frozen matcher can use new generator evidence without retraining; improvements over the old pool also include the newcomers' own capabilities.

3. Implementation and supplementary material

Thank you for identifying these differences. The updated supplement gives the training objective and the settings used to compute the reported results.

Training. During warm-up, utility regression is combined with a gap-weighted probe-ranking loss, using weight 0.2 and a utility-gap threshold of 0.05. This term was omitted from the released training code and is now included. The QPR architecture is unchanged.

Capability profiles. The main experiments use uniform probe means; Equation 6 describes the weighted estimator. Both are now supported, and the supplement specifies which was used. With complete profiles and normalized shared probe weights, the centering term cancels when ranking generators. This explains why the choice of mean need not change the ranking, although it does not establish equivalence for missing outcomes or different training procedures.

Cost evaluation. The supplement distinguishes validation-based cost selection from the settings used for Table 2. On MR-RouteBench, scores are standardized and the penalty is scaled using the score and cost ranges in the evaluation set. On MICoBench, the matcher and semantic estimates are combined, with the mixing and cost coefficients selected using MICoBench labels. This is an in-benchmark calibration result, rather than an independent external-test result. The original release's per-query cost normalization is documented separately.

We also include the precomputed features, predictions for all three training seeds, query and model identifiers, and quality/cost matrices for the three main benchmarks. These allow the model choices and their scores to be checked without generating images or calling an evaluator. The original main-table weights could not be recovered, so those comparisons are reproduced from the saved predictions. For onboarding, the three trained models are provided; evaluating them reproduces the reported qualities and win/tie/loss counts exactly. The accompanying instructions describe both procedures.

Reviewer_95xrOpenReview ↗

raise my score from 2 to3.

Thank you for the author's response; it largely cleared up my doubts, so I have decided to raise my score to 3.

Reviewer_6RBUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper deals with the task of routing between models for multi-reference guided generation, where the aim is to select the model that can best produce an image following a given prompt and a set of multiple images that should affect the generated content (e.g., subjects that should appear in the image, styles to apply to it etc.).

To do this routing, the authors pre-evaluate the models on a calibration set with a fixed number of example probes. They then use features extracted from image and text encoders for the new query case, retrieve nearby probes according to their similarity to these features, and then pass the new query features and the nearby probe features through a learned router that predicts importance weights over the probes. Finally these weights are used to determine the best candidate generator models according to their performance on said probes. A key advantage of this approach is that new models can also be added to the pool by adding their results on the probes to the table, and the authors demonstrate that this works experimentally.

This routing mechanism is also, in my opinion, the core contribution of the paper.

Critical strengths
  1. The paper tackles a task with clear practical importance, and its solution for the task is reasonable and validated by significant experimental results including human evaluation.

  2. There is an extreme level of detail in the work that should easily facilitate reproduction. I could likely point an agent at the text of the work (and at the dataset) and reproduce virtually all of it. I'm not sure we are allowed to do this in practice so I didn't check, but either way the wealth of information and details is welcome.

Critical weaknesses

Unfortunately I feel like this work is far below the ICLR bar. Here is a list of the biggest issues I have with it:

  1. The paper doesn't do much new compared to prior (often uncited) work. The core idea of routing for model choice in image generation models appears in a range of prior work from 2-3 years ago (e.g., DiffusionGPT which the authors cite, and ComfyBench and ComfyGen which the authors do not). Some of those papers actually have calibration set based model routing as well (e.g., ComfyGen-IC) so this approach not new either. If anything, the prior works already deal with a larger or more complex scenario of combining different models into pipelines, which this work does not. The core differences between this method and such prior work are: (1) The specialization to multi-reference image based generation, but there is nothing inherently different in this setup and the authors actually just average over the embeddings of multiple references so the actual method doesn't even seem to focus on this difference. (2) The design of the query model, which is indeed interesting but its not trivial to me if its better, and if this was the main point of the paper (that current retrieval approaches are sub-optimal and that this is the right way to do it), I'd expect the story of the paper to focus on that and analyze why current retrieval approaches are bad, where they fail etc. In other words, at the very least the paper needs significant re-writing to focus on the one part in this work which is, to the best of my knowledge, actually new.

  2. As an aside to the above, the references in the paper could do with another pass. For example, the title of the paper and the intro heavily focus on reference-based image generation but the paper does not cite any of the canonical works in the field (DreamBooth, Textual Inversion, CustomDiffusion, IP Adapters, OminiControl etc.). This is actually more minor because the paper is actually a model routing paper and the setup of reference-based generation is almost inconsequential to the work.

  3. A lot of the metrics in the paper are aggregated scores over different requirements (reference fidelity, prompt alignment, visual quality) which are combined in entirely arbitrary ways. I can appreciate the need for a single score for the retrieval model, but I would still have liked to see how this method performs on each of the individual factors of the score and whether your arbitrary choice for combining them affects how well the router performs with respect to each of the options. Some of this information could potentially be gleaned from the supplementary tables but its certainly not trivial.

  4. The text of the paper appears to be heavily AI generated, with very wordy and repetitive paragraphs that often don't seem to provide additional info. At the same time, there are annotations used in the paper which are only defined in the supplementary if at all (for examples R_30B and R_8B, "Validation-scaled" cost, human regret and more). A lot of similarly important information is only found in the appendix (e.g., short descriptions of what the baselines the authors compare to actually do). While I suspect the use of LLMs goes far beyond the "sentence-level polishing" outlined in the AI-use statement (I believe a model wrote most if not all of the paper text), I can't prove this and thus did not penalize for it. Readability and annotation definitions however did affect my score. I suspect an agent wouldn't have had as much trouble reading the paper because most of these details are in there somewhere.

  5. It would have been nice to see visual comparisons or additional analysis on the experiments in table 2. What do the generated samples with the other routers look like? What are their failure modes that this approach fixes? Why is this the case?

  6. The paper links to an anonymous repo that was modified on september 30th, after the paper deadline.

Questions for authors

I am unlikely to flip my score to a positive one, but at the very least I'd like to know how this router compares to other routers across the different attributes you consider (R, P, V), and what if any are the failure modes of existing routers that this method overcomes.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6RBU: Part 1/2

Thank you for the detailed review and for pointing to the learned query-to-probe matcher as the part that deserves a clearer analysis. Below, we address the related work, component-wise results, presentation, and visual comparisons in the order raised.

1. Related work and the focus of the method

The methodological distinction is the learned weighting of individual probes, shared across generator outcome columns. ComfyGen-IC also uses prior performance evidence, but summarizes workflows by prompt categories and uses an LLM for selection. ComfyBench studies agents that construct and execute collaborative workflows. Our setting selects a frozen generator before rendering, with a matcher whose output dimension does not depend on the number of candidates. Adding a generator supplies another evidence column rather than another learned output head.

These works belong in the related-work discussion. Our contribution is this decision rule and its retraining-free extension, rather than image-generation routing or offline calibration in general. Workflow construction is a broader capability studied by prior work and is outside our experimental scope. Relevant references are ComfyGen and ComfyBench.

Both text and reference inputs contribute in the submitted ablation: text-only scores 51.08 ± 0.04, reference-only 51.43 ± 0.01, and the full input 51.64 ± 0.05 on Analysis Validation. Mean pooling is a simple way to supply visual context; it does not explicitly preserve reference order or entity bindings. The input ablation supports multimodal conditioning, while the learned probe weighting is the algorithmic point of the paper.

Your suggested reference-conditioning citations are also useful background: DreamBooth, Textual Inversion, Custom Diffusion, IP-Adapter, and OminiControl. They help distinguish progress in the underlying generators from the selection problem studied here.

2. Performance on individual quality factors

The gain over Best Fixed is mainly in reference fidelity and instruction fulfillment. To make the trade-off explicit, we joined the archived quality-only decisions with the existing component scores on the same 1,000 MR-RouteBench queries. R is reference fidelity, P holistic prompt following, A atomic requirement satisfaction, S the combined semantic factor, and V visual quality. Values are percentage points.

Quality-only policyRPASVFinal-Q
Best Fixed83.6985.6769.0877.1572.6147.87
Direct Predictor86.8187.0075.6381.3469.0649.26
MLP86.4186.5874.6280.5770.0649.46
ZOOTER-Adapt86.6387.0275.2981.1869.3749.32
KNN86.5786.7975.2481.0268.8948.80
PRkNN86.5586.6875.3481.0668.8248.75
RouterDC-Adapt86.4886.6974.8280.7569.8349.36
SCOPE-Adapt85.9886.6874.9480.8068.7748.40
Uniform Semantic-Probe86.5487.0075.0781.0469.6249.32
MRG-Router-Q86.7887.0275.3381.2169.3349.30

Compared with Best Fixed, ours gains 3.09 points in R, 6.25 in A, and 4.06 in S, with a 3.28-point decrease in V. This explains why an image can be more faithful to the requested composition without scoring higher on standalone visual quality. The comparison also makes differences among routers visible: MLP has higher V and slightly higher Final-Q, while ours has higher R and A. All entries use the same quality-only objective; Final-Q averages per-output products, not the product of the displayed column means.

The aggregation check and replies on presentation and visual comparisons follow in Part 2.

AuthorsOpenReview ↗

Response to Reviewer 6RBU: Part 2/2

This continues Part 1 with the aggregation check, definitions, and visual comparisons.

2. Performance on individual quality factors (continued)

The aggregation is intended to penalize failure in a required dimension. For example, a visually polished image that drops a reference subject should not be compensated entirely by its appearance. We examined the role of this choice by rescoring the same fixed routing decisions under alternative aggregations:

Fixed-decision scoring ruleMRG-Router-QOriginal Best Fixed
R × S × V49.3047.87
mean(R, S, V)79.1177.82
geometric mean(R, S, V)77.8976.73
R × P × V, without the atomic term52.2952.21

Arithmetic and geometric averaging preserve a positive difference over the original fixed model. Removing the atomic term makes the difference much smaller. Thus, the evaluation places meaningful weight on explicit compositional requirements. This is a fixed-policy sensitivity check: neither the routers nor the fixed-model identity were reselected for the alternative objectives.

3. Readability and definitions

The paper would be easier to follow with shorter descriptions and definitions placed alongside their first use. R30B and R8B refer to the reference-fidelity judge branches associated with Qwen3-VL-30B and Qwen3-VL-8B. Human regret is the highest human mean score for a query minus the human mean score of the evaluator-selected output, on the original 1–10 scale. Short baseline descriptions also belong with the main comparison, rather than only in the appendix.

For cost, validation-scaled refers to the standard experimental finalizer's use of a scale derived from validation data. The submitted cost-aware rows also involve row-specific table-generation settings, described in our implementation response. Those settings need to be distinguished from the standard rule. We appreciate the point about readability and take responsibility for making the method and evaluation definitions clear.

4. Visual comparisons and the behavior of different routers

The qualitative comparison is provided in Appendix Figure 5. It shows the original queries and selected outputs, with the router selections labeled. Two examples connect directly to the component analysis:

ExampleOurs / SCOPE selectionOther selections
Figure 5, row 3Final-Q 0.76KNN / RouterDC / PRkNN: 0.74
Figure 5, row 4Final-Q 0.53KNN / ZOOTER / PRkNN: 0.39

These examples illustrate differences in reference preservation and compositional fulfillment. The proposed mechanism addresses the evidence-selection step: probes that are close in embedding space need not be equally informative about relative generator quality. QPR learns to reweight that neighborhood using routing supervision. The figures illustrate this motivation; the component table gives the aggregate evidence. Neither analysis identifies a failure mode exclusive to one baseline, and the first row of Figure 5 also illustrates a quality–cost trade-off.

I thank the authors for their response. I have read through it (again with some difficulty because it appears to be heavily LLM generated with many of the same faults as the core paper).

The breakdown of improvements into different factors is interesting, and I think seeing the tradeoffs between routers is worthwhile, though the gaps here are small which makes them a bit harder to judge. Can you also explain how this Final-Q score differs from the Avg.Q in the paper which has different numbers for some models? For example MLP has an Avg.Q score of 46.29 on MR-RouteBench in table 2 (lower than your 49.3), while it has a Final-Q score of 49.46 in the rebuttal (higher than your 49.3). If this is a different way of aggregating the numbers, then does this not show that some of your "wins" in the paper is largely dependent on the way you aggregate the scores?

The visual examples are a good improvement and do indeed show that your approach usually picks the better performing models (it could be nice to also see which model is which in each case but I believe we can already tell that you didn't just learn 'always pick nano banana' or something of the sort because of price differences).

On the flip side, the remarks that prior methods address more complex scenarios (workflows rather than just a single model choice) is not a big point in your favor since a single model is an instance of a workflow. Your algorithmic approach doesn't actually care about this distinction either and could probably be applied to the more general case. This is indeed the same family of work and I still think the existence of many such prior papers hurts the novelty of this one.

Finally, my main concern remains that the core story of the paper is largely decoupled from its chief novelty and I'm not sure the level of change needed here to focus on the novel part and the experiments it requires is reasonable for a rebuttal.

I await your response to the Q score question and will see what other reviewers have to say before I decide if to increase my score to a 2.

AuthorsOpenReview ↗

Response to Reviewer 6RBU

Thank you for following up and for your feedback on the component analysis and visual examples. We first clarify the Q scores, then address the distinction from prior work.

1. Final-Q and Avg.Q use the same scoring rule; the routing objective differs.

For each generated output, Final-Q is Q = R × S × V. Avg.Q is the mean of these per-output scores for the outputs selected by a router, reported in percentage points and averaged over three training seeds. The Final-Q column in our previous response reports exactly this same mean. We should have called that column Avg.Q to keep the terminology consistent.

The MLP row in Table 2 uses cost-aware selection, whereas the component table in our response uses quality-only selection for every router. The 49.30 entry you mention is the separately labeled MRG-Router-Q row under “Quality-only routing” in Table 2. The relevant results on the same 1,000 MR-RouteBench test queries are:

MethodSelection objectiveAvg.QCost (USD / 1K queries)
MLPQuality–cost46.296.52
MLPQuality only49.4672.83
MRG-RouterQuality–cost46.936.03
MRG-Router-QQuality only49.3056.78
Best FixedFixed generator47.87134.00

For MLP, removing the cost penalty changes which generator is selected for each query. Its average quality rises from 46.29 to 49.46, while its average cost rises from 6.52 to 72.83. The test queries, candidate quality scores, and quality aggregation are unchanged; these are different selections, not different ways of scoring the same selected outputs.

Under the same quality-only objective, MLP does have a slightly higher average quality than ours, 49.46 versus 49.30. Our corresponding average cost is lower, 56.78 versus 72.83, although these are not matched-budget results. The quality-only result supports an improvement over Best Fixed, from 47.87 to 49.30, rather than superiority over every router.

This explains the numerical discrepancy, but your broader question about aggregation is still relevant. Our earlier sensitivity analysis rescored fixed selections under alternative rules. It supports a narrower observation about those selections, not a claim that all router rankings remain unchanged under any aggregation rule.

2. The methodological contribution is the shared query-to-probe weighting, not the restriction to single-model selection.

We agree that a single generator is a special case of a workflow, and that these prior papers belong to the same line of research. Our distinction should therefore rest on the decision mechanism. The matcher learns which individual probes are informative for a query and applies the same learned weights to every candidate's probe outcomes. A new generator can then be added through another outcome column, without changing or retraining the matcher. Uniform probe retrieval and the learned reweighting are compared in the component analysis; onboarding tests the extension to new candidates.

We appreciate your point that this mechanism should be more central to the paper's narrative. The revised presentation should connect the motivation, method, and analysis around it, rather than rely on the narrower task scope as an argument for novelty. We also agree that naming the generators in the visual examples would make the selections easier to interpret.

Reviewer_7dbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

MRG-Router chooses one frozen multi-reference generator before rendering. Generators are profiled on a shared bank of 512 probes. A request retrieves similar probes, a small network reweights them, and a cost term can be subtracted. On MR-RouteBench, quality-only routing is slightly above the best fixed model. The larger reported gain is a lower bill when the pool includes a commercial model.

Critical strengths

The same queries are run on all ten generators, with cost recorded per query. The gap between one fixed model and the per-query oracle is measured.

Critical weaknesses
  1. I do not see a new technical idea. Deciding whether an object matches a reference is handed to a pretrained VLM, with no change to that model. The rest is frozen encoders, retrieval over past queries, and a small reweighting network. The pipeline is an assembly of existing pieces.

  2. This setting has little practical value. I have worked in this setting, and the user data I have supports the behavior I describe: users who pay already choose the stronger model. Nano Banana is already inexpensive, and the reported saving is against always calling it. A probe bank, a pool of live generators, and a learned router do not create a use that these users need.

Questions for authors
  1. What is new here, beyond retrieval over past queries and an off-the-shelf VLM for object similarity?

  2. Given that users who pay already select the stronger model, and that Nano Banana is already inexpensive, where does this setting have practical value?

A convincing answer would make me consider raising score.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7dbc

Thank you for your clear summary and for highlighting the value of the query-by-generator measurements. Your questions concern the routing mechanism itself and its practical use; we address them in that order.

1. What is learned beyond retrieval and reference similarity?

The methodological contribution is a learned query-to-probe weighting rule shared by all generators. The VLM provides offline quality labels; it is not used to compare candidate outputs when a request is routed. At inference, the system receives the instruction and reference images, selects one generator, and then renders an image.

Semantic retrieval and QPR have different roles. Retrieval finds probes with similar content. QPR learns which of those probes are useful for estimating the relative performance of the available generators. This separates content similarity from decision relevance: two historical queries can be similarly close to the incoming request yet provide different evidence about which model to choose.

For complete profiles, the resulting generator ranking can be expressed as

\qquad \sum_{p}\alpha_\theta(x,p)=1.$$ Here, the learned weights depend on the query and probes, while the generator is represented by its measured outcome column. Appending a new generator's probe outcomes therefore expands the candidate pool without changing the matcher parameters. The new generator still needs offline profiling, but not another matcher-training stage. The submitted ablations help assess this design. On Analysis Validation, full multimodal reweighting scores 51.64, compared with 50.97 for uniform aggregation, 51.08 for text-only input, and 51.43 for reference-only input. These controlled comparisons support using both modalities to weight the probe evidence. For a matched-objective test comparison, we also reanalyzed the archived decisions on the same 1,000 MR-RouteBench queries. All policies below optimize quality rather than mixing quality-only and cost-aware operating points. | Quality-only policy | Quality | Ours minus policy | Paired 95% CI | |---|---|---|---| | Best Fixed | 47.87 | +1.43 | [0.65, 2.20] | | Reference-count lookup | 49.08 | +0.22 | [-0.40, 0.84] | | KNN | 48.80 | +0.50 | [0.07, 0.96] | | Uniform Semantic-Probe | 49.32 | -0.02 | [-0.29, 0.24] | | MRG-Router-Q | 49.30 | N/A | N/A | The quality gain over Best Fixed is 1.43 points, and the gain over quality-only KNN is 0.50 points. The reference-count lookup and uniform probe aggregation are also strong baselines. Thus, the controlled ablation supports the learned weighting in its analysis setting, while the test comparison shows that simple probe-based alternatives remain competitive. The two allocations answer complementary questions. The lookup is fitted on the designated training rows. Intervals use 10,000 paired query bootstrap resamples, after averaging selected quality over three training seeds for each query. They describe uncertainty for the archived models and split, rather than accounting for earlier configuration or split exploration. The contribution is therefore the shared, learned evidence-weighting mechanism and its support for an evolving generator pool. Frozen encoders and similarity retrieval are the components it builds on; the dense benchmark makes its behavior and alternatives measurable. ## 2. When is routing useful if users already choose a strong model? Routing is most relevant to a service handling many heterogeneous requests, where the best model on average need not be the best choice for every request. On the archived MR-RouteBench test set, Best Fixed scores 47.87, the quality-only router scores 49.30, and the per-query oracle scores 56.61. These values measure the complementarity that remains after selecting the strongest average model. For such a service, the useful decisions include choosing a model suited to the current request, changing the quality–cost operating point, and incorporating a new generator without retraining the matcher. This differs from asking an individual user to maintain a live pool of models. A user satisfied with one generator and its price may reasonably keep that fixed choice; the service setting is where the additional profiling infrastructure can be amortized. The monetary comparison also depends on deployment assumptions. The reported savings use the paper's declared proprietary-output pricing proxy and measured self-hosted operating points. Cheap fixed generators already account for substantial savings, so the percentage should be read under that pricing setup rather than as a universal benefit of learning. The practical aim is to use model complementarity before rendering and to keep the selector usable as the pool changes.
AuthorsOpenReview ↗

Response to Reviewer 7dbc

Thank you again for raising the questions about technical novelty and practical value. We would like to add a direct comparison with fixed generators that does not depend on API pricing, and clarify how it relates to the proposed mechanism.

Practical value beyond a lower generation bill. Your observation that paying users often prefer a strong model is important. The question we can test is whether choosing the strongest generator on average already captures the available quality. On the same 1,000 MR-RouteBench test queries, using the same automatic quality measure throughout:

Quality-only policyAvg.QMRG-Router-Q minus policy
Best Fixed: NanoBanana47.87+1.43
Best self-hosted fixed: SenseNova-8step46.26+3.04
MRG-Router-Q49.30N/A

The last column is MRG-Router-Q minus the fixed policy. These comparisons use no cost penalty. Their paired 95% intervals are [0.65, 2.20] and [2.39, 3.69] points, respectively, using 10,000 query bootstrap resamples after averaging differences over three training seeds. They quantify uncertainty for the evaluated models and split, not for model or split selection.

Thus, the observed benefit is not solely a saving relative to an expensive API: request-dependent selection improves average quality over both the strongest fixed candidate and the strongest self-hosted fixed candidate in this evaluation. This does not establish a need for every individual user. The intended use is a service that already offers multiple generators and handles repeated, heterogeneous requests. The service maintains the profiles and selector; users need not manage the pool themselves. Our evidence concerns model complementarity on these requests, rather than a claim about how all paying users behave.

What the learned mechanism contributes. The design choice is to learn weights over historical probes, rather than learn a separate output head for each generator. Retrieval identifies similar queries; the matcher learns how to combine their measured outcomes for a routing decision. Crucially, the same weights are applied to every generator's outcome column. An unseen generator can therefore become selectable after profiling it on the existing probes, without changing the matcher's parameters or existing profile columns. The pretrained VLM supplies offline supervision; it does not inspect current-query candidate outputs during routing.

The fixed-model comparison above tests the value of request-dependent selection, not the isolated effect of learned reweighting. The submitted component study provides complementary evidence on Analysis Validation: uniform aggregation scores 50.97 and the full method 51.64. The onboarding experiments address a separate property: whether the trained matcher remains usable when new generators enter the pool. Together, these explain the intended contribution more precisely than treating frozen encoders, retrieval, or VLM judging individually as new techniques.

We hope this clarifies why the work is not premised only on commercial-model pricing: it studies selection from measured generator behavior before rendering, including how to retain that selector as the available generators change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.