- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Before rendering, the paper routes each multi-reference generation query to one of several available generators. The authors constructed MR-RouteBench (comprising 9,000 queries and 10 generators, complete with VLM-based evaluation scores and cost data). Each generator’s capabilities are profiled using 512 probe queries. For a new query, the router retrieves similar probe queries and employs a lightweight re-weighting network to predict the quality of each generator, factoring in a cost penalty. New generators can be added simply by running them on the probe set, without requiring retraining.
Critical strengths
- The dense query-by-generator result matrix itself is quite useful. I can reproduce the anchor rows in Table 2 exactly using public data.
- The design is simple and suitable for open pools; adding a generator does not require retraining.
Critical weaknesses
- The performance gains are modest, and a simple baseline accounts for most of them. I verified a few points using public data:
- On the test set, a quality-only router outperforms "Best Fixed" by 1.43 points, closing about 16% of the gap to the oracle. The 33.8% figure in Fig. 1c relies on an unspecified data split (Best Fixed: 48.4, Oracle: 58.2); these numbers match neither the validation set nor the test set.
- Simply selecting a generator via a lookup table based on the number of reference images (K) yields a score of 49.08 (at a cost of $80/1K), whereas MRG-Router-Q achieves 49.30 (at $57/1K).
- In cost-aware settings, consistently using SenseNova-8step yields 46.26 at $6.05/1K, while the router yields 46.93 at $6.03/1K. The claim of "up to 96.0% savings" (MICoBench: $5.33 vs. $134/1K) is derived by billing "NanoBanana" at "Pro" rates—a method Appendix A.6 terms a "conservative proxy." A cheap open-source model can already achieve most of these savings.
- On the MultiBanana dataset, the cost-aware router performs slightly worse than "Best Fixed" in terms of quality (75.91 vs. 75.94); the Q–C variant also lags slightly (75.30 vs. 75.41) while being more expensive ($6.35 vs. $6.10/1K).
-
The gains from integrating new models stem primarily from the models themselves rather than the routing mechanism. On a shared batch of 200 queries, using Wan2.7 alone yields a score of 48.87 (Table 10). Adding Wan to the routing pool raises the score to 49.06, but including both new models results in a score of exactly 48.87.
-
All scores are derived from the author's own evaluator, and the router was trained using the same evaluation system. Its alignment with human judgment is moderate (τb = 0.528), and the gain of approximately 1 point has not been validated against human assessment.
Questions for authors
- How does MRG-Router-Q compare to the lookup-table method (based on reference map counts) and KNN? Please provide pairwise confidence intervals.
- On the dataset used to evaluate the new model, how often does the router actually outperform the "always use Wan2.7" strategy?
- There are three discrepancies between the published code and the appendix: the probe-ranking loss is missing; profile estimation uses uniform weights rather than the post-stratification weights from Eq. 6; and costs are normalized per query rather than following Eq. 16. Which version produced the results in Table 2? I cannot reproduce the router-related rows myself because the features, checkpoints, and prediction results are not included in the release package.
2 comments
raise my score from 2 to3.
Thank you for the author's response; it largely cleared up my doubts, so I have decided to raise my score to 3.
Response to Reviewer 95xr
Thank you for these questions. Below we compare the quality-only policies, report the query-level onboarding results, and clarify the implementation used in the experiments.
1. Quality-only routing versus the lookup and KNN
On the same 1,000 MR-RouteBench queries, MRG-Router-Q scores 49.30. The reference-count lookup is fitted on the designated training rows and selects NanoBanana for K=2/3 and SenseNova-8step for K=4/5. KNN below also uses a quality-only objective.
Intervals use 10,000 paired query bootstrap resamples after averaging selected quality over three training seeds. The intervals quantify query-sampling uncertainty for the evaluated models and split; they do not account for model or split selection. The difference from KNN has a positive interval; the lookup remains competitive.
2. How often does routing beat always-Wan?
The mean per-run win rate is 16.3% with Wan onboarding and 27.3% with both newcomers, each evaluated on the same 200 queries. A win means strictly higher automatic quality than Wan's output, not merely selecting a different generator.
Always-Wan scores 48.8681. The per-seed win/tie/loss counts are 33/139/28, 31/139/30, and 34/138/28 for Wan onboarding; 55/95/50, 54/95/51, and 55/94/51 for dual onboarding. Each triplet sums to 200. Intervals resample queries after averaging seed-level differences, rather than treating repeated runs as 600 independent queries.
The aggregate quality is close to always-Wan. The onboarding result demonstrates that the frozen matcher can use new generator evidence without retraining; improvements over the old pool also include the newcomers' own capabilities.
3. Implementation and supplementary material
Thank you for identifying these differences. The updated supplement gives the training objective and the settings used to compute the reported results.
Training. During warm-up, utility regression is combined with a gap-weighted probe-ranking loss, using weight 0.2 and a utility-gap threshold of 0.05. This term was omitted from the released training code and is now included. The QPR architecture is unchanged.
Capability profiles. The main experiments use uniform probe means; Equation 6 describes the weighted estimator. Both are now supported, and the supplement specifies which was used. With complete profiles and normalized shared probe weights, the centering term cancels when ranking generators. This explains why the choice of mean need not change the ranking, although it does not establish equivalence for missing outcomes or different training procedures.
Cost evaluation. The supplement distinguishes validation-based cost selection from the settings used for Table 2. On MR-RouteBench, scores are standardized and the penalty is scaled using the score and cost ranges in the evaluation set. On MICoBench, the matcher and semantic estimates are combined, with the mixing and cost coefficients selected using MICoBench labels. This is an in-benchmark calibration result, rather than an independent external-test result. The original release's per-query cost normalization is documented separately.
We also include the precomputed features, predictions for all three training seeds, query and model identifiers, and quality/cost matrices for the three main benchmarks. These allow the model choices and their scores to be checked without generating images or calling an evaluator. The original main-table weights could not be recovered, so those comparisons are reproduced from the saved predictions. For onboarding, the three trained models are provided; evaluating them reproduces the reported qualities and win/tie/loss counts exactly. The accompanying instructions describe both procedures.