- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors study personalized model routing for video generation. They build a dataset(TasteRoute-3K) with individual preference labels , which separately annotates video quality and subjective user preferences. In addition, the authors propose a personalized video generation router (TasteRoute). It selects the best video generator for a user based on the input request, user preferences, and budget limit. Experiments show that this method meets user preferences while reducing generation costs.
Critical strengths
This paper studies the practical problem of routing video generation models across video quality, user preferences, and generation costs, showing strong engineering value. In addition, the authors build a dataset with individual preference labels, which provides useful data support for personalized video generation research.
Critical weaknesses
1. Limited statistical evidence. With 9 annotators and a 15% per-annotator test split (roughly a dozen groups each), most differences in Fig. 3B and Fig. 4 are within 1–2 NDCG points, and no confidence intervals or significance tests are reported. The conclusion's claim that TasteRoute "outperforms every personalized baseline at every budget cap" is not clearly supported: the curves in Fig. 3B largely overlap, and PersonalizedRouter-GNN appears comparable in Fig. 12B.
2. Modest value of learned routing. The generic router (39.9% Top-1) barely exceeds a single global-best generator (36.6%) and kNN (39.7%).
3. Personalization does not match the motivation. Each annotator gets an independently trained model, so e_u reduces to a per-user bias with no cross-user sharing.
4. Data construction may bias the analysis.
- Pair sampling prioritizes similarly ranked generators, which likely deflates agreement rates and biases the BT fit.
- Duplicate endpoints of the same model are treated as distinct candidates, inflating disagreement and flattening the winner distribution.
- FLUX 3 generated some I2V reference images while also being a candidate.
5. Weak cost-saving claim.
- Prices are per second while budgets are per video, and the cost computation is unspecified.
- The 8–12% saving is measured against the most expensive alternative and amounts to roughly $0.05 per video.
6. Evaluation and presentation issues.
- NDCG structurally penalizes cascades (Octo-Human-Label reaches 58% Top-1 but ranks last on NDCG), yet it is the primary metric.
- Fig. 3 and Fig. 4 use different NDCG definitions.
Questions for authors
- Personalized ground truth and significance. How is personalized ground truth defined when the router selects a generator the annotator never ranked? How many test pools/pairs does each annotator have? Please provide confidence intervals or significance tests for the main differences in Fig. 3B, Fig. 4, and Fig. 14.
- Top-1 definition. In Table 3 (λ = 0), Top-1 decreases as the budget increases (47.01% at 0.5 vs. 42.51% at 1.0), whereas it increases in Fig. 12B. Section 6 also states that an unaffordable best generator remains the Top-1 target. Please reconcile these definitions and subsets.
- Cost and pool construction. How is per-video cost computed from per-second prices and duration? Do videos within a pool share duration and aspect ratio? How were the generators in each pool selected, and is that selection correlated with price?
- Joint model. Is the "jointly trained router" in Fig. 5 a multi-user model different from TasteRoute-Personal? If so, please report its results in the Fig. 3B and Fig. 14 settings.
- Sampling bias. What are the agreement rates and the 34–55% statistic when computed only on randomly sampled groups?
5 comments
Addressing the 4 Questions
We sincerely thank the reviewer for the thorough and constructive comments. And also please refer to the revised paper submission for any updated Figures. We'll present newly added tables or experiments here instead.
Q1 Personalized ground truth and significance.
- Personalized ground truth : How is personalized ground truth defined when the router selects a generator the annotator never ranked? If the annotator never ranked a video before, we will never know how the annotator would prefer this video more or not, unless the annotator “labeled” it or we estimated a pseudo label for it. The first part is what we have done in this work, so in our dataset the testing and ground truth will always be made from annotators’ preference rankings. Even in an online learning scenario, we shall collect annotations of new generators’ videos to define personalized ground truth. Under our current evaluation setting, the latter part is not supported as this involves training a video quality judge, hence out of the scope of our focus in personalized hard video routing setup. We leave this for future work.
- Q1 Personalized ground truth :How many test pools and pairs does each annotator have? We evaluate personalization with grouped 5-fold cross-validation. Near-duplicate prompts are grouped so they never fall in different folds, and every method is retrained on each fold, so every annotator–prompt case is held out exactly once. This gives 1,523 test pools and 15,690 pairwise comparisons in total. | Annotator| Test pools | Pairwise comparisons | Mean pool size| | -------- | ------- | -------- | ------- | | A1 | 322 | 4,644 | 6.4 | | A2 | 254 | 2,438 | 4.9 | | A3 | 314 | 4,391 | 6.0 | | A4 | 85 | 281 | 2.9 | | A5 | 112 | 624 | 3.7 | | A6 | 122 | 888 | 4.2 | | A7 | 107 | 716 | 4.1 | | A8 | 102 | 881 | 4.6 | | A9 | 105 | 827 | 4.4 | |Total|1,523|15,690|5.1|
- Significance results: we added confidence intervals in Figure-3 and Fig 14. Overall, TasetRoute achieves comparable results with all baselines at different budgets. While TasteRoute outperform all baselines except kNN in opposite preferences, and TasteRoute outperform kNN in "shared preferences only", it shows TasteRoute to be the only model that better learns shared and opposite preferences at the same time.
Q2 Top-1 definition:
Thank you for carefully finding out about this issue. Yes, the Top-1 definition is different from what figure 12 used. To better align all our metrics, we change the top-1 definition of Table 3 to be the same as figure 12 does now: the original Table 3 uses the affordable generators under the budget as the denominator of the Top-1. Now we change to the whole generator pool. In this way the trend would be increasing instead because more and more affordable generators will be counted into the prediction pool, which make Table 3, Top-1 by λ (cost savings unchanged) align with Fig 12 trends. Please refer to the revised Table 3 in rebuttal submission version paper.
Q3 Cost and pool construction:
- Per-video cost computation from per-second prices and duration? Sharing of duration and aspect ratio?
Thank you for the question. In Table 4, each generator's cost is mainly based on video duration required ranging from 4s, 6s, 10s ( depending on the original input requirements ) most of the models follows the following pricing rule: Per-second pricing: cost = rate × duration. Example: Veo 3.1 Fast at 720p is $0.08/s × 6 s = $0.48. Per-video pricing (e.g., Wan 2.2-5B): the listed flat price, $0.15. Base plus extra seconds (Kling 2.5): base price for 5 s, plus a per-second charge beyond that. All videos in the same pool follow the same input and video length requirements, they are all the same duration and aspect ratio.
- Selection of generators in each pool? Whether that selection is correlated with price? We included all commercially available T2V/I2V APIs on fal.ai / Vertex AI. The pools range from average $0.06 to $1.45 per video, so cheap, mid-range and premium models all appear in the same pool. The generators in each pool are selected based on prior rounds of labeling, we cluster similar performance generators in each pool with 2 overlapping anchor generators between pools included. We did a calculation of pool size and average price and found spearman correlation of -0.05 showing there’s no correlation between selection and price.
Q4 Joint model.
Yes, they are different. We updated the single personalized tasteRoute trained jointly as “Shared TasteRoute” in Fig 3B and FIg 14. As we see the shared version is slightly worse than the separated trained versions, we think the simple jointly trained models suffer more on those disagreements from annotations, and separately training could completely avoid the influence from disagreement in personalization. Please refer to the revised paper fig and last question of reviewer pqaz for more detail.
Q5 Sampling bias issue
Thank you very much for the question. We believe there are reasons and contexts we need to explain before answering why our samples prioritize similarly ranked groups: In the nearly 30 generators, there are wide different gaps of model generation capability. E.g. In terms of acceptability scored by expert annotators, grok-imagine-video-v1.5 (for T2V) and veo-3.1-lite-generate-001 (for T2V) has 87% and 76% accepted quality that has no motion blurring, subject/object inconsistency and text blurring issues at all in the generated videos. However, generators like pika-v2.2, ltx-video-13b-distilled and wan-v2.2-a14b have very low accepted quality (9%, 10%, 26%), in which wan-v2.2-a14b failed mostly on motion artifacts ( 30% failure rate vs 0 to 4% in Veo and grok), and ltx-video-13b-distilled failed mostly on human realism (25% failure rate vs 1 to 4% in veo and grok), not to mention pika has 70% on motion failure rate. Given such huge capability gaps, randomly selecting generators for annotation groups would waste our labeling effort and not make personalization accurate for similar ranked generators that are actually hard to rank because of human subjective judges and their similar qualities. Although we don’t have the fully randomly sampled group’s annotations to examine, we expect the agreements of randomly sampled annotations will be much higher as there will be more pairs of significantly capability-different generators annotated together, which are very easy to judge. But we need to emphasize that in this way it won’t benefit the personalization training. We will rephrase our wording to emphasize the importance of the method for generator selection in annotation groups.
Addressing the 6 Weakness Points
We sincerely thank the reviewer for the thorough and constructive comments. Please refer to the revised submission for the updated figures and numbers requested during the rebuttal, as well as our related responses to other reviewers.
We will rephrase our conclusions to “comparable” or “competitive” instead of “outperform” relative to personalized baselines, consistent with the abstract and main text. We also updated Fig. 3 and Figs. 11–14 with new baselines and confidence intervals in the revised version. For general routing, TasteRoute is competitive with baselines, but our main focus is personalization in video routing: TasteRoute outperforms kNN in personalized routing at $1.5 cap (Fig. 3-B), but TasteRoute outperforms kNN in shared preferences from the experiment we added in reviewer kh1f 's weakness 1, and all baselines except same-user kNN in the opposite-preference tests (Fig. 14), showing personalized TasteRoute is the only method that learns both shared and opposite preferences better than other baselines.
We also add a shared-embedding TasteRoute to Fig. 3. It is slightly worse than the separately trained version, indicating that opposite preferences harm rather than benefit joint training. The conclusions still hold: it remains competitive in general and personalized routing (Fig. 3-B) and significantly outperforms all baselines except same-user kNN and GNN on opposite preferences. TasteRoute can thus be trained separately or jointly; we view joint training as a trade-off between personalization bias in training and personalized performance.
Please refer to our response to Question 3.