acceptodds
Under review as a conference paper at ICLR 2027

TasteRoute: Personalized Routing for Video Generation

20 views

Abstract

Rapid progress in video generation has produced a growing ecosystem of models that differ substantially in capability and generation cost. This raises a natural question: can each request be efficiently routed to an appropriate model? We first investigate whether video router can be learned from the generation input alone. We find that once outputs clear a basic quality bar, annotators' own favorites agree with the consensus pick only 34-55% of the time. Motivated by this observation, we introduce TasteRoute, a personalized video-generation router that selects a generator jointly based on input request, user preferences, and available generation budget. Across text-to-video and image-to-video settings, TasteRoute achieves competitive preference-routing performance with strong simple baselines while reducing average generation cost, with larger savings under higher budget caps. Finally, we release TasteRoute-3k, a human-annotated dataset containing multi-model video comparisons, quality judgments, preference rankings, and user-profile signals to facilitate future research on personalized and cost-aware video routing.

Primary area:
generative models
Keywords:
video routing, personalization, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_wqHMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study personalized model routing for video generation. They build a dataset(TasteRoute-3K) with individual preference labels , which separately annotates video quality and subjective user preferences. In addition, the authors propose a personalized video generation router (TasteRoute). It selects the best video generator for a user based on the input request, user preferences, and budget limit. Experiments show that this method meets user preferences while reducing generation costs.

Critical strengths

This paper studies the practical problem of routing video generation models across video quality, user preferences, and generation costs, showing strong engineering value. In addition, the authors build a dataset with individual preference labels, which provides useful data support for personalized video generation research.

Critical weaknesses

1. Limited statistical evidence. With 9 annotators and a 15% per-annotator test split (roughly a dozen groups each), most differences in Fig. 3B and Fig. 4 are within 1–2 NDCG points, and no confidence intervals or significance tests are reported. The conclusion's claim that TasteRoute "outperforms every personalized baseline at every budget cap" is not clearly supported: the curves in Fig. 3B largely overlap, and PersonalizedRouter-GNN appears comparable in Fig. 12B.

2. Modest value of learned routing. The generic router (39.9% Top-1) barely exceeds a single global-best generator (36.6%) and kNN (39.7%).

3. Personalization does not match the motivation. Each annotator gets an independently trained model, so e_u reduces to a per-user bias with no cross-user sharing.

4. Data construction may bias the analysis.

  • Pair sampling prioritizes similarly ranked generators, which likely deflates agreement rates and biases the BT fit.
  • Duplicate endpoints of the same model are treated as distinct candidates, inflating disagreement and flattening the winner distribution.
  • FLUX 3 generated some I2V reference images while also being a candidate.

5. Weak cost-saving claim.

  • Prices are per second while budgets are per video, and the cost computation is unspecified.
  • The 8–12% saving is measured against the most expensive alternative and amounts to roughly $0.05 per video.

6. Evaluation and presentation issues.

  • NDCG structurally penalizes cascades (Octo-Human-Label reaches 58% Top-1 but ranks last on NDCG), yet it is the primary metric.
  • Fig. 3 and Fig. 4 use different NDCG definitions.
Questions for authors
  1. Personalized ground truth and significance. How is personalized ground truth defined when the router selects a generator the annotator never ranked? How many test pools/pairs does each annotator have? Please provide confidence intervals or significance tests for the main differences in Fig. 3B, Fig. 4, and Fig. 14.
  2. Top-1 definition. In Table 3 (λ = 0), Top-1 decreases as the budget increases (47.01% at 0.5 vs. 42.51% at 1.0), whereas it increases in Fig. 12B. Section 6 also states that an unaffordable best generator remains the Top-1 target. Please reconcile these definitions and subsets.
  3. Cost and pool construction. How is per-video cost computed from per-second prices and duration? Do videos within a pool share duration and aspect ratio? How were the generators in each pool selected, and is that selection correlated with price?
  4. Joint model. Is the "jointly trained router" in Fig. 5 a multi-user model different from TasteRoute-Personal? If so, please report its results in the Fig. 3B and Fig. 14 settings.
  5. Sampling bias. What are the agreement rates and the 34–55% statistic when computed only on randomly sampled groups?
5 comments
AuthorsOpenReview ↗

Addressing the 6 Weakness Points

We sincerely thank the reviewer for the thorough and constructive comments. Please refer to the revised submission for the updated figures and numbers requested during the rebuttal, as well as our related responses to other reviewers.

Weakness 1 and 2, Limited statistical evidence and Modest value in generic routing:

We will rephrase our conclusions to “comparable” or “competitive” instead of “outperform” relative to personalized baselines, consistent with the abstract and main text. We also updated Fig. 3 and Figs. 11–14 with new baselines and confidence intervals in the revised version. For general routing, TasteRoute is competitive with baselines, but our main focus is personalization in video routing: TasteRoute outperforms kNN in personalized routing at $1.5 cap (Fig. 3-B), but TasteRoute outperforms kNN in shared preferences from the experiment we added in reviewer kh1f 's weakness 1, and all baselines except same-user kNN in the opposite-preference tests (Fig. 14), showing personalized TasteRoute is the only method that learns both shared and opposite preferences better than other baselines.

Personalization does not match the motivation:

We also add a shared-embedding TasteRoute to Fig. 3. It is slightly worse than the separately trained version, indicating that opposite preferences harm rather than benefit joint training. The conclusions still hold: it remains competitive in general and personalized routing (Fig. 3-B) and significantly outperforms all baselines except same-user kNN and GNN on opposite preferences. TasteRoute can thus be trained separately or jointly; we view joint training as a trade-off between personalization bias in training and personalized performance.

Data construction bias:

  1. Bias toward similarly ranked groups: We emphasize the importance of our label-group construction and the pitfalls of randomly allocating generators for annotation. First, pairwise ranking is widely used in quality evaluation (e.g., Artificial Analysis Video Generation Arena, Chatbot Arena) and recent routing work (RouteLLM, RouteT2I) to reduce prompt-difficulty bias and improve human judgment reliability over absolute scores. Second, under a limited budget, our initial round allocated more annotations to similarly ranked generators, helping the model distinguish similar-quality generators by subjective preferences. The 30 generators already span a wide human-judged acceptance rate (from 87% for veo-3.1-lite-generate-001 to 9% for pika-v2.2), so fewer strong-vs-weak pairs do not harm personalization training, as annotator agreement on such pairs is naturally much higher than on near-quality pairs. We thus consider this a reasonable choice under budget constraints, which, if anything, deflates user agreement by emphasizing harder pairs. We will clarify this in the revision.
  2. Duplicate endpoints of the same model: These endpoints use different generation configurations. The *-generate-001 suffix denotes 4-second videos for veo-3.1, veo-3.1-fast and veo-3.1-lite; endpoints without it generate 6–10 second videos. They never co-occur for the same prompt in our annotations. They are similar in quality but always produce videos of different lengths, so they are never identical and do not inflate personalized user agreement. We will rename them in the revision.
  3. FLUX 3 image and video models: These are different models. Reference images are generated with FLUX 3 Image Klein (5.6B, open-weight), while FLUX 3 video is a closed-weight video generation model. We apologize for the confusion and will rephrase L496 (Ethics statement).

Cost-saving claim:

Please refer to our response to Question 3.

Evaluation and presentation issues:

  1. Octo-Human-Label is a cascading “upper-bound” baseline: starting from the cheapest generator, it uses human labels to check whether the N selected generators include the ground truth. It targets top-1 prediction rather than the full ranking, so it naturally performs poorly on NDCG. Top-1 and NDCG capture different aspects; NDCG is more holistic, and we also report top-1 and top-3 scores in the paper.
  2. Thank you for pointing this out. Yes, the NDCG definitions differed; we have aligned the calculation with Fig. 3 using the full pool.
AuthorsOpenReview ↗

Addressing the 4 Questions

We sincerely thank the reviewer for the thorough and constructive comments. And also please refer to the revised paper submission for any updated Figures. We'll present newly added tables or experiments here instead.

Q1 Personalized ground truth and significance.

  • Personalized ground truth : How is personalized ground truth defined when the router selects a generator the annotator never ranked? If the annotator never ranked a video before, we will never know how the annotator would prefer this video more or not, unless the annotator “labeled” it or we estimated a pseudo label for it. The first part is what we have done in this work, so in our dataset the testing and ground truth will always be made from annotators’ preference rankings. Even in an online learning scenario, we shall collect annotations of new generators’ videos to define personalized ground truth. Under our current evaluation setting, the latter part is not supported as this involves training a video quality judge, hence out of the scope of our focus in personalized hard video routing setup. We leave this for future work.
  • Q1 Personalized ground truth :How many test pools and pairs does each annotator have? We evaluate personalization with grouped 5-fold cross-validation. Near-duplicate prompts are grouped so they never fall in different folds, and every method is retrained on each fold, so every annotator–prompt case is held out exactly once. This gives 1,523 test pools and 15,690 pairwise comparisons in total. | Annotator| Test pools | Pairwise comparisons | Mean pool size| | -------- | ------- | -------- | ------- | | A1 | 322 | 4,644 | 6.4 | | A2 | 254 | 2,438 | 4.9 | | A3 | 314 | 4,391 | 6.0 | | A4 | 85 | 281 | 2.9 | | A5 | 112 | 624 | 3.7 | | A6 | 122 | 888 | 4.2 | | A7 | 107 | 716 | 4.1 | | A8 | 102 | 881 | 4.6 | | A9 | 105 | 827 | 4.4 | |Total|1,523|15,690|5.1|
  • Significance results: we added confidence intervals in Figure-3 and Fig 14. Overall, TasetRoute achieves comparable results with all baselines at different budgets. While TasteRoute outperform all baselines except kNN in opposite preferences, and TasteRoute outperform kNN in "shared preferences only", it shows TasteRoute to be the only model that better learns shared and opposite preferences at the same time.

Q2 Top-1 definition:

Thank you for carefully finding out about this issue. Yes, the Top-1 definition is different from what figure 12 used. To better align all our metrics, we change the top-1 definition of Table 3 to be the same as figure 12 does now: the original Table 3 uses the affordable generators under the budget as the denominator of the Top-1. Now we change to the whole generator pool. In this way the trend would be increasing instead because more and more affordable generators will be counted into the prediction pool, which make Table 3, Top-1 by λ (cost savings unchanged) align with Fig 12 trends. Please refer to the revised Table 3 in rebuttal submission version paper.

Q3 Cost and pool construction:

  • Per-video cost computation from per-second prices and duration? Sharing of duration and aspect ratio?

Thank you for the question. In Table 4, each generator's cost is mainly based on video duration required ranging from 4s, 6s, 10s ( depending on the original input requirements ) most of the models follows the following pricing rule: Per-second pricing: cost = rate × duration. Example: Veo 3.1 Fast at 720p is $0.08/s × 6 s = $0.48. Per-video pricing (e.g., Wan 2.2-5B): the listed flat price, $0.15. Base plus extra seconds (Kling 2.5): base price for 5 s, plus a per-second charge beyond that. All videos in the same pool follow the same input and video length requirements, they are all the same duration and aspect ratio.

  • Selection of generators in each pool? Whether that selection is correlated with price? We included all commercially available T2V/I2V APIs on fal.ai / Vertex AI. The pools range from average $0.06 to $1.45 per video, so cheap, mid-range and premium models all appear in the same pool. The generators in each pool are selected based on prior rounds of labeling, we cluster similar performance generators in each pool with 2 overlapping anchor generators between pools included. We did a calculation of pool size and average price and found spearman correlation of -0.05 showing there’s no correlation between selection and price.

Q4 Joint model.

Yes, they are different. We updated the single personalized tasteRoute trained jointly as “Shared TasteRoute” in Fig 3B and FIg 14. As we see the shared version is slightly worse than the separated trained versions, we think the simple jointly trained models suffer more on those disagreements from annotations, and separately training could completely avoid the influence from disagreement in personalization. Please refer to the revised paper fig and last question of reviewer pqaz for more detail.

AuthorsOpenReview ↗

Q5 Sampling bias issue

Thank you very much for the question. We believe there are reasons and contexts we need to explain before answering why our samples prioritize similarly ranked groups: In the nearly 30 generators, there are wide different gaps of model generation capability. E.g. In terms of acceptability scored by expert annotators, grok-imagine-video-v1.5 (for T2V) and veo-3.1-lite-generate-001 (for T2V) has 87% and 76% accepted quality that has no motion blurring, subject/object inconsistency and text blurring issues at all in the generated videos. However, generators like pika-v2.2, ltx-video-13b-distilled and wan-v2.2-a14b have very low accepted quality (9%, 10%, 26%), in which wan-v2.2-a14b failed mostly on motion artifacts ( 30% failure rate vs 0 to 4% in Veo and grok), and ltx-video-13b-distilled failed mostly on human realism (25% failure rate vs 1 to 4% in veo and grok), not to mention pika has 70% on motion failure rate. Given such huge capability gaps, randomly selecting generators for annotation groups would waste our labeling effort and not make personalization accurate for similar ranked generators that are actually hard to rank because of human subjective judges and their similar qualities. Although we don’t have the fully randomly sampled group’s annotations to examine, we expect the agreements of randomly sampled annotations will be much higher as there will be more pairs of significantly capability-different generators annotated together, which are very easy to judge. But we need to emphasize that in this way it won’t benefit the personalization training. We will rephrase our wording to emphasize the importance of the method for generator selection in annotation groups.

Reviewer_oyB2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TasteRoute, a personalized routing framework for video generation that aims to select an appropriate video generator based on the input request, individual user preferences, and generation budget. To support this task, the authors construct TasteRoute-3k, a human-annotated dataset containing videos generated by multiple models, quality assessments, and individual preference rankings. The proposed framework uses pretrained multimodal embeddings, user and generator representations, and ranking objectives to predict generator preferences without performing video generation during routing. Experiments across text-to-video and image-to-video settings show that TasteRoute achieves competitive routing performance compared with existing baselines, while personalized routing provides modest improvements in preference alignment and reduces average generation costs under certain budget constraints. The authors further analyze how personalization benefits vary with budget availability, the amount of user feedback, and disagreement among annotators.

Critical strengths

1 The paper introduces TasteRoute-3k, a human-annotated dataset covering multiple video generators, quality assessments, and individual preference rankings. 2 The paper provides useful analyses of how personalization performance varies with budget constraints, the amount of user feedback, and disagreement among annotators. In particular, the results offer insights into when personalized routing is beneficial and when its advantages become limited. These analyses help provide a more nuanced understanding of personalized video generation routing beyond overall performance comparisons.

Critical weaknesses

1 While personalized video generation routing is an interesting problem, its practical significance could be better justified. In particular, it remains unclear whether modeling individual user preferences provides sufficient additional value over prompt-based routing or simply allowing users to select their preferred generation models. Given the additional effort required to collect user preference data and train personalized routers, a clearer demonstration of the practical benefits of personalization would strengthen the significance of this work. 2 Although the paper evaluates TasteRoute against several routing baselines, the performance improvements are relatively modest. As shown in Figure 3, TasteRoute achieves performance close to the simple kNN baseline across different budget constraints. Moreover, the personalized router achieves only 55.2% accuracy on conflicting-preference pairs, compared with 52.4% for same-user kNN. These results raise questions about whether the additional complexity of the proposed approach is justified by its empirical gains. Reporting statistical significance for the relatively small improvements would further strengthen the experimental conclusions. 3 Human preferences are subjective and may change over time, across different tasks, or under different usage scenarios. However, TasteRoute mainly relies on historical preference annotations to train an independent router for each user, implicitly assuming that these preferences are sufficiently stable to predict future choices. Although the model incorporates prompt information, the paper does not explicitly investigate the temporal consistency or stability of individual preferences.

Questions for authors

1 Could the authors further justify the practical necessity of personalized routing, particularly given that individual preferences may change across tasks and over time?

2 Given the modest improvements over simple baselines, could the authors clarify what evidence supports the effectiveness and generalizability of the proposed personalization framework?

4 comments
AuthorsOpenReview ↗

Addreesing the 3 Weakness Points

We sincerely thank the reviewer for the thorough and constructive feedbacks. And please refer to the revised paper submission for any updated Figures or digtis that were asked during the rebuttal session, or other reviewer's related response from us.

insufficient additional value over prompt-based routing or simply allowing users to select their preferred generation models

  • Allowing users to select their preferred generation models: No single model wins more than 9.5% of prompt sets (Table 2), and even the top consensus model (Veo 3.1 Lite) loses on 5 of 12 prompt attributes in T2V (and 8 of 12 in I2V, Figs. 1 & 9). Because video generation takes minutes and costs up to $1.45 per clip, users cannot manually trial-and-error across 30 models.
  • Prompt-based routing : At the same time, prompt-only consensus routing hits a hard ceiling because consensus matches an individual user's favorite only 34–55% of the time on clean videos (Section 4).

Additional complexity of TasteRoute and significant test.

TasteRoute is actually a simple MLP neural network with personal ranking head output for scoring the generators. It has much simpler complexity than other baselines like personalizeGNN and RouteLLM. But TasteRoute still gets comparable results to them, and even has significant results on opposite preference over them (please see the updated Fig. 14), while other baselines do not. Thus we think it’s not because of learning with additional complexity, but because of the training on preference and video prompts, which kNN and tasteRoute both benefits from, which other baselines only partly learn from. (Please refer to the updated Fig 3, 11-13 with the added confidence interval)

Temporal conssitency or stability of individual ppreferences

Although we may need to emphasize that our work does not focus on long term preference drift on video routing: analyzing preference transferring across tasks or finding labels’ temporal consistency is not our focus.

We would like to confirm with the reviewer what “tasks” in the question refers to video prompt inquiries? The question “Does preferences change across tasks” means preference changes with input prompt? If so, all evaluations use grouped cross-validation over normalized prompt text, so every test prompt is unseen during training. There are no shared prompts or source images between train and test. Test prompts share at most 8.4% of word trigrams with their closest training prompt (median 2%), and none has a near-duplicate in training (max Qwen3-Embedding cosine 0.91, median 0.58). Attribute composition also varies across folds by up to 18pp (e.g., human-centric prompts). Yet we can still see the personalization works in those unseen tasks as Fig. 3-B and Fig 14 shows. As for temporal consistency, we collected our labels across two months, and we simply order our annotations by time and train partially on the consecutively chronological data, and compare the results trained on the earliest and the most recent data of the same amount. For each expert, the test set is their latest 30% of prompt groups. The models train on either their earliest or their most recent n% of the earlier labels. The validation set (for early stopping) is the same for both, and results are averaged over 2 seeds.

The 25% gap reflects generators added during annotation: 43% of the generators in the test pools never appear in the earliest 25% window, versus 26% in the most recent one. At 50% and above the unseen shares are similar (19–23% at 50%, 12–15% at 75%), and the gap disappears. Scaling from earliest to latest: accuracy rises from 32.5 to about 37 between 25% and 50% of the data, then levels off. No gain over the population model on future labels: TasteRoute trained on all earlier labels is −2.0 versus a population ranking model built from the other 8 experts (p=0.46). This is consistent with the 5-fold tie we found earlier. Cheap per-expert model (User-BT): it leans toward recent labels (+1.4 to +3.1, significant only at 75%). It goes through the same pool-turnover effect, so we wouldn't present it as drift. Based on the results, we believe we still can only claim stability within an under-2-month collection window.

Training windowTasteRoute, earliestTasteRoute, most recentΔ (most recent − earliest)
25%32.537.7 +5.2 [+0.4, +10.2] p=0.04
50% (pre-declared main comparison)36.838.4 +1.7 [−3.9, +7.1] p=0.55
75%36.734.0 −2.7 [−6.7, +1.2] p=0.19
100%37.6——
AuthorsOpenReview ↗

Addressing the 2 Questions

We sincerely thank the reviewer for the thorough and constructive feedbacks. And please refer to the revised paper submission for any updated Figures or digtis.

Could the authors further justify the practical necessity of personalized routing, particularly given that individual preferences may change across tasks and over time?

As we mentioned in Weakness3, we may need to emphasize that our work does not focus on long term preference drift on video routing : analyzing preference transferring across tasks or finding labels’ temporal consistency is not our focus. In the previous Weakness response, yet we still provided a further analysis on the difference between train and test dataset while personalization is still generalizable. And we also provided an extra experiment on chronological sequential data training from the head vs that from the tail of the trainign data sequence, and we can still claim stability within an under-2-month collection window because of insiginificant difference performance on testing dataset. Please refer to more detail in the Weakness part.

Given the modest improvements over simple baselines, could the authors clarify what evidence supports the effectiveness and generalizability of the proposed personalization framework?

To simply put, we found significant improvement of Personalized TasteRoute from same-user kNN on "shared preference" in reviewer kh1f 's weakness 1 rebuttal, but insignificant improvement of Personalized TasteRoute from same-user kNN on "opposite preference", while TasteRoute especially outperform other baselines on "opposite preference" significantly. These results support the effective personalization of Personalized TasteRoute, especially on opposite preferences training, and maintaining competitive routing capability on shared preferences. And since we are testing on totall fresh and held-out testing datset from training dataset, this also shows the generalizability of TasteRoute to show effectiveness on unseen prompts.

Thank you for your review and the time you have taken to evaluate our paper. We have submitted a rebuttal addressing your comments, and we would be grateful for the opportunity to further clarify any remaining concerns or questions you may have.

We kindly invite you to engage in a discussion, as your insights would be extremely valuable in improving the clarity and quality of our work. We also appreciate the Area Chair's assistance in initiating this discussion.

We thank the reviewer again for their time and consideration. As the rebuttal period is approaching its end, we wanted to kindly follow up and ask whether our response has addressed your concerns or if there are any remaining questions we could clarify.

We would greatly appreciate your feedback and consideration of our additional results.

Reviewer_pqazOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether, given a prompt (and optionally a reference image), one can pick the right video generator for a particular user without going over a per-video budget. To study this, the authors build TasteRoute-3K: 394 requests, around 3K videos from 30 generator endpoints, all labeled by 9 annotators. Each annotator first rates whether each video is acceptable, then ranks the videos by personal taste. The most interesting finding comes from this data. Annotators mostly agree on which videos are broken, but once both videos are acceptable, their agreement drops to near chance. The router itself is fairly simple: a frozen Qwen3-VL embedding of the request, learned embeddings for generators and users, and small MLP heads trained with a pairwise Bradley–Terry loss plus an auxiliary quality loss. It performs slightly better than kNN, and the personalized version also ends up choosing somewhat cheaper generators at higher budget caps.

Critical strengths

The analysis of individual vs. consensus preference. The finding that annotators agree on broken videos (83–85%) but are close to chance on acceptable ones (57–62%) is useful and fairly convincing. So is the finding that the consensus pick matches a given person’s favorite only 34–55% of the time. Together they give a concrete reason why one leaderboard ranking cannot serve every user, which is the paper’s main motivation. This is the part I found most valuable.

The released dataset and its annotation design. Acceptability is rated separately from personal taste, each annotator’s own ranking is kept rather than averaged away, and there are questionnaire and onboarding signals. Few public video preference datasets keep this kind of information. It could help future work on personalized evaluation.

Critical weaknesses
  1. Limited originality; key prior work is omitted. The router uses a frozen query embedding, a learned model embedding and a Bradley–Terry loss, which is essentially the matrix-factorization router of RouteLLM (Ong et al., ICLR 2025). The user-conditioning design (a per-user embedding table, with a fallback to a generalized prediction) closely follows personalized preference models for image generation such as PAM∃LA (arXiv:2604.07427), PIPE (ICML 2026) and PPD (CVPR 2025). None of these are cited. The claim that video routing is “unexplored” also overlooks cost-aware text-to-image routing (CATImage, arXiv:2506.14753; RouteT2I, arXiv:2411.13787) and a publicly released budget-capped video router (Runway Media Router, July 2026). Once this prior work is taken into account, the contribution amounts largely to applying a known recipe to a new modality.

  2. Personalization rests on nine annotators, and prompt conditioning shows no clear benefit. One model is trained per annotator, and there is no evidence that the approach generalizes beyond these nine individuals. Moreover, Hierarchical User-BT appears to be prompt-independent, yet TasteRoute-Personal is reported only as “competitive” with it (Section 8.2). This suggests that conditioning on the prompt adds little beyond a per-user prior over generators, which undermines the paper’s central claim.

  3. Marginal gains without statistical support. The improvement over kNN is small (42.0% vs. 39.7% Top-1). Results are averaged over only three seeds, with no confidence intervals or significance tests, and the per-annotator test sets are small. Key personalization differences (55.2% vs. 52.4% and 52.2%) may lie within noise. In addition, the Introduction and Conclusion claim superiority over all personalized baselines at every budget cap, which is inconsistent with Section 8.2.

  4. The labels confound generator quality with sampling noise, and the cost savings are not a methodological result. Each generator contributes a single video per prompt, so the labels mix generator quality with generation randomness. This matters most in exactly the regime where annotator agreement is already near chance. It is also unclear whether each annotator saw every generator in a pool. Finally, the router is trained without cost supervision, so the reported cost savings reflect these annotators’ preference for cheaper models rather than any property of the method.

Questions for authors
  1. Comparison with RouteLLM. How does TasteRoute differ, conceptually and empirically, from a RouteLLM-style matrix-factorization router with an added user embedding? Could you include this as a baseline?

  2. Hierarchical User-BT. Is this baseline prompt-independent? If so, what is the gap between it and TasteRoute-Personal at each budget cap, and is that gap statistically significant?

  3. Sampling noise. Several models appear under two endpoint IDs (e.g., Veo 3.1 Lite and Veo 3.1 Fast). How often do annotators prefer one of these duplicate endpoints over the other? This would give an estimate of how much of the routing target is sampling noise rather than generator quality.

  4. Coverage of personal ground truth. Did each annotator see every generator in a pool? If not, how is an annotator’s top choice defined for generators they never judged?

  5. Candidate set. The router only chooses among generators labeled for each pool. How would performance change if it had to choose among all 30 endpoints, as it would in deployment?

6 comments
AuthorsOpenReview ↗

Addressing Weakness Points

We sincerely thank the reviewer for the thorough and constructive comments. And please refer to the revised paper submission for any updated Figures or digtis that were asked during the rebuttal session.

Weakness 1: Limited originality

We thank the reviewer for these references and will discuss all of them in Section 8. Our novelty is not in any single component (e.g., the BT loss) but in the problem setting and the data that make it studiable:

  1. Task: personalized, pre-generation hard routing for video under a per-video budget. One generator must be chosen before any video exists. Top video generators are mostly closed APIs, ruling out post-generation scoring, weight adaptation, or trial and error (a clip takes minutes and up to $1.45).
  2. Data: TasteRoute-3k, to our knowledge the first video preference dataset separating rubric-based acceptability (objective defects) from each professional annotator's own ranking (subjective taste). It shows consensus matches an individual's favorite only 34–55% of the time on acceptable videos.
  3. Finding: personalization matters where users disagree. On opposite-preference pairs, TasteRoute significantly outperforms pooled routers, including the RouteLLM-style MF baseline we added (updated Fig. 14).

Each cited work covers only part of this setting:

WorkModalityRoutes before generationMultiple (closed) generatorsPer-userIndividual human labels
RouteLLMText✓✓✗✗
RouteT2I / CATImageImage✓Partial✗✗
PAM∃LAImage✗ (reward model)✗✓✓
PIPE / PPDImage✗ (adapts one open model)✗✓— / ✓
Runway Media RouterVideo✓✓✗ (undisclosed)✗ (undisclosed)
TasteRoute (ours)Video✓✓✓✓
  • PAM∃LA trains a post-generation personalized reward model (used for prompt optimization); it scores existing images and does not choose among generators.
  • PIPE and PPD adapt a single open-weight diffusion model per user (IP-Adapter / cross-attention embeddings with Diffusion-DPO). Both need weight access, so they do not apply to closed video APIs chosen before generation.
  • RouteT2I and CATImage route image generation without personalization: edge-vs-cloud switching and in-pipeline routing with automated metrics, respectively.
  • Runway Media Router (July 2026) is concurrent with our submission and is a closed commercial feature with no published method, dataset, or personalization; its release reinforces the demand for an open benchmark like TasteRoute-3k.

Weakness 2 Generalization across 9 annotators.

Please refer to the weakness 2 "Narrow validation" we address in response to reviewer kh1f's review.

Weakness 3 Marginal gains without statistics support:

We added confidence interval and more baselines in Fig. 3 and Fig. 14 in the rebuttal version of the paper, please refer to the updated figures there. And after we added CI, we found that the difference between Personal TasteRoute and same-user kNN is significant (p value=0.05) at $1.5. And we further found there significant difference on shraed preferences only dataset in reviewer kh1f's Weakness 1 experiment. Considering the opposite preference experiment (Fig.14), Personalize TasteRoute outperforms all of the baselines significantly except kNN. While kNN falls in an intermediate state, yet kNN does not significantly differ from the general routing ceiling as well. The overall results suggest that kNN does not fully learn shared preference as well as TasteRoute does, while kNN still manage to learn something on opposite preference, but not significant enough, and TasteRoute learn bettern on opposite preference than most of the baselines do.

Weakness 4-1 Sampling noise:

Due to the limited resource of labeling efforts and budgets we have, we carefully prioritize generation samples of similar ranked generators for annotations to reduce wasteful annotations on too-easy pairs made by possible randomness, and spent more than 10,000 USD to generate videos, for the largest amount allow us to achieve diverse tastes from annotations. Please see more detail about what we have addrees for reviewer wqHM 's Weakness 4 data construction bias for more detail about the context of the anchor generators sampling strategy.

Weakness 4-2 cost supervision:

We have another training with cost experiment in section 8.1 (its numbers are also updated due to metrics alignment), which reflects the routing based on costs and preferences as well. The TasteRoute does provide settings for cost sensitive training scenario.

AuthorsOpenReview ↗

Addressing the 5 Questions

We sincerely thank the reviewer for the thorough and constructive comments. And please refer to the revised paper submission for any updated Figures or digtis that were asked during the rebuttal session, or other reviewer's related rebuttal.

Comparison with RouteLLM

We added RouteLLM in updateed Figure 3, Figure 11-13, and FIgure 14. Conceptually, RouteLLM is an additive user embedding in a bilinear MF that can only add a fixed per-user offset (a low-rank User-BT). Pooled training pulls users with unusual tastes toward the consensus, and TasteRoute avoids that because it can decide to trains on each user's own data separately or jointly. The disagreement probe (Fig.14) supports this. Emperically, in Figure 14 show that RouteLLM is almost not able to learn opposite preference, while both separately and jointly trained TasteRoute models can outperform RouteLLM on this. And in overall personalized routing, TasteRoute has slightly higher not not significantly NDCG score than RouteLLM over all budgets. This suggest TasteRoute's advantage on learning prefernece disagreement more.

Hierarchical User-BT

Yes, it’s a prompt-independent baseline. It scores generators with a population Bradley–Terry model plus per-user residuals for each modality (T2V/I2V), and takes no prompt input. Its predictions are therefore constant for a given user, modality and generator pair. In the overall personalization routing experiment, TasteRoute has slightly higher but comparable results to User-BT’s at each budget cap. However, where the two methods differ is mostly in handling users who disagree with each other. On pairs where two annotators judged the same videos and preferred opposite ones, TasteRoute-Personal recovers both users' preferences in 23.2% of cases versus 8.4% for User-BT (+14.8 pp; 95% CI [+11.7, +18.9]; [+4.1, +26.2], which exist a significant difference. User-BT gives both users the same ranking in 82.6% of these cases, because its hierarchical prior shrinks per-user residuals toward the population ranking.

Sampling Noise:

Thank you for pointing out the issues, we did not make the tasks description clearly, the redundant model name series “Veo” vary with the generation setting depending on the postfix. Here the *-generate-001 postfix refers to a generation setting that mainly generates videos of length 4 seconds only, and Veo without the postfix refers to a generation setting that generates 6, 8, and 10 seconds videos. And after rechecking, there are no same endpoints or even the same models that share the same prompt group in our labeled dataset, so there will be no chance that annotators prefer one of these duplicate endpoints over the other. And therefore there will be no such sampling noise or bias.

Coverage of personal ground truth:

No. To keep the annotation load manageable, each prompt's videos were split into groups of about five, with "anchor" videos shared between groups. Each annotator ranked one or more of these groups. On average, an annotator judged 6.3 of the 8.5 generators available for a prompt, and 49% of annotator–prompt pairs cover the full pool. As for Personalized TasteRoute to score generators that are never seen by an annotator, the answer depends on how we train the personalized TasteRoute. If we train each single TasteRoute for each annotator, we do not fill in preferences for generators an annotator never judged. But for a jointly trained personalized TasteRoute (short as shared TasteRoute), all annotators’ preferences are trained in the personal head MLP, which enables the model to score a generator for an annotator that has never seen but annotated by others before. We will state this definition explicitly in the revision.

Candidate set:

The router does not need any changes to choose among all endpoints. It learns one shared representation for every generator, trained on all 30 across prompts, so it can score every endpoint for any prompt, including generators that were never labeled for that prompt. We restrict evaluation to the labeled pool (8.5 generators per prompt on average) because that is the only place we have human ground truth. Grading a choice outside the pool would require human labels that do not exist. Although we don’t have the personalized annotations for all generators for each annotator to evaluate the personalized results when applying 30 generator pools, we still update the mean of 9 annotator results of the Shared TasteRoute in Fig 3-A, personalized results in Fig. 3-B, and full NDCG Top-1,Top-3 results in Fig. 11-13 respectively. The results show almost the same NDCG, and slightly improved Top-1, Top-3 scores when the model can predict the 30 endpoints. Second, deployment compares fewer than 30 endpoints, because each task (T2V or I2V) has only its compatible subset.

Thank you for your review and the time you have taken to evaluate our paper. We have submitted a rebuttal addressing your comments, and we would be grateful for the opportunity to further clarify any remaining concerns or questions you may have.

We kindly invite you to engage in a discussion, as your insights would be extremely valuable in improving the clarity and quality of our work. We also appreciate the Area Chair's assistance in initiating this discussion.

Reviewer_kh1fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TasteRoute selects a video generator based on the request, user preferences, and budget. The paper contributes a 3,128-video dataset annotated by nine people. Text-to-video and image-to-video evaluations report general routing comparable to kNN, with modest personalization gains and lower generation costs at higher budget caps.

Critical strengths
  • A dataset that preserves individual preferences: TasteRoute-3k contains 3,128 annotated videos, with individual rankings and separate acceptability judgments from nine annotators. This supports studying personalization without reducing everyone’s preferences to a single consensus.

  • Useful evidence about when personalization matters: The paper documents lower agreement on clean videos and evaluates routing across budgets and feedback amounts against strong retrieval baselines. These analyses clarify both the opportunity for personalization and its current limitations, providing value beyond the proposed router’s modest gains.

Critical weaknesses
  • Limited demonstrated advantage on the central task: General routing performs similarly to kNN, while accuracy on opposing-preference pairs is only 55.2% versus 52.4% for same-user kNN. This leaves limited evidence of a substantial methodological advance in resolving individual preferences.

  • Narrow validation: Results rely on nine annotators, a fixed generator catalog, and one recorded video per request–generator pair. Whether the routing gains and cost savings persist for other users or fresh generation samples remains unestablished, limiting practical generalizability.

Questions for authors
  • What are the paired confidence intervals for improvements over same-user kNN, accounting for dependence within annotators and prompt pools? This would establish whether the modest gains are statistically reliable.

  • Do the routing gains persist on fresh generations for held-out requests? This requires a new experiment and would test whether the router predicts generator preferences beyond the particular recorded samples.

4 comments
AuthorsOpenReview ↗

Addressing the 2 Weakness Points

We sincerely thank the reviewer for the thorough and constructive comments. And please refer to the revised paper submission for any updated Figures.

Llimited advance in resolving individual preferences:

Actually in the main results (Updated Fig 3), we found Personalized TasteRoute is significantly better than same-user kNN (+1.47pp, p value=0.05) at $1.5 cap, indicating that Personalized TasteRoute can still better capture overall preferences than kNN does. The opposite preference test is a way we want to observe how accurate Personalized TasteRoute is on opposite preferences. The insignificance between 55.2% and 52.4% just indicates that kNN can capture opposite preferences as well but not significantly. To verify our point, we further examined the performance differences of same-user kNN and Personalized TasteRoute on “shared preference” only and found significant difference between them (57.3% vs 63.4%, p value = 0.0003), which shows that kNN is not as good at capturing shared preference as TasteRoute is. We will further rephrase our statement for the updated Fig. 14 to describe this difference to avoid misunderstanding.

Narrow validation:

  1. Relying on a fixed breadth of the Generator Catalog: Our candidate pool consist of 26 distinct video models across 12 providers (Appendix A, Table 1), including 5 open-weight models, 9 commercial marketplace models, and first-party APIs, covering both T2V and I2V across a 24x price spectrum ( 0.01 USD/s to 0.242 USD /s). Not to mention routing works like RouteT2I (Xin et al.) only use 7 image generators in their training dataset. As related work Li et al. mentions vision evaluation works like vHELM only uses 19 generators of VLM. We prioritized video models we would use in the real world and are not optimizing for the goal of being able to route to large sums of video models. Besides the wide capability and price spectrums we observe, we can effortlessly add new generator category to our routing in an online learning scenario, either by human labeling with our target users or pseudo labeling with a learned scorers based on rubric scores, preferences pair and newly generated video of new generators.
  2. Annotator Cohort Selection & Depth over Shallow Crowdsourcing: Our nine annotators were specifically selected to represent primary real-world user groups of video generation platforms: nine professional video creators/editors/professions work in this area. We also undergo rubric calibration on objective technical defects, to avoid calibrating their aesthetic preferences so their authentic personal tastes were preserved and high quality. We can also add new annotators online with initial questionnaire embedding as a cold start.
  3. As for persistence for fresh generations, we would like to emphasize that our testing prompts are already held-out prompts that never exist in the training prompt group. The testing we did is already on fresh generations that are never trained or seen before. We did a cost calculation, the total cost including human annotations (hiring of video editing experts) has well exceeded $10,000 USD. We base on this cost and further found TasteRoute to save more costs with a cost-performance trade-off experiment on the same “held-out fresh testing generations” as well in Table 3. Lastly, we would like to clarify that in real world use, we do not expect commercial products to follow the exact process to collect their user preferences, rather our goal is to collect a clean dataset as the first step to study video routing tasks.
AuthorsOpenReview ↗

Addressing the 2 Questions

We sincerely thank the reviewer for the thorough and constructive comments. And also please refer to the revised paper submission for any updated Figures. We'll present newly added tables or experiments here instead.

Q1 Confidence intervals for improvements over same-user kNN:

The updated Figure 3-B shows the confidence interval, and as we rebuttaled in Weakness 1, there is a significant difference of performance between tasteRoute and same-user kNN when considering whether capturing only shared preferences. As for other non-significant baselines in Fig 3-B, we also want to emphasize that TasteRoute outperforms all of them when considering only opposite preferences. We hope the results from the updated Fig 3B, the updated Fig 14 and extra shared preference tests could bring more reliable results between TasteRoute and other baselines now.

Q2 Persistence on fresh generations for held-out requests:

As we mentioned in Weakness 2, our testing data is already held-out prompts from the training prompts. In addition, we also implement the scaling of training data across time in reviewer oyB2’s question. The results show that even if we trained routers on the earliest 50% and the latest 50% annotation data across two-month labeling duration, and tested on the extra latest 30% testing data, there is still no significant difference between their performance differences on the freshest and latest 30% testing data, showing persistence on fresh generation in some way.

scaling of data across time:

We collected our labels across two months, and we simply order our annotations by time and train partially on the consecutively chronological data, and compare the results trained on the earliest and the most recent data of the same amount. For each expert, the test set is their latest 30% of prompt groups. The models train on either their earliest or their most recent n% of the earlier labels. The validation set (for early stopping) is the same for both, and results are averaged over 2 seeds. The 25% gap reflects generators added during annotation: 43% of the generators in the test pools never appear in the earliest 25% window, versus 26% in the most recent one. At 50% and above the unseen shares are similar (19–23% at 50%, 12–15% at 75%), and the gap disappears. Scaling from earliest to latest: accuracy rises from 32.5 to about 37 between 25% and 50% of the data, then levels off. No gain over the population model on future labels: TasteRoute trained on all earlier labels is −2.0 versus a population ranking model built from the other 8 experts (p=0.46). This is consistent with the 5-fold tie we found earlier. Cheap per-expert model (User-BT): it leans toward recent labels (+1.4 to +3.1, significant only at 75%). It goes through the same pool-turnover effect, so we wouldn't present it as drift. Based on the results, we believe we still can only claim stability within an under-2-month collection window.

Training windowTasteRoute, earliestTasteRoute, most recentΔ (most recent − earliest)
25%32.537.7 +5.2 [+0.4, +10.2] p=0.04
50% (pre-declared main comparison)36.838.4 +1.7 [−3.9, +7.1] p=0.55
75%36.734.0 −2.7 [−6.7, +1.2] p=0.19
100%37.6——

Thank you for your review and the time you have taken to evaluate our paper. We have submitted a rebuttal addressing your comments, and we would be grateful for the opportunity to further clarify any remaining concerns or questions you may have.

We kindly invite you to engage in a discussion, as your insights would be extremely valuable in improving the clarity and quality of our work. We also appreciate the Area Chair's assistance in initiating this discussion.

We thank the reviewer again for their time and consideration. As the rebuttal period is approaching its end, we wanted to kindly follow up and ask whether our response has addressed your concerns or if there are any remaining questions we could clarify.

We would greatly appreciate your feedback and consideration of our additional results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

Dear Area Chair and Reviewers,

We thank all reviewers for their constructive feedback and the AC for facilitating the discussion. Below we summarize our contributions, how we addressed the main concerns, and the discussion status.

Novelty and contributions

  1. TasteRoute-3k. To our knowledge, the first video generation dataset that separates rubric-based acceptability (objective defects) from each professional annotator's personal ranking (subjective taste): 3,128 videos from 26 generators (12 providers, T2V and I2V, 24× price range), annotated by 9 professional video creators. All four reviewers recognized its value.
  2. Evidence that one leaderboard cannot serve every user. Annotators agree on broken videos (83–85%) but are near chance on acceptable ones (57–62%), and the consensus pick matches an individual's favorite only 34–55% of the time. Reviewers kh1f, pqaz and oyB2 found this analysis useful.
  3. Personalized, pre-generation (hard) routing. Because video generation is slow and costly, TasteRoute picks a generator for a specific user before generation, given only the request and a budget. On held-out prompts, it is the only method that is significantly better than same-user kNN on shared-preference pairs and significantly better than all other baselines on opposite-preference pairs.

Strengths noted by reviewers

  • Dataset preserving individual tastes (all four).
  • Individual-vs-consensus analysis (kh1f, pqaz, oyB2).
  • When personalization helps, across budgets, feedback and disagreement (kh1f, oyB2).
  • Practical value of cost-aware video routing (wqHM).

Main concerns and how we addressed them

  1. Statistical support (all reviewers). We added 95% CIs and significance tests (Fig. 3, Figs. 11–14) over 1,523 test pools / 15,690 pairs under grouped 5-fold CV. TasteRoute beats same-user kNN at the $1.5 cap (+1.47 pp, p = 0.05, marginal) and on shared-preference pairs (63.4% vs. 57.3%, p = 0.0003), and beats all other baselines on opposite-preference pairs (e.g., +14.8 pp over User-BT, 95% CI [+11.7, +18.9]); kNN is not significantly different there. We also replaced "outperforms" with "comparable/competitive" where appropriate. → Takeaway: the gains are statistically supported where personalization matters most, and our claims are now calibrated to the evidence.

  2. Originality (pqaz). We added RouteLLM as a baseline; it barely recovers opposite preferences. Related work now covers RouteT2I, CATImage, PAM∃LA, PIPE and PPD, and clarifies that our setting routes before generation, across mostly closed-source video APIs, under a per-video budget. → Takeaway: TasteRoute is not RouteLLM plus a user embedding; it resolves the user disagreements that a pooled MF router averages away.

  3. Value of personalization (oyB2, pqaz, wqHM). We added a jointly trained Shared TasteRoute. Against the prompt-independent User-BT, TasteRoute is comparable on overall NDCG but recovers both users' preferences on disagreement pairs far more often (23.2% vs. 8.4%). → Takeaway: prompt-conditioned personalization adds value precisely where users disagree, which a per-user prior cannot capture.

  4. Generalization (kh1f, oyB2). All test prompts are held out (≤8.4% trigram overlap, no near-duplicates). A new chronological experiment shows no significant earliest-vs-latest difference at the pre-declared 50% window (p = 0.55), supporting stability within our two-month collection window. → Takeaway: personalization generalizes to unseen prompts and remains stable over the collection period.

  5. Data construction (pqaz, wqHM). Similar-rank sampling avoids spending the budget on easy pairs (acceptance ranges from 87% to 9%). Veo variants are different duration configs that never share a pool; the FLUX image and video models are distinct. → Takeaway: annotation effort is concentrated on hard, taste-driven pairs, which is where personalization signal lies.

  6. Cost and metrics (wqHM). We specified per-video cost, confirmed that pools share duration and aspect ratio and pool selection is uncorrelated with price (Spearman −0.05), and aligned the NDCG/Top-1 definitions across all figures and Table 3. → Takeaway: cost and metric comparisons are now consistent and reproducible.

Discussion status

Reviewer pqaz (confidence 4) stated that our response "has addressed all of my concerns" and raised their score, and Reviewer wqHM stated that their "main concerns have been adequately addressed." Reviewers kh1f and oyB2 have not yet responded; their main concerns (statistical support, modest gains, generalization) overlap with those that pqaz and wqHM considered resolved, so we believe the same revisions address them. We welcome any further questions.

We believe TasteRoute-3k and our analysis are a solid first step toward personalized video routing. Thank you again.

Sincerely, The Authors of Paper 39846

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.