acceptodds
Under review as a conference paper at ICLR 2027

WanPEBench: Towards Unified Evaluation of Prompt Enhancers for Video Generation

Abstract

Modern video generators can realize increasingly complex visual narratives, positioning the prompt enhancer (PE) as a critical bridge from concise user instructions and multimodal references to structured cinematic plans. However, existing PE evaluation relies on rendered videos, imposing substantial computational and human costs, slowing PE training and iteration, and conflating PE quality with downstream generator behavior. To address this gap, we introduce WanPEBench, the first unified benchmark for direct PE evaluation across text-to-video, image-to-video, and reference-to-video prompt enhancement. It comprises 1,100 expert-verified cases and 1,005 visual assets, spanning 35 fine-grained tasks with diverse temporal, cinematic, audiovisual, and multi-reference requirements. In addition, we develop WanPEBench evaluation, an evidence-grounded framework that combines modality-aware fact extraction with rubric-based assessment across 24 criteria. Our systematic evaluation of representative open- and closed-source PE methods reveals an emerging shift from fine-grained descriptive expansion toward intent-preserving cinematic planning, while the caption-reconstruction and forward-refinement methods show complementary strengths in cinematic coverage and semantic fidelity or internal coherence, respectively. Human validation shows that WanPEBench scores align closely with expert judgments of enhanced prompts and downstream videos from Wan3.0 and MiniMax-H3, indicating that prompt-level evaluation reliably reflects downstream utility.

Primary area:
datasets and benchmarks
Keywords:
Prompt Enhancer, Video Generation, Evidence-Grounded Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vT9rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WanPEBench, a benchmark for evaluating video prompt enhancers directly from their outputs without generating videos for every evaluation. It comprises 1,100 expert-verified cases across text-to-video, image-to-video, and reference-to-video enhancement, with 1,005 visual assets and 35 task types. The evaluation framework combines cached, modality-aware evidence extraction with 24 criteria organized into Semantic Fidelity, Internal Consistency, and Directorial Creation. Experiments compare ten enhancers on text-only inputs and smaller subsets on visually conditioned inputs, and analyze temporal complexity, reference complexity, and output length. A human study compares automatic system scores with expert rankings of enhanced prompts and downstream videos generated by Wan3.0 and MiniMax-H3. The paper reports high rank correlations and argues that prompt-level evaluation provides an efficient indicator of downstream utility.

Critical strengths
  1. The benchmark brings text-only, first-frame-conditioned, and multi-reference prompt enhancement into one evaluation framework, covering temporal structure, audio requirements, and heterogeneous reference roles. This is a useful scope for studying enhancer behavior beyond conventional text rewriting. Direct prompt evaluation also offers a plausible way to diagnose omissions and contradictions without requiring video generation for every evaluation.
  2. Separating fidelity, consistency, and directorial planning helps distinguish different failure modes, while sharing extracted reference evidence across methods is a sensible consistency measure.
Critical weaknesses
  1. The practical downstream significance of the evaluation scores is insufficiently established. Prompt enhancement is intended to improve generated videos, but Table 5 reports system-level rank correlations rather than the magnitude of improvement over original prompts. Although the human study includes a no-enhancement baseline, it does not report downstream preference win/tie/loss rates or effect sizes with uncertainty. Appendix D.1 computes correlations over 11, 4, and 3 systems for T2V, I2V, and R2V, respectively, including that baseline; the perfect I2V/R2V correlations consequently have limited resolution. Such correlations do not establish how much video quality improves or whether the metric can identify beneficial, ineffective, or harmful enhancements on individual inputs. The evidence supports aggregate ordering, but not yet reliable feedback for selecting or refining particular prompts. It therefore remains unclear whether the measured score differences correspond to practically meaningful video improvements rather than primarily compliance with a detailed prompt-writing rubric. This gap limits the paper's central claim of reliable downstream utility.

  2. The core scoring protocol is not sufficiently reproducible or independently validated. Appendix B provides criterion descriptions and output schemas, but not a reproducible mapping from minor/major issues to the 0-100 dimension scores, complete evaluator prompts, or calibrated severity rules. The treatment of non-applicable criteria also requires clarification. Appendix B.1 defines pass rates over applicable cases, yet the T2V DLI and OTI percentages in Table 2 are consistent with a denominator of all 500 cases despite supplied examples lacking dialogue or on-screen-text requirements. This is a reporting ambiguity, not proof of incorrect computation, but counting non-applicable cases as passes would inflate the apparent success rates on those capabilities. Furthermore, the underspecified Direct VLM Scoring baseline does not isolate the benefits of evidence extraction and rubric decomposition. These gaps concern the benchmark's central measurement procedure and prevent a clear assessment of what its scores measure and how robustly they can be reproduced.

Questions for authors
  1. For each generator and task, how much do videos produced from enhanced prompts improve over those produced from the original prompts? Please report preference win/tie/loss rates or other effect-size estimates with confidence intervals from the existing human study, and identify settings where enhancement offers little benefit or degrades results. This would clarify the practical significance beyond rank correlation.

  2. For matched inputs, do the proposed scores correctly predict expert video preferences between alternative enhancements and between enhanced and original prompts? Can the existing annotations support this analysis, including cases where the preferred choice is no enhancement? Please report uncertainty and specify unique-input counts, repeated judgments, and the treatment of "both good" and "both bad." This tests the claimed usefulness for prompt selection rather than only system ranking.

  3. What are the exact severity-to-score rule and the numerator/denominator for each criterion? Please explain how not_present and failed evaluations are handled, reconcile the DLI/OTI denominator ambiguity, and provide the complete evaluator prompts and scoring implementation. If dimension scores are generated directly by the judge rather than computed, what rules calibrate them and ensure consistency with the criterion-level judgments?

  4. What evidence separates the benefits of factual grounding and rubric decomposition from a preference for particular writing styles? A same-judge, matched-input ablation would help attribute the gains over Direct VLM Scoring, while meaning-preserving formatting or verbosity changes would test score robustness. Appendix C.2 compares different cases grouped by output length and therefore does not isolate this latter effect from input difficulty.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vT9r (Part 1/4)

Dear Reviewer vT9r, we sincerely thank you for recognizing the value of our unified evaluation scope, direct prompt-level diagnosis, and evidence-grounded decomposition of prompt-enhancement quality. We appreciate your constructive comments and questions, and we provide our detailed responses below.


W1&Q1: Practical downstream improvements over original prompts

A1: Thank you for this important suggestion. System-level rank correlations alone may not quantify the magnitude or direction of downstream quality changes relative to the original prompts. We therefore report the human preference outcomes between videos generated from each enhanced prompt and its corresponding original prompt.

For each system, we evaluate 50 cases and report the enhanced-win / tie / original-win rates. We additionally report case-level preference consistency, which measures whether the prompt-level score ordering agrees with downstream human video preference. A comparison is inconsistent when humans explicitly prefer the video generated from the lower-scoring prompt. The reported uncertainty is the 95% Wilson score confidence interval [1] for this consistency estimate. The results for the three tasks are as follows.

System pair (T2V)Enhanced win / tie / original winPreference consistency95% Wilson CI
WanPE-397B vs. original prompt62% / 24% / 14%82.0%[69.2%, 90.2%]
WanPE-35B vs. original prompt50% / 30% / 20%88.0%[76.2%, 94.4%]
H3-Context-IR vs. original prompt48% / 24% / 28%92.0%[81.2%, 96.8%]
LTX-2.5-PE vs. original prompt38% / 34% / 28%84.0%[71.5%, 91.7%]
SCMaPR vs. original prompt36% / 40% / 24%92.0%[81.2%, 96.8%]
LingBot-Video-PE vs. original prompt34% / 36% / 30%82.0%[69.2%, 90.2%]
VPO vs. original prompt20% / 54% / 26%78.0%[64.8%, 87.2%]
Prompt-A-Video-Cog vs. original prompt22% / 48% / 30%82.0%[69.2%, 90.2%]
Prompt-A-Video-OS vs. original prompt28% / 42% / 30%80.0%[67.0%, 88.8%]
RAPO vs. original prompt8% / 40% / 52%90.0%[78.6%, 95.7%]
Overall34.6% / 37.2% / 28.2%85.0%–
System pair (I2V)Enhanced win / tie / original winPreference consistency95% Wilson CI
H3-Context-IR vs. original prompt70% / 20% / 10%90.0%[78.6%, 95.7%]
LTX-2.5-PE vs. original prompt56% / 28% / 16%84.0%[71.5%, 91.7%]
LingBot-Video-PE vs. original prompt50% / 32% / 18%82.0%[69.2%, 90.2%]
Overall58.7% / 26.7% / 14.7%85.3%–
System pair (R2V)Enhanced win / tie / original winPreference consistency95% Wilson CI
H3-Context-IR vs. original prompt74% / 16% / 10%90.0%[78.6%, 95.7%]
LTX-2.5-PE vs. original prompt54% / 30% / 16%84.0%[71.5%, 91.7%]
Overall64.0% / 23.0% / 13.0%87.0%–

These results reveal substantial heterogeneity hidden by system-level rank correlations. Every I2V and R2V enhancer wins more often than it loses. In T2V, WanPE-397B, WanPE-35B, and H3-Context-IR provide clear benefits, whereas LingBot-Video-PE and Prompt-A-Video-OS are close to parity. VPO and Prompt-A-Video-Cog produce more original-prompt wins than enhanced-prompt wins, while RAPO substantially degrades downstream quality. These weak or negative results mainly occur for prompt enhancers originally designed for earlier video generators, whose limited context capacity and instruction-following ability encouraged descriptive rewriting that may compress or omit details when the original prompt is already highly informative. When applied to modern generators, these legacy enhancers may compress or discard important details from already informative prompts, thereby yielding little benefit or even degrading downstream quality.

Preference consistency remains high across both beneficial and harmful settings, reaching 85.0% for T2V, 85.3% for I2V, and 87.0% for R2V. We have added these win/tie/loss results and uncertainty estimates to the revised paper (Appendix D.3) to clarify the practical benefits and limitations of prompt enhancement beyond rank correlation.

[1] Wilson E B. Probable inference, the law of succession, and statistical inference[J]. Journal of the American Statistical Association, 1927, 22(158): 209-212.

AuthorsOpenReview ↗

Response to Reviewer vT9r (Part 2/4)

W1&Q2: Matched-input prompt-selection reliability and human-study details

A2: We sincerely appreciate this constructive suggestion. The enhanced-versus-original results in A1 show that the proposed scores predict expert video preferences on matched inputs, rather than merely reflecting system-level rankings. Using I2V and R2V as examples, for each matched input, we compare the ordering of two candidate prompts induced by the proposed scores with experts’ pairwise preference between their generated videos. This comparison covers both pairs of alternative enhancements and enhanced-versus-original pairs. The results are shown below.

System pair (I2V)Preference consistency95% Wilson CI
H3-Context-IR vs. LTX-2.5-PE82.0%[69.2%, 90.2%]
H3-Context-IR vs. LingBot-Video-PE86.0%[73.8%, 93.0%]
H3-Context-IR vs. original prompt90.0%[78.6%, 95.7%]
LTX-2.5-PE vs. LingBot-Video-PE76.0%[62.6%, 85.7%]
LTX-2.5-PE vs. original prompt84.0%[71.5%, 91.7%]
LingBot-Video-PE vs. original prompt82.0%[69.2%, 90.2%]
Overall83.3%–
System pair (R2V)Preference consistency95% Wilson CI
H3-Context-IR vs. LTX-2.5-PE82.0%[69.2%, 90.2%]
H3-Context-IR vs. original prompt90.0%[78.6%, 95.7%]
LTX-2.5-PE vs. original prompt84.0%[71.5%, 91.7%]
Overall85.3%–

After removing the original-prompt baseline, preference consistency remains 81.3% for I2V and 82.0% for R2V. Thus, the case-level agreement is not driven solely by an obviously weaker baseline and also holds when selecting between alternative enhancements.

The existing annotations also retain cases in which experts prefer no enhancement. Importantly, preference consistency remains high even for ineffective or harmful enhancements. For example, experts prefer the original-prompt video to the RAPO-enhanced video in 52% of T2V cases, while the score ordering has 90.0% consistency with these preferences. This indicates that the evaluation does not merely reward more detailed prompt writing and can assign lower scores to harmful enhancements in agreement with downstream human preferences.

We have expanded Appendix D with the complete human-study protocol and agreement statistics.

Exact Study Counts. We randomly sample 50 cases per task and evaluate every system pair, yielding 2,750 T2V, 300 I2V, and 150 R2V comparisons. Each comparison is independently evaluated by three annotators.

Inter-rater agreement and exclusions. After accounting for the randomized left/right presentation, all three annotators agree on 85.0% of comparisons. We exclude 3 of 60 annotators due to low agreement or systematic positional bias and reassign their comparisons to maintain three valid independent judgments per comparison.

Tie handling in Bradley–Terry Fitting. We treat both-good and both-bad outcomes as ties. A decisive outcome contributes one win to the preferred system, whereas a tie contributes 0.5 wins to each system. We fit

$ p_{kj}=\sigma(b_k-b_j), \qquad \sum_k b_k=0,\qquad BT_k=100\sigma(b_k). $

Overall, the existing matched-input annotations show that the proposed scores align with expert preferences for both alternative-enhancement and enhanced-versus-original comparisons, including cases favoring no enhancement, supporting their case-level utility for prompt selection beyond system ranking. We have added the above experimental analyses and human-study details to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer vT9r (Part 3/4)

W2&Q3: Complete evaluator prompts, scoring implementation and DLI/OTI denominator ambiguity

A3: Thank you for identifying this lack of clarity. Dimension scores are not freely assigned by the evaluator. The evaluator assigns each criterion a status and severity label based on supporting evidence. The score is then computed deterministically from these labels, as detailed below.

(1) Deterministic severity-to-score mapping. For each dimension, MM and mm denote the numbers of criteria labeled major and minor. A minor issue affects local details without changing the overall interpretation, whereas a major issue affects core semantics, coherence, or directorial feasibility. Each criterion is counted once at its highest severity. The per case dimension score is computed as

$ S(M,m)=100 \text{ if } M=0,m=0;\quad \max(70,94-5m) \text{ if } M=0,m\geq1;\quad \max(0,75-15M-5m) \text{ if } M\geq1. $

DimensionNo issueIssue statusInapplicable
Semantic Fidelityconsistentlossy or alterednot_present
Internal Consistencyconsistentambiguous or inconsistentnot_present
Directorial CreationcompletedeficientNot used

Lossy/altered, ambiguous/inconsistent, and deficient denote semantic omission/change, uncertainty/contradiction, and incomplete directorial design, respectively. Issues are graded minor or major by impact.

(2) Applicability rules. A Semantic Fidelity criterion applies only when relevant content appears in the source instruction, visual evidence, or enhanced prompt. Otherwise, it is labeled not_present and excluded from MM and mm. Internal Consistency follows the same rule when an omitted element is unnecessary for coherence. All nine Directorial Creation criteria are assessed for every prompt, but not every directorial element is required. An omission is penalized only when the element is necessary to realize the requested video, given its duration, format, and use of reference assets.

For I2V, only certain and instruction-relevant ImageFacts are used. For R2V, only AssetFacts relevant to each instruction-specified asset role are applied. Uncertain, null, or irrelevant facts are excluded.

(3) DLI/OTI denominator ambiguity. We thank the reviewer for identifying this reporting ambiguity and clarify the definition and computation of these percentages below. The DLI and OTI percentages in Table 2 are penalty-based, all-case non-violation rates rather than pass rates conditioned on criterion applicability. For criterion cc, we compute

$ R_{\mathrm{NV}}(c)= \frac{N_{\mathrm{no\ issue}}(c)+N_{\mathrm{not_present}}(c)} {N_{\mathrm{valid}}(c)}, $

where Nno issueN_{\mathrm{no\ issue}} counts applicable cases without a criterion-level violation. A not_present label indicates that the input contains no corresponding requirement to violate. It is counted as a non-violation and contributes no minor or major penalty. Criterion instances labeled minor or major are counted as violations. Accordingly, NvalidN_{\mathrm{valid}} is 500 for T2V and 300 for I2V and R2V. These percentages measure the benchmark-wide absence of criterion violations rather than performance conditioned on cases containing dialogue or on-screen text. Failed or malformed evaluations are retried. If they remain invalid, they are excluded from NvalidN_{\mathrm{valid}} and reported separately.

Following the reviewer’s suggestion, we additionally report DLI and OTI rates using a shared criterion-relevant denominator, defined as the cases in which the original prompt or any evaluated enhanced prompt contains the corresponding content. The results are shown in the three tables below.

Metric (T2V)WanPE-397BWanPE-35BH3-Context-IRSCMaPRLTX-2.5-PELingBot-Video-PEVPOPrompt-A-Video-CogPrompt-A-Video-OSRAPO
DLI85.64%70.30%82.67%50.99%51.49%30.69%41.09%47.03%42.57%32.18%
OTI84.88%91.28%86.63%85.47%74.42%55.23%68.02%74.42%70.35%38.95%
Metric (I2V)H3-Context-IRLTX-2.5-PELingBot-Video-PE
DLI64.13%40.22%25.00%
OTI79.93%70.07%69.01%
Metric (R2V)H3-Context-IRLTX-2.5-PE
DLI66.44%39.60%
OTI55.00%39.17%

Across all three tasks, the shared-denominator results preserve the system rankings in Table 2 while excluding cases that are universally inapplicable to DLI or OTI, confirming that the conclusions are not driven by the original denominator choice. We have added these shared-denominator DLI/OTI results to the revised paper (Appendix B.3).

(4) Complete rubrics and public release. The revised Supplementary Material includes the complete T2V, I2V, and R2V evaluation prompts, ImageFacts and AssetFacts extraction prompts, applicability rules, and JSON schemas.

Reviewer_QC5TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WanPEBench, a benchmark for directly evaluating video prompt enhancers (PE) at the prompt level, without requiring downstream video rendering. It covers three settings — T2V-PE, I2V-PE, and R2V-PE — with 1,100 expert-verified cases and 1,005 visual assets spanning 35 fine-grained task types, 12 languages, and varying temporal/reference complexity. The evaluation framework combines modality-aware fact extraction (GPT-5.4 for first-frame images, Qwen3.5-Omni-Plus for reference assets) with rubric-based assessment across 24 criteria organized into three dimensions: Semantic Fidelity (9), Internal Consistency (6), and Directorial Creation (9). Ten PE systems are evaluated, yielding four findings: a shift toward cinematic planning, complementary strengths between caption-reconstruction and forward-refinement paradigms, degradation under temporal/reference complexity, and Spearman correlations above 0.8 with expert rankings at both prompt and downstream-video levels.

Critical strengths
  1. The core idea — decoupling PE evaluation from downstream rendering — addresses a real and costly bottleneck. PE development currently requires video synthesis for every iteration, which is expensive and conflates PE quality with generator capability. A prompt-level benchmark that correlates with downstream outcomes would meaningfully accelerate PE research. The paper's framing of this problem is convincing, and the reported alignment with downstream video rankings across two different generators (Wan3.0, MiniMax-H3) provides initial support for the premise.

  2. Substantial and carefully curated dataset with broad coverage. The 1,100 cases spanning 35 task types, 12 languages, three conditioning modalities, and varying temporal/reference complexity represent real engineering effort. The four-stage pipeline with model-based verification followed by ten domain experts is reasonably rigorous, and the complexity stratification (event stages, requested shots, reference count/modality) enables the diagnostic analyses in Tables 3–4 that a flat benchmark could not support.

Critical weaknesses
  1. The benchmark's Directorial Creation dimension appears structurally biased toward the authors' own PE paradigm, which undermines its credibility as a neutral evaluation standard. Nine of the 24 criteria reward detailed cinematic specification — precisely what caption-reconstruction training produces. WanPE is trained by reconstructing full video captions from compressed prompts, so it naturally emits storyboard-style output. Unsurprisingly, WanPE-397B/35B lead Directorial Creation (78.97/78.15) while SCMaPR and VPO — which prioritize intent preservation — score 49.18 and 33.07, and RAPO collapses to 0.93. The paper interprets this as "complementary capability profiles," but an equally plausible reading is that the rubric encodes the authors' preferred notion of a good prompt. For a benchmark paper whose value depends on being an impartial yardstick, this coupling between the measurement instrument and one of the measured systems is a serious concern that the paper does not acknowledge or control for.

  2. The claim of generator-independence is conceptually in tension with how PE actually works. The paper positions generator-independent evaluation as a feature, but prompt enhancement is inherently generator-specific: each video model is trained on a particular caption distribution, and the "best" enhanced prompt is the one closest to that model's training distribution. LingBot-Video-PE explicitly maps prompts into model-specific structured captions, and its low scores here (Semantic Fidelity 41.61, Directorial Creation 48.66) may reflect mismatch with WanPEBench's rubric rather than genuine PE deficiency. The downstream validation uses only Wan3.0 and MiniMax-H3 — both of which plausibly share caption conventions with the industry PE systems that score highest. Validation on a generator with a markedly different caption style (e.g., an academic open-source model trained on short captions) would be necessary to support the generality claim.

Questions for authors
  1. How many expert annotators participated in the human alignment study, what was their inter-annotator agreement, and how many items were ranked per correlation computation? A Spearman ρ of 1.000 over ~11 systems warrants clarification on sample size and tie handling.

  2. Was the rubric design finalized before or after evaluating WanPE? Given that WanPE-397B leads Directorial Creation and that this dimension's criteria closely mirror caption-reconstruction output characteristics, can you provide evidence that the rubric was not implicitly tuned toward this paradigm — e.g., by having rubric designers blind to the PE systems, or by reporting results under an alternative weighting of the three dimensions?

  3. Can you validate downstream alignment on a video generator with a substantially different caption distribution from Wan3.0/MiniMax-H3? This would test whether prompt-level scores transfer beyond generators that share conventions with the top-scoring PE systems.

  4. All benchmark prompts are synthesized by Gemini 3.1 Pro rather than collected from real users. How well does this synthetic distribution reflect actual user prompting behavior, and would results change on a held-out set of real user prompts?

5 comments
AuthorsOpenReview ↗

Response to Reviewer QC5T (Part 1/3)

Dear Reviewer QC5T, we sincerely thank you for recognizing that decoupling PE evaluation from downstream rendering addresses a real and costly bottleneck, and for highlighting our substantial, carefully curated dataset with broad coverage. We appreciate your constructive comments and questions, and we provide our detailed responses below.


W1 & Q2: Rubric independence and impartiality

R1: Thank you for raising this concern. We finalized all 24 criteria and rubrics before running inference with or evaluating any PE system, so the rubric designers had no access to the evaluated systems' outputs or scores.

The criteria were independently derived from consultations with 20 general users and short-video creators and cross-checked against the publicly documented video-generation capabilities of Seedance 2.5, Kling 4.0, and Wan3.0. The nine criteria identified by the reviewer were included because camera design, lighting, editing, sound, and related aspects are fundamental to real-world video creation, not because they resemble caption-reconstruction outputs.

Accordingly, DC evaluates the completeness and specificity of a PE's directorial content creation, not its resemblance to any prompt format. Prompt length, cinematic terminology, timestamps, and multi-shot structure are not rewarded by themselves. Appropriate single-take, static-camera, and silent designs remain valid, while irrelevant elaboration is penalized. Semantic Fidelity (SF) and Internal Consistency (IC) separately assess intent preservation and logical coherence.

We further clarify that WanPEBench reports SF, IC, and DC separately without an aggregate score or overall ranking. This dimension-wise evaluation reveals the strengths and weaknesses of different PE systems across diverse content-creation scenarios, providing guidance for the targeted optimization of individual models. In the T2V-PE setting, SCMaPR, VPO, and WanPE-397B lead SF, IC, and DC, respectively, revealing complementary strengths rather than a universal winner. Future PE systems should integrate all three capabilities to produce faithful, coherent, and professionally enriched prompts aligned with user preferences.

AuthorsOpenReview ↗

Response to Reviewer QC5T (Part 2/3)

W2 & Q3: Generator-independent evaluation and downstream transfer across caption distributions

R2: We clarify that the term “generator independence” in the main paper refers to evaluation-stage decoupling, not generator-agnostic prompt enhancement. Each PE is designed for its downstream DiT, but video-level evaluation cannot determine whether a poor result stems from the PE's failure to faithfully interpret and elaborate the user's intent or the DiT's failure to realize the enhanced prompt. WanPEBench independently evaluates PE outputs for accurate attribution.

As modern DiTs increasingly follow rich instructions, PEs play a central role in interpreting underspecified intent and developing audiovisual content. WanPEBench fills this gap as the first unified benchmark for directly evaluating video PEs through Semantic Fidelity (SF), Internal Consistency (IC), and Directorial Creation (DC), measuring intent preservation, logical coherence, and the completeness of user-aligned content creation.

For downstream video generation, PE outputs should match the target generator's training-caption distribution, and the generator's native PE is typically best aligned with its input conventions. Their output conventions differ: LingBot-Video-PE uses structured JSON with camera and timestamped-action fields, H3-Context-IR uses tagged "integrated_multimodal_description" prose with separate audio fields, and WanPE uses a global description followed by timestamped shots.

However, WanPEBench evaluates criterion-relevant content rather than serialization. To test the reviewer's concern, we use GPT-5.4 to convert every LingBot-Video-PE output from structured JSON into an equivalent descriptive natural-language prompt while preserving its semantics, entities, attributes, temporal information, and camera specifications, and then reevaluate it using the same protocol.

TaskPrompt representationSFICDC
T2VOriginal LingBot JSON output41.6190.9648.66
T2VRefined natural-language output43.4888.5046.71
T2VChange (refined minus original)+1.87-2.46-1.95
I2VOriginal LingBot JSON output45.7091.1437.47
I2VRefined natural-language output43.2389.1437.07
I2VChange (refined minus original)-2.47-2.00-0.40

All absolute changes are below 3 points, with a mean of 1.86. Because the evaluator extracts criterion-level evidence under fixed rubrics and computes dimension scores from those labels, these results show robustness to surface formatting and no systematic preference for JSON or prose. Thus, LingBot-Video-PE's lower scores reflect its criterion-level performance in WanPEBench rather than a mismatch between WanPEBench's rubric and its structured JSON format.

Moreover, although surface format does not materially affect WanPEBench scoring, it may affect downstream video generation. The human-alignment experiments reported in the paper already control for this factor. Before video generation, GPT-5.4 converts each PE output into the target DiT's expected format. Thus, the reported human comparisons assess content differences identified by WanPEBench rather than accidental compatibility with the target generator's input format.

Finally, following the reviewer's suggestion, we validate on the open-source LTX-2.5 generator, whose caption distribution and input conventions differ substantially from those of Wan3.0 and MiniMax-H3. Spearman correlations between WanPEBench and downstream human rankings reach 0.836, 0.845, and 0.918 for SF, IC, and DC on T2V, respectively, and 1.0 for all three dimensions on I2V (LTX-2.5 does not support our multi-reference R2V setting), extending prompt-level alignment beyond generators that share the conventions of the highest-scoring industrial PEs.

These results show that WanPEBench evaluates PE-intrinsic quality without favoring a specific prompt format, while its rankings remain predictive across generators with different caption distributions. We have incorporated these additional experiments and analyses into the revised paper.

AuthorsOpenReview ↗

Response to Reviewer QC5T (Part 3/3)

Q1: Human alignment study details

A1: Thank you for requesting these details. We first clarify that Table 5 does not report ρ=1.000\rho=1.000 over approximately 11 systems. For T2V, correlations across 11 systems range from 0.818 to 0.955 for enhanced-prompt alignment and from 0.809 to 0.945 for downstream-video alignment. Perfect correlations occur only for I2V and R2V, whose rankings include 4 and 3 systems, respectively, including the original-prompt baseline. These smaller pools reflect the nascent state of I2V-PE and R2V-PE research and, to our knowledge, include all systems available at the time.

Study size. We randomly sample 50 cases per task and compare every system pair within each case, yielding 2,750 T2V, 300 I2V, and 150 R2V comparisons. Three annotators independently evaluate each comparison. Each correlation in Table 5 compares system rankings for one task and dimension across 11 T2V, 4 I2V, or 3 R2V systems.

Inter-rater agreement. After correcting for randomized left/right presentation, all three annotators agree on 85.0% of comparisons. We exclude 3 of 60 annotators for low agreement or systematic positional bias and reassign their comparisons to maintain three valid independent judgments per comparison.

Tie handling in Bradley–Terry Fitting. We treat both-good and both-bad outcomes as ties. A decisive outcome contributes one win to the preferred system, whereas a tie contributes 0.5 wins to each system. We fit

$ p_{kj}=\sigma(b_k-b_j), \qquad \sum_k b_k=0,\qquad BT_k=100\sigma(b_k). $

We have added these study sizes, agreement and exclusion statistics, correlation sample sizes, and tie-handling details to Appendix D of the revised paper.


Q4: Synthetic-to-real prompt alignment and real-user prompt validation

A4: WanPEBench synthesis is anchored to real-user language rather than unconstrained element combination. We invited 50 users to contribute video requests and included a subset as few-shot style examples in the Gemini 3.1 Pro system prompt. Gemini then synthesizes each candidate request from a sampled specification comprising a task, target duration, and (category,subject,scene,style,audio)(\text{category}, \text{subject}, \text{scene}, \text{style}, \text{audio}) tuple, while varying instruction specificity and temporal structure.

To assess whether the synthetic prompts capture real-user prompting characteristics, we compare 500 synthetic T2V prompts generated with real-user requests as few-shot style examples against real-user T2V prompts disjoint from those few-shot examples. We compare token-length distributions using normalized 1-Wasserstein distance [1] and the distributions of prompting-style characteristics using Jensen-Shannon divergence [2]. The low distances of 0.144 and 0.079, respectively, indicate close alignment between synthetic and real-user prompting styles.

Following the reviewer's suggestion, we further evaluate all PE systems on the held-out real-user prompts across the three dimensions. The results are as follows:

SystemSFICDC
WanPE-397B74.4094.0479.36
WanPE-35B73.5294.2678.66
H3-Context-IR68.8695.3077.36
LTX-2.5-PE73.4490.0258.60
SCMaPR93.8095.4650.74
LingBot-Video-PE43.1890.3242.30
VPO62.7297.3431.54
Prompt-A-Video-Cog62.1491.8432.90
Prompt-A-Video-OS60.1694.8421.28
RAPO15.6089.561.00

Compared with the 500 synthetic T2V prompts reported in the paper, the leading system in each dimension remains unchanged. No system moves by more than one position in any dimension, showing that evaluation on held-out real-user prompts does not materially change the system rankings. We have added this analysis to Appendix A.3 of the revised paper.

[1] Villani C. Optimal transport: old and new[M]. Berlin: springer, 2009.

[2] Lin J. Divergence measures based on the Shannon entropy[J]. IEEE Transactions on Information theory, 1991, 37(1): 145-151.

Reviewer_99ahOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WanPEBench directly evaluates video prompt enhancers across text-to-video, image-to-video, and reference-to-video settings. It contains 1,100 curated cases and uses extracted visual facts and 24 criteria to assess semantic fidelity, internal consistency, and directorial creation. Experiments compare ten enhancers and validate system rankings against expert judgments of prompts and videos from two generators.

Critical strengths

The motivation is clear and addresses a practical bottleneck: evaluating a prompt enhancer through rendered videos is expensive and makes it difficult to separate rewriting errors from generator failures. The framework is complete, covering T2V, I2V, and R2V with 1,100 curated cases and a detailed criterion taxonomy. Separating semantic fidelity, internal consistency, and directorial creation is particularly useful because a fluent, coherent rewrite can still change the user’s intent. The authors also include original prompts as a baseline and compare against expert judgments at both prompt and video levels using two downstream generators, rather than validating the evaluator only against another automatic metric.

Critical weaknesses

The main concern I have is about the evidence supporting downstream utility. Table 5 reports correlations between system-level rankings, not agreement on individual prompt improvements. The I2V and R2V pools contain only four and three systems, respectively, including the original-prompt baseline, so perfect rank correlations provide limited evidence of general predictive reliability. The study is useful, but case-level agreement, uncertainty intervals, and sensitivity to removing clearly weaker systems are needed before concluding that prompt-level evaluation can reliably replace downstream validation.

Another issue is that the numerical scoring procedure is not fully specified. The paper says dimension scores aggregate the number and severity of criterion-level issues, but Appendix B mainly gives qualitative definitions and JSON fields containing a 0–100 score. It does not clearly explain whether that score is calculated deterministically or assigned directly by the judge, nor how minor/major issues and inapplicable criteria affect it. This matters when small numerical differences determine system rankings.

The fact-extraction stage also needs a more direct evaluation. Shared ImageFacts and AssetFacts make comparisons consistent, but an incorrect shared fact can still penalize every enhancer that follows the actual reference. This is particularly relevant to identity bindings, temporal order, and audiovisual reference roles. The paper does not isolate extraction accuracy from rubric quality through human-corrected facts or an ablation that applies the same rubrics directly to the original assets.

In terms of construct validity, Directorial Creation may favor a particular form of cinematic elaboration. A concise request can be appropriate for a capable generator without specifying editing, lighting, sound, or a detailed storyboard. The low DC score of original prompts therefore does not by itself establish that they need enhancement. The length-stratified analysis is helpful, but compares different cases and cannot rule out preferences for particular structures or writing styles. A controlled comparison with compact but sufficient prompts would be more convincing.

Finally, the human-validation protocol is difficult to audit in detail. Appendix D reports 60 experts and approximate pair counts, but not the number of unique cases, repeated judgments per comparison, inter-rater agreement, or annotator exclusions. It also does not explain how “both good” and “both bad” responses enter the Bradley–Terry model. These details are important for interpreting the rankings, while measured evaluation cost and latency would help substantiate the practical efficiency claim.

Questions for authors

1.Could the authors report case-level agreement with downstream human preferences, confidence intervals for the system correlations, and sensitivity to removing the original-prompt baseline or clearly weaker systems, especially for the small I2V/R2V pools? 2.How exactly are criterion labels converted into dimension-level 0–100 scores? Could the authors provide the mapping, applicability rules, and repeated-judge variability, or clarify that the numerical score is directly generated by the evaluator? 3.Could the authors evaluate a subset with human-corrected reference facts and compare fact-based evaluation against direct asset inspection under the same rubrics? This would isolate the contribution and failure modes of evidence extraction. 4.Could the authors compare compact but fully sufficient prompts with cinematic expansions of the same requests, including simple single-take or explicitly silent cases, and check whether DC preferences match downstream human preferences? 5.Could the authors provide exact human-study counts, inter-rater agreement, exclusion statistics, and tie handling in Bradley–Terry fitting, together with measured evaluation cost and latency relative to video-based evaluation?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 99ah (Part 1/3)

Dear Reviewer 99ah, we sincerely thank you for recognizing WanPEBench’s practical value, comprehensive design, and expert-validated evaluation framework. We appreciate your constructive comments and questions, and we provide our detailed responses below.


W1&Q1: Case-level agreement, confidence intervals, and sensitivity analyses

A1: Thank you for this concern. Given the limited number of available I2V and R2V prompt enhancement systems, system-level Spearman correlations have limited resolution. Following your suggestion, we conduct additional case-level analyses from three perspectives.

(1) Case-level protocol. For each setting, we analyze 50 randomly sampled cases by comparing score rankings with downstream video preferences. We report case-level preference consistency, where a case is inconsistent only if humans prefer the video generated from the lower-scoring prompt. Each estimate includes a 95% Wilson score confidence interval (CI) from the binomial score test [1].

(2) Case-level results. The results are summarized below.

System pair (I2V)Preference consistency95% Wilson CI
H3-Context-IR vs. LTX-2.5-PE82.0%[69.2%, 90.2%]
H3-Context-IR vs. LingBot-Video-PE86.0%[73.8%, 93.0%]
H3-Context-IR vs. original prompt90.0%[78.6%, 95.7%]
LTX-2.5-PE vs. LingBot-Video-PE76.0%[62.6%, 85.7%]
LTX-2.5-PE vs. original prompt84.0%[71.5%, 91.7%]
LingBot-Video-PE vs. original prompt82.0%[69.2%, 90.2%]
Overall83.3%–
System pair (R2V)Preference consistency95% Wilson CI
H3-Context-IR vs. LTX-2.5-PE82.0%[69.2%, 90.2%]
H3-Context-IR vs. original prompt90.0%[78.6%, 95.7%]
LTX-2.5-PE vs. original prompt84.0%[71.5%, 91.7%]
Overall85.3%–

(3) Uncertainty, sensitivity, and downstream utility. Using 95% Wilson CIs and leave-one-system-out analysis (excluding each system in turn), consistency remains 80.7–86.0% for I2V and 82.0–90.0% for R2V, including 81.3% and 82.0% after removing the original-prompt baseline. These results show that prompt-score ordering aligns with downstream human video preferences at the case level and is not driven by the baseline or any single system.

We have added these case-level analyses to Appendix D.2 of the revised paper.

[1] Wilson E B. Probable inference, the law of succession, and statistical inference[J]. Journal of the American Statistical Association, 1927, 22(158): 209-212.


W2&Q2: Deterministic scoring, applicability rules, and complete evaluation rubrics

A2: Thank you for highlighting this ambiguity. Dimension scores are not freely assigned by the evaluator. The evaluator assigns each criterion a status and severity label based on supporting evidence. The score is then computed deterministically from these labels, as below.

(1) Deterministic severity-to-score mapping. For each dimension, MM and mm denote the numbers of criteria labeled major and minor. A minor issue affects local details without changing the overall interpretation, whereas a major issue affects core semantics, coherence, or directorial feasibility. Each criterion is counted once at its highest severity. The per case dimension score is computed as

$ S(M,m)=100 \text{ if } M=0,m=0;\quad \max(70,94-5m) \text{ if } M=0,m\geq1;\quad \max(0,75-15M-5m) \text{ if } M\geq1. $

DimensionNo issueIssue statusInapplicable
Semantic Fidelityconsistentlossy or alterednot_present
Internal Consistencyconsistentambiguous or inconsistentnot_present
Directorial CreationcompletedeficientNot used

Lossy/altered, ambiguous/inconsistent, and deficient denote semantic omission/change, uncertainty/contradiction, and incomplete directorial design, respectively. Issues are graded minor or major by impact.

(2) Applicability rules. A Semantic Fidelity criterion applies only when relevant content appears in the source instruction, visual evidence, or enhanced prompt. Otherwise, it is labeled not_present and excluded from MM and mm. Internal Consistency follows the same rule when an omitted element is unnecessary for coherence. All nine Directorial Creation criteria are assessed for every prompt, but not every directorial element is required. An omission is penalized only when the element is necessary to realize the requested video, given its duration, format, and use of reference assets.

For I2V, only certain and instruction-relevant ImageFacts are used. For R2V, only AssetFacts relevant to each instruction-specified asset role are applied. Uncertain, null, or irrelevant facts are excluded.

(3) Complete rubrics and public release. The revised Supplementary Material includes the complete T2V, I2V, and R2V evaluation prompts, ImageFacts and AssetFacts extraction prompts, applicability rules, and JSON schemas.

AuthorsOpenReview ↗

Response to Reviewer 99ah (Part 2/3)

W3 & Q3: Human-corrected facts and direct asset inspection

A3: Following the reviewer’s suggestion, we conduct two complementary experiments using human-corrected reference facts and direct asset inspection under the same rubrics.

(1) Evaluation with human-corrected facts. We randomly sample 50 cases from each of I2V and R2V. Ten annotators inspect the original assets and correct inaccurate or unsupported ImageFacts and AssetFacts while preserving their schemas. Verification covers identity and attribute bindings for both tasks, as well as temporal order, audiovisual source bindings, and cross-asset correspondences for R2V. We then reevaluate the enhanced prompts using the same evaluator, rubrics, and scoring procedure.

Each entry below is reported as modality-aware extracted facts / human-corrected facts.

Model (I2V)SFICDC
H3-Context-IR64.28 / 59.7698.32 / 96.2477.04 / 73.30
LTX-2.5-PE60.18 / 56.2890.64 / 91.5850.62 / 53.06
LingBot-Video-PE49.58 / 45.5491.94 / 93.0842.60 / 42.54
Model (R2V)SFICDC
H3-Context-IR63.00 / 58.1295.38 / 95.5874.92 / 74.84
LTX-2.5-PE47.34 / 44.9887.14 / 85.3633.38 / 38.10

All score changes after human correction are within 5 points, and the system ranking remains unchanged in every dimension. Thus, the scores and rankings are robust to human correction of the extracted facts.

(2) Fact-based evaluation versus direct asset inspection. We further compare three evidence conditions on the same subset:

  1. Human-corrected facts: manually corrected ImageFacts or AssetFacts;
  2. Modality-aware extracted facts: facts produced using our modality-specific extraction checklists;
  3. Direct assets: direct inspection of reference images or videos without structured facts.

We hold the evaluated outputs, evaluator, rubrics, and scoring procedure fixed, varying only the evidence representation. We report case-level consistency between the resulting pairwise system preferences and human judgments.

TaskEvidence conditionPreference consistency
I2VHuman-corrected facts86.3%
I2VModality-aware extracted facts83.3%
I2VDirect assets73.7%
R2VHuman-corrected facts88.7%
R2VModality-aware extracted facts85.3%
R2VDirect assets70.0%

The results consistently show Human-corrected facts > Modality-aware extracted facts > Direct assets. Human correction changes human-preference agreement by less than 5 percentage points, showing that the facts constructed using our modality-specific extraction checklists perform comparably to human-corrected facts. In contrast, modality-aware extracted facts outperform direct asset inspection by 9.6 percentage points on I2V and 15.3 percentage points on R2V, demonstrating the benefit of explicit and shared evidence representations. We have included this result in the Section 4.4 and Appendix D.4.


W4&Q4: Directorial Creation score and downstream human preference agreement for compact but sufficient prompts

A4: We appreciate this important suggestion. DC measures explicit directorial control, not basic prompt executability or verbosity. This distinction is increasingly relevant as modern video generators such as Seedance 2.5, Kling 4.0, and Wan3.0 support longer narratives, multi-shot continuity, camera control, editing, and native audiovisual generation. Prompts can now specify not only what appears, but also how the content unfolds through camera, lighting, pacing, transitions, and sound.

As specified in our publicly available rubric in the revised Supplementary Material, DC does not reward prompt length, timestamp syntax, formatting, or cinematic terminology by themselves. Each judgment requires concrete criterion-level evidence. Single-take, static-camera, and explicitly silent designs are treated as valid.

Following the reviewer’s suggestion, we selected 50 compact but fully sufficient cases from each task setting, including single-action, single-take, static-camera, and silent requests. We compared each original prompt with an expansion of the same request using the same downstream DiT. The results are as follows:

SettingSystem pairMean DC (Origin → PE)Human preference consistency
T2VOrigin vs. WanPE-397B27.64 → 83.8080.0%
I2VOrigin vs. H3-Context-IR28.92 → 78.0886.0%
R2VOrigin vs. H3-Context-IR21.24 → 77.4482.0%

Human preference consistency measures case-level agreement between DC ordering and preferences for the generated videos. The results show that when a faithful expansion adds explicit directorial control to the same request, the preference captured by DC generally agrees with downstream human video preference. These results have been included in Appendix D.5 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 99ah (Part 3/3)

W5&Q5: Human-study details and evaluation efficiency

A5: Thank you for requesting these details. We have expanded Appendix D with the complete study protocol and efficiency analysis.

Exact Study Counts: We randomly sample 50 cases per task and evaluate every system pair, yielding 2,750 T2V, 300 I2V, and 150 R2V comparisons. Each comparison is independently evaluated by three annotators.

Inter-rater agreement and exclusions: After accounting for the randomized left/right presentation, all three annotators agree on 85.0% of comparisons. We exclude 3 of 60 annotators due to low agreement or systematic positional bias and reassign their comparisons to maintain three valid independent judgments per comparison.

Tie handling in Bradley–Terry Fitting: We treat both-good and both-bad outcomes as ties. A decisive outcome contributes one win to the preferred system, whereas a tie contributes 0.5 wins to each system. We fit

$ p_{kj}=\sigma(b_k-b_j), \qquad \sum_k b_k=0,\qquad BT_k=100\sigma(b_k). $

Evaluation cost and latency. The API overhead of our prompt-level evaluation is negligible relative to video generation. (GPT-5.4 costs USD 2.50/15.00 per million input/output tokens, while Qwen3.5-Omni-Plus costs approximately USD 1.00/7.50 per million visual/audio input tokens and USD 5.60 per million text output tokens.) In our setup, generating one 15-second video with MiniMax-H3 requires four A100 GPUs and approximately 14–15 minutes. Alternatively, Wan3.0 API generation costs approximately USD 1.20 per 15-second 720p video at its standard rate of USD 0.08 per second. Video-based evaluation additionally requires pairwise human assessment. By avoiding video generation and video-level annotation, our method substantially reduces evaluation latency, computational cost, and human effort.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Responses and Revisions

Following the author-reviewer discussion period, we are grateful to have received positive and constructive feedback from the reviewers. During the rebuttal phase, the main concerns raised by the reviewers and our corresponding responses and revisions are summarized below:

  1. Downstream Utility: We added enhanced-win/tie/original-win rates, case-level preference consistency between alternative enhancements and between enhanced and original prompts, and sensitivity analyses. These results show that WanPEBench evaluation scores reliably reflect expert judgments of both enhanced prompts and downstream videos across generators. (revised in Section 4.4, Appendices D.2 and D.3)

  2. Evaluation Protocol and Human Study Details: We clarified the deterministic severity-to-score mapping, applicability rules, DLI/OTI denominator definitions, and Bradley–Terry tie handling. We also provided human study statistics and included the complete evaluation rubrics in the publicly available Supplementary Material. (revised in Appendices B.1, B.3 and D.1)

  3. Factual Grounding and Rubric Decomposition: We conducted same-judge, matched-input ablations comparing direct asset inspection with structured ImageFacts/AssetFacts and Direct VLM Scoring with our 24-criterion rubric-based evaluation, isolating the respective contributions of factual grounding and rubric decomposition. (revised in Section 4.4 and Appendix D.4)

  4. Robustness to Output Style and Real-User Generalization: We conducted a same-input JSON-to-prose perturbation, compared the distributions of synthetic and real-user prompts, and evaluated all PE systems on held-out real-user prompts. The results show WanPEBench’s robustness to output style, close synthetic-to-real prompt alignment, and stable system rankings on real-user prompts. (revised in Section 4.4 and Appendix A.3)

  5. Evaluation Cost and Latency: We measured the cost and latency of prompt-level evaluation relative to video-based evaluation, showing that WanPEBench substantially reduces computational cost, latency, and human effort by avoiding video generation and annotation. (revised in Appendix D.6)

  6. Rubric Independence and Impartiality: We clarified that all criteria and rubrics were finalized before evaluating any PE system, with the criteria independently derived through consultations with general users and short-video creators. (revised in Appendix B.2)

We sincerely thank all the reviewers and the Area Chair for their time, effort, and valuable feedback throughout the review process.

Best regards,
The Submission 6831 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.