- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces WanPEBench, a benchmark for evaluating video prompt enhancers directly from their outputs without generating videos for every evaluation. It comprises 1,100 expert-verified cases across text-to-video, image-to-video, and reference-to-video enhancement, with 1,005 visual assets and 35 task types. The evaluation framework combines cached, modality-aware evidence extraction with 24 criteria organized into Semantic Fidelity, Internal Consistency, and Directorial Creation. Experiments compare ten enhancers on text-only inputs and smaller subsets on visually conditioned inputs, and analyze temporal complexity, reference complexity, and output length. A human study compares automatic system scores with expert rankings of enhanced prompts and downstream videos generated by Wan3.0 and MiniMax-H3. The paper reports high rank correlations and argues that prompt-level evaluation provides an efficient indicator of downstream utility.
Critical strengths
- The benchmark brings text-only, first-frame-conditioned, and multi-reference prompt enhancement into one evaluation framework, covering temporal structure, audio requirements, and heterogeneous reference roles. This is a useful scope for studying enhancer behavior beyond conventional text rewriting. Direct prompt evaluation also offers a plausible way to diagnose omissions and contradictions without requiring video generation for every evaluation.
- Separating fidelity, consistency, and directorial planning helps distinguish different failure modes, while sharing extracted reference evidence across methods is a sensible consistency measure.
Critical weaknesses
-
The practical downstream significance of the evaluation scores is insufficiently established. Prompt enhancement is intended to improve generated videos, but Table 5 reports system-level rank correlations rather than the magnitude of improvement over original prompts. Although the human study includes a no-enhancement baseline, it does not report downstream preference win/tie/loss rates or effect sizes with uncertainty. Appendix D.1 computes correlations over 11, 4, and 3 systems for T2V, I2V, and R2V, respectively, including that baseline; the perfect I2V/R2V correlations consequently have limited resolution. Such correlations do not establish how much video quality improves or whether the metric can identify beneficial, ineffective, or harmful enhancements on individual inputs. The evidence supports aggregate ordering, but not yet reliable feedback for selecting or refining particular prompts. It therefore remains unclear whether the measured score differences correspond to practically meaningful video improvements rather than primarily compliance with a detailed prompt-writing rubric. This gap limits the paper's central claim of reliable downstream utility.
-
The core scoring protocol is not sufficiently reproducible or independently validated. Appendix B provides criterion descriptions and output schemas, but not a reproducible mapping from minor/major issues to the 0-100 dimension scores, complete evaluator prompts, or calibrated severity rules. The treatment of non-applicable criteria also requires clarification. Appendix B.1 defines pass rates over applicable cases, yet the T2V DLI and OTI percentages in Table 2 are consistent with a denominator of all 500 cases despite supplied examples lacking dialogue or on-screen-text requirements. This is a reporting ambiguity, not proof of incorrect computation, but counting non-applicable cases as passes would inflate the apparent success rates on those capabilities. Furthermore, the underspecified Direct VLM Scoring baseline does not isolate the benefits of evidence extraction and rubric decomposition. These gaps concern the benchmark's central measurement procedure and prevent a clear assessment of what its scores measure and how robustly they can be reproduced.
Questions for authors
-
For each generator and task, how much do videos produced from enhanced prompts improve over those produced from the original prompts? Please report preference win/tie/loss rates or other effect-size estimates with confidence intervals from the existing human study, and identify settings where enhancement offers little benefit or degrades results. This would clarify the practical significance beyond rank correlation.
-
For matched inputs, do the proposed scores correctly predict expert video preferences between alternative enhancements and between enhanced and original prompts? Can the existing annotations support this analysis, including cases where the preferred choice is no enhancement? Please report uncertainty and specify unique-input counts, repeated judgments, and the treatment of "both good" and "both bad." This tests the claimed usefulness for prompt selection rather than only system ranking.
-
What are the exact severity-to-score rule and the numerator/denominator for each criterion? Please explain how
not_presentand failed evaluations are handled, reconcile the DLI/OTI denominator ambiguity, and provide the complete evaluator prompts and scoring implementation. If dimension scores are generated directly by the judge rather than computed, what rules calibrate them and ensure consistency with the criterion-level judgments? -
What evidence separates the benefits of factual grounding and rubric decomposition from a preference for particular writing styles? A same-judge, matched-input ablation would help attribute the gains over Direct VLM Scoring, while meaning-preserving formatting or verbosity changes would test score robustness. Appendix C.2 compares different cases grouped by output length and therefore does not isolate this latter effect from input difficulty.
5 comments
Response to Reviewer vT9r (Part 2/4)
W1&Q2: Matched-input prompt-selection reliability and human-study details
A2: We sincerely appreciate this constructive suggestion. The enhanced-versus-original results in A1 show that the proposed scores predict expert video preferences on matched inputs, rather than merely reflecting system-level rankings. Using I2V and R2V as examples, for each matched input, we compare the ordering of two candidate prompts induced by the proposed scores with experts’ pairwise preference between their generated videos. This comparison covers both pairs of alternative enhancements and enhanced-versus-original pairs. The results are shown below.
| System pair (I2V) | Preference consistency | 95% Wilson CI |
|---|---|---|
| H3-Context-IR vs. LTX-2.5-PE | 82.0% | [69.2%, 90.2%] |
| H3-Context-IR vs. LingBot-Video-PE | 86.0% | [73.8%, 93.0%] |
| H3-Context-IR vs. original prompt | 90.0% | [78.6%, 95.7%] |
| LTX-2.5-PE vs. LingBot-Video-PE | 76.0% | [62.6%, 85.7%] |
| LTX-2.5-PE vs. original prompt | 84.0% | [71.5%, 91.7%] |
| LingBot-Video-PE vs. original prompt | 82.0% | [69.2%, 90.2%] |
| Overall | 83.3% | – |
| System pair (R2V) | Preference consistency | 95% Wilson CI |
|---|---|---|
| H3-Context-IR vs. LTX-2.5-PE | 82.0% | [69.2%, 90.2%] |
| H3-Context-IR vs. original prompt | 90.0% | [78.6%, 95.7%] |
| LTX-2.5-PE vs. original prompt | 84.0% | [71.5%, 91.7%] |
| Overall | 85.3% | – |
After removing the original-prompt baseline, preference consistency remains 81.3% for I2V and 82.0% for R2V. Thus, the case-level agreement is not driven solely by an obviously weaker baseline and also holds when selecting between alternative enhancements.
The existing annotations also retain cases in which experts prefer no enhancement. Importantly, preference consistency remains high even for ineffective or harmful enhancements. For example, experts prefer the original-prompt video to the RAPO-enhanced video in 52% of T2V cases, while the score ordering has 90.0% consistency with these preferences. This indicates that the evaluation does not merely reward more detailed prompt writing and can assign lower scores to harmful enhancements in agreement with downstream human preferences.
We have expanded Appendix D with the complete human-study protocol and agreement statistics.
Exact Study Counts. We randomly sample 50 cases per task and evaluate every system pair, yielding 2,750 T2V, 300 I2V, and 150 R2V comparisons. Each comparison is independently evaluated by three annotators.
Inter-rater agreement and exclusions. After accounting for the randomized left/right presentation, all three annotators agree on 85.0% of comparisons. We exclude 3 of 60 annotators due to low agreement or systematic positional bias and reassign their comparisons to maintain three valid independent judgments per comparison.
Tie handling in Bradley–Terry Fitting. We treat both-good and both-bad outcomes as ties. A decisive outcome contributes one win to the preferred system, whereas a tie contributes 0.5 wins to each system. We fit
$ p_{kj}=\sigma(b_k-b_j), \qquad \sum_k b_k=0,\qquad BT_k=100\sigma(b_k). $
Overall, the existing matched-input annotations show that the proposed scores align with expert preferences for both alternative-enhancement and enhanced-versus-original comparisons, including cases favoring no enhancement, supporting their case-level utility for prompt selection beyond system ranking. We have added the above experimental analyses and human-study details to the revised paper.
Response to Reviewer vT9r (Part 3/4)
W2&Q3: Complete evaluator prompts, scoring implementation and DLI/OTI denominator ambiguity
A3: Thank you for identifying this lack of clarity. Dimension scores are not freely assigned by the evaluator. The evaluator assigns each criterion a status and severity label based on supporting evidence. The score is then computed deterministically from these labels, as detailed below.
(1) Deterministic severity-to-score mapping. For each dimension, and denote the numbers of criteria labeled major and minor. A minor issue affects local details without changing the overall interpretation, whereas a major issue affects core semantics, coherence, or directorial feasibility. Each criterion is counted once at its highest severity. The per case dimension score is computed as
$ S(M,m)=100 \text{ if } M=0,m=0;\quad \max(70,94-5m) \text{ if } M=0,m\geq1;\quad \max(0,75-15M-5m) \text{ if } M\geq1. $
| Dimension | No issue | Issue status | Inapplicable |
|---|---|---|---|
| Semantic Fidelity | consistent | lossy or altered | not_present |
| Internal Consistency | consistent | ambiguous or inconsistent | not_present |
| Directorial Creation | complete | deficient | Not used |
Lossy/altered, ambiguous/inconsistent, and deficient denote semantic omission/change, uncertainty/contradiction, and incomplete directorial design, respectively. Issues are graded minor or major by impact.
(2) Applicability rules. A Semantic Fidelity criterion applies only when relevant content appears in the source instruction, visual evidence, or enhanced prompt. Otherwise, it is labeled not_present and excluded from and . Internal Consistency follows the same rule when an omitted element is unnecessary for coherence. All nine Directorial Creation criteria are assessed for every prompt, but not every directorial element is required. An omission is penalized only when the element is necessary to realize the requested video, given its duration, format, and use of reference assets.
For I2V, only certain and instruction-relevant ImageFacts are used. For R2V, only AssetFacts relevant to each instruction-specified asset role are applied. Uncertain, null, or irrelevant facts are excluded.
(3) DLI/OTI denominator ambiguity. We thank the reviewer for identifying this reporting ambiguity and clarify the definition and computation of these percentages below. The DLI and OTI percentages in Table 2 are penalty-based, all-case non-violation rates rather than pass rates conditioned on criterion applicability. For criterion , we compute
$ R_{\mathrm{NV}}(c)= \frac{N_{\mathrm{no\ issue}}(c)+N_{\mathrm{not_present}}(c)} {N_{\mathrm{valid}}(c)}, $
where counts applicable cases without a criterion-level violation. A not_present label indicates that the input contains no corresponding requirement to violate. It is counted as a non-violation and contributes no minor or major penalty. Criterion instances labeled minor or major are counted as violations. Accordingly, is 500 for T2V and 300 for I2V and R2V. These percentages measure the benchmark-wide absence of criterion violations rather than performance conditioned on cases containing dialogue or on-screen text. Failed or malformed evaluations are retried. If they remain invalid, they are excluded from and reported separately.
Following the reviewer’s suggestion, we additionally report DLI and OTI rates using a shared criterion-relevant denominator, defined as the cases in which the original prompt or any evaluated enhanced prompt contains the corresponding content. The results are shown in the three tables below.
| Metric (T2V) | WanPE-397B | WanPE-35B | H3-Context-IR | SCMaPR | LTX-2.5-PE | LingBot-Video-PE | VPO | Prompt-A-Video-Cog | Prompt-A-Video-OS | RAPO |
|---|---|---|---|---|---|---|---|---|---|---|
| DLI | 85.64% | 70.30% | 82.67% | 50.99% | 51.49% | 30.69% | 41.09% | 47.03% | 42.57% | 32.18% |
| OTI | 84.88% | 91.28% | 86.63% | 85.47% | 74.42% | 55.23% | 68.02% | 74.42% | 70.35% | 38.95% |
| Metric (I2V) | H3-Context-IR | LTX-2.5-PE | LingBot-Video-PE |
|---|---|---|---|
| DLI | 64.13% | 40.22% | 25.00% |
| OTI | 79.93% | 70.07% | 69.01% |
| Metric (R2V) | H3-Context-IR | LTX-2.5-PE |
|---|---|---|
| DLI | 66.44% | 39.60% |
| OTI | 55.00% | 39.17% |
Across all three tasks, the shared-denominator results preserve the system rankings in Table 2 while excluding cases that are universally inapplicable to DLI or OTI, confirming that the conclusions are not driven by the original denominator choice. We have added these shared-denominator DLI/OTI results to the revised paper (Appendix B.3).
(4) Complete rubrics and public release. The revised Supplementary Material includes the complete T2V, I2V, and R2V evaluation prompts, ImageFacts and AssetFacts extraction prompts, applicability rules, and JSON schemas.
Response to Reviewer vT9r (Part 1/4)
Dear Reviewer vT9r, we sincerely thank you for recognizing the value of our unified evaluation scope, direct prompt-level diagnosis, and evidence-grounded decomposition of prompt-enhancement quality. We appreciate your constructive comments and questions, and we provide our detailed responses below.
W1&Q1: Practical downstream improvements over original prompts
A1: Thank you for this important suggestion. System-level rank correlations alone may not quantify the magnitude or direction of downstream quality changes relative to the original prompts. We therefore report the human preference outcomes between videos generated from each enhanced prompt and its corresponding original prompt.
For each system, we evaluate 50 cases and report the enhanced-win / tie / original-win rates. We additionally report case-level preference consistency, which measures whether the prompt-level score ordering agrees with downstream human video preference. A comparison is inconsistent when humans explicitly prefer the video generated from the lower-scoring prompt. The reported uncertainty is the 95% Wilson score confidence interval [1] for this consistency estimate. The results for the three tasks are as follows.
These results reveal substantial heterogeneity hidden by system-level rank correlations. Every I2V and R2V enhancer wins more often than it loses. In T2V, WanPE-397B, WanPE-35B, and H3-Context-IR provide clear benefits, whereas LingBot-Video-PE and Prompt-A-Video-OS are close to parity. VPO and Prompt-A-Video-Cog produce more original-prompt wins than enhanced-prompt wins, while RAPO substantially degrades downstream quality. These weak or negative results mainly occur for prompt enhancers originally designed for earlier video generators, whose limited context capacity and instruction-following ability encouraged descriptive rewriting that may compress or omit details when the original prompt is already highly informative. When applied to modern generators, these legacy enhancers may compress or discard important details from already informative prompts, thereby yielding little benefit or even degrading downstream quality.
Preference consistency remains high across both beneficial and harmful settings, reaching 85.0% for T2V, 85.3% for I2V, and 87.0% for R2V. We have added these win/tie/loss results and uncertainty estimates to the revised paper (Appendix D.3) to clarify the practical benefits and limitations of prompt enhancement beyond rank correlation.
[1] Wilson E B. Probable inference, the law of succession, and statistical inference[J]. Journal of the American Statistical Association, 1927, 22(158): 209-212.