- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates whether video generation agents that rewrite or iteratively refine prompts can outperform simply resampling from a fixed prompt under the same video-generation budget. The authors compare nine prompting/refinement methods, including prompt-enhancement methods, iterative agents, simple counterparts, and human-written prompts, across up to six video generators. Each method is allocated 20 generations per generator, with videos used for iterative feedback counted toward the same budget. The main finding is that agentic refinement generally does not consistently outperform best-of-N sampling from a Simple Prompt: improvements in some dimensions often coincide with degradation in others, particularly action fulfillment, and gains do not consistently transfer across generators. The paper validates these conclusions using automatic evaluation over semantic and aesthetic dimensions, together with blinded human pairwise judgments.
Critical strengths
-
Counting all generated videos, including those used only for iterative feedback, creates a substantially fairer comparison between agentic refinement and simple resampling than evaluations that allow agents to use additional generations without charging them to the same budget. The fact that additional LLM/VLM calls are not counted actually makes the comparison somewhat favorable to the agents.
-
Evaluating methods across several open-source and proprietary generators helps reveal that prompt-refinement gains can be highly generator-dependent rather than universally transferable. This is an important empirical observation for claims about general-purpose video-generation agents.
-
Comparing published methods not only against Simple Prompt but also against Simple Rewrite, Simple Iterative, and Human Prompt helps disentangle whether performance gains arise from sophisticated algorithmic machinery or merely from rewriting and repeated sampling.
-
The appendix explicitly states which components of RAPO, RAPO++, VPO, and VQQA were reproduced or adapted. This transparency makes the empirical results easier to interpret, even though the adaptations themselves also introduce limitations.
Critical weaknesses
- All methods are evaluated on a single reference-conditioned request involving two characters, one scene, and a fixed multi-step action sequence. This is insufficient evidence for broad claims about whether video-generation agents generally outperform resampling. The chosen task may also be particularly sensitive to prompt rewriting because dropping even one action can strongly reduce action-fulfillment scores. A broader collection of requests covering different levels of complexity, action types, camera control, style, and reference dependence would be necessary to establish external validity.
- Each method-generator condition uses only 20 samples, while many conclusions rely on the maximum-scoring sample or Top-5 Mean. Such order statistics can have substantial sampling variance. The paper does not convincingly quantify uncertainty over expected best-of-N performance through repeated pools, bootstrap resampling, or confidence intervals. Consequently, relatively small differences between methods are difficult to interpret as stable effects.
- The appendix explicitly notes that the normalized radar area depends on the ordering of the generator axes because adjacent scores are multiplied. An aggregate statistic whose value changes under a permutation of otherwise identical dimensions is difficult to justify as a central quantitative comparison. Generator-wise paired differences, average ranks, or an order-invariant aggregate would be more defensible.
- RAPO is evaluated through a specific instruction-rewrite branch, RAPO++ through a reproduction of its SSPO test-time stage, VPO uses the released optimizer without reproducing its training stages, and VQQA is also reproduced in the authors’ framework. Therefore, statements such as “published agents do not beat Simple Prompt” should be interpreted cautiously, since part of the observed performance may arise from adaptation choices rather than intrinsic limitations of the original systems.
- Human raters compare individual method videos against individual Simple Prompt videos rather than directly evaluating the best selected outputs from two equal-size candidate pools. This evaluates pairwise candidate quality, but it does not exactly validate the paper’s primary claim about which strategy produces the best result under an N-sample budget.
Questions for authors
How sensitive are the main conclusions to the choice of the single evaluation request? Do the authors have results on additional requests, even on a smaller subset of generators and methods, that could demonstrate whether the observed ranking is stable across tasks? Second, can the authors provide uncertainty estimates for Maximum and Top-5 Mean, for example through repeated N=20 subsampling or bootstrap confidence intervals? Third, how sensitive are the conclusions in Table 3 to replacing radar area with an order-invariant aggregation metric? Finally, for the reproduced or partially adapted methods such as RAPO++, VQQA, and RAPO, what evidence verifies that the implementations recover the behavior or performance trends reported in the original papers?
4 comments
# Response to Reviewer z4ts (2/2)
W4 & Q4. Adaptation and validation of the reproduced methods
This concern has two parts: whether our reproductions are faithful, and whether the methods should be retrained for each generator.
Reproduction. Several methods do not release complete code, so we reproduced each one as closely as the paper and the available code allow. Section 6.2 and Appendix D.3 (Table 17) list all retained components and differences. Two results suggest that the reproductions work as intended. VQQA and RAPO++ both outperform Simple Iterative, the naive loop they are designed to improve on (area sums +2.2 and −19.7 vs. −27.8; Table 3a). VPO improves human-judged visual quality (55.4% win rate; Table 4), which is its stated objective. We are also running each reproduction in its original setting to check whether it recovers the reported trends. We will post these results during the discussion if they are ready, and otherwise include them in the final version. We will release all code, prompts, and videos.
Retraining. We do not retrain the methods for each generator. Most of them claim to work across generators: VPO reports "strong generalization across video generation models", RAPO++ describes itself as "model-agnostic", VQQA as "strictly model-agnostic", and SCMAPR as "training-free and model-agnostic". Our evaluation tests these claims on generators the methods were not developed on.
We do not dispute that an enhancer trained for a specific generator can help; providers train their own for this reason. Generator-specific retraining is costly for individual creators, motivating evaluation without generator-specific retraining. Our conclusion therefore concerns the documented implementations transferred to a new generator, not the intrinsic limits of the original systems.
We have revised the text to state this (Section 6.2).
W5. Human evaluation does not compare the best outputs
We agree. The existing study compares individual videos of each method with individual Simple Prompt videos. We are running an additional study in which 10 raters compare the best video of each method pool with the best video of the Simple Prompt pool on the same generator. We will post the results during the discussion if they are ready, and otherwise include them in the final version.
Follow-up on Our Response and Revised Manuscript
Dear Reviewer z4ts,
Thank you again for your time and thoughtful feedback on our paper.
We would like to gently follow up on our rebuttal and revised manuscript. When you have a chance, we would appreciate your thoughts on whether the updates address your concerns.
Please let us know if any questions remain. We would be happy to provide further clarification.
Best regards, The Authors
Thank you for the detailed response and additional experiments, which have addressed some of my original concerns. However, the limited generalizability across tasks and generators, insufficient validation of reproduced methods, and lack of direct human evaluation of the best outputs remain important limitations. Therefore, despite these improvements, I maintain my original recommendation.
Response to Reviewer z4ts (1/2)
We thank the reviewer for the detailed feedback. In the revision, we added four requests, repeated experiments with confidence intervals, and an order-invariant aggregate. The four questions correspond to W1–W4, so we answer each pair together.
W1 & Q1. Single request and sensitivity to the request
We agree that one request cannot support a general claim. We added four requests on the three open-source generators: a single-person action, a non-violent object interaction, a structured multi-shot request, and a complex sporting action (Section 5.4; Appendix D.1.1; Appendix C.1, Tables 6–9).
The overall finding persists: agents do not consistently outperform Simple Prompt. The exception is that PE has an advantage on structured prompts, while iterative agents still show no consistent advantage across tasks (Tables 6–9).
The three-request average below shows that all three iterative methods have lower Area Sums than Simple Prompt at both Maximum and Top-5 Mean, while SCMAPR is near parity (+0.5 and +1.0).
Three-request average (a1, a2, c; Section 5.4, Table 5). For each generator, the corresponding selected dimension scores are averaged equally across the three requests before computing radar areas. SP gives absolute normalized areas (%); other entries are percentage-point differences from SP.
Maximum
Top-5 Mean
HP: Human Prompt; SR: Simple Rewrite; SI: Simple Iterative.
Action losses persist even on the short, single-person task a1: with Hunyuan-1.5, VPO's Maximum video scores 8 on action fulfillment versus 10 for Simple Prompt (Table 6).
W2 & Q2. Uncertainty of Maximum and Top-5 Mean
We agree that order statistics from 20 samples have high variance. We repeated selected conditions with independent pools and report the mean and confidence interval of Maximum and Top-5 Mean (Section 5.4; Appendix C.7.2, Table 14).
We repeated Simple Iterative, RAPO++, and VQQA on all three open-source generators. The original request has three GPT-scored runs and two additional Claude-scored runs per condition; task a1 has three runs under each evaluator. Table 14 summarizes the 19 updates, excluding the initial video. For example, the original-task GPT Maximum intervals for VQQA (27.86 with 95% CI [25.68, 30.03]) and Simple Iterative (24.79 [21.59, 28.00]) overlap. The repeated runs also preserve the best-versus-final gap, supporting the finding that refinement does not reliably improve later outputs.
Our claim is that no agent shows a consistent gain over Simple Prompt. We do not interpret small point-estimate differences as reliable rankings.
The revised abstract and conclusion state this as "no consistent advantage over Simple Prompt".
We keep Maximum and Top-5 Mean as the main statistics because a creator keeps the best videos of a pool, not an average one.
W3 & Q3. Radar area depends on the axis order
We agree. We added an order-invariant additive score (Appendix C.4, Table 12) and refer to it in Section 5.1.
The six-generator Maximum ranking is unchanged. Small Top-5 differences for Human Prompt and SCMAPR change sign, but all three iterative final-round averages remain below Simple Prompt Maximum. The main conclusions therefore persist under this order-invariant score.
(Continued in Part 2/2: W4–W5 and Q4.)