acceptodds
Under review as a conference paper at ICLR 2027

Can Video Generation Agents Beat Simply Trying Again?

Abstract

A better video may come from a better prompt—or simply another try. Video generation agents revise prompts before generation or after inspecting results, yet they spend the same resource as best-of-N (BoN) sampling: generation attempts. Existing evaluations rarely compare agents against BoN with a simple prompt under the same budget, leaving unclear whether reported gains would survive such a comparison. We therefore ask: given the same number of attempts, can an agent beat simply trying again? We evaluate nine methods on a single reference-conditioned request across up to six video generators, giving each method 20 generations per generator and counting every video an agent produces, including those used only for feedback. Published agents are compared with Simple Prompt, which repeatedly samples a minimal prompt, and with a simple counterpart in their own family. The answer is largely no. No agent's best videos improve on Simple Prompt across all dimensions; gains in one dimension come with losses in another, especially in action fulfillment. More elaborate agents fare no better than Simple Prompt even when they improve on their simple counterparts, expert-written prompts perform on par, and the few gains rarely transfer across generators. Iterative agents also do not improve round by round: their final-round videos fall well below their best ones, and picking the best round turns them into BoN with a changing prompt. These results suggest that agents and simply trying again are far closer than published evaluations imply, and that agents should be evaluated against matched-budget BoN on the target generator.

Primary area:
datasets and benchmarks
Keywords:
video generation, generative models, prompt optimization, multimodal agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z4tsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates whether video generation agents that rewrite or iteratively refine prompts can outperform simply resampling from a fixed prompt under the same video-generation budget. The authors compare nine prompting/refinement methods, including prompt-enhancement methods, iterative agents, simple counterparts, and human-written prompts, across up to six video generators. Each method is allocated 20 generations per generator, with videos used for iterative feedback counted toward the same budget. The main finding is that agentic refinement generally does not consistently outperform best-of-N sampling from a Simple Prompt: improvements in some dimensions often coincide with degradation in others, particularly action fulfillment, and gains do not consistently transfer across generators. The paper validates these conclusions using automatic evaluation over semantic and aesthetic dimensions, together with blinded human pairwise judgments.

Critical strengths
  1. Counting all generated videos, including those used only for iterative feedback, creates a substantially fairer comparison between agentic refinement and simple resampling than evaluations that allow agents to use additional generations without charging them to the same budget. The fact that additional LLM/VLM calls are not counted actually makes the comparison somewhat favorable to the agents.

  2. Evaluating methods across several open-source and proprietary generators helps reveal that prompt-refinement gains can be highly generator-dependent rather than universally transferable. This is an important empirical observation for claims about general-purpose video-generation agents.

  3. Comparing published methods not only against Simple Prompt but also against Simple Rewrite, Simple Iterative, and Human Prompt helps disentangle whether performance gains arise from sophisticated algorithmic machinery or merely from rewriting and repeated sampling.

  4. The appendix explicitly states which components of RAPO, RAPO++, VPO, and VQQA were reproduced or adapted. This transparency makes the empirical results easier to interpret, even though the adaptations themselves also introduce limitations.

Critical weaknesses
  1. All methods are evaluated on a single reference-conditioned request involving two characters, one scene, and a fixed multi-step action sequence. This is insufficient evidence for broad claims about whether video-generation agents generally outperform resampling. The chosen task may also be particularly sensitive to prompt rewriting because dropping even one action can strongly reduce action-fulfillment scores. A broader collection of requests covering different levels of complexity, action types, camera control, style, and reference dependence would be necessary to establish external validity.
  2. Each method-generator condition uses only 20 samples, while many conclusions rely on the maximum-scoring sample or Top-5 Mean. Such order statistics can have substantial sampling variance. The paper does not convincingly quantify uncertainty over expected best-of-N performance through repeated pools, bootstrap resampling, or confidence intervals. Consequently, relatively small differences between methods are difficult to interpret as stable effects.
  3. The appendix explicitly notes that the normalized radar area depends on the ordering of the generator axes because adjacent scores are multiplied. An aggregate statistic whose value changes under a permutation of otherwise identical dimensions is difficult to justify as a central quantitative comparison. Generator-wise paired differences, average ranks, or an order-invariant aggregate would be more defensible.
  4. RAPO is evaluated through a specific instruction-rewrite branch, RAPO++ through a reproduction of its SSPO test-time stage, VPO uses the released optimizer without reproducing its training stages, and VQQA is also reproduced in the authors’ framework. Therefore, statements such as “published agents do not beat Simple Prompt” should be interpreted cautiously, since part of the observed performance may arise from adaptation choices rather than intrinsic limitations of the original systems.
  5. Human raters compare individual method videos against individual Simple Prompt videos rather than directly evaluating the best selected outputs from two equal-size candidate pools. This evaluates pairwise candidate quality, but it does not exactly validate the paper’s primary claim about which strategy produces the best result under an N-sample budget.
Questions for authors

How sensitive are the main conclusions to the choice of the single evaluation request? Do the authors have results on additional requests, even on a smaller subset of generators and methods, that could demonstrate whether the observed ranking is stable across tasks? Second, can the authors provide uncertainty estimates for Maximum and Top-5 Mean, for example through repeated N=20 subsampling or bootstrap confidence intervals? Third, how sensitive are the conclusions in Table 3 to replacing radar area with an order-invariant aggregation metric? Finally, for the reproduced or partially adapted methods such as RAPO++, VQQA, and RAPO, what evidence verifies that the implementations recover the behavior or performance trends reported in the original papers?

4 comments
AuthorsOpenReview ↗

Response to Reviewer z4ts (1/2)

We thank the reviewer for the detailed feedback. In the revision, we added four requests, repeated experiments with confidence intervals, and an order-invariant aggregate. The four questions correspond to W1–W4, so we answer each pair together.

W1 & Q1. Single request and sensitivity to the request

We agree that one request cannot support a general claim. We added four requests on the three open-source generators: a single-person action, a non-violent object interaction, a structured multi-shot request, and a complex sporting action (Section 5.4; Appendix D.1.1; Appendix C.1, Tables 6–9).

The overall finding persists: agents do not consistently outperform Simple Prompt. The exception is that PE has an advantage on structured prompts, while iterative agents still show no consistent advantage across tasks (Tables 6–9).

The three-request average below shows that all three iterative methods have lower Area Sums than Simple Prompt at both Maximum and Top-5 Mean, while SCMAPR is near parity (+0.5 and +1.0).

Three-request average (a1, a2, c; Section 5.4, Table 5). For each generator, the corresponding selected dimension scores are averaged equally across the three requests before computing radar areas. SP gives absolute normalized areas (%); other entries are percentage-point differences from SP.

Maximum

DimensionSPHPSRRAPOSCMAPRVPOSIRAPO++VQQA
Entity/Obj.89.10.0-6.1-6.4-4.1-2.1-16.1-20.4-1.4
Action82.8-12.3+0.1-19.4+0.1-11.7-12.1-7.8-13.6
Ident./Appear.63.9+5.6+5.6+1.8+7.4-1.9-5.9-14.9+3.1
VADB Total34.1-4.8-2.5-3.6-3.0-1.9-3.4-3.7+0.4
Area Sum269.9-11.6-3.0-27.6+0.5-17.6-37.6-46.7-11.6

Top-5 Mean

DimensionSPHPSRRAPOSCMAPRVPOSIRAPO++VQQA
Entity/Obj.84.2-2.0-3.6-6.50.0-2.0-19.1-24.4-0.7
Action66.5-9.5-8.1-18.3+2.2-4.4-28.0-8.6-6.4
Ident./Appear.63.4+0.9+0.5-1.0+0.9-6.0-9.5-14.7-1.5
VADB Total31.9-3.4-1.3-2.2-2.1-1.1-0.8-2.1-1.5
Area Sum246.1-14.0-12.5-28.0+1.0-13.5-57.4-49.8-10.1

HP: Human Prompt; SR: Simple Rewrite; SI: Simple Iterative.

Action losses persist even on the short, single-person task a1: with Hunyuan-1.5, VPO's Maximum video scores 8 on action fulfillment versus 10 for Simple Prompt (Table 6).

W2 & Q2. Uncertainty of Maximum and Top-5 Mean

We agree that order statistics from 20 samples have high variance. We repeated selected conditions with independent pools and report the mean and confidence interval of Maximum and Top-5 Mean (Section 5.4; Appendix C.7.2, Table 14).

We repeated Simple Iterative, RAPO++, and VQQA on all three open-source generators. The original request has three GPT-scored runs and two additional Claude-scored runs per condition; task a1 has three runs under each evaluator. Table 14 summarizes the 19 updates, excluding the initial video. For example, the original-task GPT Maximum intervals for VQQA (27.86 with 95% CI [25.68, 30.03]) and Simple Iterative (24.79 [21.59, 28.00]) overlap. The repeated runs also preserve the best-versus-final gap, supporting the finding that refinement does not reliably improve later outputs.

Our claim is that no agent shows a consistent gain over Simple Prompt. We do not interpret small point-estimate differences as reliable rankings.

The revised abstract and conclusion state this as "no consistent advantage over Simple Prompt".

We keep Maximum and Top-5 Mean as the main statistics because a creator keeps the best videos of a pool, not an average one.

W3 & Q3. Radar area depends on the axis order

We agree. We added an order-invariant additive score (Appendix C.4, Table 12) and refer to it in Section 5.1.

The six-generator Maximum ranking is unchanged. Small Top-5 differences for Human Prompt and SCMAPR change sign, but all three iterative final-round averages remain below Simple Prompt Maximum. The main conclusions therefore persist under this order-invariant score.

(Continued in Part 2/2: W4–W5 and Q4.)

AuthorsOpenReview ↗

# Response to Reviewer z4ts (2/2)

W4 & Q4. Adaptation and validation of the reproduced methods

This concern has two parts: whether our reproductions are faithful, and whether the methods should be retrained for each generator.

Reproduction. Several methods do not release complete code, so we reproduced each one as closely as the paper and the available code allow. Section 6.2 and Appendix D.3 (Table 17) list all retained components and differences. Two results suggest that the reproductions work as intended. VQQA and RAPO++ both outperform Simple Iterative, the naive loop they are designed to improve on (area sums +2.2 and −19.7 vs. −27.8; Table 3a). VPO improves human-judged visual quality (55.4% win rate; Table 4), which is its stated objective. We are also running each reproduction in its original setting to check whether it recovers the reported trends. We will post these results during the discussion if they are ready, and otherwise include them in the final version. We will release all code, prompts, and videos.

Retraining. We do not retrain the methods for each generator. Most of them claim to work across generators: VPO reports "strong generalization across video generation models", RAPO++ describes itself as "model-agnostic", VQQA as "strictly model-agnostic", and SCMAPR as "training-free and model-agnostic". Our evaluation tests these claims on generators the methods were not developed on.

We do not dispute that an enhancer trained for a specific generator can help; providers train their own for this reason. Generator-specific retraining is costly for individual creators, motivating evaluation without generator-specific retraining. Our conclusion therefore concerns the documented implementations transferred to a new generator, not the intrinsic limits of the original systems.

We have revised the text to state this (Section 6.2).

W5. Human evaluation does not compare the best outputs

We agree. The existing study compares individual videos of each method with individual Simple Prompt videos. We are running an additional study in which 10 raters compare the best video of each method pool with the best video of the Simple Prompt pool on the same generator. We will post the results during the discussion if they are ready, and otherwise include them in the final version.

AuthorsOpenReview ↗

Follow-up on Our Response and Revised Manuscript

Dear Reviewer z4ts,

Thank you again for your time and thoughtful feedback on our paper.

We would like to gently follow up on our rebuttal and revised manuscript. When you have a chance, we would appreciate your thoughts on whether the updates address your concerns.

Please let us know if any questions remain. We would be happy to provide further clarification.

Best regards, The Authors

Thank you for the detailed response and additional experiments, which have addressed some of my original concerns. However, the limited generalizability across tasks and generators, insufficient validation of reproduced methods, and lack of direct human evaluation of the best outputs remain important limitations. Therefore, despite these improvements, I maintain my original recommendation.

Reviewer_sVfdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether video-generation agents improve over best-of-N sampling when every method receives the same number of video-generation attempts. It compares different prompting methods on one reference-conditioned creative request across various generators, allocating videos to each method-generator condition and counting videos used for feedback toward that budget. Evaluation covers entity fulfillment, action fulfillment, identity/appearance persistence, and aesthetics, supplemented by blinded human comparisons. The reported results show no consistent advantage for the evaluated agents over repeatedly sampling the fixed Simple Prompt; gains depend on the generator and metric, and later refinement rounds often perform worse than earlier candidates. The contribution is an empirical critique of evaluation practice rather than a new generation algorithm.

Critical strengths
  1. The matched-budget question is important and practically relevant. Counting feedback videos addresses a real confound: improvements attributed to agentic refinement may instead come from additional generation attempts. Including Simple Rewrite and Simple Iterative also helps distinguish the value of a published pipeline from that of a simpler procedure in the same family. This evaluation principle is useful even if the empirical conclusions require a narrower scope.

  2. The study exposes meaningful trade-offs and documents implementation scope. The cross-generator comparison and separate semantic and visual judgments show why aesthetic improvements should not be treated as improved task completion. It is interesting. Paper also provides a traceable analysis of failures throughout the iterative process. The paper reports performance at the best and final iterations, tracks performance across iterations, and presents concrete examples of how prompts evolve through successive revisions.

Critical weaknesses

1、The paper uses an excessive variety of colors, making its presentation visually cluttered and obscuring the main findings. In the tables, too many values are highlighted in red; highlighting only the most informative results would help readers identify the key takeaways. The main text also uses multiple colors, including red, green, blue and etc. In addition, Figure 1 appears to contain informal annotations, which give it an unfinished appearance. For a submission to a top conference, a more consistent and polished presentation would be appropriate. Points requiring emphasis should be explained clearly in the text. Several methods mentioned in lines 76–77, including VPO, RAPO, SCMAPR, and VQQA, appear to lack citations or labels.

2、Video generation agents encompass a broad range of approaches, whereas this paper appears to focus primarily on prompt rewriting and evaluates only a limited set of agent capabilities. Other potentially valuable capabilities, such as multi-shot planning, coordination across shots, and structured control, are not adequately examined. The current experiments therefore do not support a general conclusion that video generation agents offer no advantage. In many practical settings, users may treat a video generation agent as a black box and care primarily about the quality of its final output, rather than the number of intermediate steps or calls it makes. A comprehensive comparison should therefore include an evaluation of the final generated videos. This is arguably one of the most important comparisons and should, at a minimum, be a central part of the experimental evaluation.

Questions for authors
  1. Scope clarification: Was the single request fixed before observing any comparative results, and are results available for other independently chosen requests?

  2. Implementation clarification: For RAPO, RAPO++, and VQQA, which original components and recommended iteration settings were retained? Is there a validation on the source method's intended configuration showing that the reproduction achieves its expected behavior before transfer to this task?

  3. Could you evaluate the approach on a broader range of agent tasks? I suspect that simply try again may be ineffective in many settings, such as tasks involving structured or constrained generation, or cases where the generator itself fails.

  4. Have you measured how many iterations Video Generation Agents typically require to produce a high-quality video? Is a fixed budget of N = 20 videos fair to this baseline, especially if it requires, for example, 40 iterations to achieve satisfactory results? Could the authors also evaluate performance using a budget based on the maximum number of iterations Video Generation Agents needs to produce a satisfactory video?

3 comments
AuthorsOpenReview ↗

# Response to Reviewer sVfd (1/2)

W1. Presentation: excessive color, unfinished Figure 1, and missing citations

We have revised the presentation as suggested. We removed unnecessary colors from the text, limited table highlighting to the key results, removed the informal annotations in Figure 1, and added citations for VPO, RAPO, SCMAPR, and VQQA at first mention (Sections 2 and 4.1; Figure 1; Tables 3–4).

W2 (scope). Agents beyond prompt refinement

Prompt enhancement benefits this structured request, but iteration still does not reliably improve later outputs.

We agree that our study covers only prompt-level agents (prompt enhancement and iterative refinement), not all video generation agents.

We have revised the abstract, introduction, and conclusion to state this scope.

We added a structured multi-shot task on the three open-source generators (Section 5.4; Appendix C.1, Table 8).

Thank you for suggesting this setting: all four prompt-enhancement methods improve the aggregate Top-5 radar area. However, all nine GPT-scored iterative trajectories finish below their own best output, so refinement does not provide sustained improvement.

Agents that use structured control signals are outside our scope; see Q3.

W2 (final output) & Q4. Does the agent's output improve with more rounds, and is N = 20 fair?

The original submission already evaluates the final-round video of each iterative agent, which is what a black-box user receives. The revised Figure 3 (Section 5.1) shows this more clearly.

The final-round video is much worse than the agent's best round. In area sum relative to the Simple Prompt Maximum, VQQA goes from +2.2 (best round) to −61.9 (final round), RAPO++ from −19.7 to −78.5, and Simple Iterative from −27.8 to −125.4 (Table 3a). The best round can occur well before the final round. For VQQA it falls between rounds 3 and 17 depending on the generator, and for Simple Iterative on Wan-3.0 it is round 0, i.e., the unrevised prompt.

All methods receive a 20-video budget, including feedback videos. We extended VQQA on Seedance-2.0-fast to 40 videos because its 20-video GPT Maximum is close to Simple Prompt (34.05 vs. 34.18; Appendix C.7.1, Table 13).

We see no monotonic upward trend in rounds 20–39. Scores fluctuate as in rounds 0–19.

Neither evaluator's best combined score improves beyond the first 20 videos, while the GPT final-output score falls from 33.25 to 31.18 (Table 13).

These results indicate that the iterative agents do not improve steadily with more rounds. Their gains come from selecting the best round, which is equivalent to best-of-N sampling with a changing prompt. A larger budget would therefore also have to be given to Simple Prompt.

Q1. Was the request fixed in advance, and do results hold on other requests?

Yes. The request was fixed before we designed the comparison and before we saw any results. We added four requests on the three open-source generators (Sections 4.2 and 5.4; Appendix D.1.1; Appendix C.1, Tables 6–9).

The four requests were chosen to vary action complexity, interaction type, and shot structure, and were fixed before their comparative experiments.

Their results are consistent with the main findings. The three-request average below shows that all three iterative methods have lower Area Sums than Simple Prompt at both Maximum and Top-5 Mean, while SCMAPR is near parity (+0.5 and +1.0). Prompt enhancement helps on the structured multi-shot request, as noted in W2.

Three-request average (a1, a2, c; Section 5.4, Table 5). For each generator, the corresponding selected dimension scores are averaged equally across the three requests before computing radar areas. SP gives absolute normalized areas (%); other entries are percentage-point differences from SP.

Maximum

DimensionSPHPSRRAPOSCMAPRVPOSIRAPO++VQQA
Entity/Obj.89.10.0-6.1-6.4-4.1-2.1-16.1-20.4-1.4
Action82.8-12.3+0.1-19.4+0.1-11.7-12.1-7.8-13.6
Ident./Appear.63.9+5.6+5.6+1.8+7.4-1.9-5.9-14.9+3.1
VADB Total34.1-4.8-2.5-3.6-3.0-1.9-3.4-3.7+0.4
Area Sum269.9-11.6-3.0-27.6+0.5-17.6-37.6-46.7-11.6

Top-5 Mean

DimensionSPHPSRRAPOSCMAPRVPOSIRAPO++VQQA
Entity/Obj.84.2-2.0-3.6-6.50.0-2.0-19.1-24.4-0.7
Action66.5-9.5-8.1-18.3+2.2-4.4-28.0-8.6-6.4
Ident./Appear.63.4+0.9+0.5-1.0+0.9-6.0-9.5-14.7-1.5
VADB Total31.9-3.4-1.3-2.2-2.1-1.1-0.8-2.1-1.5
Area Sum246.1-14.0-12.5-28.0+1.0-13.5-57.4-49.8-10.1

HP: Human Prompt; SR: Simple Rewrite; SI: Simple Iterative.

(Continued in Part 2/2: Q2–Q4.)


AuthorsOpenReview ↗

# Response to Reviewer sVfd (2/2)

Q2. Which components of RAPO, RAPO++, and VQQA were retained, and do the reproductions behave as intended?

Several methods do not release complete code, so we reproduced each one as closely as the paper and the available code allow. Section 6.2 and Appendix D.3 (Table 17) list all retained components and differences. We use RAPO's instruction-rewrite branch, RAPO++'s SSPO test-time stage, VQQA's core loop (question generation, question answering, and prompt refinement), without its original Global Selection or adaptive stopping, and VPO's released optimizer without retraining. Each iterative method uses one initial video and 19 updates (see W2 & Q4).

Two results suggest that the reproductions work as intended. VQQA and RAPO++ both outperform Simple Iterative, the naive loop they are designed to improve on (area sums +2.2 and −19.7 vs. −27.8; Table 3a). VPO improves human-judged visual quality (55.4% win rate; Table 4), which is its stated objective.

We are also running each reproduction in its original setting to check whether it recovers the reported trends. We will post these results during the discussion if they are ready, and otherwise include them in the final version. We will release all code, prompts, and videos.

We have also revised the text to state that our conclusions apply to these implementations with the listed differences.

Q3. Would simple resampling fail on broader or more constrained tasks?

We agree that our conclusion may not hold in such settings. Our study covers agents that act only through the text prompt and conceptual image input. Within this scope, the structured multi-shot request (W2) identifies a setting where prompt enhancement helps.

However, we agree that the resampling is likely to be less effective in two cases:

  1. Strongly constrained tasks, e.g., motion transfer from an untextured blockout or editing a masked region. Here an agent can provide control signals (pose, depth, masks, reference video) that resampling cannot.
  2. Systematic generator failure. Resampling cannot produce content that the generator never generates. A prompt-only agent has the same limitation.

These settings are outside our scope, as clarified in Section 3.1.

Q4. Is N = 20 fair to agents that may need more iterations?

Answered together with W2 (final output) in Part 1/2.

AuthorsOpenReview ↗

Follow-up on Our Response and Revised Manuscript

Dear Reviewer sVfd, Thank you again for your time and thoughtful feedback on our paper. We would like to gently follow up on our rebuttal and revised manuscript. When you have a chance, we would greatly appreciate your thoughts on whether the additional experiments and clarifications have addressed your concerns. Please let us know if any questions remain. We would be happy to provide further clarification. Best regards, The Authors

Reviewer_UrHoOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether video generation agents are truly superior to repeatedly sampling and selecting the best results using simple prompts, given a fixed video generation budget. The authors compared methods such as Simple Prompt, Human Prompt, Prompt Enhancement, and Iterative Prompt across up to six video generators. Each method generated 20 videos, which were evaluated based on action completion, entity and scene satisfaction, identity consistency, and visual quality. The experiments show that the advantages of existing agents typically appear only for specific generators or on a single metric, and are often accompanied by a decline in action completeness; iterative refinement also failed to demonstrate consistent improvement from round to round. Overall, in the complex reference-based video tasks used in this paper, existing video generation agents rarely outperform simple best-of-N sampling consistently under the same generation budget.

Critical strengths
  1. The research question is clear and has practical significance. The paper provides a fair comparison between video-generating agents and Simple Prompt’s best-of-N sampling under the same generation budget, highlighting key issues that existing evaluations may overlook.

  2. The experimental design is relatively comprehensive. In addition to existing prompt enhancement and iterative methods, the authors have incorporated Simple Rewrite, Simple Iterative, and Human Prompt, which help distinguish the additional benefits of complex agents from the effects of simple prompt rewriting or resampling.

  3. The evaluation evidence is relatively comprehensive. The authors examine semantic satisfaction, action completion, identity consistency, and visual quality simultaneously, and by combining automated scoring, blind human evaluations, iterative trajectories, and qualitative video case studies, they effectively reveal the trade-off between improved visual quality and reduced action semantics.

  4. Cross-generator analysis offers valuable insights. Experiments show that the benefits of prompt optimization are highly dependent on the specific video generator and are difficult to transfer consistently, providing valuable empirical insights into the limitations of existing video generation agents.

Critical weaknesses
  1. The experimental task is relatively simple. The main conclusions are drawn from a single scene featuring multiple characters, reference images, and long action sequences; while this illustrates the agent’s limitations on this specific task, it is not sufficient to support general conclusions about video-generating agents.

  2. The selection of the best video relies on the same set of automated evaluations. The GPT evaluator is responsible for both scoring the videos and selecting the “Maximum” and “Top-5” from among 20 candidates, which may amplify selection bias caused by evaluation noise. Although the paper provides human blind evaluations, it currently lacks an independent selector, confidence intervals, or more robust statistics for the overall candidate pool.

  3. The trajectories of the iterative methods lack sufficient repetition. Each method primarily shows one trajectory of 19 iterations for each generator; therefore, a single instance of random generation or incorrect feedback could affect all subsequent rounds. While the paper can support the claim that “these trajectories do not exhibit stable round-by-round improvement,” conclusions regarding the general behavior of iterative methods should still be drawn with greater caution.

Questions for authors
  1. Could the author provide additional information or discuss the results for different types of task, such as single person actions, short action sequences, or non-violent object interactions to determine whether the current conclusions are specific to this particular scenario?

  2. For the Maximum and Top-5 calculations, is it permissible to use a separate evaluation model or manual evaluation to select candidates, and to report the confidence interval, mean, and the proportion of cases in which all key actions were successfully completed?

3 comments
AuthorsOpenReview ↗

# Response to Reviewer UrHo

We thank the reviewer for the constructive feedback. We added four requests, a second evaluator, and repeated iterative trajectories. We answer Q1 and Q2 together with W1 and W2.

W1 & Q1. Single task and other task types

We added four requests varying action complexity, interaction type, and shot structure: placing a record (a short single-person action sequence), a non-violent notebook interaction, a structured multi-shot request, and a complex sporting action. All were fixed before the additional comparative experiments.

The findings are consistent across scenarios: in most evaluated settings, iterative methods do not substantially improve video quality, with prompt enhancement in the structured-prompt setting being the exception. On simple task a1 with Hunyuan-1.5, however, VPO's Maximum video still has a lower action score than Simple Prompt (8 vs. 10). The additional requests broaden the evidence (Section 5.4; Appendix C.1, Tables 6–9; task definitions in Appendix D.1.1; video examples in Appendix A).

The three-request average below shows that all three iterative methods have lower Area Sums than Simple Prompt at both Maximum and Top-5 Mean, while SCMAPR is near parity (+0.5 and +1.0).

Three-request average (a1, a2, c; Section 5.4, Table 5). For each generator, the corresponding selected dimension scores are averaged equally across the three requests before computing radar areas. SP gives absolute normalized areas (%); other entries are percentage-point differences from SP.

Maximum

DimensionSPHPSRRAPOSCMAPRVPOSIRAPO++VQQA
Entity/Obj.89.10.0-6.1-6.4-4.1-2.1-16.1-20.4-1.4
Action82.8-12.3+0.1-19.4+0.1-11.7-12.1-7.8-13.6
Ident./Appear.63.9+5.6+5.6+1.8+7.4-1.9-5.9-14.9+3.1
VADB Total34.1-4.8-2.5-3.6-3.0-1.9-3.4-3.7+0.4
Area Sum269.9-11.6-3.0-27.6+0.5-17.6-37.6-46.7-11.6

Top-5 Mean

DimensionSPHPSRRAPOSCMAPRVPOSIRAPO++VQQA
Entity/Obj.84.2-2.0-3.6-6.50.0-2.0-19.1-24.4-0.7
Action66.5-9.5-8.1-18.3+2.2-4.4-28.0-8.6-6.4
Ident./Appear.63.4+0.9+0.5-1.0+0.9-6.0-9.5-14.7-1.5
VADB Total31.9-3.4-1.3-2.2-2.1-1.1-0.8-2.1-1.5
Area Sum246.1-14.0-12.5-28.0+1.0-13.5-57.4-49.8-10.1

HP: Human Prompt; SR: Simple Rewrite; SI: Simple Iterative.

W2 & Q2. Evaluator dependence and additional statistics

Second-evaluator check. Claude opus-5.5 independently scores the same generated videos on the three open-source generators. We compare whole-pool distributions, including medians and middle-50% ranges. The broad patterns persist: Simple Iterative's median is below Simple Prompt in all tasks under both assessors (Section 5.4; Appendix C.2, Table 10).

Whole-pool statistics. We additionally report complete-action rates from Claude's five action checkpoints and run-level average scores for iterative methods. These additional statistics support the same overall finding: agents do not consistently outperform Simple Prompt. For example, on a1, RAPO completes all checkpoints in 65.0% of videos versus 85.0% for Simple Prompt, averaged over the three generators. (Appendix C.3, Table 11; iterative averages in Appendix C.7.2, Table 14).

Confidence intervals. We report pointwise 95% intervals for repeated-trajectory statistics over updates 1–19 and multiway-bootstrap intervals for human win rates. The latter appear in the existing main-text Table 4; Appendix E explains their calculation (Appendix C.7.2, Table 14; Section 5.2, Table 4; Appendix E). The best-versus-final gap persists across repeated runs: VQQA's original-task GPT mean Maximum is 27.86 versus 22.07 at the final update.

W3. One trajectory per iterative method

We added two trajectories for each iterative method on each open-source generator for the original request, giving three trajectories per condition. Across all GPT-scored trajectories (three methods × three generators × three runs), the final output is below the best output. We also repeated task a1 three times per condition: the final output is below the best in 26/27 GPT-scored trajectories and 27/27 Claude-scored trajectories; the remaining GPT trajectory ties its best at the final round. These results support the narrower observation that refinement does not reliably improve outputs round by round (Section 5.4; Appendix C.7.2, Table 14; trajectory plots in Appendix C.7).

Reviewer_UrHoOpenReview ↗

Official Review of Submission9300 by Reviewer UrHo

We thank the authors for the additional tasks, the second-evaluator analysis, and the repeated trajectories, which address part of my concerns. However, the Maximum and Top-5 candidates are still selected by the same GPT evaluator, and the repeated iterative runs (three per method) remain limited in number, so I will maintain my original rating of 2.

AuthorsOpenReview ↗

Follow-Up to Reviewer UrHo

Table 10 in the previous revision reported Claude opus-5.5 scores. To clarify, Claude independently scored all candidate videos, and the Claude-based Maximum and Top-5 Mean were selected entirely according to Claude scores, without using GPT-based candidate selection. To make the Claude results for the original task easier to inspect, we removed the side-by-side comparison with GPT from the revised Table 10. We also added two dedicated radar plots in Appendix F (Figures 28 and 29) showing the Claude-based Maximum, Top-5 Mean, and Final Round comparisons across all six generators.

We ran each iterative method three times on each of the three open-source generators. Assuming independent runs within each condition, averaging three runs reduces the sampling variance of the estimated mean by 66.7% relative to a single run. On the original task, the final round scored below the best round in all 27 GPT-scored trajectories. For VQQA, the mean within-run gap was 5.79 points (approximate 95% CI [3.21, 8.37]). This provides preliminary evidence that the gap between the best and final outputs persists across repeated runs under the tested conditions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Paper Revision

Dear Area Chair and Reviewers, Thank you for your feedback. Our additional experiments show where prompt enhancement helps and why its gains do not hold across tasks. Revisions are highlighted in orange. Main text

  • Section 5.4 adds four requests on three open-source generators. Averaged over a1, a2, and c, all three iterative methods score below Simple Prompt in total radar area for both Maximum and Top-5 Mean (Table 5). On the structured multi-shot request, all four prompt enhancers improve Top-5 total radar area (Table 8). (All reviewers)
  • Sections 3.2 and 5.1 clarify that feedback videos count toward the shared 20-video budget. Figure 3 compares each iterative method's best and final outputs. (sVfd)
  • Section 5.2, Table 4 adds confidence intervals for human win rates. Appendix E details the calculation. (UrHo)
  • Sections 3.1 and 6.2 limit our claims to prompt-level methods evaluated without generator-specific retraining. Table 17 documents the implementation differences. (sVfd, z4ts) Appendix
  • Appendices A, C.1 and D.1.1 give visual examples, per-request results, and the four prompts fixed before the experiments. (All reviewers)
  • Table 10 and Figures 28–29 (Appendix F) report independent Claude-based evaluations of all candidate videos, including Maximum, Top-5 Mean, and Final Round comparisons across six generators. Claude-based selections are made independently of GPT scores.
  • Table 11 reports complete-action rates. (UrHo)
  • Table 12 sums the four scores to remove dependence on radar-axis order. The six-generator ranking for best videos is unchanged. (z4ts)
  • Table 13 extends VQQA on Seedance-2.0-fast from 20 to 40 videos. Neither evaluator finds a higher best score after video 20. (sVfd)
  • Table 14 reports confidence intervals across repeated runs using updates 1–19. On the original request, the final video scores below the best in all 27 GPT-scored runs. (UrHo, z4ts) Ongoing work. We are checking the reproductions in their original settings and running a 10-rater study comparing each method's best video with Simple Prompt's best on the same generator. We will share results during discussion if ready, otherwise in the final version. We will release the code, prompts, and videos.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.