- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper constructs an image-reference benchmark for multi-shot narrative video generation, using three conditions to disentangle the effects of asset reference, storyboard reference, and storyboard-as-first-frame conditioning, and validates them with 7-dimension/40-subdimension automatic evaluation and human pairwise preference. Overall, the problem is clear, the data construction is complete, and the evaluation dimensions are solid. The benchmark claims to be meaningful and important, and it is important for community development; however, the conclusions are somewhat weak and do not provide actionable design guidance.
Critical strengths
- The problem is important and has long-term importance for the community.
- The data are rich and the evaluation dimensions are rich, covering three styles and multiple reference conditions, and preserving the complete pipeline for subsequent diagnosis and reuse; the shot-level and sequence-level stratification and the 7-dimension/40-subdimension evaluation provide good diagnostic granularity.
- The evaluation results are generally consistent with human evaluation.
Critical weaknesses
- The paper treats reference conditions as discrete categories (SA-R2V / A-R2V / I2V), but does not analyze the characteristics of the references themselves, such as reference quality, compatibility with the shot, and information type. It feels that this paper is not image-controlled but condition-controlled; the authors should make this more explicit, especially by highlighting the contrast in related work.
- Data construction and human evaluation both rely on AI/human hybrid pipelines, but the paper does not report screening criteria, screener backgrounds, screening consistency, or the proportion screened out, nor does it report inter-annotator agreement or annotator expertise for human evaluation. For a benchmark paper, this is basic information for usability and credibility.
- How are the 7 dimensions defined? How correlated are they? What is the basis for the weights?
- The reference conditions have unequal information content. Are there no reference-free or text-only baselines?
- Is "systematic" more like "comprehensive"? Where is the systematicity reflected?
- The generation configuration is not reported, e.g., how many videos are generated per shot per condition?
- The phrasing of the "stronger first-frame anchoring" conclusion is inconsistent. For example, the contribution section says: "Stronger first-frame anchoring does not reliably improve generation quality"; the conclusion says: "Our results show that stronger first-frame anchoring improves generation only when the underlying model can effectively continue from a static reference"; and Section 4.3 says: "neither stronger first-frame anchoring nor a larger number of references alone predicts generation quality".
- The n=100/99/99 in Table 8 needs to be explained relative to the "100-story controlled experiment".
Questions for authors
- The paper treats reference conditions as discrete categories (SA-R2V / A-R2V / I2V), but does not analyze the characteristics of the references themselves, such as reference quality, compatibility with the shot, and information type. It feels that this paper is not image-controlled but condition-controlled; the authors should make this more explicit, especially by highlighting the contrast in related work.
- Data construction and human evaluation both rely on AI/human hybrid pipelines, but the paper does not report screening criteria, screener backgrounds, screening consistency, or the proportion screened out, nor does it report inter-annotator agreement or annotator expertise for human evaluation. For a benchmark paper, this is basic information for usability and credibility.
- The reference conditions have unequal information content. Are there no reference-free or text-only baselines?
3 comments
Maintain my opinion
Thank you very much for your detailed response. I have learned a lot, but my concern is still that the Image Reference is not clear enough. I still maintain my rating.
Thank you for taking the time to carefully consider our response. We will clarify the scope of image reference in the Introduction, and we sincerely appreciate your feedback.
Thank you for the thoughtful review and constructive feedback.
W1, Q1 (what is controlled). ReelBench controls the image reference input: as Section 4.2 states, it holds stories, shot descriptions, video model and protocol fixed across 100 stories while varying only the image reference input format (SA-R2V, A-R2V, I2V); the abstract likewise describes controlled comparisons among image reference conditions within a story, so the three conditions are defined by which images are supplied. On the three properties you list: information type is what the three conditions vary; quality is examined in Appendix E, where raising storyboard quality by +3.64 transfers negatively to video (-2.23, p=1.2e-9; -0.57, p=2.5e-4); shot compatibility is secured at construction, where shot descriptions were manually reviewed for executability (Section 3.1) and asset/storyboard usability was checked (see W2/Q2). Appendix F adds an observational stratification by asset count (1–2 vs 7–10).
W2, Q2 (screening and human evaluation protocols). Inter-annotator reliability and multi-stage data screening: Following Appendix B.3, each pair is judged blindly—hiding model/condition labels and randomizing left-right order after practice calibration—by 3 independent domain-trained researchers in generative vision and AI, achieving high consensus across all 260 valid pairs (80 controlled, 180 cross-model):
In dataset construction, two experienced narrative-video creators jointly screened and refined the initial corpus along three criteria: narrative completeness, requiring closed event progression and resolution; causal coherence, requiring clear motivation and cross-shot setup; and visual expressibility, grounding key plot points in observable actions, interactions, and state changes, alongside asset/storyboard usability checks. Of 100 initial stories, 85% were retained with shot-level refinement and 15% were replaced during quality screening and genre balancing, expanding 500 draft shots to 642 base shots (1,926 across the three visual styles; Section 3.1).
W4, Q3 (information content & text-only baseline). Reference-free T2V baseline under matched shot scripts: All conditions share identical shot action and camera scripts with minimal interface-specific reference tags, varying only pixel-level visual references to reflect deployment choices; nonetheless, following your suggestion and Reviewer Kvro's, we added text-only T2V and storyboard-only R2V (S-R2V) on the 20-story set (Vidu Q2):
T2V ranks lowest overall (trailing SA-R2V by 4.10, p=6.8e-4, 18/20 stories) and ties single-reference modes: without 2D constraints, T2V yields freer motion, but suffers severe identity drift and cross-shot discontinuity, confirming pixel references are essential for identity preservation.
W3 (dimensions, correlations, and weight robustness). Low inter-dimension redundancy and complete weight invariance: The 7 dimensions and 40 sub-items (Section 3.3, Appendix A) show moderate story-level Pearson correlations (mean r=0.38, 0.17 [PP–NC] to 0.69 [CQ–VC]; IA–CQ 0.42, VQ–PP 0.34). As stated in Appendix A.2, six core dimensions carry equal weight (0.15), while Cinematic Quality is 0.10 due to partial framing/lighting overlap with VQ; replacing them with uniform 1/7 weights leaves all results unchanged:
W5–W8 (systematicity, config, phrasing, n=99). Systematicity: Factorial single-variable control—reference mode and storyboard quality varied one at a time under fixed scripts. Generation config: 1 video/shot/condition; each shot is a 5.0 s clip (150 frames at 30 fps or 120 at 24 fps, native 1920x1080 or 1280x720 per interface, judged at 4 fps; Sections 3.1, 4.1, Appendix A.2). Phrasing: Sections 1, 4.3, and 5 will unify to: the benefit of first-frame anchoring is bounded by the model's temporal continuation capability from a static reference. n=100/99/99: All 100 stories have full scripts and references; 1 story's I2V clip was blocked by platform safety moderation while SA-R2V and A-R2V succeeded, which is why the I2V rows of Table 8 carry n=99.