acceptodds
Under review as a conference paper at ICLR 2027

ReelBench: Towards Systematic Evaluation of Multi-Shot Narrative Video Generation under Image Reference

Abstract

Multi-shot narrative video generation increasingly relies on image references, including asset and storyboard images, to maintain visual and narrative consistency across independently generated shots. Yet existing benchmarks primarily assess final video quality or rank model capabilities, offering limited insight into how different forms of image conditioning affect generation. We introduce ReelBench, an evaluation framework and dataset for image-guided multi-shot narrative video generation. ReelBench comprises 300 story-style pairs with associated asset and storyboard images across three visual styles: Pixar-style animation, live-action realism, and 2D anime. It supports controlled comparisons among image reference conditions within the same story and evaluates generated videos along seven dimensions and 40 subdimensions: instruction alignment, character quality, visual quality, physical plausibility, visual continuity, cinematic quality, and narrative coherence. Our experiments show that stronger visual anchoring through first-frame conditioning does not consistently improve overall quality, and that reference complexity affects physical plausibility more strongly than overall scores. ReelBench agrees with human overall preferences in over 90% of pairwise comparisons.

Primary area:
datasets and benchmarks
Keywords:
multi-shot narrative video evaluation, visual reference evaluation, text-asset dataset, video generation benchmark, human preference alignment

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ck4POpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper constructs an image-reference benchmark for multi-shot narrative video generation, using three conditions to disentangle the effects of asset reference, storyboard reference, and storyboard-as-first-frame conditioning, and validates them with 7-dimension/40-subdimension automatic evaluation and human pairwise preference. Overall, the problem is clear, the data construction is complete, and the evaluation dimensions are solid. The benchmark claims to be meaningful and important, and it is important for community development; however, the conclusions are somewhat weak and do not provide actionable design guidance.

Critical strengths
  1. The problem is important and has long-term importance for the community.
  2. The data are rich and the evaluation dimensions are rich, covering three styles and multiple reference conditions, and preserving the complete pipeline for subsequent diagnosis and reuse; the shot-level and sequence-level stratification and the 7-dimension/40-subdimension evaluation provide good diagnostic granularity.
  3. The evaluation results are generally consistent with human evaluation.
Critical weaknesses
  1. The paper treats reference conditions as discrete categories (SA-R2V / A-R2V / I2V), but does not analyze the characteristics of the references themselves, such as reference quality, compatibility with the shot, and information type. It feels that this paper is not image-controlled but condition-controlled; the authors should make this more explicit, especially by highlighting the contrast in related work.
  2. Data construction and human evaluation both rely on AI/human hybrid pipelines, but the paper does not report screening criteria, screener backgrounds, screening consistency, or the proportion screened out, nor does it report inter-annotator agreement or annotator expertise for human evaluation. For a benchmark paper, this is basic information for usability and credibility.
  3. How are the 7 dimensions defined? How correlated are they? What is the basis for the weights?
  4. The reference conditions have unequal information content. Are there no reference-free or text-only baselines?
  5. Is "systematic" more like "comprehensive"? Where is the systematicity reflected?
  6. The generation configuration is not reported, e.g., how many videos are generated per shot per condition?
  7. The phrasing of the "stronger first-frame anchoring" conclusion is inconsistent. For example, the contribution section says: "Stronger first-frame anchoring does not reliably improve generation quality"; the conclusion says: "Our results show that stronger first-frame anchoring improves generation only when the underlying model can effectively continue from a static reference"; and Section 4.3 says: "neither stronger first-frame anchoring nor a larger number of references alone predicts generation quality".
  8. The n=100/99/99 in Table 8 needs to be explained relative to the "100-story controlled experiment".
Questions for authors
  1. The paper treats reference conditions as discrete categories (SA-R2V / A-R2V / I2V), but does not analyze the characteristics of the references themselves, such as reference quality, compatibility with the shot, and information type. It feels that this paper is not image-controlled but condition-controlled; the authors should make this more explicit, especially by highlighting the contrast in related work.
  2. Data construction and human evaluation both rely on AI/human hybrid pipelines, but the paper does not report screening criteria, screener backgrounds, screening consistency, or the proportion screened out, nor does it report inter-annotator agreement or annotator expertise for human evaluation. For a benchmark paper, this is basic information for usability and credibility.
  3. The reference conditions have unequal information content. Are there no reference-free or text-only baselines?
3 comments

Thank you for the thoughtful review and constructive feedback.

W1, Q1 (what is controlled). ReelBench controls the image reference input: as Section 4.2 states, it holds stories, shot descriptions, video model and protocol fixed across 100 stories while varying only the image reference input format (SA-R2V, A-R2V, I2V); the abstract likewise describes controlled comparisons among image reference conditions within a story, so the three conditions are defined by which images are supplied. On the three properties you list: information type is what the three conditions vary; quality is examined in Appendix E, where raising storyboard quality by +3.64 transfers negatively to video (-2.23, p=1.2e-9; -0.57, p=2.5e-4); shot compatibility is secured at construction, where shot descriptions were manually reviewed for executability (Section 3.1) and asset/storyboard usability was checked (see W2/Q2). Appendix F adds an observational stratification by asset count (1–2 vs 7–10).

W2, Q2 (screening and human evaluation protocols). Inter-annotator reliability and multi-stage data screening: Following Appendix B.3, each pair is judged blindly—hiding model/condition labels and randomizing left-right order after practice calibration—by 3 independent domain-trained researchers in generative vision and AI, achieving high consensus across all 260 valid pairs (80 controlled, 180 cross-model):

Metric (260 valid pairs, 3 raters/pair)Majority AgreementPairwise agreementUnanimous (3/3)Fleiss' kappa
Overall human preference94.1%88.2%82.3% (214/260)0.773

In dataset construction, two experienced narrative-video creators jointly screened and refined the initial corpus along three criteria: narrative completeness, requiring closed event progression and resolution; causal coherence, requiring clear motivation and cross-shot setup; and visual expressibility, grounding key plot points in observable actions, interactions, and state changes, alongside asset/storyboard usability checks. Of 100 initial stories, 85% were retained with shot-level refinement and 15% were replaced during quality screening and genre balancing, expanding 500 draft shots to 642 base shots (1,926 across the three visual styles; Section 3.1).

W4, Q3 (information content & text-only baseline). Reference-free T2V baseline under matched shot scripts: All conditions share identical shot action and camera scripts with minimal interface-specific reference tags, varying only pixel-level visual references to reflect deployment choices; nonetheless, following your suggestion and Reviewer Kvro's, we added text-only T2V and storyboard-only R2V (S-R2V) on the 20-story set (Vidu Q2):

Condition (n=20)SA-R2VA-R2VS-R2VI2VT2V (no ref)
Overall70.1567.2567.0166.4466.04

T2V ranks lowest overall (trailing SA-R2V by 4.10, p=6.8e-4, 18/20 stories) and ties single-reference modes: without 2D constraints, T2V yields freer motion, but suffers severe identity drift and cross-shot discontinuity, confirming pixel references are essential for identity preservation.

W3 (dimensions, correlations, and weight robustness). Low inter-dimension redundancy and complete weight invariance: The 7 dimensions and 40 sub-items (Section 3.3, Appendix A) show moderate story-level Pearson correlations (mean r=0.38, 0.17 [PP–NC] to 0.69 [CQ–VC]; IA–CQ 0.42, VQ–PP 0.34). As stated in Appendix A.2, six core dimensions carry equal weight (0.15), while Cinematic Quality is 0.10 due to partial framing/lighting overlap with VQ; replacing them with uniform 1/7 weights leaves all results unchanged:

Weighting SchemePairwise flips (n=260)Human agreementControlled SA / A / I2V
Paper weights (6x0.15 + 0.10)—92.7% (241/260)71.29 / 67.34 / 67.01
Uniform weights (1/7 each)0 / 260 (0.0%)92.7% (241/260)71.54 / 67.63 / 67.36

W5–W8 (systematicity, config, phrasing, n=99). Systematicity: Factorial single-variable control—reference mode and storyboard quality varied one at a time under fixed scripts. Generation config: 1 video/shot/condition; each shot is a 5.0 s clip (150 frames at 30 fps or 120 at 24 fps, native 1920x1080 or 1280x720 per interface, judged at 4 fps; Sections 3.1, 4.1, Appendix A.2). Phrasing: Sections 1, 4.3, and 5 will unify to: the benefit of first-frame anchoring is bounded by the model's temporal continuation capability from a static reference. n=100/99/99: All 100 stories have full scripts and references; 1 story's I2V clip was blocked by platform safety moderation while SA-R2V and A-R2V succeeded, which is why the I2V rows of Table 8 carry n=99.

Reviewer_ck4POpenReview ↗

Maintain my opinion

Thank you very much for your detailed response. I have learned a lot, but my concern is still that the Image Reference is not clear enough. I still maintain my rating.

Thank you for taking the time to carefully consider our response. We will clarify the scope of image reference in the Introduction, and we sincerely appreciate your feedback.

Reviewer_qbRVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReelBench evaluates image-guided multi-shot narrative video generation using 300 story-style pairs and 1,926 shots with aligned scripts, assets, and storyboards. It compares asset references, combined asset and storyboard references, and storyboard-based I2V. A VLM scores seven dimensions at shot and adjacent-shot levels. Experiments report model-dependent reference benefits, limited transfer from static storyboard optimization, and stronger associations between asset complexity and physical errors. Validation includes human preferences, paired tests, repeated scoring, and alternative evaluators.

Critical strengths
  1. The dataset aligns stories, shot descriptions, assets, storyboards, and generated videos across three styles. This enables comparisons within the same narrative and provides a useful resource for studying reference design beyond model rankings.
  2. The Vidu Q2 comparison uses matched stories with paired tests, confidence intervals, and effect sizes. Cross-model results reveal that reference benefits vary by model, while repeated scoring and evaluator replacement provide initial robustness evidence.
Critical weaknesses
  1. The Visual Continuity prompt explicitly excludes cross-shot consistency and instructs the evaluator to ignore differences in character appearance, clothing, and scene layout. It largely measures visual appeal across two shots. Although Character Quality checks reference identity, the framework does not clearly establish systematic coverage of cross-shot identity and object-state continuity. Moreover, all sequence-level tasks see only adjacent pairs, limiting direct assessment of long-range narrative dependencies. These gaps affect the validity of a benchmark whose central purpose is multi-shot evaluation.
  2. The reported 95.6% overall agreement is computed on the non-tied subset of 160 video pairs; the 3,840 judgments also include multiple dimensions and annotators. The tie threshold, valid counts, and agreement by score gap or comparison type are not reported. More importantly, there is no comparison with direct holistic scoring by the same VLM. Evaluator replacement supports robustness across judges, but does not establish the benefit of dimensional decomposition and aggregation. Overall preference agreement also does not validate the claimed fine-grained diagnoses.
Questions for authors
  1. Which metrics explicitly penalize cross-shot identity, clothing, prop-state, and scene inconsistencies, given the exclusions in Figure 13? Can you provide targeted tests of these errors, including inconsistencies spanning nonadjacent shots?
  2. With the same VLM and comparable input budget, how does ReelBench compare with direct holistic scoring in human agreement and error diagnosis? This comparison would help establish whether the proposed decomposition is necessary.
  3. What tie threshold was used, and what are the valid pair counts, tie rates, and confidence intervals? Please report agreement separately for the 40 controlled-reference pairs and for small-score-gap comparisons.
3 comments

Thank you for the rigorous and constructive review.

W1, Q1 (which metrics penalize cross-shot inconsistencies, and tests across nonadjacent shots). Metrics explicitly penalizing identity, clothing, prop-state, and scene inconsistencies: Cross-shot identity, clothing, and prop appearance errors are explicitly penalized in Character Quality (VLM-B, Appendix A.2, Figure 9) by sub-items B1 (identity) and B3 (costume & props), while script-specified scene and prop-state transitions are penalized per shot by Instruction Alignment (Figure 8). Because every shot in a story is generated from and evaluated against the same shared character and prop asset sheets, any appearance drift in Shot k directly lowers B1 and B3 in that shot. Visual Continuity (Figure 13) evaluates perceptual cut smoothness across shots.

Targeted multi-condition test and sub-item attribution: Across the 20-story multi-condition set (Vidu Q2, evaluated with identical asset sheets passed to VLM-B):

Metric (n=20)SA-R2VA-R2VI2VT2V (no ref)SA vs. ASA vs. I2V
B1 Identity76.471.666.443.7+4.8 (p=0.013)+10.0 (p=2.6e-5)
B3 Costume & Props75.566.664.342.7+8.9 (p=1.2e-5)+11.2 (p=1.4e-6)
CQ Composite75.470.867.662.0+4.6 (p=9.7e-4)+7.8 (p=8.4e-6)

Of the seven CQ sub-items (B1–B7, where B2/B4–B7 assess single-shot anatomy, expression, prominence, gaze, and rendering), the SA-R2V vs. A-R2V gain concentrates on B3 (+8.9) and B1 (+4.8) while anchor-independent items barely shift (B5 +2.3, B6 +1.6). Under storyboard-only R2V (S-R2V), B1 reaches 70.8 but B3 drops to 58.5, confirming that asset sheets specifically govern clothing and prop-state fidelity.

Inconsistencies spanning nonadjacent shots: Because B1/B3 score every shot against the same story-level anchor, within-story standard deviation across all shots in a story—spanning nonadjacent pairs such as Shot 2 vs. Shot 7—reflects cross-shot drift. Across all 100 controlled stories, within-story CQ SD drops from 13.60 (I2V) and 10.32 (A-R2V) to 8.72 (SA-R2V) (paired p=1.3e-14); on the 20-story set, within-story B3 SD likewise drops from 17.13 (I2V) and 15.50 (A-R2V) to 11.71 (SA-R2V) (p<0.01).

W2, Q3 (tie threshold, valid counts, CIs, and subset/gap breakdown). Unified tie protocol and expanded stratified agreement: Paper and expanded study share one protocol: each pair is judged by 3 independent annotators on Overall and 7 dimensions; automatic ties are set at Overall gap epsilon=1 (calibrated on a separate sub-1-point pool, last row of the table), and 3-rater "Similar" majorities or 1-1-1 splits are excluded as human ties/splits. The resulting valid counts, tie rates, and Wilson CIs by subset and score gap:

Subset (Overall)Valid pairs3-vote totalAuto-tie (<1)Human-tie/splitAgreement95% Wilson CI
Controlled (ref. modes)80841388.8% (71/80)[80.0%, 94.0%]
Cross-model18019613394.4% (170/180)[90.1%, 97.0%]
Pooled total26028014692.7% (241/260)[88.9%, 95.3%]
└ gap [1, 2)910—177.8%[45.3, 93.7]
└ gap [2, 5)121123—289.3%[82.5, 93.6]
└ gap >=5130133—396.9%[92.4, 98.8]
gap <1 (separate calibration pool)1214—266.7% (8/12)[39.1%, 86.2%]

The gap rows split the 266 pooled pairs with gap >=1; the other 14 are automatic ties. The last row is the separate sub-1-point pool that set epsilon=1 and is not part of the 260.

Q2 (direct holistic scoring under matched VLM and input budget). Matched-budget holistic comparison: Following your suggestion, we evaluated the same 260 pairs with qwen3.5-plus in single-pass holistic mode under an identical input budget—feeding the full concatenated story video sampled at the same 4 fps, the same character/scene asset sheets, and all shot scripts:

Scoring Mode (qwen3.5-plus, matched inputs)Valid comparableHolistic tiesControlledCross-modelPooled [95% CI]McNemar p
ReelBench 7-dim decomposition223089.2%94.9%93.3% [89.2, 95.9]5.6e-10
Direct holistic VLM scoring22335 (13.6%)70.8%72.2%71.7% [65.5, 77.3]—

Of the 260 pairs, 2 could not be scored (platform moderation) and 35 of the other 258 were holistic ties, leaving 223 comparable pairs (ReelBench: 93.3% here vs. 92.7% on all 260).

Holistic scoring drops human alignment by 21.6 percentage points, produces undecidable equal scores on 13.6% of pairs, and yields one opaque scalar, whereas ReelBench's 7 dimensions each align with human preferences (77.0%–90.5%, mean r=0.38) to localize errors.

I have carefully read the authors' rebuttal. The authors have provided extensive additional experimental results, which have substantially alleviated several of my previous concerns. I have also read the comments from the other reviewers. Taking all of these into consideration, I am willing to raise my score from 2 to 3. I have no further questions or concerns at this point.

Thank you for taking the time to carefully review our rebuttal and additional experiments. We sincerely appreciate your constructive feedback and are grateful for your reconsideration and support.

Reviewer_z6XFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReelBench, an evaluation framework and dataset for image-guided multi-shot narrative video generation. The authors construct 300 story-style pairs with corresponding asset images, storyboard images, and generated videos across three visual styles. ReelBench evaluates generated videos along seven dimensions and 40 subdimensions at both shot and sequence levels. The authors compare three image reference conditions: asset-only (A-R2V), storyboard-and-asset (SA-R2V), and image-to-video (I2V). Experiments across multiple video generation models demonstrate that combining storyboard and asset references can improve generation quality, while storyboard-based I2V does not consistently outperform other reference conditions. The proposed evaluation framework also shows high agreement with human preferences.

Critical strengths
  • I think evaluating the effects of different image reference conditions on multi-shot narrative video generation is an interesting and practical direction. Comparing different reference conditions using the same stories and shot descriptions provides useful insights into how asset and storyboard references affect generation quality.

  • The construction of a multi-shot video benchmark with aligned asset images, storyboards, and generated videos across three visual styles is a valuable contribution. Unlike existing multi-shot benchmarks, ReelBench enables controlled comparisons of different image reference conditions within the same narrative.

  • The authors provide extensive experiments across multiple video generation models and reference conditions. The human preference evaluation and storyboard optimization experiments also offer interesting observations about the limitations of static image references.

Critical weaknesses
  • The effect of first-frame conditioning is not fully isolated. While the comparison between SA-R2V and A-R2V reasonably demonstrates the benefit of adding storyboard references, I2V differs from the other settings in both the available reference information and the generation interface. In particular, I2V uses a storyboard as either the first-frame condition or a strong visual reference, depending on the model. Therefore, the results do not fully isolate the effect of first-frame conditioning itself. Additional controlled comparisons would help support the conclusions regarding the effectiveness of different reference conditions.

  • The human preference validation could be more thoroughly reported. The authors report 95.6% agreement with human preferences, but only 40 of the 160 evaluated video pairs come from the controlled reference-condition experiment. Furthermore, automatic and human ties are excluded from the agreement calculation, while the number of valid pairs is not reported. Although the evaluator replacement experiments demonstrate promising robustness, reporting agreement separately for reference-condition comparisons and different score-gap ranges would provide stronger evidence for the reliability of the proposed evaluation framework.

Questions for authors
  • Could the authors clarify how I2V is implemented for each evaluated video generation model? In particular, does the storyboard serve as the exact first frame or as a reference image?

  • Since I2V differs from A-R2V and SA-R2V in both the available reference information and the generation interface, could the authors clarify how the effect of first-frame conditioning can be isolated? A more controlled comparison would help support the conclusions regarding different reference conditions.

  • Could the authors report the human preference agreement separately for the controlled reference-condition experiment and the cross-model comparisons? It would also be helpful to report the number of valid pairs after excluding automatic and human ties.

3 comments

Thank you for the constructive review and insightful suggestions.

W1, Q2 (isolating first-frame conditioning). Direct first-frame isolation via storyboard-only R2V (S-R2V): Within the same R2V interface, SA-R2V vs. A-R2V isolates the benefit of adding storyboard composition (+2.89, 95% CI [2.36, 3.43], dz=1.07). First-frame I2V APIs do not accept auxiliary asset sheets, and in production a storyboard is supplied to R2V alongside asset sheets rather than alone, since R2V treats it as a soft global reference rather than a start frame. Nonetheless, following your suggestion, we ran S-R2V on the 20-story set (Vidu Q2) to compare directly against I2V under the identical single storyboard image:

Condition (Vidu Q2, n=20)SA-R2VA-R2VS-R2VI2V
Overall70.1567.2567.0166.44
Frame-0 median SSIM vs. storyboard0.5980.5480.5580.937
Paired diff vs. I2V+3.71 (p=1.1e-4)+0.81 (p=0.31)+0.57 (p=0.51)—

Soft-reference behavior and complementary gains vs. losses: Frame-0 SSIM against the input storyboard falls below 0.9 in 123 of 124 S-R2V clips (median 0.558 vs. 0.937 in I2V), confirming that S-R2V acts as a soft global reference like SA-R2V rather than locking frame 0. Comparing S-R2V with I2V under the identical storyboard reveals opposing trade-offs: locking frame 0 in I2V enforces exact initial layout and boosts Instruction Alignment (+9.3), whereas releasing that rigid lock in S-R2V allows freer motion, boosting Visual Quality (+7.4) and Physical Plausibility (+2.5). Because these gains and losses cancel out in Overall (+0.57, p=0.51), first-frame locking alone yields no net overall gain; true improvements come from combining storyboard composition with character asset sheets (SA-R2V).

Q1 (per-model I2V implementation). Strong initial-frame anchoring across all evaluated I2V interfaces relative to R2V: Across our benchmark, SA-R2V does not anchor frame 0 to the storyboard (median frame-0 SSIM 0.598), whereas every evaluated I2V interface in Table 3 invokes the model's dedicated first-frame mode and exhibits substantially stronger frame-0 fidelity (median SSIM 0.706–0.937):

Evaluated I2V Interface (Table 3)ClipsMedian SSIMSSIM > 0.9Frame-0 Behavior
Vidu Q2 (img2video, controlled / cross)128 / 3840.937 / 0.93396.9% / 89.3%Pixel-exact start frame
Vidu Q3 (img2video, start frame)3840.92983.3%Pixel-exact start frame
LTX-Video 2.3 (first-frame I2V)3840.93185.9%Pixel-exact start frame
Wan2.2-TI2V-5B (first-frame TI2V)3840.7490.5%Strong layout anchor with redraw
Seedance 2.0 (role=first_frame)340.7060%Strong layout anchor with redraw

All evaluated I2V interfaces enforce strong start-frame conditioning compared with R2V, with Vidu Q2/Q3 and LTX-2.3 preserving frame 0 at the pixel level and Wan2.2/Seedance 2.0 preserving layout while lightly redrawing pixels. We will add this table to Appendix C and clarify Section 4.1.

W2, Q3 (human preference agreement by subset and score gap). Threshold calibration and expanded stratified human agreement: Threshold calibration used a separate set of 36 sub-1-point pairs outside the 280 (12 with a 3-rater directional majority): human-model agreement was 66.7% [39.1%, 86.2%], confirming that differences below 1 point are perceptually ambiguous for both humans and VLMs; we therefore set the automatic tie threshold at epsilon=1, while counting 3-rater "Similar" majorities or 1-1-1 splits separately as human ties/splits. Following reviewer feedback, we expanded the blind study to 80 controlled and 180 cross-model valid pairs:

Subset (Overall)3-vote pairsAuto-tie (<1)Human-tie/splitValid pairsAgreement95% Wilson CI
Controlled (ref. conditions)84138088.8% (71/80)[80.0%, 94.0%]
Cross-model19613318094.4% (170/180)[90.1%, 97.0%]
Pooled total28014626092.7% (241/260)[88.9%, 95.3%]
└ gap [1, 2)10—1977.8% (7/9)[45.3%, 93.7%]
└ gap [2, 5)123—212189.3% (108/121)[82.5%, 93.6%]
└ gap >= 5133—313096.9% (126/130)[92.4%, 98.8%]

The three gap rows split the 266 pooled pairs with gap >=1 (the other 14 are automatic ties). The sub-1-point calibration pool is separate and outside the 280: 36 pairs had votes, 12 with a 3-rater directional majority, agreement 66.7% (8/12, [39.1%, 86.2%]) versus 93.2% for gap >=2, which is how we chose epsilon=1; these pairs are not added to the 260.

Thank you for the clarifications and additional experiments, which resolve most of my concerns. I appreciate the benchmark’s focus on a practical problem and the value of its findings within the evaluated configurations. However, the multiple reference conditions and model-specific interfaces make the setup relatively complex, and it remains unclear how broadly the findings generalize beyond these configurations. Overall, I maintain my score.

Thank you for your follow-up and for carefully considering our clarifications and additional experiments. We are glad that our responses helped address most of your concerns.

Reviewer_KvroOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

ReelBench’s genuinely new contribution is a parallel-condition dataset for multi-shot narrative video. The same 100 stories and 642 shots, with fixed shot text, come with matched asset images and storyboards in three visual styles, plus generated videos under three reference conditions (assets only, assets plus storyboard, storyboard as first frame). This turns image references from a fixed input into a controlled experimental variable, which prior benchmarks like MSVBench, ViStoryBench and UniVBench did not offer: they rank models, while ReelBench lets you ask which references help, for which model, in which style. It also treats storyboard quality as an upstream variable, and shows that improving a storyboard’s static quality does not reliably improve the video. That points to a gap: the field lacks references and metrics that encode motion, contact and state transitions rather than just appearance and composition.

Critical strengths
  1. The research question is practically relevant: which references to feed a shot-by-shot pipeline are a real decision that practitioners face. A parallel-condition dataset is a useful resource that doesn’t currently exist.

  2. The statistics are careful, such as paired tests with BH correction, effect sizes and CIs, repeat runs, evaluator swaps, and an openly released dataset and code.

  3. Two findings are genuinely useful. Storyboards mainly rescue the worst shots, and higher storyboard static quality does not translate to video quality.

Critical weaknesses
  1. The controlled study isn’t cleanly controlled, so the headline claim isn’t isolated. There is no text-only baseline. SA-R2V vs I2V changes two factors at once: assets present and storyboard role. The I2V and R2V conditions use different API endpoints. Only off-the-shelf models are tested.

  2. Weak evidence for the most interesting claims. Main effects of about 3 points sit near the judge’s run-to-run noise (MAE about 1.4–1.8). The storyboard-optimization result has no significance test. The asset-count analysis is confounded with scene complexity and rests on 8–12 stories per extreme bin. The human validation of the condition comparisons uses only 40 pairs, and the agreement rate excludes ties.

  3. The conclusion overclaims. It says first-frame anchoring helps “only when the model can effectively continue from a static reference,” but no model in the study shows I2V winning, and “continuation ability” is never measured.

  4. The metrics don’t match their names, which undercuts a multi-shot benchmark. Visual Continuity explicitly ignores cross-shot consistency. Narrative Coherence only sees adjacent shot pairs. So the core multi-shot property, identity and state continuity across the story, is not measured directly.

  5. The novelty is overstated relative to MSVBench. MSVBench already reported that image references can hurt physical plausibility compared with T2V. ReelBench’s Table 1 also misrepresents MSVBench’s reference controls.

Questions for authors
  1. Missing baseline: Can you please add a text-only (T2V) condition?

  2. Confounded conditions: SA-R2V vs I2V changes both asset presence and storyboard role, via different API endpoints. Can you add “I2V + assets” or storyboard-only R2V to isolate first-frame anchoring?

  3. Cross-shot consistency: Visual Continuity explicitly ignores cross-shot appearance changes, and Narrative Coherence sees only adjacent pairs. How does ReelBench measure identity and state consistency across a full story?

  4. Novelty and scope: MSVBench already reported that I2V references can hurt physical plausibility, and your anchoring claim is tested only on off-the-shelf APIs. What is new beyond MSVBench, and would the finding hold for a storyboard-trained model like STAGE?

3 comments

Thank you for the detailed review and constructive suggestions.

W1, Q1, Q2 (baselines and condition isolation). Scope and five-condition controlled study: ReelBench compares image-reference conditioning paradigms; text prompts alone cannot specify pixel-exact character or scene appearance, making asset sheets (A-R2V) or storyboards the minimal visual anchors (stated in Section 4.1), while first-frame I2V APIs do not accept auxiliary assets. Nonetheless, following your suggestion, we added text-only T2V and storyboard-only R2V (S-R2V) on the 20-story set with Vidu Q2:

Condition (n=20)SA-R2VA-R2VS-R2VI2VT2V
Overall70.1567.2567.0166.4466.04
Paired vs. SA-R2V—-2.89 (p=1.1e-4)-3.14 (p=6.9e-4)-3.71 (p=1.1e-4)-4.10 (p=6.8e-4)

First-frame trade-off (S-R2V vs. I2V): Both receive the identical storyboard, yet in 123/124 clips S-R2V does not place it at frame 0 (frame-0 SSIM <= 0.9, median 0.558 vs. 0.937 in I2V), acting as a soft guide. Locking frame 0 in I2V enforces initial layout and raises Instruction Alignment (+9.3 over S-R2V), whereas releasing that rigid lock in S-R2V yields freer motion, raising Visual Quality (+7.4) and Physical Plausibility (+2.5); these offset in Overall (+0.57, p=0.51), consistent in magnitude and direction with the 100-story A-R2V vs. I2V difference (+0.54, p=0.054). Complementarity within R2V: SA-R2V beats both A-R2V and S-R2V under the same R2V endpoint, confirming composition and character sheets are jointly required. Why text-only T2V falls short: T2V ranks lowest overall (trailing SA-R2V in 18/20 stories); freed from 2D constraints, it yields smoother single-shot motion but suffers severe identity drift and cross-shot discontinuity, so text alone cannot preserve appearance across shots.

Q3, W4 (cross-shot consistency). Unified input-anchor measurement across the full story: All shots share one unified anchor—the pre-generated asset sheets and shot scripts. Character Quality checks every shot against these shared sheets via B1 (identity) and B3 (costume/props) (Appendix A.2, Figure 9), penalizing drift regardless of shot distance; across 100 stories, this cuts within-story CQ dispersion from 13.60 (I2V) to 8.72 (SA-R2V, p=1.3e-14).

W2 (strength of evidence). Statistically decisive main effects: Both 95% CIs lie entirely above zero—SA-R2V beats A-R2V by +2.89 ([2.36, 3.43]) and I2V by +3.43 ([2.84, 4.02]); averaged over 100 paired stories, the retest noise of a single video (MAE 1.4–1.8) shrinks well below these gaps. Significant negative transfer from storyboard optimization: Paired t-tests on Table 16 (Appendix E) confirm significant drops under SA-R2V (-2.23, p=1.2e-9, n=100) and I2V (-0.57, p=2.5e-4, n=99). Asset count as a load measure: Each shot depicts exactly one scene (Section 4.3), so the scene is held fixed and asset count varies the character and prop load rather than scene diversity; from 1–2 to 7–10 assets, PP drops 6.51 ([-9.40, -3.50]), 5.67 ([-8.31, -3.09]), and 4.84 ([-8.52, -1.15]). Expanded human validation: Blind 3-rater evaluation covers 80 controlled and 180 cross-model valid pairs (epsilon=1), reaching 88.8% [80.0, 94.0] and 94.4% [90.1, 97.0] agreement (pooled 92.7%).

W3 (overclaim). Direct test of continuation and unified phrasing: In Appendix E (Sections 4.2–4.3), raising I2V first-frame static quality by +3.64 yields -0.57 on video Overall (p=2.5e-4): a better static frame did not yield a better video, which we attribute to post-frame motion continuation; we will align all sections to: the benefit of first-frame anchoring is bounded by the model's temporal continuation capability from a static reference.

W5, Q4 (novelty vs. MSVBench and STAGE). Controlled isolation beyond system ranking: PP is our lowest dimension across all model-reference pairs and drops ~3x faster than Overall as asset count grows (Section 4.3; Appendix F, Table 18). Unlike MSVBench, which evaluates multi-shot generation broadly across systems, ReelBench fixes stories and the backbone to isolate how the reference form affects each quality dimension. Our Table 1 legend classified references by input type, making multi-setting benchmarks read as single-setting; we revised the legend and corrected the MSVBench entry. STAGE: STAGE feeds the start-end frames predicted by its trained STEP2 into an off-the-shelf video model, the interface type ReelBench evaluates, so our findings on image-reference conditioning bear on that generation step; ReelBench's parallel references can also be used to evaluate such storyboard-trained pipelines.

Reviewer_KvroOpenReview ↗

Thank the authors for their efforts!

Thank you so much for your substantial efforts. I have read your rebuttal in detail. I appreciate your efforts, and I don't have any further questions.

I will engage in the AC-Reviewers discussion period to fairly assess your work, taking into consideration the other reviewers' concerns.

Regarding the score and assessment, I will determine my final score after the AC-Reviewers' discussion period.

Thanks again for your keen efforts!

Thank you for carefully reviewing our rebuttal. We sincerely appreciate your thoughtful consideration and continued attention to our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.