acceptodds
Under review as a conference paper at ICLR 2027

Interactive Video Storytelling with Contextual Grounding

Abstract

Interactive video storytelling enables viewers to actively steer how a video unfolds. However, once we allow viewers to intervene during generation, a new challenge arises: The viewer's request can have latent dependencies on both the grounding source and the current rendered video state. These dependencies may not be explicitly stated in any individual input, but emerge only when the source, rendered history, and new viewer intent are considered jointly. Existing interactive video generation systems primarily emphasize following viewer instructions, while source-grounded video generation methods focus on aligning generated content with an external narrative or knowledge source. This leaves a fundamental question underexplored: What context should a generation model ground on during interactive continuation, and how can heterogeneous, unstructured inputs be transformed into such grounding context? In this work, we formulate contextual grounding as the process of transforming heterogeneous input context into an executable constraint model for video generation. To address this challenge, we introduce VideoTaleInteract, which performs Structured Context Abstraction to recover grounded states and cross-context dependencies, Generation Constraints Induction to project relevant dependencies into candidate-specific constraints, and Constrained Video Generation to enforce these constraints through planning, verification, revision, and rendering. Across three video generation backbones, VideoTaleInteract consistently achieves the highest overall composite score, reaching an average absolute improvement of 10.3 points over the strongest per-backbone baselines. Detailed analysis further shows gains across both narrative and knowledge grounding, and reveals remaining challenges in dependency extraction, and faithful realization during video rendering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Video Generation, Agentic Filmmaking, Multimodal Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kqhDOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies interactive video storytelling with contextual grounding. Given an external narrative or knowledge source, an already rendered video prefix, and a new viewer request, the task is to produce a continuation that follows the request while remaining faithful to the source and consistent with the visible history. The proposed VideoTaleInteract framework treats this problem as context compilation. It first abstracts the inputs into structured video, story, and source states with cross-context alignments. It then derives dependencies among these states and activates candidate-specific generation constraints. These constraints are used throughout an iterative plan–verify–revise–render process, including both script-level verification and verification of the final rendered video. The paper also introduces IVS-Bench, which contains narrative- and knowledge-grounded interactive continuation instances. Across three video-generation backbones, VideoTaleInteract obtains the strongest reported semantic composite score, with an average improvement of 10.3 points over the strongest corresponding baselines. The paper further provides ablations, qualitative examples, failure analysis, and a human validation study for the automatic evaluators.

Critical strengths
  1. The paper identifies and formalizes an important gap between interactive generation and source-grounded storytelling. Existing interactive video systems generally emphasize responsiveness to the latest input, while existing storytelling systems usually begin from a fixed prompt, script, or narrative plan. This paper makes a useful distinction: after part of a video has been rendered, a new request must be interpreted jointly with both the persistent source and the concrete state already shown to the viewer. The separation among source faithfulness, story continuity, and interaction fulfillment is conceptually clear and provides a useful formulation for future work.

  2. The proposed context-compilation interface is technically meaningful and empirically promising. In particular, the use of a shared dependency representation for candidate-conditioned constraint activation, script repair, and post-rendering verification is a coherent design. Applying the same grounded dependencies before and after video synthesis addresses the important fact that a valid script does not guarantee faithful visual realization. The consistent gains across narrative and knowledge grounding and across three different video backbones provide encouraging evidence that the framework captures a reusable reasoning problem rather than renderer-specific behavior.

Critical weaknesses
  1. The motivation would benefit from a sharper definition of what is genuinely interactive and what requires joint contextual reasoning. The current benchmark appears to evaluate one viewer intervention followed by a continuation, rather than a sustained multi-turn interaction. It also assumes that the rendered prefix is a sufficient record of all previous instructions. This assumption may not hold when earlier requests introduce goals, preferences, or factual context that are not fully visible in the rendered video. Moreover, prior benchmarks already cover neighboring aspects of the problem: StoryBench includes action execution and story continuation from a conditioning video, while WBench evaluates multi-turn interaction adherence and consistency. The paper’s distinctive motivation is therefore not interactive continuation by itself, but the joint reconciliation of an external source, a rendered history, and a new request. I find this distinction valuable, but it should be stated more narrowly and supported with a clearer taxonomy of cases in which no individual input is sufficient.

  2. The novelty of the individual method components relative to recent agentic video-generation systems is not yet fully isolated. Hierarchical story and shot planning already appear in MovieAgent; structured elaboration and self-validation appear in VideoGen-of-Thought; multimodal self-refinement is central to Co-Director; and ViMax explicitly uses dependency-aware state tracking and VLM-guided monitoring. Against this background, planning, state tracking, verification, and iterative repair are not individually new. The most novel part of VideoTaleInteract seems to be the construction of cross-context dependencies among the source, prefix, and request, followed by candidate-conditioned activation of those dependencies and their reuse at both the script and rendered-video levels. However, the paper describes this mechanism largely at a conceptual and prompt-template level. A more precise account of how dependencies are constructed and activated, together with targeted ablations against simpler state-tracking and generic plan–verify–repair variants, would make the methodological contribution substantially clearer.

Questions for authors
  1. Motivation and task composition: What proportion of IVS-Bench instances genuinely require joint reasoning over all three inputs—the source, video prefix, and viewer request—as opposed to being solvable primarily from one or two of them? Could the authors provide a taxonomy of cross-context dependency types and report leave-one-context-out results? Such an analysis would directly substantiate the paper’s central motivation.

  2. Scope of interaction: Is the intended task fundamentally single-intervention continuation, or is VideoTaleInteract expected to support repeated multi-turn interaction? If it is intended to be multi-turn, how are previous user goals or instructions preserved when they are not completely observable in the rendered prefix? Clarifying this point would help distinguish the proposed setting from both story continuation benchmarks and multi-turn interactive world-model benchmarks.

  3. Methodological novelty: How exactly are the dependency set (R) and the activation function implemented? Are dependency types predefined, induced freely by the VLM, or constrained by a schema? What behavior would be lost if the system used a generic structured summary followed by a standard verifier–repair loop? A more explicit comparison with dependency-aware state tracking and self-refinement in recent agentic video-generation systems would materially affect my assessment of the method’s novelty.

4 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer kqhD

We thank the reviewer for the thoughtful review and are encouraged by the positive assessment. We address each concern below.


1. Scope: single intervention vs. multi-turn interaction (W1, Q2)

We agree that multi-turn interaction is an interesting and important setting. The task in this paper is single-intervention continuation. Its distinctive element is not interactive continuation by itself, but the joint reconciliation of an external source, a rendered history, and a new request. We revise the introduction and Sec. 2.2 to state this.

We restricted the benchmark to a single intervention deliberately, for four reasons.

First, fixed checklists for objective evaluation. With a single intervention, every method receives the same source, the same rendered prefix, and the same request. The checklists for source faithfulness, story continuity, and interaction fulfillment can therefore be written once per instance, before any method is run. Every method is scored on the same questions, so a difference in score reflects a difference in the continuation. The judges can also be validated against human raters on those same checks (App. F.3).

Second, in a multi-turn session, the evaluation protocol is not being shared across method. From the second turn on, each method continues its own video, so the prefixes diverge. Which source facts are already shown and which are still pending also depends on that method's earlier output, so the faithfulness and fulfillment checks change with it. The checklists would have to be regenerated per method and per turn. Methods would then be scored on different questions, by a rubric that depends on the outputs it grades. The number of possible histories also grows exponentially with the number of turns, so they cannot be enumerated and annotated in advance.

Third, the requests cannot stay fixed either. A second request written in advance may be already satisfied, impossible, or meaningless after a particular method's first continuation. Adapting the requests requires a human or simulated viewer, which gives each method different requests and adds the simulator as a confound. Supplying a shared reference prefix at every turn avoids both problems, but that is the single-intervention setting repeated.

Lastly, error attribution. A late-turn score mixes the current decision with inherited errors: a failure at turn 3 may be a reasoning failure at turn 3 or damage carried over from turn 1.

The single intervention is the fundamental step of any longer session, so we see measuring it under controlled conditions as the prerequisite for multi-turn evaluation in the future.


2. How often are all three inputs needed, and what happens when one is removed? (W1, Q1)

By dataset construction, each instance is built so that every input carries its own obligations. The request is written at an interruption point, and its requirements are mapped to source facts not yet shown in the prefix. The SF, SC, and IF checklists are derived from the source, the prefix, and the request respectively.

In practice, we further confirm this with experiment that withheld one input at a time on the 16 Omni instances (8 narrative, 8 knowledge), with the judges keeping the full task. Changes are in points relative to the full-context method.

Input withheldJCS, narrativeJCS, knowledgeOwn metric (pooled)Other metrics (pooled)
Source−18.2−22.3SF −41IF −14
Rendered prefix−7.6−8.7SC −13.5
Viewer request−2.1−31.7IF −23SF −24

All three input-removal comparisons were significant after Holm correction (adjusted p < 0.05). Each input is needed for its own metric, and the cross-metric effects show why the reasoning must be joint. The relative small narrative JCS change for the request hides two opposite effects: The continuation that ignores the request has fewer obligations to reconcile with the prefix, so the request is therefore needed for interaction fulfillment in both settings, and JCS understates this on narrative instances. This trade-off is the joint-reasoning problem the paper targets. We report the two settings and the per-metric changes separately in the revision.

Usage by the method. In 95% of our method's outputs, the constraints activated by the final script cited evidence from the source, the prefix, and the request.

Taxonomy of dependencies. The table gives the share of dependencies by relation type (rounded, so shares sum to 101%). We note that the dependencies schema is not shared with the method.

Dependency typeShare of dependencies
Temporal precedence47%
Persistence17%
Source support13%
Compatibility12%
Causal prerequisite9%
State transition3%

On average, the system constructed 20 dependencies per instance.

AuthorsOpenReview ↗

[2/2] Response to Reviewer kqhD

3. How R and activation are implemented (W2, Q3)

Thanks reviewer for the suggestion, we add the following implementation details to the appendix:

  • LLM-derived dependencies are generated by LLMs freely (at most 30), and must cite existing predicate IDs as antecedents.
  • Activation is one verifier (LLM) call per candidate script. It instantiates each dependency the script makes relevant into a constraint (scope, condition, relation, evidence IDs).
  • The constraints are evaluated on the script and on relations extracted from the rendered clips, and they drive targeted revision and shot-level re-rendering.

4. What is lost with a generic summary and a standard verify–repair loop? (W2, Q3)

We added the request ablation by replacing our structured context representation and dependency-guided checks with a generic structured summary and a generic verify–repair loop. This lowered JCS by 4.4 points. Both variants can verify and revise a candidate continuation, so the gap persists even when the comparison method has an opportunity to repair its output.

The generic variant loses following behaviors. Its summary has no evidence links, so its verifier can flag an inconsistency but cannot trace it to the fact that caused it, and the repair feedback is not targeted. It has no candidate-conditioned activation, so nothing determines which obligations a specific proposed action triggers. We add the per-metric breakdown of this comparison to Sec. 5.2.

Section 5.2 gives complementary evidence on narrative-grounded instances. Replacing the structured state with a free-form summary lowered JCS by 7.60 points. One-Pass Planning, which keeps the structured state but omits candidate-specific constraint induction, lowered it by 4.36 points. The 4.4-point figure comes from a separate comparison that changes representation and verification together, so the three figures should not be read as additive, and none isolates a single component.


5. What distinguishes VideoTaleInteract from prior agentic video systems? (W2, Q3)

We agree that planning, state tracking, verification, and iterative repair are not individually new. Our contribution is to determine which dependencies become relevant to a particular proposed continuation. VideoTaleInteract aligns evidence from the external source, rendered prefix, and viewer request, activates the constraints triggered by the proposed action, and checks those constraints in both the script and the rendered video.

External sourceRendered-prefix stateMid-video requestSource–prefix–request alignmentCandidate-conditioned constraintsVerification and refinement
MovieAgentscript–––––
VideoGen-of-Thought–––––self-validation
Co-Directorbrief/assets––––multimodal self-refinement
ViMaxscreenplaydependency-aware state (own shots)–––VLM monitoring
Ours✓✓✓✓✓script and rendered video, same constraints

This comparison is at the level of design. Co-Director and ViMax are recent preprints (April and June 2026) and are not among our evaluated baselines. We add the table and this discussion to Appendix A.

Reviewer_kqhDOpenReview ↗

Follow-up on Rebuttal and Rating Update

Thank you for the detailed rebuttal and additional experiments. The clarifications on task scope, joint contextual reasoning, and dependency mechanisms, along with the new ablations, have satisfactorily addressed my main concerns.

I appreciate the authors' efforts and have raised my rating to 4.

Dear reviewer KqhD,

We are pleased to hear we have solved all your concerns!

We sincerely thank you again for your careful evaluation and suggestions :D

Reviewer_9TbxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines a task where a viewer interrupts a generated video with a request, and the continuation must respect three things: the request, a grounding source (a story or a document), and the video prefix already shown. VideoTaleInteract extracts typed predicates from all three inputs into video, story, and source states. It builds cross-context dependencies, turns the ones a candidate script actually triggers into constraints, and then runs a plan, verify, revise, render loop that checks both the script and the rendered output. The authors also build IVS-Bench from CoQA passages: 250 sources, 1,004 requests, and 2,008 instances in two visual styles. Evaluation uses three backbones (Omni-1.1-Flash, Veo 3.1, MiniMax-H3) and four agentic baselines. Two LLM judges score source faithfulness, story continuity, interaction fulfillment, visual consistency, and quality, and the paper reports a 10.3-point average composite gain.

Critical strengths
  1. The task is new and well motivated. A viewer request can conflict with both what the source supports and what the prefix has already shown, and the paper frames this clearly.
  2. Activating a constraint only when a candidate script makes it relevant, with evidence links attached, is a sensible design. It makes failures traceable and gives the revision step targeted feedback. The robustness across backbones (composite range under 6 points vs. over 30 for Direct Generation) suggests the structure does real work.
Critical weaknesses

1 .The evaluation is far smaller than the benchmark. Appendix B shows that only 16 instances per backbone (8 narrative, 8 knowledge) were evaluated, out of 2,008. With so few samples the confidence intervals are wide and overlap: on MiniMax-H3 the result is 78.8 [75.5, 81.4] vs. 75.4 [68.2, 81.7]. That does not support the “consistently highest” claim, and the benchmark contribution is mostly unused. 2. The comparison does not control for compute. VideoTaleInteract runs iterative script and output repair with extra VLM calls, and the authors acknowledge that spending was capped, not matched. The gains could come partly from extra test-time compute.

Questions for authors

1 .Can you report results on a substantially larger portion of IVS-Bench (for example, several hundred instances), with per-backbone significance tests? 2. What are the average number of VLM calls, repair rounds, and wall-clock time per continuation, compared with the baselines? How do the baselines perform with a matched budget, for example best-of-N selection? 3. (Withdrawn) Clarification: Table 1 includes values like 6.3 and 93.8, which look like averages over very few items. Is each cell computed over 8 instances?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9Tbx

We thank the reviewer for the constructive review. We are encouraged that reviewer finds that: "the task is new and well motivated", the method is "a sensible design" that "makes failures traceable and gives the revision step targeted feedback", as well as the robustness across backbones "suggests the structure does real work".

We address each concern below.


1. Evaluation scale and per-backbone significance (W1, Q1)

We conducted additional experiments with all five methods on 277 more instances (277 instance X 2 visual style X 2 backbones X 5 methods = 5540 evaluated video) from IVS-Bench on Veo 3.1 and Omni, and 40 instances (40 instance X 2 visual style X 5 methods = 400 evaluated video) on MiniMax-H3 due to the slow inference time of this model and the computational budget limits.

BackboneOursBest baselineΔJCS [95% CI]Holm p
Veo 3.184.979.4+5.6 [3.9, 7.3]0.001
Omni86.481.0+5.4 [3.5, 6.7]0.001
MiniMax-H380.375.3+5.1 [0.1, 10.0]0.063

On Veo and Omni, our method is significantly better than every baseline on SF, SC, IF and JCS. The margin is smaller than in the 16-instance Table 1, but the improvement is still significant on all metrics (p ≤ 0.031). We updated the paper with the table of new results. On MiniMax, our method is also leading but less significant compared to other backbone. This is due to the limitation of MiniMax poor video generation capability, which often is not able to fully visualize the method’s output. We have softened the relevant claim in the paper.


2. Compute, repair rounds and matched-budget baselines (W2, Q2)

First, following the reviewer's request, we report the cost below for every method.

Method$ / inst. (Omni / Veo)LLM callsLLM tokens in / outRender attempts / shotWall-clock (s, Omni / Veo)
Ours4.02 / 14.8816–18~80k / ~42k1.4–1.61,179 / 1,535
Direct4.48 / 19.336–8~35k / ~5k1.8–2.0851 / 1,290
MM-StoryAgent3.91 / 15.1117–19~55k / ~16k1.61,076 / 1,469
MovieAgent3.65 / 14.719–10~37k / ~10k1.5–1.6970 / 1,213
VideoGen-of-Thought3.82 / 14.769–11~35k / ~7k1.5–1.7992 / 1,251

Second, we’d like to clarify that the max-allowed VLM calls and video generation calls are equivalent for our method and all baselines. Some baselines might stop early. To further address the reviewer's concern, we conducted an additional experiment to match the token budgets, we let each baseline render best-of-N candidates until it had used as many LLM tokens as our method with 40 instances.

CompositeOmni: original → budget-matchedVeo: original → budget-matched
Ours87.788.1
VideoGen-of-Thought81.2 → 80.382.7 → 84.1
MovieAgent78.9 → 78.483.5 → 80.9
MM-StoryAgent74.1 → 71.572.0 → 73.6
Direct66.3 → 67.664.7 → 67.2

With the extra budget, our method stays first (all CIs exclude 0).


3. Judge bias from the shared model family (W2)

To test this, we re-scored 368 videos with 7 judges, keeping the checklists, frames and transcripts identical.

JudgeOmni: rank / Δ vs. bestVeo: rank / Δ vs. best
Gemini 3.8 Flash (paper)1 / +5.81 / +4.2
Claude Opus 5 (paper)1 / +7.21 / +4.0
Claude Opus 5.51 / +5.01 / +4.2
Claude Sonnet 4.61 / +6.41 / +3.8
Gemma 4 31B1 / +6.41 / +4.0
Gemini 4 Argon1 / +6.81 / +4.0
Qwen3-VL-32B1 / +7.91 / +1.3

The gain does not depend on a same-family judge.

Additionally, we conducted an anonymized human evaluation where five human annotators scored 70 entries using the identical 0–4 scale criteria, completely blind to the generation method:

Mean JCS on rated items (Omni + Veo)ItemsHumanJudges
Ours1488.586.0
VideoGen-of-Thought1484.280.0
MovieAgent1482.177.9
MM-StoryAgent1475.876.6
Direct1467.571.3

The human preferences match the automatic judge rank-ordering, confirming our approach as top-performing


4. Clarification on Table 1 cells (Q3)

Yes, each narrative or knowledge cell in Table 1 averages across 8 evaluation instances. Each instance's score is itself a mean of 0–4 checklist ratings, normalized to 0–100 and averaged over the two judges (App. D.3). Values such as 6.3 or 93.8 are therefore means, not success counts. Table 1 will be replaced by the larger-scale results in Point 1.

Thank you for reporting the detailed experimental results. I will raise my rating by one level.

We are pleased to hear that we have addressed all of your concerns! We sincerely thank you again for your careful evaluation and suggestions.

Reviewer_UhdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates interactive video continuation as a contextual-grounding problem. Given a grounding source, a previously rendered video prefix, and a new viewer request, the system must generate a continuation that satisfies the request while remaining faithful to the source and consistent with the established video state. The authors propose VideoTaleInteract, which first performs Structured Context Abstraction to extract video, story, and source states with evidence links. It then constructs cross-context dependencies and activates only those relevant to a candidate continuation. These dependencies guide script planning, verification, revision, rendering, and output-level repair. The paper also introduces IVS-Bench, containing narrative- and knowledge-grounded synthetic video-continuation instances. The evaluation measures source faithfulness, story continuity, interaction fulfillment, joint constraint satisfaction, visual consistency, and video quality. Across three video backbones and four adapted baselines, VideoTaleInteract obtains the highest reported composite score. Ablations suggest that structured state representation and candidate-specific constraint induction both contribute to the gains. A small human-validation study reports moderate-to-strong agreement between automatic and human judgments.

Critical strengths
  1. A clear contextual formulation of interactive video continuation. The paper separates what is established by the source, what has been visibly established by the prefix, and what is requested by the viewer. The dependency representation and candidate-conditioned activation provide a principled interface between contextual reasoning and video generation.

  2. Evaluation across multiple semantic dimensions and video backbones. The benchmark distinguishes source faithfulness, story continuity, interaction fulfillment, visual consistency, and video quality rather than collapsing everything into a single instruction-following score. Evaluating the method with three different video backbones and including structured ablations are useful strengths.

  3. Explicit attention to verification and failure modes. The method checks both the planned script and the rendered output, and the paper acknowledges that extraction, dependency inference, and rendering can all fail. The human validation of the automatic metrics is also a positive step.

Critical weaknesses
  1. The benchmark is synthetic, small per backbone, and potentially aligned with the proposed representation. IVS-Bench is constructed from 250 CoQA passages, converted into scene-based stories and interaction instructions with substantial generative-model assistance. The main comparison evaluates only 16 instances per backbone, and the evaluated instances may differ across backbones. Synthetic source transformations, instructions, assets, and evaluation checklists may favor a pipeline explicitly designed around source facts and structured dependencies. The results therefore do not yet establish performance on naturally occurring viewer requests or human-authored interactive video tasks.

  2. Baseline comparisons do not isolate contextual reasoning from implementation and compute differences. The proposed system can use up to 48 agent calls, multiple planning and repair stages, image candidates, and output-level re-rendering. The baselines are adapted to a common endpoint but lose several native components, including latent propagation, smoothing, and some synchronization modules. The appendix explicitly states that token usage, number of calls, retries, and latency are not matched. Thus, the reported gains establish an advantage for the complete evaluated configuration, but not necessarily for structured contextual grounding itself.

  3. The interactive claim is only partially tested. The benchmark uses a fixed 24-second prefix and a fixed 48-second, two-scene continuation. There is no evaluation of repeated user interventions, long-horizon state accumulation, response latency, or whether a viewer can actually steer generation in an interactive setting. The formative user study has only nine participants, with seven included in the learning-gain analysis and four providing experience ratings; it is exploratory rather than evidence for the proposed system’s interactive utility.

  4. The composite metric does not represent strict joint satisfaction. JCS is the arithmetic mean of source faithfulness, story continuity, and interaction fulfillment, so a low score in one dimension can be offset by high scores in another. The authors report a conjunctive variant with the same ordering, which is helpful, but the headline composite still does not directly measure whether all three constraints are simultaneously satisfied. In addition, the benchmark score excludes visual consistency and video quality from the overall composite.

Questions for authors
  1. Can the authors evaluate VideoTaleInteract on a held-out set of independently human-authored sources and viewer requests, preferably with source-matched paired instances across backbones? This would help determine whether the results are specific to the synthetic benchmark construction.

  2. Please report measured model calls, token usage, wall-clock latency, rendering cost, and retry counts for every method, together with a compute-matched ablation of structured context and dependency induction. How much of the gain remains when the proposed system and baselines receive comparable planning and repair budgets?

  3. Can the authors test multiple sequential viewer interventions on the same evolving video, including requests that conflict with prior state or source constraints? This would more directly validate the interactive and long-context claims.

3 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer Uhdt

We thank the reviewer for the careful review. We are encouraged that the reviewer finds we give "a clear contextual formulation of interactive video continuation", and the evaluation covers multiple semantic dimensions and video backbones. We address the reviewer's concern point-by-point below.


1. Benchmark evaluation scale concern (W1, Q1)

We conducted additional experiments with all five methods on 277 more instances (277 instance X 2 visual style X 2 backbones X 5 methods = 5540 evaluated video) from IVS-Bench on Veo 3.1 and Omni, and 40 instances (40 instance X 2 visual style X 5 methods = 400 evaluated video) on MiniMax-H3 due to the computational budget limits.

BackboneOursBest baselineΔJCS [95% CI]Holm p
Veo 3.184.979.4+5.6 [3.9, 7.3]0.001
Omni86.481.0+5.4 [3.5, 6.7]0.001
MiniMax-H380.375.3+5.1 [0.1, 10.0]0.063

On Veo and Omni, our method is significantly better than every baseline on SF, SC, IF and JCS. The margin is smaller than in the 16-instance Table 1, but the improvement is still significant on all metrics (p ≤ 0.031). We updated the paper with the table of new results. On MiniMax, our method is also leading but less significant compared to other backbone. This is due to the limitation of MiniMax poor video generation capability, which often is not able to fully visualize the method’s output. We have softened the relevant claim in the paper.

Lastly, following the reviewer's request, we restricted the analysis to instances scored for all five methods on both backbones to check source matching, so every comparison uses the identical set. We report our method gain over best baseline:

Matched setVeoOmni
197 instances+5.8 [4.1, 7.5]+5.2 [3.5, 7.0]

2. Synthetic construction and potential judge bias (W1, Q1)

We'd like to clarify how the evaluation is kept separate from our method. The SF checklists are derived from CoQA's human-written QA annotations. Each checklist is reviewed by two human evaluators. All checklists are fixed before any method is run, and no method sees them.

Regarding the concern on the evaluator bias on source-facts and dependencies, If the evaluation setup systematically favored fact-centric architectures, the primary gains would manifest in source faithfulness. Relative to the top baseline for each metric, SF shows a 2.1-point advantage (92.7 vs. 90.6), whereas story continuity exhibits a substantially larger 13.6-point margin (81.3 vs. 67.7). This differential indicates that performance gains extend beyond source-fact alignment alone.

Additionally, we conducted an anonymized human evaluation where five human annotators scored 70 entries using the identical 0–4 scale criteria, completely blind to the generation method:

Mean JCS on rated items (Omni + Veo)ItemsHumanJudges
Ours1488.586.0
VideoGen-of-Thought1484.280.0
MovieAgent1482.177.9
MM-StoryAgent1475.876.6
Direct1467.571.3

The human preferences match the automatic judge rank-ordering, confirming our approach as top-performing


3. Measured compute and budget-matched baselines (W2, Q2)

First, following the reviewer's request, we report the cost below for every method.

Method$ / inst. (Omni / Veo)LLM callsLLM tokens in / outRender attempts / shotWall-clock (s, Omni / Veo)
Ours4.02 / 14.8816–18~80k / ~42k1.4–1.61,179 / 1,535
Direct4.48 / 19.336–8~35k / ~5k1.8–2.0851 / 1,290
MM-StoryAgent3.91 / 15.1117–19~55k / ~16k1.61,076 / 1,469
MovieAgent3.65 / 14.719–10~37k / ~10k1.5–1.6970 / 1,213
VideoGen-of-Thought3.82 / 14.769–11~35k / ~7k1.5–1.7992 / 1,251

Second, we’d like to clarify that the max-allowed VLM calls and video generation calls are equivalent for our method and all baselines. Some baselines might stop early. To further address the reviewer's concern, we conducted an additional experiment to match the token budgets, we let each baseline render best-of-N candidates until it had used as many LLM tokens as our method with 40 instances.

CompositeOmni: original → budget-matchedVeo: original → budget-matched
Ours87.788.1
VideoGen-of-Thought81.2 → 80.382.7 → 84.1
MovieAgent78.9 → 78.483.5 → 80.9
MM-StoryAgent74.1 → 71.572.0 → 73.6
Direct66.3 → 67.664.7 → 67.2

Even with the extra budget for the baselines, our method outperforms them (all CIs exclude 0).

Third, regarding the native components omitted from the adapted baselines: these operate inside each system’s original renderer and cannot be expressed through a shared video backbone, which exposes no latent access.

AuthorsOpenReview ↗

[2/2] Response to Reviewer Uhdt

4. How much does structured contextual grounding contribute? (W2, Q2)

We tested additional ablation of whether a generic summary and a generic verify–repair loop could achieve the same performance as the full method. Replacing our structured context representation and dependency-guided checks with these generic alternatives lowered the Mean Semantic Score by 4.40 percentage points.

The ablations in Section 5.2 provide complementary evidence. On narrative-grounded instances, replacing the structured state with a free-form summary lowered the Mean Semantic Score by 7.60 points. One-Pass Planning lowered the score by 4.36 points. Together, these results support the contribution of structured grounding within our pipeline.

In addition, the separate token-matched comparison in Point 3 addresses how the adapted baselines perform with additional sampling budget.


5. Multiple sequential interventions (W3, Q3)

First, we'd like to clarify that the benchmark uses a single intervention by design, so that each continuation can be scored against fixed checklists for objective and fair comparison. We underscore this design choice in the revised paper, and will clarify that multi-turn setting is out of the scope of this paper.

Second, to examine how the method handles repeated interventions, we ran six exploratory multi-turn sessions per method on each backbone. Each session contained four human-written viewer requests.

BackboneFirst-turn Mean Semantic Score, oursFirst-turn Mean Semantic Score, VGoTCross-turn state consistency, oursCross-turn state consistency, VGoT
Omni87.383.666.065.6
Veo91.886.264.662.5

Three viewer requests conflicted with an earlier request in the same session. Our method successfully handled two of these three conflicts, while VGoT handled none. We report these exploratory results in the revised appendix.


6. Question and suggestion on metrics (W4)

We thank the reviewer for the suggestion. We rename JCS to Mean Semantic Score and add the conjunctive metrics below to the main table.

In addition, we report the conjunctive metric:

Metric (Veo / Omni)OursBest baselineΔ (Holm p)
pass@75: SF, SC and IF all ≥ 75 (%)61.6 / 68.746.4 / 47.4+15.5 / +21.9 (≤ 0.001)
pass@75, generation failures counted as 0 (%)60.9 / 60.446.4 / 45.3+14.6 / +14.9 (≤ 0.005)
min(SF, SC, IF)75.6 / 77.766.5 / 67.5+9.1 / +9.9 (0.001)
Composite including VC and VQ80.7 / 83.177.4 / 79.1+3.2 / +3.8 (0.001)

We kept VC and VQ outside the headline composite because frame-sampled VQ is largely bounded by the backbone. Including them leaves the ranking unchanged.


7. The role of the formative study, and evaluation on human-written requests (W1, W3)

We'd like to clarify that the formative study only motivates the task. It is not used to evaluate the method, and we will say so explicitly in Sec. 3.

We evaluate VideoTaleInteract’s performance separately through comparisons with the four baselines. To test whether the observed gains persist when people write their own requests, we evaluated 35 participant-written requests across 10 video instances.

BackboneMethodSFSCIFNKComposite [95% CI]
Veo 3.1Direct56.074.471.074.059.966.9 [63.6, 73.3]
MM-StoryAgent65.871.884.674.074.174.0 [70.1, 78.1]
VideoGen-of-Thought76.477.285.674.285.679.9 [75.5, 83.1]
MovieAgent81.873.888.281.980.581.2 [78.0, 84.0]
VideoTaleInteract89.084.491.084.092.588.2 [86.3, 91.1]
Omni 1.1 FlashDirect54.674.670.573.958.866.4 [60.3, 74.5]
MM-StoryAgent66.973.486.576.175.175.6 [71.3, 80.5]
VideoGen-of-Thought76.972.989.580.279.379.7 [77.0, 84.6]
MovieAgent82.174.688.979.684.281.9 [79.7, 84.8]
VideoTaleInteract85.187.088.085.388.186.7 [84.2, 90.6]

VideoTaleInteract achieved the highest composite on both backbones. VideoGen-of-Thought achieved the highest interaction fulfillment on Omni (89.5 versus our 88.0). As discussed in the limitation analysis (Sec. 5.3), this is consistent with a trade-off built into our design. Methods that condition more directly on the latest instruction can score higher on IF in such cases, but at a cost to the other two dimensions: on Omni, VideoGen-of-Thought is 8.2 points below our method on source faithfulness (76.9 vs. 85.1) and 14.1 points below on story continuity (72.9 vs. 87.0). The IF gap itself is small (1.5 points), while our method leads on IF on Veo (91.0 vs. 88.2).

AuthorsOpenReview ↗

Following up on the rebuttal response

Dear Reviewer Uhdt,

We sincerely thank the reviewer for their thoughtful feedback and constructive suggestions, which have helped us substantially strengthen the paper.

We hope these additional experiments and clarifications have satisfactorily addressed the reviewer's concerns and provided stronger evidence for our contributions. We would be truly grateful if the reviewer could kindly reconsider their assessment.

Thank you again for your time, careful, and valuable feedback throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.