- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies interactive video storytelling with contextual grounding. Given an external narrative or knowledge source, an already rendered video prefix, and a new viewer request, the task is to produce a continuation that follows the request while remaining faithful to the source and consistent with the visible history. The proposed VideoTaleInteract framework treats this problem as context compilation. It first abstracts the inputs into structured video, story, and source states with cross-context alignments. It then derives dependencies among these states and activates candidate-specific generation constraints. These constraints are used throughout an iterative plan–verify–revise–render process, including both script-level verification and verification of the final rendered video. The paper also introduces IVS-Bench, which contains narrative- and knowledge-grounded interactive continuation instances. Across three video-generation backbones, VideoTaleInteract obtains the strongest reported semantic composite score, with an average improvement of 10.3 points over the strongest corresponding baselines. The paper further provides ablations, qualitative examples, failure analysis, and a human validation study for the automatic evaluators.
Critical strengths
-
The paper identifies and formalizes an important gap between interactive generation and source-grounded storytelling. Existing interactive video systems generally emphasize responsiveness to the latest input, while existing storytelling systems usually begin from a fixed prompt, script, or narrative plan. This paper makes a useful distinction: after part of a video has been rendered, a new request must be interpreted jointly with both the persistent source and the concrete state already shown to the viewer. The separation among source faithfulness, story continuity, and interaction fulfillment is conceptually clear and provides a useful formulation for future work.
-
The proposed context-compilation interface is technically meaningful and empirically promising. In particular, the use of a shared dependency representation for candidate-conditioned constraint activation, script repair, and post-rendering verification is a coherent design. Applying the same grounded dependencies before and after video synthesis addresses the important fact that a valid script does not guarantee faithful visual realization. The consistent gains across narrative and knowledge grounding and across three different video backbones provide encouraging evidence that the framework captures a reusable reasoning problem rather than renderer-specific behavior.
Critical weaknesses
-
The motivation would benefit from a sharper definition of what is genuinely interactive and what requires joint contextual reasoning. The current benchmark appears to evaluate one viewer intervention followed by a continuation, rather than a sustained multi-turn interaction. It also assumes that the rendered prefix is a sufficient record of all previous instructions. This assumption may not hold when earlier requests introduce goals, preferences, or factual context that are not fully visible in the rendered video. Moreover, prior benchmarks already cover neighboring aspects of the problem: StoryBench includes action execution and story continuation from a conditioning video, while WBench evaluates multi-turn interaction adherence and consistency. The paper’s distinctive motivation is therefore not interactive continuation by itself, but the joint reconciliation of an external source, a rendered history, and a new request. I find this distinction valuable, but it should be stated more narrowly and supported with a clearer taxonomy of cases in which no individual input is sufficient.
-
The novelty of the individual method components relative to recent agentic video-generation systems is not yet fully isolated. Hierarchical story and shot planning already appear in MovieAgent; structured elaboration and self-validation appear in VideoGen-of-Thought; multimodal self-refinement is central to Co-Director; and ViMax explicitly uses dependency-aware state tracking and VLM-guided monitoring. Against this background, planning, state tracking, verification, and iterative repair are not individually new. The most novel part of VideoTaleInteract seems to be the construction of cross-context dependencies among the source, prefix, and request, followed by candidate-conditioned activation of those dependencies and their reuse at both the script and rendered-video levels. However, the paper describes this mechanism largely at a conceptual and prompt-template level. A more precise account of how dependencies are constructed and activated, together with targeted ablations against simpler state-tracking and generic plan–verify–repair variants, would make the methodological contribution substantially clearer.
Questions for authors
-
Motivation and task composition: What proportion of IVS-Bench instances genuinely require joint reasoning over all three inputs—the source, video prefix, and viewer request—as opposed to being solvable primarily from one or two of them? Could the authors provide a taxonomy of cross-context dependency types and report leave-one-context-out results? Such an analysis would directly substantiate the paper’s central motivation.
-
Scope of interaction: Is the intended task fundamentally single-intervention continuation, or is VideoTaleInteract expected to support repeated multi-turn interaction? If it is intended to be multi-turn, how are previous user goals or instructions preserved when they are not completely observable in the rendered prefix? Clarifying this point would help distinguish the proposed setting from both story continuation benchmarks and multi-turn interactive world-model benchmarks.
-
Methodological novelty: How exactly are the dependency set (R) and the activation function implemented? Are dependency types predefined, induced freely by the VLM, or constrained by a schema? What behavior would be lost if the system used a generic structured summary followed by a standard verifier–repair loop? A more explicit comparison with dependency-aware state tracking and self-refinement in recent agentic video-generation systems would materially affect my assessment of the method’s novelty.
4 comments
[2/2] Response to Reviewer kqhD
3. How R and activation are implemented (W2, Q3)
Thanks reviewer for the suggestion, we add the following implementation details to the appendix:
- LLM-derived dependencies are generated by LLMs freely (at most 30), and must cite existing predicate IDs as antecedents.
- Activation is one verifier (LLM) call per candidate script. It instantiates each dependency the script makes relevant into a constraint (scope, condition, relation, evidence IDs).
- The constraints are evaluated on the script and on relations extracted from the rendered clips, and they drive targeted revision and shot-level re-rendering.
4. What is lost with a generic summary and a standard verify–repair loop? (W2, Q3)
We added the request ablation by replacing our structured context representation and dependency-guided checks with a generic structured summary and a generic verify–repair loop. This lowered JCS by 4.4 points. Both variants can verify and revise a candidate continuation, so the gap persists even when the comparison method has an opportunity to repair its output.
The generic variant loses following behaviors. Its summary has no evidence links, so its verifier can flag an inconsistency but cannot trace it to the fact that caused it, and the repair feedback is not targeted. It has no candidate-conditioned activation, so nothing determines which obligations a specific proposed action triggers. We add the per-metric breakdown of this comparison to Sec. 5.2.
Section 5.2 gives complementary evidence on narrative-grounded instances. Replacing the structured state with a free-form summary lowered JCS by 7.60 points. One-Pass Planning, which keeps the structured state but omits candidate-specific constraint induction, lowered it by 4.36 points. The 4.4-point figure comes from a separate comparison that changes representation and verification together, so the three figures should not be read as additive, and none isolates a single component.
5. What distinguishes VideoTaleInteract from prior agentic video systems? (W2, Q3)
We agree that planning, state tracking, verification, and iterative repair are not individually new. Our contribution is to determine which dependencies become relevant to a particular proposed continuation. VideoTaleInteract aligns evidence from the external source, rendered prefix, and viewer request, activates the constraints triggered by the proposed action, and checks those constraints in both the script and the rendered video.
| External source | Rendered-prefix state | Mid-video request | Source–prefix–request alignment | Candidate-conditioned constraints | Verification and refinement | |
|---|---|---|---|---|---|---|
| MovieAgent | script | – | – | – | – | – |
| VideoGen-of-Thought | – | – | – | – | – | self-validation |
| Co-Director | brief/assets | – | – | – | – | multimodal self-refinement |
| ViMax | screenplay | dependency-aware state (own shots) | – | – | – | VLM monitoring |
| Ours | ✓ | ✓ | ✓ | ✓ | ✓ | script and rendered video, same constraints |
This comparison is at the level of design. Co-Director and ViMax are recent preprints (April and June 2026) and are not among our evaluated baselines. We add the table and this discussion to Appendix A.
Follow-up on Rebuttal and Rating Update
Thank you for the detailed rebuttal and additional experiments. The clarifications on task scope, joint contextual reasoning, and dependency mechanisms, along with the new ablations, have satisfactorily addressed my main concerns.
I appreciate the authors' efforts and have raised my rating to 4.
Dear reviewer KqhD,
We are pleased to hear we have solved all your concerns!
We sincerely thank you again for your careful evaluation and suggestions :D
[1/2] Response to Reviewer kqhD
We thank the reviewer for the thoughtful review and are encouraged by the positive assessment. We address each concern below.
We agree that multi-turn interaction is an interesting and important setting. The task in this paper is single-intervention continuation. Its distinctive element is not interactive continuation by itself, but the joint reconciliation of an external source, a rendered history, and a new request. We revise the introduction and Sec. 2.2 to state this.
We restricted the benchmark to a single intervention deliberately, for four reasons.
First, fixed checklists for objective evaluation. With a single intervention, every method receives the same source, the same rendered prefix, and the same request. The checklists for source faithfulness, story continuity, and interaction fulfillment can therefore be written once per instance, before any method is run. Every method is scored on the same questions, so a difference in score reflects a difference in the continuation. The judges can also be validated against human raters on those same checks (App. F.3).
Second, in a multi-turn session, the evaluation protocol is not being shared across method. From the second turn on, each method continues its own video, so the prefixes diverge. Which source facts are already shown and which are still pending also depends on that method's earlier output, so the faithfulness and fulfillment checks change with it. The checklists would have to be regenerated per method and per turn. Methods would then be scored on different questions, by a rubric that depends on the outputs it grades. The number of possible histories also grows exponentially with the number of turns, so they cannot be enumerated and annotated in advance.
Third, the requests cannot stay fixed either. A second request written in advance may be already satisfied, impossible, or meaningless after a particular method's first continuation. Adapting the requests requires a human or simulated viewer, which gives each method different requests and adds the simulator as a confound. Supplying a shared reference prefix at every turn avoids both problems, but that is the single-intervention setting repeated.
Lastly, error attribution. A late-turn score mixes the current decision with inherited errors: a failure at turn 3 may be a reasoning failure at turn 3 or damage carried over from turn 1.
The single intervention is the fundamental step of any longer session, so we see measuring it under controlled conditions as the prerequisite for multi-turn evaluation in the future.
By dataset construction, each instance is built so that every input carries its own obligations. The request is written at an interruption point, and its requirements are mapped to source facts not yet shown in the prefix. The SF, SC, and IF checklists are derived from the source, the prefix, and the request respectively.
In practice, we further confirm this with experiment that withheld one input at a time on the 16 Omni instances (8 narrative, 8 knowledge), with the judges keeping the full task. Changes are in points relative to the full-context method.
All three input-removal comparisons were significant after Holm correction (adjusted p < 0.05). Each input is needed for its own metric, and the cross-metric effects show why the reasoning must be joint. The relative small narrative JCS change for the request hides two opposite effects: The continuation that ignores the request has fewer obligations to reconcile with the prefix, so the request is therefore needed for interaction fulfillment in both settings, and JCS understates this on narrative instances. This trade-off is the joint-reasoning problem the paper targets. We report the two settings and the per-metric changes separately in the revision.
Usage by the method. In 95% of our method's outputs, the constraints activated by the final script cited evidence from the source, the prefix, and the request.
Taxonomy of dependencies. The table gives the share of dependencies by relation type (rounded, so shares sum to 101%). We note that the dependencies schema is not shared with the method.
On average, the system constructed 20 dependencies per instance.