acceptodds
Under review as a conference paper at ICLR 2027

Weave Forcing: Compositional Memory Routing for Interactive Long Video Generation

Abstract

Recent advances in autoregressive video generation have improved temporal consistency over extended durations, yet interactive storytelling requires more than continuous scene extension: a new shot may combine characters and backgrounds from different historical shots. Whole prompt retrieval can overlook the distinct reference needs of individual components, while directly combining all historical memories may introduce unrelated visual content. To address these problems, we present **Weave Forcing**, a training-free framework for compositional memory reuse in interactive long video generation. First, we use an LLM for semantic slot routing to decompose user prompts into character and background descriptions and explicitly select suitable historical references for each component. To isolate the required content, masked memory weaving uses contrasting attention maps conditioned on semantic slots to construct refined semantic masks, selectively exposing relevant tokens from compressed historical KV memories to guide the generation of the current shot. We further introduce coverage adaptive RoPE to adjust temporal offsets and memory retention according to no, partial, or full reference coverage, addressing visual artifacts observed when incomplete historical references are positioned close to the current generation. Extensive experiments demonstrate that Weave Forcing improves cross-shot subject and background consistency while maintaining competitive visual quality and text alignment.

Primary area:
generative models
Keywords:
Video Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PCSxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Weave Forcing proposes a training-free compositional memory routing framework aimed at improving cross-shot character and background consistency in interactive long video generation. The method first uses an LLM to decompose the current shot prompt into semantic components, such as characters and backgrounds, and retrieves the corresponding historical shots for each component. It then employs Contrastive Slot Attention to construct Semantic KV Masks, selectively combining relevant KV memories from different historical shots while suppressing irrelevant visual information. Additionally, the authors introduce Coverage Adaptive RoPE, which adjusts temporal position offsets and memory retention strategies according to the availability of historical references, balancing the preservation of historical content with the generation of new content.

Critical strengths
  1. The paper presents a clear and well-motivated problem formulation. It identifies the limitations of existing historical memory retrieval strategies in compositional multi-shot video generation, where retrieving a single historical shot may miss relevant components, while reusing multiple complete shots may introduce irrelevant visual information. This motivation is clearly explained and supported by intuitive examples.

  2. The proposed framework operates without additional model training and can be integrated into existing autoregressive video generation architectures through semantic memory routing, KV-cache masking, and temporal position adjustment.

Critical weaknesses
  1. Recent video generation systems, such as Seedance 2.0 and MiniMax H3, already support multi-shot video generation using explicit character and scene references, along with previously generated video content. Such workflows can maintain character identity and scene consistency across shots through explicit visual references, without requiring semantic retrieval and recombination of historical KV caches. In contrast, Weave Forcing introduces a relatively complex historical memory retrieval and fusion mechanism to address a similar compositional consistency problem. However, the paper does not sufficiently justify the necessity of adopting a KV cache-based approach, nor does it demonstrate unique advantages over existing reference-conditioned video generation workflows in terms of generation quality, efficiency, or controllability. Consequently, the practical motivation and applicability of the proposed framework remain insufficiently justified.

  2. The proposed LLM-based Semantic Memory Router primarily relies on textual descriptions of historical shots for retrieval, without verifying whether the corresponding characters or scenes were actually generated in those videos. Video generation models do not always follow textual instructions accurately. If a historical shot fails to generate a specified character or omits key visual elements, its KV cache may lack the required visual information, even if the router correctly identifies the corresponding shot based on its textual description. Therefore, correct retrieval at the textual level does not necessarily guarantee valid visual references, potentially leading to the repeated reuse of incorrect or incomplete historical memories in subsequent video generation.

  3. Weave Forcing selects only one historical shot for each semantic slot, rather than aggregating all relevant historical shots, but this design choice is not adequately justified. Different historical shots featuring the same character may provide complementary visual information, such as the character's appearance from different viewpoints and in different poses. The current strategy may therefore discard valuable information, and the paper does not investigate whether aggregating multiple relevant historical memories could further improve cross-shot consistency.

  4. Attention maps do not necessarily correspond to accurate semantic segmentation, especially when multiple subjects occlude one another, when characters and backgrounds are semantically correlated, or when small objects occupy only a few tokens. Inaccurate semantic masks may lead to the mixing of attributes across different characters. In fact, the authors demonstrate such a failure case in Figure 9 of Appendix E, where clothing attributes are incorrectly transferred between different characters.

  5. Although Semantic KV Masking as a key innovation, the experimental results show that its contribution to subject consistency is relatively limited, while the most substantial improvement comes from semantic-level historical memory retrieval. This suggests that the proposed framework relies heavily on the external LLM for effective memory routing. Furthermore, the paper does not provide a dedicated ablation study to independently evaluate the contribution of Coverage Adaptive RoPE.

Questions for authors

The paper proposes Coverage Adaptive RoPE. Specifically, when no historical references or only partial historical references are available, the authors fix the temporal RoPE offset at J=45J=45. When both character and background references are available, the offset is determined based on prompt embedding similarity within the range [1,6][1,6]. However, the paper does not sufficiently justify the choice of J=45J=45 or explain why the range [1,6][1,6] is selected for full coverage. It also lacks sensitivity analyses comparing different fixed offsets and adaptive offsets. Furthermore, since the paper simultaneously adjusts the retention duration of historical KV memories, the existing experiments make it difficult to independently assess the actual contribution of Coverage Adaptive RoPE. Please see weakness for other questions.

4 comments

We thank the reviewer for the comments and respond to each point below.

Q1. Comparison with reference-conditioned video workflows

We use Self Forcing as an autoregressive baseline and build on its generation framework. We acknowledge that our method may not match the overall visual quality of recent reference-conditioned systems such as Seedance 2.0 and MiniMax H3. Its advantage lies in a different operating setting: the generator achieves 15.42 FPS in our evaluation (Appendix A.4), while autoregressive inference and KV-cache updates support continuous long-video generation.

Building on this efficient foundation, Weave Forcing improves cross-shot consistency in interactive long-video generation. For each new instruction, it selects historical KV memories for the requested people and background and masks unrelated content. This provides control over which earlier elements are recalled and combined in a new shot, while retaining the throughput and continuous-generation capability of the autoregressive backbone.

Q2. Risk of a mismatch between text-based routing and generated content

Our shot-level prompts explicitly introduce the people and scenes that later shots may need to recall. We also provide detailed instructions to the LLM for extracting semantic slots and selecting matching historical sources. In our evaluated cases, the LLM rarely misinterpreted these instructions, and we did not observe a missing target in a historical shot being repeatedly retrieved and reused incorrectly. CSA subsequently identifies relevant regions in the selected shot’s actual historical KV memory.

We agree that, if a historical shot fails to depict a target element, its KV memory may be an unreliable reference. Therefore, we designed detailed LLM prompts that specify how to identify the requested character and background slots, match them to earlier shots, and select the corresponding historical references, thereby reducing errors in text-based memory routing.

Q3. Why each semantic slot selects one historical shot

Selecting one source for each semantic slot makes its historical reference explicit and limits cache size and potential conflicts between appearances from different viewpoints or poses. Different slots can still select different historical shots, allowing the current shot to combine people and backgrounds introduced separately. We agree that multiple views of the same person may provide complementary information. Aggregating several historical memories for one slot is a worthwhile extension for future work.

Q4. Attention masks and multi-person attribute-binding errors

Contrastive Slot Attention (CSA) subtracts the attention response to non-target content from the response to the target slot to identify the semantic region to retain. Morphological opening and closing remove isolated responses and improve the mask’s spatial coherence. Nevertheless, attention maps do not guarantee accurate instance separation, particularly when people overlap, subjects and backgrounds are semantically related, or a small object occupies only a few tokens.

Figure 12 in Appendix E illustrates this limitation through a clothing-attribute transfer error. The failure has two coupled causes: an imperfect mask may retain tokens from an adjacent person, leaving their historical appearances insufficiently separated; and a prompt involving several people and attributes can create competition for attention, causing an attribute to be assigned to the wrong person. More discriminative instance-level masks and explicit multi-person attribute binding are promising directions for addressing this failure mode.

Q5. Contribution of Semantic KV Masking and dependence on the LLM

We agree that, in the current quantitative results, removing semantic separation has a smaller effect on subject consistency than removing LLM routing. The two components serve different purposes: LLM routing determines which historical shot to retrieve, whereas the semantic KV mask determines which tokens within that shot can be reused. The component ablations in Section 5.3 show:

SettingInter-shot subjectInter-shot backgroundText alignment
Without LLM routing0.4420.7340.230
Without semantic separation0.5010.7780.220
Full method0.5130.8110.237

Figure 5 further compares direct concatenation (Concat), weighted fusion, and semantic KV masking. Direct concatenation is more prone to introducing irrelevant background content, whereas semantic masking better isolates the requested people and scene. Quantitative metrics may not fully capture such localized visual differences, so the numerical changes should be considered alongside the qualitative results. Together, the ablation scores and visual comparisons support the effectiveness of semantic KV masking.

Q6. Offset settings in Coverage-adaptive RoPE

When neither the people nor the background have historical references, or when only part of the requested content has a reference, the available history may not fully match the current instruction. We therefore use a larger temporal offset, J = 45, to limit interference from mismatched content. When both types of references are available, we set the offset within [1, 6] according to prompt-embedding similarity s:

J = round(1 + 5 × clip((0.99 − s) / (0.99 − 0.5), 0, 1))

A higher similarity places the relevant historical reference closer to the current queries. To evaluate this design, we change only the offset setting while keeping routing, semantic masking, memory decay, and eviction unchanged:

SettingIntra-shot subjectIntra-shot backgroundInter-shot subjectInter-shot backgroundText alignment
Uniformly large offset0.7760.9280.4690.7450.226
Coverage-adaptive RoPE0.7710.9230.5130.8110.237

A uniformly large offset reduces inter-shot subject and background consistency. A uniformly small offset causes visual drift and unnaturally saturated colors in the no-coverage and partial-coverage cases, so these visibly unstable settings are excluded from the quantitative table. These experiments demonstrate the effectiveness of our Coverage-adaptive RoPE design. The results have been added to Section 5.3, Ablation Study.

Thank you to the authors for addressing all my questions. My concerns have been fully resolved, and I am therefore willing to raise my score to 3.

Dear Reviewer PCSx,

Thank you for taking the time to read our rebuttal. We are glad that our responses addressed your concerns, and we greatly appreciate your updated score.

Best regards,
The Authors

Reviewer_Ld8FOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a training-free framework for memory weaving in interactive long video generation. The memory weaving relies on the proposed contrastive slot attention (CSA) to query the required context from the KV mask. The paper also uses LLM for semantic slot routing, which decomposes user prompts into character and background descriptions. In addition to that, the paper also proposes a variant of RoPE to address the temporal offsets and memory retention problems.

Critical strengths

The main strength of this work is the proposed contrastive slot attention (CSA), which ensures that the target prompt can activate both the requested component and its correlated surroundings accurately.

The other strength of this work is the proposed coverage-adaptive RoPE, which was used to regulate historical guidance.

The overall training-free framework could also be considered another strength of this work, as it proposes a memory weaving approach for cross-shot scenarios.

Critical weaknesses

The most critical weakness of this paper is the lack of video demonstration results. There is no supplementary package that includes such video demos. This is a big concern, as this work is about video generation. Merely using sample frames in Figure 4 is not sufficient. These results should be accompanied by individual video files.

Though the title of this paper states that this work is for interactive long video generation, the paper does not clearly present the solutions for this "interactive" and "long". "Interactive" is usually positioned in conversational video generation, where multiple subjects talk and interact with each other. The paper does not clearly show such interactions when designing the proposed method. In Figure 2, the interaction seems to be a simple concatenation of multiple subjects. Moreover, the paper does not show video generation results such as 30s, 60s, and 120s. It does not have to be strictly like that. But at least, the paper needs to show the results across different durations.

There is no ablation study regarding the coverage-adaptive RoPE, including no coverage. partial coverage, and full coverage. Their performance differences are unclear.

The proposed contrastive slot attention (CSA) is a major contribution of this work, and it relies on the morphological opening and closing (Soille et al., 1999), which are rather old operations. There are concerns about whether such morphological operations are effective in complex scenarios. The paper does not seem to present relevant analysis to show that these morphological operations are sufficient.

To illustrate the CSA, the paper could also benefit from providing more visual results regarding the use of token-level attention maps in Eqns. 6 and 7. Does that correspond to the attention maps given in top-right of Figure 2?

Some minor suggestions. Figure 1 is not a very intuitive way of showing the motivation for compositional memory reuse with simple selections of different figures.

Questions for authors

Why are there no video demo results included to accompany the visual comparisons in Figure 4? Similarly, that also applies to the Figure 5.

How to validate that the used morphological operations are sufficient to generate the accurate mask that is critical to the proposed framework?

What are the ablation study results for the coverage-adaptive RoPE?

What is the meaning of "interactive" video generation in this work?

What are the results for long video generations, and how can the proposed memory weaving specifically address that?

6 comments

We thank the reviewer for the questions and respond to each point below.

Q1. Video demonstrations for Figures 4 and 5

The static frames in the paper facilitate comparisons of appearance in selected shots, but they cannot fully convey motion or shot transitions. We have organized the complete videos corresponding to Figures 4 and 5 on an anonymous demonstration page. The Figure 4 videos compare methods under the same sequence of shot-level instructions, while the Figure 5 videos compare the cache-fusion strategies.

Q2. Morphological operations and semantic masks

Morphological opening and closing remove isolated responses and fill small holes, improving the spatial coherence of the masks and making them easier to inspect. We compared generation with and without these operations. The videos are difficult to distinguish visually, and the absolute changes in the evaluated metrics are all below 0.05.

The key mechanism in Contrastive Slot Attention (CSA) is the difference between the attention maps for the target slot and competing content, which helps identify the semantically relevant region. Morphological opening and closing are auxiliary post-processing steps, we include them to make mask construction more robust, they are not the key contribution of CSA.

Q3. Ablation of Coverage-adaptive RoPE

To evaluate Coverage-adaptive RoPE, we change only the temporal offset setting while keeping routing, semantic masking, memory decay, and eviction unchanged. Using a uniformly large offset reduces inter-shot subject consistency from 0.513 to 0.469 and inter-shot background consistency from 0.811 to 0.745:

SettingIntra-shot subjectIntra-shot backgroundInter-shot subjectInter-shot backgroundText alignment
Uniformly large offset0.7760.9280.4690.7450.226
Coverage-adaptive RoPE0.7710.9230.5130.8110.237

With a uniformly small offset, the no-coverage and partial-coverage cases exhibit visual drift and unnaturally saturated colors. We therefore exclude these visibly unstable settings from the quantitative table. The results and analysis have been added to Section 5.3, Ablation Study, of the revised manuscript.

Q4. Meaning of “interactive” video generation

By “interactive,” we mean that the system accepts new instructions shot by shot. After a shot has been generated, the user can specify the people, setting, or actions for the next shot. The system then retrieves relevant historical memories and generates the next shot accordingly. Compared with a setting in which a complete script is specified in advance and the video is extended along the timeline, our setting emphasizes recombining people and backgrounds from different earlier shots in response to new instructions during generation.

Q5. Long-video results and memory weaving

We further evaluated 120-second videos and added a comparison with the 60-second results to Appendix D.1 of the revised manuscript:

DurationIntra-shot subjectIntra-shot backgroundInter-shot subjectInter-shot backgroundAesthetic qualityImaging qualityText alignment
60 s0.7710.9230.5130.8110.6110.7500.237
120 s0.7800.8230.4830.7280.6110.7480.178

At 120 seconds, aesthetic quality, imaging quality, and intra-shot subject consistency remain close to the 60-second results, while intra-shot background consistency, inter-shot consistency, and text alignment decline. Comparison with other methods evaluated at 60 seconds in Fig.8 of the revised manuscript shows that our 120-second results remain competitive.

Autoregressive long-video baselines such as Self Forcing primarily emphasize continued use of historical context. Our memory weaving instead selects historical sources for the people and background required by the current instruction, exposes only the relevant historical KV tokens through semantic masks, and progressively decays and removes these references as the current shot develops. This allows the model to recall earlier content when needed and better preserve subject and background consistency across shots.

In addition to our responses to the reviewer’s questions above, we address the two points raised in Critical Weaknesses below.

1. Visualization of CSA attention maps

The top-right of Figure 2 illustrates the token-level attention maps used by CSA and the resulting semantic mask. To provide examples from generated videos, we have added two visualizations in Appendix B.4 Fig.6 of the revised manuscript. They show the character and background attention maps associated with Eqs. (6)–(8) and the resulting semantic KV masks. In these examples, the masks localize the intended regions, illustrating how they can reduce interference from unrelated content when generating a new shot.

2. Motivation illustrated in Figure 1

Thank you for the suggestion. Figure 1 illustrates the limitations of different ways to reuse historical memory. Reusing too much history can introduce irrelevant content, whereas selecting a single historical shot based on global similarity can omit a character or background required by the current instruction. Our method instead selects the relevant historical sources and semantic regions separately for the new shot, allowing the required content to be combined.

Dear Reviewer Ld8F,

Thank you for your thoughtful feedback. As the discussion period ends in three days, we would be grateful if you could review our rebuttal and check whether your concerns are addressed. Thanks again for your time.

Best regards, The Authors

Reviewer_7NxvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Weave Forcing is a training-free framework for reusing characters and backgrounds across video shots. Built on Self Forcing with Wan2.1-T2V-1.3B, it uses an LLM to route semantic slots to historical shots, contrastive attention to mask compressed KV memories, and coverage-adaptive RoPE with memory decay to regulate reference influence. Experiments on 100 six-shot narratives report improved cross-shot consistency over the evaluated baselines, with competitive visual quality and text alignment.

Critical strengths
  1. A well-motivated formulation of compositional recall. Selecting references separately for each character and background directly addresses a limitation of whole-shot retrieval. Combining this routing with selective KV access provides a coherent way to recover content from different historical shots while limiting unrelated content. This is a useful contribution to interactive video generation.
  2. Encouraging gains without additional training. The method achieves the highest reported cross-shot subject and background consistency, with competitive quality and text alignment. Removing LLM routing reduces these scores from 0.513/0.811 to 0.442/0.734, supporting the value of semantic source selection. These results provide evidence for the practical utility of the overall framework.
Critical weaknesses
  1. The subject metric takes maximum DINOv2 similarity without explicit one-to-one matching or a penalty for extra people. The cross-group duplicate suppression threshold of 0.5 is also unvalidated. Thus, appearance similarity may not reliably establish that every requested identity and its attributes are preserved. No human evaluation corroborates the ranking. Excluding runtime failures and undefined sequences without reporting valid sample counts further complicates comparison. This matters because correct composition is the central claim.
  2. The evidence for individual components is incomplete. Coverage-adaptive RoPE is a main contribution, but Tables 2-3 do not isolate positional adaptation from memory decay and eviction. The background consistency gain from semantic masking over direct concatenation is also small: 0.809 to 0.811. Results use one random seed and provide no uncertainty estimates. Consequently, the independent benefit of the positional policy and the reliability of smaller gains remain unclear.
Questions for authors
  1. Does the ranking persist under human assessment or identity-aware matching that checks all requested people and attribute assignments? Please justify the 0.5 duplicate suppression threshold and report valid sample counts for each method, including whether the ranking holds on a common evaluation subset.
  2. With routing, masking, decay, and eviction held fixed, how does coverage-adaptive RoPE compare with fixed small and large offsets? Please specify the offset mapping and key mask parameters, and clarify whether the main gains persist across random seeds.
  3. What is the measured time from a new user instruction to the first generated frame, including LLM calls, mask construction, and memory transfer? Appendix A4 gives only a coarse per-call estimate, leaving actual interactive responsiveness unclear.
3 comments

We thank the reviewer for the constructive questions. We respond to the evaluation protocol, Coverage-adaptive RoPE, and inference cost below.

Q1. Subject-consistency evaluation

We acknowledge that the YOLO11n- and DINOv2-based metric cannot fully verify whether every target person appears or whether attributes are assigned to the correct identities. Figure 4 provides complementary qualitative evidence. We inspected the person segmentation masks when designing the metric and introduced cross-group duplicate suppression: if the DINOv2 similarity between reference persons assigned to different groups exceeds 0.5, the latter group receives a score of zero. This prevents repeated generation of the same person from artificially increasing the score.

We have added the scores and valid-video and valid-sample counts for three thresholds to Appendix C.2, Table 5:

Method0.3 ScoreValid videosValid samples0.5 ScoreValid videosValid samples0.7 ScoreValid videosValid samples
EchoShot0.318941390.338941530.35194159
CineTrans0.305971360.354971630.35697165
Self Forcing0.356961040.405961090.42596129
LongLive0.459971060.489971150.53297130
Rolling Forcing0.389981380.435981610.44598167
Infinity-RoPE0.445991210.495991490.51999166
Reward Forcing0.442971090.472971220.50997136
ShotStream0.385971280.478971500.46897167
Echo Forcing0.386981410.431981620.43198167
Ours0.447991350.513991560.53499166

Our method ranks first at thresholds 0.5 and 0.7 and second at 0.3. Thus, while the scores vary with the threshold, the conclusion that our method achieves strong cross-shot subject consistency remains supported.

Q2. Coverage-adaptive RoPE

We compare offset settings while keeping routing, semantic masking, memory decay, and eviction unchanged. The uniformly large-offset result has been added to Section 5.3, Ablation Study, under “Coverage adaptive RoPE” in Table 3.

SettingIntra-shot subjectIntra-shot backgroundInter-shot subjectInter-shot backgroundText alignment
Uniformly large offset0.7760.9280.4690.7450.226
Coverage-adaptive RoPE0.7710.9230.5130.8110.237

A uniformly large offset reduces inter-shot subject and background consistency. With a uniformly small offset, the no-coverage and partial-coverage cases exhibit visual drift and unnaturally saturated colors; we therefore exclude these visibly unstable settings from the quantitative table.

Under full coverage, prompt similarity maps the temporal offset to [1,6]: as similarity increases from 0.5 to 0.99, the offset decreases from 6 to 1. The semantic mask uses a binarization threshold of 0.6, followed by morphological opening and closing with a kernel size of 3 on a 30 × 52 token grid.

Q3. Inference cost

We have added the inference time for a complete 60-second video to Appendix A.4, Table 4:

ComponentValue
LLM calls106.34 s
Generation throughput15.42 FPS
Video denoising62.26 s
Total inference time183.87 s

The autoregressive denoising stage retains a throughput of 15.42 FPS and takes 62.26 seconds for the 60-second video. LLM calls take 106.34 seconds and constitute the largest measured component of the total inference time.

Thank you to the authors for their clarifications in the rebuttal. The additional experimental results have addressed my concerns, and I am happy to maintain my score of 3.

Dear Reviewer 7Nxv,

Thank you for reviewing our rebuttal and the additional experimental results. We greatly appreciate your time and thoughtful consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.