acceptodds
Under review as a conference paper at ICLR 2027

EncoreMem: Re-encounter Verification of Region Memory for Training-Free Long Video Generation

Abstract

Autoregressive video models generate a long video one chunk at a time, conditioning each chunk on a cache of what has already been generated. Keeping the earliest frames in this cache helps limit degradation. These frames hold only what appeared at the start, so the rest of the cache must decide which later content to keep and for how long. Existing methods choose what to keep by its current or predicted relevance, or by learned representations. We instead keep stored content only if later frames, which also follow the current prompt, return to it. We introduce EncoreMem, which stores newly appearing parts of frames after they persist over several chunks and keeps each part provisionally. Whether a part returns is detected with the features of the generator, and when memory fills, the part that returned least often in recent chunks is replaced. The rule needs no training, additional model, retrieval at read time, or cache rebuild when the prompt changes. In minute-long video generation with both multi and single prompts, EncoreMem achieves competitive overall performance.

Primary area:
generative models
Keywords:
long video generation, autoregressive video diffusion, KV cache, memory, training-free, streaming generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dsjrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a KV-cache management method to efficiently maintain region-level consistency in both single-prompt and multi-prompt cases. The author novely propose to perform region-based KV cache selection for long-term consistency.

Critical strengths
  • The region-based memory selection is simple and reasonable.
  • The method is training-free so can be applied to any autoregressive foundation models.
Critical weaknesses
  • the quantitative results shown in Tab.1 is no solid. Self-forcing has highest consistency score. Other metrics is only comparable with Deep Forcing, no significant improvements.
  • The author claims the method can work under multi-prompt settings。 But qualitative figures in both paper and appendix don’t really show the ability to maintain consistency in response to prompt change。 I suggest the author to add results such as 'a man stay in the room first, then go out in a garden, and come back to the home' to see whethe the settings in the home stays the same.
Questions for authors
  • What is the extra cost time during inference compared to baseline, e.g., self-forcing? As fewer tokens are kept but a selection process added, whether the inference latency increased or decreased.
  • The core difference between ENCOREMEM and Deep Forcing? Deep Forcing keep Top-K similar tokens in KV-cache. If the targeted region repeatedly occur, it should also be kept in DeepForcing’s KV cache. Can the author provide some visual comparison on the tokens being kept by ENCOREMEM and Deep Forcing?
6 comments
AuthorsOpenReview ↗

Response to the Reviewer dsjr [1/3]

Dear Reviewer dsjr, The authors sincerely appreciate your concrete suggestions, especially the returning-scene example, which helped us test the method where memory matters most. Your questions on Table 1, on returning scenes and on the difference from Deep Forcing helped the authors add the missing evidence for the main claim. We are also thankful for your acknowledgement of the strengths of this work, namely region-based memory selection that is simple and reasonable, and a training-free design applicable to autoregressive foundation models. We address your concerns and questions in the threads below.

W1) Comparison with Self Forcing and Deep Forcing in Table 1.

Answer1: The Consistency of Self Forcing [G1] is high because its frames barely change over a long video, which does not indicate a better long video.

VBench [G6] consistency measures how similar the frames within a two-second window are, so a video that changes less scores higher, whether or not the video follows the prompt. Self Forcing is the base generator for autoregressive video generation and is trained on clips of 7 chunks (about 5 s). A one-minute rollout goes far beyond this length, so the generator keeps redrawing the scene that has already been drawn. The frames therefore change less and less, which raises Consistency. In Table D, Self Forcing changes least over a multi-prompt rollout although the prompt changes five times (Subject -0.34, Background -0.17), and its Aesthetic drops most (-7.80).

In the multi-prompt setting of Table 1, this means that the frames keep the earlier scene instead of following the new prompt (Figure 1). Prompt agreement falls by 10.7 points from the first to the last segment (from 26.06 to 15.38), against 1.8 points for EncoreMem (Table 1). This is why the base generator is not ranked in Table 1 (Sec. 4.2). Its Aesthetic and Quality are also the lowest among the compared methods (57.39 and 80.88).

Answer2: Against Deep Forcing [G2], EncoreMem is higher on most metrics in each of the 3 seeds, with about half the attended tokens. Over 3 seeds, EncoreMem also keeps prompt agreement best among all methods (Δ6−1\Delta_{6-1}).

In Table 1 (one seed), the paired 95% intervals of the differences in Consistency and Aesthetic over the 100 prompts exclude zero (Sec. 4.2, L315-316).

Table R9. Comparison over 3 seeds. The multi-prompt block has all methods (Self Forcing on 1 seed), and the single-prompt block has Deep Forcing. Runs beyond those of the paper are run for this response. * means that the paired 95% interval of the difference from EncoreMem excludes zero. Multi-prompt metrics are as in Table 1 (prompt agreement by ViCLIP [G7]), single-prompt metrics are VBench dimensions, and CLIP-T uses CLIP [G8].

Multi-promptConsistencyAestheticQualityViCLIP 1ViCLIP 2ViCLIP 3ViCLIP 4ViCLIP 5ViCLIP 6Δ6−1\Delta_{6-1}
Self Forcing97.20*57.39*80.88*26.0622.79*21.11*18.48*17.13*15.38*-10.68*
MemFlow96.8859.03*81.29*26.4124.57*23.52*23.3623.9023.20*-3.21*
LongLive96.7960.6781.86*26.3125.1924.5324.33*24.8123.99*-2.32*
Deep Forcing96.13*60.02*83.32*26.1625.10*24.29*23.7623.97*24.08*-2.08*
EncoreMem (Ours)96.7460.8783.1126.2025.5324.6523.8324.4524.61-1.60
Single-promptDynamicImagingMotionBackgroundAestheticSubjectOverallCLIP-T 60-0 s
Deep Forcing64.84*68.39*97.73*95.57*59.81*95.86*23.60*-1.34*
EncoreMem (Ours)48.3470.0798.2596.2561.0097.0023.94-0.93

In each of the 3 seeds EncoreMem is higher than Deep Forcing in Consistency, Aesthetic, prompt agreement in segments 2, 3, 5 and 6, Δ6−1\Delta_{6-1} and each single-prompt metric except Dynamic degree.

Deep Forcing has a higher Dynamic degree (64.84 vs. 48.34), but this motion comes with lower prompt agreement. The prompt agreement of Deep Forcing falls more as the video goes on, both with a single prompt (CLIP-T 60-0 s of -1.34 vs. -0.93) and with multiple prompts (Δ6−1\Delta_{6-1} of -2.08 vs. -1.60), and both differences are significant (Table R9). Over 3 seeds, EncoreMem also keeps prompt agreement better than the training-based LongLive [G3] and MemFlow [G4] (Δ6−1\Delta_{6-1} of -1.60 vs. -2.32 and -3.21), with higher Quality and Consistency on par.

EncoreMem achieves this with about half the attended tokens of Deep Forcing (14,157 vs. 29,640) and less GPU memory (17.4 GiB vs. 20.9 GiB). EncoreMem also keeps higher quality over 240 s (Reviewer Ws4y, Table R8) and is significantly better when earlier prompts return (W2).

We have added these new results with accompanying discussions in Table F in App. E.4 of the revised PDF.

AuthorsOpenReview ↗

Response to the Reviewer dsjr [2/3]

W2) Does a scene stay the same when a prompt change brings it back?

We sincerely thank you for proposing a scenario that tests the point where the memory should matter most.

Answer: In the suggested scene, when the video leaves the room and comes back, the room stays more consistent than with Deep Forcing. On a benchmark where each of six scenes returns 30 s later, EncoreMem also keeps the background and faces of returning scenes significantly closer to their first appearance than Deep Forcing.

For the suggested scene, we wrote scripts in which a person is in a room, goes out to another place for 30 s, and comes back to the room. The places include a market, a subway station, a forest and a beach (all runs for this response). When the videos of both methods leave the room and return, the room on return is significantly closer to the first room for EncoreMem than for Deep Forcing. The background similarity with the person masked is 0.768 vs. 0.681. Figure F shows an example. Deep Forcing returns to a different room. Without memory, the video relies on recent frames, so the market stalls remain in the room on return. A memory of whole frames keeps the room when the prompt asks for the market. EncoreMem goes to the market, comes back, and keeps the first room.

To test returns at scale, we show the six prompts of each multi-prompt script for 5 s each and then again in the same order (12 segments, 11 switches, 30 scripts), so each prompt returns 30 s later, after five others. For example, one script follows a detective in a rain-soaked alley through six prompts (examining a footprint, meeting an informant, receiving a note, reading the clue, watching the informant leave, and standing up with the clue), and the same six prompts are then shown again.

Table R10. Returning prompts (30 scripts, 3 seeds, all runs for this response). * means that the paired bootstrap 95% interval of the difference from Deep Forcing excludes zero. Prompt agreement is the ViCLIP score, background similarity is the cosine similarity of DINOv2 [G10] features with detected persons masked, and face identity is the ArcFace [G9] similarity of detected faces.

MethodPrompt agreement (ViCLIP, 12 segments)Background similarity of returning segmentsFace identity of returning segments
Deep Forcing22.780.8840.393
EncoreMem (Ours)23.47*0.925*0.429*

EncoreMem follows the prompts better than Deep Forcing and keeps the background and face of a returning segment closer to their first appearance.

On the benchmark of Table 1, comparing the face in the last segment with the first (ArcFace, 47 prompts with a detected face), EncoreMem keeps the face better than Deep Forcing and as well as LongLive and MemFlow, while its prompt agreement declines less than that of LongLive and MemFlow (Table R11).

Table R11. Face identity and prompt agreement on the 47 prompts of Table 1 with a detected face. All methods use 3 seeds except Self Forcing, which uses 1. EncoreMem and the added seeds are run for this response. * means that the paired 95% interval of the difference from EncoreMem excludes zero.

MethodIdentity, segment 6 to segment 1ViCLIP 2-6Δ6−1\Delta_{6-1}
Self Forcing0.105*19.46*-10.38*
MemFlow0.39523.63*-3.62*
LongLive0.39624.47-2.48*
Deep Forcing0.368*24.44-1.73
EncoreMem (Ours)0.39924.59-1.55

We have added these new results with accompanying discussions in Table 5 in Sec. 4.6, Figure F in App. E.5 and Table G in App. E.6 of the revised PDF.

AuthorsOpenReview ↗

Response to the Reviewer dsjr [3/3]

Q1) Does inference cost increase or decrease relative to Self Forcing?

Answer: Thanks to region-level storage, EncoreMem uses less GPU memory than Self Forcing (17.4 GiB vs. 19.5 GiB at peak), with latency similar to Self Forcing.

We measured all methods on the same GPU with 60-s clips. EncoreMem reads fewer than half the tokens that Self Forcing reads (14,157 vs. 32,760 at chunk 40), which shortens the forward passes of the generator (57.3 vs. 70.1 s per clip). Storing, verification, replacement and capture together add 7.5 s per clip (verification alone 1.3 s, measured in a separate run with timing enabled). EncoreMem also needs the least GPU memory during generation (17.4 GiB vs. 19.5 GiB for Self Forcing).

We have added these new results with accompanying discussions in Table 6 in Sec. 4.6 and Table E in App. E.3 of the revised PDF (see also the cost summary in the General Response).

Q2) Deep Forcing would also keep a region that occurs repeatedly. What is the core difference, and can the kept tokens be compared visually?

Answer1: Deep Forcing keeps what the current chunk attends to, while EncoreMem keeps what later frames actually come back to.

We agree that Deep Forcing keeps a recurring region whenever the current queries attend to it. Deep Forcing ranks tokens by the attention of the recent queries and recomputes this at each compression, so content the current chunk does not attend to is dropped even if it appeared many times before. What will be needed again cannot be known when content first appears. EncoreMem therefore stores each newly appearing region first and confirms the region once later frames show the region again. This check uses only frames that have already been generated. A confirmed region then stays available while it is out of view, regardless of the attention of the current chunk, because Re-encounter Replacement removes a confirmed region only when the memory is full and a new region needs the slot (Sec. 3.4).

This difference shows when an earlier prompt returns. On the returning-prompt benchmark of W2, none of the tokens Deep Forcing selects by attention (Top-C) come from the segment where the content first appeared, and 69% come from the previous segment, which shows a different prompt and is no longer relevant. In contrast, EncoreMem keeps the background and faces of returning segments closer to their first appearance (Table R10). Within a single prompt, Deep Forcing also drifts further from the prompt over a minute (CLIP-T 60-0 s of -1.34 vs. -0.93 over 3 seeds, Table R9).

Answer2: Deep Forcing keeps scattered fragments, while EncoreMem stores a few connected regions (Figure E). This matches the returning-scene results of W2, where the room stays more consistent with EncoreMem (Figure F).

Figure E in App. E.4 shows, at chunks 20, 30 and 40 on the same prompt and seed, the tokens Deep Forcing keeps and the regions EncoreMem stores. In this figure, the 3,120 tokens kept by Deep Forcing at chunk 40 are scattered over 1,685-2,505 connected pieces (the largest covering at most 8.8% of them), while EncoreMem stores 5-9 regions forming 4-11 pieces (the largest covering 17-43%).

We have added these new results with accompanying discussions in Figure E in App. E.4 of the revised PDF.

Thank you for the insightful questions, which have helped us strengthen the comparison with Deep Forcing and the evaluation of returning scenes.

Reviewer_Ws4yOpenReview ↗
Rating
4
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

EncoreMem is a training-free memory mechanism for long video generation. It provisionally stores newly appearing regions after they persist across several chunks, then uses subsequent reappearances to guide retention and replacement. It requires no additional model or training and achieves competitive performance on minute-long videos with single or multiple prompts.

Critical strengths
  1. The paper addresses the practical challenge of preserving historical content while allowing scenes and actions to evolve. In the scene-recall experiments, the full method shows less layout repetition than the frame-memory baseline.

  2. The method clearly separates memory admission, verification, and replacement, allowing initial storage decisions to be revised using subsequent re-encounter evidence.

  3. The framework requires no additional training or auxiliary models and uses representations from the frozen generator. Prompt changes do not require explicit cache reconstruction.

  4. The experiments cover single-prompt, multi-prompt, and scene-recall settings, with competitive overall performance. Memory visualizations and event statistics illustrate the method’s behavior, and the paper discusses trade-offs among consistency, motion, and prompt adherence.

Critical weaknesses
  1. Comparisons with simpler memory policies are missing.

Additional comparisons under the same region representation and cache budget would help clarify the benefit of re-encounter-based retention and replacement.

  1. The benefit of re-encounter verification requires clearer evidence.

In Table 2, removing RV leads to relatively small changes in prompt alignment and motion, leaves layout repetition unchanged, and increases raw crop similarity. Although RV changes memory release and replacement behavior, its contribution to final generation quality remains insufficiently demonstrated. The w/o RV configuration also retains re-encounter-based replacement, so it does not test the full contribution of re-encounter evidence.

  1. Generality and computational efficiency require further evaluation.

Quantitative experiments focus on a single generator and sixty-second videos, while longer videos are evaluated qualitatively. Comparisons across generators and generation lengths, together with runtime and memory measurements, would clarify the method’s applicability and computational cost.

Questions for authors

Please refer to the Critical Weaknesses section above.

7 comments
AuthorsOpenReview ↗

Response to the Reviewer Ws4y [1/4]

Dear Reviewer Ws4y, The authors sincerely appreciate your positive assessment and precise suggestions, which directly shaped our additional experiments. Your requests for simpler baselines and clearer evidence on re-encounter verification helped the authors separate what the memory stores from how it decides what to keep. We are also thankful for your acknowledgement of the strengths of this work, namely the practical problem of keeping past content while scenes evolve, a clear separation of admission, verification and replacement, no training or cache reconstruction at prompt changes, and broad experiments with memory visualizations. We address your comments in the threads below.

W1) Comparison with simpler memory policies to clarify the benefit of re-encounter-based retention and replacement.

Answer1: As suggested, we compared our rule with simpler replacement rules under the same regions and budget. Without re-encounter verification, more slots are occupied by regions that do not return and need not be remembered. With re-encounter verification (RV), these regions are released, and their slots go to regions that appear again and are worth remembering (W2, Table R7).

With verification disabled, we replace by FIFO or at random. With storage and verification kept, we replace by LRU or LFU of re-encounters.

Table R4. Replacement rules (scene recall on 30 clips and multi-prompt on 40 prompts, seed 1). The scene-recall values of the 14-slot EncoreMem and w/o RV rows are those of Table 2, and the other values are run for this response. * means that the paired 95% interval of the difference from EncoreMem with the same number of slots excludes zero. The multi-prompt values of LRU and LFU at 4 slots come from a second run with different initial noise, so they are compared with the EncoreMem row of that run. On scene recall, crop similarity and layout repetition use DINOv2 [G10] features, optical flow uses RAFT [G11], Dynamic is the VBench [G6] dynamic degree, and ViCLIP [G7] is the agreement with the repeated prompt. The multi-prompt metrics are those of Table 1.

SlotsRuleRecall crop sim.Recall flowRecall DynamicRecall ViCLIPRecall layout rep. ↓\downarrowMulti ConsistencyMulti AestheticMulti QualityMulti ViCLIP 2-6Multi Δ6−1\Delta_{6-1}
14EncoreMem (RV + re-encounter)0.6154.3828.3310.230.68896.7060.4482.7624.15-1.44
14w/o RV (re-encounter only)0.6324.3226.3310.000.68896.7160.2682.7324.24-1.41
14FIFO0.6444.6323.6710.340.68496.7160.2482.6924.22-1.57
14Random0.6124.1726.0010.410.68396.7760.2582.6624.21-1.56
14LRU (with RV)0.6404.8727.6710.050.68396.6760.4482.8024.16-1.61*
14LFU (with RV)0.6075.0428.0010.320.68296.7060.3282.62*24.10-1.59
4EncoreMem (RV + re-encounter)0.6082.7822.679.050.69296.9460.8182.4624.33-1.18
4w/o RV (re-encounter only)0.6282.7726.009.260.69196.9660.6782.4624.28-1.36
4FIFO0.5962.9621.678.660.68996.9060.8882.4324.17-1.74
4Random0.6233.2025.339.580.68796.9160.8182.5224.17-1.46
4EncoreMem (second run)-----97.0260.8982.5823.74-1.69
4LRU (with RV)0.6092.7624.009.060.69296.99*60.9082.5823.75-1.62
4LFU (with RV)0.6272.7223.009.350.69197.0060.9982.5723.80-1.73

The simpler rules did not significantly improve on ours in our tests. With 14 slots, LRU and LFU are each significantly lower on one metric. Without RV, more slots are occupied by regions that do not return (W2).

We have added these new results with accompanying discussions in Table J in App. E.9 of the revised PDF.

AuthorsOpenReview ↗

Response to the Reviewer Ws4y [2/4]

W1) Comparison with simpler memory policies to clarify the benefit of re-encounter-based retention and replacement (continued).

Answer2: Regions are the better choice. With the same number of tokens, regions give higher quality and prompt agreement than more recent frames and repeat earlier layouts less than whole frames.

We compare regions with two simpler choices. The first spends the same attended tokens on more recent frames (3 sink and 7 recent frames, S3/R7), and the second stores whole frames in the memory (frame memory). Table R5 shows the results.

Table R5. What the memory stores (same attended tokens, hence the same attention cost). ViCLIP 2-6 is the mean prompt agreement of segments 2-6, after the first switch. Δ6−1\Delta_{6-1} and CLIP-T 60-0 s are the last value minus the first, and closer to zero means less forgetting of the prompt. CLIP-T uses CLIP [G8], and the other metrics are as in Table R4.

Multi-prompt (100 prompts, seed 1, Regions as in Table 1 and S3/R7 run for this response)

StoredConsistencyAestheticQualityViCLIP 2-6Δ6−1\Delta_{6-1}
Regions (EncoreMem)96.8160.9583.0124.42-1.80
Recent frames (S3/R7)96.9859.3882.3423.76-2.94

Single-prompt (128 prompts, seed 1, both rows run for this response)

StoredDynamicImagingMotionBackgroundAestheticSubjectOverallCLIP-T 60-0 s
Regions (EncoreMem)49.6470.3298.2496.2460.7696.9523.78-0.82
Recent frames (S3/R7)49.5368.5598.0696.1658.7896.9423.58-2.19

Scene recall (30 clips) and lingering (20 prompts in which an object leaves). The scene-recall rows are those of Table 2, and lingering is run for this response. Crop similarity rises as motion falls (Sec. 4.4). Lingering is the rate at which the object is drawn again.

StoredCrop sim.Optical flowDynamicViCLIPLayout repetition ↓\downarrowLingering ↓\downarrow
Regions (EncoreMem)0.6154.3828.3310.230.6880.61
Whole frames (frame memory)0.7252.3427.676.410.8120.89
Nothing (w/o memory)0.6711.9624.008.990.7080.74

At the same tokens, regions improve Aesthetic, Quality and prompt agreement after the first switch (ViCLIP 2-6). Regions also keep the prompt over a single-prompt minute better than recent frames (CLIP-T 60-0 s of -0.82 vs. -2.19). The paired 95% intervals of these differences exclude zero. S3/R7 has slightly higher Consistency because more recent frames make neighboring frames more alike.

Compared with whole frames, regions follow a repeated prompt more (ViCLIP 10.23 vs. 6.41), copy earlier layouts less (0.688 vs. 0.812), keep more motion (optical flow 4.38 vs. 2.34) and draw a departed object again less often (0.61 vs. 0.89). Frame memory has the highest crop similarity because it moves least.

We have added these new results with accompanying discussions in Table I in App. E.8 of the revised PDF.

AuthorsOpenReview ↗

Response to the Reviewer Ws4y [3/4]

W2) What is the contribution of re-encounter verification (RV)?

Answer: RV does not change average quality (Sec. 4.4). The role of RV is to keep the memory up to date. RV releases content that does not return, so those slots stay available for content that does.

We agree that the Table 2 ablation cannot show the contribution of RV, for two reasons. First, w/o RV keeps re-encounter-based replacement. Second, with 14 slots replacement is rare (Table 3).

We therefore did three things. (a) We removed re-encounter evidence entirely, using FIFO and random replacement with verification disabled (Table R4). (b) We varied the budget (Table R6). (c) We measured the failures RV is designed to prevent, namely lingering (Table R6) and a stress test in which a person leaves while several objects pass through (Reviewer iJjo, Table R1).

Table R6. RV on and off across budgets and targeted measurements (seed 1). The recall ViCLIP of the 14-slot rows is that of Table 2, and the other values are run for this response. * means that the paired 95% interval of the on-off difference excludes zero.

SlotsRVRecall ViCLIPMulti ViCLIP 2-6Lingering ↓\downarrow
14on10.2324.150.61
14off10.0024.240.63
7on10.8224.090.61
7off10.5524.210.64
4on9.0524.330.63
4off9.2624.280.62
2on9.7324.110.73
2off9.2824.130.73

We also measured what the memory holds on the standard benchmarks (Table R7). These are the slots held by regions that do not come back and how often stored regions are replaced.

Table R7. Memory state with RV on and off on the standard benchmarks (14 slots, all rows run for this response, * as above).

BenchmarkPromptsSeedsRVSlots held by regions not re-encountered ↓\downarrowStored regions re-encountered within the last 8 chunksReplacements per clip ↓\downarrow
Multi-prompt (Table 1)1001on0.24*0.918*2.83*
Multi-prompt (Table 1)1001off0.310.8793.68
Multi-prompt (subset of Table 1)408on0.25*0.921*2.66*
Multi-prompt (subset of Table 1)408off0.330.8854.14
Scene recall (Table 2)303on0.29*0.932*4.83*
Scene recall (Table 2)303off0.350.8986.71
Returning prompts (Table R10)301on0.25*0.934*2.60*
Returning prompts (Table R10)301off0.340.8844.30

RV keeps the memory up to date on the four benchmarks we tested (Table R7). Fewer slots are held by regions that are not re-encountered (0.24 vs. 0.31 on the multi-prompt benchmark of Table 1). More stored regions were re-encountered recently (0.918 vs. 0.879), and stored regions are replaced less often (2.83 vs. 3.68 replacements per clip). This holds over 8 seeds, on scene recall (Table 2) and on returning prompts (Reviewer dsjr, W2).

The difference grows when the memory is contested (Reviewer iJjo, Table R1). Without RV, passing objects hold more slots (6.25 vs. 4.85 of 14, and 5.65 vs. 4.65 of 7). More stored regions are also replaced at each budget (1.60, 1.85, 3.55 and 3.95 more per clip at 14, 10, 7 and 4 slots). The paired 95% intervals exclude zero. With RV, the person region is kept in more clips at 14, 10 and 7 slots (15/16 vs. 13/16, 10/15 vs. 9/16, and 6/16 vs. 3/16).

RV does not change how often a departed object is drawn again at the budgets we tested (Lingering in Table R6, 0.61 vs. 0.63 at 14 slots).

We now state this role of RV in Sec. 4.4 (L456-462) of our revised PDF, and we have added these new results with accompanying discussions in Tables L and M in App. E.10.

Reviewer_iJjoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents EncoreMem, a training-free memory mechanism for autoregressive long video generation. It stores newly appearing regions rather than complete frames, provisionally retains them, and uses later re-encounters in the generator's key space to confirm or replace stored content. Experiments show competitive generation quality and less layout repetition than baselines.

Critical strengths
  1. I think storing regions rather than whole frames is an interesting idea, and it is surprising to see that even if you drop tokens, which actually puts inference outside the model's training distribution, it still works.

  2. The method uses only the frozen generator's features and requires no additional external recognition model. The experiments include reasonable baselines, and the overall results are competitive.

Critical weaknesses
  1. The first concern is whether recent recurrence is a golden policy: a background may appear frequently, while an important character may disappear for a long interval before returning and risk being forgotten.

  2. This method is training-free, and a big concern is the scalability (usefulness?) of this method itself. Can this method be applied to other causal models or just the Self Forcing backbone model specifically? And how to apply this approach during training, and whether there are benefits to applying this method during training, are unknown.

Questions for authors

I was expecting that dropping tokens might improve the throughput a bit. Could this paper focus on this part, for example by measuring how many tokens you can drop without a large degradation in performance?

6 comments
AuthorsOpenReview ↗

Response to the Reviewer iJjo [1/3]

Dear Reviewer iJjo, The authors sincerely appreciate your thoughtful review and the two concerns you raised, which led directly to the new experiments below. Your questions on subjects that stay absent for a long time and on generality beyond Self Forcing helped the authors test the memory where it is most likely to fail and show that it carries over to other generators. We are also thankful for your acknowledgement of the strengths of this work, namely storing regions rather than whole frames as an interesting idea that works even when tokens are dropped, the use of only the features of the frozen generator, and reasonable baselines with competitive results. We address your concerns and questions in the threads below.

W1) A background recurs often, while an important subject may be absent for a long interval. Does re-encounter-based retention keep such a subject rather than the background?

Answer: A recurring background does not take over the memory, and a confirmed subject (a subject seen again in at least 6 of the 8 chunks after storage) is not removed by absence alone. At the default budget, the person region stays in memory in 15 of 16 stress-test clips.

A region is stored only for content that the sink [G12] and the stored regions do not already explain (Sec. 3.2), so a recurring background is not stored again and again. On the multi-prompt benchmark (Table 1), background regions hold 2.42 of the 7.95 occupied slots on average, and their share falls from 38% in the first segment to 27% in the last.

A stored region is checked over the 8 chunks (about 6 s) after storage. A region seen again in at least 6 of these chunks is confirmed, and a region seen in fewer chunks is released. Absence alone does not remove a confirmed region, which becomes a candidate for replacement only when a new region must be stored into a full memory (Sec. 3.4). Content that leaves the view during these 8 chunks and is seen in fewer than 6 of them is therefore treated like passing content, and the content is stored again if the content returns. We now state this in App. D.

We measured how much stored content survives an absence in an ordinary case and in a hard case. The ordinary case is the scene recall benchmark of the paper (Table 2, A-B-C-A'-B'-C'), where a scene returns about 20 s later. There, 81 of the 97 regions stored during B and 59 of the 85 stored during C were still stored and read when the scene returned, so most of what is stored for a scene stays in memory while that scene is absent. The hard case is a stress test in which other content competes for the slots of a confirmed subject. A person leaves for 30-40 s while three other objects pass through one after another. For example, a woman sits on a bench reading a book on a park path and then walks out of the frame, a dog, a cyclist and a flock of birds pass one after another, and the woman comes back to the bench in the last segment.

Table R1. Stress test (20 scripts, seed 1, all rows run for this response). Counts are over the clips in which a region of the person was stored before it left. With re-encounter verification (RV) on and off, these are 16 and 16 of 20 clips at 14 slots, 15 and 16 at 10, 16 and 16 at 7, and 13 and 13 at 4. * means that the paired 95% interval of the on-off difference excludes zero.

SlotsRVPerson region kept through the absenceSlots held by passing objects when the person returns ↓\downarrowReplacements per clip ↓\downarrow
14 (default)on15/164.85*1.10*
14off13/166.252.70
10on10/155.004.00*
10off9/165.705.85
7on6/164.65*8.95*
7off3/165.6512.50
4on1/133.5510.25*
4off0/133.6014.20

At the default budget the region of the person stays in memory through the whole absence in 15 of 16 clips. The person stays in view for more than the 8 chunks after storage, so the region of the person is confirmed. A passing object is seen in fewer than 6 of the 8 chunks before leaving, so RV releases the region of the object. The person is lost more often only when the memory is cut below the default (kept in 10/15, 6/16 and 1/13 clips at 10, 7 and 4 slots), and without RV more stored regions are replaced at each budget. This shows that re-encounter-based retention keeps a subject through a long absence when the memory has room, and that RV keeps this room by releasing the passing objects.

We have added these new results with accompanying discussions in Table L in App. E.10 of the revised PDF.

AuthorsOpenReview ↗

Response to the Reviewer iJjo [2/3]

W2) Can EncoreMem be applied to other causal models and used during training?

Answer1: Added without retraining to CausVid [G5], a causal model trained with a different procedure from Self Forcing, and to Deep Forcing [G2], which uses a different cache policy, EncoreMem improves both.

Most recent long-video methods, including all our baselines (LongLive [G3], MemFlow [G4], Deep Forcing), are built on Self Forcing [G1], which is why the main experiments use it. To go beyond Self Forcing, we add EncoreMem unchanged, with the thresholds calibrated on Self Forcing, to CausVid (a causal model distilled with a different procedure) and to Deep Forcing (same weights, different cache policy).

Table R2. EncoreMem added at inference time, without retraining (single-prompt, 128 prompts). CausVid uses seed 1 and Deep Forcing 3 seeds. All rows are run for this response except Deep Forcing native. * means that the paired 95% interval of the difference between the two settings of a model excludes zero, marked on the higher one. The first seven columns are VBench [G6] dimensions, and CLIP-T 60-0 s is the CLIP [G8] score of the last frame minus that of the first frame.

ModelSettingDynamicImagingMotionBackgroundAestheticSubjectOverallCLIP-T 60-0 s
CausVidnative44.5865.8398.1597.05*59.7597.9921.71-2.14
CausVid+ EncoreMem55.86*67.58*98.2396.6159.7697.9722.87*-1.04*
Deep Forcingnative64.8468.3997.7395.5759.8195.8623.60-1.34
Deep Forcing+ EncoreMem (same budget)64.8368.75*97.7295.6460.10*96.02*24.00*-0.66*

On CausVid our memory raises Dynamic, Imaging and Overall and halves the loss of prompt agreement over a minute (-1.04 vs. -2.14), with Background consistency 0.4 points lower as Dynamic rises by 11 points. On Deep Forcing our memory keeps the cache budget and the motion (Dynamic 64.83 vs. 64.84) while improving Imaging, Aesthetic, Subject, Overall and prompt retention.

More generally, EncoreMem needs only what a KV-cache generator already computes, namely the clean latents of each frame, the keys and values of one layer, and control over the rotary positions of cached tokens. EncoreMem can therefore be added to a causal generator as long as its weights are not trained for a specific cache. LongLive and MemFlow are fine-tuned together with their own caches (a frame sink with KV re-caching at prompt switches, and a retrieved memory bank), so we compare with them as baselines.

Answer2: EncoreMem is designed to be training-free, and it is effective on several models without any training.

Our goal is a training-free method, and EncoreMem is designed for it. As the answer above and the paper show, applying our cache management to several models without training is effective (Table R2). As the reviewer points out, changing the memory management during training in the way our method does is a meaningful research direction, and the fact that EncoreMem already works in a plug-and-play, training-free way shows its potential. As it goes beyond the goal and scope of this work and may need a different design, we would like to leave it as future work.

We have added these new results with accompanying discussions in Table 7 in Sec. 4.6 of the revised PDF.

AuthorsOpenReview ↗

Response to the Reviewer iJjo [3/3]

Q) Does dropping tokens improve throughput, and how many tokens can be dropped without a large loss in quality?

Answer: EncoreMem reads fewer than half of the tokens that Self Forcing reads, which shortens the forward passes by about 18%. Fewer memory slots save little more time and make returning scenes repeat earlier layouts more, so we use 14 slots.

Because EncoreMem stores only regions of newly appearing content, it attends to the fewest tokens among the compared methods (General Response). To check how far the tokens can be reduced, we vary the number of memory slots (14, 7, 4, 2, 0) with everything else fixed. Slots that are not used are not read, so the number of attended tokens falls with the number of slots.

Table R3. Memory budget vs. quality and cost. Quality is measured on scene recall as in Table 2 (30 clips, seed 1). The quality of the 14-slot row is that of Full in Table 2, and the other rows with memory are run for this response. Crop similarity rises as motion falls (Sec. 4.4). Cost is measured as in the General Response, with attended tokens at chunk 70 as a range over 5 clips. Crop similarity and layout repetition use DINOv2 [G10] features, optical flow uses RAFT [G11], Dynamic is the VBench dynamic degree, and ViCLIP [G7] is the agreement with the repeated prompt.

SlotsAttended tokens at chunk 70s / clipPeak GiBCrop sim.Optical flowDynamicViCLIPLayout repetition ↓\downarrow
14 (default)12,957-19,130105.817.40.6154.3828.3310.230.688
712,957-16,282108.216.70.6203.3125.6710.820.688
412,957-14,124102.916.20.6082.7822.679.050.692
211,533-12,606101.915.80.6212.7826.339.730.698
0 (no memory)10,92086.110.90.6711.9624.008.990.708

Reading fewer tokens shortens the forward passes of the generator by about 18% relative to Self Forcing (57.3 vs. 70.1 s per clip). This saving is larger than the 7.5 s per clip that storing, verification, replacement and capture take.

With 14 and 7 slots the video follows a repeated prompt more closely than without memory (ViCLIP 10.23 and 10.82 vs. 8.99). With fewer slots a returning scene repeats earlier layouts more (layout repetition 0.688 at 14 slots vs. 0.708 without memory). We attribute this to the generator relying more on the sink and recent frames when less earlier content is stored.

Because a slot holds a region of newly appearing content rather than a frame, each added slot uses little GPU memory (Sec. 3.2). From 2 to 14 slots, peak GPU memory rises by only 1.6 GiB and time by about 4 s, while a returning scene repeats earlier layouts less (0.698 at 2 slots, 0.688 at 14). Even with 14 slots, EncoreMem uses less peak GPU memory than Self Forcing (17.4 GiB vs. 19.5 GiB) and reads the fewest tokens among the compared methods, with latency similar to Self Forcing. More slots therefore improve the video at a small cost in GPU memory and time, so we keep 14 slots as the default.

We have added these new results with accompanying discussions in Table 6 in Sec. 4.6, Table E in App. E.3 and Table H in App. E.7 of the revised PDF.

Thank you for the insightful questions, which have helped us show more clearly how the memory keeps an absent subject and how it applies to other models.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response [1/2]

We sincerely thank all reviewers (iJjo, Ws4y, dsjr) for their careful reviews and valuable feedback, which have substantially improved our work. Below, we summarize the strengths highlighted by the reviewers, restate our main contributions, highlight the key responses and new results, and list the other revisions of the manuscript. All changes in the revised PDF are highlighted in blue. The references cited in our individual responses follow in the next comment, and the details are in our response to each reviewer.

Strengths of our work, as highlighted by reviews

  1. [ iJjo, dsjr ] Region-level memory is an interesting and reasonable idea. Storing regions rather than whole frames works even when tokens are dropped.
  2. [ iJjo, Ws4y, dsjr ] Training-free design. EncoreMem uses only features of the frozen generator ( iJjo ), needs no training ( Ws4y, dsjr ), and can be applied to autoregressive foundation models ( dsjr ).
  3. [ Ws4y ] Clear separation of admission, verification and replacement, with no cache reconstruction at prompt changes.
  4. [ iJjo, Ws4y ] Broad evaluation with reasonable baselines and competitive results.

Main contributions of our work

  • Region-level storage. EncoreMem stores the regions of newly appearing content, delimited with the features of the frozen generator, so a returning scene can follow a new prompt without copying an earlier layout.
  • Deferred retention by re-encounter. A newly stored region is kept provisionally. The region is confirmed if the frames generated after its storage show the region again, and released otherwise, so what stays in memory is decided by what the rollout produces.
  • Training-free memory at no extra cost. All decisions use tensors that the frozen generator already computes, at about the same speed as the base generator.

Highlighted responses and new results

  • ( iJjo, Ws4y ) Subjects absent for a long time. In a stress test, the region of a person survives a 30-40 s absence in 15 of 16 clips at the default budget (App. E.10).
  • ( Ws4y ) Role of re-encounter verification (RV). RV keeps the memory on content that later frames re-encounter. More of the stored regions were re-encountered recently (0.918 vs. 0.879), and stored regions are kept longer (2.83 vs. 3.68 replacements per clip, App. E.10).
  • ( Ws4y ) Comparison with simpler memory policies. At the same tokens, regions give higher quality and prompt agreement than recent frames and repeat earlier layouts less than whole frames. At the same regions, the simpler replacement rules did not significantly improve on ours (App. E.8 and E.9).
  • ( dsjr ) Comparison with Deep Forcing. EncoreMem is higher in each of the 3 seeds on Consistency, Aesthetic, prompt agreement in four of six segments and each single-prompt metric except Dynamic, with half the attended tokens (App. E.4). Over 3 seeds, EncoreMem also keeps prompt agreement best among the compared methods (Δ6−1\Delta_{6-1} of -1.60 vs. -2.08 to -3.21).
  • ( dsjr ) Returning to an earlier scene. When a video leaves a room and comes back, the room on return is significantly closer to the first room with EncoreMem than with Deep Forcing (background similarity 0.768 vs. 0.681). On returning prompts, background similarity is 0.925 vs. 0.884 and face identity 0.429 vs. 0.393 (3 seeds, Sec. 4.6 and App. E.5).
  • ( iJjo, Ws4y ) Generalization to other models and longer videos. EncoreMem is added to CausVid and Deep Forcing without retraining, and tested on 240-s rollouts (Sec. 4.6 and App. E.11).
  • ( iJjo, Ws4y, dsjr ) Cost. Thanks to region-level storage, EncoreMem uses the least GPU memory among the compared methods (17.4 GiB peak vs. 19.5 GiB for Self Forcing and 20.9 GiB for Deep Forcing). EncoreMem reads the fewest tokens, and the latency of EncoreMem is similar to Self Forcing (Sec. 4.6, Table 6).

Other revisions of the manuscript

  1. Page 6, Lines 282-284 (Sec. 3.4). We stated when a long absence costs a stored region its slot (iJjo).
  2. Page 6, Lines 316-317 (Sec. 4.2). We added pointers to the 3-seed comparison and the returning-scene experiments (dsjr).
  3. Page 9, Lines 456-462 (Sec. 4.4). We stated what verification changes in memory (iJjo, Ws4y).
  4. Pages 9-10, Lines 480-522 (Sec. 4.6, Tables 5-7). We added returning prompts, the cost on the same GPU, and other generators to the main text (iJjo, Ws4y, dsjr).
  5. Page 7 (Table 1). We renamed the prompt-agreement metric ViCLIP score, as in Table 2.
  6. Page 20, Lines 1077-1080 (App. D). We stated that content leaving the view before it is confirmed is released and stored again if it returns (iJjo).
  7. Pages 19-20 and 22-28 (App. B.4, E.3-E.11). We added the benchmarks, metrics and results of all new experiments.

We hope these revisions address all concerns and further clarify the significance of this work.

AuthorsOpenReview ↗

General Response [2/2]

Global reference

[G1] Huang et al., "Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion," 2025.

[G2] Yi et al., "Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression," 2026.

[G3] Yang et al., "LongLive: Real-time Interactive Long Video Generation," 2026.

[G4] Ji et al., "MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives," 2025.

[G5] Yin et al., "From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CausVid)," 2025.

[G6] Huang et al., "VBench: Comprehensive Benchmark Suite for Video Generative Models," 2024.

[G7] Wang et al., "InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (ViCLIP)," 2024.

[G8] Radford et al., "Learning Transferable Visual Models From Natural Language Supervision (CLIP)," 2021.

[G9] Deng et al., "ArcFace: Additive Angular Margin Loss for Deep Face Recognition," 2019.

[G10] Oquab et al., "DINOv2: Learning Robust Visual Features without Supervision," 2024.

[G11] Teed and Deng, "RAFT: Recurrent All-Pairs Field Transforms for Optical Flow," 2020.

[G12] Xiao et al., "Efficient Streaming Language Models with Attention Sinks," 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.