- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a KV-cache management method to efficiently maintain region-level consistency in both single-prompt and multi-prompt cases. The author novely propose to perform region-based KV cache selection for long-term consistency.
Critical strengths
- The region-based memory selection is simple and reasonable.
- The method is training-free so can be applied to any autoregressive foundation models.
Critical weaknesses
- the quantitative results shown in Tab.1 is no solid. Self-forcing has highest consistency score. Other metrics is only comparable with Deep Forcing, no significant improvements.
- The author claims the method can work under multi-prompt settings。 But qualitative figures in both paper and appendix don’t really show the ability to maintain consistency in response to prompt change。 I suggest the author to add results such as 'a man stay in the room first, then go out in a garden, and come back to the home' to see whethe the settings in the home stays the same.
Questions for authors
- What is the extra cost time during inference compared to baseline, e.g., self-forcing? As fewer tokens are kept but a selection process added, whether the inference latency increased or decreased.
- The core difference between ENCOREMEM and Deep Forcing? Deep Forcing keep Top-K similar tokens in KV-cache. If the targeted region repeatedly occur, it should also be kept in DeepForcing’s KV cache. Can the author provide some visual comparison on the tokens being kept by ENCOREMEM and Deep Forcing?
6 comments
Response to the Reviewer dsjr [2/3]
W2) Does a scene stay the same when a prompt change brings it back?
We sincerely thank you for proposing a scenario that tests the point where the memory should matter most.
Answer: In the suggested scene, when the video leaves the room and comes back, the room stays more consistent than with Deep Forcing. On a benchmark where each of six scenes returns 30 s later, EncoreMem also keeps the background and faces of returning scenes significantly closer to their first appearance than Deep Forcing.
For the suggested scene, we wrote scripts in which a person is in a room, goes out to another place for 30 s, and comes back to the room. The places include a market, a subway station, a forest and a beach (all runs for this response). When the videos of both methods leave the room and return, the room on return is significantly closer to the first room for EncoreMem than for Deep Forcing. The background similarity with the person masked is 0.768 vs. 0.681. Figure F shows an example. Deep Forcing returns to a different room. Without memory, the video relies on recent frames, so the market stalls remain in the room on return. A memory of whole frames keeps the room when the prompt asks for the market. EncoreMem goes to the market, comes back, and keeps the first room.
To test returns at scale, we show the six prompts of each multi-prompt script for 5 s each and then again in the same order (12 segments, 11 switches, 30 scripts), so each prompt returns 30 s later, after five others. For example, one script follows a detective in a rain-soaked alley through six prompts (examining a footprint, meeting an informant, receiving a note, reading the clue, watching the informant leave, and standing up with the clue), and the same six prompts are then shown again.
Table R10. Returning prompts (30 scripts, 3 seeds, all runs for this response). * means that the paired bootstrap 95% interval of the difference from Deep Forcing excludes zero. Prompt agreement is the ViCLIP score, background similarity is the cosine similarity of DINOv2 [G10] features with detected persons masked, and face identity is the ArcFace [G9] similarity of detected faces.
| Method | Prompt agreement (ViCLIP, 12 segments) | Background similarity of returning segments | Face identity of returning segments |
|---|---|---|---|
| Deep Forcing | 22.78 | 0.884 | 0.393 |
| EncoreMem (Ours) | 23.47* | 0.925* | 0.429* |
EncoreMem follows the prompts better than Deep Forcing and keeps the background and face of a returning segment closer to their first appearance.
On the benchmark of Table 1, comparing the face in the last segment with the first (ArcFace, 47 prompts with a detected face), EncoreMem keeps the face better than Deep Forcing and as well as LongLive and MemFlow, while its prompt agreement declines less than that of LongLive and MemFlow (Table R11).
Table R11. Face identity and prompt agreement on the 47 prompts of Table 1 with a detected face. All methods use 3 seeds except Self Forcing, which uses 1. EncoreMem and the added seeds are run for this response. * means that the paired 95% interval of the difference from EncoreMem excludes zero.
| Method | Identity, segment 6 to segment 1 | ViCLIP 2-6 | |
|---|---|---|---|
| Self Forcing | 0.105* | 19.46* | -10.38* |
| MemFlow | 0.395 | 23.63* | -3.62* |
| LongLive | 0.396 | 24.47 | -2.48* |
| Deep Forcing | 0.368* | 24.44 | -1.73 |
| EncoreMem (Ours) | 0.399 | 24.59 | -1.55 |
We have added these new results with accompanying discussions in Table 5 in Sec. 4.6, Figure F in App. E.5 and Table G in App. E.6 of the revised PDF.
Response to the Reviewer dsjr [3/3]
Q1) Does inference cost increase or decrease relative to Self Forcing?
Answer: Thanks to region-level storage, EncoreMem uses less GPU memory than Self Forcing (17.4 GiB vs. 19.5 GiB at peak), with latency similar to Self Forcing.
We measured all methods on the same GPU with 60-s clips. EncoreMem reads fewer than half the tokens that Self Forcing reads (14,157 vs. 32,760 at chunk 40), which shortens the forward passes of the generator (57.3 vs. 70.1 s per clip). Storing, verification, replacement and capture together add 7.5 s per clip (verification alone 1.3 s, measured in a separate run with timing enabled). EncoreMem also needs the least GPU memory during generation (17.4 GiB vs. 19.5 GiB for Self Forcing).
We have added these new results with accompanying discussions in Table 6 in Sec. 4.6 and Table E in App. E.3 of the revised PDF (see also the cost summary in the General Response).
Q2) Deep Forcing would also keep a region that occurs repeatedly. What is the core difference, and can the kept tokens be compared visually?
Answer1: Deep Forcing keeps what the current chunk attends to, while EncoreMem keeps what later frames actually come back to.
We agree that Deep Forcing keeps a recurring region whenever the current queries attend to it. Deep Forcing ranks tokens by the attention of the recent queries and recomputes this at each compression, so content the current chunk does not attend to is dropped even if it appeared many times before. What will be needed again cannot be known when content first appears. EncoreMem therefore stores each newly appearing region first and confirms the region once later frames show the region again. This check uses only frames that have already been generated. A confirmed region then stays available while it is out of view, regardless of the attention of the current chunk, because Re-encounter Replacement removes a confirmed region only when the memory is full and a new region needs the slot (Sec. 3.4).
This difference shows when an earlier prompt returns. On the returning-prompt benchmark of W2, none of the tokens Deep Forcing selects by attention (Top-C) come from the segment where the content first appeared, and 69% come from the previous segment, which shows a different prompt and is no longer relevant. In contrast, EncoreMem keeps the background and faces of returning segments closer to their first appearance (Table R10). Within a single prompt, Deep Forcing also drifts further from the prompt over a minute (CLIP-T 60-0 s of -1.34 vs. -0.93 over 3 seeds, Table R9).
Answer2: Deep Forcing keeps scattered fragments, while EncoreMem stores a few connected regions (Figure E). This matches the returning-scene results of W2, where the room stays more consistent with EncoreMem (Figure F).
Figure E in App. E.4 shows, at chunks 20, 30 and 40 on the same prompt and seed, the tokens Deep Forcing keeps and the regions EncoreMem stores. In this figure, the 3,120 tokens kept by Deep Forcing at chunk 40 are scattered over 1,685-2,505 connected pieces (the largest covering at most 8.8% of them), while EncoreMem stores 5-9 regions forming 4-11 pieces (the largest covering 17-43%).
We have added these new results with accompanying discussions in Figure E in App. E.4 of the revised PDF.
Thank you for the insightful questions, which have helped us strengthen the comparison with Deep Forcing and the evaluation of returning scenes.
Response to the Reviewer dsjr [1/3]
Dear Reviewer
dsjr, The authors sincerely appreciate your concrete suggestions, especially the returning-scene example, which helped us test the method where memory matters most. Your questions on Table 1, on returning scenes and on the difference from Deep Forcing helped the authors add the missing evidence for the main claim. We are also thankful for your acknowledgement of the strengths of this work, namely region-based memory selection that is simple and reasonable, and a training-free design applicable to autoregressive foundation models. We address your concerns and questions in the threads below.Answer1:The Consistency of Self Forcing [G1] is high because its frames barely change over a long video, which does not indicate a better long video.VBench [G6] consistency measures how similar the frames within a two-second window are, so a video that changes less scores higher, whether or not the video follows the prompt. Self Forcing is the base generator for autoregressive video generation and is trained on clips of 7 chunks (about 5 s). A one-minute rollout goes far beyond this length, so the generator keeps redrawing the scene that has already been drawn. The frames therefore change less and less, which raises Consistency. In Table D, Self Forcing changes least over a multi-prompt rollout although the prompt changes five times (Subject -0.34, Background -0.17), and its Aesthetic drops most (-7.80).
In the multi-prompt setting of Table 1, this means that the frames keep the earlier scene instead of following the new prompt (Figure 1). Prompt agreement falls by 10.7 points from the first to the last segment (from 26.06 to 15.38), against 1.8 points for EncoreMem (Table 1). This is why the base generator is not ranked in Table 1 (Sec. 4.2). Its Aesthetic and Quality are also the lowest among the compared methods (57.39 and 80.88).
Answer2:Against Deep Forcing [G2], EncoreMem is higher on most metrics in each of the 3 seeds, with about half the attended tokens. Over 3 seeds, EncoreMem also keeps prompt agreement best among all methods (Δ6−1).In Table 1 (one seed), the paired 95% intervals of the differences in Consistency and Aesthetic over the 100 prompts exclude zero (Sec. 4.2, L315-316).
Table R9. Comparison over 3 seeds. The multi-prompt block has all methods (Self Forcing on 1 seed), and the single-prompt block has Deep Forcing. Runs beyond those of the paper are run for this response. * means that the paired 95% interval of the difference from EncoreMem excludes zero. Multi-prompt metrics are as in Table 1 (prompt agreement by ViCLIP [G7]), single-prompt metrics are VBench dimensions, and CLIP-T uses CLIP [G8].
In each of the 3 seeds EncoreMem is higher than Deep Forcing in Consistency, Aesthetic, prompt agreement in segments 2, 3, 5 and 6, Δ6−1 and each single-prompt metric except Dynamic degree.
Deep Forcing has a higher Dynamic degree (64.84 vs. 48.34), but this motion comes with lower prompt agreement. The prompt agreement of Deep Forcing falls more as the video goes on, both with a single prompt (CLIP-T 60-0 s of -1.34 vs. -0.93) and with multiple prompts (Δ6−1 of -2.08 vs. -1.60), and both differences are significant (Table R9). Over 3 seeds, EncoreMem also keeps prompt agreement better than the training-based LongLive [G3] and MemFlow [G4] (Δ6−1 of -1.60 vs. -2.32 and -3.21), with higher Quality and Consistency on par.
EncoreMem achieves this with about half the attended tokens of Deep Forcing (14,157 vs. 29,640) and less GPU memory (17.4 GiB vs. 20.9 GiB). EncoreMem also keeps higher quality over 240 s (Reviewer
Ws4y, Table R8) and is significantly better when earlier prompts return (W2).We have added these new results with accompanying discussions in Table F in App. E.4 of the revised PDF.