acceptodds
Under review as a conference paper at ICLR 2027

Cache locally, Recompute Selectively for Streaming Video Understanding

13 views

Abstract

Streaming video understanding (SVU) requires video-LLMs to process incoming video continuously while answering user queries with low latency. Recent methods reduce query-time cost by prefilling frames into KV caches during idle time and reusing the cached KVs at query time. To bound per-frame computation over unbounded video streams, these methods restrict the context available during prefill. This creates a fundamental context mismatch: the restricted context may omit information needed to properly contextualize video tokens, yet, still contain information irrelevant to the eventual query, which is unknown during prefill. We propose FrameBlend, a simple yet effective plug-and-play method to address this mismatch through selective KV recomputation, without requiring any additional training. At query time, FrameBlend re-prefills selected video tokens using query-relevant context and replaces their cached KVs with recomputed ones. Crucially, this recomputation enables shorter attention windows during idle-time prefill, substantially reducing prefill cost while improving accuracy. Experiments across three SVU frameworks and five streaming and offline benchmarks demonstrate accuracy gains of up to 3.9 pp. On RVS-Ego, FrameBlend reduces per-frame prefill FLOPs by over 20% and total processing time by 14-59% for retrieval-based frameworks, including the cost of query-time recomputation. These results demonstrate that a simple change in how contextualization is distributed between idle time and query time can improve the accuracy-efficiency trade-off for SVU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, KV Construction, Selective Recomputation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tie2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FrameBlend, a training-free method for streaming video understanding that combines shorter attention windows during idle-time prefill with selective KV recomputation after a query arrives. Its QTS strategy selects recomputation targets based on retrieval frequency across layers or layer–head pairs. The selected tokens are jointly re-prefilled, and the recomputed KVs replace their corresponding cached entries. The method is evaluated across ReKV, LiveVLM, and HERMES on two streaming and three offline benchmarks, primarily using LLaVA-OneVision-7B, with additional streaming evaluations using Qwen2.5-VL-7B.

Critical strengths
  1. FrameBlend exploits the information obtained after the arrival of the problem to fix the cache representation, thereby reducing the context required for idle phase processing. For retrieval-based frameworks, reusing existing retrieval results directly to select recomputed objects is an attractive solution. Table 5 supports the value of combining the two components: RVS-Ego improves accuracy from 59.3 to 62.4 on ReKV, while either component alone achieves lower accuracy than their combination. Despite the simplicity of the design, this is a useful systematic contribution.

  2. The experiments in this paper include retrieval-based and compression-based systems, covering both streaming and offline tasks. For example, HERMES improves accuracy on MLVU from 60.2 to 64.1. Additional Qwen2.5-VL experiments also provide evidence beyond the single backbone model. The paper also reports FLOPs, actual runtime, first token latency, and memory overhead. These results help to evaluate the practical value of the method and understand its benefits and costs simultaneously.

Critical weaknesses
  1. The robustness of the reported improvement effects has not been fully verified. Section 5.1 and Appendix B indicate that the pre-filled window was tuned separately for each framework-benchmark combination, but it does not specify which validation set partitioning was adopted, nor does it clarify whether the final evaluation labels or the feedback from the evaluation server affected this choice. Many overall improvement amplitudes are small, such as the increase from 57.3 to 57.5 for HERMES on OVO-Bench, and from 68.4 to 68.7 for ReKV on MLVU, and no uncertainty estimation was provided. Therefore, it is unclear how consistent these improvements will remain under independently selected, fixed window settings.

  2. This paper has not fully demonstrated the unique contribution of its proposed selection strategy in terms of methodology. As acknowledged in related studies, selective key-value re-computation has been explored before, so its contribution largely depends on the allocation method of the proposed context and query guidance target selection. Although Table 5 evaluated the individual and combined effects of shorter windows and re-computation, it did not separate the role of QTS itself. Table 4 provides a useful comparison with CacheBlend, but no effective re-computation budget was reported, and there was no comparison with a simple selection strategy under matching conditions. Therefore, these results more clearly support the effectiveness of the overall method than the specific advantages provided by QTS.

Questions for authors
  1. Under the condition that the pre-filling window and the recomputation budget remain unchanged, how much additional accuracy can QTS provide compared to simple strategies such as random selection or sorting by aggregated normalized retrieval scores? Please also report the effective recomputation budget used in the CacheBlend comparison as presented in Table 4.

  2. In Table 5, adding recomputation alone reduces accuracy on RVS-Ego and RVS-Movie. How do the authors explain these degradations, and under what conditions does recomputation complement a shorter prefill window?

  3. Which data partitioning methods and evaluation metrics were used when selecting the pre-filling window? Was the final evaluation label or the results from the evaluation server used in the selection process? Can the authors report the window results based on the non-overlapping validation data selection and attach the paired confidence intervals for the accuracy differences?

  4. Figures 3 and 6 indicate that shorter pre-filling windows can improve accuracy, but changing the window length also alters the cache representation, which may affect the set of retrieved tokens. Can the authors isolate the proposed context interference reduction effect by fixing the retrieved and recalculated token sets and only varying the pre-filling window? Does this conclusion still hold under the limited recomputation budget used in the main experiments?

6 comments

Q1/W2. QTS vs. Simple Baselines

Q1. Under the condition that the pre-filling window and the recomputation budget remain unchanged, how much additional accuracy can QTS provide compared to simple strategies such as random selection or sorting by aggregated normalized retrieval scores? Please also report the effective recomputation budget used in the CacheBlend comparison as presented in Table 4.

W2. This paper has not fully demonstrated the unique contribution of its proposed selection strategy in terms of methodology. As acknowledged in related studies, selective key-value re-computation has been explored before, so its contribution largely depends on the allocation method of the proposed context and query guidance target selection. Although Table 5 evaluated the individual and combined effects of shorter windows and re-computation, it did not separate the role of QTS itself. Table 4 provides a useful comparison with CacheBlend, but no effective re-computation budget was reported, and there was no comparison with a simple selection strategy under matching conditions. Therefore, these results more clearly support the effectiveness of the overall method than the specific advantages provided by QTS.

1. Comparing with simple baselines

We appreciate the reviewer’s concern regarding the role of QTS itself and comparison with simple selection strategies. As shown in the table below, we conducted the suggested experiments and show that QTS consistently surpasses simple baselines. We set a recomputation budget of K=7840 and a fixed prefill window size of 384 (=128*3).

More specifically, ‘score sum/mean’ select tokens based on the sum/mean of their layer-wise normalized retrieval scores, considering only the layers in which each token is retrieved. We also report coverage, defined as the proportion of KVs in the decoding context that are replaced through recomputation (i.e., measuring how much of the decoding context is updated by recomputation). As shown in the table, QTS achieves the best overall accuracy and the highest coverage among the compared strategies. These results support the effectiveness of frequency-based selection for identifying recomputation targets.

MethodRVS-Ego Acc.RVS-Ego Cov.RVS-Movie Acc.RVS-Movie Cov.OVO-Bench Acc.OVO-Bench Cov.
LLaVA-OV-7B
ReKV57.6–41.0–52.3–
ReKV + short window58.3–42.4–51.6–
ReKV + FrameBlend (QTS)58.836.944.246.052.470.0
ReKV + FrameBlend (Score sum)58.436.543.645.652.068.9
ReKV + FrameBlend (Score mean)58.422.442.635.151.965.5
ReKV + FrameBlend (Random)58.012.342.620.952.157.9

2. Recomputation budget of CacheBlend in Table 4

In addition, to address the reviewer’s question, we clarify that CacheBlend's recomputation budget matches FrameBlend's K=7840 in Table 4. Following the official implementation of CacheBlend (link) which selects recomputation targets based on a full recomputation of the first layer, we selected 7,840 tokens for recomputation in subsequent layers. Thus, both methods use the same number of recomputation targets, although CacheBlend basically has additional overhead of calculating full recomputation to the first layer, which is a necessary operation for CacheBlend. We will clarify this implementation detail in the revised manuscript.


Q2. Short Windows and Recomputation

Q2. In Table 5, adding recomputation alone reduces accuracy on RVS-Ego and RVS-Movie. How do the authors explain these degradations, and under what conditions does recomputation complement a shorter prefill window?

We believe that this is due to the contextual interference (c.f. our response to Q4). Whenever the model fails to retrieve relevant frames—e.g., due to overly long prefill windows during the prefill—recomputing KV with highly irrelevant frames may in fact hurt the model performance by incorporating unnecessary or query-irrelevant information from the other frames. This is in contrast with what happens with correctly retrieved frames (e.g., with shorter prefill window), where the information is enhanced with rich context.

RVS-Ego and RVS-Movie seem to be more prone to such retrieval failures, as these datasets consist of egocentric and movie videos with many blurry frames (due to camera shake) or frequent scene transitions. Such characteristics may introduce substantial redundant or irrelevant information during prefill.

Q3/W1. Validation-Set Tuning and Fixed Windows (1/2)

Q3. Which data partitioning methods and evaluation metrics were used when selecting the pre-filling window? Was the final evaluation label or the results from the evaluation server used in the selection process? Can the authors report the window results based on the non-overlapping validation data selection and attach the paired confidence intervals for the accuracy differences?

W1. The robustness of the reported improvement effects has not been fully verified. Section 5.1 and Appendix B indicate that the pre-filled window was tuned separately for each framework-benchmark combination, but it does not specify which validation set partitioning was adopted, nor does it clarify whether the final evaluation labels or the feedback from the evaluation server affected this choice. Many overall improvement amplitudes are small, such as the increase from 57.3 to 57.5 for HERMES on OVO-Bench, and from 68.4 to 68.7 for ReKV on MLVU, and no uncertainty estimation was provided. Therefore, it is unclear how consistent these improvements will remain under independently selected, fixed window settings.

1. Using prefill window size tuned on validation set

We agree with the reviewer's concern that the robustness of the reported improvement effects has not been fully verified. Therefore, We tuned prefill window size with separated validation set, and show that FrameBlend still improves overall accuracy. Since most of the benchmarks used in our evaluation do not provide official validation splits, the results in the original paper were obtained by selecting prefill window sizes within {196, 384 (=128*3), 512, 1024, 2048, 4096, 8192}, based on the final accuracy. We agree that this evaluation protocol may overestimate the accuracy gains of FrameBlend. To directly mitigate this concern, we conduct additional experiments using separate validation sets for window size tuning.

Specifically, we adopt the following evaluation protocol:

  • Validation/test split: We randomly split each benchmark into equally sized validation and test sets (50:50), select the prefill window size using only the validation set, and report accuracy exclusively on the held-out test set.
  • Repeated evaluation: We repeat this procedure with 100 different random seeds and report the mean test accuracy across all splits.
  • Window size search space: We use the same candidate window sizes across all settings: {196, 384 (=128*3), 512, 1024, 2048, 4096, 8192}. So it’s significantly shorter than the original window sizes (15000 for RekV, 12000 for LiveVLM). For HERMES, we exclude 8192 because its original context size is 6000.
  • EgoSchema: Since only 500 of the approximately 5,000 samples are publicly available, we exceptionally use these 500 samples for window selection and report accuracy on the full benchmark using the official evaluation server. RVS-Ego/Movie: Since the previously used GPT-3.5-turbo-0125 evaluator has recently been deprecated, we re-evaluate all RVS results using GPT-6-luna, averaging scores over three evaluation runs.

The test-set accuracy tuned with a separated validation set is in the table below. "revised" denotes the results obtained using the validation-set evaluation protocol described above, while "original" denotes results using the window sizes selected in the original paper. And we also reported the standard deviation of average accuracy occurred by test-val set splitting.

(continued in the next comment)

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-7B
ReKV57.6041.1552.3157.9268.4160.0356.24 ± 0.41
ReKV + FrameBlend (original)58.4640.9153.7659.4868.7460.2756.94 ± 0.44 (+0.70)
ReKV + FrameBlend (revised)57.8943.8553.2759.1368.5460.1357.14 ± 0.45 (+0.90)
LiveVLM59.1844.2653.4357.6866.9960.5857.02 ± 0.46
LiveVLM + FrameBlend (original)61.9044.0054.6858.1368.1960.5457.91 ± 0.44 (+0.89)
LiveVLM + FrameBlend (revised)61.4843.4754.5257.7867.8460.3357.57 ± 0.48 (+0.55)
HERMES58.0546.7457.2755.7960.0459.6756.26 ± 0.47
HERMES + FrameBlend (original)60.4348.2957.4258.7863.9861.2858.36 ± 0.43 (+2.10)
HERMES + FrameBlend (revised)59.8147.6857.1158.5463.6361.2858.01 ± 0.45 (+1.75)

Q3/W1. Validation-Set Tuning and Fixed Windows (2/2)

Q3. Which data partitioning methods and evaluation metrics were used when selecting the pre-filling window? Was the final evaluation label or the results from the evaluation server used in the selection process? Can the authors report the window results based on the non-overlapping validation data selection and attach the paired confidence intervals for the accuracy differences?

W1. The robustness of the reported improvement effects has not been fully verified. Section 5.1 and Appendix B indicate that the pre-filled window was tuned separately for each framework-benchmark combination, but it does not specify which validation set partitioning was adopted, nor does it clarify whether the final evaluation labels or the feedback from the evaluation server affected this choice. Many overall improvement amplitudes are small, such as the increase from 57.3 to 57.5 for HERMES on OVO-Bench, and from 68.4 to 68.7 for ReKV on MLVU, and no uncertainty estimation was provided. Therefore, it is unclear how consistent these improvements will remain under independently selected, fixed window settings.

The results above show that FrameBlend continues to improve accuracy over the baselines in most cases, even when window sizes are selected using a separate validation set. Moreover, the average accuracy gains remain comparable to those obtained with the original test-set tuning protocol. These findings indicate that the improvements of FrameBlend are not merely by test-set tuning, but persist under a proper evaluation protocol. Considering that FrameBlend also reduces overall computational cost, we believe these results support the effectiveness of our approach.

We will revise the accuracy results reported in the paper to use the test-set accuracy obtained by validation-set evaluation protocol instead of the original test-set tuning procedure. We appreciate the reviewer for this valuable comment, which prompted us to adopt a more rigorous evaluation protocol and reliably evaluate the effectiveness of our method.

2. Using fixed prefill window size

In addition, we also evaluate FrameBlend using a single fixed window size across all benchmarks for each framework. We show that FrameBlend still improves the average accuracy, even with a single fixed window size.

Since the results reported in the original paper use prefill window sizes tuned for each framework–benchmark pair, we evaluate FrameBlend using a single fixed window size across all benchmarks for each framework. In addition, we further evaluate FrameBlend using fixed window sizes separated for streaming and offline benchmarks within each framework. This setting can be practical, as streaming and offline benchmarks represent different application scenarios and have different temporal context requirements. As shown in the tables below, FrameBlend still improves the average accuracy of all three frameworks in both tables, without benchmark-specific window size tuning.

(ReKV: 2048, LiveVLM: 2048, HERMES: 512)

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-7B
ReKV57.641.052.357.868.460.056.18
ReKV + FrameBlend58.043.353.858.967.860.056.97 (+0.79)
LiveVLM59.344.153.557.767.060.657.02
LiveVLM + FrameBlend62.043.853.258.167.360.457.48 (+0.46)
HERMES57.946.757.355.860.259.756.26
HERMES + FrameBlend59.348.257.458.764.160.858.09 (+1.83)

(ReKV: 384/8192, LiveVLM: 1024/4096, HERMES: 512/512 for streaming/offline benchmarks, respectively)

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-7B
ReKV57.641.052.357.868.460.056.18
ReKV + FrameBlend58.844.352.459.468.760.257.30 (+1.11)
LiveVLM59.344.153.557.767.060.657.02
LiveVLM + FrameBlend61.943.753.758.068.160.457.67 (+0.65)
HERMES57.946.757.355.860.259.756.26
HERMES + FrameBlend59.348.257.458.764.160.858.09 (+1.83)
Reviewer_gQKmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FrameBlend for streaming methods that prefill KVs in idle time and reuse them at query time. It uses a short idle-time window of 196 to 8192 tokens, which is tuned for each framework and benchmark. At query time, it re-prefills the K=7840 tokens retrieved by the most layers. Over three frameworks and five benchmarks, accuracy changes by −0.1 to +3.9 points, while TTFT on RVS-Ego rises from 0.03–0.17 s to 0.84–1.04 s. The authors claim consistent gains and a better accuracy–efficiency trade-off.

Critical strengths
  1. With recomputation, a short idle window is better than the default one on OVO-Bench, RVS-Ego and RVS-Movie in Fig. 6. This interesting finding is the main strength of the paper.
  2. The method is plug-and-play for ReKV, LiveVLM and HERMES, and the gains also hold on Qwen2.5-VL-7B. Thus, it can be used with different frameworks and backbones.
Critical weaknesses
  1. The idle window in Table 6 is tuned for each framework and benchmark. In Fig. 6, no single window lets FrameBlend outperform the ReKV baseline in all six subfigures. Recomputation alone also gives only −0.6 to +0.9 points in Table 5. Since no validation split is described, the gains may come from test-set tuning, which is the main reason for my rating.
  2. In Figs. 3 and 6, K=∞ is used, so all retrieved KVs are recomputed. Then the idle window can only change which frames are retrieved. Thus, the sweeps cannot separate a retrieval effect from the interference in the KVs used for decoding, as suggested in Section 4.2. This leaves the main explanation of the paper untested by these sweeps.
Questions for authors
  1. What is the accuracy with one fixed window on all benchmarks at the default K=7840? Since 2048 and 384 for ReKV are not in the Fig. 6 grid, how were they chosen?
  2. If the retrieved set is fixed and only the idle window is varied at the default K=7840, does accuracy still change?
  3. At 0.5 FPS, per-frame prefill takes only 1.4% to 4.1% of the frame interval. Can accuracy and TTFT be reported under different query frequencies and at 2 to 4 FPS?
  4. Was CacheBlend's recomputation budget matched to K=7840 in Table 4?
7 comments

Q1/W1. Fixed Windows and Validation-Set Tuning (1/2)

Q1. What is the accuracy with one fixed window on all benchmarks at the default K=7840? Since 2048 and 384 for ReKV are not in the Fig. 6 grid, how were they chosen?

W1. The idle window in Table 6 is tuned for each framework and benchmark. In Fig. 6, no single window lets FrameBlend outperform the ReKV baseline in all six subfigures. Recomputation alone also gives only −0.6 to +0.9 points in Table 5. Since no validation split is described, the gains may come from test-set tuning, which is the main reason for my rating.

1. Using fixed prefill window size

We agree with the reviewer’s concern regarding the overestimation of accuracy gains by benchmark-specific window size tuning. Therefore, we show that FrameBlend still improves the average accuracy, even with a single fixed window size.

Since the results reported in the original paper use prefill window sizes tuned for each framework–benchmark pair, we evaluate FrameBlend using a single fixed window size across all benchmarks for each framework. In addition, we further evaluate FrameBlend using fixed window sizes separated for streaming and offline benchmarks within each framework. This setting can be practical, as streaming and offline benchmarks represent different application scenarios and have different temporal context requirements. As shown in the tables below, FrameBlend still improves the average accuracy of all three frameworks in both tables, without benchmark-specific window size tuning.

(ReKV: 2048, LiveVLM: 2048, HERMES: 512)

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-7B
ReKV57.641.052.357.868.460.056.18
ReKV + FrameBlend58.043.353.858.967.860.056.97 (+0.79)
LiveVLM59.344.153.557.767.060.657.02
LiveVLM + FrameBlend62.043.853.258.167.360.457.48 (+0.46)
HERMES57.946.757.355.860.259.756.26
HERMES + FrameBlend59.348.257.458.764.160.858.09 (+1.83)

(ReKV: 384/8192, LiveVLM: 1024/4096, HERMES: 512/512 for streaming/offline benchmarks, respectively)

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-7B
ReKV57.641.052.357.868.460.056.18
ReKV + FrameBlend58.844.352.459.468.760.257.30 (+1.11)
LiveVLM59.344.153.557.767.060.657.02
LiveVLM + FrameBlend61.943.753.758.068.160.457.67 (+0.65)
HERMES57.946.757.355.860.259.756.26
HERMES + FrameBlend59.348.257.458.764.160.858.09 (+1.83)

Q1/W1. Fixed Windows and Validation-Set Tuning (2/2)

Q1. What is the accuracy with one fixed window on all benchmarks at the default K=7840? Since 2048 and 384 for ReKV are not in the Fig. 6 grid, how were they chosen?

W1. The idle window in Table 6 is tuned for each framework and benchmark. In Fig. 6, no single window lets FrameBlend outperform the ReKV baseline in all six subfigures. Recomputation alone also gives only −0.6 to +0.9 points in Table 5. Since no validation split is described, the gains may come from test-set tuning, which is the main reason for my rating.

2. Using prefill window size tuned on validation set

Furthermore, we tuned prefill window size with separated validation set, and show that FrameBlend still improves overall accuracy. Since most of the benchmarks used in our evaluation do not provide official validation splits, the results in the original paper were obtained by selecting prefill window sizes within {196, 384 (=128*3), 512, 1024, 2048, 4096, 8192}, based on the final accuracy. We agree that this evaluation protocol may overestimate the accuracy gains of FrameBlend. To directly mitigate this concern, we conduct additional experiments using separate validation sets for window size tuning.

Specifically, we adopt the following evaluation protocol:

  • Validation/test split: We randomly split each benchmark into equally sized validation and test sets (50:50), select the prefill window size using only the validation set, and report accuracy exclusively on the held-out test set.
  • Repeated evaluation: We repeat this procedure with 100 different random seeds and report the mean test accuracy across all splits.
  • Window size search space: We use the same candidate window sizes across all settings: {196, 384 (=128*3), 512, 1024, 2048, 4096, 8192}. So it’s significantly shorter than the original window sizes (15000 for RekV, 12000 for LiveVLM). For HERMES, we exclude 8192 because its original context size is 6000.
  • EgoSchema: Since only 500 of the approximately 5,000 samples are publicly available, we exceptionally use these 500 samples for window selection and report accuracy on the full benchmark using the official evaluation server.
  • RVS-Ego/Movie: Since the previously used GPT-3.5-turbo-0125 evaluator has recently been deprecated, we re-evaluate all RVS results using GPT-6-luna, averaging scores over three evaluation runs.

The test-set accuracy tuned with a separated validation set is in the table below. "revised" denotes the results obtained using the validation-set evaluation protocol described above, while "original" denotes results using the window sizes selected in the original paper.

The results show that FrameBlend continues to improve accuracy over the baselines in most cases, even when window sizes are selected using a separate validation set. Moreover, the average accuracy gains remain comparable to those obtained with the original test-set tuning protocol. These findings indicate that the improvements of FrameBlend are not merely by test-set tuning, but persist under a proper evaluation protocol. Considering that FrameBlend also reduces overall computational cost, we believe these results support the effectiveness of our approach.

We will revise the accuracy results reported in the paper to use the test-set accuracy obtained by validation-set evaluation protocol instead of the original test-set tuning procedure. We appreciate the reviewer for this valuable comment, which prompted us to adopt a more rigorous evaluation protocol and reliably evaluate the effectiveness of our method.

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-7B
ReKV57.6041.1552.3157.9268.4160.0356.24
ReKV + FrameBlend (original)58.4640.9153.7659.4868.7460.2756.94 (+0.70)
ReKV + FrameBlend (revised)57.8943.8553.2759.1368.5460.1357.14 (+0.90)
LiveVLM59.1844.2653.4357.6866.9960.5857.02
LiveVLM + FrameBlend (original)61.9044.0054.6858.1368.1960.5457.91 (+0.89)
LiveVLM + FrameBlend (revised)61.4843.4754.5257.7867.8460.3357.57 (+0.55)
HERMES58.0546.7457.2755.7960.0459.6756.26
HERMES + FrameBlend (original)60.4348.2957.4258.7863.9861.2858.36 (+2.10)
HERMES + FrameBlend (revised)59.8147.6857.1158.5463.6361.2858.01 (+1.75)

Q2/W2. Isolating Contextual Interference

Q2. If the retrieved set is fixed and only the idle window is varied at the default K=7840, does accuracy still change?

W2. In Figs. 3 and 6, K=∞ is used, so all retrieved KVs are recomputed. Then the idle window can only change which frames are retrieved. Thus, the sweeps cannot separate a retrieval effect from the interference in the KVs used for decoding, as suggested in Section 4.2. This leaves the main explanation of the paper untested by these sweeps.

We appreciate the reviewer’s concern regarding the actual effect of interference in the KVs. We conducted a suggested experiment, and show that even with the retrieval set is fixed, the accuracy is still increased by using the short prefill window.

First of all, our explanation for the benefits of shorter prefill windows involves two separate effects: (1) improved retrieval performance (as shown in Figure 3 and 6) and (2) reduced contextual interference in the KVs used for decoding. We agree with the concern that Figure 3 and 6 do not isolate these two effects. Specifically, since these experiments use K=∞\infty, all retrieved KVs are recomputed, so varying the prefill window size only affects which tokens are retrieved. Therefore, these results demonstrate that shorter prefill windows can improve retrieval performance.

To isolate the second effect, we conduct additional experiments following the reviewer's suggestion. Specifically, we fix both the retrieved token set and the recomputation targets using the retrieval results from vanilla ReKV. And we vary only the prefill window size with the default recomputation budget of (K=7840). Under this setting, since the retrieved tokens and recomputation targets are fixed, the varying window sizes only affect the amount of context encoded in the retrieved but non-recomputed KVs. As shown in Figure 8 (we added the figures in manuscript), reducing the prefill window size (i.e., reducing encoded context) can improve accuracy, even with the retrieval set fixed. This supports our claim that reducing the amount of context can mitigate the interference, which may be caused by unnecessarily encoded information during prefill.

We incorporate these additional controlled experiments into the revised manuscript and clarify the distinct contributions of retrieval performance and contextual interference in Appendix H. We appreciate the reviewer for this valuable suggestion, which strengthens the empirical support for our explanation.

Reviewer_bQBTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FrameBlend, a training-free and plug-and-play method for streaming video understanding. The key idea is to use a shorter attention window during idle-time KV prefill and selectively recompute the KVs of query-relevant tokens once a query arrives, thereby reducing the mismatch between prefill-time and query-time context. Experiments across multiple SVU frameworks and benchmarks show that FrameBlend can improve or maintain accuracy while reducing overall computation and processing latency.

Critical strengths
  1. The paper identifies a clear and practically relevant problem in streaming video understanding: KV caches are constructed before future queries are known, which can create a mismatch between the context used during prefill and the context that is actually useful at query time.

  2. The method is simple and practical. FrameBlend is training-free, can be integrated into existing SVU frameworks such as ReKV, LiveVLM, and HERMES, and reuses query-relevant retrieval signals to select recomputation targets with little additional machinery.

  3. The paper demonstrates a favorable accuracy-efficiency trade-off. By combining shorter idle-time prefill windows with selective query-time recomputation, FrameBlend improves accuracy on multiple benchmarks while reducing per-frame prefill FLOPs and overall latency.

  4. The ablation study provides reasonable support for the proposed design. Using only a shorter window or only recomputation gives mixed results, while combining both consistently performs best, which supports the claimed complementarity between the two components.

Critical weaknesses
  1. FrameBlend does not directly address long-term KV memory growth in potentially unbounded video streams. Its scalability still depends on the underlying SVU framework for KV compression, eviction, or offloading, so the method mainly improves cache contextualization rather than solving the broader long-term memory management problem.

  2. Query-time recomputation introduces a noticeable increase in time-to-first-token (TTFT). Although the overall latency is reduced due to cheaper idle-time prefill, this trade-off may be undesirable in applications with strict response-latency requirements.

  3. The prefill window size is still manually tuned for each framework-benchmark pair. The lack of an adaptive window selection mechanism may limit the practicality of the method under changing video distributions or deployment conditions.

  4. The methodological novelty is moderate. Selective KV recomputation has already been explored in prior work such as CacheBlend and EPIC, and the main novelty of FrameBlend lies in query-aware target selection and redistributing computation between idle time and query time rather than introducing a fundamentally new recomputation mechanism.

  5. The current evaluation is limited to 7B-scale backbone models. It remains unclear whether the same accuracy-efficiency trade-off will hold for substantially smaller or larger video-LLMs.

Questions for authors

see weaknesses

5 comments

W1. Scope Clarification

W1. FrameBlend does not directly address long-term KV memory growth in potentially unbounded video streams. Its scalability still depends on the underlying SVU framework for KV compression, eviction, or offloading, so the method mainly improves cache contextualization rather than solving the broader long-term memory management problem.

As the reviewer points out, FrameBlend does not directly address long-term KV memory growth, as our primary focus is cache contextualization rather than memory management. While mitigating memory growth is an important challenge in SVU, we emphasize that cache contextualization through recomputation is a distinct and complementary problem. Importantly, FrameBlend is orthogonal to prior approaches and can be integrated into SVU frameworks without modifying their underlying mechanisms. That is, FrameBlend improves accuracy while reducing overall computational cost, preserving the benefits of existing memory management strategies (e.g., KV compression, eviction, and offloading). We therefore position FrameBlend as a complementary contribution that improves the accuracy–efficiency trade-off of existing SVU frameworks, rather than as a replacement for long-term memory management techniques.


W2. TTFT Trade-off and Practical Applicability

W2. Query-time recomputation introduces a noticeable increase in time-to-first-token (TTFT). Although the overall latency is reduced due to cheaper idle-time prefill, this trade-off may be undesirable in applications with strict response-latency requirements.

We appreciate the reviewer for highlighting this important trade-off. First of all, the main advantage of FrameBlend is improving accuracy while reducing total inference cost.

We agree that FrameBlend should be carefully employed in applications with strict TTFT requirements. However, FrameBlend is particularly beneficial in scenarios which include intense idle-time prefill (e.g., long video, high-fps video, multi-camera), infrequent queries, or high-accuracy requirements. Such scenarios are relevant to practical SVU applications, such as physical AI, where high-fps video streams from multiple cameras may need to be processed over extended periods. In these settings, reducing overall computation can be crucial for reducing GPU resource requirements, energy consumption and thermal issues.

W3. Fixed Window Size

W3. The prefill window size is still manually tuned for each framework-benchmark pair. The lack of an adaptive window selection mechanism may limit the practicality of the method under changing video distributions or deployment conditions.

We agree with the reviewer’s concern regarding the practicality of the method. Therefore, we show that FrameBlend still improves the average accuracy, even with a single fixed window size.

Since the results reported in the original paper use prefill window sizes tuned for each framework–benchmark pair, we additionally evaluate FrameBlend using a single fixed window size across all benchmarks for each framework. In addition, we further evaluate FrameBlend using fixed window sizes separated for streaming and offline benchmarks within each framework. This setting can be practical, as streaming and offline benchmarks represent different application scenarios and have different temporal context requirements. As shown in the tables below, FrameBlend still improves the average accuracy of all three frameworks in both tables, without benchmark-specific window size tuning.

However, as the reviewer properly mentioned, developing an adaptive window selection mechanism that dynamically adjusts to different video distributions or deployment conditions can be a promising direction for future work.

(ReKV: 2048, LiveVLM: 2048, HERMES: 512)

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-7B
ReKV57.641.052.357.868.460.056.18
ReKV + FrameBlend58.043.353.858.967.860.056.97 (+0.79)
LiveVLM59.344.153.557.767.060.657.02
LiveVLM + FrameBlend62.043.853.258.167.360.457.48 (+0.46)
HERMES57.946.757.355.860.259.756.26
HERMES + FrameBlend59.348.257.458.764.160.858.09 (+1.83)

(ReKV: 384/8192, LiveVLM: 1024/4096, HERMES: 512/512 for streaming/offline benchmarks, respectively)

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-7B
ReKV57.641.052.357.868.460.056.18
ReKV + FrameBlend58.844.352.459.468.760.257.30 (+1.11)
LiveVLM59.344.153.557.767.060.657.02
LiveVLM + FrameBlend61.943.753.758.068.160.457.67 (+0.65)
HERMES57.946.757.355.860.259.756.26
HERMES + FrameBlend59.348.257.458.764.160.858.09 (+1.83)

W4. Novelty of Recomputation Strategy

W4. The methodological novelty is moderate. Selective KV recomputation has already been explored in prior work such as CacheBlend and EPIC, and the main novelty of FrameBlend lies in query-aware target selection and redistributing computation between idle time and query time rather than introducing a fundamentally new recomputation mechanism.

We appreciate the reviewer’s concern regarding the methodological novelty.

As the reviewer correctly points out, selective KV recomputation has already been explored in prior works (e.g., CacheBlend and EPIC). Nevertheless, our contribution is not simply applying existing recomputation techniques to SVU, but designing an efficient, query-aware recomputation strategy tailored to the unique constraints of SVU. Existing recomputation methods either rely on costly operations to identify recomputation targets or use query-agnostic selection strategies, making them less suitable for SVU. Apart from these approaches, we propose a lightweight, frequency-based selection mechanism (QTS) specifically designed for SVU. QTS can utilize KV retrieval results already available in existing SVU frameworks and selects recomputation targets based on their frequency across layers. This enables QTS to identify query-relevant tokens without costly additional operations. Moreover, as demonstrated in Table 4, FrameBlend consistently outperforms CacheBlend in both accuracy and added TTFT across all three streaming benchmarks, validating the effectiveness of our SVU-specific design.

W5. Different Model Scales

W5. The current evaluation is limited to 7B-scale backbone models. It remains unclear whether the same accuracy-efficiency trade-off will hold for substantially smaller or larger video-LLMs.

To mitigate the reviewer’s concern, we evaluate FrameBlend on LLaVA-OV-0.5B, and show that accuracy–efficiency trade-off still holds across different model scales. Efficiency results are measured on RVS-Ego using a single NVIDIA A100 80GB GPU.

As shown in the table below, FrameBlend still improves average accuracy on both frameworks, even with the 0.5B-scale backbone, while also reducing the total cost. These results suggest that the benefits of FrameBlend extend beyond 7B-scale models to substantially smaller backbones.

For larger models, we are also working toward evaluating a 32B-scale backbone. but these experiments have not yet been completed due to limited computational resources. We will make every effort to report the results within the rebuttal period.

MethodRVS-EgoRVS-MovieOVO-BenchVideo-MMEMLVUEgoSchemaAverage
LLaVA-OV-0.5B
ReKV49.3135.9141.7644.4953.8831.4642.80
ReKV + FrameBlend51.3636.0140.7044.5254.3733.5943.43 (+0.62)
HERMES50.5437.3043.2444.4551.7731.1943.08
HERMES + FrameBlend52.8737.4243.1844.1752.4831.4343.59 (+0.51)
MethodTFLOPs/fms/fRecomp. (s)TTFT (s)Total TFLOPsTotal Latency (s)
LLaVA-OV-0.5B
ReKV1.0325.5–0.121095.729.6
ReKV + FrameBlend0.7925.50.100.23853.528.9
HERMES0.9718.8–0.021032.421.8
HERMES + FrameBlend0.8516.00.240.29916.917.3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.