- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes StreamBalance, a fixed-capacity memory framework for ultra-long streaming video understanding, where past observations must be compressed before future queries are known. The approach combines a state-conditioned optimal transport mechanism for assigning incoming visual tokens to memory slots with a retention mechanism that uses learnable per-slot parameters to control content merging and preservation. A lightweight query-aware retriever selects memory cells for answer generation by a frozen multimodal language model. The memory modules and retriever are trained using downstream question-answering supervision.
Critical strengths
- The paper addresses an important problem: preserving distant evidence in streaming videos under a fixed memory budget and without knowing future queries.
- The reported results show substantial improvements over strong streaming baselines, with evaluation across multiple backbones and benchmarks.
- The ablations cover the main components and design choices, providing useful evidence that the gains come from improved memory quality rather than a larger retriever.
Critical weaknesses
- Algorithm 1 computes the STE surrogate but branches on the hard decision, leaving the gradient path through the memory update unclear.
- Appendix E introduces a learnable projection W that is missing from the main method and parameter lists.
- Results are inconsistent across tables, e.g. ReKV scores 37.4 on StreamingBench in Table 1 but 69.1 in Table 9.
- Qwen3-VL backbone achieves relatively modest gains on most benchmarks.
- The comparisons rely heavily on older baselines, leaving competitiveness against recent VLMs unclear.
Questions for authors
- Could the authors clarify how gradients flow through the STE surrogate when Algorithm 1 branches on the hard decision?
- Could the authors define the projection introduced in Appendix E and specify where it is used and how it is trained?
- Could the authors reconcile the inconsistent results, particularly ReKV’s StreamingBench scores of 37.4 in Table 1 and 69.1 in Table 9?
- Could the authors explain the relatively modest gains with Qwen3-VL on the ultra-long benchmarks and clarify when the method is most beneficial?
- Could the authors include comparisons with more recent baselines under matched evaluation budgets?
5 comments
Official Comment by Authors (2/3)
W4/Q4: Gains with Qwen3-VL and when StreamBalance helps most.
We first clarify that the gains with Qwen3-VL are not modest on most benchmarks: such as StreamBalance improves it by +16.5 on OVOBench, +5.4 on StreamingBench. The smaller gap on EgoLife-QA (+3.7) reflects how differently the two models observe the stream. EgoLife-QA was designed for offline agentic systems; the base Qwen3-VL model sees 64 frames sampled uniformly over the history, tens of minutes apart on day-long videos, so the brief evidence often falls between them, whereas StreamBalance, like prior streaming methods, processes every frame at 1 fps, stores it in memory, and reasons over the compressed memory.
Such compression is what makes this benchmark hard for streaming models: prior streaming methods on Qwen2.5-VL fall below their own backbone (e.g., HERMES 30.9 vs. 35.4; Table 1b), likely because distant evidence is lost under compression. StreamBalance is the only streaming method that surpasses its backbone on every backbone we evaluate by a good margin on this ultra-long benchmark. Our goal with such ultra-long benchmarks is to show that even under extreme compression, a streaming model can answer questions over ultra-long videos from the actual answer-bearing evidence, retained across very long histories, without collapsing as the evidence lies further back, whereas the backbone, with its sparse sampled view, largely answers without it. The +3.7 gap thus compares a backbone that largely answers without the evidence with a streaming model that answers from it under extreme compression.
To verify that its answers come from the retained evidence, we replace the frames within ±120 s of each question's annotated evidence timestamp with black frames, keeping all other frames and timestamps unchanged (w/o evidence).
Table W4: EgoLife-QA by certificate length (Qwen3-VL-8B backbone). Values in parentheses: change from removing the evidence.
| Method | <2h | 2–6h | 6–24h | >24h | Overall |
|---|---|---|---|---|---|
| Blind (no video) | 32.3 | 31.8 | 25.6 | 27.8 | 29.4 |
| Qwen3-VL-8B | 42.6 | 37.2 | 39.5 | 34.9 | 38.6 |
| Qwen3-VL-8B w/o evidence | 41.0 (−1.6) | 35.3 (−1.9) | 39.1 (−0.4) | 33.4 (−1.5) | 37.2 (−1.4) |
| + StreamBalance (Ours) | 44.8 | 43.1 | 41.7 | 39.4 | 42.3 |
| + StreamBalance (Ours) w/o evidence | 35.7 (−9.1) | 32.5 (−10.6) | 28.4 (−13.3) | 30.1 (−9.3) | 31.7 (−10.6) |
Removing the evidence changes Qwen3-VL by only 1.4 points, consistent with the evidence rarely being among its sampled frames, and its accuracy fluctuates non-monotonically with certificate length, mirroring prior streaming methods in Fig. 4 of the main paper. In contrast, StreamBalance drops by 10.6 points, close to the blind model, at every length, showing that its accuracy comes from the evidence it retains and retrieves, and it decays gently as the evidence lies further in the past. StreamBalance is therefore most beneficial when the answer depends on specific evidence far in the past, which it consistently retains and retrieves.
Official Comment by Authors (3/3)
W5/Q5: Clarification on baseline recency and evaluation budgets.
We clarify that our direct comparators are recent in every table (1a, 1b, 2). On general streaming benchmarks (Table 1a), every streaming method built on our backbones was published in 2025–2026 at a top CV/ML/NLP venue, e.g., StreamReady (CVPR'26), HERMES (ACL'26), QueryStream (ICLR'26), and InfiniPot-V (NeurIPS'25). On ultra-long benchmarks (Table 1b), we additionally compare with recent agentic systems, e.g., WorldMM (CVPR'26) and M3-Agent (ICLR'26), and on offline long-video benchmarks (Table 2), with recent methods such as Streamo (CVPR'26), TimeSearch-R (ICLR'26), and StreamingTom (CVPR'26).
As a streaming method, StreamBalance's direct comparators are these prior streaming methods, which we compare under a matched evaluation protocol: within each backbone block, all methods share the same backbone and the same 1 fps streaming input. Memory budgets, however, cannot be matched directly: each method retains a different unit of memory (e.g., KV caches, pruned visual tokens, or compressed memory cells), so the same nominal size is not comparable across methods, and changing it would depart from each method's own reported best configuration; we therefore keep every method's memory budget at its best reported setting. Offline MLLMs, in contrast, are not direct comparators and cannot be evaluated under a matched streaming budget, as their context length cannot hold the unbounded 1 fps stream. Following prior streaming work, they are reported for completeness only, from the benchmarks' official papers, under the standard protocol of at most 64 frames sampled from the 1 fps video. Their recency therefore does not affect our comparisons or conclusions.
As requested, we additionally include more recent models under the same protocol (below): two more 2026 streaming methods, FluxMem and MuKV (official reported results), and a more recent backbone, Qwen3.5-4B (Feb'26; evaluated by us). StreamBalance outperforms both new streaming methods by a clear margin and yields large gains on the new backbone.
Table W5: Comparison with more recent models. FluxMem and MuKV: official reported results. ‡: evaluated by us under the official protocol.
| Method | Release | OVOBench (Real-Time) | StreamingBench (Real-Time) |
|---|---|---|---|
| Recent streaming methods (Qwen2.5-VL-7B backbone) | |||
| FluxMem | CVPR'26 | 67.2 | 76.4 |
| MuKV | CVPR'26 | – | 64.4 |
| StreamBalance (Ours) | – | 75.1 | 82.1 |
| More recent backbone | |||
| Qwen3.5-4B‡ | Feb'26 | 68.0 | 77.0 |
| + StreamBalance (Ours) | – | 82.5 | 83.2 |
Official Comment by Authors (1/3)
We thank the reviewer for their feedback on our paper. We have incorporated it and revised the manuscript accordingly (changes marked in blue). We address the concerns below.
W1/Q1: Gradient flow through the STE surrogate.
We apologize for the confusion caused by Algorithm 1. Its "if/else" shows what the update does in forward pass, and is exactly what runs at inference. Let d=1[sim≥τj] (d=1: merge, d=0: allocate), with sim=cos(ΔjW,ck∗W) (definition of W explained in W2/Q2). During training, the written content is not chosen by the branch: we compute both outcomes,
cmerge=(1−λj)ck∗+λjΔj,calloc=Δj,
and select between them using the STE surrogate d^=d+sim−sg[sim], while d determines which cell receives the result:
cnew=d^⋅cmerge+(1−d^)⋅calloc.
Forward pass. Since d^=d∈0,1, cnew is exactly cmerge when d=1 and exactly calloc when d=0, as in the hard branch of Algorithm 1.
Backward pass. The loss depends on d^ through cnew, with ∂cnew/∂d^=cmerge−calloc. Applying the chain rule through d^ and sim gives the gradient of the merge decision:
∂W∂L=∂L/∂d^(∂cnew∂L)⊤(cmerge−calloc)⋅=1 (STE)∂sim∂d^⋅∂W∂sim
The first factor compares the two outcomes for the written content: if moving it toward cmerge would lower the loss, the update increases sim, favoring merging; otherwise, it decreases sim, favoring allocation. This signal arises at every write, whether the forward pass merged or allocated, and the STE passes it through sim to W, the incoming content Δj (hence the transport plan and its parameters), and the stored cell ck∗, so all parameters governing the merge decision are trained by the downstream loss. As is standard for straight-through estimators, this signal is an approximation: the discrete steps of the update, i.e., which cell is written and which two cells are consolidated when a slot is full, are index selections, which are inherently non-differentiable; they have no parameters of their own and are determined by sim.
For simplicity of presentation, we showed only the hard choice in the main text and Algorithm 1, since this is what the update computes in the forward pass and at inference. Additionally, Appendix E started from ∂L/∂d^ and omitted the first factor, which shows how d^ enters the memory update. We understand this may have made the gradient path unclear, and have revised the manuscript accordingly (changes in blue):
W2/Q2: Clarification of the learnable projection W.
W, mentioned in Appendix E, is a learnable linear projection used to compute the retention similarity in Eq. 5, sim=cos(ΔjW,ck∗W), i.e., the two features are compared after a shared projection. As it is a standard detail, we had omitted it from the main description for brevity. W lets the downstream loss shape what counts as redundant, which is what makes the merge criterion learnable. It is used only for comparison, so stored cells remain in the original feature space, and it is trained with the VL loss through the STE path derived in Appendix E. We have revised the Method section and Appendix E to clarify W, and listed it among the learnable parameters in Algorithms 1 and 2.
W3/Q3: Reconciling numeric typographical issues.
The ReKV entry in Table 1 was a transcription error, which we have corrected; the correct StreamingBench score is 69.1, as reported in Appendix Table 9 (original, Table 14 in revised). While verifying this, we cross-checked all baseline entries against their original sources and corrected one more transcription error in Table 1: the OVOBench overall scores of Qwen2.5-VL and TimeChat-Online, which appear in adjacent rows, had been swapped. Importantly, these corrections affect only baseline entries copied from prior work: all StreamBalance results are consistent, and no correction changes any ranking or conclusion. All corrected entries are marked in blue in the revised manuscript.