acceptodds
Desk rejected at ICLR 2027

Remembering What Matters: Balanced Memory for Ultra-Long Streaming Videos

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) for streaming videos compress an unbounded stream of past frames into a fixed-size memory to stay within a limited LLM context budget. However, existing methods often produce unbalanced memory representations: heuristic writes induce a recency bias that coarsens older content, while attention-based writes concentrate on a few dominant events; both losing distant evidence as streams grow longer. Thus, we reformulate streaming memory compression as a **balanced temporal coverage** problem and propose **StreamBalance**, whose fixed-size memory is fully learnable and trained end-to-end through two mechanisms. A *state-aware, entropy-regularized optimal transport* formulation that routes tokens to their best-fit memory slots under a capacity constraint that keeps utilization even on write, and *adaptive decay-rate-aware retention mechanism* in which each slot learn its own persistence: fast slots merging recurring redundancy, and slow slots preserving rare events; so answer-bearing evidence is neither crowded out on write nor decayed away over time. Together, these mechanisms let StreamBalance set a new state of the art on two streaming benchmarks, such as % overall gain on OVOBench. On two ultra-long benchmarks spanning up to hours, it maintains nearly stable accuracy where prior methods collapse, outperforming prior streaming model by % on EgoLife-QA at lesser compute and lower latency. Despite being trained for streaming, it further generalizes competitively to three offline long-video benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming video understanding, balanced memory, learnable compression, MLLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JvD8OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes StreamBalance, a fixed-capacity memory framework for ultra-long streaming video understanding, where past observations must be compressed before future queries are known. The approach combines a state-conditioned optimal transport mechanism for assigning incoming visual tokens to memory slots with a retention mechanism that uses learnable per-slot parameters to control content merging and preservation. A lightweight query-aware retriever selects memory cells for answer generation by a frozen multimodal language model. The memory modules and retriever are trained using downstream question-answering supervision.

Critical strengths
  • The paper addresses an important problem: preserving distant evidence in streaming videos under a fixed memory budget and without knowing future queries.
  • The reported results show substantial improvements over strong streaming baselines, with evaluation across multiple backbones and benchmarks.
  • The ablations cover the main components and design choices, providing useful evidence that the gains come from improved memory quality rather than a larger retriever.
Critical weaknesses
  • Algorithm 1 computes the STE surrogate but branches on the hard decision, leaving the gradient path through the memory update unclear.
  • Appendix E introduces a learnable projection W that is missing from the main method and parameter lists.
  • Results are inconsistent across tables, e.g. ReKV scores 37.4 on StreamingBench in Table 1 but 69.1 in Table 9.
  • Qwen3-VL backbone achieves relatively modest gains on most benchmarks.
  • The comparisons rely heavily on older baselines, leaving competitiveness against recent VLMs unclear.
Questions for authors
  • Could the authors clarify how gradients flow through the STE surrogate when Algorithm 1 branches on the hard decision?
  • Could the authors define the projection WW introduced in Appendix E and specify where it is used and how it is trained?
  • Could the authors reconcile the inconsistent results, particularly ReKV’s StreamingBench scores of 37.4 in Table 1 and 69.1 in Table 9?
  • Could the authors explain the relatively modest gains with Qwen3-VL on the ultra-long benchmarks and clarify when the method is most beneficial?
  • Could the authors include comparisons with more recent baselines under matched evaluation budgets?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for their feedback on our paper. We have incorporated it and revised the manuscript accordingly (changes marked in blue). We address the concerns below.

W1/Q1: Gradient flow through the STE surrogate.

We apologize for the confusion caused by Algorithm 1. Its "if/else" shows what the update does in forward pass, and is exactly what runs at inference. Let d=1[sim≥τj]d=\mathbf{1}[\text{sim}\ge\tau_j] (d=1d=1: merge, d=0d=0: allocate), with sim=cos⁡(ΔjW,ck∗W)\text{sim}=\cos(\Delta_j W, c_{k^*}W) (definition of WW explained in W2/Q2). During training, the written content is not chosen by the branch: we compute both outcomes,

cmerge=(1−λj)ck∗+λjΔj,calloc=Δj,c^{\text{merge}} = (1-\lambda_j) c_{k^*} + \lambda_j\Delta_j, \qquad c^{\text{alloc}} = \Delta_j,

and select between them using the STE surrogate d^=d+sim−sg[sim]\hat d = d + \text{sim} - \mathrm{sg}[\text{sim}], while dd determines which cell receives the result:

cnew=d^⋅cmerge+(1−d^)⋅calloc.c^{\text{new}} = \hat d\cdot c^{\text{merge}} + (1-\hat d)\cdot c^{\text{alloc}}.

Forward pass. Since d^=d∈0,1\hat d=d\in\\{0,1\\}, cnewc^{\text{new}} is exactly cmergec^{\text{merge}} when d=1d=1 and exactly callocc^{\text{alloc}} when d=0d=0, as in the hard branch of Algorithm 1.

Backward pass. The loss depends on d^\hat d through cnewc^{\text{new}}, with ∂cnew/∂d^=cmerge−calloc\partial c^{\text{new}}/\partial\hat d = c^{\text{merge}}-c^{\text{alloc}}. Applying the chain rule through d^\hat d and sim\text{sim} gives the gradient of the merge decision:

∂L∂W=(∂L∂cnew)⊤(cmerge−calloc)⏟∂L/∂d^⋅∂d^∂sim⏟=1 (STE)⋅∂sim∂W\frac{\partial\mathcal{L}}{\partial W} = \underbrace{\Big(\frac{\partial\mathcal{L}}{\partial c^{\text{new}}}\Big)^{\top}\big(c^{\text{merge}}-c^{\text{alloc}}\big)}_{\partial\mathcal{L}/\partial\hat d} \cdot \underbrace{\frac{\partial\hat d}{\partial\text{sim}}}_{=1 \text{ (STE)}} \cdot \frac{\partial\text{sim}}{\partial W}

The first factor compares the two outcomes for the written content: if moving it toward cmergec^{\text{merge}} would lower the loss, the update increases sim\text{sim}, favoring merging; otherwise, it decreases sim\text{sim}, favoring allocation. This signal arises at every write, whether the forward pass merged or allocated, and the STE passes it through sim\text{sim} to WW, the incoming content Δj\Delta_j (hence the transport plan and its parameters), and the stored cell ck∗c_{k^*}, so all parameters governing the merge decision are trained by the downstream loss. As is standard for straight-through estimators, this signal is an approximation: the discrete steps of the update, i.e., which cell is written and which two cells are consolidated when a slot is full, are index selections, which are inherently non-differentiable; they have no parameters of their own and are determined by sim\text{sim}.

For simplicity of presentation, we showed only the hard choice in the main text and Algorithm 1, since this is what the update computes in the forward pass and at inference. Additionally, Appendix E started from ∂L/∂d^\partial\mathcal L/\partial\hat d and omitted the first factor, which shows how d^\hat d enters the memory update. We understand this may have made the gradient path unclear, and have revised the manuscript accordingly (changes in blue):

  1. Sec. 3.2 now includes the expression for cnewc^{\text{new}} and explains its forward and backward behavior.
  2. Appendix E now derives the first factor ∂L/∂d^\partial\mathcal L/\partial\hat d, completing the chain rule.
  3. Algorithm 1 now computes both outcomes during training and writes cnewc^{\text{new}}, showing how d^\hat d is used.

W2/Q2: Clarification of the learnable projection WW.

WW, mentioned in Appendix E, is a learnable linear projection used to compute the retention similarity in Eq. 5, sim=cos⁡(ΔjW,ck∗W)\text{sim}=\cos(\Delta_j W, c_{k^*} W), i.e., the two features are compared after a shared projection. As it is a standard detail, we had omitted it from the main description for brevity. WW lets the downstream loss shape what counts as redundant, which is what makes the merge criterion learnable. It is used only for comparison, so stored cells remain in the original feature space, and it is trained with the VL loss through the STE path derived in Appendix E. We have revised the Method section and Appendix E to clarify WW, and listed it among the learnable parameters in Algorithms 1 and 2.

W3/Q3: Reconciling numeric typographical issues.

The ReKV entry in Table 1 was a transcription error, which we have corrected; the correct StreamingBench score is 69.1, as reported in Appendix Table 9 (original, Table 14 in revised). While verifying this, we cross-checked all baseline entries against their original sources and corrected one more transcription error in Table 1: the OVOBench overall scores of Qwen2.5-VL and TimeChat-Online, which appear in adjacent rows, had been swapped. Importantly, these corrections affect only baseline entries copied from prior work: all StreamBalance results are consistent, and no correction changes any ranking or conclusion. All corrected entries are marked in blue in the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

W4/Q4: Gains with Qwen3-VL and when StreamBalance helps most.

We first clarify that the gains with Qwen3-VL are not modest on most benchmarks: such as StreamBalance improves it by +16.5 on OVOBench, +5.4 on StreamingBench. The smaller gap on EgoLife-QA (+3.7) reflects how differently the two models observe the stream. EgoLife-QA was designed for offline agentic systems; the base Qwen3-VL model sees 64 frames sampled uniformly over the history, tens of minutes apart on day-long videos, so the brief evidence often falls between them, whereas StreamBalance, like prior streaming methods, processes every frame at 1 fps, stores it in memory, and reasons over the compressed memory.

Such compression is what makes this benchmark hard for streaming models: prior streaming methods on Qwen2.5-VL fall below their own backbone (e.g., HERMES 30.9 vs. 35.4; Table 1b), likely because distant evidence is lost under compression. StreamBalance is the only streaming method that surpasses its backbone on every backbone we evaluate by a good margin on this ultra-long benchmark. Our goal with such ultra-long benchmarks is to show that even under extreme compression, a streaming model can answer questions over ultra-long videos from the actual answer-bearing evidence, retained across very long histories, without collapsing as the evidence lies further back, whereas the backbone, with its sparse sampled view, largely answers without it. The +3.7 gap thus compares a backbone that largely answers without the evidence with a streaming model that answers from it under extreme compression.

To verify that its answers come from the retained evidence, we replace the frames within ±120 s of each question's annotated evidence timestamp with black frames, keeping all other frames and timestamps unchanged (w/o evidence).

Table W4: EgoLife-QA by certificate length (Qwen3-VL-8B backbone). Values in parentheses: change from removing the evidence.

Method<2h2–6h6–24h>24hOverall
Blind (no video)32.331.825.627.829.4
Qwen3-VL-8B42.637.239.534.938.6
Qwen3-VL-8B w/o evidence41.0 (−1.6)35.3 (−1.9)39.1 (−0.4)33.4 (−1.5)37.2 (−1.4)
+ StreamBalance (Ours)44.843.141.739.442.3
+ StreamBalance (Ours) w/o evidence35.7 (−9.1)32.5 (−10.6)28.4 (−13.3)30.1 (−9.3)31.7 (−10.6)

Removing the evidence changes Qwen3-VL by only 1.4 points, consistent with the evidence rarely being among its sampled frames, and its accuracy fluctuates non-monotonically with certificate length, mirroring prior streaming methods in Fig. 4 of the main paper. In contrast, StreamBalance drops by 10.6 points, close to the blind model, at every length, showing that its accuracy comes from the evidence it retains and retrieves, and it decays gently as the evidence lies further in the past. StreamBalance is therefore most beneficial when the answer depends on specific evidence far in the past, which it consistently retains and retrieves.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

W5/Q5: Clarification on baseline recency and evaluation budgets.

We clarify that our direct comparators are recent in every table (1a, 1b, 2). On general streaming benchmarks (Table 1a), every streaming method built on our backbones was published in 2025–2026 at a top CV/ML/NLP venue, e.g., StreamReady (CVPR'26), HERMES (ACL'26), QueryStream (ICLR'26), and InfiniPot-V (NeurIPS'25). On ultra-long benchmarks (Table 1b), we additionally compare with recent agentic systems, e.g., WorldMM (CVPR'26) and M3-Agent (ICLR'26), and on offline long-video benchmarks (Table 2), with recent methods such as Streamo (CVPR'26), TimeSearch-R (ICLR'26), and StreamingTom (CVPR'26).

As a streaming method, StreamBalance's direct comparators are these prior streaming methods, which we compare under a matched evaluation protocol: within each backbone block, all methods share the same backbone and the same 1 fps streaming input. Memory budgets, however, cannot be matched directly: each method retains a different unit of memory (e.g., KV caches, pruned visual tokens, or compressed memory cells), so the same nominal size is not comparable across methods, and changing it would depart from each method's own reported best configuration; we therefore keep every method's memory budget at its best reported setting. Offline MLLMs, in contrast, are not direct comparators and cannot be evaluated under a matched streaming budget, as their context length cannot hold the unbounded 1 fps stream. Following prior streaming work, they are reported for completeness only, from the benchmarks' official papers, under the standard protocol of at most 64 frames sampled from the 1 fps video. Their recency therefore does not affect our comparisons or conclusions.

As requested, we additionally include more recent models under the same protocol (below): two more 2026 streaming methods, FluxMem and MuKV (official reported results), and a more recent backbone, Qwen3.5-4B (Feb'26; evaluated by us). StreamBalance outperforms both new streaming methods by a clear margin and yields large gains on the new backbone.

Table W5: Comparison with more recent models. FluxMem and MuKV: official reported results. ‡: evaluated by us under the official protocol.

MethodReleaseOVOBench (Real-Time)StreamingBench (Real-Time)
Recent streaming methods (Qwen2.5-VL-7B backbone)
FluxMemCVPR'2667.276.4
MuKVCVPR'26–64.4
StreamBalance (Ours)–75.182.1
More recent backbone
Qwen3.5-4B‡Feb'2668.077.0
+ StreamBalance (Ours)–82.583.2
Reviewer_24yiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

StreamBalance casts streaming memory as a balanced temporal coverage problem. A fixed bank (128 slots × 8 cells) is written by a state-aware, entropy-regularized optimal-transport assignment with learned per-slot capacity; each slot learns a decay rate that controls merge-vs-allocate; a query-aware retriever feeds the top-64 cells to a frozen MLLM, and the memory is trained end-to-end from the downstream loss. Experiments cover OVO-Bench, StreamingBench, EgoLife-QA, Ego-R1 Bench, and three offline benchmarks.

Critical strengths

The framing is crisp and the diagnosis is evidenced: Fig. 6 shows heuristic memories collapsing toward recency and attention-based writers concentrating arbitrarily, while the proposed bank keeps coverage flat.

Results are consistent across three backbones and five benchmarks; Backward-split gains are large (e.g., 44.7 → 71.9 with Qwen2.5-VL), and EgoLife-QA remains stable up to 44 h.

Ablations are unusually thorough: each mechanism, alternative writers (Expert Choice, TTM, Q-Former), fixed-hierarchy retention, OT solver, decay rates, temporal metadata, capacity, and retrieval-K sensitivity. Overhead is modest (Sinkhorn ≈1.8 ms/frame vs. 46.5 ms ViT encoding), and the paper separates write cost, TTFT, and memory.

Critical weaknesses
  1. Baseline protocol is unclear. Most StreamingBench/OVO-Bench numbers appear to be quoted from prior papers with different backbones, frame rates, prompts, and training budgets. It is not stated which were re-run; a "reported vs. reproduced" table is needed for the SOTA claim.
  2. No variance. Single run, no seeds or confidence intervals. Several reported margins are only a few points on small benchmarks (OVO-Bench, Ego-R1 Bench), so their significance is unclear.
  3. Coverage–accuracy link is correlational. Fig. 6 shows that coverage differs, not that it explains the accuracy gap. Uniform coverage is also a strong prior that could hurt queries needing fine detail in one short span; stratify accuracy by certificate length and coverage, and discuss failure cases.
  4. Efficiency claims are inconsistent. The abstract says "1.2× less compute", §4.3 says "1.2× less memory"; "8× lower latency" is TTFT only, while ViT encoding dominates end-to-end cost. Report end-to-end latency/throughput and clarify the reference method.
  5. Benchmark subset. Removing HiREST-sourced videos from OVO-Bench ASI changes the test set; the † scores are not directly comparable to published numbers unless baselines are recomputed on the same subset.
  6. Minor numeric issues. The Qwen2.5-VL base is 46.5 in Table 1 but 47.7 in Table 8; the "+8.3 on EgoLife-QA" appears to be over the full-history backbone (35.4), not the best streaming baseline (31.5). Please reconcile.
Questions for authors
  1. Which baselines were re-run under your exact protocol, and with what frame sampling and memory budget?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/4)

We thank the reviewer for their positive assessment of and feedback on our paper. We have incorporated the feedback and revised the manuscript accordingly (changes marked in blue). We address the concerns below.

W1/Q1: Baseline protocol clarification.

All numbers in Tables 1a and 1b are the official scores reported in each method's own paper, except the streaming results on the ultra-long benchmarks (Table 1b). Since no streaming results existed on these benchmarks, we ran those ourselves using each method's released code and default configuration; they are now marked in the revised tables. Within each block, all methods share the same backbone and the standard 1 fps streaming input (Sec. 4, Appendix F), and all benchmarks are evaluated with their official prompts and scoring scripts. Trained methods build on a shared core of public video instruction-tuning data, most commonly LLaVA-Video-178K (e.g., TimeChat-Online, StreamForest, and StreamBridge), as does ours.

The methods are thus comparable in every respect except what each retains of the stream, which is precisely what makes each method distinct. Each method's retention strategy and its budget are designed to work best in its own setup, so changing them would mean evaluating a different model rather than the method as proposed. Moreover, since methods retain different units of memory (e.g., a KV cache for HERMES, pruned visual tokens for TimeChat-Online, frame embeddings for StreamReady, and compressed cells for ours), they cannot be brought to a common memory size, as the same nominal size holds different content for each. We have added this description of the evaluation protocol to the revised manuscript (Appendix F, in blue).

W2: Statistical significance of the reported margins.

We use greedy decoding (temperature 0), so evaluation is deterministic for a given trained model. As retraining with multiple seeds was not feasible within the rebuttal period, we assess the significance of the reported margins with respect to the test-set sample using 95% bootstrap confidence intervals and two-sided pp-values (1,000 resamples of the test questions). For baselines reproduced by us, we use a paired test on the margin; otherwise, we compare the baseline's reported score against our bootstrap distribution. The margins over the best prior method are significant on all benchmarks except one (p≤0.012p \le 0.012), including OVOBench (p<0.001p < 0.001 with both backbones) and Ego-R1 Bench with Qwen2.5-VL (+10.0, p=0.004p = 0.004). The exception is Ego-R1 Bench with Qwen2-VL (+2.8, p=0.39p = 0.39), where the benchmark's small size (300 questions) limits statistical power. We have added this analysis to the revised manuscript (Table 12 Appendix, in blue).

Table W2a: Significance of results. * = paired test against a baseline reproduced by us.

Benchmark (backbone)nnOurs (95% CI)Best priorMarginMargin 95% CIpp-value
OVOBench (Qwen2-VL)278074.6 [72.6, 76.8]68.2 (StreamReady)+6.4–<0.001
OVOBench (Qwen2.5-VL)278068.0 [66.2, 70.0]55.6 (Streamo)+12.4–<0.001
EgoLife-QA (Qwen2-VL)50038.5 [34.9, 41.8]31.5 (StreamReady)+7.0[+1.6, +11.2]*0.012*
EgoLife-QA (Qwen2.5-VL)50043.7 [39.4, 48.5]30.9 (HERMES)+12.8[+7.3, +17.8]*<0.001*
Ego-R1 Bench (Qwen2-VL)30032.6 [28.2, 37.5]29.8 (StreamReady)+2.8[−3.5, +9.5]*0.39*
Ego-R1 Bench (Qwen2.5-VL)30038.3 [33.2, 44.0]28.3 (HERMES)+10.0[+3.2, +16.5]*0.004*
AuthorsOpenReview ↗

Official Comment by Authors (2/4)

W3: Coverage–accuracy link and failure cases.

Beyond the correlation in Fig. 6, we provide two interventional results. First, our ablations change only the write mechanism, with everything else fixed (Table 3(d), Fig. 3), which lowers memory balance (utilization) and accuracy together, most on long-range splits. Second, removing the answer-bearing evidence from the stream (blacking out frames within ±120 s of each question's annotated evidence timestamp) drops StreamBalance from 42.3 to 31.7 on EgoLife-QA, close to the blind model (29.4), while the base model changes by only 1.4 points, showing that our gains come from retaining and retrieving the evidence.

Table W3a: Evidence removal on EgoLife-QA (Qwen3-VL-8B). Values in parentheses: change from removing the evidence.

Method<2h2–6h6–24h>24hOverall
Blind (no video)32.331.825.627.829.4
Qwen3-VL-8B42.637.239.534.938.6
Qwen3-VL-8B w/o evidence41.0 (−1.6)35.3 (−1.9)39.1 (−0.4)33.4 (−1.5)37.2 (−1.4)
+ StreamBalance (Ours)44.843.141.739.442.3
+ StreamBalance (Ours) w/o evidence35.7 (−9.1)32.5 (−10.6)28.4 (−13.3)30.1 (−9.3)31.7 (−10.6)

As suggested, we further stratify accuracy by certificate length together with coverage for the three memories of Fig. 6. As certificates grow longer, the coverage of the heuristic and attention-based memories drops overall, and their accuracy drops with it, whereas StreamBalance maintains both, consistent with coverage being what enables distant evidence to be recalled.

Table W3b: Coverage and accuracy by certificate length on EgoLife-QA (Qwen2.5-VL-7B). Cov: mean per-question coverage; Acc: accuracy (%). Overall: mean over certificate-length groups.

Memory<2h Cov<2h Acc2–6h Cov2–6h Acc6–24h Cov6–24h Acc>24h Cov>24h AccOverall CovOverall Acc
Heuristic0.8843.20.7337.80.6922.90.4515.20.6929.8
Attention-based0.7846.00.8143.20.6436.40.5829.60.7038.8
StreamBalance (Ours)0.9449.20.9344.40.9541.80.9339.30.9543.7

Regarding fine detail: balanced coverage is over time, not uniform in content: a distinct event receives its own cell rather than being merged away, and cells in slow slots change little after being written (Fig. 7), so short-span detail is preserved. Consistently, StreamBalance is also best on the shortest certificates (<2h) and on OVOBench Real-Time, where detail within a short span matters most.

To discuss failure cases, we categorize all incorrect answers of StreamBalance on EgoLife-QA by where the answer-bearing evidence is lost. The largest share (53%) are reasoning errors, where the evidence is retrieved but the frozen LLM still answers incorrectly; these largely involve questions that require identifying specific people or aggregating multiple events. Memory-side failures are smaller: the evidence is not retained (18%) or not retrieved (21%), while fine-grained detail consolidated with similar content, the case raised by the reviewer, accounts for only 8% of errors. We have added this analysis to the revised manuscript (Appendix Table 5, 11, in blue).

Table W3c: Breakdown of StreamBalance errors on EgoLife-QA (Qwen2.5-VL-7B). Evidence cell: a memory cell whose interval overlaps the annotated evidence timestamp (±120 s). Over-merged: the evidence cell has been merged more than 20 times since the evidence was written, leaving only a small fraction of the original content.

Failure typeErrors (n / %)
Evidence not in memory51 / 18%
Evidence over-merged (fine detail)22 / 8%
Evidence not retrieved59 / 21%
Reasoning error (evidence retrieved)149 / 53%
AuthorsOpenReview ↗

Official Comment by Authors (3/4)

W4: Clarification on efficiency metrics and end-to-end latency.

We clarify that "1.2×" refers to peak GPU memory (VRAM) usage and "8×" to TTFT, both relative to HERMES at 16K frames. We have revised the abstract and Sec. 4.3 to state this consistently.

In streaming, the video is never processed as a whole. Frames are encoded as they arrive, and a query may come at any point, so streaming latency measures how fast a model answers a query given only what it has already observed, without revisiting past frames. The ViT cost of past frames is thus paid well before the query, not while answering it. Moreover, with the same frozen backbone, this per-frame ViT cost is identical across methods. What differs is TTFT (memory retrieval, LLM prefilling, and first-token generation), which grows with stream length for prior methods due to what they retain and retrieve from (a growing token bank, a large KV cache, or a heavy retriever), whereas StreamBalance reads from a fixed-size memory.

For completeness, we report the end-to-end latency once a query arrives (current-frame ViT and storage + TTFT) below. Since the shared ViT cost is a constant added to every method, it bounds the end-to-end speedup, as accelerating part of a pipeline is limited by the part left unchanged. This is precisely why streaming work compares TTFT, which isolates the cost in which methods actually differ. Even so, StreamBalance is 3.3× faster end-to-end than HERMES and over 20× faster than the other methods. We have added this analysis to the revised manuscript (Sec. 4.3, in blue).

Table W4: End-to-end latency once a query arrives at 16K observed frames. All methods use the same Qwen2.5-VL-7B backbone; StreamReady, originally built on Qwen2-VL, is reproduced on it (‡). Storage: overhead of writing the current frame into each method's memory. Response latency: TTFT. Our speedup = end-to-end latency of the method / that of StreamBalance.

MethodViT (ms)Storage (ms)Response latency (ms)End-to-end latency (ms)Our speedup
TimeChat-Online46.51.267746821.7106.1×
StreamReady‡46.55.413361387.921.6×
HERMES46.535128209.53.3×
StreamBalance (Ours)46.51.81664.3–
Reviewer_6ZapOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method called "StreamBalance" Its main motivation is to strike a balance between the heuristic timestep-based retention rule, and the over-focus using attention scores. In this way, the authors propose an approach of fully learnable memory mechanism based on the formulation of optimal transport. Under this design, the memory also absorbs the idea of slow and fast systems to memorizing different types of events. Finally, the authors conducted experiments on Egolife and OVOBench to support the effectiveness of their approach.

Critical strengths
  1. The overall problem is real, and the issues with the earlier approaches exist.
  2. The fully end-to-end learnable way of memory is definitely better than designing manual heuristics. The formulation is also overall reasonable.
  3. The performance comparison on the benchmarks supports the benefits of the approach.
Critical weaknesses
  1. The benefit beyond simple policies that already avoid recency bias should require more comparison. The motivation centers on retaining evidence across the full history. The comparisons with hierarchical memories and attention-based writers are useful, but analysis and comparison with other baselines like reservoir-sampling or simple temporal-coverage baseline would be useful. Such a comparison would test whether learned content-dependent allocation and retention add value beyond spreading memory across the past.

In addition, it might be useful to explain mechanistically, or use some qualitative examples to show why the model CAN learn such behaviors.

  1. I have some confusion about the state-aware assignment explanation. The paper attributes substantial gains to pressure/staleness terms that redirect writes between slots. However, these terms add the same offset to every token assigned to a given slot. If the balanced transport enforces a prescribed total for each slot, those offsets cannot change the converged assignment. Table 3 nevertheless reports a large effect from removing them. The actual formulation might be better to explain this discrepancy?

  2. Balanced slot use and timestamp coverage do not fully support preservation of old answer evidence. The results in Figure 6 measures first-write timestamps even though stored features can change through later merging, e.g., memory can retain an old timestamp while losing its original content. Showing that distant answer evidence remains accessible would strengthen the connection between the proposed memory behavior and the reported gains. Also related to my first point, it is better to have some mechanistic explanation here.

Questions for authors

See above

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for their positive assessment of and feedback on our paper. We have incorporated the feedback and revised the manuscript accordingly (changes marked in blue). We address the concerns below.

W1: Comparison with content-agnostic coverage policies, and why the behavior is learnable.

To test whether learned, content-dependent allocation and retention add value beyond spreading memory across the past, we compare with the two baselines suggested by the reviewer, which decide what to keep without regard to content, under the same memory budget, retriever, and training:

  1. Reservoir sampling, which stores every incoming cell until the memory is full and then admits each new cell with a probability that decreases with the number of cells seen so far, replacing a random stored cell, so that every past cell is equally likely to be retained.
  2. Uniform temporal subsampling, a simple temporal-coverage baseline, which keeps evenly spaced frames and halves their density whenever the memory is full.

Both cover the full history uniformly by construction, yet fall clearly behind StreamBalance in accuracy, by up to 12.6 points on $\geq$6h certificates. Spreading memory evenly in time thus does not suffice, likely because it spends capacity on recurring content and misses rare events. Coverage is necessary but not sufficient, and learned content-dependent allocation and retention are what make it effective.

Table W1: Comparison with content-agnostic coverage policies (Qwen2.5-VL-7B, same memory budget, retriever, and training).

Write / retentionOVOBench BackOVOBench OverallEgoLife-QA <6hEgoLife-QA ≥6hEgoLife-QA Overall
Reservoir sampling59.560.236.429.533.1
Uniform temporal subsampling64.163.735.132.834.0
StreamBalance (Ours)71.968.045.242.143.7

Regarding why such behavior can be learned: every memory decision receives a learning signal from the answer loss, so the model is directly trained on what to keep. (i) Routing is differentiable through the Sinkhorn plan, so the loss shapes which content each slot receives via the slot anchors, state terms, and capacity. (ii) Retention has an exact gradient through the merge strength λj\lambda_j: when a slot merges, the gradient lowers its decay rate αj\alpha_j if preserving the stored content would have helped a later answer, and raises it if refreshing with new content would have helped. (iii) The merge-vs-allocate decision receives a gradient through the STE, indicating whether moving the written content toward merging or allocating would have lowered the loss. Since each slot has its own αj\alpha_j and receives different content, these signals differ across slots, so slots specialize into fast and slow ones. Our results confirm this: learned decay rates are strongly correlated with the temporal horizon each slot retains (Fig.7, ρ=−0.89\rho=-0.89), with slow slots holding content from further back, and a single shared decay rate cannot replace per-slot rates.

W2: Clarification on the state-aware cost formulation.

We agree with the reviewer's analysis: if Sinkhorn enforced the slot marginal exactly, slot-only cost terms would be absorbed by the column scaling and leave PP unchanged. StreamBalance, however, uses a soft slot-side marginal. With G=exp⁡(−C^/ε)G=\exp(-\hat C/\varepsilon), each Sinkhorn iteration in Eq. (3) performs

ui←ai(Gv)i,vj←(bj(G⊤u)j)κ,where κ=0.85,u_i \leftarrow \frac{a_i}{(Gv)_i}, \qquad v_j \leftarrow \left(\frac{b_j}{(G^\top u)_j}\right)^{\kappa}, \qquad \text{where } \kappa=0.85,

and returns P=diag(u)Gdiag(v)P=\mathrm{diag}(u)G\mathrm{diag}(v), where aia_i is the mass of token ii. The token-side update is exact, so every token's mass is fully distributed, while the exponent κ\kappa on the slot-side update pulls slot loads toward the capacity bb without forcing them to equal it. This is the behavior described in the text ("pulled toward matching", L198; "balanced toward its learned capacity", L237). Since Eq. (3) presents the transport step at a high level, as P=OT(C^,b)P=\mathrm{OT}(\hat C, b), the individual Sinkhorn updates, and hence κ\kappa, were not shown explicitly. In the revision, we describe the soft slot-side update after Eq. (3) and give the full update in Appendix F (in blue).

Although the pressure/staleness offset is shared by all tokens of a slot, under the soft marginal it is only partially compensated by the column scaling, so it shifts that slot's total intake: down for heavily used slots (pressure) and up for dormant ones (staleness). Since each token's mass is fully distributed, the displaced mass moves between slots, changing the routing, and the resulting change in slot loads wjw_j also changes write decisions (wj>c/Nkw_j>c/N_k). The state terms thus have no effect only with an exact slot marginal (κ=1\kappa=1); under our soft marginal they change the routing, which explains the effect of removing them in Table 3(b).

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W3: Preservation of distant answer evidence.

We clarify that Fig. 6 is intended to show where in time the memory's content originates, i.e., that writing is not recency-biased; preservation of that content is supported by the retention rule and verified directly below.

Mechanistically, complementing W1 (how slots learn to specialize), the retention rule (Eq. 5) explains why content survives: a cell is updated only by similar content, and only slightly so in slow slots, whose low decay rate keeps the merge strength small. Dissimilar content is allocated to a new cell rather than overwriting an existing one, and when a slot is full, only its two most similar cells are merged. A distinct, rare event is therefore not overwritten by dissimilar later content.

To verify this directly, we remove the answer-bearing evidence from the stream by blacking out the frames within ±120 s of each EgoLife-QA question's annotated evidence timestamp, keeping all other frames and timestamps unchanged. If the memory retained only an old timestamp but had lost the original content, removing the evidence would have little effect. Instead, StreamBalance drops sharply to near the blind model at every certificate length, including the longest (>24h), whereas the base model is almost unaffected (−1.4). This shows that the answer-bearing content itself, not just its timestamp, is retained in memory and used at query time.

Table W3a: Evidence removal on EgoLife-QA (Qwen3-VL-8B). Values in parentheses: change from removing the evidence.

Method<2h2–6h6–24h>24hOverall
Blind (no video)32.331.825.627.829.4
Qwen3-VL-8B42.637.239.534.938.6
Qwen3-VL-8B w/o evidence41.0 (−1.6)35.3 (−1.9)39.1 (−0.4)33.4 (−1.5)37.2 (−1.4)
+ StreamBalance (Ours)44.843.141.739.442.3
+ StreamBalance (Ours) w/o evidence35.7 (−9.1)32.5 (−10.6)28.4 (−13.3)30.1 (−9.3)31.7 (−10.6)

Consistently, our failure analysis shows that evidence lost to repeated merging accounts for only 8% of StreamBalance's errors on EgoLife-QA, while the majority (53%) occur even though the evidence was retained and retrieved, where the frozen LLM still answers incorrectly.

Table W3b: Breakdown of StreamBalance errors on EgoLife-QA (Qwen2.5-VL-7B). Evidence cell: a memory cell whose interval overlaps the annotated evidence timestamp (±120 s). Over-merged: the evidence cell has been merged more than 20 times since the evidence was written, leaving only a small fraction of the original content.

Failure typeErrors (n / %)
Evidence not in memory51 / 18%
Evidence over-merged (fine detail)22 / 8%
Evidence not retrieved59 / 21%
Reasoning error (evidence retrieved)149 / 53%

Together, these results connect the memory's behavior to the reported gains: distant answer-bearing content is retained and used, rather than only its timestamp. We have added the evidence-removal analysis and the failure analysis to the revised manuscript (Table 6, 11 Appendix).

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 6Zap,

We thank you again for your positive assessment and constructive feedback. We hope our responses, including the comparison with reservoir sampling and uniform temporal coverage, the clarification of the soft Sinkhorn marginal, and the evidence-removal analysis, have addressed your concerns. We would be happy to clarify anything further. Thank you.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers and Chairs,

We sincerely thank all reviewers for their time and constructive feedback, which helped us improve the paper. We briefly summarize the reviews and our revisions (changes marked in blue in the revised manuscript).

Strengths. All reviewers found the problem important and the framing of streaming memory as balanced temporal coverage well motivated (6Zap, 24yi, JvD8), the fully learnable, end-to-end memory an improvement over manual heuristics (6Zap), the gains consistent across three backbones and several benchmarks with stable accuracy on ultra-long streams (6Zap, 24yi, JvD8), and the ablations thorough (24yi, JvD8).

Main concerns and revisions.

  • Method clarity (JvD8, 6Zap): we clarified how gradients flow through the straight-through estimator, defined the learnable projection WW, and made the soft Sinkhorn marginal explicit, which explains the effect of the state-aware cost.
  • Retention of distant evidence (6Zap, 24yi): an evidence-removal experiment shows that StreamBalance's accuracy drops to near the blind floor without the answer-bearing evidence, while the base model is almost unaffected; a failure analysis shows that over-merging accounts for only 8% of errors.
  • Additional comparisons (6Zap, JvD8): StreamBalance clearly outperforms reservoir sampling and uniform temporal coverage, and a newer Qwen3.5 backbone.
  • Evaluation rigor (24yi, JvD8): we clarified the evaluation protocol, added bootstrap significance tests, end-to-end latency, and a HiREST subset analysis, and corrected transcription errors in baseline entries, none of which changes any ranking or conclusion.

Following the rebuttal, Reviewer JvD8 noted that most concerns were addressed and raised their score; we also thank Reviewers 6Zap and 24yi for their positive assessment of the paper. We are happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.