acceptodds
Withdrawn from ICLR 2027

Learning What to Keep: Causal Memory Compression for Streaming Audio-Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming omnimodal systems often need to answer questions about earlier events displayed. Retaining the full history becomes increasingly expensive as the stream grows, forcing the system to decide what information to keep or delete. We in- troduce Causal Adaptive Memory Compression (CAMC), a method for manag- ing past-information retention. CAMC allocates a limited memory budget across stream segments, audio and video, and individual tokens. It uses two signals: historical novelty, which measures how different new content is from stored infor- mation, and temporal change, which measures how different it is from prior ob- servations. A dynamic budget enables CAMC to retain useful information while staying close to an overall retention target. Additionally, we present StreamRe- call, a streaming audio-video multiple-choice benchmark that evaluates whether information from earlier events remains accessible when queried later. CAMC achieves the highest average answer accuracy evaluated while retaining a fraction of the original audio-video history.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal token compression, Streaming, multimodal understanding, OmniLLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VzhxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Causal Adaptive Memory Compression (CAMC), a training-free method for streaming audio-video understanding. It uses historical novelty and temporal change to adaptively allocate memory budgets across video segments, modalities, and individual tokens. The authors also introduce StreamRecall, a benchmark for evaluating delayed access to historical audio-video evidence. Experiments on three benchmarks show competitive performance under different token retention budgets.

Critical strengths
  1. The paper addresses a practical problem in streaming audio-video understanding. The proposed compression strategy is simple, training-free, and explicitly considers both historical novelty and temporal changes.

  2. The experiments cover three benchmarks and several recent compression methods under different retention budgets. The ablation studies support the usefulness of the proposed signals. The gold-evidence analysis is also interesting, revealing that historical retrieval remains a major bottleneck even without compression.

Critical weaknesses
  1. Limited novelty. The method mainly combines existing techniques, including cosine-similarity-based novelty estimation, temporal difference scoring, adaptive budget allocation, and Top-K token selection. While the overall design is reasonable, the technical contribution over existing compression methods is relatively incremental.

  2. Modest improvements and unclear computational overhead. Although CAMC achieves the highest average accuracy, the gains over the strongest baselines are relatively small (0.81 and 0.60 points at the two retention levels). The benefit of adaptive modality allocation is also inconsistent across benchmarks. Moreover, the paper lacks a detailed analysis of compression overhead and how the historical similarity computation scales with stream duration.

Questions for authors
  1. Can the authors report the computational overhead of CAMC, including compression latency and memory usage, especially as the historical memory grows?

  2. Given the relatively small gains, how much does the three-level adaptive allocation improve over simpler compression strategies using only historical novelty or temporal change?

  3. Can the authors evaluate CAMC on another Omni-LLM backbone to verify whether the improvements generalize beyond Qwen2.5-Omni-7B?

1 comment

We thank the reviewer for a thorough review and for recognizing the gold-evidence analysis. We respond to the weaknesses and questions below.

W1 — Novelty. We agree that each primitive (cosine novelty, temporal difference, Top-K selection) is simple; the contribution lies in the problem formulation and in how the primitives are composed under its constraints. We formalize streaming memory formation as causal, query-blind and irreversible allocation at three levels (chunk, modality, token), a setting that most existing compressors violate by seeing the whole clip, conditioning on the question, or keeping a recoverable pool. Within it, historical novelty is measured against everything already retained, which detects recurrences that temporal signals cannot, and the same two signals drive both the audio/video split and the token ranking. The revised version simplifies the design further — a fixed per-chunk budget replaces the adaptive controller, making retention exactly comparable across methods — and we regard the fact that a method this simple is the most consistent performer across six settings as the point, not a weakness.

W2 — Size of the gains, modality allocation, overhead.

Gains. We now report paired, video-clustered bootstrap intervals with Holm correction for every comparison, and we frame the claim as an aggregate one. Over the six benchmark × regime cells our method has the highest average in both regimes and overall and is within one point of the best method in every cell, whereas every baseline falls several points behind the best in at least one cell, and the best baseline changes from cell to cell. Under tight retention our method is significantly better than random retention and OmniZip on the delayed-recall benchmark; no baseline is significantly better than our method anywhere. For a compressor that must be chosen before the workload is known, this robustness is the relevant property, and no baseline has it. Modality allocation. The revised ablations are run under the fixed budget so that all variants retain exactly the same number of tokens. The content-dependent split contributes most where the budget is tight; at generous budgets the allocation matters less, and we state this explicitly rather than claiming a uniform benefit. Overhead. See Q1.

Q1 — Computational overhead and scaling. The history is summarized by one normalized key vector per modality per chunk, so the novelty computation is a single matrix–vector product per chunk (one maximum cosine similarity over the stored keys); its cost grows linearly with the number of chunks but is negligible relative to encoding and generation — a 34-minute stream yields ~140 keys. Memory for keys is likewise a few hundred kilobytes per hour of stream. The supplementary material reports measured query latency and memory per method against accuracy: our method sits within the latency range of the compared compressors, well below the uncompressed model, while achieving the best mean accuracy rank.

Q2 — Three-level allocation vs. single signals. The component ablations answer this directly. Under a fixed budget, removing temporal change costs 1.4 points and removing historical novelty 0.8 points on StreamingBench at tight retention, and the uniform modality split costs a further 0.8; together the components account for most of the ~4-point margin over random retention there. Either signal alone recovers much of the gain at generous budgets, which is expected for two correlated cues, but both are needed where the budget binds. Moreover, since the budget is now fixed, the random-retention baseline in Table 1 is exactly “our budget with random token selection”; the gap to it (3–5 points under tight retention on the delayed-recall benchmark) is the contribution of the ranking.

Q3 — Other backbones. The revised supplementary material adds results on two further omni-modal backbones with different audio and vision front-ends, together with an analysis of baseline portability: embedding-only methods (ours, random retention, VidCom2) transfer unchanged, whereas methods that depend on backbone-specific attention maps, sidecars or a trained selector cannot be faithfully ported without re-deriving them per backbone. The method’s behaviour on the additional backbones is consistent with the main results, and we consider this portability a practical advantage of a training-free, embedding-based design.

We thank the reviewer again; the paired statistics, the fixed-budget ablations, the overhead analysis and the additional backbones all follow from this review.

Reviewer_z4F8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CAMC, a training-free method that allocates historical audio-video retention across chunks, modalities, and tokens using historical novelty and temporal change under a cumulative retention target. It also introduces StreamRecall, a 750-question benchmark with timestamped supporting events for delayed queries. Experiments on three benchmarks show that CAMC achieves competitive performance under constrained memory budgets.

Critical strengths
  1. The proposed method is training-free and potentially easy to integrate. Additionally, the paper introduces StreamRecall, a benchmark that enables a more fine-grained evaluation of streaming memory.
Critical weaknesses
  1. The proposed method has limited novelty, as it mainly combines existing techniques from prior work. Additionally, the paper lacks comparisons and evaluations across other streaming video understanding models. Furthermore, the absence of an intuitive framework diagram or qualitative case studies makes the proposed approach somewhat difficult to understand.
Questions for authors

See weakness and:

  1. The proposed method uses mean pooling over all tokens to represent each audio or video chunk. Is this design reasonable? Could it potentially obscure brief but important events?
  2. CAMC relies on comparisons with previously stored historical representations, which continue to accumulate over time. Would this introduce substantial computational overhead as the video length increases?
1 comment

We thank the reviewer for the feedback and respond to each point.

Novelty. The contribution is not a new scoring primitive but a new problem formulation and a method built around its constraints. We formalize streaming memory formation as causal, query-blind and irreversible token allocation at three levels (chunk, modality, token) — a setting that most existing compressors do not satisfy, because they see the whole clip, condition on the question, or keep a recoverable pool. Within that setting, the combination of historical novelty (comparison against everything already retained, which detects recurrences that purely temporal signals miss) with temporal change, applied both to the audio/video split and to token ranking, is, to our knowledge, not present in prior work; the ablations show that both signals contribute where the memory budget is tight. The revised version also makes the budget allocation simpler and fully comparable across methods (a fixed per-chunk budget), which we consider a strength rather than a loss of novelty.

Comparisons across other streaming models. The main experiments use a single frozen omni-modal backbone so that seven compressors can be compared at exactly matched retention. The supplementary material of the revised version adds results on two further omni-modal backbones with different audio and vision front-ends, together with an analysis of which baselines can be ported faithfully to a new backbone: embedding-only methods (ours, random, VidCom2) transfer directly, whereas methods relying on backbone-specific attention maps, sidecars or trained selectors (OmniZip, OmniPack, VisionZip-OM, SEATS, OmniSIFT) cannot be transferred without re-deriving or re-training them for each backbone. We believe this portability is itself an important practical property of a training-free, embedding-based design.

Diagram and qualitative examples. The revised paper includes a full framework overview figure (stream and encoders → content signals, budget split and token selection → causal memory → query-time retrieval and answering), and the supplementary material adds qualitative examples of retained versus pruned content and of delayed-recall questions.

Q1 — Mean pooling and brief events. The pooled chunk key is used only for the chunk-level signals that drive the audio/video split. Token retention is decided by per-token scores: each token is compared against the stored history and the previous chunk individually, and a local-distinctiveness term explicitly favours tokens that differ from their own chunk’s mean. A brief event inside a 15-s chunk therefore produces high-scoring tokens even when the chunk mean is dominated by background, and it is retained at the token level. The gold-evidence analysis supports this: evidence events in the benchmark have a median duration of a few seconds, and restoring the retained chunks that overlap them raises accuracy to 80%+, so the brief events are present in memory.

Q2 — Computational overhead over long streams. The history is represented by one normalized key vector per modality per chunk; the novelty computation is a maximum cosine similarity over these keys, i.e. a single matrix–vector product per chunk whose cost grows linearly with the number of chunks but remains negligible next to the encoder and LLM (a 34-minute stream has ~140 keys of a few thousand dimensions). We report measured query latency against accuracy in the supplementary material: our method lies within the latency range of the compared compressors, well below the uncompressed model, and far from the slowest baselines, while achieving the best mean accuracy rank.

We thank the reviewer for prompting the overview figure and the backbone-portability discussion, both of which are now part of the paper.

Reviewer_2dhwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to retain useful information from streaming audio-video inputs when future questions are unknown. The authors propose Causal Adaptive Memory Compression (CAMC), a training-free method that makes memory retention decisions at three levels: how many tokens to keep from each chunk, how to divide the budget between audio and video, and which individual tokens to retain. These decisions are based on historical novelty and temporal change, together with a cumulative budget controller. The paper also introduces StreamRecall, a benchmark containing 750 questions from 424 videos, with annotated timestamps for supporting evidence. Experiments on three streaming audio-video benchmarks show that CAMC achieves the highest average accuracy among the compared methods at both evaluated retention levels. The authors further analyze how access to historical evidence affects question-answering performance.

Critical strengths

1. The streaming setting is meaningful, and the method is designed around its main constraints.

The paper considers a practical setting where memory must be constructed before future questions are known, and discarded information cannot be recovered later. CAMC addresses this by adjusting retention across chunks, modalities, and tokens. The cumulative budget controller also provides a clear way to manage memory usage over time. I think this is a reasonable approach to a problem that is different from standard offline token compression.

2. The analysis of historical evidence access provides a useful finding beyond the compression results.

The gold-evidence experiment in Section 6.4 is particularly interesting. Table 3 shows that even the uncompressed model improves from 55.07% to 83.20% when it is given direct access to the annotated evidence. This suggests that locating and using the right historical information is itself a major challenge, even when that information has not been removed by compression. I find this analysis valuable because it helps separate the problem of preserving information from the problem of accessing it later.

Critical weaknesses

1. The performance improvements over the strongest baselines are quite small, and their reliability is not sufficiently established.

Although CAMC achieves the highest average accuracy, its advantage over the best competing methods is limited in several cases. For example, at approximately 43% retention in Table 1, CAMC achieves 55.41% compared with 55.21% for VidCom2 on StreamingBench, and 53.20% compared with 52.80% on StreamRecall. On LVOmniBench, CAMC and OmniSIFT both achieve 37.91%.

These results show that CAMC is competitive, but it is difficult to determine whether the small differences are reliable without paired statistical analysis. Also, while the paper states that achieved retention is reported for each method, Table 1 mainly provides the target retention levels and marks some mismatched results without listing the exact achieved retention values. Since the main claim is about obtaining better accuracy under a limited memory budget, I think the comparison would be more convincing with uncertainty estimates and exact memory usage for each method.

2. The evidence validation process for StreamRecall does not fully establish that the annotated target event is sufficient to answer each question.

StreamRecall is an important contribution because its evidence timestamps are used to study how performance changes with historical distance. However, Appendix A.3 states that questions are generated using the target event together with five neighboring events before and after it. The automatic evidence verification also uses a video segment covering this wider context. Passing this verification does not necessarily mean that the annotated target event alone contains all the information needed to answer the question. This distinction matters because questions are placed after the target event, while some of the surrounding context used during generation and verification could occur after the query boundary.

The human validation procedure also raises a question. An item is removed only when both annotators mark it as invalid, and the reported agreement is 75.3%. The paper does not describe an additional review of cases where the annotators disagree. I am not suggesting that the benchmark necessarily contains incorrect questions or future information leakage. However, the current validation process leaves some uncertainty about the accuracy of its evidence labels. Since the evidence-age and gold-evidence experiments rely on these labels, I think this issue needs stronger support.

Questions for authors

Q1. Could the authors report the exact achieved retention for each method in Table 1 and provide paired confidence intervals for the closest accuracy comparisons? Resampling at the video level would be helpful because several questions may come from the same video. This would make it easier to judge whether CAMC's small improvements over the strongest baselines are reliable.

Q2. For StreamRecall, how did the authors verify that the annotated target event contains sufficient evidence to answer each question, rather than relying on the wider context used during generation and verification? It would also be useful to know how many questions were retained despite disagreement between the two annotators, and whether these cases received any further review.

Q3. Table 2 evaluates historical novelty, temporal change, and modality allocation, but does not separately examine the effects of adaptive chunk budgeting and token selection. Could the authors compare CAMC with a fixed per-chunk budget using the same token ranking, and with CAMC's adaptive budgets using random token selection within each modality? These comparisons would help clarify which part of the proposed design contributes most to the improvement.

1 comment

We thank the reviewer for a careful and constructive review. We address the two weaknesses and the three questions in turn.

W1 / Q1 — Reliability of margins and achieved retention.

Paired statistics. All comparisons are now assessed with paired, video-clustered percentile bootstrap intervals (10,000 resamples, resampling at the video level as the reviewer suggests) and a video-level sign-flip test, with Holm correction within each regime. Under tight retention, our method is significantly better than random retention and OmniZip on the delayed-recall benchmark, and significantly better than OmniSIFT at the main regime; no baseline is significantly better than our method in any cell. Aggregate evidence. Beyond per-cell numbers, the revised paper reports per-regime and overall averages over the six benchmark × regime cells and a robustness analysis of each method’s gap to the best method in each cell. Our method has the highest average in both regimes and overall and is within one point of the best method in every cell, whereas every baseline falls several points behind the best in at least one cell and the best-performing baseline changes from cell to cell. We consider this consistency, rather than any single cell, the central empirical result. Exact retention. Our method now uses a fixed per-chunk budget by default, so it retains exactly the target fraction at every point of the stream; baselines are run at matched retention, and the achieved retention of every method, benchmark and regime is tabulated in the supplementary material, including the cases where a baseline’s own allocation rule deviates from the target.

W2 / Q2 — Evidence sufficiency and annotator disagreement. We have strengthened the validation protocol. Items on which the two annotators disagreed were audited by two authors; wording was corrected where needed and items whose answer could not be grounded in the annotated target event alone were removed. All experiments were re-run on the resulting set, and the conclusions are unchanged. On the reviewer’s specific concern about the generation context: the query boundary is always placed after the target event and only content before it is available to any method at evaluation time, so no future content can be used. Moreover, the gold-evidence diagnostic restores only the compressed chunks overlapping the annotated target interval and raises accuracy to 80–83% for every method, including the uncompressed model — direct empirical evidence that the annotated event is sufficient for the large majority of questions. The revised benchmark section states the generation context and this evidence explicitly.

Q3 — Separating budget adaptivity from token selection. We ran the requested comparison. With the same token ranking, a fixed per-chunk budget matches the adaptive controller across the three benchmarks; since it additionally guarantees exact retention matching and removes a tuned component, the revised method adopts it as the default and reports the adaptive controller as an ablation. For the second comparison, under a fixed budget the random-retention baseline in Table 1 is precisely “our budget with random token selection within each modality”; the gap to it (about four points under tight retention on StreamingBench and 3–5 points on the delayed-recall benchmark) measures the contribution of the ranking, and the component ablations show that both scoring signals contribute where the budget is tight.

We thank the reviewer again; the paired analysis, the strengthened validation and the fixed-budget default all follow from this review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.