- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Causal Adaptive Memory Compression (CAMC), a training-free method for streaming audio-video understanding. It uses historical novelty and temporal change to adaptively allocate memory budgets across video segments, modalities, and individual tokens. The authors also introduce StreamRecall, a benchmark for evaluating delayed access to historical audio-video evidence. Experiments on three benchmarks show competitive performance under different token retention budgets.
Critical strengths
-
The paper addresses a practical problem in streaming audio-video understanding. The proposed compression strategy is simple, training-free, and explicitly considers both historical novelty and temporal changes.
-
The experiments cover three benchmarks and several recent compression methods under different retention budgets. The ablation studies support the usefulness of the proposed signals. The gold-evidence analysis is also interesting, revealing that historical retrieval remains a major bottleneck even without compression.
Critical weaknesses
-
Limited novelty. The method mainly combines existing techniques, including cosine-similarity-based novelty estimation, temporal difference scoring, adaptive budget allocation, and Top-K token selection. While the overall design is reasonable, the technical contribution over existing compression methods is relatively incremental.
-
Modest improvements and unclear computational overhead. Although CAMC achieves the highest average accuracy, the gains over the strongest baselines are relatively small (0.81 and 0.60 points at the two retention levels). The benefit of adaptive modality allocation is also inconsistent across benchmarks. Moreover, the paper lacks a detailed analysis of compression overhead and how the historical similarity computation scales with stream duration.
Questions for authors
-
Can the authors report the computational overhead of CAMC, including compression latency and memory usage, especially as the historical memory grows?
-
Given the relatively small gains, how much does the three-level adaptive allocation improve over simpler compression strategies using only historical novelty or temporal change?
-
Can the authors evaluate CAMC on another Omni-LLM backbone to verify whether the improvements generalize beyond Qwen2.5-Omni-7B?
We thank the reviewer for a thorough review and for recognizing the gold-evidence analysis. We respond to the weaknesses and questions below.
W1 — Novelty. We agree that each primitive (cosine novelty, temporal difference, Top-K selection) is simple; the contribution lies in the problem formulation and in how the primitives are composed under its constraints. We formalize streaming memory formation as causal, query-blind and irreversible allocation at three levels (chunk, modality, token), a setting that most existing compressors violate by seeing the whole clip, conditioning on the question, or keeping a recoverable pool. Within it, historical novelty is measured against everything already retained, which detects recurrences that temporal signals cannot, and the same two signals drive both the audio/video split and the token ranking. The revised version simplifies the design further — a fixed per-chunk budget replaces the adaptive controller, making retention exactly comparable across methods — and we regard the fact that a method this simple is the most consistent performer across six settings as the point, not a weakness.
W2 — Size of the gains, modality allocation, overhead.
Gains. We now report paired, video-clustered bootstrap intervals with Holm correction for every comparison, and we frame the claim as an aggregate one. Over the six benchmark × regime cells our method has the highest average in both regimes and overall and is within one point of the best method in every cell, whereas every baseline falls several points behind the best in at least one cell, and the best baseline changes from cell to cell. Under tight retention our method is significantly better than random retention and OmniZip on the delayed-recall benchmark; no baseline is significantly better than our method anywhere. For a compressor that must be chosen before the workload is known, this robustness is the relevant property, and no baseline has it. Modality allocation. The revised ablations are run under the fixed budget so that all variants retain exactly the same number of tokens. The content-dependent split contributes most where the budget is tight; at generous budgets the allocation matters less, and we state this explicitly rather than claiming a uniform benefit. Overhead. See Q1.
Q1 — Computational overhead and scaling. The history is summarized by one normalized key vector per modality per chunk, so the novelty computation is a single matrix–vector product per chunk (one maximum cosine similarity over the stored keys); its cost grows linearly with the number of chunks but is negligible relative to encoding and generation — a 34-minute stream yields ~140 keys. Memory for keys is likewise a few hundred kilobytes per hour of stream. The supplementary material reports measured query latency and memory per method against accuracy: our method sits within the latency range of the compared compressors, well below the uncompressed model, while achieving the best mean accuracy rank.
Q2 — Three-level allocation vs. single signals. The component ablations answer this directly. Under a fixed budget, removing temporal change costs 1.4 points and removing historical novelty 0.8 points on StreamingBench at tight retention, and the uniform modality split costs a further 0.8; together the components account for most of the ~4-point margin over random retention there. Either signal alone recovers much of the gain at generous budgets, which is expected for two correlated cues, but both are needed where the budget binds. Moreover, since the budget is now fixed, the random-retention baseline in Table 1 is exactly “our budget with random token selection”; the gap to it (3–5 points under tight retention on the delayed-recall benchmark) is the contribution of the ranking.
Q3 — Other backbones. The revised supplementary material adds results on two further omni-modal backbones with different audio and vision front-ends, together with an analysis of baseline portability: embedding-only methods (ours, random retention, VidCom2) transfer unchanged, whereas methods that depend on backbone-specific attention maps, sidecars or a trained selector cannot be faithfully ported without re-deriving them per backbone. The method’s behaviour on the additional backbones is consistent with the main results, and we consider this portability a practical advantage of a training-free, embedding-based design.
We thank the reviewer again; the paired statistics, the fixed-budget ablations, the overhead analysis and the additional backbones all follow from this review.