Learning What to Keep: Causal Memory Compression for Streaming Audio-Video Understanding

Streaming omnimodal systems often need to answer questions about earlier events displayed. Retaining the full history becomes increasingly expensive as the stream grows, forcing the system to decide what information to keep or delete. We in- troduce Causal Adaptive Memory Compression (CAMC), a method for manag- ing past-information retention. CAMC allocates a limited memory budget across stream segments, audio and video, and individual tokens. It uses two signals: historical novelty, which measures how different new content is from stored infor- mation, and temporal change, which measures how different it is from prior ob- servations. A dynamic budget enables CAMC to retain useful information while staying close to an overall retention target. Additionally, we present StreamRe- call, a streaming audio-video multiple-choice benchmark that evaluates whether information from earlier events remains accessible when queried later. CAMC achieves the highest average answer accuracy evaluated while retaining a fraction of the original audio-video history.