Understanding Streaming Memory for Progressive Context Expansion

Streaming video understanding requires models to reason over continuously growing history under strict online computation budgets. Existing memory systems rely on increasingly complex compression, organization, and retrieval mechanisms, yet their tightly coupled designs obscure the factors that truly govern performance and efficiency. We systematically revisit streaming memory, disentangle its key design choices, and derive principles for effective memory utilization. Based on these insights, we propose TRACE (Triggered Retrieval and Adaptive Context Expansion), a training-free progressive memory framework that optimizes Recent, Short, and Long memory levels and expands historical access according to predictive uncertainty. TRACE starts from low-cost recent context and introduces additional memory only when needed. On OVO-Bench, TRACE improves Qwen3.5-9B by 9.13 percentage points overall, with gains of 10.88 points on real-time perception and 6.81 points on backward tracing, while reducing per-query latency from 19.10 s to 1.86 s. Consistent gains on StreamingBench and across multiple VLM backbones demonstrate the effectiveness of selective and progressive memory access for balancing long-horizon reasoning and efficiency.