Structure-Preserving Visual Token Compression for Streaming Video Understanding via Bilateral Grid Splatting

Streaming video understanding requires Video MLLMs to process incoming frames in real time, but rapidly accumulating visual tokens incur a severe computational bottleneck. Visual token compression offers a remedy, yet existing methods fall short in the streaming on two fronts: the unbounded frame stream forces aggressive compression under which their accuracy collapses, and they require a temporal boundary prior unavailable in the streaming setting. To bridge this gap, we propose BGStream, a training-free compression method based on the bilateral grid that 1) preserves the semantic structure of visual features even under aggressive retention ratios, and 2) splats each incoming frame's tokens into the grid on the fly, without accessing the full video. However, since generalizing the bilateral grid to visual tokens is non-trivial, we ground this design in a theoretical analysis based on the Johnson–Lindenstrauss lemma. On StreamingBench and OVO-Bench, BGStream outperforms prior compression methods while remaining compatible with KV cache optimization.