- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BGStream, a training-free token compression method for streaming video MLLMs. Visual features are randomly projected to 4-D keys, and tokens in the same cell are averaged. Each grid covers 32 frames. A FIFO buffer keeps salient tokens from the last 5 frames, and a Johnson–Lindenstrauss analysis is given for the projection. At about 5% retention, BGStream outperforms seven compression baselines on StreamingBench and OVO-Bench. The authors claim that it preserves the structure under aggressive compression and is complementary to KV-cache methods.
Critical strengths
- The splatting is additive and does not need the full video, so it fits streaming. The compression step itself is also cheap, which makes the method practical.
- At about 5% retention, BGStream outperforms seven compression baselines on three backbones, and it still leads when the token count is matched for each query. This strong result is the main reason for my positive rating.
Critical weaknesses
- In Table 15, FIFO-only already outperforms every compression baseline in Tables 1 and 2. It gets 67.96 against 65.84 on StreamingBench and 53.76 against 48.84 on OVO-Bench. The grid only adds 0.64 and 0.82 points, so its own gain is small. Since the grid is the main contribution, this is an important concern.
- Theorem B.6 needs Δ>τ_grid+2R. Since Δ≤D+2R always holds, τ_grid must be below D. However, the only τ_grid construction in Appendix B.2 gives more than 2D at N=6272, d_s=4 and s_r=1/3. The purity guarantee is therefore not shown to hold in the actual setting. This weakens the theoretical contribution of the paper.
Questions for authors
- How were the baselines changed from their default configurations in Appendix D.3 to reach 5% retention, and can they be given the same FIFO buffer?
- Since Appendix D.3 removes spatial partitioning, the grid becomes a 4-D random projection with 3 bins per axis plus averaging. This is closer to random-projection hashing than to a bilateral grid. Can it be compared with k-means at the same cell count, and how are position ids assigned to grid tokens?
- In Tables 1, 2 and 14, the KV-cache combinations are slower and less accurate than BGStream alone. What is the KV memory of BGStream alone?
- Is the 524 ms encoder cost in Table 18 paid at query time, and what does ReKV's 13 s latency include?
4 comments
Response to Reviewer d9XX (N/N)
Q2. Relation to hashing, k-means, position ids
We agree that, with spatial partitioning removed, our routing step, a Gaussian projection followed by per-axis quantization, is closely related to random-projection hashing [1]. At the same time, we view this routing step as one component of a broader framework: the bilateral grid assigns each token to a cell, accumulates it into a homogeneous (sum, count) pair, and summarizes each cell into one representative token, so that frames are incorporated as they arrive and grids can be merged under a token budget (Appendix C.2). Within this framework, the routing step can be instantiated in several ways, such as PCA that gives comparable accuracy at about 70× the projection time (Table 18 in the revised manuscript). The k-means comparison suggested by the reviewer fits naturally into this view as another routing: at the same cell count, it is comparable on StreamingBench and 1.7 points lower on OVO-Bench, and its compression takes 2.87 s per query versus 95.4 ms for the grid, since it fits clusters iteratively over all features of a group. The framework thus accommodates different routings, and we use the random projection because it is the simplest one that keeps the framework single-pass and additive in a stream. Regarding position ids, grid tokens follow the flattened cell-index order within chronologically ordered groups, followed by the FIFO tokens, and the LLM assigns consecutive positions to this sequence; original patch indices and timestamps are not used as position ids.
[1] Locality-sensitive hashing scheme based on p-stable distributions, Datar et al., ACM’24
Q3. KV memory
BGStream alone requires 0.03 GiB of KV cache on StreamingBench and OVO-Bench, essentially the same as the combination in Table 14 (in the revised manuscript, Table 15). We would like to explain why we nevertheless report the combination. BGStream and KV-cache methods bound different things: BGStream reduces the tokens that enter the LLM, and its token budget caps the context, which at = 18,000 corresponds to about 1 GiB of KV cache. On the other hand, a KV-cache method caps memory at a chosen budget independent of , by offloading or evicting entries rather than merging them. The combination is intended for long streams under such a memory cap, where it keeps the KV-cache method's bound while being much faster than that method alone (Tables 1, 2 and 15). However, the benchmark videos are far shorter than the point at which either bound binds, so at these lengths BGStream alone and the combination use the same memory and the combination only pays the overhead of the KV-cache method, as the reviewer observed.
We have revised the paper accordingly: Sections 2 and 5.2 and Appendix A.1 now describe KV-cache optimization as compatible with BGStream rather than as complementary in general, and Appendix D.11 reports the KV memory of BGStream alone in Table 15 together with a short explanation of why it matches the combination at these stream lengths.
Q4. Latency
Yes. In our measurement protocol, all frames up to the query are encoded when the query is issued, so the 524 ms in Table 18 (in the revised manuscript, Table 21) is the visual encoder and adaptor cost over the full prefix, and it is included in the per-query latency of every method (in deployment, this cost is paid as frames arrive). Similarly, ReKV's 13 s includes the same encoder and adaptor cost. The remaining time comes from two sources. First, all visual tokens pass through the LLM to build the KV cache, so the attention cost grows quadratically with the number of visual tokens. Second, the past KV entries are offloaded, and the query-relevant ones are loaded back to GPU memory at query time. BGStream avoids both by reducing the visual tokens before they enter the LLM, which is where its latency advantage comes from.
We have clarified the measurement protocol and this breakdown in Appendix D.14.
Thank you for the response and new experiments. Although the margin over baselines with the same FIFO buffer is modest and Theorem B.6 does not certify the default setting, BGStream remains fast and still leads in the controlled comparisons, so I will keep my score of 3.
We sincerely appreciate your valuable feedback and your engagement in the discussion, both of which have helped us further improve our paper.
Response to Reviewer d9XX (1/N)
W1/Q1. The gain over FIFO-only and the baseline setup
We agree that the gain over FIFO-only in Table 15 (in the revised manuscript, Table 16) is modest, and we address this in two steps: first the baseline configuration behind Tables 1 and 2, then where the benefit of the grid shows. Regarding the configuration in Appendix D.3, the baselines follow the hyperparameters of their original papers, with only the retention ratio set to 5% to match BGStream, and they do not use a FIFO buffer.
As for whether the baselines can be given the same FIFO buffer, yes: we gave each baseline the same FIFO buffer and the same retention ratio as BGStream, so that only the compression of past frames differs. For this comparison, we use the baselines that, like BGStream, require no temporal boundary prior and are thus applicable in the streaming setting (VisionZip, ToMe, and StreamingTOM). As shown in the table below, with the FIFO buffer the baselines reach the level of FIFO-only: VisionZip and ToMe do not improve over it on OVO-Bench, StreamingTOM improves slightly, and BGStream gives the largest gain on both benchmarks. We have clarified the baseline configuration in Appendix D.3, added this comparison to Appendix D.12, and referred to it in Section 5.2.
We would also like to reiterate our main contribution: a grid that operates without a temporal prior, remains effective under aggressive retention ratios, and removes temporal redundancy as a by-product of splatting (Section 4.2). Since most questions in these benchmarks concern the recent context, the grid's gain is small in overall accuracy; it is consistent, however, in the comparison above, and it is on the tasks that require past frames (Backward Tracing in Table 2, offline benchmarks in Appendix D.8) that the role of the grid becomes visible.
W2. Theorem B.6
We agree that the explicit construction at the bottom of page 26 in Appendix B.2, τgrid=srD2log(δ2N(N−1)ds), satisfies the grid-width probability constraint in Eq. (23) but is too conservative to simultaneously satisfy the geometric separation condition of Theorem B.6 under our default configuration. This construction was intended to establish the existence of a deterministic bound satisfying Eq. (23), rather than the joint feasibility of all the theorem's conditions, and the proof of Theorem B.6 does not depend on it.
A sharper, realization-wise criterion follows from Lemma B.5 and Eq. (26): for a fixed projection P, every non-empty grid cell is pure whenever ΔP>2RP+sr∥W(P)∥2, where ΔP and RP denote the minimum inter-cluster center distance and the maximum within-cluster radius after projection, respectively. Cross-cluster points are separated by at least ΔP−2RP, whereas points sharing a cell are at most sr∥W(P)∥2 apart by Lemma B.5, so this condition rules out cross-cluster collisions. However, even this criterion does not hold on our features [ΔP−2RP=X vs. sr∥W(P)∥2=Y], which is consistent with the purity below 1 in Table 5. The difficulty is therefore not specific to the construction: Theorem B.6 does not certify our default configuration, and neither does Theorem B.8, which relies on the same separation condition (Eq. 28).
We nevertheless believe the conditional results retain their theoretical value. Theorem B.6 identifies sufficient conditions linking grid width, cluster separation, and within-cluster drift to the absence of cross-cluster collisions, and Appendix B.3 characterizes how such contamination affects the aggregate SNR and representation error under its stated assumptions.