- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how positional information should be preserved when tokens are merged, pooled, retained, or evicted in multimodal/video models. The authors observe that RoPE-style positional encodings assume point tokens, whereas after compression a token represents a support measure (an interval or region). Using only the token centre can therefore break positional consistency across merges and make results depend on the merge tree.
Critical strengths
The paper addresses a timely and real problem: how positional information should remain consistent when tokens are merged, pooled, or compressed in multimodal models.
Critical weaknesses
Unclear contribution positioning. It is not clear whether the main contribution is a positional-encoding theory, a representational analysis, or a practical compression method. This ambiguity weakens the paper’s focus.
Limited theoretical novelty and strong assumptions. Using Fourier moments/kernel mean embeddings to represent supports is not entirely new. The analysis relies on strong assumptions such as known support measures, shared nonnegative merge weights, a fixed finite frequency bank, and expected rotary interactions rather than the actual softmax behavior.
Large gap between theory and practice. The algebraic guarantees cover idealized expected kernels, but actual attention is nonlinear, multi-layer, content-dependent, and often involves heterogeneous supports. The paper does not sufficiently bridge this gap.
Weak empirical evidence. Several key comparisons are not statistically significant or become insignificant after matched controls. Some native-interface results show no improvement, and one shows a significant degradation. The near-chance adaptation grid further leaves the practical ranking unresolved.
Concerns about baseline fairness and implementation. Comparisons with PPE, Sync-RoPE, and RoTE involve different selectors, adaptations, support constructions, and readouts. This makes it difficult to attribute differences to the proposed representation itself.
The identifiability results are relatively simple and of limited practical impact. The center-only ambiguity essentially reduces to an alternating nullspace, and the two-dimensional construction is mostly illustrative.
Writing and organization need improvement. The main claims and evidence are difficult to follow, and the heavy appendix material does not compensate for the weak empirical conclusions in the main text.
Questions for authors
See weakness.
2 comments
I thank the authors for the detailed clarifications and added controls. The contribution is clearer and the empirical comparison fairer. I am updating my rating to weak acceptance.
What the minimal-state theorem adds to Fourier integration, how it applies inside trained attention, and what the matched baselines establish.
What is the contribution? We study the positional state a compressor must pass from one merge to the next. The grouping rule comes from the compressor. Fourier integration gives a code for a known support. Our question is what must survive once the members have been discarded. Theorem 1 characterises the smallest continuous state sufficient for the backbone's expected rotary interactions. We revised Sections 1–2 to state this claim up front and test its consequence in trained readers (Table 2).
For positions on an interval of nonzero length and M distinct positive frequencies, a point probe can recover each moment's real and imaginary parts by choosing its query and key. Mixtures of 2M+1 suitably chosen points span a simplex of dimension 2M in these coordinates. A continuous sufficient summary must distinguish all points in that simplex, so it needs at least 2M real coordinates. The M complex moments attain the bound for arbitrary mixtures. On finite mixtures, an affine update and agreement with RoPE on points fix their values uniquely. A uniform interval takes just two parameters, but merging two separated intervals leaves that family.
What must the compressor know? At ingestion it needs the source positions and the nonnegative scalar weights used to pool content. Under mass-weighted pooling, subsequent merges need only the child summaries. We carry represented mass and the unnormalised phase sums. Two children with the same mean phase can represent one and ten patches, respectively. Their masses determine how much each contributes when the next patch arrives. The selector may depend on content, and different tokens may have entirely different supports. The guarantee compares merge orders that preserve the final members and effective weights.
What about softmax and later layers? For fixed pooled queries and keys, independent draws from the two supports give an expected rotary logit determined by their moments. For the actual network, compare two executions on the same compressed tokens. One recomputes moments from the original members and the other adds child states. They present the same multipliers to the first layer. Its logits, softmax weights and output therefore agree, and the argument repeats at every deterministic layer in exact arithmetic. Both paths run the same nonlinear attention on the pooled content. In five trained EPIC readers, three merge trees reproduced all 15,360 direct-encoding predictions. A separate similarity selector produced 1,717 disconnected supports. Fitting one interval across the outer endpoints gave a maximum moment error of 1.05. Adding the child states gave an error below 4.6×10−16 and reproduced all 5,120 predictions.
Which empirical differences can we attribute? In Section 4.5 we gave Qwen2.5-VL-7B the same online selector, 1,024 slots, training data and five paired seeds across all three interfaces. We verified matching pooled content and centroids on every scored item. PPE can reselect from the original coordinates. Aperture can discard them. On MVBench, Aperture/PPE/centre-only scored 63.83/63.92/63.49%. On the 2,498 available StreamingBench visual questions they scored 71.39/71.84/71.56%. All four paired accuracy intervals include zero. The storage difference grows with the stream. At the longest prefix Aperture carries 1.01 MiB of mass and moments, while this PPE implementation retains 17.91 MiB of coordinates for reselection (Figure 7).
On the in-distribution EPIC replay, Aperture scores 49.22% against 43.01% for centre-only. Under the offline video-QA selector, PPE leads stored support on MVBench, 67.12% to 65.63%. The temporal controls answer a different question. Matching gain RMS reduces the Aperture advantage over MN-sinc from 5.97 to 2.14 points at 3B and from 5.49 to 0.73 at 7B. Both matched intervals include zero (Table 3). A reader can therefore calibrate its gains at readout while preserving the raw state for the next merge. Appendix F distinguishes our MN-sinc and width-capped controls from published RoTE and Sync-RoPE and checks both published operators against reference implementations. The near-chance onset grid supplies no reliable ranking.
When does identifiability help? It gives a simple rule for deciding when to store width. With contiguous intervals and a known left boundary, the centres recover every width recursively. With centres alone, alternating width changes preserve every observed centre while changing the answer to a support query (Figure 4). An observed total span resolves the ambiguity for odd chain lengths. For even lengths the alternating changes sum to zero. The planar example shows why a known outer boundary alone need not identify a rectangular tiling. For a reader that only needs an approximation, Section 2 also bounds the absolute error in the centre's complex multiplier at frequency θ by θ2σ2/2 for positional variance σ2.