acceptodds
Under review as a conference paper at ICLR 2027

APERTURE: MERGE-CONSISTENT ROTARY STATES FOR COMPRESSED TOKENS

11 views

Abstract

Token compression combines content from several positions, yet rotary position embeddings usually assign the merged token one coordinate. We ask what positional information must survive later merges. Aperture stores Fourier moments of the token's weighted support at the model's rotary frequencies. We prove that these moments have minimal real dimension among continuous states sufficient for the selected expected rotary interactions. Represented mass makes updates additive; attention normalisation remains a separate readout choice. Uniform intervals give a centre rotation times a sinc gain. We characterise when centres determine interval widths and construct matched examples where they do not. Numerical checks verify the weighted-support implementation. In trained temporal readers, compression transfer varies with gain calibration and feature placement. In a prespecified native video question-answering comparison, stored support reaches accuracy versus for the deployed merging rule. These results separate exact positional preservation under compression from downstream benefit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Token Compression, Rotary Position Embeddings, Fourier Moments, Support-Aware Position Encoding, Merge Consistency, Multimodal Transformers, Positional State, Token Merging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3zNAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how positional information should be preserved when tokens are merged, pooled, retained, or evicted in multimodal/video models. The authors observe that RoPE-style positional encodings assume point tokens, whereas after compression a token represents a support measure (an interval or region). Using only the token centre can therefore break positional consistency across merges and make results depend on the merge tree.

Critical strengths

The paper addresses a timely and real problem: how positional information should remain consistent when tokens are merged, pooled, or compressed in multimodal models.

Critical weaknesses

Unclear contribution positioning. It is not clear whether the main contribution is a positional-encoding theory, a representational analysis, or a practical compression method. This ambiguity weakens the paper’s focus.

Limited theoretical novelty and strong assumptions. Using Fourier moments/kernel mean embeddings to represent supports is not entirely new. The analysis relies on strong assumptions such as known support measures, shared nonnegative merge weights, a fixed finite frequency bank, and expected rotary interactions rather than the actual softmax behavior.

Large gap between theory and practice. The algebraic guarantees cover idealized expected kernels, but actual attention is nonlinear, multi-layer, content-dependent, and often involves heterogeneous supports. The paper does not sufficiently bridge this gap.

Weak empirical evidence. Several key comparisons are not statistically significant or become insignificant after matched controls. Some native-interface results show no improvement, and one shows a significant degradation. The near-chance adaptation grid further leaves the practical ranking unresolved.

Concerns about baseline fairness and implementation. Comparisons with PPE, Sync-RoPE, and RoTE involve different selectors, adaptations, support constructions, and readouts. This makes it difficult to attribute differences to the proposed representation itself.

The identifiability results are relatively simple and of limited practical impact. The center-only ambiguity essentially reduces to an alternating nullspace, and the two-dimensional construction is mostly illustrative.

Writing and organization need improvement. The main claims and evidence are difficult to follow, and the heavy appendix material does not compensate for the weak empirical conclusions in the main text.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

What the minimal-state theorem adds to Fourier integration, how it applies inside trained attention, and what the matched baselines establish.

What is the contribution? We study the positional state a compressor must pass from one merge to the next. The grouping rule comes from the compressor. Fourier integration gives a code for a known support. Our question is what must survive once the members have been discarded. Theorem 1 characterises the smallest continuous state sufficient for the backbone's expected rotary interactions. We revised Sections 1–2 to state this claim up front and test its consequence in trained readers (Table 2).

For positions on an interval of nonzero length and MM distinct positive frequencies, a point probe can recover each moment's real and imaginary parts by choosing its query and key. Mixtures of 2M+12M+1 suitably chosen points span a simplex of dimension 2M2M in these coordinates. A continuous sufficient summary must distinguish all points in that simplex, so it needs at least 2M2M real coordinates. The MM complex moments attain the bound for arbitrary mixtures. On finite mixtures, an affine update and agreement with RoPE on points fix their values uniquely. A uniform interval takes just two parameters, but merging two separated intervals leaves that family.

What must the compressor know? At ingestion it needs the source positions and the nonnegative scalar weights used to pool content. Under mass-weighted pooling, subsequent merges need only the child summaries. We carry represented mass and the unnormalised phase sums. Two children with the same mean phase can represent one and ten patches, respectively. Their masses determine how much each contributes when the next patch arrives. The selector may depend on content, and different tokens may have entirely different supports. The guarantee compares merge orders that preserve the final members and effective weights.

What about softmax and later layers? For fixed pooled queries and keys, independent draws from the two supports give an expected rotary logit determined by their moments. For the actual network, compare two executions on the same compressed tokens. One recomputes moments from the original members and the other adds child states. They present the same multipliers to the first layer. Its logits, softmax weights and output therefore agree, and the argument repeats at every deterministic layer in exact arithmetic. Both paths run the same nonlinear attention on the pooled content. In five trained EPIC readers, three merge trees reproduced all 15,360 direct-encoding predictions. A separate similarity selector produced 1,717 disconnected supports. Fitting one interval across the outer endpoints gave a maximum moment error of 1.05. Adding the child states gave an error below 4.6×10−164.6\times10^{-16} and reproduced all 5,120 predictions.

Which empirical differences can we attribute? In Section 4.5 we gave Qwen2.5-VL-7B the same online selector, 1,024 slots, training data and five paired seeds across all three interfaces. We verified matching pooled content and centroids on every scored item. PPE can reselect from the original coordinates. Aperture can discard them. On MVBench, Aperture/PPE/centre-only scored 63.83/63.92/63.49%. On the 2,498 available StreamingBench visual questions they scored 71.39/71.84/71.56%. All four paired accuracy intervals include zero. The storage difference grows with the stream. At the longest prefix Aperture carries 1.01 MiB of mass and moments, while this PPE implementation retains 17.91 MiB of coordinates for reselection (Figure 7).

On the in-distribution EPIC replay, Aperture scores 49.22% against 43.01% for centre-only. Under the offline video-QA selector, PPE leads stored support on MVBench, 67.12% to 65.63%. The temporal controls answer a different question. Matching gain RMS reduces the Aperture advantage over MN-sinc from 5.97 to 2.14 points at 3B and from 5.49 to 0.73 at 7B. Both matched intervals include zero (Table 3). A reader can therefore calibrate its gains at readout while preserving the raw state for the next merge. Appendix F distinguishes our MN-sinc and width-capped controls from published RoTE and Sync-RoPE and checks both published operators against reference implementations. The near-chance onset grid supplies no reliable ranking.

When does identifiability help? It gives a simple rule for deciding when to store width. With contiguous intervals and a known left boundary, the centres recover every width recursively. With centres alone, alternating width changes preserve every observed centre while changing the answer to a support query (Figure 4). An observed total span resolves the ambiguity for odd chain lengths. For even lengths the alternating changes sum to zero. The planar example shows why a known outer boundary alone need not identify a rectangular tiling. For a reader that only needs an approximation, Section 2 also bounds the absolute error in the centre's complex multiplier at frequency θ\theta by θ2σ2/2\theta^2\sigma^2/2 for positional variance σ2\sigma^2.

I thank the authors for the detailed clarifications and added controls. The contribution is clearer and the empirical comparison fairer. I am updating my rating to weak acceptance.

Reviewer_WWajOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates an important question in multimodal token merging for VLMs and omnimodal models on how to assign correct RoPE embeddings to these tokens after merging. Classical token merging methods usually give the merged token only one position (for example, the centre of the merged cluster). This keeps the centre but loses information about how the merged content is spread over the original positions. To address this, the authors propose Aperture. It describes each merged token by its weighted support (the original positions and their merging weights) and stores the Fourier moments of this support at the model's rotary frequencies. The moments are merged with the same weights as the token content, so the result does not depend on the order of merges. For a uniform interval, this code is RoPE at the interval centre multiplied by a sinc gain. The authors prove that this state has the minimal dimension among continuous states that preserve the expected rotary interactions, and they analyse when token centres are enough to recover token widths. Numerical checks confirm that different merge orders give the same state. The experiments include synthetic tasks, a custom task on EPIC-Kitchens features (with a small reader and with Qwen2.5-Omni 3B and 7B), and Qwen2.5-VL-7B on the commonly used benchmarks MVBench and Video-MME. The results are mixed. Width information helps the small EPIC-Kitchens model when the test conditions match training. However, under unseen compression policies and on video question answering, the method does not improve over the simple centre-only baseline. On MVBench and Video-MME, it is also worse than the existing PPE method.

Critical strengths
  1. The paper studies an important problem and proposes a simple method. A merged token usually gets only one position. The authors describe its position as a weighted support and store the Fourier moments of this support at the model's rotary frequencies. A merge is just a sum with no learned parameters.
  2. The theory is careful. The authors show that this state has the minimal dimension and that sinc is the only interval code consistent with repeated merging of equal halves. They also analyse when token centres are enough to recover token widths.
  3. The evaluation is transparent, even though the results are mixed: the main comparisons follow a protocol fixed in advance, and negative results are reported openly. The authors also build a synthetic task where only the sampling rate changes, which allows a controlled test of transfer.
Critical weaknesses
  1. The method gives no clear practical gain. Under unseen compression policies on EPIC-Kitchens, the method is not better than the centre-only baseline, and in the Qwen2.5-Omni study the differences from centre-only are not significant at the main test budget. A significant gain over centre-only appears only for the 3B model at an additional, stronger compression level. On MVBench, the method is almost equal to centre-only and worse than PPE in all five seeds.
  2. The main advantage is not tested in trained models. The method is designed to update positions after each merge without keeping the original positions. However, in the trained experiments, the code is computed from known intervals or from the stored membership of each token.
  3. The evaluation is limited. The main task is custom, it is built on EPIC-Kitchens features with automatic labels, and the test set has only 3 participants and 16 videos. There are only two standard benchmarks (MVBench and Video-MME), with one model and one retention level.
Questions for authors
  1. Can you show an experiment with repeated or streaming merging where the original token membership is not stored? How does Aperture compare with PPE and centre-only in this setting?
  2. Stored support and PPE use the same membership information, but PPE is better on MVBench and Video-MME. What explains this difference?
  3. Have you tested the method on high-resolution images with similarity-based merging (e.g., DocVQA or ChartQA) or on streaming benchmarks (e.g., OVO-Bench or StreamingBench), where tokens with different widths are more common?
2 comments
AuthorsOpenReview ↗

A trained streaming test without member history, a comparison with PPE and centre-only, and the reason identical members yield different rotary readouts.

1. Repeated merging without the original members. Yes. We trained Qwen2.5-VL-7B to answer video questions while repeatedly merging visual tokens into 1,024 live slots. We have added the update and the trained comparison to Section 4.5. When a patch arrives, Aperture accumulates its mass and its phase at each rotary frequency. Its positional state for the next merge is the mass and raw phase sums. The selector uses current pooled content, and attention reads the sums divided by mass. The next merge never accesses an original-member list.

We fitted Aperture, PPE and centre-only separately on the same 4,096 examples with five paired seeds. All three use the same content-based DPC-KNN selector, centroid order, source-time clock and frozen visual tower and merger. Each arm selects its checkpoint on the same video-disjoint validation questions. We checked the pooled content and centroids on every scored item. PPE receives each cluster's original coordinates and can reselect its K=8K=8 positions after every merge.

Mean accuracy over five seedsAperturePPECentre-only
MVBench, 4,000 questions63.83%63.92%63.49%
StreamingBench visual, 2,498 questions71.39%71.84%71.56%

On MVBench, Aperture minus PPE is −0.09 percentage points (95% paired-seed interval [−0.62, +0.44]). On StreamingBench it is −0.45 [−1.20, +0.30]. The corresponding differences from centre-only are +0.34 and −0.17 points. All four intervals include zero. The readouts retain different information at these scores. Centre-only recovers the centroid rotation at the next question. Aperture can recover the mean rotation of all contributing patches at each frequency. PPE can still choose among individual source positions because this implementation keeps their coordinates.

For SS live slots and NN patches seen, Aperture uses 1032S1032S bytes for mass and moments. PPE uses 8S+24N8S+24N bytes for mass and three coordinates per patch. At capacity the crossover is 43,691 patches. Aperture used less positional storage on 1,953 of the 2,498 scored prefixes in Figure 7. At the longest prefix it carried 1.01 MiB, while PPE retained 17.91 MiB of member coordinates alone. All arms also carry 28 MiB of pooled content at capacity. The byte comparison covers positional metadata.

Table 2 also tests the hidden states directly in trained EPIC readers. A similarity selector produced 1,717 disconnected supports among 16,384 final tokens. Adding child moments agreed with a member-list reference within 4.6×10−164.6\times10^{-16}, and all five readers agreed at every checked block output. Fitting one interval between a cluster's outer endpoints gave a maximum moment error of 1.05. This checks the positional claim even in clusters with gaps, on which a single width would assign weight to patches that never entered the cluster.

2. Why does PPE score higher offline with the same members? Consider two equally weighted patches with opposite phases at one frequency. Aperture averages the phases within that channel, so the multiplier is zero. PPE assigns a selected member position to a group of channels. Whichever patch is selected, that channel still rotates at unit magnitude. Thus supplying the very same members to both operators can suppress a query–key contribution in one model and retain it in the other. PPE preserves the unit-length rotations the pretrained backbone saw. That is a plausible explanation for its offline advantage. The offline comparison changes both phase assignment and magnitude, so their individual contributions to the score gap remain open.

With the deployed offline selector at one-quarter retention, PPE beat stored support on all five MVBench seeds (67.12% versus 65.63%, Holm-adjusted p=0.00047p=0.00047), and scored 62.24% versus 61.46% on Video-MME. Figure 6 also shows the offline comparison with a fixed 2×22\times2 grid selector. PPE also leads with membership fixed by that grid.

3. Which of the proposed benchmarks did we test? We chose StreamingBench because later questions on the same video force the model to reuse a summary after more patches have arrived. This extends the custom EPIC test (three participants and 16 videos) to visual questions from 499 source-video identities. Patches enter chronologically at 2 fps. Each question sees the available prefix back to the start of the video. The visual state persists between questions and the language KV cache resets. Two damaged prefixes leave 2,498 of 2,500 released questions scorable for every arm. Three timestamps extend beyond the source duration. Excluding those three changes each five-seed mean by less than 0.02 percentage points (Appendix G). We tested StreamingBench among the four benchmarks you named. DocVQA, ChartQA and OVO-Bench remain untested. The supplement includes per-seed correct counts and storage at each scored question.

Thank you for adding the StreamingBench experiment. It directly addresses my main concern about whether repeated merging works in a trained model without retaining the original patch coordinates. The reduction in positional memory on long video prefixes is a concrete benefit, even though I don't see a clear gain in QA accuracy. I'm raising my rating to weak accept.

Reviewer_7LjKOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Aperture, which lets merged tokens retain information about the distribution of the positions they cover, rather than recording only a single centre point. It encodes this positional information as a set of averaged Rotary Position Embeddings (RoPE), which can be updated additively when tokens are merged. Experiments show that Aperture preserves positional information more completely.

Critical strengths
  1. The paper has a clear structure, moving from the proposed representation to theoretical analysis.
  2. It distinguishes between preserving positional information and improving model performance.
Critical weaknesses
  1. The paper is overly elaborate, and its main experimental findings are not sufficiently clear.
  2. The experimental settings are inconsistent and some results are near chance, making it difficult to determine whether Aperture is truly effective.
Questions for authors
  1. Could the authors explain why the extensive appendix is not accompanied by clearer arguments in the main paper?
3 comments
AuthorsOpenReview ↗

The revised main text shows when exact merging preserves a trained reader's answers, when position helps prediction, and what it costs in streaming QA.

Your distinction between preserving position and improving prediction is the one we used to reorganise the paper. We brought the trained-reader replay and the streaming measurements into the ten-page main text so a reader can follow the argument from Figure 1 to Table 2, then judge predictive performance in Figure 5 and the cost of storing positions in Figure 7. The proofs, training details and secondary comparisons are in the appendix.

What does a trained reader actually preserve? Figure 1 starts with two equal patches at −a-a and aa. Their centre is zero, just like a single patch at the origin. At rotary frequency θ\theta, however, the two mean phases are cos⁡(θa)\cos(\theta a) and 11. Aperture stores the mass and weighted phase sums. A later merge adds these quantities, including when the patches are separated by a gap. Theorem 1 explains the size of the phase vector. For arbitrary mixtures, its normalised moments attain the minimum dimension for a continuous state that recovers the chosen rotary interactions. Mass supplies the relative weights for the next merge. Table 2 tests the consequence directly. We fixed five trained EPIC readers, the pooled content and the final tokens on 1,024 windows. We changed only how the final rotary multipliers were obtained. Across three different merge trees, all 15,360 predictions matched direct encoding. The largest absolute change in a final logit was 4.68×10−54.68\times10^{-5} in float32. A second replay used a content-based selector that produced 1,717 disconnected supports. Incremental and direct computation agreed on all 5,120 answers. Checking each window matters here because a change in which questions a reader answers correctly can leave aggregate accuracy untouched.

When does that information improve prediction? To measure predictive benefit, we trained separate readers with different positional inputs. On the same in-distribution EPIC replay, Aperture located the onset correctly on 49.22% of windows versus 43.01% for centre-only. The paired gain was 6.21 percentage points (five-seed 95% interval [2.50, 9.92], exact sign-flip p=0.0625p=0.0625). A scalar width feature also helps when training and testing both use coarsening. Figure 5 tests whether that gain survives a change in compression policy, and whether it matters where the model receives width. Under held-out masking and eviction, the scalar feature averaged 3.44 points below centre-only, whereas exact Aperture averaged 0.20 points below it. The within-seed comparison favoured rotary placement by 3.25 points (95% interval [1.55, 4.94], exact p=0.0625p=0.0625). These are descriptive comparisons over five training seeds on a fixed test set.

How should the near-chance results be read? The two large-model onset studies use different readouts. In Section 4.4 a shared content-similarity score helps each model find the event. At the primary held-out budget (b=4b=4), the bin-accuracy differences from centre-only are +2.59 points at 3B and +0.18 at 7B, with both intervals spanning zero (Table 3). At the exploratory wider budget (b=8b=8), the 3B model selects the onset-bearing token more often on 20 of 24 paired seeds, gaining 3.76 points on average (Holm-adjusted p=0.012p=0.012). Section 4.5 removes the content-similarity aid and trains a readout to predict one of twelve time bins. The methods cluster near the 1/12 chance level. Table 4 gives the prespecified contrasts and their uncertainty. The near-chance scores offer little leverage for choosing a positional interface. The conclusion about merge fidelity rests on the same-model replay, where each prediction has a direct reference.

What does this buy in video QA? We adapted Qwen2.5-VL-7B with three readouts under one online content selector. All three get the same pooled patches and 1,024 live slots. On 4,000 MVBench questions, Aperture, PPE and centre-only average 63.83%, 63.92% and 63.49% over five seeds (Table 5). In the same order they score 71.39%, 71.84% and 71.56% on 2,498 available StreamingBench visual questions. The four paired accuracy intervals span zero. In the separate offline MVBench comparison, PPE leads stored support 67.12% to 65.63%. Figure 6 compares readouts within each offline selector. Figure 7 compares them after adaptation to repeated online merging.

As an online cluster grows, Aperture can still supply its exact mean phases from a fixed set of sums. The PPE baseline keeps the original coordinates to choose positions again at the next merge. At the longest StreamingBench prefix, 782,640 source patches have been pooled into 1,024 slots. Aperture's mass and moments occupy 1.01 MiB. PPE's retained coordinates occupy 17.91 MiB. The crossover occurs at 43,691 patches at full capacity. On 1,953 of the 2,498 prefixes in Figure 7, preserving the exact mean phases used less positional memory than retaining the coordinates for repeated PPE selection.

Dear Reviewer 7LjK,

As the discussion deadline approaches, we would be very grateful for any further thoughts or feedback on our rebuttal and additional experiments. We sincerely appreciate the time and effort you have devoted to reviewing our work and would be happy to address any remaining concerns or questions.

Best regards,

Authors

I thank the authors for their response, which has resolved some of my questions. I am willing to increase my score, but I feel the abstract and introduction sections remain too short.

Comments on the paper

1 comment
Area_Chair_xdwUOpenReview ↗

Discussion Phase

As the discussion period is rapidly coming to a close (Nov 18 AoE), I would like to remind the reviewers to please participate in discussion and acknowledge any comments posted by the authors. A secondary, important reminder: full disclosure of any AI usage is required and the reviewer must include their own human-written assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.