acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Sequence Compression with Test-time Regression

Abstract

Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sequence Compression, Multimodal Token Compression, Large Language Model, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8szFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates sequence compression as an input-specific ridge regression problem. It uses the fitted coefficients as a compressed representation and evaluates this operator in block-wise sparse attention and visual token merging. The results show clear gains in long-context retrieval and stronger visual compression.

Critical strengths

(1) The method has a clear mechanism: the regression solution accounts for correlations among tokens, while the dual-form solve is practical for the small blocks studied here. (2) The retrieval gains are substantial, and the visual-merging results are especially encouraging at the higher compression ratio.

Critical weaknesses

(1) The comparisons do not fully isolate the value of modeling token correlations. A parameter- and training-matched compressor that explicitly models token interactions would be a more decisive baseline. (2) The language experiments use one 345M model, and the visual experiments use one main backbone. Multiple runs would help establish the stability of the smaller gains, particularly for 2×2 merging. The relation to HiLS-Attention is clear because this paper replaces its block compressor. It shares a test-time regression perspective with MesaNet, but uses the solution directly as a compressed representation.

Questions for authors

Could the authors compare against an interaction-aware compressor under matched parameters and training budget, and report results across multiple random seeds? This would clarify whether the gains specifically come from the proposed regression mechanism.

2 comments

Thank you for your constructive comments and for recognizing that our method "has a clear mechanism: the regression solution accounts for correlations among tokens, while the dual-form solve is practical for the small blocks studied here," and that "the retrieval gains are substantial, and the visual-merging results are especially encouraging at the higher compression ratio." We have conducted all suggested experiments and added them to the revised manuscript (highlighted in blue). We address each question below.

Weakness1 & Question1 (Part 1): Comparison with parameter- and training-matched interaction-aware baselines to isolate the value of the regression mechanism.

A1: Excellent suggestion! To isolate the specific value of our regression formulation and closed-form linear system solve (G−1v\boldsymbol{G}^{-1}\boldsymbol{v}) from general second-order token interaction modeling, we designed and evaluated two parameter- and training-matched interaction-aware baselines under the 3×33 \times 3 visual token compression setting in Appendix E.2 (Table 10):

  1. Gram-MLP: Computes the exact same 3×33 \times 3 cosine-similarity Gram matrix, flattens it (R81\mathbb{R}^{81}), and passes it through a parameter-matched MLP (81→23→981 \rightarrow 23 \rightarrow 9) followed by softmax to predict aggregation weights α\boldsymbol{\alpha}.
  2. Gram-Interaction Pool: Retains the exact learnable diagonal scaling D\boldsymbol{D} and scalar target projection v\boldsymbol{v} of TTR-Compressor, but replaces the closed-form linear system solver G−1v\boldsymbol{G}^{-1}\boldsymbol{v} with direct Gram-interaction scoring α=softmax(Gv)\boldsymbol{\alpha} = \text{softmax}(\boldsymbol{G}\boldsymbol{v}).
Method (3×33 \times 3 Token Compression)MMBenchMMStarRWQAMMMUAI2DOCRBenchInfoVQAChartQAMMSIERQAAvg.
MLP74.7450.8964.3144.4474.6855.9035.1663.6026.0033.0052.27
Gram-MLP (Interaction-Aware)75.9551.1865.7545.7876.5257.3037.9566.1228.9034.2553.97
Gram-Interaction Pool (Interaction-Aware)75.6951.1164.8445.3376.0758.5038.7366.5626.4036.0053.92
TTR-Compressor (Ours)76.4652.4065.6245.8975.2358.7038.2767.1626.8036.5054.30
TTR-Compressor (w/ Conv) (Ours)76.6351.1665.8847.4475.1660.0039.7067.1628.9036.5054.85

Both interaction-aware baselines (Gram-MLP at 53.97 and Gram-Interaction Pool at 53.92) outperform baseline that do not explicitly capture second-order correlations, confirming the value of modeling inter-token correlations. Crucially, they remain inferior to both TTR-Compressor (54.30) and TTR-Compressor (w/ Conv) (54.85), verifying the specific advantage of our ridge-regression formulation and its exact closed-form inverse solution over heuristic interaction scoring.

Weakness2 & Question1 (Part 2): Scaling up the language model size, evaluating multi-seed stability for 2×22\times 2 visual merging, and clarifying the relationship with HiLS-Attention and MesaNet.

A2: We have added both the multi-seed stability evaluation for 2×22 \times 2 visual merging and scaled-up 1B language model experiments to the revised manuscript:

  1. Multi-Seed Evaluation for 2×22 \times 2 Visual Merging (Appendix E.3, Table 11): We extended the single-seed evaluation in Table 4 (seed 42) with two additional independent training runs (seeds 43 and 44). Averaged across all three random seeds, TTR-Compressor and TTR-Compressor (w/ Conv) consistently outperform the MLP baseline by +0.93 (56.22 vs. 55.29) and +1.08 (56.37 vs. 55.29) points:
Method (3-Seed Avg: 42, 43, 44)MMBenchMMStarRWQAMMMUAI2DOCRBenchInfoVQAChartQAMMSIERQAAvg.
MLP77.2951.6764.7046.2676.1962.2742.2971.2826.1034.8355.29
TTR-Compressor77.2952.9165.0146.4876.7963.1043.8973.2127.2336.2556.22
TTR-Compressor (w/ Conv)76.3552.8166.3646.0476.7763.3045.8572.9527.4735.8356.37
  1. Scaling Up Language Model Size (1B Parameters, 100B Tokens) (Section 5.1, Table 3): To verify that our gains hold at larger scales, we trained a 1B-parameter model on 100B tokens and compared TTR-Compressor with the strongest baseline (Attention-Weighted Pooling) on RULER across 4K–32K context lengths:
Model (1B params, 100B tokens)S-NIAH-1 (4K/8K/16K/32K)S-NIAH-2 (4K/8K/16K/32K)S-NIAH-3 (4K/8K/16K/32K)MK-NIAH-1 (4K/8K/16K/32K)Avg.
Attention-Weighted Pooling0.978 / 0.972 / 0.984 / 0.9901.000 / 0.982 / 0.860 / 0.7320.898 / 0.856 / 0.456 / 0.5500.494 / 0.358 / 0.336 / 0.3140.7350
TTR-Compressor1.000 / 1.000 / 1.000 / 1.0001.000 / 0.956 / 0.862 / 0.7500.980 / 0.918 / 0.740 / 0.6620.626 / 0.442 / 0.446 / 0.4120.7996

TTR-Compressor achieves a substantial improvement (0.7996 vs. 0.7350), and its advantage widens at longer context lengths (16K and 32K).

  1. Relationship with HiLS-Attention and Distinctions from MesaNet:
    • Relation to HiLS-Attention: HiLS-Attention is a hierarchical sparse attention framework that incorporates architectural designs orthogonal to our focus, whereas our work specifically investigates the token compression algorithm. To ensure a strictly fair comparison across compression algorithms in language modeling, both TTR-Compressor and all competing baselines are evaluated within the exact same HiLS-Attention framework, differing only in the block compressor module. (Moreover, our visual token merging experiments operate outside HiLS-Attention, demonstrating that TTR-Compressor is a general-purpose compression primitive.)
    • Key Distinctions from MesaNet: While sharing the high-level perspective of test-time regression, TTR-Compressor fundamentally differs from MesaNet in four aspects:
      1. Compression Primitive vs. Fast-Weight Memory: MesaNet fits a matrix-shaped fast-weight state that must be queried by subsequent vectors for readout, whereas TTR-Compressor reframes test-time regression as a sequence compression primitive, directly using the fitted regression coefficient vector h⋆∈Rd\boldsymbol{h}^\star \in \mathbb{R}^d as the compressed token representation.
      2. Exact Dual Closed-Form vs. Approximate Iterative Primal Solver: MesaNet relies on approximate iterative primal conjugate gradient (CG) steps, whereas TTR-Compressor obtains an exact closed-form solution in the dual space.
      3. Explicit Token-to-Token Gram Matrix: At the core of TTR-Compressor is the explicit construction of the token-to-token Gram matrix in the dual space, which directly captures pairwise token correlations to suppress redundancy—a mechanism absent in MesaNet.
      4. Minimalist Self-Supervised Parameterization: MesaNet employs standard dual-branch high-dimensional linear projections to generate multi-dimensional key and value vectors. In contrast, TTR-Compressor introduces a highly compact self-supervised regression formulation: it directly normalizes input features as regression keys (kt\boldsymbol{k}_t) and uses a single vector a\boldsymbol{a} to project tokens into scalar regression targets (vtv_t).

We sincerely appreciate your time and constructive comments, which have greatly improved the quality of our manuscript!

Reviewer_EgVsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TTR-Compressor, a method for sequence compression, where a sequence of TT dd-dimensional vectors is compressed into a single dd-dimensional vector. For each sequence, it solves a ridge regression problem and uses the fitted coefficients as the compressed vector.

The approach is motivated by a limitation the authors identify in common sequence compressors such as mean pooling, projection of concatenated tokens, and attention-weighted pooling: tokens are weighted without accounting for redundancy between the tokens. The authors argue that this can lead to the dilution of distinctive information. In the regression solution, tokens are coupled together through their Gram matrix, so redundant tokens have diminishing rather than additive influence. The authors support this claim with an analysis in the appendix.

The solution is given in closed form, and the regression problem is trained end-to-end through this solution. The authors assume T<dT < d, and thus propose a dual formulation, at a cost of O(T2d+T3)\mathcal{O}(T^2 d + T^3) rather than the primal's O(Td2+d3)\mathcal{O}(T d^2 + d^3).

The authors evaluate the proposed method on two tasks: block-wise sparse attention for efficient language modeling, and visual token merging for efficient multimodal understanding. On said tasks, they show the superiority of their method.

Critical strengths
  1. The method is principled with sound theoretical support. The use of the ridge-regression coefficients as the compressed representation follows directly from the stated motivation and is properly formalized and derived.

  2. The method consistently improves average performance over the baselines on the evaluated tasks, while maintaining comparable efficiency and added parameters.

Critical weaknesses
  1. Language modeling evaluation is limited to the NIAH task. The results are based on four NIAH tasks from RULER at a single model scale (345M). NIAH is a reasonable diagnostic task, but it does not demonstrate general-purpose success on realistic tasks. The short context experiments in Table 2 do not exercise the compressor at all, as the authors note. Those results support the model's short-context common-sense reasoning capabilities, which are on par with baseline methods, but do not demonstrate that the long-term reasoning/understanding capabilities of the model are preserved/improved. Moreover, S-NIAH-1 is a best case for a redundancy suppressing compressor, since the haystack includes many repeated sentences. I would expect a much larger and diverse evaluation suite (e.g. the full RULER suite and LongBench).

  2. Baselines are narrow in both applications. The authors only compare with simple baselines they implement, but a broader comparison is needed to prove the efficacy of the method. For block-sparse attention, the only compressors compared are mean pooling, concatenation with projection, and the attention pooling. Established block compressors are missing, such as NSA (Yuan et al., 2025), Quest (Tang et al., 2024) and Landmark Attention (Mohtashami & Jaggi, 2023). More broadly, learned compression tokens such as gist tokens (Mu et al., 2023), ICAE (Ge et al., 2024) and Activation Beacon (Zhang et al., 2025) are absent from both the related work and the experiments. For visual merging, several established baselines exist as well, as cited in the paper: C-Abstractor (Cha et al., 2024), Q-Former (Li et al., 2023) and ToMe (Bolya et al., 2023). Other standard alternatives are also missing: TokenPacker (Li et al., 2025) among trained projectors, and redundancy-aware reduction methods such as DivPrune (Alvar et al., 2025) and VisionZip (Yang et al., 2025).

  3. Relevant prior work is missing. The proposed method generalizes Generalized Max Pooling (Murray & Perronnin, 2014). GMP defines the pooled vector as the ridge solution arg⁡min⁡h∥Kh−1∥22+λ∥h∥22\arg\min_h \lVert Kh - \mathbf{1}\rVert_2^2 + \lambda \lVert h\rVert_2^2, computes it in the dual, and is motivated by the same goal of keeping frequent descriptors from dominating the pooled representation. Deep GMP (Christlein et al., 2019) extends GMP by learning λ\lambda in an end-to-end manner. TTR-Compressor adds learned targets, key normalization, and a learned diagonal penalty. The learned targets are not ablated (v=1v = \mathbf{1}, with the same normalization and penalty, like GMP).

Questions for authors
  1. The paper claims that the method's gains are due to the coupling of tokens through the Gram matrix (Section 5.1). The comparison against additive baselines supports this claim. However, the baselines also differ from TTR-Compressor in the assignment of token weights (uniform/softmax rather than learned targets) and, as far as I understand, in using unnormalized keys. Could you compare against additive pooling with the same normalized keys and learned targets but without the matrix inverse, i.e., h=K⊤v=∑tvtkth = K^\top v = \sum_t v_t k_t? This would differ from TTR-Compressor only in the token coupling and would directly test your hypothesis, if I understand correctly.
6 comments

Thank you for your constructive comments and for recognizing that our method is "principled with sound theoretical support," that the formulation "follows directly from the stated motivation and is properly formalized and derived," and that it "consistently improves average performance over the baselines on the evaluated tasks, while maintaining comparable efficiency and added parameters." We have incorporated the suggested experiments into the revised manuscript (highlighted in blue) and address each point below.

Question1: Ablation study comparing against additive pooling with the same normalized keys and learned targets (h=K⊤v\boldsymbol{h} = \boldsymbol{K}^\top \boldsymbol{v}) without the Gram matrix inverse.

A1: Thank you for this insightful suggestion. We have added this exact ablation to Section 5.4 (Table 7). We evaluate the uncoupled additive variant h=K⊤v=∑t=1Tvtkt\boldsymbol{h} = \boldsymbol{K}^\top \boldsymbol{v} = \sum_{t=1}^T v_t \boldsymbol{k}_t using the exact same normalized keys (kt=xt/∥xt∥2\boldsymbol{k}_t = \boldsymbol{x}_t / \|\boldsymbol{x}_t\|_2) and learned targets (vt=a⊤xtv_t = \boldsymbol{a}^\top \boldsymbol{x}_t) without the Gram matrix inverse:

ModelS-NIAH-1 (4K/8K/16K/32K)S-NIAH-2 (4K/8K/16K/32K)S-NIAH-3 (4K/8K/16K/32K)MK-NIAH-1 (4K/8K/16K/32K)Avg.
TTR-Compressor0.996 / 0.996 / 0.992 / 0.6500.982 / 0.566 / 0.288 / 0.1260.770 / 0.376 / 0.188 / 0.0660.528 / 0.302 / 0.212 / 0.1100.5092
w/o Gram matrix (K⊤v\boldsymbol{K}^\top\boldsymbol{v})0.934 / 0.844 / 0.836 / 0.5520.928 / 0.572 / 0.244 / 0.1000.680 / 0.224 / 0.038 / 0.0960.538 / 0.324 / 0.176 / 0.1020.4493

Removing the Gram matrix inverse causes a substantial performance drop from 0.5092 to 0.4493 (-5.99 points). Furthermore, in Section 5.4 (Figure 4), we visualize the effective weights α\boldsymbol{\alpha} on real images, showing that redundant background/sleeve patches are strongly down-weighted while distinctive tokens retain high weights. Together, these results demonstrate the importance of the Gram matrix in explicitly modeling inter-token relationships and suppressing redundant information.

Weakness1: Limited language modeling evaluation (single 345M model scale and four NIAH tasks); need broader evaluations on the full RULER suite and LongBench.

A2: To thoroughly evaluate TTR-Compressor beyond basic needle retrieval and the 345M scale, we have added two major sets of experiments to the revised manuscript:

  1. Full RULER Suite and LongBench (Appendix E.4, Tables 12 and 13): We compare TTR-Compressor against the strongest baseline (Attention-Weighted Pooling) across the remaining tasks of the RULER suite and the full LongBench benchmark:
Method (Remaining RULER Tasks)MK-NIAH-2 (8K/16K)MK-NIAH-3 (8K/16K)MQ-NIAH (8K/16K)MV-NIAH (8K/16K)CWE (8K/16K)FWE (8K/16K)VT (8K/16K)QA-Hotpot (8K/16K)QA-SQuAD (8K/16K)Avg.
Attention-Weighted Pooling0.002 / 0.0000.000 / 0.0000.166 / 0.0630.188 / 0.0830.006 / 0.0050.191 / 0.1140.002 / 0.0080.122 / 0.1040.060 / 0.0530.065
TTR-Compressor0.002 / 0.0000.000 / 0.0000.191 / 0.1260.206 / 0.1300.008 / 0.0030.153 / 0.0650.000 / 0.0020.134 / 0.1040.075 / 0.0720.071
Method (LongBench)Single-Doc QAMulti-Doc QASummarizationFew-Shot LearningSynthetic TasksCode CompletionOverall Score
Attention-Weighted Pooling0.04760.04260.09550.09650.01670.04790.0578
TTR-Compressor0.04660.05400.10050.09760.02680.09010.0693

On the extended RULER suite, while several highly complex distraction/tracking subsets (e.g., MK-NIAH-2/3, CWE, and VT) yield near-zero accuracy for both methods and offer limited discriminative signal, TTR-Compressor consistently outperforms Attention-Weighted Pooling on subsets with meaningful evaluation signal (MQ-NIAH, MV-NIAH, QA-Hotpot, and QA-SQuAD) and achieves a higher overall average (0.071 vs. 0.065, a 9.2% relative improvement). Similarly, on LongBench, TTR-Compressor surpasses Attention-Weighted Pooling in 5 out of 6 categories as well as in the overall score (0.0693 vs. 0.0578, a 19.9% relative improvement).

  1. Scaling Up to a 1B Model on 100B Tokens: In Section 5.1 (Table 3), we scale up both TTR-Compressor and the strongest baseline (Attention-Weighted Pooling) to a 1B-parameter model trained on 100B tokens. TTR-Compressor consistently outperforms Attention-Weighted Pooling (0.7996 vs. 0.7350 average accuracy), with the gap widening at 16K and 32K lengths:
Model (1B params, 100B tokens)S-NIAH-1 (4K/8K/16K/32K)S-NIAH-2 (4K/8K/16K/32K)S-NIAH-3 (4K/8K/16K/32K)MK-NIAH-1 (4K/8K/16K/32K)Avg.
Attention-Weighted Pooling0.978 / 0.972 / 0.984 / 0.9901.000 / 0.982 / 0.860 / 0.7320.898 / 0.856 / 0.456 / 0.5500.494 / 0.358 / 0.336 / 0.3140.7350
TTR-Compressor1.000 / 1.000 / 1.000 / 1.0001.000 / 0.956 / 0.862 / 0.7500.980 / 0.918 / 0.740 / 0.6620.626 / 0.442 / 0.446 / 0.4120.7996

Weakness 2: Limited baseline comparisons.

A3: Thank you for your suggestion. We have expanded the comparisons in both applications and will clarify the relationship to the additional methods you suggested.

1. Block-wise sparse attention. We have added NSA and Quest. We also clarify that our existing “Attention-Weighted Pooling” baseline is the compressor used in Landmark Attention.

Method (RULER)S-NIAH-1 (4K/8K/16K/32K)S-NIAH-2 (4K/8K/16K/32K)S-NIAH-3 (4K/8K/16K/32K)MK-NIAH-1 (4K/8K/16K/32K)Avg.
NSA0.870 / 0.752 / 0.554 / 0.5520.770 / 0.136 / 0.040 / 0.0540.710 / 0.376 / 0.096 / 0.0880.316 / 0.190 / 0.042 / 0.0440.3494
Quest1.000 / 0.994 / 0.996 / 0.9920.508 / 0.074 / 0.034 / 0.0300.398 / 0.178 / 0.038 / 0.0560.238 / 0.126 / 0.036 / 0.0280.3579
HiLS-Attention (Attention-Weighted Pooling)0.944 / 0.914 / 0.876 / 0.7400.926 / 0.274 / 0.136 / 0.0600.340 / 0.082 / 0.026 / 0.0260.474 / 0.258 / 0.124 / 0.0720.3920
TTR-Compressor0.996 / 0.996 / 0.992 / 0.6500.982 / 0.566 / 0.288 / 0.1260.770 / 0.376 / 0.188 / 0.0660.528 / 0.302 / 0.212 / 0.1100.5092

TTR achieves the highest average accuracy among these methods, exceeding NSA, Quest, and HiLS by 15.98, 15.13, and 11.72 percentage points, respectively. These additional comparisons complement our controlled compressor study in Section 5.1, where hierarchical softmax, query calibration, and HoPE are held fixed to isolate the effect of the compression operator.

2. Visual token merging. We have expanded the comparisons under the same 3×33 \times 3 spatial compression setting to include Honeybee's C-Abstractor, and TokenPacker. Regarding Q-Former, its standard design uses a fixed number of learned global queries, whereas our setting compresses each local spatial group at a fixed ratio. We therefore prioritize TokenPacker as a region-to-point attention-based comparator more closely aligned with this setting.

MethodAverage score ↑
MLP52.27
Honeybee (C-Abstractor)51.43
TokenPacker53.00
TTR-Compressor54.30
TTR-Compressor (w/ Conv)54.85

Both TTR variants outperform all listed baselines on average, including alternatives that explicitly model pairwise token interactions. Per-benchmark results are provided in Appendix E.2, Table 10.

3. Comparison to broader compression methods. We agree that these works deserve explicit discussion, and we have added a dedicated "More Related Work" section in the Appendix G of the revised manuscript to cite and discuss them in detail:

  • Gist tokens, ICAE, and Activation Beacon primarily propose new architectural paradigms for efficient LLM context processing (via gist tokens, memory slots, or beacon activations), which are orthogonal to our evaluated block-wise sparse attention setting. Furthermore, their underlying token compression mechanism relies on attention-based aggregation, which directly corresponds to the Attention-Weighted Pooling baseline evaluated in our experiments.
  • ToMe, DivPrune, and VisionZip address visual token reduction from different angles: DivPrune performs token pruning (selection) rather than token merging, whereas ToMe merges tokens via weighted averaging where the weights correspond to the number of original tokens each merged token represents—a concept that does not exist in our single-stage, fixed-grid merging setting, making a direct comparison inapplicable. VisionZip groups redundant tokens based on semantic similarity and fuses tokens within each group via mean pooling. Its compression algorithm, mean pooling, has already been used as a baseline for comparison in our experiments.
Reviewer_VuYUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTR-Compressor, which compresses a block of tokens by fitting a small ridge regression to them and using the fitted coefficients as the compressed representation. The keys are normalized tokens, the targets are a learned projection of the tokens, and the regularizer is a learnable diagonal matrix. The regression is solved in closed form in the dual, while the target projection and regularizer are trained end-to-end. The authors argue that the Gram matrix lets the operator down-weight redundant tokens. The method is evaluated as the block compressor in sparse attention for language models (RULER) and as the visual token merger in a multimodal LLM.

Critical strengths
  1. The idea is simple and well motivated. The operator has a closed-form solution, very few extra parameters and a clean interpretation in Appendix B. The added training and inference cost is small (Fig. 3 and Fig. 4).
  2. The method gives clear gains in two quite different settings. On RULER it clearly improves over attention-weighted pooling, the strongest baseline (average 0.51 vs. 0.39 in Table 1). In the visual merger it beats the MLP merger, with larger gains under 3×3 merging and on text-heavy benchmarks such as OCRBench and InfoVQA (Table 3).
Critical weaknesses
  1. The main explanation is not tested directly. The paper attributes the gains to the Gram matrix suppressing redundant tokens, but TTR also differs from the baselines in its normalized keys and learned targets. The obvious control, using the same keys and targets without the Gram matrix (the large-λ\lambda limit noted in Appendix B), is not reported. Even λ=1\lambda=1 in Table 6 already beats attention-weighted pooling, so part of the gain may come from the key and target design rather than the coupling.
  2. All results are from single runs, and several conclusions rest on small or mixed differences. For example, the learnable diagonal is close to a fixed λ\lambda in Table 6, and the convolutional variant is not always better than plain TTR in Table 1. Some claims, such as the "collapse" without key normalization, are stronger than the tables show.
  3. The visual comparison is narrow. Only the standard MLP merger is used as a baseline. Mean pooling and attention pooling, which are used in the language setting, are missing, as are mergers cited in the paper such as ToMe [1], Honeybee [2] and DeCo [3]. The headline claim of matching the MLP merger with 55.6% fewer tokens also relies on the convolutional variant, while no baseline gets the same convolution.
  4. The language evidence is limited to one 345M model and four synthetic needle-in-a-haystack tasks. Apart from the simplest task, most methods score low beyond 8K. It is unclear whether the gains carry over to larger models or more realistic long-context tasks.

References

[1] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. "Token Merging: Your ViT But Faster." International Conference on Learning Representations (ICLR), 2023.

[2] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. "Honeybee: Locality-enhanced Projector for Multimodal LLM." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[3] Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. "DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models." arXiv preprint arXiv:2405.20985, 2024.

Questions for authors
  1. Can you report results with the same keys and targets but without the Gram matrix, in both settings? Is there any evidence on real data that redundant tokens are actually down-weighted?
  2. Can you report results over multiple seeds, at least for the main tables, so we can tell which differences are reliable?
  3. How does TTR compare with mean pooling and attention pooling as a visual merger, and with at least one of [2] or [3]? What does the MLP merger get with the same convolution?
  4. Do the gains hold for a larger model or for RULER tasks beyond needle retrieval?
3 comments

Thank you for your constructive comments and for recognizing that our idea is "simple and well motivated," has a "closed-form solution, very few extra parameters and a clean interpretation in Appendix B," incurs "small added training and inference cost," and "gives clear gains in two quite different settings." We have conducted all suggested experiments and incorporated them into the revised manuscript (highlighted in blue). We address each point below.

Weakness1 & Question1: Isolating the effect of the Gram matrix and empirical evidence of redundancy suppression on real data.

A1: Great suggestion! We have added both the suggested ablation and the real-data visualization to the revised paper:

  1. Isolating the Gram matrix (h=K⊤v\boldsymbol{h} = \boldsymbol{K}^\top \boldsymbol{v}): In Section 5.4 (Table 7), we evaluate an uncoupled additive variant using the exact same normalized keys (kt=xt/∥xt∥2\boldsymbol{k}_t = \boldsymbol{x}_t / \Vert\boldsymbol{x}_t\Vert_2) and learned targets (vt=a⊤xtv_t = \boldsymbol{a}^\top \boldsymbol{x}_t) without the Gram matrix inverse, computing h=K⊤v=∑t=1Tvtkt\boldsymbol{h} = \boldsymbol{K}^\top \boldsymbol{v} = \sum_{t=1}^T v_t \boldsymbol{k}_t. We conduct this ablation on both block-wise sparse attention (RULER) and 3×33 \times 3 visual token merging. Removing the Gram matrix leads to a substantial performance drop in both sparse attention and visual token merging settings, demonstrating the importance of the Gram matrix in explicitly modeling inter-token relationships and suppressing redundant information:
Method (RULER)S-NIAH-1 (4K/8K/16K/32K)S-NIAH-2 (4K/8K/16K/32K)S-NIAH-3 (4K/8K/16K/32K)MK-NIAH-1 (4K/8K/16K/32K)Avg.
TTR-Compressor0.996 / 0.996 / 0.992 / 0.6500.982 / 0.566 / 0.288 / 0.1260.770 / 0.376 / 0.188 / 0.0660.528 / 0.302 / 0.212 / 0.1100.5092
w/o Gram matrix (K⊤v\boldsymbol{K}^\top\boldsymbol{v})0.934 / 0.844 / 0.836 / 0.5520.928 / 0.572 / 0.244 / 0.1000.680 / 0.224 / 0.038 / 0.0960.538 / 0.324 / 0.176 / 0.1020.4493
Method (3×33 \times 3 Visual Merging)MMBenchMMStarRWQAMMMUAI2DOCRBenchInfoVQAChartQAMMSIERQAAvg.
TTR-Compressor76.4652.4065.6245.8975.2358.7038.2767.1626.8036.5054.30
w/o Gram matrix (K⊤v\boldsymbol{K}^\top\boldsymbol{v})75.7750.5263.7945.1175.6555.9038.1866.6426.4035.0053.31
  1. Empirical evidence of redundancy suppression on real data: In Section 5.4 (Figure 4), we visualize the effective weights α\boldsymbol{\alpha} within 3×33 \times 3 visual token groups on real images. Uniform background patches and repetitive sleeve patches are strongly suppressed (blue, low weight), whereas distinctive foreground objects (flowers, text characters, and human hands) receive substantially higher weights (red). This provides direct empirical evidence of input-adaptive deduplication.

Weakness2 & Question2: Multi-seed evaluation to verify the statistical stability of the performance gains.

A2: We have added a multi-seed evaluation in Appendix E.3 (Table 11). Given limited computational resources and time, we selected the 2×22 \times 2 visual token merging setting, where the performance margins are relatively smaller, and extended the single-seed runs in Table 4 (seed 42) with two additional independent seeds (43 and 44). The 3-seed results confirm that our gains over MLP remain consistent, and the convolutional variant remains superior to plain TTR-Compressor:

Method (3-Seed Avg: 42, 43, 44)MMBenchMMStarRWQAMMMUAI2DOCRBenchInfoVQAChartQAMMSIERQAAvg.
MLP77.2951.6764.7046.2676.1962.2742.2971.2826.1034.8355.29
TTR-Compressor77.2952.9165.0146.4876.7963.1043.8973.2127.2336.2556.22
TTR-Compressor (w/ Conv)76.3552.8166.3646.0476.7763.3045.8572.9527.4735.8356.37

Furthermore, following your advice, we have revised our wording from "collapse" without key normalization to "substantial performance degradation."

Weakness3 & Question3: Extended visual token merger baselines.

A3: We have added these extended baseline comparisons under the 3×33 \times 3 visual token compression setting in Appendix E.2 (Table 10). Specifically, we compare TTR-Compressor against:

  • Mean Pooling (DeCo): parameter-free uniform averaging within each 3×33 \times 3 window;
  • MLP (w/ Conv): equipping the standard MLP merger with the exact same 3×33 \times 3 convolutional layer used in our convolutional variant;
  • Honeybee (C-Abstractor): a representative locality-enhanced convolutional projector;
  • TokenPacker (Attn. Pool): a strong attention-weighted pooling baseline for visual token merging.
Method (3×33 \times 3 Token Compression)MMBenchMMStarRWQAMMMUAI2DOCRBenchInfoVQAChartQAMMSIERQAAvg.
Mean Pooling (DeCo)75.2650.8463.7944.5675.3655.0035.2265.7626.4033.2552.54
MLP74.7450.8964.3144.4474.6855.9035.1663.6026.0033.0052.27
MLP (w/ Conv)75.0050.8961.9646.5675.1355.7036.4064.8426.3034.0052.68
Honeybee (C-Abstractor)75.9547.9559.3546.0074.0051.1034.4763.5227.5034.5051.43
TokenPacker (Attn. Pool)75.8651.2563.4046.3374.6456.1036.6965.4426.5033.7553.00
TTR-Compressor76.4652.4065.6245.8975.2358.7038.2767.1626.8036.5054.30
TTR-Compressor (w/ Conv)76.6351.1665.8847.4475.1660.0039.7067.1628.9036.5054.85

Equipping MLP with the same 3×33 \times 3 convolution yields only a modest gain (52.27 →\rightarrow 52.68), and both TTR-Compressor (54.30) and TTR-Compressor (w/ Conv) (54.85) substantially outperform Mean Pooling (52.54), MLP (w/ Conv) (52.68), Honeybee (51.43), and TokenPacker (53.00).

Weakness4 & Question4: Scaling to larger language models and evaluating on broader long-context tasks beyond needle retrieval.

A4: Great suggestion! We have added both larger-scale model experiments and broader long-context evaluations to the revised manuscript:

  1. Scaling up model size (1B parameters) and training budget (100B tokens): In Section 5.1 (Table 3), we scale up both TTR-Compressor and the strongest baseline (Attention-Weighted Pooling) to a 1B-parameter model trained on 100B tokens. TTR-Compressor consistently outperforms Attention-Weighted Pooling (0.7996 vs. 0.7350 average accuracy), with the gap widening at 16K and 32K lengths:
Model (1B params, 100B tokens)S-NIAH-1 (4K/8K/16K/32K)S-NIAH-2 (4K/8K/16K/32K)S-NIAH-3 (4K/8K/16K/32K)MK-NIAH-1 (4K/8K/16K/32K)Avg.
Attention-Weighted Pooling0.978 / 0.972 / 0.984 / 0.9901.000 / 0.982 / 0.860 / 0.7320.898 / 0.856 / 0.456 / 0.5500.494 / 0.358 / 0.336 / 0.3140.7350
TTR-Compressor1.000 / 1.000 / 1.000 / 1.0001.000 / 0.956 / 0.862 / 0.7500.980 / 0.918 / 0.740 / 0.6620.626 / 0.442 / 0.446 / 0.4120.7996
  1. Full RULER suite and LongBench: In Appendix E.4 (Tables 12 and 13), we evaluate on the remaining RULER tasks and the LongBench benchmark. On the extended RULER suite, while several highly complex distraction/tracking subsets (e.g., MK-NIAH-2/3, CWE, and VT) yield near-zero accuracy for both methods and offer limited discriminative signal, TTR-Compressor consistently outperforms Attention-Weighted Pooling on subsets with meaningful evaluation signal (MQ-NIAH, MV-NIAH, QA-Hotpot, and QA-SQuAD) and achieves a higher overall average (0.071 vs. 0.065, a 9.2% relative improvement). Similarly, on LongBench, TTR-Compressor surpasses Attention-Weighted Pooling in 5 out of 6 categories as well as in the overall score (0.0693 vs. 0.0578, a 19.9% relative improvement):
Model (RULER Extended, 8K/16K)MK-NIAH-2MK-NIAH-3MQ-NIAHMV-NIAHCWEFWEVTQA-HotpotQA-SQuADAvg.
Attention-Weighted Pooling0.002 / 0.0000.000 / 0.0000.166 / 0.0630.188 / 0.0830.006 / 0.0050.191 / 0.1140.002 / 0.0080.122 / 0.1040.060 / 0.0530.065
TTR-Compressor0.002 / 0.0000.000 / 0.0000.191 / 0.1260.206 / 0.1300.008 / 0.0030.153 / 0.0650.000 / 0.0020.134 / 0.1040.075 / 0.0720.071
Model (LongBench)Single-Doc QAMulti-Doc QASummarizationFew-Shot LearningSynthetic TasksCode CompletionOverall Score
Attention-Weighted Pooling0.04760.04260.09550.09650.01670.04790.0578
TTR-Compressor0.04660.05400.10050.09760.02680.09010.0693

Finally, we would like to sincerely thank you once again for your time and valuable feedback, which have significantly strengthened our work!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers, AC, SAC, and PC,

Thank you for your efforts in organizing ICLR and coordinating the review process. We sincerely thank all reviewers for their insightful comments and constructive feedback.

We are grateful that our work was recognized for its simple, principled, and well-motivated formulation, sound theoretical support with an exact dual closed-form solution, clear mechanism for capturing inter-token correlations, minimal parameter and computational overhead, and consistent gains across both block-wise sparse attention and visual token merging.

Our responses focused on several core concerns:

Isolating the regression mechanism and redundancy suppression
Ablations against uncoupled pooling without the Gram matrix inverse (h=K⊤v\boldsymbol{h} = \boldsymbol{K}^\top \boldsymbol{v}), constant targets (v=1\boldsymbol{v} = \boldsymbol{1}), and parameter-matched interaction baselines (Gram-MLP and Gram-Interaction Pool) confirm the distinct value of our exact ridge-regression solve. Real-image weight visualizations further provide direct empirical evidence of input-adaptive redundancy suppression.

Scaling up model size and broader long-context evaluations
Scaling to a 1B-parameter model trained on 100B tokens yields substantial gains on RULER (0.7996 vs. 0.7350), with the margin widening at 16K–32K lengths. Evaluations on the remaining RULER suite and LongBench further show +9.2% and +19.9% relative improvements, respectively.

Expanded baseline comparisons
TTR-Compressor outperforms NSA and Quest in sparse attention, and surpasses DeCo, MLP (w/ Conv), Honeybee, and TokenPacker in visual token merging.

Statistical stability
Three-seed evaluations on 2×22 \times 2 visual token merging verify consistent, stable improvements (+0.93 and +1.08 points) over the MLP baseline.

Theoretical positioning and related work
We clarified our architectural role within HiLS-Attention, detailed key distinctions from MesaNet and GMP/Deep GMP, and expanded discussions on broader context and visual token reduction paradigms.

New results and discussions have been incorporated into the revised manuscript (highlighted in blue) to make our claims and findings more rigorous, comprehensive, and transparent.

Below, we address each reviewer's specific concerns in detail. We extend our heartfelt thanks to all reviewers for their recognition and valuable feedback. We welcome further discussion and look forward to continued engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.