- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formulates sequence compression as an input-specific ridge regression problem. It uses the fitted coefficients as a compressed representation and evaluates this operator in block-wise sparse attention and visual token merging. The results show clear gains in long-context retrieval and stronger visual compression.
Critical strengths
(1) The method has a clear mechanism: the regression solution accounts for correlations among tokens, while the dual-form solve is practical for the small blocks studied here. (2) The retrieval gains are substantial, and the visual-merging results are especially encouraging at the higher compression ratio.
Critical weaknesses
(1) The comparisons do not fully isolate the value of modeling token correlations. A parameter- and training-matched compressor that explicitly models token interactions would be a more decisive baseline. (2) The language experiments use one 345M model, and the visual experiments use one main backbone. Multiple runs would help establish the stability of the smaller gains, particularly for 2×2 merging. The relation to HiLS-Attention is clear because this paper replaces its block compressor. It shares a test-time regression perspective with MesaNet, but uses the solution directly as a compressed representation.
Questions for authors
Could the authors compare against an interaction-aware compressor under matched parameters and training budget, and report results across multiple random seeds? This would clarify whether the gains specifically come from the proposed regression mechanism.
2 comments
Weakness2 & Question1 (Part 2): Scaling up the language model size, evaluating multi-seed stability for visual merging, and clarifying the relationship with HiLS-Attention and MesaNet.
A2: We have added both the multi-seed stability evaluation for visual merging and scaled-up 1B language model experiments to the revised manuscript:
- Multi-Seed Evaluation for Visual Merging (Appendix E.3, Table 11): We extended the single-seed evaluation in Table 4 (seed 42) with two additional independent training runs (seeds 43 and 44). Averaged across all three random seeds, TTR-Compressor and TTR-Compressor (w/ Conv) consistently outperform the MLP baseline by +0.93 (56.22 vs. 55.29) and +1.08 (56.37 vs. 55.29) points:
| Method (3-Seed Avg: 42, 43, 44) | MMBench | MMStar | RWQA | MMMU | AI2D | OCRBench | InfoVQA | ChartQA | MMSI | ERQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MLP | 77.29 | 51.67 | 64.70 | 46.26 | 76.19 | 62.27 | 42.29 | 71.28 | 26.10 | 34.83 | 55.29 |
| TTR-Compressor | 77.29 | 52.91 | 65.01 | 46.48 | 76.79 | 63.10 | 43.89 | 73.21 | 27.23 | 36.25 | 56.22 |
| TTR-Compressor (w/ Conv) | 76.35 | 52.81 | 66.36 | 46.04 | 76.77 | 63.30 | 45.85 | 72.95 | 27.47 | 35.83 | 56.37 |
- Scaling Up Language Model Size (1B Parameters, 100B Tokens) (Section 5.1, Table 3): To verify that our gains hold at larger scales, we trained a 1B-parameter model on 100B tokens and compared TTR-Compressor with the strongest baseline (Attention-Weighted Pooling) on RULER across 4K–32K context lengths:
| Model (1B params, 100B tokens) | S-NIAH-1 (4K/8K/16K/32K) | S-NIAH-2 (4K/8K/16K/32K) | S-NIAH-3 (4K/8K/16K/32K) | MK-NIAH-1 (4K/8K/16K/32K) | Avg. |
|---|---|---|---|---|---|
| Attention-Weighted Pooling | 0.978 / 0.972 / 0.984 / 0.990 | 1.000 / 0.982 / 0.860 / 0.732 | 0.898 / 0.856 / 0.456 / 0.550 | 0.494 / 0.358 / 0.336 / 0.314 | 0.7350 |
| TTR-Compressor | 1.000 / 1.000 / 1.000 / 1.000 | 1.000 / 0.956 / 0.862 / 0.750 | 0.980 / 0.918 / 0.740 / 0.662 | 0.626 / 0.442 / 0.446 / 0.412 | 0.7996 |
TTR-Compressor achieves a substantial improvement (0.7996 vs. 0.7350), and its advantage widens at longer context lengths (16K and 32K).
- Relationship with HiLS-Attention and Distinctions from MesaNet:
- Relation to HiLS-Attention: HiLS-Attention is a hierarchical sparse attention framework that incorporates architectural designs orthogonal to our focus, whereas our work specifically investigates the token compression algorithm. To ensure a strictly fair comparison across compression algorithms in language modeling, both TTR-Compressor and all competing baselines are evaluated within the exact same HiLS-Attention framework, differing only in the block compressor module. (Moreover, our visual token merging experiments operate outside HiLS-Attention, demonstrating that TTR-Compressor is a general-purpose compression primitive.)
- Key Distinctions from MesaNet: While sharing the high-level perspective of test-time regression, TTR-Compressor fundamentally differs from MesaNet in four aspects:
- Compression Primitive vs. Fast-Weight Memory: MesaNet fits a matrix-shaped fast-weight state that must be queried by subsequent vectors for readout, whereas TTR-Compressor reframes test-time regression as a sequence compression primitive, directly using the fitted regression coefficient vector as the compressed token representation.
- Exact Dual Closed-Form vs. Approximate Iterative Primal Solver: MesaNet relies on approximate iterative primal conjugate gradient (CG) steps, whereas TTR-Compressor obtains an exact closed-form solution in the dual space.
- Explicit Token-to-Token Gram Matrix: At the core of TTR-Compressor is the explicit construction of the token-to-token Gram matrix in the dual space, which directly captures pairwise token correlations to suppress redundancy—a mechanism absent in MesaNet.
- Minimalist Self-Supervised Parameterization: MesaNet employs standard dual-branch high-dimensional linear projections to generate multi-dimensional key and value vectors. In contrast, TTR-Compressor introduces a highly compact self-supervised regression formulation: it directly normalizes input features as regression keys () and uses a single vector to project tokens into scalar regression targets ().
We sincerely appreciate your time and constructive comments, which have greatly improved the quality of our manuscript!
Thank you for your constructive comments and for recognizing that our method "has a clear mechanism: the regression solution accounts for correlations among tokens, while the dual-form solve is practical for the small blocks studied here," and that "the retrieval gains are substantial, and the visual-merging results are especially encouraging at the higher compression ratio." We have conducted all suggested experiments and added them to the revised manuscript (highlighted in blue). We address each question below.
A1: Excellent suggestion! To isolate the specific value of our regression formulation and closed-form linear system solve (G−1v) from general second-order token interaction modeling, we designed and evaluated two parameter- and training-matched interaction-aware baselines under the 3×3 visual token compression setting in Appendix E.2 (Table 10):
Both interaction-aware baselines (Gram-MLP at 53.97 and Gram-Interaction Pool at 53.92) outperform baseline that do not explicitly capture second-order correlations, confirming the value of modeling inter-token correlations. Crucially, they remain inferior to both TTR-Compressor (54.30) and TTR-Compressor (w/ Conv) (54.85), verifying the specific advantage of our ridge-regression formulation and its exact closed-form inverse solution over heuristic interaction scoring.