acceptodds
Under review as a conference paper at ICLR 2027

Cross-Modal Patch Distillation with Budget-Enforced Any-Resolution Routing for Long-Context Compression

Abstract

Large language models (LLMs) are severely bottlenecked by long-context inputs during inference due to the quadratic complexity of self-attention and linear Key-Value cache growth. In this work, we introduce Cross-Modal Patch Distillation (CMPD), a context compression framework that maps extended textual contexts into a continuous cross-modal patch manifold prior to the transformer feedforward layers, under the hypothesis that visual manifolds can represent abstract information more efficiently than textual manifolds. CMPD leverages a model's native multimodal pathways to construct budget-constrained 2D patch grids, transforming long documents into a compact sequence of continuous cross-modal latent patches. To preserve critical verbatim tokens (e.g., proper nouns, dates, and numerical identifiers) without spatial low-pass degradation, CMPD dynamically routes query-relevant lexical anchors via a parameter-free Query-Centroid Anchor Routing mechanism. To support dynamic token budgets at inference time without retraining, we optimize CMPD in two stages: (1) multimodal manifold anchoring via a cross-modal contrastive objective on dense captioning data, and (2) causal task distillation across a diverse mixture of long-context benchmarks with stochastic budget sampling. Extensive evaluations across Qwen3.5-0.8B, Qwen3.5-9B, and Gemma-4-E4B on LongBench-v2, ZeroSCROLLS, and RULER demonstrate that CMPD achieves up to context compression, consistently outperforming discrete prompt compression and pruning baselines (such as LLMLingua) on complex multi-hop reasoning and long-form generative summarization while matching or exceeding uncompressed full-context performance. CMPD establishes a scalable and theoretically grounded paradigm for efficient long-context reasoning in modern foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Context Compression, Long Context

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wEXkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Cross-Modal Patch Distillation (CMPD), a framework for long-context compression that maps textual inputs into compact continuous latent patches using a query-conditioned cross-attention bottleneck. CMPD is trained in two stages: cross-modal alignment with visual features from image-caption data, followed by task-oriented distillation on long-context benchmarks with stochastic budget sampling. Experiments on Qwen3.5 and Gemma 4 models demonstrate improvements over the LLMLingua baseline under several compression budgets.

Critical strengths

The paper addresses the efficiency–performance trade-off in long-context compression, which is an important problem for practical LLM deployment.

Critical weaknesses
  1. The motivation for introducing visual representations into a text-only compression task is not convincing. The paper does not clearly explain why aligning textual latents with natural-image features should improve text compression, or whether this introduces an unnecessary modality gap.

  2. The experimental comparisons are too limited to support the paper's central motivation and claimed contributions. The evaluation primarily focuses on LLMLingua, without direct comparisons to representative learned continuous context compression methods under comparable settings. Furthermore, CMPD substantially underperforms the full-context baseline on Qwen3.5-9B (34.79% vs. 55.20%), and the RULER results reveal significant limitations in exact-information retrieval. These findings weaken the claimed effectiveness and generalizability.

Questions for authors
  1. The method selects isolated words based on embedding similarity and appends them to the compressed representation. Such disconnected lexical tokens may lose their original semantic relationships and introduce ambiguity. Further analysis is needed to establish whether this strategy reliably preserves useful information without disrupting downstream reasoning. More importantly, the lack of a Stage 1 removal ablation makes it impossible to determine whether cross-modal alignment provides any meaningful benefit over direct task-oriented training.

  2. The paper lacks a motivation figure and an overall architecture diagram. The connections between the two training stages, cross-modal alignment, latent compression, anchor routing, and inference are not clearly presented, making the method difficult to understand and evaluate. Given the complexity of the proposed framework, this is a substantial presentation issue rather than a minor writing concern.

  3. The efficiency analysis mainly emphasizes theoretical complexity and KV-cache reduction, without sufficiently demonstrating end-to-end runtime benefits. More comprehensive latency, throughput, and peak memory comparisons, including the compression overhead, are needed to support the practical efficiency claims.

  4. The paper emphasizes preserving spatial and chronological structure through a 2D latent manifold. However, the proposed cross-attention compressor allows each latent query to attend to the entire document, without explicitly enforcing spatial locality or correspondence to textual regions. It remains unclear how the claimed spatial structure emerges from this mechanism or why it should improve text compression.

  5. The core compression mechanism remains a conventional cross-attention bottleneck. Simply assigning 2D positional coordinates to latent queries does not necessarily establish meaningful spatial structure, especially when each query can attend to the entire document. The paper lacks sufficient evidence that the proposed 2D visual formulation is more effective than an equivalent 1D continuous latent compressor.

  6. The paper overlooks several highly relevant studies, including DeepSeek-OCR, VIST, and VisInContext, which have already explored visual representations for efficient text compression and extended-context processing. (DeepSeek-OCR: Contexts Optical Compression, Vision-centric Token Compression in Large Language Model, VisInContext: Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning)

4 comments
AuthorsOpenReview ↗

Thanks for actionable suggestions

We thank Reviewer wEXk for a thorough, high-expertise review examining modality motivation, continuous baselines, full-context and retrieval limits, anchor routing, Stage 1 removal, architectural presentation, 2D vs. 1D structure, end-to-end runtime, and visual-text literature.

Highlighted 2D Grid Design

Addressing your assessment that architectural presentation was a substantial issue rather than a minor writing concern, we overhauled Sections 1–3:

  • Clarified Core Narrative (Sections 1–2.1): We reframed CMPD explicitly as Coordinate-Parameterized 2D Visual Grid Completion—reusing a VLM's resolution-agnostic G×GG \times G 2D coordinate domain (M=G2M = G^2) to synthesize continuous patches directly from text embeddings in one cross-attention pass (Eqs. 1–3), rather than claiming natural images are semantically richer than text.
  • Visual & Algorithmic Transparency (Figure 2, Algorithm 1, Table 1): Added an end-to-end architecture diagram (Figure 2), unified inference pseudocode (Algorithm 1), explicit batched anchor-alignment and position-indexing equations (Eqs. 4–7), and a stage-wise gradient-flow map (Table 1), alongside concise finding-focused prose in Section 3.

W1. Clarified Motivation for Visual-Space Alignment vs. Modality Gap (Tables 3–5)

"The paper does not clearly explain why aligning textual latents with natural-image features should improve text compression, or whether this introduces an unnecessary modality gap."

Our revised narrative and three controlled experiments (Global Response Tables A1 & A3) directly clarify what visual-grid formulation and Stage 1 alignment do:

  1. Why a 2D Coordinate Grid (Section 3.3.1, Table 3): Holding Qwen3.5-0.8B, Stage 1+2 data, and s=0\mathbf{s}=\mathbf{0} fixed, 1D sinusoidal queries (Pure 1D) collapse on SCROLLS GovReport (4.28%–4.78% ROUGE-1), whereas factoring the same MM slots into a G×GG \times G 2D sinusoidal grid (Pure 2D) restores GovReport to 18.20%–18.65% (20.73% with Hybrid offset s\mathbf{s}).
  2. Stage 1 Removal Proves No Harmful Modality Gap (Section 3.3.5, Table 4): Training No Stage 1 (random-init patchifier trained directly in Stage 2) reaches 26.04% / 26.04% / 26.64% on LBv2 and 18.86% / 19.35% / 19.30% on GovReport, whereas Stage 1 Only (frozen backbone without Stage 2 / LoRA) drops to 21.87%–22.66% LBv2 and 0.84%–1.68% GovReport. Thus, the 2D coordinate grid + Stage 2 task distillation drive downstream reading, while Stage 1 visual alignment acts as a helpful +1.00 to +2.19 LBv2 and +1.02 to +1.43 ROUGE-1 initialization (CMPD: 27.04% / 28.23% / 28.43% LBv2; 19.93% / 20.37% / 20.73% GovReport).
  3. Natural-Image (PixelProse) vs. Rendered-Document (Wikipedia) Stage 1 (Section 3.3.4, Table 4 & Table 5): Replacing PixelProse with 400,000 rendered Wikipedia page images paired with their exact text yields 26.24% / 26.44% / 27.04% LBv2 and 21.35% / 21.41% / 21.40% GovReport (+0.67 to +1.42 ROUGE-1 over PixelProse), while synthesized patches match ViT-encoded Wikipedia page tokens at 0.783 mean (0.829–0.833 median) cosine similarity (Table 5), confirming insensitivity to natural-image vs. rendered-page alignment corpora.

W2. Continuous Baselines, Qwen3.5-9B Gap (34.79% vs. 55.20%), & RULER Limits

  1. Learned Continuous Baselines (Tables A1–A2, Tables 2–3): Added ICAE and ComprExIT (disclosing Llama-1B/8B base-model differences) plus same-backbone (Qwen3.5-0.8B) Pure 1D and 1D Soft-Prompt baselines.
  2. Candid Analysis of Qwen3.5-9B (34.79% vs. 55.20%) & 131k RULER (Section 3.3.8, Table 12): Explicitly highlighted in the Abstract, Introduction, Section 3.2, and Appendix J. While CMPD matches uncompressed Full Context on 0.8B (28.43% vs. 26.10%) and Gemma-4 (28.83% vs. 28.43% LBv2; 19.03% vs. 25.30% GovReport) and beats 9B LLMLingua-2 (34.79% vs. 25.84%), aggressive front-end compression loses fine-grained multi-hop cues that 9B exploits under full attention (compounded by 32k non-thinking lm-eval vs. the official 256k thinking-mode 55.20% score). Similarly, on full 131,072-token RULER (0.8B), CMPD reaches 68.00%–72.53% on FWE and 8.80%–9.73% on SQuAD QA (7.54%–7.90% 12-task mean), but random-string NIAH/VT stays near zero (0.00%–2.90%) because context-free passcodes lack cosine alignment with qc\mathbf{q}_c and are smoothed by softmax pooling.

Q1. Whole-Word Anchor Contextualization (Anchorless) & Stage 1 Removal (No Stage 1)

  1. Contextualized Whole-Word Spans + Anchorless Ablation (Sections 2.2–2.3 & 3.3.6, Table 4): Rather than appending disconnected subwords, CMPD groups subwords into chronological whole-word spans wj\\{w_j\\} sorted in document order \text{orig\\_idx}(\cdot), contextualizes them over Edoc\mathbf{E}_{\text{doc}} via shared FiLM + Cross-Attention (Eq. 5, supervised by Stage 1 fidelity Lfid\mathcal{L}_{\text{fid}}), and assigns source positions \text{orig\\_idx}(\cdot) in Stage 2 (Eq. 7) with 3030\\% stochastic anchor dropout. Training an Anchorless variant (00 anchors in Stage 2 and inference; Table A3 / Table 4) reaches 25.25% / 25.05% / 24.85% LBv2 and 19.80% / 19.51% / 18.93% GovReport (87%–99% of CMPD), confirming the 2D grid carries core document semantics; adding Query-Centroid whole-word anchors consistently lifts LBv2 by +1.79 / +3.18 / +3.58 points (27.04% / 28.23% / 28.43%) and GovReport by +0.13 / +0.86 / +1.80 ROUGE-1 as MM grows (whereas unconditioned learned-gate anchors collapse GovReport to 0.43% on 9B, Appendix H Table 15).
  2. Stage 1 Removal Ablation (Section 3.3.5, Table 4): Completed in Table A3 / Table 4: No Stage 1 achieves 26.04% / 26.04% / 26.64% LBv2 and 18.86% / 19.35% / 19.30% GovReport, while Stage 1 Only scores 21.87%–22.66% LBv2 and 0.84%–1.68% GovReport—showing Stage 1 provides a modest +1.0 to +2.2 point initialization benefit while Stage 2 distillation is essential.

Q2. Overall Architecture Diagram (Figure 2) & Stage-to-Inference Connections

Added Figure 2 (Section 2), Algorithm 1, Eqs. 1–7, and Table 1, explicitly connecting G×GG \times G coordinate grid initialization, Query-Centroid whole-word selection, single-pass batched FiLM cross-attention, Stage 1 (LS1\mathcal{L}_{S1} on patchifier) vs. Stage 2 (LS2\mathcal{L}_{S2} on patchifier, projector, and LoRA), and contiguous inference indexing peval(r)=rp_{\text{eval}}(r)=r.

Q3. End-to-End Runtime, Compression Overhead, Throughput, & Peak Memory

Detailed in Global Response Part 2 (Appendix B, Table 8). At M=3000,T=512M=3000, T=512, CMPD's single-layer compression + routing takes 4.3 ms (8k) to 20.7 ms (128k) (98.7x–235.6x faster than ComprExIT/ICAE/LLMLingua-2 compression overhead), achieving 50.9–68.7 ms Total TTFT (2.5x–73.5x faster than Full Context and 71.6x faster than LLMLingua-2 at 128k), 52.7 tok/s decoding, and 3.27 GB Peak VRAM (4.15x below Full Context).

Q4 & Q5. How 2D Grid Coordinates Operate Under Global Cross-Attention & 1D Comparisons

  1. Why Global Cross-Attention + 2D Query Geometry (Q4, Sections 2.1 & 2.3): Hard local window masks would block cross-segment synthesis (0.8B Multi-Doc QA: 28.73% CMPD vs. 24.12% Pure 1D and 16.98%–18.98% chunked ComprExIT/ICAE-8B; Table 11). As clarified in Section 2.3, 2D coordinates gi=(⌊i/G⌋,i mod G)\mathbf{g}_i = (\lfloor i/G \rfloor, i \bmod G) parameterize the query generator ΩG(i)\boldsymbol{\Omega}_G(i) (Eq. 1) prior to cross-attention, factoring M=G2M = G^2 queries across a compact 2D grid [0,G−1]2[0, G-1]^2 (G∈31,54,70G \in \\{31, 54, 70\\}) rather than stretching indices to [0,M−1][0, M-1] along 1D, while slot diversity and ordering are reinforced by Stage 1 off-diagonal decorrelation Ldecorr\mathcal{L}_{\text{decorr}} (Eq. 8) and Stage 2 row-stride source positions ptrain(r)=⌊ur⋅S⌋p_{\text{train}}(r) = \lfloor u_r \cdot S \rfloor (Eq. 7).
  2. Same-Backbone 2D vs. Equivalent 1D Compressors (Q5, Section 3.3.1, Table 3): On Qwen3.5-0.8B (Table A1), holding s=0\mathbf{s}=\mathbf{0} fixed, Pure 2D lifts GovReport from Pure 1D's 4.28%–4.78% collapse to 18.20%–18.65% (+13.8 ROUGE-1) and edges Pure 1D on LBv2 (23.06%–23.66% vs. 22.47%–23.26%; Pure 1D leads short Reasoning-Sensitive choices 27.97% and 131k RULER FWE 81.33%–82.27%, while Hybrid 2D nearly doubles RULER SQuAD QA 8.80%–9.73% vs. 4.13%–5.00%). Combining s\mathbf{s} (B=1B=1) with the 2D grid (CMPD) beats static 1D Soft-Prompt slots on LBv2 (27.04%–28.43% vs. 26.24%–27.63%) and GovReport (19.93%–20.73% vs. 19.50%–19.57%) with zero-shot budget scaling.

Q6. Visual-Text Compression Studies (VisInContext, VIST, DeepSeek-OCR)

Added citations and discussion of VisInContext (Wang et al., NeurIPS 2024), VIST (Xing et al., NeurIPS 2025), and DeepSeek-OCR (Wei et al., 2025) in Section 5: unlike optical pipelines that render text into RGB images for a multi-layer ViT at fixed resolution, CMPD completes a dynamic G×GG \times G latent grid directly from embeddings in 4.3–20.7 ms, trading off exact optical character fidelity on random strings.

Thank you for the authors' response and additional experiments. However, I still find the overall motivation and technical presentation difficult to follow, and the revisions do not sufficiently clarify the rationale behind the proposed design. Therefore, I maintain my original score.

AuthorsOpenReview ↗

Thank you for your thoughtful follow-up.

Prompted by your question on why a 2D representation outperforms its 1D counterpart, we elevated the geometric mechanism, non-prolonged multi-row relative distance, into Section 1 (Introduction) as our Key Observation and Figure 1, paired with Figure 3 (Section 2.1) and a new general R×WR \times W / G×GG \times G multi-row diagnostic suite (R∈{2,4,8,16,32,64,100}R \in \{2, 4, 8, 16, 32, 64, 100\}; Appendix K Tables 15–17):

1. Why 2D outperforms 1D ? (a) Non-prolonged multi-row distance (Section 1 & Appendix K) + (b) Native PDF-page representation (Figure 3).

  • (a) Concatenating RR rows in 1D prolongs early-row distance to R⋅WR \cdot W, whereas an R×WR \times W 2D grid bounds spatial distance to O(M)\mathcal{O}(\sqrt{M}): Suppose a compressed memory organizes M=R⋅WM = R \cdot W slots across RR rows or chunks of width WW. Flattening or concatenating chunks into a 1D sequence pushes Row 11 to relative distance R⋅W=MR \cdot W = M from the query while Row RR sits adjacent to the query; in a 2D visual grid under VLM mRoPE, all RR rows share identical temporal and column distances to the query, shrinking inter-row positional separation by W×W\times (up to 100×100\times at 100×100100 \times 100). Across R∈{2,4,8,16}R \in \{2, 4, 8, 16\} compressed GovReport rows (M≤8,192M \le 8{,}192), fixed-M=4,096M=4{,}096 aspect ratios (2×2048→64×642 \times 2048 \to 64 \times 64), and square grids up to 100×100100 \times 100 (M=10,000M = 10{,}000), 1D concatenation decays First-Row / Last-Row query attention to 0.74–0.77—and in chunk-concatenated 1D ICAE/ComprExIT (1B & 8B) across R=2→32R = 2 \to 32 chunks (M=256→4,096M = 256 \to 4{,}096), Chunk-0 / Last-Chunk attention drops to 0.29–0.64 (dragging ComprExIT-1B multi-depth retrieval from 62.5% down to 19.8%; Table 17). Conversely, 2D mRoPE keeps First-Row / Last-Row attention flat at 0.99–1.02 (4.2×4.2\times–28.1×28.1\times lower cross-row variance; Table 15). On the compressor side (Eq. 1), Pure 1D stretches sinusoidal indices across [0,M−1][0, M-1] (20×20\times past Mtrain=256M_{\text{train}}=256), causing 41.1× top/bottom slot collapse on GovReport (Table 16), whereas factoring MM into a M×M\sqrt{M} \times \sqrt{M} 2D grid keeps slot balance compact (1.4×–1.8×); switching only inference decoder positions to explicit 2D mRoPE zero-shot adds a further +0.68 to +1.04 ROUGE-1 as MM scales (1k→5k1\text{k} \to 5\text{k}).
  • (b) Visual alignment to the corresponding rendered PDF (Figure 3, Section 2.1): Because VLMs are pre-trained with 2D mRoPE over document/image patch grids, Figure 3 shows that CMPD's text-synthesized 2D grid matches Qwen3.5-0.8B's native ViT representation of the corresponding rendered PDF/page (0.783 mean / 0.829–0.833 median cosine similarity; 53.3–53.6 ROUGE-1 / 22.6–23.1 Corpus BLEU output agreement) in 4.3–20.7 ms without pixel rendering, outperforming Pure 1D across every generative metric.

2. Strict apples-to-apples comparisons & off-the-shelf ICAE/ComprExIT checkpoints.

  • Strict same-backbone 1D vs. 2D (Table 3, Qwen3.5-0.8B): Every row in Table 3 shares the identical Qwen3.5-0.8B backbone, cross-attention bottleneck, Stage 1+2 data/steps, LoRA config, and Query-Centroid router—varying only query geometry (Eqs. 1–2). With s=0\mathbf{s}=\mathbf{0} and zero extra parameters, switching from Pure 1D to Pure 2D lifts GovReport ROUGE-1 from 4.45 / 4.78 / 4.28% to 18.58 / 18.65 / 18.20% and edges LBv2 at every budget (23.66 / 23.26 / 23.06% vs. 23.26 / 22.66 / 22.47%). Likewise, CMPD (Hybrid 2D) uses fewer query parameters (11 shared vector vs. Mmax⁡M_{\max} static slots) yet beats same-backbone 1D Soft-Prompt in all 6/6 settings (27.04 / 28.23 / 28.43% vs. 26.24 / 26.84 / 27.63% LBv2; 19.93 / 20.37 / 20.73% vs. 19.57 / 19.50 / 19.50% GovReport) and LLMLingua-2 in all 18/18 settings (Table 2).
  • Off-the-shelf ICAE & ComprExIT checkpoints (Table 2): We did not train ICAE or ComprExIT on downstream tasks; we evaluated the publicly released 4×4\times SFT checkpoints (Jiang-nan/{icae,comprexit}-{1b,8b}-4x-sft, Ye et al., 2026) directly under the identical lm-eval harness. Consistent with the multi-row 1D distance prolongation above, concatenating more 1D chunks degrades ICAE-8B (26.04% →\to 22.66%) and ComprExIT-8B (25.25% →\to 23.46%) as MM grows, whereas CMPD's unified 2D grid improves or holds steady (27.04% →\to 28.43% on 0.8B; 34.59–34.79% on 9B).
Reviewer_yaYkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for compressing long textual contexts into latent representations aligned with a multimodal model’s visual input space. The results show better accuracy preservation than the baseline.

Critical strengths

The paper introduces a cross-modal compression approach that utilizes the potential of multimodal models for long context processing, which is an interesting idea.

Critical weaknesses

The evaluation pays much to discuss problem diagnosis and ablation, but has not enough experiments on the overall method performance. The only end-to-end method evaluation, Table 2, has only one old baseline LLMLingua, which was released in 2023. There are many newer methods like LLMLingua-2, LongLLMLingua, and so on. This paper needs to include them (at least three more baselines), otherwise it is hard to prove the proposed method has the state-of-the-art performance. In addition, the efficiency of the proposed method is not evaluated against prior compression methods, which is a problem since the paper emphasizes the computational complexity in Section 2.6.

Questions for authors

Could the authors add the necessary experiments above?

2 comments
AuthorsOpenReview ↗

Thanks for actionable suggestions

We thank Reviewer yaYk for constructive feedback on improving overall presentation clarity, expanding end-to-end baseline coverage, and benchmarking runtime efficiency against prior compression methods.

1. Notification on Presentation Revision & Clarified 2D Grid Narrative

"Presentation: 2: Generally understandable, but requires substantial effort in places. The evaluation pays much to discuss problem diagnosis and ablation, but has not enough experiments on the overall method performance."

To make the paper substantially easier to follow, we restructured Sections 1–3 around our 2D Visual Grid Completion design:

  • Section 2 now includes an end-to-end architecture diagram (Figure 2), unified pseudocode (Algorithm 1), explicit equations for G×GG \times G 2D coordinate grid completion and batched whole-word anchor routing (Eqs. 1–7), and a stage-wise parameter update table (Table 1).
  • Section 3.2 (Main Results) has been rewritten to synthesize high-level end-to-end empirical takeaways across five compression baselines concisely, rather than overloading the text with diagnostic prose and redundant score lists.

2. Expanding End-to-End Baselines ("At Least Three More Baselines")

"The only end-to-end method evaluation, Table 2, has only one old baseline LLMLingua, which was released in 2023. There are many newer methods like LLMLingua-2, LongLLMLingua, and so on. This paper needs to include them (at least three more baselines)..."

We corrected the label on our discrete baseline—which already used LLMLingua-2 (Pan et al., ACL 2024)—and added four learned continuous compression baselines (five baselines total) across M∈{1000,3000,5000}M \in \{1000, 3000, 5000\} and T=512T=512 (Global Response Tables A1–A2; Section 3.2 Table 2 and Section 3.3.1 Table 3 in the revised manuscript):

  1. LLMLingua-2 (Pan et al., ACL 2024) — same-backbone discrete pruning on Qwen3.5-0.8B, Qwen3.5-9B, and Gemma-4-E4B (Table 2).
  2. ICAE (Ge et al., ICLR 2024) — 4×4\times SFT continuous chunk compressor on Llama-3.2-1B and Llama-3.1-8B (Table 2, with explicit base-model and uncompressed Full Context disclosures).
  3. ComprExIT (Ye et al., 2026) — Optimal Transport + inter-layer fusion continuous compressor on Llama-3.2-1B and Llama-3.1-8B (Table 2).
  4. Pure 1D Continuous Compressor (s=0\mathbf{s}=\mathbf{0}) — controlled same-backbone 1D sinusoidal query baseline on Qwen3.5-0.8B (Table 3).
  5. 1D Soft-Prompt Compressor (Semantic Slot) — controlled same-backbone static 1D learnable-slot baseline on Qwen3.5-0.8B (Table 3).

Across all 9 same-backbone LongBench-v2 settings and all 9 SCROLLS GovReport settings, CMPD outperforms LLMLingua-2, surpasses both controlled same-backbone 1D continuous baselines on Qwen3.5-0.8B, and avoids the multi-chunk concatenation degradation observed as MM grows in ICAE and ComprExIT.

3. End-to-End Efficiency Evaluation Against Prior Compression Methods

"In addition, the efficiency of the proposed method is not evaluated against prior compression methods, which is a problem since the paper emphasizes the computational complexity in Section 2.6 [now Section 2.5]. Could the authors add the necessary experiments above?"

Yes. Global Response Part 2 (Appendix B, Table 8 in the revised manuscript) benchmarks warmed-up wall-clock latency (Comp. + Route, Backbone Prefill, Total TTFT), 6464-token decoding throughput (tok/s), and Peak VRAM (GB) across N∈{8k,32k,64k,128k}N \in \{8\text{k}, 32\text{k}, 64\text{k}, 128\text{k}\} at M=3000,T=512M=3000, T=512 against Full Context, LLMLingua-2, ICAE-1B, and ComprExIT-1B:

  • Multi-layer chunk encoders shift rather than eliminate prefill latency, requiring 2,043.5–4,876.5 ms of compression overhead at N=128kN=128\text{k} (4,876.5 ms for LLMLingua-2's 24-layer encoder, 4,219.6 ms for ICAE's 16-layer LoRA encoder, and 2,043.5 ms for ComprExIT's 16-layer + Sinkhorn OT encoder).
  • In contrast, CMPD's single-layer pre-backbone cross-attention and vectorized cosine-centroid anchor routing take only 4.3–20.7 ms (98.7x–235.6x lower compression overhead at 128k128\text{k}), achieving 68.7 ms Total TTFT at 128k128\text{k} (73.5$\times$ faster than Full Context and 71.6$\times$ faster than same-backbone LLMLingua-2), preserving full 52.7 tok/s decoding speed, and cutting Peak VRAM from 13.57 GB to 3.27 GB (4.15x reduction).

Dear Reviewer, thank you for your time and effort in reviewing this submission. As the discussion period is ongoing, I would appreciate it if you could join the discussion and engage with the authors' rebuttal at your earliest convenience.

Reviewer_6iAbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a "multi-modal" approach for LLM compression, which compresses LLM context into a 2D singal with mROPE where the downstream reads the image to answer the questions.

The framework is further equipped with other types of tokens which prove to be useful for the compression performance. The whole framework is trained via two stages: alignment stage and then task oriented fine-tuning.

The authors test their proposal on various benchmarks with Qwen models at different sizes. The paper establishes its advantage against LLMingua and has done its ablation studies that justifies its compressed representation choices (which composes of 3 types of tokens including the 2D representation token).

Critical strengths

I personally haven't seen tentatives to incorporate the compression as a 2D signal.

In terms of experiments, the paper presents experiments that validate the paper's compression design by carefully ablating through each slots.

Critical weaknesses

The paper's main proposal is hybird compression or soft compression where the main slot needs to be learned. In this case, the comparison is too far from being complete. The paper is similar to ICAE which probably first proposes to train as two stages and followed by various works including 500xcompressor, SAC, etc. Even the sparse compressor that the authors compared are not SOTA as LLMLingua was almost one of the first approaches proposed for LLMcompression. Without such comparison, the readers won't be able to tell if any advantage is offered by compression into 2D approaches.

The language of the paper is not precise and looks like AI generated which does not yet quality ICLR bar. Examples are: 033 theoretically grounded (the study and the domain is largely empirical), 153 "spatial low-pass filter that can smooth out verbatim named entities, numerical figures, and critical keywords." where it is not sure the low-pass filter is correct and if the mechanism is smoothing, etc.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Thanks for actionable suggestions

We thank Reviewer 6iAb for noting the novelty of formulating context compression as a 2D signal and our slot-level ablations, and for direct criticism on presentation clarity, imprecise phrasing, and missing soft/sparse baselines.

1. Paper Framing and Precise Terminology

We agree that our initial submission suffered from imprecise, metaphorical phrasing and substantially rewrote the manuscript around 2D Visual Grid Completion:

  1. Clarified 2D Grid Formulation (Sections 1–2, Figure 2, Algorithm 1, Table 1): Rather than relying on abstract metaphors, Section 2 now formalizes the pipeline algebraically across three explicit stages:

    • (i) 2D Coordinate Grid Initialization & Completion (Section 2.1, Eqs. 1–3): A resolution-agnostic G×GG \times G query grid ΩG(i)=MLPgen(PE2D(gi))+s\boldsymbol{\Omega}_G(i) = \text{MLP}_{\text{gen}}(\text{PE}_{2\text{D}}(\mathbf{g}_i)) + \mathbf{s} (M=G2M = G^2) attends over document token embeddings via a single FiLM-conditioned cross-attention layer.
    • (ii) Query-Centroid Anchor Routing (Section 2.2, Eqs. 4–5): Stopword-filtered, cosine-selected whole-word spans are aligned within the same batched cross-attention pass.
    • (iii) Unified Sequence Assembly & Gradient Flow (Section 2.3, Eqs. 6–7, Algorithm 1, Table 1): The resulting representations are assembled into a unified sequence, accompanied by an explicit stage-wise gradient map (Table 1).

    Additionally, Section 3.2 and Sections 3.3.4–3.3.6 have been revised into clear, finding-focused prose to emphasize empirical behavior over metaphorical framing.

  2. Terminology Refinement (e.g., "Theoretically Grounded" and "Spatial Low-Pass Filter"): We have completely removed these phrases and now explicitly state the accuracy trade-offs in the Abstract, Introduction, Section 3.2, and Conclusion. While "spatial low-pass filter" is a standard computer vision concept that naturally describes smoothing over our 2D visual grid representation, we appreciate that replacing it with the underlying mathematical formulation improves precision and avoids ambiguity. Accordingly, in Sections 2.2 and 3.3.6, we replace the metaphor with the exact algebraic mechanism: softmax cross-attention forms each visual patch as a convex combination of token value vectors (∑j=1Nαijvj\sum_{j=1}^N \alpha_{ij}\mathbf{v}_j, where αij≥0\alpha_{ij} \ge 0 and ∑jαij=1\sum_j \alpha_{ij}=1), capturing document-wide semantics while averaging over isolated verbatim identifiers (such as names, dates, and exact figures). Our Anchorless ablation (Section 3.3.6, Table 4) empirically isolates this effect: continuous 2D patches alone retain 87%–93% of LongBench-v2 performance (24.85%–25.25%) and 91%–99% of GovReport performance (18.93%–19.80%), whereas whole-word anchors restore fine-grained lexical precision (+1.79 to +3.58 on LongBench-v2 and +0.13 to +1.80 ROUGE-1 as MM increases).

2. Comparisons to Two-Stage Soft Compressors (ICAE, ComprExIT), Same-Backbone 1D Baselines, and LLMLingua-2

We addressed both sparse and learned soft compression in Sections 3.1–3.3, Tables 2–4, and Global Response Tables A1–A3:

  1. Recent Sparse Baseline (LLMLingua-2): Our discrete baseline already used LLMLingua-2 (xlm-roberta-large-meetingbank, Pan et al., ACL 2024; previously mislabeled "LLMLingua") across all three backbones (Table 2), where CMPD leads across all 9 LBv2 and all 9 GovReport settings while cutting 128k128\text{k} TTFT by 71.6×71.6\times (68.7 ms vs. 4,920.3 ms; Table 8).
  2. Comparison to Two-Stage 1D Soft Compressors (ICAE & ComprExIT): We evaluated 4×4\times SFT checkpoints of ICAE (Ge et al., ICLR 2024) and ComprExIT (Ye et al., 2026) across M∈1k,3k,5kM \in \\{1\text{k}, 3\text{k}, 5\text{k}\\} (Table 2, disclosing Llama-1B/8B base-model differences). ICAE and ComprExIT run multi-layer LLM encoders over 512512-token chunks (4,219.6 ms and 2,043.5 ms compression latency at 128k128\text{k} vs. CMPD's single-layer 20.7 ms) and concatenate 8→24→408 \to 24 \to 40 compressed 1D spans at larger budgets, degrading LBv2 as MM grows (ICAE-8B: 26.04% -> 22.66%; ComprExIT-8B: 25.25% -> 23.46%; ComprExIT-1B: 24.65% -> 19.09%). Even on GovReport, where ICAE-8B leads at M=1kM=1\text{k} (15.51% vs. 14.46%), expanding MM degrades ICAE-8B (14.23%–14.32%) while CMPD's global 2D grid improves to 14.91%. Moreover, unlike ICAE's mandatory billion-token text autoencoding, our No Stage 1 ablation (Table 4) reaches 26.04%–26.64% LBv2 and 18.86%–19.35% GovReport from random patchifier initialization, showing Stage 1 is a +1.0 to +2.2 point initializer rather than a hard requirement.
  3. Unconfounded Same-Backbone Advantage of the 2D Grid (Section 3.3.1, Table 3): On Qwen3.5-0.8B with identical cross-attention, training data, LoRA, and anchor router: (a) holding s=0\mathbf{s}=\mathbf{0} fixed, Pure 1D collapses on GovReport (4.28%–4.78%), whereas Pure 2D restores GovReport to 18.20%–18.65%; and (b) Hybrid 2D (CMPD) outperforms static 1D Soft-Prompt slots on LBv2 (27.04%–28.43% vs. 26.24%–27.63%) and GovReport (19.93%–20.73% vs. 19.50%–19.57%) while scaling zero-shot across budgets MM.
Reviewer_6iAbOpenReview ↗

Thanks for clarifying my questions

The paper has clarified its baslines and has added tw stage 1D soft compressors (that ICAE is added with SFT checkpionts (I don't know if the authors trained them for the tasks or they are taken directly from ICAE)), in any case, the proposed method seems to perform better than those baselines.

I believe what the authors have put in the rebuttal would strengthen the paper, although I don't fully get why 2D representation performs better than its 1D counterpart and I wonder we have strict apple to apple comparison.

Based on the submission, I don't change my score. Again, I believe the rebuttals are in good shape and should strengthen the paper in its message and clarity.

Prompted by your question on why a 2D representation outperforms its 1D counterpart, we elevated the geometric mechanism, non-prolonged multi-row relative distance, into Section 1 (Introduction) as our Key Observation and Figure 1, paired with Figure 3 (Section 2.1) and a new general R×WR \times W / G×GG \times G multi-row diagnostic suite (R∈{2,4,8,16,32,64,100}R \in \{2, 4, 8, 16, 32, 64, 100\}; Appendix K Tables 15–17):

1. Why 2D outperforms 1D ? (a) Non-prolonged multi-row distance (Section 1 & Appendix K) + (b) Native PDF-page representation (Figure 3).

  • (a) Concatenating RR rows in 1D prolongs early-row distance to R⋅WR \cdot W, whereas an R×WR \times W 2D grid bounds spatial distance to O(M)\mathcal{O}(\sqrt{M}): Suppose a compressed memory organizes M=R⋅WM = R \cdot W slots across RR rows or chunks of width WW. Flattening or concatenating chunks into a 1D sequence pushes Row 11 to relative distance R⋅W=MR \cdot W = M from the query while Row RR sits adjacent to the query; in a 2D visual grid under VLM mRoPE, all RR rows share identical temporal and column distances to the query, shrinking inter-row positional separation by W×W\times (up to 100×100\times at 100×100100 \times 100). Across R∈{2,4,8,16}R \in \{2, 4, 8, 16\} compressed GovReport rows (M≤8,192M \le 8{,}192), fixed-M=4,096M=4{,}096 aspect ratios (2×2048→64×642 \times 2048 \to 64 \times 64), and square grids up to 100×100100 \times 100 (M=10,000M = 10{,}000), 1D concatenation decays First-Row / Last-Row query attention to 0.74–0.77—and in chunk-concatenated 1D ICAE/ComprExIT (1B & 8B) across R=2→32R = 2 \to 32 chunks (M=256→4,096M = 256 \to 4{,}096), Chunk-0 / Last-Chunk attention drops to 0.29–0.64 (dragging ComprExIT-1B multi-depth retrieval from 62.5% down to 19.8%; Table 17). Conversely, 2D mRoPE keeps First-Row / Last-Row attention flat at 0.99–1.02 (4.2×4.2\times–28.1×28.1\times lower cross-row variance; Table 15). On the compressor side (Eq. 1), Pure 1D stretches sinusoidal indices across [0,M−1][0, M-1] (20×20\times past Mtrain=256M_{\text{train}}=256), causing 41.1× top/bottom slot collapse on GovReport (Table 16), whereas factoring MM into a M×M\sqrt{M} \times \sqrt{M} 2D grid keeps slot balance compact (1.4×–1.8×); switching only inference decoder positions to explicit 2D mRoPE zero-shot adds a further +0.68 to +1.04 ROUGE-1 as MM scales (1k→5k1\text{k} \to 5\text{k}).
  • (b) Visual alignment to the corresponding rendered PDF (Figure 3, Section 2.1): Because VLMs are pre-trained with 2D mRoPE over document/image patch grids, Figure 3 shows that CMPD's text-synthesized 2D grid matches Qwen3.5-0.8B's native ViT representation of the corresponding rendered PDF/page (0.783 mean / 0.829–0.833 median cosine similarity; 53.3–53.6 ROUGE-1 / 22.6–23.1 Corpus BLEU output agreement) in 4.3–20.7 ms without pixel rendering, outperforming Pure 1D across every generative metric.

2. Strict apples-to-apples comparisons & off-the-shelf ICAE/ComprExIT checkpoints.

  • Strict same-backbone 1D vs. 2D (Table 3, Qwen3.5-0.8B): Every row in Table 3 shares the identical Qwen3.5-0.8B backbone, cross-attention bottleneck, Stage 1+2 data/steps, LoRA config, and Query-Centroid router—varying only query geometry (Eqs. 1–2). With s=0\mathbf{s}=\mathbf{0} and zero extra parameters, switching from Pure 1D to Pure 2D lifts GovReport ROUGE-1 from 4.45 / 4.78 / 4.28% to 18.58 / 18.65 / 18.20% and edges LBv2 at every budget (23.66 / 23.26 / 23.06% vs. 23.26 / 22.66 / 22.47%). Likewise, CMPD (Hybrid 2D) uses fewer query parameters (11 shared vector vs. Mmax⁡M_{\max} static slots) yet beats same-backbone 1D Soft-Prompt in all 6/6 settings (27.04 / 28.23 / 28.43% vs. 26.24 / 26.84 / 27.63% LBv2; 19.93 / 20.37 / 20.73% vs. 19.57 / 19.50 / 19.50% GovReport) and LLMLingua-2 in all 18/18 settings (Table 2).
  • Off-the-shelf ICAE & ComprExIT checkpoints (Table 2): We did not train ICAE or ComprExIT on downstream tasks; we evaluated the publicly released 4×4\times SFT checkpoints (Jiang-nan/{icae,comprexit}-{1b,8b}-4x-sft, Ye et al., 2026) directly under the identical lm-eval harness. Consistent with the multi-row 1D distance prolongation above, concatenating more 1D chunks degrades ICAE-8B (26.04% →\to 22.66%) and ComprExIT-8B (25.25% →\to 23.46%) as MM grows, whereas CMPD's unified 2D grid improves or holds steady (27.04% →\to 28.43% on 0.8B; 34.59–34.79% on 9B).

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response

We thank Reviewers 6iAb, yaYk, and wEXk for their actionable suggestions. We revised the manuscript to highlight our 2D Visual Grid Completion design, added four continuous baselines (five total), added Stage-removal (No Stage 1, Stage 1 Only), dataset, and Anchorless ablations (Table 4), and profiled latency, compression overhead, throughput, and VRAM (Appendix B, Table 8).

1. Highlighting the 2D Grid Design

To resolve presentation ambiguity and imprecise phrasing, we restructured Sections 1–3 around three explicit mechanisms:

  1. Coordinate-Parameterized 2D Grid (Sections 1–2.1, Eqs. 1–3): 1D compressors either slice static soft-prompt tables (fixing max budget) or concatenate chunk-wise 1D spans (degrading as budgets grow), while optical methods rasterize text into RGB images for a multi-layer ViT. Instead, CMPD initializes a resolution-agnostic G×GG \times G blank visual grid (M=G2M = G^2) from 2D sinusoidal coordinates ΩG(i)=MLPgen(PE2D(gi))+s\boldsymbol{\Omega}_G(i) = \text{MLP}_{\text{gen}}(\text{PE}_{2\text{D}}(\mathbf{g}_i)) + \mathbf{s} (Eq. 1, B=1B=1) and completes it from document embeddings in one FiLM-conditioned cross-attention pass (Eqs. 2–3), scaling zero-shot to any budget MM.
  2. Batched Whole-Word Routing & Architecture Diagram (Figure 2, Sections 2.2–2.4, Eqs. 4–9): Because softmax cross-attention averages document tokens via convex combinations and smooths over isolated strings (names, dates, figures), CMPD pairs ΩG\boldsymbol{\Omega}_G in the same single-layer batched cross-attention pass with parameter-free Query-Centroid whole-word anchors (Eqs. 4–5). Figure 2, Algorithm 1, Eqs. 6–7 (segment-wise training vs. contiguous inference positions), and Table 1 (gradient flow) connect Stage 1, Stage 2 LoRA distillation, and inference.
  3. Precise Prose & Findings-First Analysis: Removed metaphorical/overstated terms (0 occurrences of "theoretically grounded" or "low-pass filter"), rewrote Section 3.2 around core findings rather than score lists, and added Section 5 visual-text comparisons (VisInContext, VIST, DeepSeek-OCR).

2. Five Baselines Across Discrete, Chunk-Wise 1D, and Same-Backbone 1D Families (T=512T=512)

LongBench-v2 and SCROLLS GovReport use max_length = 32,768 (trailing T=512T=512); RULER uses full 131,072-token contexts. Because ICAE (Ge et al., ICLR 2024) and ComprExIT (Ye et al., 2026) couple compressors to Llama-3.2-1B/3.1-8B (22.07%/30.00% Full Context LBv2 vs. 26.10%/55.20% official Qwen3.5 scores; 9B uses 256k thinking mode), we separate Table A1 (same-backbone, Tables 2–3) from Table A2 (Llama scaling, Table 2):

Table A1: Same-Backbone Results (Qwen3.5-0.8B, 9B, Gemma-4-E4B; M∈1k,3k,5k,T=512M \in \\{1\text{k}, 3\text{k}, 5\text{k}\\}, T=512, in %)

MethodLBv2 (0.8B)LBv2 (9B)LBv2 (Gemma⋄^\diamond)GovRep (0.8B)GovRep (9B)GovRep (Gemma⋄^\diamond)RULER (0.8B^{\text{¶}})
Full Context26.10♯^\sharp55.20♯^\sharp28.43——25.30—
LLMLingua-224.25/25.84/25.4524.65/25.84/20.2821.07/21.87/21.472.89/8.26/12.573.37/7.35/8.761.46/7.53/14.06—
1D Soft-Prompt26.24/26.84/27.63——19.57/19.50/19.50———
Pure 1D (s=0\mathbf{s}=\mathbf{0})23.26/22.66/22.47——4.45/4.78/4.28——8.33/8.34/8.44
Pure 2D (s=0\mathbf{s}=\mathbf{0})23.66/23.26/23.0632.01/31.61/30.22—18.58/18.65/18.2017.55/18.84/18.80§^\S——
CMPD (Ours)27.04/28.23/28.4334.59/34.79/34.5928.03/28.83/28.4319.93/20.37/20.7314.46/14.58/14.9118.74/18.96/19.037.90/7.63/7.54

♯^\sharpOfficial Qwen3.5 scores (9B: 256k thinking vs. 32k non-thinking lm-eval). ⋄^\diamondStage-2 LoRA-merged. §^\SStage-1 learned gating (Table 15). ^{\text{¶}}Full 131k (Table 12).

Table A2: Within-Backbone Budget Scaling on Llama-3.2-1B & Llama-3.1-8B (M∈1k,3k,5k,T=512M \in \\{1\text{k}, 3\text{k}, 5\text{k}\\}, T=512, in %)

Method$^\dagger$LBv2 (Llama-1B)LBv2 (Llama-8B)GovReport (Llama-1B)GovReport (Llama-8B)
Full Context†^\dagger22.0730.00⋆^\star——
ICAE†^\dagger (4×4\times SFT)23.86/22.66/24.0626.04/24.06/22.660.99/0.67/0.7015.51/14.23/14.32
ComprExIT†^\dagger (4×4\times SFT)24.65/20.68/19.0925.25/24.45/23.466.20/5.59/5.402.60/2.68/2.76

3. New Stage-Removal, Dataset, and Anchorless Ablations (Sections 3.3.4–3.3.6, Table 4)

Table A3 isolates training stages, alignment corpora, and routed anchors on Qwen3.5-0.8B (T=512T=512):

Table A3: Training Stage, Alignment Corpus, and Anchor Routing Ablations on Qwen3.5-0.8B (Table 4)

Training Variant (Qwen3.5-0.8B, T=512T=512)LBv2 (1k/3k/5k1\text{k} / 3\text{k} / 5\text{k})SCROLLS GovReport (1k/3k/5k1\text{k} / 3\text{k} / 5\text{k})
Wiki Stage 1 (400k rendered Wikipedia pages; Stage 2 unchanged)26.24 / 26.44 / 27.04%21.35 / 21.41 / 21.40%
LongBench-v1 Stage 2 (Stage 1 unchanged; narrower Stage 2 data)24.65 / 24.65 / 24.25%12.91 / 11.82 / 11.02%
No Stage 1 (random-init patchifier trained directly in Stage 2)26.04 / 26.04 / 26.64%18.86 / 19.35 / 19.30%
Stage 1 Only (visual alignment only; frozen LLM without LoRA)21.87 / 22.66 / 22.47%1.68 / 0.99 / 0.84%
Anchorless (00 anchor words in Stage 2 training & inference)25.25 / 25.05 / 24.85%19.80 / 19.51 / 18.93%
CMPD (Full Hybrid 2D, Ours)27.04 / 28.23 / 28.43%19.93 / 20.37 / 20.73%

Key Empirical Findings (Tables A1–A3):

  1. 2D Grid vs. 1D Sequence & Chunk Scaling: CMPD beats same-backbone LLMLingua-2 across all 18 LBv2/GovReport cells (Table A1). Holding s=0\mathbf{s}=\mathbf{0} fixed, switching queries from 1D sinusoidal (Pure 1D) to 2D grid (Pure 2D) restores GovReport (4.28%–4.78% →\to 18.20%–18.65%), while Hybrid 2D beats 1D Soft-Prompt slots and scales zero-shot across MM without the chunk-concatenation drop of ComprExIT/ICAE-8B (Table A2; on GovReport, ICAE-8B leads CMPD-9B at M=1kM=1\text{k}, 15.51% vs. 14.46%, before declining at M≥3kM \ge 3\text{k} where CMPD-9B overtakes it at 14.58%–14.91%).
  2. Stage 2 + 2D Grid Drive Reading; Stage 1 & Anchors Add Initialization & Precision (Table A3): No Stage 1 reaches 26.04%–26.64% LBv2 and 18.86%–19.35% GovReport (-1.00 to -2.19 LBv2; -1.02 to -1.43 ROUGE-1 vs. CMPD), whereas Stage 1 Only collapses on GovReport (0.84%–1.68%): the 2D coordinate grid + Stage 2 distillation teach the decoder to read patches, while Stage 1 gives a modest, corpus-insensitive (Wiki vs. PixelProse) initialization gain. Anchorless retains 87%–93% LBv2 and 91%–99% GovReport, while whole-word anchors add +1.79 to +3.58 LBv2 accuracy as MM grows.
  3. Candid Limitations: CMPD trails official 256k thinking-mode Qwen3.5-9B Full Context (34.79% vs. 55.20% LBv2) and scores 0.00%–2.90% on 131k RULER random-string NIAH/VT (Section 3.3.8, Appendix J).

4. End-to-End Runtime, Compression Overhead, Throughput, & Peak VRAM (Appendix B, Table 8)

Warmed-up latency, 6464-token decode speed, and peak VRAM on an NVIDIA L40S (bfloat16, M=3000,T=512M=3000, T=512):

NNMethod (Backbone)Comp.+Route (ms)Prefill (ms)Total TTFT (ms)SpeedupDecode (tok/s)‡^\ddaggerVRAM (GB)
8kFull / LLMLingua-2 (Qwen-0.8B)— / 279.4128.1 / 42.4128.1 / 321.71.0×1.0\times / 0.4×0.4\times52.5 / 51.82.18 / 3.97
ICAE-1B / ComprExIT-1B (Llama-1B)208.5 / 121.246.6 / 36.3255.1 / 157.5—132.1 / 140.33.29 / 3.90
CMPD (Qwen-0.8B)4.346.650.92.5$\times$52.61.84
32kFull / LLMLingua-2 (Qwen-0.8B)— / 1,180.5757.9 / 38.5757.9 / 1,219.11.0×1.0\times / 0.6×0.6\times52.8 / 52.34.42 / 4.66
ICAE-1B / ComprExIT-1B (Llama-1B)980.2 / 657.857.4 / 49.71,037.6 / 707.4—132.2 / 137.94.23 / 3.71
CMPD (Qwen-0.8B)7.047.554.513.9$\times$52.41.99
64kFull / LLMLingua-2 (Qwen-0.8B)— / 2,402.91,800.2 / 34.91,800.2 / 2,437.81.0×1.0\times / 0.7×0.7\times51.5 / 52.17.53 / 4.66
ICAE-1B / ComprExIT-1B (Llama-1B)2,161.3 / 1,020.157.6 / 50.82,218.9 / 1,070.9—132.9 / 139.74.35 / 3.77
CMPD (Qwen-0.8B)11.247.558.630.7$\times$52.72.42
128kFull / LLMLingua-2 (Qwen-0.8B)— / 4,876.55,049.6 / 43.85,049.6 / 4,920.31.0×1.0\times / 1.03×1.03\times52.9 / 52.013.57 / 4.66
ICAE-1B / ComprExIT-1B (Llama-1B)4,219.6 / 2,043.557.7 / 50.74,277.4 / 2,094.1—132.5 / 140.24.38 / 3.80
CMPD (Qwen-0.8B)20.747.968.773.5$\times$52.73.27

4. Why 2D outperforms 1D ? (a) Non-prolonged multi-row distance (Section 1 & Appendix K) + (b) Native PDF-page representation (Figure 1).

  • (a) Concatenating RR rows in 1D prolongs early-row distance to R⋅WR \cdot W, whereas an R×WR \times W 2D grid bounds spatial distance to O(M)\mathcal{O}(\sqrt{M}): Suppose a compressed memory organizes M=R⋅WM = R \cdot W slots across RR rows or chunks of width WW. Flattening or concatenating chunks into a 1D sequence pushes Row 11 to relative distance R⋅W=MR \cdot W = M from the query while Row RR sits adjacent to the query; in a 2D visual grid under VLM mRoPE, all RR rows share identical temporal and column distances to the query, shrinking inter-row positional separation by W×W\times (up to 100×100\times at 100×100100 \times 100). Across R∈{2,4,8,16}R \in \{2, 4, 8, 16\} compressed GovReport rows (M≤8,192M \le 8{,}192), fixed-M=4,096M=4{,}096 aspect ratios (2×2048→64×642 \times 2048 \to 64 \times 64), and square grids up to 100×100100 \times 100 (M=10,000M = 10{,}000), 1D concatenation decays First-Row / Last-Row query attention to 0.74–0.77—and in chunk-concatenated 1D ICAE/ComprExIT (1B & 8B) across R=2→32R = 2 \to 32 chunks (M=256→4,096M = 256 \to 4{,}096), Chunk-0 / Last-Chunk attention drops to 0.29–0.64 (dragging ComprExIT-1B multi-depth retrieval from 62.5% down to 19.8%; Table 17). Conversely, 2D mRoPE keeps First-Row / Last-Row attention flat at 0.99–1.02 (4.2×4.2\times–28.1×28.1\times lower cross-row variance; Table 15). On the compressor side (Eq. 1), Pure 1D stretches sinusoidal indices across [0,M−1][0, M-1] (20×20\times past Mtrain=256M_{\text{train}}=256), causing 41.1× top/bottom slot collapse on GovReport (Table 16), whereas factoring MM into a M×M\sqrt{M} \times \sqrt{M} 2D grid keeps slot balance compact (1.4×–1.8×); switching only inference decoder positions to explicit 2D mRoPE zero-shot adds a further +0.68 to +1.04 ROUGE-1 as MM scales (1k→5k1\text{k} \to 5\text{k}).
  • (b) Visual alignment to the corresponding rendered PDF (Figure 3, Section 2.1): Because VLMs are pre-trained with 2D mRoPE over document/image patch grids, Figure 3 shows that CMPD's text-synthesized 2D grid matches Qwen3.5-0.8B's native ViT representation of the corresponding rendered PDF/page (0.783 mean / 0.829–0.833 median cosine similarity; 53.3–53.6 ROUGE-1 / 22.6–23.1 Corpus BLEU output agreement) in 4.3–20.7 ms without pixel rendering, outperforming Pure 1D across every generative metric.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.