- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Cross-Modal Patch Distillation (CMPD), a framework for long-context compression that maps textual inputs into compact continuous latent patches using a query-conditioned cross-attention bottleneck. CMPD is trained in two stages: cross-modal alignment with visual features from image-caption data, followed by task-oriented distillation on long-context benchmarks with stochastic budget sampling. Experiments on Qwen3.5 and Gemma 4 models demonstrate improvements over the LLMLingua baseline under several compression budgets.
Critical strengths
The paper addresses the efficiency–performance trade-off in long-context compression, which is an important problem for practical LLM deployment.
Critical weaknesses
-
The motivation for introducing visual representations into a text-only compression task is not convincing. The paper does not clearly explain why aligning textual latents with natural-image features should improve text compression, or whether this introduces an unnecessary modality gap.
-
The experimental comparisons are too limited to support the paper's central motivation and claimed contributions. The evaluation primarily focuses on LLMLingua, without direct comparisons to representative learned continuous context compression methods under comparable settings. Furthermore, CMPD substantially underperforms the full-context baseline on Qwen3.5-9B (34.79% vs. 55.20%), and the RULER results reveal significant limitations in exact-information retrieval. These findings weaken the claimed effectiveness and generalizability.
Questions for authors
-
The method selects isolated words based on embedding similarity and appends them to the compressed representation. Such disconnected lexical tokens may lose their original semantic relationships and introduce ambiguity. Further analysis is needed to establish whether this strategy reliably preserves useful information without disrupting downstream reasoning. More importantly, the lack of a Stage 1 removal ablation makes it impossible to determine whether cross-modal alignment provides any meaningful benefit over direct task-oriented training.
-
The paper lacks a motivation figure and an overall architecture diagram. The connections between the two training stages, cross-modal alignment, latent compression, anchor routing, and inference are not clearly presented, making the method difficult to understand and evaluate. Given the complexity of the proposed framework, this is a substantial presentation issue rather than a minor writing concern.
-
The efficiency analysis mainly emphasizes theoretical complexity and KV-cache reduction, without sufficiently demonstrating end-to-end runtime benefits. More comprehensive latency, throughput, and peak memory comparisons, including the compression overhead, are needed to support the practical efficiency claims.
-
The paper emphasizes preserving spatial and chronological structure through a 2D latent manifold. However, the proposed cross-attention compressor allows each latent query to attend to the entire document, without explicitly enforcing spatial locality or correspondence to textual regions. It remains unclear how the claimed spatial structure emerges from this mechanism or why it should improve text compression.
-
The core compression mechanism remains a conventional cross-attention bottleneck. Simply assigning 2D positional coordinates to latent queries does not necessarily establish meaningful spatial structure, especially when each query can attend to the entire document. The paper lacks sufficient evidence that the proposed 2D visual formulation is more effective than an equivalent 1D continuous latent compressor.
-
The paper overlooks several highly relevant studies, including DeepSeek-OCR, VIST, and VisInContext, which have already explored visual representations for efficient text compression and extended-context processing. (DeepSeek-OCR: Contexts Optical Compression, Vision-centric Token Compression in Large Language Model, VisInContext: Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning)
4 comments
Q1. Whole-Word Anchor Contextualization (Anchorless) & Stage 1 Removal (No Stage 1)
- Contextualized Whole-Word Spans + Anchorless Ablation (Sections 2.2–2.3 & 3.3.6, Table 4): Rather than appending disconnected subwords, CMPD groups subwords into chronological whole-word spans sorted in document order \text{orig\\_idx}(\cdot), contextualizes them over via shared FiLM + Cross-Attention (Eq. 5, supervised by Stage 1 fidelity ), and assigns source positions \text{orig\\_idx}(\cdot) in Stage 2 (Eq. 7) with stochastic anchor dropout. Training an Anchorless variant ( anchors in Stage 2 and inference; Table A3 / Table 4) reaches 25.25% / 25.05% / 24.85% LBv2 and 19.80% / 19.51% / 18.93% GovReport (87%–99% of CMPD), confirming the 2D grid carries core document semantics; adding Query-Centroid whole-word anchors consistently lifts LBv2 by +1.79 / +3.18 / +3.58 points (27.04% / 28.23% / 28.43%) and GovReport by +0.13 / +0.86 / +1.80 ROUGE-1 as grows (whereas unconditioned learned-gate anchors collapse GovReport to 0.43% on 9B, Appendix H Table 15).
- Stage 1 Removal Ablation (Section 3.3.5, Table 4): Completed in Table A3 / Table 4: No Stage 1 achieves 26.04% / 26.04% / 26.64% LBv2 and 18.86% / 19.35% / 19.30% GovReport, while Stage 1 Only scores 21.87%–22.66% LBv2 and 0.84%–1.68% GovReport—showing Stage 1 provides a modest +1.0 to +2.2 point initialization benefit while Stage 2 distillation is essential.
Q2. Overall Architecture Diagram (Figure 2) & Stage-to-Inference Connections
Added Figure 2 (Section 2), Algorithm 1, Eqs. 1–7, and Table 1, explicitly connecting coordinate grid initialization, Query-Centroid whole-word selection, single-pass batched FiLM cross-attention, Stage 1 ( on patchifier) vs. Stage 2 ( on patchifier, projector, and LoRA), and contiguous inference indexing .
Q3. End-to-End Runtime, Compression Overhead, Throughput, & Peak Memory
Detailed in Global Response Part 2 (Appendix B, Table 8). At , CMPD's single-layer compression + routing takes 4.3 ms (8k) to 20.7 ms (128k) (98.7x–235.6x faster than ComprExIT/ICAE/LLMLingua-2 compression overhead), achieving 50.9–68.7 ms Total TTFT (2.5x–73.5x faster than Full Context and 71.6x faster than LLMLingua-2 at 128k), 52.7 tok/s decoding, and 3.27 GB Peak VRAM (4.15x below Full Context).
Q4 & Q5. How 2D Grid Coordinates Operate Under Global Cross-Attention & 1D Comparisons
- Why Global Cross-Attention + 2D Query Geometry (Q4, Sections 2.1 & 2.3): Hard local window masks would block cross-segment synthesis (0.8B Multi-Doc QA: 28.73% CMPD vs. 24.12% Pure 1D and 16.98%–18.98% chunked ComprExIT/ICAE-8B; Table 11). As clarified in Section 2.3, 2D coordinates parameterize the query generator (Eq. 1) prior to cross-attention, factoring queries across a compact 2D grid () rather than stretching indices to along 1D, while slot diversity and ordering are reinforced by Stage 1 off-diagonal decorrelation (Eq. 8) and Stage 2 row-stride source positions (Eq. 7).
- Same-Backbone 2D vs. Equivalent 1D Compressors (Q5, Section 3.3.1, Table 3): On
Qwen3.5-0.8B(Table A1), holding fixed, Pure 2D lifts GovReport from Pure 1D's 4.28%–4.78% collapse to 18.20%–18.65% (+13.8 ROUGE-1) and edges Pure 1D on LBv2 (23.06%–23.66% vs. 22.47%–23.26%; Pure 1D leads short Reasoning-Sensitive choices 27.97% and 131k RULER FWE 81.33%–82.27%, while Hybrid 2D nearly doubles RULER SQuAD QA 8.80%–9.73% vs. 4.13%–5.00%). Combining () with the 2D grid (CMPD) beats static 1D Soft-Prompt slots on LBv2 (27.04%–28.43% vs. 26.24%–27.63%) and GovReport (19.93%–20.73% vs. 19.50%–19.57%) with zero-shot budget scaling.
Q6. Visual-Text Compression Studies (VisInContext, VIST, DeepSeek-OCR)
Added citations and discussion of VisInContext (Wang et al., NeurIPS 2024), VIST (Xing et al., NeurIPS 2025), and DeepSeek-OCR (Wei et al., 2025) in Section 5: unlike optical pipelines that render text into RGB images for a multi-layer ViT at fixed resolution, CMPD completes a dynamic latent grid directly from embeddings in 4.3–20.7 ms, trading off exact optical character fidelity on random strings.
Thank you for the authors' response and additional experiments. However, I still find the overall motivation and technical presentation difficult to follow, and the revisions do not sufficiently clarify the rationale behind the proposed design. Therefore, I maintain my original score.
Thank you for your thoughtful follow-up.
Prompted by your question on why a 2D representation outperforms its 1D counterpart, we elevated the geometric mechanism, non-prolonged multi-row relative distance, into Section 1 (Introduction) as our Key Observation and Figure 1, paired with Figure 3 (Section 2.1) and a new general / multi-row diagnostic suite (; Appendix K Tables 15–17):
1. Why 2D outperforms 1D ? (a) Non-prolonged multi-row distance (Section 1 & Appendix K) + (b) Native PDF-page representation (Figure 3).
- (a) Concatenating rows in 1D prolongs early-row distance to , whereas an 2D grid bounds spatial distance to : Suppose a compressed memory organizes slots across rows or chunks of width . Flattening or concatenating chunks into a 1D sequence pushes Row to relative distance from the query while Row sits adjacent to the query; in a 2D visual grid under VLM mRoPE, all rows share identical temporal and column distances to the query, shrinking inter-row positional separation by (up to at ). Across compressed GovReport rows (), fixed- aspect ratios (), and square grids up to (), 1D concatenation decays
First-Row / Last-Rowquery attention to0.74–0.77—and in chunk-concatenated 1DICAE/ComprExIT(1B&8B) across chunks (),Chunk-0 / Last-Chunkattention drops to0.29–0.64(draggingComprExIT-1Bmulti-depth retrieval from62.5%down to19.8%;Table 17). Conversely, 2D mRoPE keepsFirst-Row / Last-Rowattention flat at0.99–1.02(– lower cross-row variance;Table 15). On the compressor side (Eq. 1),Pure 1Dstretches sinusoidal indices across ( past ), causing41.1×top/bottom slot collapse on GovReport (Table 16), whereas factoring into a 2D grid keeps slot balance compact (1.4×–1.8×); switching only inference decoder positions to explicit 2D mRoPE zero-shot adds a further+0.68to+1.04ROUGE-1 as scales (). - (b) Visual alignment to the corresponding rendered PDF (
Figure 3,Section 2.1): Because VLMs are pre-trained with 2D mRoPE over document/image patch grids,Figure 3shows that CMPD's text-synthesized 2D grid matchesQwen3.5-0.8B's native ViT representation of the corresponding rendered PDF/page (0.783mean /0.829–0.833median cosine similarity;53.3–53.6ROUGE-1 /22.6–23.1Corpus BLEU output agreement) in4.3–20.7 mswithout pixel rendering, outperformingPure 1Dacross every generative metric.
2. Strict apples-to-apples comparisons & off-the-shelf ICAE/ComprExIT checkpoints.
- Strict same-backbone 1D vs. 2D (
Table 3,Qwen3.5-0.8B): Every row inTable 3shares the identicalQwen3.5-0.8Bbackbone, cross-attention bottleneck, Stage 1+2 data/steps, LoRA config, and Query-Centroid router—varying only query geometry (Eqs. 1–2). With and zero extra parameters, switching from Pure 1D to Pure 2D lifts GovReport ROUGE-1 from4.45 / 4.78 / 4.28%to18.58 / 18.65 / 18.20%and edges LBv2 at every budget (23.66 / 23.26 / 23.06%vs.23.26 / 22.66 / 22.47%). Likewise, CMPD (Hybrid 2D) uses fewer query parameters ( shared vector vs. static slots) yet beats same-backbone 1D Soft-Prompt in all 6/6 settings (27.04 / 28.23 / 28.43%vs.26.24 / 26.84 / 27.63%LBv2;19.93 / 20.37 / 20.73%vs.19.57 / 19.50 / 19.50%GovReport) and LLMLingua-2 in all 18/18 settings (Table 2). - Off-the-shelf
ICAE&ComprExITcheckpoints (Table 2): We did not trainICAEorComprExITon downstream tasks; we evaluated the publicly released SFT checkpoints (Jiang-nan/{icae,comprexit}-{1b,8b}-4x-sft, Ye et al., 2026) directly under the identicallm-evalharness. Consistent with the multi-row 1D distance prolongation above, concatenating more 1D chunks degradesICAE-8B(26.04%22.66%) andComprExIT-8B(25.25%23.46%) as grows, whereas CMPD's unified 2D grid improves or holds steady (27.04%28.43%on0.8B;34.59–34.79%on9B).
Thanks for actionable suggestions
We thank Reviewer wEXk for a thorough, high-expertise review examining modality motivation, continuous baselines, full-context and retrieval limits, anchor routing, Stage 1 removal, architectural presentation, 2D vs. 1D structure, end-to-end runtime, and visual-text literature.
Highlighted 2D Grid Design
Addressing your assessment that architectural presentation was a substantial issue rather than a minor writing concern, we overhauled Sections 1–3:
Sections 1–2.1): We reframed CMPD explicitly as Coordinate-Parameterized 2D Visual Grid Completion—reusing a VLM's resolution-agnostic G×G 2D coordinate domain (M=G2) to synthesize continuous patches directly from text embeddings in one cross-attention pass (Eqs. 1–3), rather than claiming natural images are semantically richer than text.Figure 2,Algorithm 1,Table 1): Added an end-to-end architecture diagram (Figure 2), unified inference pseudocode (Algorithm 1), explicit batched anchor-alignment and position-indexing equations (Eqs. 4–7), and a stage-wise gradient-flow map (Table 1), alongside concise finding-focused prose in Section 3.W1. Clarified Motivation for Visual-Space Alignment vs. Modality Gap (
Tables 3–5)Our revised narrative and three controlled experiments (Global Response Tables A1 & A3) directly clarify what visual-grid formulation and Stage 1 alignment do:
Section 3.3.1, Table 3): HoldingQwen3.5-0.8B, Stage 1+2 data, and s=0 fixed, 1D sinusoidal queries (Pure 1D) collapse on SCROLLS GovReport (4.28%–4.78%ROUGE-1), whereas factoring the same M slots into a G×G 2D sinusoidal grid (Pure 2D) restores GovReport to18.20%–18.65%(20.73%with Hybrid offset s).Section 3.3.5, Table 4): TrainingNo Stage 1(random-init patchifier trained directly in Stage 2) reaches26.04% / 26.04% / 26.64%on LBv2 and18.86% / 19.35% / 19.30%on GovReport, whereasStage 1 Only(frozen backbone without Stage 2 / LoRA) drops to21.87%–22.66%LBv2 and0.84%–1.68%GovReport. Thus, the 2D coordinate grid + Stage 2 task distillation drive downstream reading, while Stage 1 visual alignment acts as a helpful+1.00to+2.19LBv2 and+1.02to+1.43ROUGE-1 initialization (CMPD:27.04% / 28.23% / 28.43%LBv2;19.93% / 20.37% / 20.73%GovReport).PixelProse) vs. Rendered-Document (Wikipedia) Stage 1 (Section 3.3.4, Table 4&Table 5): ReplacingPixelProsewith 400,000 rendered Wikipedia page images paired with their exact text yields26.24% / 26.44% / 27.04%LBv2 and21.35% / 21.41% / 21.40%GovReport (+0.67to+1.42ROUGE-1 over PixelProse), while synthesized patches match ViT-encoded Wikipedia page tokens at0.783mean (0.829–0.833median) cosine similarity (Table 5), confirming insensitivity to natural-image vs. rendered-page alignment corpora.W2. Continuous Baselines,
Qwen3.5-9BGap (34.79%vs.55.20%), & RULER LimitsTables A1–A2,Tables 2–3): Added ICAE and ComprExIT (disclosingLlama-1B/8Bbase-model differences) plus same-backbone (Qwen3.5-0.8B) Pure 1D and 1D Soft-Prompt baselines.Qwen3.5-9B(34.79%vs.55.20%) &131kRULER (Section 3.3.8, Table 12): Explicitly highlighted in the Abstract, Introduction, Section 3.2, and Appendix J. While CMPD matches uncompressed Full Context on0.8B(28.43%vs.26.10%) andGemma-4(28.83%vs.28.43%LBv2;19.03%vs.25.30%GovReport) and beats9BLLMLingua-2(34.79%vs.25.84%), aggressive front-end compression loses fine-grained multi-hop cues that9Bexploits under full attention (compounded by32knon-thinkinglm-evalvs. the official256kthinking-mode55.20%score). Similarly, on full131,072-token RULER (0.8B), CMPD reaches68.00%–72.53%on FWE and8.80%–9.73%on SQuAD QA (7.54%–7.90%12-task mean), but random-string NIAH/VT stays near zero (0.00%–2.90%) because context-free passcodes lack cosine alignment with qc and are smoothed by softmax pooling.