acceptodds
Withdrawn from ICLR 2027

TokenGraft: Diagnosing and Improving Visual Token Compression via Independent Recovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual token compression reduces the visual sequence processed by vision–language models, but performance–budget curves leave one question unresolved: does a compressor use the available token slots effectively? At the same final token count, we compare two allocations of the available budget. One uses only compressor-produced tokens (native expansion), while the other combines fewer compressed tokens with selected pre-compression visual tokens (original-token recovery). Across multiple compressors, even simple recovery rules reveal a recoverable gap: this hybrid representation can outperform native expansion. We propose TokenGraft, a training-free, text-query-independent method that restores visually important original tokens from locations under-covered by the compressed representation. On Qwen2.5-VL, TokenGraft improves all 27 main grounding comparisons across three compressors, three retention rates, and RefCOCO, RefCOCO+, and RefCOCOg, with gains up to 33.57 percentage points. Fixed-total-budget sweeps further show how reallocating slots from compression to recovery improves accuracy. These gains extend to InternVL2.5, LLaVA-OneVision, and CV-Bench, with magnitudes that depend on the backbone, compressor, and task. Under our paired runtime protocol, recovery adds less than 1% full-request time across all three compressors. These results establish token allocation, alongside budget size, as a practical design choice for visual token compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Visual Token Compression, Token Recovery, Visual Grounding, Spatial Coverage

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ypsUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to allocate a fixed visual-token budget between compressed representations and recovered original tokens.

TokenGraft selects original tokens using visual importance and spatial distance from the compressed representation, combines their percentile ranks, and inserts the selected tokens with their original features and positional information. Recovery requires no training and does not depend on the text query.

The evaluation compares recovery with native compression at matched budgets across three compressors, three VLM backbones, referring-expression grounding, and CV-Bench.

Critical strengths
  1. The budget comparison is carefully controlled. Recovery and native expansion use the same final token count, and Nuwa also matches intermediate budgets. This makes the observed gains informative about representation choices under a fixed budget.

  2. Recovery has credible benefits beyond the weakest baseline. Nuwa improves by 1+ percentage points across the three Qwen grounding datasets at 12.5% base retention, with positive paired intervals. All nine nonzero InternVL fixed-budget allocations also improve significantly. These results support the usefulness of recovery in several settings.

  3. Conditioning recovery on the compressed representation has direct support. Removing base reference locations reduces VisionZip and PPE accuracy by 1.16 and 5.49 points in Table 10. This supports selecting tokens that complement the existing representation.

  4. The method is simple to apply and the evaluation is reasonably transparent. It requires no parameter updates, and the recovered set can be reused across questions about the same image. Image-clustered paired intervals, invalid-output handling, and the ordering control help assess the results reliably.

Critical weaknesses
  1. The contribution over existing recovery and selection methods is not sufficiently established. Recoverable Compression already restores original visual tokens. TopV includes uniform recovery after pruning and a recovery ablation under the same FLOPs-reduction target. SCOPE already combines saliency and coverage through greedy selection. These works are cited, but the distinction from their recovery and selection mechanisms needs more explanation. The paper also omits Reroute, which studies recoverable routing under matched budgets, although its layer-wise re-entry differs from TokenGraft's interface-level recovery. These connections make the novelty of budget allocation and the proposed selector less clear.

  2. Recovery and sequence ordering are not fully separated in the main evidence. Table 11 shows that sorting alone explains most of the approximately 20-point OV1/PPE improvement. TokenGraft adds only 0.47, 1.24, and 0.16 points over sorted expansion, with only the RefCOCO+ interval strictly above zero. The existing control is useful, but Qwen/PPE lacks the same comparison despite producing the largest main-study gains. This leaves the contribution of original-token recovery to those gains unresolved.

  3. The combined selector does not show a clear advantage over simpler recovery rules. In Table 9, TokenGraft is significantly worse than importance-only recovery for PPE, does not significantly outperform spatial-only recovery for VisionZip, and has no significant advantage over the three simple rules for Nuwa. A common rule without compressor-specific tuning is convenient, but the added value of the percentile product remains weakly supported.

  4. Transfer and runtime results place limits on the practical contribution. VisionZip and Nuwa have no significant gains in the OV1 sweeps, and only two PPE allocations have positive overall intervals in the CV-Bench fixed-budget sweep. The reported sub-1% request-time increment is supported for the measured workload, but Table 14 shows larger relative increases in time to first token and no end-to-end speedup over Dense for PPE or Nuwa. These results narrow the demonstrated benefit of a general recovery module.

Questions for authors
  1. How much of the Qwen/PPE improvement remains when native expansion uses the same original-anchor ordering as TokenGraft? This would directly affect my interpretation of the largest gains.

  2. What does the budget-allocation diagnosis establish beyond TopV's recovery ablation? A comparison with uniform recovery or a closely related saliency-coverage rule, using the same candidates, ordering, and final budget, would help establish the incremental contribution.

  3. Given Table 9, what evidence shows that the percentile-product selector adds value beyond a simpler recovery rule across compressors?

Reviewer_NhGoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how a fixed visual-token budget should be split in VLM token compression. At the same final count K+A, it compares letting the compressor produce all K+A tokens (“native expansion”) with keeping K compressed tokens and adding back A original pre-compression tokens (“recovery”). It reports that recovery often wins and calls the difference a “recoverable gap.” The proposed TokenGraft picks the A original tokens greedily, using the rank product of ViT attention importance and distance to the compressor’s current spatial references, then inserts them with their original features and positions. On Qwen2.5-VL with VisionZip, PPE and Nuwa, it improves all 27 RefCOCO/+/g grounding comparisons (up to +33.6 pp), extends to InternVL2.5, LLaVA-OneVision and CV-Bench with mixed gains, and adds under 1% request time.

Critical strengths
  1. The matched-budget comparison is a clean idea. Comparing allocations at the same final token count is a fairer test than a standard accuracy–budget curve.

  2. The statistics are careful. The paper uses paired, image-clustered bootstrap intervals, reports where intervals cross zero, and includes random and single-signal selector controls. The runtime protocol is also well designed.

Critical weaknesses
  1. The headline gains mostly come from a broken baseline. All the +18 to +34 pp gains are on PPE, which gets only 33% at r=25%, while Nuwa gets 81% at the same budget. Random recovery alone gives +22 pp. On LLaVA-OneVision, the authors show that simply re-sorting PPE’s output by original position recovers about +20 pp, and TokenGraft adds only 0.2–1.2 pp on top. This sorting control is not run for the main Qwen experiments, so the large Qwen gains may also be an ordering artifact. InternVL+VisionZip at 36.7% looks similarly suspect. On the compressors that work normally (VisionZip and Nuwa on Qwen), gains are 0.1–3 pp and vanish at r=37.5%. On LLaVA-OneVision they are not significant.

  2. Limited novelty and a missing key comparison. Combining saliency with spatial coverage is already done in SCOPE, SCoRe, CoIn and DivPrune, and recovering dropped tokens is done in Recoverable Compression. The selector controls also do not support the combined rule: spatial-only ties TokenGraft on VisionZip and Nuwa, and importance-only beats it on PPE. Most importantly, the “recoverable gap” only shows that some compressors are suboptimal. The paper never compares against a strong single-method selector (e.g., a saliency + coverage method choosing all K+A tokens). Without that, it is unclear whether hybrid allocation itself matters. Evaluation is also mostly RefCOCO grounding, with no standard VQA benchmarks, and the CV-Bench gains are small.

Questions for authors

Please run the “Expanded + sorting” control for PPE on Qwen2.5-VL, and check VisionZip on InternVL. How much of the gain remains after fixing token order?

How does TokenGraft compare with a saliency + coverage selector (e.g., SCOPE or DivPrune) that picks all K+A tokens directly from the original sequence?

Why does the combined rank product not beat the single signals in Table 9? Is there any setting where the combination is clearly needed?

Can you report results on standard VQA benchmarks (e.g., GQA, TextVQA, MME, POPE), where most compression methods are usually evaluated?

Reviewer_Ubn7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates budget allocation in visual token compression for vision-language models (VLMs) via a matched-budget diagnostic. At a fixed total budget of K+AK+A visual tokens, the authors compare "native expansion" (generating all K+AK+A tokens via the compressor) against "original-token recovery" (combining KK compressed tokens with AA restored pre-compression tokens), observing a "recoverable gap" where hybrid representations outperform native expansion. Based on this observation, the authors propose TokenGraft, a training-free, text-query-independent recovery module that iteratively selects AA original tokens by maximizing the product of empirical cumulative ranks of vision-encoder attention importance (sis_i) and 2D Euclidean distance to the nearest compressed reference location (di(t)d_i^{(t)}), before interleaving the recovered tokens with the compressed sequence using original sequence index anchors. Experiments are conducted primarily on Qwen2.5-VL-7B, InternVL2.5-8B, and LLaVA-OneVision-7B across RefCOCO/RefCOCO+/RefCOCOg grounding benchmarks and CV-Bench using VisionZip, PPE, and Nuwa as base compressors.

Critical strengths
  1. Clean Diagnostic Formulation and Rigorous Statistical Reporting: Framing token compression as a slot-allocation problem between K+AK+A native compressed tokens and KK compressed plus AA original tokens cleanly isolates representation quality from budget size. The use of 10,000 image-clustered paired bootstrap confidence intervals across evaluations is commendable.
  2. Tuning-Free Cross-Compressor Integration: By combining visual importance and spatial complementarity via parameter-free empirical CDF rank products and native positional anchors, TokenGraft integrates across heterogeneous compressors (VisionZip, PPE, Nuwa) and VLM backbones without training.
Critical weaknesses
  1. Questionable Practical Rationale ("Over-Compressing then Recovering") and Narrow Task Scope:

    • Under a fixed target budget B=K+AB = K + A, intentionally over-compressing to K=B−AK = B - A tokens and running an O(AM2)\mathcal{O}(AM^2) post-hoc recovery loop to restore AA raw tokens is a convoluted workaround rather than a fundamental solution. When hybrid recovery beats native CK+A(X)\mathcal{C}_{K+A}(X), it merely diagnoses a design flaw in the base compressor (e.g., token merging blurring spatial coordinates needed for box regression, or clustering scrambling sequence order). Rather than applying a two-stage patch with a task- and compressor-sensitive hyperparameter AA, it is far more direct to preserve ordering and spatial coverage within a single-stage K+AK+A compressor. Moreover, for drop-only pruners that already optimize saliency and coverage on raw tokens (e.g., SCOPE, SCoRe, TopV), compressing to KK and recovering AA via importance and spatial distance is conceptually redundant.
    • Evaluations are heavily biased toward bounding-box grounding (RefCOCO/+/g), where restoring unmerged patch coordinates naturally helps box regression. On multiple-choice VQA (CV-Bench), gains vanish on VisionZip and Nuwa (-0.32 to +0.28 pp, all 95% CIs crossing zero), and standard VLM benchmarks (e.g., MME, MMBench, GQA, POPE, TextVQA) are entirely omitted.
  2. Headline Gains (+33.57 pp) Stem from PPE's Sequence-Ordering Bug, While Gains on Properly Ordered Compressors Are Marginal:

    • The advertised +33.57 pp gain occurs exclusively on PPE, whose official DPC clustering scrambles token order. Table 11 reveals that simply sorting PPE's native K+AK+A output by original sequence anchors (Expanded + sorting) without recovering any tokens jumps RefCOCO accuracy from 22.14% to 42.40% (+20.26 pp), leaving TokenGraft with a statistically insignificant +0.47 pp gain (95% CI [-0.51, +1.46]) on RefCOCO and +0.16 pp (95% CI [-0.54, +0.86]) on RefCOCOg. Yet this critical Expanded + sorting control is omitted for Qwen2.5-VL/PPE in Tables 1–2.
    • On properly ordered compressors (VisionZip and Nuwa), gains are slight or statistically insignificant: +0.09 to +0.36 pp at r=37.5%r=37.5\% on Qwen2.5-VL (Table 5, all CIs crossing zero), and -0.14 to +0.65 pp across all settings on LLaVA-OneVision (Tables 6 and 8, all CIs crossing zero).
  3. Rank-Product Selector Fails to Beat Single-Signal Baselines:

    • In Tables 1 and 9 (RefCOCO, r=25%,A=16r=25\%, A=16), the proposed rank-product selector fails to statistically significantly beat both single-signal baselines on any compressor: it underperforms Importance-only on PPE (-1.38 pp, CI [-2.59, -0.16]), underperforms Spatial-only on Nuwa (-0.29 pp) while barely differing from Random recovery (+0.19 pp, CI [-0.37, +0.76]), and only marginally exceeds Spatial-only on VisionZip (+0.31 pp, CI [-0.53, +1.12]).
  4. Misleading Overhead Metric and Slower/Heavier Pipeline Than Uncompressed Inference:

    • Claiming "<1% overhead" (Table 4) divides prefill recovery latency by 48-token decoding time (~2.2–2.6 s); measured on prefill latency (TTFT, Table 14), TokenGraft adds +9.1% to +10.8% overhead on VisionZip and PPE. Worse, compared to the uncompressed Dense model in Table 14, PPE + TokenGraft and Nuwa + TokenGraft run 17.69% and 2.28% slower in full-request time, and all three TokenGraft pipelines allocate 219–287 MiB MORE peak GPU memory than uncompressed inference (with PPE reconstructing full visual attention, adding +301 MiB).
Questions for authors
  1. Rationale of Over-Compressing then Recovering: Under a fixed budget K+AK+A, why not directly incorporate order preservation and spatial coverage into a single-stage K+AK+A compressor rather than over-compressing to KK and recovering AA tokens? How does TokenGraft perform on drop-only coverage-aware pruners (e.g., SCOPE, SCoRe) and general VQA benchmarks (e.g., MME, MMBench, GQA, TextVQA)?
  2. Controls and Selector Justification: What is the accuracy of Expanded + sorting for PPE on Qwen2.5-VL and InternVL2.5, and how do the authors justify the rank-product selector when it fails to statistically significantly beat single-signal baselines (Importance-only on PPE and Spatial-only on Nuwa/VisionZip) in Table 9?
  3. Efficiency vs. Uncompressed Model: In Table 14, why do the TokenGraft pipelines allocate more peak GPU memory and run slower (for PPE and Nuwa) than the uncompressed Dense baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.