acceptodds
Under review as a conference paper at ICLR 2027

SC-VTP: State-Consistent Visual Token Pruning via Deletion-Consistent Training

Abstract

Visual-token pruning requires selector training to reflect deployed computation. In hybrid decoders, attention masks leave discarded tokens active in convolutional history and recurrent memory. State-Consistent Visual Token Pruning (SC-VTP) coordinates key access, history advancement and memory decay/write with one differentiable retention gate. A compositional contract specifies when binary- gated execution recovers native deletion. With a frozen backbone and a 0.52M-parameter selector trained on 26,143 DocVQA questions, three paired seeds improve official-validation ANLS over attention-only training by 5.74 points on average. A separately trained Qwen3.8-27B pair improves ANLS by 10.38 points. Crossed soft/hard evaluation and trained component ablations connect execution fidelity to selector utility. The original seed reaches 93.67 ANLS versus 93.05 for native-deletion policy gradients and transfers to 84.18% TextVQA accuracy. Frozen scorers retain their attention-only advantage at 25% and 75% retention. On 32 hardware-test sources, full-input/SC-VTP geometric latency ratios are 1.16 and 1.04 for one-token and 32-token generation, including encoding and selection. SC-VTP connects deletion-consistent operators to measured pruning performance in hybrid decoders.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Token Pruning, Document Understanding, Hybrid Vision-Language Models, Differentiable Selection, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3fpMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the mismatch between training-time attention masking and deployment-time physical token deletion in hybrid vision-language decoders with recurrent states and convolutional history. The authors propose SC-VTP, which introduces a shared differentiable retention gate that simultaneously controls attention access, history updates, and recurrent memory transitions to make soft training better approximate hard deletion. Experiments on DocVQA and TextVQA with Qwen hybrid decoders show that SC-VTP improves over attention-only token pruning and achieves better performance retention under reduced visual token budgets.

Critical strengths
  1. The paper identifies a real systems-level issue in visual token pruning. Attention masking and physical deletion are often treated as equivalent, while this paper carefully analyzes how recurrent states and convolutional history can introduce additional discrepancies.

  2. The deletion-consistency formulation is technically well developed. The module-level contract and binary endpoint analysis provide a clearer formalization of what it means for a pruning operator to match native deletion.

Critical weaknesses
  1. The paper demonstrates that attention-only training produces a mismatch with hard deletion, but does not isolate how much of the final accuracy improvement comes from correcting this mismatch versus simply providing a stronger selector training signal. A comparison where the selector objective is identical but only the execution operator changes would better support the central claim.

  2. The theoretical contribution establishes a sufficient condition for binary endpoint equivalence, but the connection between this property and improved task performance is not rigorously established. Matching the deletion operator does not necessarily imply learning a better token ranking, and the paper does not provide evidence that execution fidelity is the limiting factor for pruning quality.

  3. The practical contribution appears narrower than the framing suggests. The proposed method requires modifying internal decoder operations, including recurrent memory and convolution history, which limits applicability to models where these components are accessible. It would be useful to evaluate whether a simpler approximation can obtain similar benefits or quantify the cases where full state coordination is necessary.

  4. The efficiency analysis does not fully support the motivation of token pruning as an acceleration technique. Table 3 shows only a modest latency improvement over full input (1.16× for one-token generation and 1.035× for 32-token generation), while introducing additional training complexity and model-specific execution changes. A stronger comparison with end-to-end optimized pruning systems would clarify the practical advantage.

  5. Minor issues:

  • The paper is unusually long in supplementary analysis, but some main-text space could be better used to discuss limitations and applicability beyond the evaluated hybrid family.
  • Several theoretical statements carefully include assumptions, but the main narrative occasionally sounds broader than what the experiments validate.
  • Figure 1 and Figure 2 are informative, although some diagrams contain many execution details that make the main intuition harder to identify quickly.
  • The paper reports extensive statistical analysis, but some conclusions are based on one trained pair even when additional seeds are discussed elsewhere.
Questions for authors

see weakness

Reviewer_kLbvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

SC-VTP trains a lightweight visual-token selector with a shared differentiable retention gate controlling attention access, convolutional history advancement, and recurrent memory decay and writes. At inference, the selected original visual features are physically gathered, with their logical positions preserved, and the native decoder runs on the compact sequence.

The paper defines a module-level deletion contract $G_m M_m(X,p)=M(G_mX,G_mp) $, and gives conditions under which this property composes across a hybrid decoder. Three paired Qwen3.5-9B training seeds improve DocVQA ANLS over attention-only selector training by an average of 5.74 points. A separately trained Qwen3.8-27B pair, TextVQA transfer, budget transfer, and soft-versus-hard execution diagnostics extend the evaluation.

Critical strengths
  1. A well-controlled primary comparison. The backbone, selector capacity, initialization, training data order, and deployment procedure are matched within each pair. Official-validation gains of 8.82, 4.75, and 3.65 ANLS across the three 9B seeds support the usefulness of changing the training operator (pp. 8–9 and 21).
  2. Mechanism-oriented experiments complement task scores. The trained 2×22\times2 history/memory ablation and fixed-scorer crossed soft/hard execution measurements investigate the specific mismatch the method is intended to address. Similar own-soft losses but substantially different hard-deletion losses are particularly informative (pp. 8–9 and Appendices H, J, and K).
  3. The paper reports meaningful negative results. Direct-deletion policy gradients outperform SC-VTP at 25%25\% retention, and lower-resolution input outperforms SC-VTP on the 27B checkpoint. These observations appropriately limit claims of universal superiority (pp. 7 and 43–46).
  4. Data accounting and transfer evaluation are substantial. The paper separates selector-training sources from evaluation sources, reports sensitivity to known historical evaluation exposure, and evaluates all 5,000 TextVQA questions without task-specific retraining.
Critical weaknesses
  1. The conceptual novelty should remain narrowly stated. Identity skipping, state retention, and masking-versus-removal mismatch have prior precedents. The distinctive contribution is the coordinated hybrid-decoder realization and its measured effect on selector learning; the compositional endpoint proof alone is not a new general theory of pruning.
  2. The finite-gate design is insufficiently explored. Binary endpoint consistency does not identify a good interpolation or guarantee useful gradients. The final gates remain soft, and the paper does not train alternative endpoint-consistent paths or evaluate temperature and annealing sensitivity. For example, ezge^{zg} and (1−z)+zeg(1-z)+ze^g agree at binary gates but differ for 0<z<10<z<1 (pp. 4–5 and 31–35).
  3. Practical efficiency gains are modest and implementation-dependent. The differentiable history implementation uses quadratic sequence storage. The measured full-input/SC-VTP end-to-end latency ratios are approximately 1.16×1.16\times for one output token and 1.035×1.035\times for 32 tokens on 32 sources, using an unfused GDN implementation. These results mainly support input-processing savings (pp. 42–43).
  4. Stronger comparisons and broader replication are still needed. The closest policy-gradient comparator uses one seed and different training costs. Several published baselines are adaptations, and the 27B study lacks policy-gradient and TextVQA results. Both pretrained checkpoints belong to the same hybrid family.
  5. Historical exposure is not fully resolved. Excluding known previously examined sources preserves the main direction, but does not create a fresh untouched evaluation set. The manuscript describes extensive archived checks; the supplied local material contains only the PDF, so those artifacts have not been independently verified here.
Questions for authors
  1. How much of the improvement depends on the chosen soft interpolation and temperature schedule? Can you compare trained exponential, convex-decay, and full-transition interpolation variants?
  2. Does the advantage over direct-deletion policy gradients persist across multiple seeds and at matched training time or compute? What happens when both methods are retrained specifically for 25%25\% retention?
  3. Can you demonstrate the effect on an independent hybrid or recurrent architecture, rather than a second checkpoint in the same family?
  4. What are the training-memory and end-to-end inference results with optimized native kernels, longer inputs, and different batch sizes?
  5. Can you provide accessible executable artifacts and a concise reproduction path for the decisive comparisons, while consolidating repeated appendix material?
Reviewer_tZCeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SC-VTP, a visual-token pruning method that coordinates attention access, convolutional history, and Gated DeltaNet (GDN) state updates during selector training. Under stated conditions, its binary gates correspond to physical token deletion. Experiments on Qwen3.x models show improvements over attention-only training, with additional ablations and evaluations across datasets and retention budgets.

Critical strengths

1.A practically relevant problem: The paper shows that attention masks alone cannot prevent discarded tokens from updating convolutional history and recurrent memory, clearly identifying the execution mismatch between training and physical deletion. 2.A comprehensive method design: SC-VTP uses a unified retention gate to control attention, convolutional history, and GDN state updates, making binary-gated execution equivalent to physical deletion under specific conditions. 3.Reasonably controlled main experiments: SC-VTP and the attention-only baseline use the same backbone, selector size, training data, and deployment budget. All three paired experiments on the 9B model report positive gains.

Critical weaknesses
  1. Limited model and task coverage: Task-level gains are evaluated only on the Qwen3.x hybrid architecture, and the datasets are limited to DocVQA and TextVQA. CPU-based synthetic tests on other mixers do not establish effectiveness on different models or tasks.
  2. On the 9B official validation set, SC-VTP exceeds PG by only 0.62 ANLS; its TextVQA gain is 0.51 percentage points and is an unadjusted secondary comparison. At 25% retention, SC-VTP trails PG by 4.13 ANLS, so its advantage under aggressive compression is not established.
  3. The paper uses an exponential-decay path but does not compare it in training against other soft interpolation paths with the same binary endpoints. An ablation is recommended.
  4. The method primarily applies state-skipping mechanisms to attention, convolution, and GDN. The paper should clarify what new properties or guarantees its deletion-consistency formulation provides beyond prior token-pruning and state-skipping methods.
  5. The paper mainly evaluates exponential decay, although it acknowledges that alternative paths with the same binary endpoints have different gradients. The gate-to-hard top-K L1 distances (0.135/0.252) also indicate a remaining soft-hard mismatch.
Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.