- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the mismatch between training-time attention masking and deployment-time physical token deletion in hybrid vision-language decoders with recurrent states and convolutional history. The authors propose SC-VTP, which introduces a shared differentiable retention gate that simultaneously controls attention access, history updates, and recurrent memory transitions to make soft training better approximate hard deletion. Experiments on DocVQA and TextVQA with Qwen hybrid decoders show that SC-VTP improves over attention-only token pruning and achieves better performance retention under reduced visual token budgets.
Critical strengths
-
The paper identifies a real systems-level issue in visual token pruning. Attention masking and physical deletion are often treated as equivalent, while this paper carefully analyzes how recurrent states and convolutional history can introduce additional discrepancies.
-
The deletion-consistency formulation is technically well developed. The module-level contract and binary endpoint analysis provide a clearer formalization of what it means for a pruning operator to match native deletion.
Critical weaknesses
-
The paper demonstrates that attention-only training produces a mismatch with hard deletion, but does not isolate how much of the final accuracy improvement comes from correcting this mismatch versus simply providing a stronger selector training signal. A comparison where the selector objective is identical but only the execution operator changes would better support the central claim.
-
The theoretical contribution establishes a sufficient condition for binary endpoint equivalence, but the connection between this property and improved task performance is not rigorously established. Matching the deletion operator does not necessarily imply learning a better token ranking, and the paper does not provide evidence that execution fidelity is the limiting factor for pruning quality.
-
The practical contribution appears narrower than the framing suggests. The proposed method requires modifying internal decoder operations, including recurrent memory and convolution history, which limits applicability to models where these components are accessible. It would be useful to evaluate whether a simpler approximation can obtain similar benefits or quantify the cases where full state coordination is necessary.
-
The efficiency analysis does not fully support the motivation of token pruning as an acceleration technique. Table 3 shows only a modest latency improvement over full input (1.16× for one-token generation and 1.035× for 32-token generation), while introducing additional training complexity and model-specific execution changes. A stronger comparison with end-to-end optimized pruning systems would clarify the practical advantage.
-
Minor issues:
- The paper is unusually long in supplementary analysis, but some main-text space could be better used to discuss limitations and applicability beyond the evaluated hybrid family.
- Several theoretical statements carefully include assumptions, but the main narrative occasionally sounds broader than what the experiments validate.
- Figure 1 and Figure 2 are informative, although some diagrams contain many execution details that make the main intuition harder to identify quickly.
- The paper reports extensive statistical analysis, but some conclusions are based on one trained pair even when additional seeds are discussed elsewhere.
Questions for authors
see weakness