- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes KCR-Transformer, which combines learnable channel masks with a regularizer based on feature-kernel complexity for structured Transformer compression. A Nyström approximation reduces the computational overhead. The method is theoretically well grounded and clearly motivated. However, its focus on conventional ViT compression feels dated, and the experiments do not sufficiently establish its effectiveness on today's larger models and more recent architectures.
Critical strengths
-
The theoretical motivation is clear. The paper connects feature-kernel complexity with generalization and derives a concrete regularization objective, providing a theoretical basis for the compression method.
-
The method combines representation regularization with structured channel selection. The selected channels can be assembled into compact dense matrices, providing a practical way to reduce inference cost.
Critical weaknesses
-
Experiments on larger and more recent models are missing. The main experiments use ViT, Swin and EfficientViT. These results are insufficient to establish whether the method extends to larger models and newer architectures beyond those studied. Experiments on a larger, recent VLM, such as Qwen3-VL-32B [1], are needed. The authors should also explain what changes to the channel masks and kernel-complexity estimator are required as model size increases.
-
Experiments on text-only Transformers are missing. Neither the kernel-complexity regularizer nor the channel-selection mechanism appears inherently specific to vision. The authors do not explain which parts of the design depend on visual inputs. If the method is general, it should be evaluated on a text-only LLM. Otherwise, the authors should explain why it cannot be adapted to that setting and justify the focus on vision.
-
The deployment evaluation remains limited. For a larger model such as Qwen3-VL-32B, please compare the uncompressed model, the proposed method and other pruning methods under the same conditions, reporting measured peak GPU memory and inference speed.
-
The visualizations do not cover larger, more recent models. The existing Grad-CAM, mask and t-SNE visualizations mainly concern conventional ViTs. Examples before and after VLM compression would help show which visual details are lost and how the generated answers change.
-
The pruning baselines are not strong enough. The main comparison methods date from 2023–2025. More recent VLM pruning methods, such as CoMP [2], should be included.
-
Recent related work and references are missing. The bibliography contains only two papers from 2025 and none from 2026. Recent foundation models and compression studies have not received sufficient discussion or experimental comparison.
-
An overview diagram is missing. A pipeline figure connecting kernel construction, complexity approximation, mask search, structural pruning and retraining would make the method easier to understand.
[1] Shuai Bai et al. Qwen3-VL Technical Report. 2025.
[2] Zimeng Wu, Yunhong Wang, Donghao Wang, and Jiaxin Chen. Collaborative Multi-Mode Pruning for Vision-Language Models. CVPR, 2026.
Questions for authors
-
Why focus on vision Transformers? I do not see a clear vision-specific design in the proposed method. Can it also be applied to a text-only Transformer?
-
The model choices and references are dated. Without experiments on recent large models, I am not convinced that the proposed method remains applicable to current architectures.