acceptodds
Under review as a conference paper at ICLR 2027

Compression of Vision Transformer by Reduction of Kernel Complexity

Abstract

Self-attention and transformer architectures have become foundational components in modern deep learning. In this work, we introduce Transformer with Kernel Complexity Reduction, or KCR-Transformer, a compact transformer block equipped with differentiable channel selection, guided by a novel and sharp theoretical generalization bound. To reduce the substantial computational cost of the MLP layers, the KCR-Transformer performs channel selection on the outputs of its self-attention layer. Furthermore, we provide a rigorous theoretical analysis establishing a tight generalization bound for networks equipped with KCR-Transformer blocks. Leveraging such strong theoretical results, the channel pruning by KCR-Transformer is conducted in a generalization-aware manner, ensuring that the resulting network retains a provably small generalization error. Our KCR-Transformer is compatible with many popular transformer networks, such as ViT and Swin, and it reduces the FLOPs of the vision transformers while maintaining or even improving the prediction accuracy. In the experiments, we replace all the transformer blocks in the vision transformers with KCR-Transformer blocks, leading to KCR-Transformer networks with different backbones. The resulting KCR-Transformers achieve superior performance on various computer vision tasks, even outperforming the original models with fewer FLOPs and parameters. The code of the KCR-Transformer is available at https://anonymous.4open.science/r/KCR-Transformer.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
KCR-Transformer, Kernel Complexity Reduction, Vision Transformer, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8r35OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KCR-Transformer, which combines learnable channel masks with a regularizer based on feature-kernel complexity for structured Transformer compression. A Nyström approximation reduces the computational overhead. The method is theoretically well grounded and clearly motivated. However, its focus on conventional ViT compression feels dated, and the experiments do not sufficiently establish its effectiveness on today's larger models and more recent architectures.

Critical strengths
  1. The theoretical motivation is clear. The paper connects feature-kernel complexity with generalization and derives a concrete regularization objective, providing a theoretical basis for the compression method.

  2. The method combines representation regularization with structured channel selection. The selected channels can be assembled into compact dense matrices, providing a practical way to reduce inference cost.

Critical weaknesses
  1. Experiments on larger and more recent models are missing. The main experiments use ViT, Swin and EfficientViT. These results are insufficient to establish whether the method extends to larger models and newer architectures beyond those studied. Experiments on a larger, recent VLM, such as Qwen3-VL-32B [1], are needed. The authors should also explain what changes to the channel masks and kernel-complexity estimator are required as model size increases.

  2. Experiments on text-only Transformers are missing. Neither the kernel-complexity regularizer nor the channel-selection mechanism appears inherently specific to vision. The authors do not explain which parts of the design depend on visual inputs. If the method is general, it should be evaluated on a text-only LLM. Otherwise, the authors should explain why it cannot be adapted to that setting and justify the focus on vision.

  3. The deployment evaluation remains limited. For a larger model such as Qwen3-VL-32B, please compare the uncompressed model, the proposed method and other pruning methods under the same conditions, reporting measured peak GPU memory and inference speed.

  4. The visualizations do not cover larger, more recent models. The existing Grad-CAM, mask and t-SNE visualizations mainly concern conventional ViTs. Examples before and after VLM compression would help show which visual details are lost and how the generated answers change.

  5. The pruning baselines are not strong enough. The main comparison methods date from 2023–2025. More recent VLM pruning methods, such as CoMP [2], should be included.

  6. Recent related work and references are missing. The bibliography contains only two papers from 2025 and none from 2026. Recent foundation models and compression studies have not received sufficient discussion or experimental comparison.

  7. An overview diagram is missing. A pipeline figure connecting kernel construction, complexity approximation, mask search, structural pruning and retraining would make the method easier to understand.

[1] Shuai Bai et al. Qwen3-VL Technical Report. 2025.

[2] Zimeng Wu, Yunhong Wang, Donghao Wang, and Jiaxin Chen. Collaborative Multi-Mode Pruning for Vision-Language Models. CVPR, 2026.

Questions for authors
  1. Why focus on vision Transformers? I do not see a clear vision-specific design in the proposed method. Can it also be applied to a text-only Transformer?

  2. The model choices and references are dated. Without experiments on recent large models, I am not convinced that the proposed method remains applicable to current architectures.

Reviewer_JYTWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KCR-Transformer, which compresses ViTs by pruning the input and output channels of the MLP with differentiable Gumbel-Softmax masks. Pruning is guided by a kernel complexity functional defined on the data-dependent feature kernel. Building on local Rademacher complexity, Theorem 3.1 gives two-sided generalization bounds in terms of the empirical loss and this complexity, and an approximate truncated nuclear norm computed by Nyström sampling serves as a trainable surrogate regularizer. Training runs in two stages, a search stage with Gumbel-Softmax and a FLOPs budget, and a retraining stage with the regularization term. Experiments cover ImageNet-1K classification on five backbones, COCO detection, ADE20K segmentation, and VQA, together with matched-sparsity and significance analyses in the appendix.

Critical strengths
  1. The experiments are comprehensive, covering five backbones and four tasks, with matched-sparsity comparisons and significance tests in the appendix.
  2. Theorem 3.1 connects channel pruning to a two-sided generalization bound based on the learned feature kernel, and Section 4.6 studies the tightness of these bounds.
Critical weaknesses
  1. The claim of generalization-aware pruning is overstated, since the algorithm optimizes a Nyström approximation of the truncated nuclear norm at a cross-validated rank, and none of these steps provably preserves the bound.
  2. The tightness study in Section 4.6 evaluates bounds computed with an approximate complexity A-KC rather than the exact KC, adding another layer of approximation to the evidence for tightness.
  3. The paper has no method overview figure, so the two-stage pipeline and the KCR-Transformer block can only be reconstructed from text and equations.
Questions for authors
  1. Please provide evidence that minimizing the Nyström-approximated truncated nuclear norm actually reduces the true kernel complexity.
  2. Please compare at matched sparsity against a generic low-rank regularizer such as a nuclear-norm penalty on the feature gram to isolate whether the gain is specific to the KC-based formulation.
  3. Please add a framework diagram of the two-stage training pipeline and the KCR-Transformer block.
Reviewer_iiZKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the KCR-Transformer: at the attention output of vision Transformers (e.g., ViT/Swin), it applies Gumbel-Softmax–based differentiable channel selection to reduce MLP computation, and uses generalization upper/lower bounds based on "kernel complexity (KC)" to guide pruning and retraining.

Critical strengths

none

Critical weaknesses

The accuracy gaps between different models in Table 1 are implausibly small. Even assuming your results come from real data, having a 2021 model and a 2025 model within 0.5 accuracy points means one of two things: either your choice of comparison targets is inappropriate, or the field is dead. Given that no code is included with the submission, I strongly suspect the other models the compare against were not adequately tuned, or that something even more serious is wrong.

The theory still relies on the old playbook of generalization upper/lower bounds, an analysis approach that has been overused to the point of abuse in the past two years. If you really want to connect this theory to your actual experimental results, you need at least some quantitative analysis — e.g., compute the bound under your setting for representative prior methods, show what accuracy upper bound this bound anchors, and then show where your method stands relative to that. Alternatively, design causal-inference-style experiments to demonstrate the correlation between the metric and accuracy. Simply comparing upward/downward trends proves nothing about a theoretical connection. Theorem 3.1 lacks any content connecting the definition of compression to your method. As a result, even if the subsequent theory is entirely correct, it cannot distinguish your method from other compression methods. Take DeepCompress-Swin-T as an example: under the theoretical framework, this method is indistinguishable from your proposed method.

Questions for authors

What are the sources of the baseline numbers and the reproduction protocol?

What is the essential difference from the classical local Rademacher complexity bound?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.