acceptodds
Withdrawn from ICLR 2027

SPD: Skeleton-Preserving Distillation from Quadratic to Linear-Time Vision Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The accuracy of Vision Transformers relies on large-scale pretraining with a quadratic-cost attention mixer. Subquadratic vision backbones avoid this cost at inference but are usually pretrained from scratch. We propose Skeleton-Preserving Distillation (SPD), which converts a pretrained ViT into a linear-time model by reusing its pretrained weights: the student inherits every component of the teacher except the quadratic mixer. We apply SPD to MambaFormer, our ViT-based backbone with a bidirectional Mamba2 mixer in place of attention, distilled from ViT teachers whose classifier path it can inherit. Using only 10% of ImageNet-1K, MambaFormer-L recovers 98% of its ViT-L teacher's accuracy at 18.6× less compute than re-training the teacher, and 99.7% with full data. With a CLIP ViT-B/16 teacher, MambaFormer-B reaches 84.25 top-1 vs. 84.30 for ViT-Linearizer at 5.5× less compute, and reaches 82.26 from only 10% of the data at 20.6× less. Trained from scratch on labels with 3.0× more compute (teacher pretraining excluded), the same model reaches 67.75. SPD also applies to semantic segmentation, where the student stays within 0.5 mIoU of the teacher on ADE20K and Cityscapes (single-scale evaluation), beats a no-teacher baseline by 2.1–2.4 mIoU, and needs 1.4× fewer inference FLOPs on Cityscapes. With fused scan kernels, MambaFormer-B matches ViT-B's throughput at 224² (batch 32) and is 1.55× faster at 1200² on an H100.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
knowledge distillation, state space models, Mamba, vision transformers, efficient inference, semantic segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JNedOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Skeleton-Preserving Distillation (SPD), which converts a pretrained ViT into a linear-time vision model by inheriting its non-mixer components and replacing self-attention with a bidirectional Mamba2 mixer. The classification pipeline combines attention-map alignment, block-output alignment, and end-to-end logit distillation. The approach is subsequently extended to semantic segmentation using an inherited decoder. The main contribution is an effective recipe for reducing the data and additional training compute required for cross-architecture conversion.

Critical strengths
  1. A practically valuable problem, addressed with a clean, reusable design. Many strong pretrained ViT checkpoints are publicly available, e.g., from ImageNet-21K and CLIP pretraining. Reusing this investment to obtain linear-time models without repeating pretraining is directly useful, particularly for high-resolution dense prediction, where attention dominates the inference cost. Inheriting all compatible non-mixer components and concentrating optimization on the replacement mixer is a sensible way to reduce the difficulty of cross-architecture transfer.

  2. Strong teacher recovery in both classification and segmentation. Distilled from only 10% of ImageNet-1K, MambaFormer-L reaches 83.35% top-1, compared with 85.06% for its ViT-L teacher, and full data raises this to 84.76%. The segmentation results show that the recipe carries over to dense prediction: under single-scale sliding-window evaluation, the student stays within 0.10 / 0.49 mIoU of the teacher on ADE20K / Cityscapes. It also outperforms label-only training from the same initialization by 2.1 / 2.4 mIoU, which confirms the value of the teacher signal.

  3. Solid and comprehensive experiments, including practical efficiency. The claims hold across two teacher families, two model sizes and two data budgets (10% / 100%), and the compute accounting is detailed and internally consistent. The efficiency argument goes beyond asymptotic complexity. Fused inference kernels, benchmarked against optimized attention backends, deliver a 1.55× throughput gain over ViT-B at 1200² on an H100. The paper also openly reports the small-batch slowdown and the extra high-resolution fine-tuning needed on Cityscapes. Finally, the ablations are informative. Removing the block-output alignment stage costs 12.59 pp top-1. Unfreezing the skeleton gains 0.32 pp at the cost of 25.4% more Stage-3 student training FLOPs, a useful practical trade-off. This rigor makes the results credible and reproducible, and it weighs strongly in my assessment.

Critical weaknesses
  1. Missing positioning against, and comparison with, TransMamba (Chen et al., arXiv:2502.15130, 2025). TransMamba transfers a pretrained DeiT into vision Mamba students (Vim, PlainMamba, VMamba). It initializes all non-SSM parameters from the teacher, aligns features through a projection, distills the forward and backward scan branches separately, and reports reduced-data training. The paper neither cites nor compares against it, although it is a vision-specific precedent, beyond MOHAWK, for weight inheritance, explicit handling of the two scan directions, and data efficiency.

  2. Incremental methodological novelty. The three-stage structure, non-mixer weight transfer and the option of freezing inherited components all have close precedents in MOHAWK, which the paper acknowledges. The contribution is therefore mainly an effective vision adaptation and its empirical characterization. However, the newly emphasized double-CLS initialization and Stage-2 attention-map regularizer lack direct ablations.

  3. The efficiency claims require qualification. The 18.6× figure compares conversion compute with a reference teacher-training schedule and presupposes that the teacher already exists. The 5.5× figure is an analytic FLOP ratio, not a measured training-time speedup. It relies on a uniform 3F_stu + F_tea approximation that does not reflect the different computation graphs across stages.

Questions for authors

1.What are the individual contributions of the double-CLS design and the Stage-2 attention-map regularizer? In addition, since Stage 1 has a negligible effect on final accuracy (Table 4), does it improve the convergence speed or training stability of Stage 2?

2.Why does Stage 1 align a softmax-normalized surrogate rather than the exact masked SSD matrix? The exact matrix could be materialized at n ≤ 578, since the teacher’s attention maps are already formed at this size.

3.Please report the measured end-to-end segmentation latency and memory for both teacher and student.

4.To substantiate the data-efficiency claim, could the authors evaluate at least one competing method (e.g., ViT-Linearizer’s objective or TransMamba) on the same 10% subset?

Reviewer_Wy1POpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a method that takes a trained ViT and replaces its self-attention layer with two Mamba layers to "mix" spatial information subquadratically. The model is trained via distillation in 3 stages: (1) attention alignment, (2) output alignment, and (3) mamba, CLS, fusion layer, and classification-head unfreezing. As a result of these 3 stages, the students outperform the teachers at substantially lower computational cost and the ViT-Linearizer baseline. The paper also shows results on segmentation. Through ablations the paper shows that stages 2 and 3 are most important.

Critical strengths

none

Critical weaknesses

The paper proposes no new architecture. Its contribution is a 3-stage distillation recipe that is essentially MOHAWK applied to vision, which is not disqualifying by itself. However, the component that gives the method its name, freezing the inherited skeleton, appears to contribute little: unfreezing it in Stage 3 improves top-1 by 0.32% while costing only 12% more Stage-3 wall-clock (6% at pipeline level). Figure 1 plots teachers with their full pre-training compute but students with distillation compute only, so placing the students “to the left of” their teachers overstates the efficiency gain.

As far as I can tell, there is an obvious baseline missing, which is to initialize the parameters, then simply train all parameters end to end with the KL divergence between student and teacher logits. I would also expect a LR sweep (and maybe even different LRs for the pre-trained vs randomly initialized modules).

Questions for authors

none

Reviewer_L4nDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Skeleton-Preserving Distillation(SPD), a method for converting pretrained ViTs into linear-time vision backbones while preserving their performance and reducing training costs. The proposed MambaFormer replaces self-attention with a bidirectional Mamba2 mixer while inheriting the remaining pretrained components from the teacher. SPD adopts a three-stage distillation pipeline that progressively aligns attention maps, attention outputs, and final logits. Experiments on classification and semantic segmentation show that the proposed method achieves performance close to the ViT teachers with substantially lower training costs and improved inference efficiency at high resolutions.

Critical strengths
  1. This work addresses the practical problem of efficiently converting pretrained ViTs into linear-time vision models. The proposed method is simple and effective.
  2. The experiments are fairly comprehensive, demonstrating the effectiveness of the method across different model sizes, image classification, and semantic segmentation tasks, with detailed efficiency analysis and ablation studies.
Critical weaknesses
  1. The proposed distillation framework is quite similar to MOHAWK, which limits its novelty. Moreover, the paper lacks ablation studies on key new components, such as Double-CLS and Attention Map Regularization.
  2. The comparison with ViT-Linearizer is not sufficiently fair. The two methods use teachers with different performance levels, making it difficult to assess the actual advantages of SPD over existing methods.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.