- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Skeleton-Preserving Distillation (SPD), which converts a pretrained ViT into a linear-time vision model by inheriting its non-mixer components and replacing self-attention with a bidirectional Mamba2 mixer. The classification pipeline combines attention-map alignment, block-output alignment, and end-to-end logit distillation. The approach is subsequently extended to semantic segmentation using an inherited decoder. The main contribution is an effective recipe for reducing the data and additional training compute required for cross-architecture conversion.
Critical strengths
-
A practically valuable problem, addressed with a clean, reusable design. Many strong pretrained ViT checkpoints are publicly available, e.g., from ImageNet-21K and CLIP pretraining. Reusing this investment to obtain linear-time models without repeating pretraining is directly useful, particularly for high-resolution dense prediction, where attention dominates the inference cost. Inheriting all compatible non-mixer components and concentrating optimization on the replacement mixer is a sensible way to reduce the difficulty of cross-architecture transfer.
-
Strong teacher recovery in both classification and segmentation. Distilled from only 10% of ImageNet-1K, MambaFormer-L reaches 83.35% top-1, compared with 85.06% for its ViT-L teacher, and full data raises this to 84.76%. The segmentation results show that the recipe carries over to dense prediction: under single-scale sliding-window evaluation, the student stays within 0.10 / 0.49 mIoU of the teacher on ADE20K / Cityscapes. It also outperforms label-only training from the same initialization by 2.1 / 2.4 mIoU, which confirms the value of the teacher signal.
-
Solid and comprehensive experiments, including practical efficiency. The claims hold across two teacher families, two model sizes and two data budgets (10% / 100%), and the compute accounting is detailed and internally consistent. The efficiency argument goes beyond asymptotic complexity. Fused inference kernels, benchmarked against optimized attention backends, deliver a 1.55× throughput gain over ViT-B at 1200² on an H100. The paper also openly reports the small-batch slowdown and the extra high-resolution fine-tuning needed on Cityscapes. Finally, the ablations are informative. Removing the block-output alignment stage costs 12.59 pp top-1. Unfreezing the skeleton gains 0.32 pp at the cost of 25.4% more Stage-3 student training FLOPs, a useful practical trade-off. This rigor makes the results credible and reproducible, and it weighs strongly in my assessment.
Critical weaknesses
-
Missing positioning against, and comparison with, TransMamba (Chen et al., arXiv:2502.15130, 2025). TransMamba transfers a pretrained DeiT into vision Mamba students (Vim, PlainMamba, VMamba). It initializes all non-SSM parameters from the teacher, aligns features through a projection, distills the forward and backward scan branches separately, and reports reduced-data training. The paper neither cites nor compares against it, although it is a vision-specific precedent, beyond MOHAWK, for weight inheritance, explicit handling of the two scan directions, and data efficiency.
-
Incremental methodological novelty. The three-stage structure, non-mixer weight transfer and the option of freezing inherited components all have close precedents in MOHAWK, which the paper acknowledges. The contribution is therefore mainly an effective vision adaptation and its empirical characterization. However, the newly emphasized double-CLS initialization and Stage-2 attention-map regularizer lack direct ablations.
-
The efficiency claims require qualification. The 18.6× figure compares conversion compute with a reference teacher-training schedule and presupposes that the teacher already exists. The 5.5× figure is an analytic FLOP ratio, not a measured training-time speedup. It relies on a uniform 3F_stu + F_tea approximation that does not reflect the different computation graphs across stages.
Questions for authors
1.What are the individual contributions of the double-CLS design and the Stage-2 attention-map regularizer? In addition, since Stage 1 has a negligible effect on final accuracy (Table 4), does it improve the convergence speed or training stability of Stage 2?
2.Why does Stage 1 align a softmax-normalized surrogate rather than the exact masked SSD matrix? The exact matrix could be materialized at n ≤ 578, since the teacher’s attention maps are already formed at this size.
3.Please report the measured end-to-end segmentation latency and memory for both teacher and student.
4.To substantiate the data-efficiency claim, could the authors evaluate at least one competing method (e.g., ViT-Linearizer’s objective or TransMamba) on the same 10% subset?