acceptodds
Under review as a conference paper at ICLR 2027

DiD It in 87 Minutes: A Label-Free Softmax-to-Linear Adaptation of Vision Transformers for Object Detection

Abstract

While linear attention is a compelling mechanism for high-resolution object detection due to its reduced cost for global token mixing, converting the Softmax-attention ViT backbone of a trained detector into a linear-attention one is not a trivial drop-in replacement. Directly swapping the attention operator leads to severe performance degradation, and generic label-free distillation, though effective for classification, often fails on detection tasks. We argue that the central challenge is detector-interface preservation: the converted backbone must reproduce the exact feature tensors expected by the fixed downstream detector, rather than merely imitating internal Softmax hidden states. To address this, we introduce Detector-Interface Distillation (DiD), a label-free conversion method that exclusively trains the linear-attention backbone by aligning detector-facing interface tensors with those of a frozen Softmax teacher. On DOTA-v1.5, DiD substantially outperforms established baselines and matches supervised, fully trained linear models. Adaptation completes in roughly 87 minutes on 4 GPUs, and the linearized backbone cuts inference latency by  62% and peak memory by  49%. We hope our findings offer the community a simple, label-free route to reusing trained Softmax detectors as efficient linear ones, and encourage interface-aware objectives in future architecture-conversion work.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Linear Vision Transformers, Linear Attention, Object Detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f47wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explains the problem of performance drop when we go from softmax attention to linear attention for more efficient computational cost and suggest usage of pre-trained network with softmax attention as teacher and have a student counterpart network with linear attention. During training only linear backbone of student is trainable and MSE loss is applied after FPN part as knowledge distillation objective to minimize the gap between teacher and student outputs after FPN.

In their experiment, they use ViT-S with RetinaNet and Faster R-CNN, ReLU and ELU-based linear attention, and DOTA-v1.5 as their datasets, with additional Pascal VOC results. Suggested approach outperforms the evaluated conversion baselines and approximately matches the corresponding supervised linear-attention models, although it remains below the original Softmax detectors.

Critical strengths
  1. The problem setting is practical and clearly defined. Reusing an existing detector without detection annotations or subsequent head fine-tuning

  2. Different placements of knowledge distillation are well studied to better demonstrate its impact.

  3. Consistent improvement for DOTA and Pascal voc datasets.

Critical weaknesses
  1. Lack of novelty. while the paper demonstrates performance gain for linear attention by new placement of knowledge distillation loss, no new component has been introduced (new distillation loss, extra mechanism to lower the gap between linear and softmax attention)

  2. Although the two detectors cover one-stage and two-stage detection, both rely on an FPN. Evaluation on a structurally different detector would strengthen the claimed generality of detector-interface preservation; otherwise, the conclusions should be limited to the tested ViT-S/FPN configurations.

  3. The headline efficiency claim is supported only by backbone measurements. Section 5.5 and Figure 3 explicitly measure backbone-only inference, while the abstract reports approximately 62% lower inference latency and 49% lower peak memory without that qualification. These measurements do not establish equivalent savings for the complete detector.

Questions for authors
  1. Does the central finding extend beyond ViT-S detectors with an FPN? Can you demonstrate that interface matching remains effective with a larger backbone or a structurally different detector, such as DETR/Deformable DETR? This would help establish whether the paper offers a general conversion principle.

  2. Does the supervision-placement advantage survive removing the apparent weighting confound? Table 2 reports 30.7 mAP under an apparently uniform reconstruction objective, whereas Table 5 reports 29.8 for uniform and 30.7 for early-heavy weighting. Can you clarify this discrepancy and provide a comparison that changes only the supervision location?

  3. What is the actual end-to-end accuracy–efficiency trade-off? since it seems the efficiency experiments only considers the backbone and not is not calculated end to end.

Reviewer_Vi2VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies label-free Softmax-to-linear attention conversion for trained object detectors. It proposes Detector-Interface Distillation (DiD), which matches teacher and student features after the neck/FPN while updating only the student backbone and keeping the downstream detector frozen. Experiments on DOTA-v1.5 show that DiD outperforms the tested label-free baselines and achieves performance comparable to supervised linear-attention training, while reducing backbone inference latency and memory usage.

Critical strengths
  1. Effective application to object detection. The paper addresses Softmax-to-linear attention conversion for trained object detectors. DiD consistently outperforms the tested label-free baselines across two detector architectures and two attention kernels, achieving performance comparable to supervised linear-attention training.

  2. Simple method with practical efficiency gains. DiD uses a straightforward feature-matching objective and updates only the backbone, leaving the neck and detection heads frozen. Conversion takes approximately 87 minutes on four GPUs, while the converted backbone reduces inference latency by about 62% and peak memory by about 49%.

Critical weaknesses
  1. Limited novelty beyond existing detector feature distillation. Prior methods such as DeFeat[1] and FGD[2] have already demonstrated the value of detection-specific feature supervision. The paper does not clearly establish what “detector-interface preservation” adds beyond applying this idea to Softmax-to-linear conversion with a frozen detector.

  2. Unclear practical motivation and incomplete baseline details. The method uses DOTA training images but ignores their available labels, without sufficiently explaining why this restriction is useful in practice. The paper also omits important training details for the Softmax teacher and the Full training baseline, particularly initialization and training schedules, making the comparison difficult to assess.

[1] Distilling Object Detectors via Decoupled Features

[2] Focal and Global Knowledge Distillation for Detectors

Questions for authors
  1. Since conversion uses in-domain DOTA training images, what practical scenario justifies ignoring their available labels?

  2. How was the Softmax detector trained? Was the backbone updated during training?

  3. If Full training copies Softmax weights before supervised training, how does it compare with training the linear-attention detector from scratch?

  4. Why does Full training consistently have lower mAP than DiD in Table 3? Can you provide a deeper analysis beyond the “comparable performance” interpretation in §5.3?

  5. Why does RetinaNet retain more accuracy after DiD conversion than Faster R-CNN? Is this difference related to the detection-head architecture?

(Please first convince me of the significance of the problem addressed in this paper before discussing its novelty and experimental results. Otherwise, I would find it difficult to justify raising my score.)

Reviewer_gCczOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the linearization of trained object detection models and proposes Detector-Interface Distillation (DiD). The method replaces Softmax attention in a ViT backbone with linear attention. With the original FPN and detection head frozen, it uses unlabeled images to align the FPN output features of the teacher and student, updating only the student backbone. The authors compare different distillation supervision locations and validate the method on DOTA-v1.5 and Pascal VOC, with additional training configuration ablations and efficiency analyses. The paper's core contribution is identifying and validating a distillation supervision location suitable for linearizing object detection models.

Critical strengths
  1. The authors conduct pioneering exploratory research on adapting object detection models to linear attention.

  2. The authors conduct a small-scale study with a complete experimental validation cycle, demonstrating an effective supervision location for linearization distillation. This provides a useful reference.

  3. The paper provides a complete set of internal ablation studies.

Critical weaknesses
  1. The motivation lacks a principled explanation. The paper identifies a failure mode when linearizing object detection models but does not explain it from the principles of linear attention.

  2. The novelty and contributions are limited.

    (1) The paper's distillation method is essentially feature distillation, which is widely accepted in the field. One of the paper's core contributions is identifying a distillation supervision location, which has already been explored in related work such as DeFeat and FGD. Meanwhile, although the paper adapts object detection models to linear attention, it does not introduce any fundamentally novel design specifically for object detection models. Its method is highly consistent with mainstream model linearization methods. The paper's novelty lies in an incremental contribution under specific restrictive conditions.

    (2) The paper studies the linearization of object detection models. However, object detection models are typically constructed by attaching a detection head to a pretrained backbone, and a large body of prior work has already studied linear backbones and the linearization of pretrained vision backbones. The paper does not discuss an efficiency comparison between using a pretrained linear backbone followed by detection fine-tuning and DiD.

    (3) No linear-attention student architecture is specifically designed to address the identified failure mode. The architecture still uses the oldest form of vanilla linear attention.

  3. The experimental coverage is insufficient.

    (1) The backbones evaluated in the paper are limited to the Small scale with 30M parameters. Experiments with larger Base and Large models are missing.

    (2) The object detection datasets used in the paper are relatively small, and the method has not been tested on the classic and challenging COCO benchmark.

  4. The paper does not specify the source of the teacher models it uses. Moreover, it only linearizes fine-tuned object detection models implemented by the authors themselves, without applying the method to existing powerful open-source pretrained object detection models, such as DETR-family models. Only applying it to such models would demonstrate the method's true practical value.

Overall, I think the paper provides comprehensive internal experiments and is well written. But its contributions do not meet the standard for acceptance at ICLR.

Questions for authors
  1. It is better to explain, from the principles of linear attention, the failure mode after linearizing object detection models and why the chosen distillation supervision location can alleviate this issue?

  2. Compared with existing research on feature distillation, FPN feature distillation, and the linearization of pretrained vision backbones, it is better to explain what specific methodological contributions or new insights can this paper provide?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.