- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper explains the problem of performance drop when we go from softmax attention to linear attention for more efficient computational cost and suggest usage of pre-trained network with softmax attention as teacher and have a student counterpart network with linear attention. During training only linear backbone of student is trainable and MSE loss is applied after FPN part as knowledge distillation objective to minimize the gap between teacher and student outputs after FPN.
In their experiment, they use ViT-S with RetinaNet and Faster R-CNN, ReLU and ELU-based linear attention, and DOTA-v1.5 as their datasets, with additional Pascal VOC results. Suggested approach outperforms the evaluated conversion baselines and approximately matches the corresponding supervised linear-attention models, although it remains below the original Softmax detectors.
Critical strengths
-
The problem setting is practical and clearly defined. Reusing an existing detector without detection annotations or subsequent head fine-tuning
-
Different placements of knowledge distillation are well studied to better demonstrate its impact.
-
Consistent improvement for DOTA and Pascal voc datasets.
Critical weaknesses
-
Lack of novelty. while the paper demonstrates performance gain for linear attention by new placement of knowledge distillation loss, no new component has been introduced (new distillation loss, extra mechanism to lower the gap between linear and softmax attention)
-
Although the two detectors cover one-stage and two-stage detection, both rely on an FPN. Evaluation on a structurally different detector would strengthen the claimed generality of detector-interface preservation; otherwise, the conclusions should be limited to the tested ViT-S/FPN configurations.
-
The headline efficiency claim is supported only by backbone measurements. Section 5.5 and Figure 3 explicitly measure backbone-only inference, while the abstract reports approximately 62% lower inference latency and 49% lower peak memory without that qualification. These measurements do not establish equivalent savings for the complete detector.
Questions for authors
-
Does the central finding extend beyond ViT-S detectors with an FPN? Can you demonstrate that interface matching remains effective with a larger backbone or a structurally different detector, such as DETR/Deformable DETR? This would help establish whether the paper offers a general conversion principle.
-
Does the supervision-placement advantage survive removing the apparent weighting confound? Table 2 reports 30.7 mAP under an apparently uniform reconstruction objective, whereas Table 5 reports 29.8 for uniform and 30.7 for early-heavy weighting. Can you clarify this discrepancy and provide a comparison that changes only the supervision location?
-
What is the actual end-to-end accuracy–efficiency trade-off? since it seems the efficiency experiments only considers the backbone and not is not calculated end to end.