acceptodds
Withdrawn from ICLR 2027

FSD: Bridging Foundation Semantics and Local Details for Real-Time Object Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-time object detection has advanced rapidly with CNN- and Transformer-based architectures. Recently, Vision Foundation Models (VFMs) have been introduced to further enhance detectors with rich pretrained semantics. Where partially overlapping semantic modeling may lead to redundant representations and limit further improvements. In fact, the structural complementarity between the VFM and detector remains largely overlooked in existing VFM-enhanced detection research. Motivated by this, we propose FSD, a VFM-enhanced real-time detector that exploits complementary semantic and spatial representations. Specifically, FSD couples a frozen EuPE semantic branch with a trainable CNN spatial pathway to combine global semantics with fine-grained spatial details. To bridge their heterogeneous representations, we further propose Multi-level Semantic-Detail Interaction (MSDI) for adaptive multi-level interaction. Finally, we design an Efficient Auxiliary Head (EAH) to promote detection-oriented adaptation of semantic features and better balance semantic and spatial-detail cues. Extensive experiments on COCO demonstrate that FSD achieves state-of-the-art performance with a favorable accuracy and efficiency trade-off. Further analyses highlight the importance of representational complementarity between VFMs and detectors.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Real-time Object Detection, Vision Foundation Models, Semantic-Detail Interaction, Multi-scale Feature Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BC2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FSD, a VFM-enhanced real-time object detector. It uses a dual-branch backbone: a frozen VFM semantic branch for high-level semantics, and a trainable CNN spatial branch for fine-grained spatial details such as edges, textures, and localization cues. The claim is that VFM-enhanced detection benefits not only from stronger pretrained features, but also from the representational complementarity between VFM semantics and CNN spatial representations.

Critical strengths
  1. Strong accuracy–efficiency trade-off FSD achieves state-of-the-art COCO AP across multiple model scales while maintaining fast inference speed. The lightweight FSD-N and FSD-S are particularly strong in balancing accuracy, trainable parameters, and latency.

  2. Comprehensive experimental analysis The paper includes ablations on key components, fusion strategy comparisons, VFM generality tests, auxiliary loss weight analysis, cross-device latency measurements, and CKA-based representation similarity analysis. The experimental analysis are solid.

Critical weaknesses
  1. Limited technical novelty, mainly engineering integration The method mainly combines existing ideas: frozen VFM + CNN detector, multi-level concatenation, C3k blocks for alignment, and an auxiliary detection head. MSDI and EAH are not fundamentally new mechanisms. The work feels more like careful system design and engineering optimization than a deep methodological breakthrough.

  2. Limited experimental scope The main experiments are on COCO. Robustness under domain shift, long-tailed distributions, open-world settings, and specialized visual domains is not thoroughly explored.

Questions for authors
  1. The “complementarity” insight is intuitive but not deeply theorized The CKA analysis shows a correlation between lower representation similarity and larger gains, but it does not establish causality or provide a predictive principle. The paper does not deeply answer when, why, or how VFM-detector complementarity arises. It would be better if author provide more experimental results to explore it.
  2. Does the concat + C3k soft alignment in MSDI truly necessary? Could a simpler, more principled, or more lightweight alignment mechanism achieve similar performance? Maybe the author could provide more method comparisons.
  3. Can the auxiliary loss be made adaptive rather than fixed at 0.5. Can it improve performance?
1 comment
AuthorsOpenReview ↗

Point-by-Point Rebuttal to Reviewer BC2s and Revisions

Weaknesses

W1. Technical novelty. While concatenation, C3k, and auxiliary supervision are established operations, FSD's central contribution is a complementarity-driven perspective on VFM-enhanced detection: rather than solely introducing stronger pretrained semantics, we explicitly exploit the distinct strengths of VFM semantics and CNN spatial details. FSD pairs frozen VFM semantics with trainable CNN details; MSDI aligns and propagates features across scales, while EAH guides semantic adaptation. New branch-wise, cross-architecture, and capacity-matched analyses support this motivation (Q1–Q2). See Secs. 1, 3, and 4 of the revised manuscript.

W2. Experimental scope. We further evaluate NEU-DET (varying defect scales and subtle textures) and GC10 (class imbalance and small defects), two industrial benchmarks beyond natural-image detection. FSD achieves 79.4/47.3 AP50/AP50:95 on NEU-DET and 76.2/39.7 on GC10, outperforming all compared CNN/DETR detectors. Against the strongest baseline, YOLO11-S, gains are +2.3/+1.7 and +1.1/+2.2 AP50/AP50:95, respectively. These results support effectiveness in specialized domains.

MethodTypeNEU AP50NEU AP50:95GC10 AP50GC10 AP50:95
Faster R-CNNCNN59.136.851.026.5
Cascade R-CNNCNN61.737.453.727.4
RetinaNet-R50CNN56.335.152.420.2
GFLCNN64.437.545.721.0
TOOD-R50CNN56.336.148.222.0
RTMDetCNN64.536.754.026.7
YOLO11-SCNN77.145.675.137.5
YOLOv12-SCNN76.944.874.937.3
RT-DETR-LDETR68.141.966.230.7
DINO-4scaleDETR60.333.447.220.2
DEIMDETR65.342.768.733.8
FSD (Ours)Hybrid79.447.376.239.7

See the revised Appendix for experimental details.

Questions for Authors

Q1. Understanding complementarity. We agree that CKA measures similarity, not causality. We added branch-wise and cross-architecture analyses to examine complementarity. First, CNN-only has higher small-object AP than EuPE-only (28.7 vs. 24.7), whereas EuPE-only performs better on medium/large objects (56.2/73.5 vs. 51.3/64.1). Integrating both yields 54.9 AP, improving APs by 10.4 and AP75 by 6.9 over EuPE-only. These results support complementary detection strengths

CNNEuPEAPAP50AP75APsAPmAPl
✓-46.663.450.328.751.364.1
-✓48.869.552.324.756.273.5
✓✓54.973.459.235.160.974.7

Second, cross-architecture comparisons and CKA show that CNN detectors exhibit lower similarity to VFM features and are associated with larger gains than Transformer detectors. Additional EuPE improves DEIMv2-S from 50.9 to 52.7 (+1.8 AP), whereas the full FSD-S improves YOLO11-S from 46.6 to 54.9 (+8.3 AP). Though architectures and integration differ, the results support our hypothesis.

DetectorConfigurationAPΔAP
YOLO11-SBaseline46.6—
FSD-SYOLO11-S + EuPE + MSDI + EAH54.9+8.3
DEIMv2-SBaseline50.9—
DEIMv2-S + EuPEAdditional EuPE integration52.7+1.8

We hope these results clarify complementarity. See Secs. 4.3–4.4 of the revised manuscript

Q2. Alignment alternatives. Using the same EuPE levels (E3–E5), we compare Add, Concat, and capacity-matched High-channel Concat (HC-Concat) with MSDI. Add/Concat reach 52.8/52.9 AP, whereas MSDI reaches 53.3 AP. HC-Concat reaches 52.6 AP with 5.378M trainable parameters versus MSDI's 5.425M, indicating gains beyond increased fusion capacity. Removing Cross-scale Complementary Feature Propagation (CFP) reduces AP to 48.9 (−4.4), underscoring hierarchical interaction.

StrategyInteractionParams (M)APAP50AP75ΔAP
Feature FusionAdd4.91352.871.157.0-0.5
Feature FusionConcat5.12652.971.357.3-0.4
Feature FusionHC-Concat5.37852.671.056.9-0.7
Feature FusionMSDI5.42553.372.157.8—
Cross-scaleMSDI w/o CFP4.74448.969.754.0-4.4
Cross-scaleMSDI5.42553.372.157.8—

These comparisons support our design choices. See Sec. 4.3 of the revised manuscript.

Q3. Adaptive auxiliary weight. We initially tried making λaux\lambda_{\mathrm{aux}} learnable during training; FSD-N achieved 53.2 AP, below 53.3 AP with fixed λaux=0.5\lambda_{\mathrm{aux}}=0.5. As this task-level weight is easily tuned, we select fixed 0.5 for simplicity after sensitivity analysis

Due to the character limit, our full point-by-point responses and additional experimental details are provided in the supplementary material

Reviewer_RvDPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FSD combines a frozen EuPE vision foundation model with a trainable CNN for real-time object detection. Multi-level Semantic-Detail Interaction (MSDI) fuses features from the two branches, while an auxiliary detection head (EAH) supervises the semantic features during training. On COCO, the authors report 53.3 AP for FSD-N and 59.3 AP for FSD-X, with respective inference latencies of 2.34 and 5.21 ms on an RTX 4090.

Critical strengths
  1. FSD achieves competitive accuracy at the reported latency. In Table 1, FSD-X reaches 59.3 AP at 5.21 ms, compared with 57.8 AP at 5.22 ms for DEIMv2-X.
  2. Table 3 provides direct evidence for using multiple EuPE layers. Combining all three layers yields 53.3 AP, a 5.2-point gain over the best single-layer configuration.
  3. Auxiliary supervision offers a practical improvement without adding a prediction head at inference. In Table 2, adding EAH to the dual-branch model with MSDI raises AP from 52.6 to 53.3.
Critical weaknesses
  1. FSD-N’s reported computational cost appears difficult to reconcile with the submitted architecture. Since the model runs EuPE-B/16 on (640\times640) inputs, the 108 GFLOPs reported for the entire detector seems unexpectedly low and requires clarification.
  2. Table 4 does not isolate the effect of architectural complementary. FSD retains a separate semantic encoder at inference, whereas the published DEIMv2-S uses a ViT-Tiny backbone distilled from DINOv3-S. The 8.3 versus 1.9 AP gains therefore reflect differences in model capacity and how the VFM is used, alongside any effect of detector architecture.
  3. I have reservations about the novelty of MSDI. Prior work such as ViT-CoMer (Xia C, Wang X, Lv F, et al. Vit-comer: Vision transformer with convolutional multi-scale feature interaction for dense predictions (CVPR). IEEE, 2024: 5493-5502.) has already explored multi-scale CNN–ViT fusion, making the methodological contribution of MSDI insufficiently differentiated.
  4. Equation (2) obscures how auxiliary supervision benefits the main detector. It sends frozen EuPE outputs directly to EAH, whereas the code places trainable adapters shared with the main branch before the auxiliary head.
  5. The localization interpretation conflicts with Table 2. Adding MSDI to the dual-branch baseline improves AP50 by 6.2 points and AP75 by 5.4 points, contrary to the claim of a larger gain at the stricter IoU threshold.
Questions for authors
  1. Could the authors provide the EuPE input resolution, per-branch FLOP counts, and TensorRT timing logs for the same FSD-N checkpoint used to obtain 53.3 AP?
  2. How much do YOLO11-S and DEIM-S each gain when both use the same frozen VFM checkpoint, feature layers, fusion module, and training budget?
  3. With the three input levels fixed and fusion capacity matched, how consistent is MSDI’s 0.4 AP gain over concatenation?
  4. Which adapter produced the main results in Table 1, and which parameters shared with the main detector receive gradients from the auxiliary loss?
2 comments
AuthorsOpenReview ↗

Point-by-Point Rebuttal to Reviewer RvDP

W1 (FLOPs). The originally reported 108 GFLOPs for FSD-N were obtained using THOP v2.0.20 at 640×640, including the frozen EuPE-B/16 branch, whose forward FLOPs remain despite freezing. The profiling script, checkpoint, and eval logs are provided in Supplementary Material R1/Q1. Further investigation revealed inconsistent FLOPs conventions and operator coverage across variants. We have standardized the profiling and updated the results (Sec. 4.1; Supplementary Folder R1/Q1)

W2 (Complementarity). We agree the 8.3 vs. 1.9 AP gains do not isolate architecture: capacities and VFM integration differ. DEIMv2-S/M use DINOv3-distilled ViTs, DEIMv2-L uses pretrained DINOv3-S, and DEIMv2 employs STA. FSD instead couples a frozen semantic encoder with a trainable CNN. FSD-S: 54.9 AP/2.62 ms vs. DEIMv2-S: 50.9/3.14; FSD-N vs. DEIMv2-M: 53.3/2.34 vs. 53.0/3.99; FSD-M vs. DEIMv2-L: 56.2/3.39 vs. 56.0/4.64. FSD has fewer trainable parameters in the latter pairs but more FLOPs and a retained VFM. Thus comparisons of raw AP improvements should not be interpreted as isolated architectural effects. Secs. 4.2/4.4 distinguish frozen feature augmentation from VFM-distilled detector backbones; CKA supports representational differences but does not establish causality.

W3 (Novelty). We acknowledge ViT-CoMer and prior CNN–ViT fusion. MSDI's distinction is scale-wise alignment of frozen VFM semantics and trainable CNN details, followed by adjacent-level CFP propagating semantics toward fine scales and spatial detail toward deep scales. Removing CFP costs 4.4 AP; MSDI exceeds parameter-matched HC-Concat by 0.7 AP. The distinction is the sequence of cross-branch interaction and subsequent cross-scale exchange, not CNN–ViT fusion itself (Sec. 3.3)

W4 (Gradient flow). Corrected Eq. (2): E~i=Ai(Ei)\widetilde E_i=\mathcal A_i(E_i), i∈{3,4,5}i\in\{3,4,5\}; Y^aux=Haux(E~3,E~4,E~5)\hat Y_{aux}=\mathcal H_{aux}(\widetilde E_3,\widetilde E_4,\widetilde E_5). EAH receives the adapted outputs, not raw frozen features. Because those same adapter outputs feed MSDI and the main prediction path, auxiliary gradients update inference-retained adapter weights as well as EAH, but not frozen EuPE (Sec. 3.4)

W5 (Localization). Corrected Sec. 4.3: adding MSDI to DB increases AP50 by 6.2 and AP75 by 5.4. We claim gains at both thresholds, not a larger AP75 gain

Q1 (Profiling). EuPE input: 640×640. The exact THOP v2.0.20 script, FSD-N 53.3-AP checkpoint and COCO val2017 pycocotools logs are supplied in Supplementary Folder R1/Q1. Sec. 4.1 details TensorRT FP16/batch-1 measurement. We have revised the manuscript at Sec. 4.1, thank again

Q2 (VFM enhancement). YOLO11-S improves 46.6→54.9 AP with FSD-S (+8.3). Adding the same frozen EuPE checkpoint/feature levels improves RT-DETRv4-S 49.7→53.2 (+3.5) and DEIMv2-S 50.9→53.7 (+2.8). These additional experiments demonstrate that EuPE can improve both Transformer-based baselines, while FSD-S shows the largest observed improvement over its CNN baseline. Different initial architectures, fusion mechanisms and VFM usage preclude causal or capacity-matched claims (Sec. 4.4)

ModelVFMVFM Arch.AP (%)Latency (ms)
FSD-SSEERRegNetY51.62.34
FSD-SDINOv3ConvNeXtV253.52.53
FSD-SEuPEViT54.92.62
FSD-SDINOv3ViT54.02.97
RT-DETRv4-SDINOv3 distillViT49.72.38
RT-DETRv4-SDINOv3 distill + EuPEViT53.23.05
DEIMv2-SDINOv3 distillViT50.93.14
DEIMv2-SDINOv3 distill + EuPEViT53.73.71

Q3 (Fusion capacity). With E3/E4/E5 fixed, MSDI yields 53.3 AP vs. Concat 52.9 and near-parameter-matched HC-Concat 52.6 (5.425M vs. 5.378M). Removing CFP drops AP to 48.9. The HC-Concat control differs by only 0.047M trainable parameters, yet trails MSDI by 0.7 AP; this reduces the likelihood that the gain stems merely from fusion capacity. The CFP ablation separately assesses cross-scale interaction. Cross-seed consistency remains untested (Sec. 4.3)

StrategyInteractionParams (M)APAP50AP75ΔAP
FusionAdd4.91352.871.157.0-0.5
FusionConcat5.12652.971.357.3-0.4
FusionHC-Concat5.37852.671.056.9-0.7
FusionOurs5.42553.372.157.8—
Cross-scaleMSDI w/o CFP4.74448.969.754.0-4.4
Cross-scaleMSDI5.42553.372.157.8—

Q4 (Adapter). All Table 1 FSD models use trainable linear adapters shared by EAH and MSDI. Auxiliary loss updates these and EAH, not frozen EuPE. Eq. (2)/Sec. 3.4 clarify the path; adapter ablations are in Sec. 4.3

AdapterAPAP50AP75
W/O Adapter53.071.757.5
MLP53.071.857.6
1×1 Conv53.172.057.6
Linear53.372.157.8

Due to the word limit, full responses and experimental details are in the supplementary material

I thank the reviewer for the response, which addresses some concerns. However, I still have concerns regarding the novelty of the method and some claims, such as the novelty about the fusion of CNN and Transformer, and the claim about the larger improvement from stricter criteria from L361. Based on the above considerations, especially given the novelty expected for an ICLR publication, I maintain my original score.

Reviewer_Ge2uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FSD, which uses MSDI to enable multiscale interactions between a frozen EuPE encoder and a CNN. By combining the semantic features of VFMs with the spatial features of CNNs, FSD effectively improves real-time object detection performance. FSD achieves improved AP on COCO and reports relatively low inference latency on the RTX 4090.

Critical strengths
  • The paper is well motivated. Leveraging the strong semantic representations of a lightweight VFM to improve real-time object detection is an interesting direction, and these representations complement the spatial inductive biases of CNNs.

  • The experimental evaluation is reasonably comprehensive, demonstrating clear improvements on COCO. The ablation studies also show the effectiveness of individual designs.

Critical weaknesses
  • The evidence to support the claim of low latency is insufficient. Compared with 22 GFLOPs for YOLO11-S, FSD-S requires 303 GFLOPs, yet their respective latencies on an RTX 4090 are 2.28 and 2.62 ms. This paper lacks a more detailed explanation of why such a large difference in computational cost translates into only a small difference in latency.

  • The gain of the semantic–detail feature fusion strategy in MSDI is modest. In Table 4, the proposed fusion scheme improves over direct concatenation by only 0.4 AP. This result provides limited evidence that the dedicated fusion strategy offers a consistent benefit over simple concatenation sufficient to justify the added complexity.

Questions for authors
  1. Why does the substantial increase in GFLOPs translate into only a small increase in measured latency? Could the authors conduct fair latency comparisons with the baseline models on other hardware platforms?

  2. What specific advantages does the semantic–detail feature fusion strategy in MSDI offer over direct concatenation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.