acceptodds
Withdrawn from ICLR 2027

Seeing Globally, Detecting Locally: Injecting and Characterising Frozen Vision Transformer Priors in Object Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Convolutional neural networks (CNNs) provide efficient local inductive biases, while Vision Transformers (ViTs) aggregate global scene context via self-attention. Efforts to unify these complementary strengths into hybrid models often require complex, architecture- or domain-specific coupling, limiting transfer across detector families and obscuring how the two priors interact. To address this, we introduce a lightweight framework that uses a frozen vision foundation model as an auxiliary feature supplier, injecting its final patch representations in one direction into the deepest stage (C5) of a host convolutional backbone. The downstream pipeline remains unchanged, so the host's own detection loss determines how local and global features are mixed. Using DINOv3 as the frozen encoder, we conduct a factorial study of host initialisation and prior availability across one- and two-stage detectors, stratified data regimes on PASCAL VOC and Palm Oil, and full-scale COCO. Across settings, frozen priors consistently accelerate learning. They substitute for and complement conventional pretraining: randomly initialised hybrids surpass pretrained baselines at full PASCAL VOC on both hosts, and pretrained hosts still gain. These benefits depend on host competence and target domain. On PASCAL VOC, pretrained hybrids outperform their baselines by 7.1 AP on YOLOv8s and 13.8 AP on Faster R-CNN at full data, and the Faster R-CNN hybrid matches its full-data baseline with an estimated 11.4× fewer labels. On Palm Oil, this advantage contracts as data increase. We further identify a failure mode in which fine-tuning a COCO-pretrained YOLO detector at its default learning rate erodes accuracy on categories it already detects, with limited data restricting recovery. Overall, this study characterises the learning dynamics that emerge when frozen ViT priors meet trainable CNN representations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Object Detection, Vision Transformers, Vision Foundation Models, CNN-Transformer Hybrids, Frozen Visual Encoders, Transfer Learning, Representation Learning, Sample Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JwULOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper suggest an approach to fuse features from frozen Vision Transformer (Dinov3) with CNN backbone to benefit from global infromation lacking in CNN backbones for object detection. Path features of Dinov3 are reshaped to have same resolution of last block of their CNN feature extractor (C5) and aslo number of channels are increased to match those of C5. After concatenation of featues conv 1x1 is applied which brings the fused feature block to same dimension as C5 which removes the need to make any adjustment for susequent layers. Through empirical studies, they demonstrate performance gain for CNN object detectors (both 1 and 2 stage CNN object detectors) on PASCAL-VOC, Palm Oil and COCO when fusing the features with frozen Dinov3.

Critical strengths
  1. Clear presentation of idea
  2. Strong empirical results covering 2 object detectors and 3 datasets.
  3. Ablation studies to further study the impact of feature fusion (impact of datasets and number of training labels)
Critical weaknesses
  1. Lack of novelty: While this paper for the first time suggest feature fusion of Dinov3 with CNN backbone for object detection but similar fusion of frozen ViT with CNN can be found in prior works:

  2. Lack of justification for the proposed design: comapring CNN + Dinov3 (frozne) with fine-tuning of DInov3 we observe that fine-tuned Dinov3 has better performance while less total parameters. The only benefit of CNN + DINOV3 (frozen) is lower trainbale parameters which comes at cost of lower performance compared to Dinov3 fine tuning.

  3. Missing ablation: despite having diverse experiments which are appreciated but there is no study showing that after training how much information is carried from C5 and how much from Dinov3.

Questions for authors
  1. Based on table 4c, fine-tuning of Dinov3 results in better performance with lower total parameters at the cost of more trainable parameters. Therefore, benefit of CNN + frozen Dinov3 is limited to only efficiency in training phase?

  2. The idea of fusing frozen vision transformer with CNN network can be traced back at prior works such as:

Zhi Cai, Yingjie Gao, Yanan Zhang, Xinzhu Ma, and Di Huang. “Towards Unbiased Source-Free Object Detection via Vision Foundation Models.” Pattern Recognition, 2026, article 113545

Xiangyu Gao, Yu Dai, Benliu Qiu, Lanxiao Wang, Heqian Qiu, and Hongliang Li. “Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection.

While your idea is not identical what fundemental novelty separates your work from prior works.

Reviewer_D5ipOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SGDL combines frozen DINOv3 features with CNN features at C5 without changing the detection heads or loss. Experiments with YOLOv8s and Faster R-CNN show faster convergence and accuracy gains that depend on initialization, training data size and dataset. The paper also shows that a lower learning rate prevents the early drop in accuracy during YOLO fine-tuning.

Critical strengths
  1. The method improves both detectors without changing their detection heads or loss. On full VOC, pretrained YOLOv8s and Faster R-CNN-R18 gain 7.1 and 13.8 AP, respectively (Table 1).
  2. The ablation studies help explain when frozen features improve detection. The experiments compare different initializations and training data sizes, and report both improvements and performance drops across datasets.
Critical weaknesses
  1. The motivation for combining a CNN with a frozen ViT is not sufficiently clear. DINOv2 and DINOv3 have already been used as detection backbones [1, 2, 3], and fine-tuning achieves higher AP in Tables 4(c) and 7(b), so the paper needs to clarify when the reduction in training memory justifies this design.
  2. The fusion method offers limited novelty compared with prior work. Frozen-DETR [4] and VMCNet [5], both cited by the authors, already use frozen features to improve detection, but SGDL does not compare fusion methods under the same experimental settings to separate the contribution of its design from that of the pretrained features.
  3. The gains need further validation with fully trained and tuned baselines. As the author claimed, the COCO models achieve only 8.9 and 11.3 AP without reaching convergence, and the YOLO comparisons are not repeated with lower learning rates even though these prevent the observed accuracy drop.

[1] Guangxing Han and Ser-Nam Lim. Few-Shot Object Detection with Foundation Models. CVPR 2024.
[2] Oriane Siméoni et al. DINOv3. Transactions on Machine Learning Research, 2026.
[3] Shihua Huang et al. Real-Time Object Detection Meets DINOv3. arXiv 2509.20787, 2025.
[4] Shenghao Fu et al. Frozen-DETR. Enhancing DETR with Image Understanding from Frozen Foundation Models. NeurIPS 2024.
[5] Xiangyu Gao et al. Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection. arXiv 2501.16981, 2025.

Questions for authors
  1. Compared with using DINOv3 as the detection backbone, what are SGDL's main contributions and advantages, and how can the effect of global context be verified?
  2. Do the findings also apply to other dense prediction tasks, such as segmentation and depth estimation?
Reviewer_zWNdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SGDL, a framework that injects features from a frozen DINOv3 Vision Transformer into a trainable CNN object detector. The ViT features are projected and aligned with the detector’s C5 feature map, then fused through concatenation and a 1×1 convolution. The detector’s neck, heads, and loss remain unchanged.

The authors evaluate SGDL using YOLOv8s and Faster R-CNN on PASCAL VOC, Palm Oil, and COCO. They study label efficiency, host initialization, fusion design, encoder choice, and computational cost, and analyze a fine-tuning instability observed in pretrained YOLO detectors.

Critical strengths
  1. A simple, reusable integration method supported by systematic experiments.

SGDL can be applied to both one-stage and two-stage detectors without redesigning their downstream detection pipelines.

The 2×2 comparison of host initialization and frozen ViT features is informative. On full PASCAL VOC, adding SGDL improves AP50:95 by 7.1 points for pretrained YOLOv8s and 13.8 points for pretrained Faster R-CNN (Table 1). The results also demonstrate meaningful improvements in label efficiency.

The ablations of injection sites, host capacities, integration routes, and frozen encoders help establish the usefulness of the approach.

  1. Useful empirical findings beyond overall detection accuracy.

The paper examines how the benefit of a frozen ViT prior changes with dataset characteristics and training-set size. The comparison between PASCAL VOC and Palm Oil provides evidence that the prior is not equally useful across target domains.

The fine-tuning erosion analysis in Section 5 is also interesting. The experiments across YOLOv8s, YOLOv6-S, and Faster R-CNN, together with learning-rate controls, provide useful observations about detector adaptation.

Critical weaknesses
  1. Limited methodological novelty relative to existing frozen-ViT fusion approaches.

SGDL uses a straightforward combination of projection, spatial alignment, concatenation, and a 1×1 convolution at C5. Related approaches, including Frozen-DETR and VMCNet, have already explored incorporating frozen transformer representations into object detectors.

The single-site interface and compatibility with different detector families are useful, but the individual operations are standard, and the paper does not introduce a new fusion objective or representation-learning principle.

The existing ablations, including the ViT-only comparisons in Table 4©, provide useful evidence. However, they do not fully establish what new generalizable design insight SGDL provides beyond demonstrating that pretrained ViT features can improve CNN detectors.

  1. Large-scale effectiveness and practical accuracy–efficiency trade-offs remain insufficiently established.

The COCO experiment in Section 4.3 uses the full dataset but only one 12-epoch run per condition. Both detectors are explicitly reported as far from convergence, with AP50:95 of 8.9 for Faster R-CNN and 11.3 for SGDL (Table 3).

The within-protocol improvement is positive, but these results do not establish that the advantage persists when the detector is trained to a competitive level.

Inference cost is another significant limitation. Appendix Table 9 shows that SGDL increases YOLOv8s computation from 14.24 to 73.05 GMACs and reduces throughput from 102.6 to 31.3 FPS.

The encoder-resolution ablation in Table 7 is helpful, but the overall accuracy–latency trade-off against competitive alternatives remains unclear.

Questions for authors
  1. What specific advantage does the proposed single-site fusion provide over simpler alternatives?

Beyond its implementation simplicity, what distinguishes SGDL from existing frozen-transformer-assisted detection methods?

A controlled comparison using the same frozen DINOv3 encoder, CNN host, and training protocol would be helpful. For example, could the authors compare the proposed C5 fusion against a simpler alternative fusion mechanism, while reporting both AP and additional inference cost?

The existing frozen-ViT-only and host-capacity comparisons are useful; the remaining question is whether the particular fusion interface provides an advantage beyond access to pretrained features.

  1. Does SGDL retain its advantage after sufficient training on COCO?

Table 3 reports 8.9 versus 11.3 AP after 12 epochs, with both models far from convergence.

Could the authors provide validation AP learning curves for both configurations, including later checkpoints under a longer matched training schedule (e.g., 24 epochs), if available?

The key question is whether the SGDL–baseline performance gap persists as both models approach convergence. If additional training is infeasible, the large-scale effectiveness claim should be limited accordingly.

  1. When is SGDL preferable under a fixed inference budget?

Table 7 shows that reducing the frozen encoder resolution from 640 to 480 pixels retains approximately 91% of the measured gain while reducing the added computation from 58.8 to 27.4 GMACs.

Could the authors report inference latency for the reduced-resolution configurations and compare their AP–latency trade-off with standalone CNN detectors of comparable inference cost?

This would help identify when the accuracy gains justify the additional frozen-encoder forward pass.

Additional Comments

  1. ViT-only comparisons: Table 4© already provides a useful comparison: SGDL obtains 38.1 AP, compared with 35.0 for a frozen ViT-backbone detector and 41.8 for fine-tuned ViTDet in the evaluated setting. These results should be considered when discussing the benefits and limitations of the proposed integration route.

  2. Interpretation of the theoretical framework: The approximation–estimation–optimisation discussion in Section 3.2 provides a useful interpretation of the observed data-scaling trends. However, the empirical curves do not uniquely identify these error components. The interpretation should remain clearly distinguished from a formal quantitative decomposition.

Overall Assessment

SGDL provides a simple integration mechanism and a useful empirical study of frozen ViT priors for object detection. The PASCAL VOC results and label-efficiency gains are meaningful, and the controlled analyses are valuable.

However, the methodological novelty is relatively limited. The incomplete COCO training and substantial inference overhead also make it difficult to establish the broader practical significance of the approach.

More convincing evidence of a distinct contribution and a competitive accuracy–efficiency trade-off would be needed to support acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.