acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Lies in Layers: Internalizing Foundation Visual Priors for Few-Shot Remote Sensing Object Detection

Abstract

Few-shot remote sensing object detection is limited by scarce annotations, hindering robust and generalizable representation learning. Existing methods increasingly leverage vision foundation models to enrich visual representations, but often rely on fixed-depth features or unified multi-layer integration, overlooking depth-dependent representation characteristics. Through a systematic layer-wise analysis of a vision foundation model, we find clear depth-dependent differences: low-to-mid layers exhibit stronger variation across spatial locations, while mid-to-high layers show stronger alignment between patch responses and object regions. Such a phenomenon implies that different network depths offer distinct advantages in spatial representation and object-aware semantics, and that exploiting these advantages may provide a lightweight solution to improve few-shot detection under limited supervision. Accordingly, we propose LayerFi-FSOD, a hierarchical foundation visual prior internalization framework that exploits foundation-model knowledge according to its depth-dependent representation characteristics. Specifically, the Adaptive Dense Prior Injection Module aggregates low-to-mid-layer dense priors to enhance multi-scale detection features, while the Dual-Grained Structural-Relational Guidance Module guides the student detector to absorb knowledge from different depths through local spatial structures and global semantic relations. Extensive experiments on the DIOR and NWPU VHR-10 datasets validate the effectiveness of LayerFi-FSOD. Notably, the vision foundation model is used only as a visual prior provider during training and is completely removed at inference, introducing no additional foundation-model inference overhead. The relevant code will be made publicly available upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Few-shot remote sensing object detection, Vision foundation models, Hierarchical visual priors, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EQ3nOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LayerFi-FSOD for few-shot remote sensing object detection. It analyzes the different characteristics of DINOv3 features across layers and introduces two modules, ADPIM and DSGM, to transfer spatial and semantic knowledge to a student detector. DINOv3 is only used during training and removed at inference.

Critical strengths
  1. The motivation is clear, and the layer-wise analysis provides useful insights into the design, from which I learned a lot.
  2. The method achieves strong results on DIOR and NWPU VHR-10, especially in low-shot settings.
  3. The ablation studies are fairly comprehensive, and removing DINOv3 at inference is a practical advantage.
Critical weaknesses
  1. Although DINOv3 is removed at inference, it is still required during training. The training time and GPU memory overhead would be useful to report.
  2. The performance appears sensitive to the choice of DINOv3 layers in Table 5. It would be helpful to further examine whether the selected layers generalize well across different datasets.
  3. The roles of ADPIM and DSGM at inference are not entirely clear. Since both modules are designed around DINOv3 features, it would be helpful to clarify how their learned knowledge is actually retained by the student after DINOv3 is removed.
Questions for authors

Please refer to the concerns raised in the Weaknesses. In particular, addressing Points 2 and 3 would be important for improving my overall assessment.

3 comments
AuthorsOpenReview ↗

Response (1/2) to Review EQ3n, we remain open to further discussion.

We greatly appreciate the reviewer’s insightful comments and provide our detailed response below. We also welcome any further questions or suggestions.

  1. Weaknesses: Although DINOv3 is removed at inference, it is still required during training. The training time and GPU memory overhead would be useful to report.

Response: We thank the reviewer for this important suggestion. We agree that although DINOv3 is removed during inference, its training overhead should still be explicitly quantified. We compare the fine-tuning time and peak GPU memory usage of LayerFi-FSOD and SAE-FSDet [1] on NWPU VHR-10 under the same experimental conditions:

MethodMetric3-shot5-shot10-shot20-shot
SAE-FSDetTime (h)0.500.691.081.90
Memory (GiB)4.704.785.275.27
mAP₅₀57.9659.4071.0285.08
LayerFi-FSODTime (h)2.102.404.506.30
Memory (GiB)16.8016.8018.1018.20
mAP₅₀63.4667.9473.3286.06

Compared with SAE-FSDet, LayerFi-FSOD requires 3.32–4.20× the fine-tuning time and 3.43–3.57× the peak GPU memory, mainly due to visual prior learning with frozen DINOv3. Importantly, this overhead occurs only during offline few-shot fine-tuning, while yielding novel-class mAP50 gains of 5.50/8.54/2.30/0.98 percentage points at 3/5/10/20-shot, respectively. DINOv3 and all auxiliary branches are completely removed at inference, and the student retains the same inference architecture as SAE-FSDet without additional overhead. The teacher-free detector requires 181.13 GFLOPs and achieves 28.06 FPS. Thus, our method represents an explicit trade-off between training cost and inference efficiency: additional offline computation improves detection under limited annotations without increasing deployment costs.

[1] Liu, Yanxing, et al. "Few-Shot Object Detection in Remote-Sensing Images via Label-Consistent Classifier and Gradual Regression." TGRS, 2024.

  1. Weaknesses: The performance appears sensitive to the choice of DINOv3 layers in Table 5. It would be helpful to further examine whether the selected layers generalize well across different datasets.

Response: We thank the reviewer for this important suggestion. We agree that it is necessary to explain the sensitivity to layer selection and validate the selected configuration across datasets. This sensitivity mainly arises from the alignment between features at different depths and the learning objectives of ADPIM and DSGM, while also being influenced by dataset characteristics. According to our layer-wise analysis, shallow-to-middle-layer features exhibit stronger inter-patch spatial variation, facilitating the construction of dense spatial priors in ADPIM, whereas middle-to-deep-layer features show stronger foreground responses, benefiting the modeling of inter-object semantic relationships in GSDM. Therefore, introducing too many deep-layer features into ADPIM may weaken spatial details, while using only the final layer or indiscriminately fusing multiple layers may fail to fully exploit the complementary advantages of representations at different depths. Furthermore, differences in object scale distributions, category composition, and scene complexity across datasets may affect the contribution of visual priors from different layers, leading to varying degrees of sensitivity to layer selection.

To further validate the cross-dataset generalization of this hierarchical division, we conduct additional layer selection experiments on DIOR Split 2 and compare the results with NWPU VHR-10 (novel-class mAP50, %):

Layers for ADPIMLayers for GSDMNWPU 3-shotNWPU 5-shotDIOR 3-shotDIOR 5-shot
{3,6,9}{12,15,18,21,24}59.3563.5416.9019.77
{3,6,9,12,15}{18,21,24}57.7061.7617.6519.34
{14,17,20,23}{2,5,8,11}59.2063.1418.1019.11
{3,6,9,12,15,18,21,24}{3,6,9,12,15,18,21,24}60.5865.8817.5820.01
{3,6,9,12}{15,18,21,24}63.4667.9419.1621.54

The results show that although the magnitude of performance variations differs across datasets, assigning shallow-to-middle-layer features to ADPIM and middle-to-deep-layer features to GSDM achieves the best performance among the evaluated configurations on both datasets. As complementary evidence of dataset-dependent variability, the three-seed results reported in our response to Reviewer AsPM (Weakness 3) show generally larger standard deviations on NWPU VHR-10 than on DIOR, indicating greater sensitivity to few-shot sample selection. These findings indicate that while the degree of sensitivity to layer selection may be influenced by dataset characteristics, the selected hierarchical allocation consistently performs best across the two evaluated datasets, supporting the importance of aligning features from different depths with the learning objectives of the corresponding modules.

AuthorsOpenReview ↗

Response (2/2) to Review EQ3n, we remain open to further discussion.

  1. Weaknesses: The roles of ADPIM and DSGM at inference are not entirely clear. Since both modules are designed around DINOv3 features, it would be helpful to clarify how their learned knowledge is actually retained by the student after DINOv3 is removed.

Response: We thank the reviewer for this suggestion. We clarify the roles of ADPIM and DSGM and how the student retains their knowledge after DINOv3 is removed.

  • ADPIM and DSGM are both excluded from inference. ADPIM constructs DINOv3 visual priors only during fine-tuning and enhances student RoI features through gated residual fusion. During inference, this fusion branch is removed, and the detector directly uses RoI features extracted from its own FPN for classification and regression. DSGM provides distillation supervision only during training, while its teacher feature extraction, relation computation, and auxiliary distillation branches are all excluded from inference. Therefore, the final detector retains only the optimized ResNet-101, FPN, and detection heads, without any DINOv3-dependent feature extraction or fusion modules.
  • Visual priors are retained in the student network parameters through gradient-based optimization during training. The residual fusion in ADPIM not only enhances detection supervision using DINOv3 [1] features but also preserves the gradient propagation path through the original student RoI features, allowing the detection loss to update the student network. More importantly, DSGM does not merely constrain the fused detection outputs: LSSD directly supervises the spatial self-similarity distribution of the student's own RoI features, while GSAD constrains the inter-proposal semantic relationships of the student's classification representations. Since the teacher relation targets are stop-gradient, the distillation errors are backpropagated through the student branches, encouraging their parameters to learn the corresponding spatial structures and semantic relationships. Thus, knowledge transfer occurs through parameter optimization during training rather than relying on teacher features at inference.

Nevertheless, introducing ADPIM during training may still cause the student to depend on external features. To examine whether this potential training–inference discrepancy affects detection performance, we compare the inference results with and without DINOv3 on NWPU VHR-10 (novel-class mAP50, %):

Method3-shot5-shot10-shot20-shot
LayerFi-FSOD w. DINOv363.0167.1672.8886.33
LayerFi-FSOD w/o DINOv363.4667.9473.3286.06

After removing DINOv3, performance improves by 0.45/0.78/0.44 percentage points under the 3/5/10-shot settings, respectively, while decreasing by only 0.27 percentage points under the 20-shot setting. These results indicate that removing the training-time visual prior branches does not cause a noticeable performance degradation, and the student can independently perform detection using its own learned representations. Together with the gradient-based optimization mechanism described above, these results support that the training guidance provided by ADPIM and DSGM can be transferred into the student's independent detection capability without requiring continued access to DINOv3 during inference.

[1] Simeoni, Oriane, et al. "DINOv3." TMLR, 2026.

  1. Questions: Please refer to the concerns raised in the Weaknesses. In particular, addressing Points 2 and 3 would be important for improving my overall assessment.

Response: We thank the reviewer for further emphasizing the importance of Points 2 and 3. To address these concerns, we provide additional experimental validation and further clarify the relevant mechanisms:

  • Regarding the sensitivity of layer selection and its cross-dataset generalization (see Weakness 2): The sensitivity mainly arises from the alignment between features at different depths and the learning objectives of ADPIM and DSGM, while also being influenced by dataset characteristics, including object scale distributions, category composition, and scene complexity. We further conduct additional experiments on DIOR Split 2. Consistent with NWPU VHR-10, the selected layer configuration achieves the best performance on both datasets, validating its cross-dataset generalization capability.
  • Regarding the knowledge retention mechanism after removing DINOv3 (see Weakness 3): We further clarify the training and inference procedures of ADPIM and DSGM and explain how visual priors are transferred into the student network parameters through gradient-based optimization. Furthermore, the inference comparison with and without DINOv3 shows that removing DINOv3 does not cause noticeable performance degradation, confirming that the student detector can perform detection independently.
Reviewer_EQ3nOpenReview ↗

Response to Authors

Thank you for your detailed response and explanation. However, taking into account the discussions with the other reviewers as well as our previous correspondence, I am inclined to maintain my original score.

Reviewer_AsPMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LayerFi-FSOD, a framework for few-shot remote sensing object detection that transfers hierarchical visual knowledge from DINOv3 into a lightweight student detector. The proposed method consists of an Adaptive Dense Prior Injection Module (ADPIM) for exploiting low-to-mid-level spatial representations and a Dual-Grained Structural-Relational Guidance Module (DSGM) for transferring local structural and global semantic relationships. The foundation model is used only during training, allowing the student detector to operate without additional inference overhead. Experiments on DIOR and NWPU VHR-10 demonstrate improvements over several existing few-shot detection methods.

Critical strengths
  1. Practical significance. The paper addresses the important challenge of leveraging pretrained foundation models for few-shot remote sensing object detection, where limited annotations make robust representation learning difficult.

  2. Efficient inference. The proposed framework transfers foundation-model knowledge during training without requiring the large teacher model at inference, providing a favorable accuracy-efficiency trade-off.

  3. Promising experimental results. The method achieves competitive performance across multiple few-shot settings, and the ablation studies provide useful evidence for the effectiveness of hierarchical feature utilization and structural distillation.

Critical weaknesses
  1. The methodological novelty is relatively limited. The proposed framework largely combines existing techniques, including multi-layer feature aggregation, adaptive routing, and relational knowledge distillation. The authors should better distinguish the contributions from prior foundation-model-based detection and distillation methods, particularly VFM-DETR and conventional structural distillation approaches.

  2. The core motivation requires stronger validation. The layer-wise analysis primarily relies on inter-patch feature variance and foreground alignment, which do not necessarily establish the claimed differences in spatial structure and semantic information. Additional investigations with different foundation models would help demonstrate the generality of these observations.

  3. The experimental improvements lack statistical reliability analysis. Few-shot detection is highly sensitive to training sample selection and random initialization. However, the paper does not report results across multiple random seeds or provide standard deviations. This is particularly important for settings where the improvements over competing methods are relatively small.

  4. The contribution of individual components could be better justified. The ablation results indicate that local spatial structure distillation contributes substantially to the improvements, while some other components provide relatively modest additional gains. More comparisons against simpler feature or relational distillation baselines would help justify the complexity of the overall framework.

  5. The evaluation of generalization and computational cost remains incomplete. Experiments are limited to two remote sensing datasets and mainly report novel-class mAP50. Additional evaluations of base-class retention, localization performance under stricter IoU thresholds, and training-time computational overhead would provide a more comprehensive assessment.

Questions for authors
  1. How does the proposed method compare with standard feature distillation or relational distillation using the same DINOv3 teacher and training budget?

  2. Can the authors report the mean and standard deviation across multiple few-shot sampling seeds, particularly for the 3-shot and 5-shot settings?

  3. Since DINOv3-based feature injection is removed at inference, could the authors further explain how the student maintains the learned representation improvements without the teacher features?

5 comments
AuthorsOpenReview ↗

Response (1/4) to Review AsPM, we remain open to further discussion.

We greatly appreciate the reviewer’s insightful comments and provide our detailed response below. We also welcome any further questions or suggestions.

  1. Weaknesses: The methodological novelty is relatively limited. The proposed framework largely combines existing techniques, including multi-layer feature aggregation, adaptive routing, and relational knowledge distillation. The authors should better distinguish the contributions from prior foundation-model-based detection and distillation methods, particularly VFM-DETR and conventional structural distillation approaches.

Response: We thank the reviewer for this valuable comment. The core contribution of LayerFi-FSOD is not simply combining multi-layer feature aggregation, adaptive routing, and relational distillation, but designing targeted visual prior construction and knowledge transfer mechanisms for few-shot remote sensing object detection based on the representation characteristics of different layers in vision foundation models. Unlike VFM-DETR's inference-time feature fusion, our framework uses hierarchical visual priors during training to optimize the student's spatial and semantic representations through task-aligned knowledge transfer. The specific distinctions from existing methods are as follows:

  • CLS-guided dense spatial prior enhancement. Unlike VFM-DETR [1], which primarily enhances foreground responses through region-wise routing, we design CAFM to explicitly exploit the semantic similarity between CLS tokens and patch tokens from shallow-to-middle layers of vision foundation models. This enables position-dependent spatial modulation of dense features, enhancing object-relevant responses while preserving spatial structural information, rather than merely aggregating multi-layer features.

  • Dynamic hierarchical selection for multi-scale detection. VFM-DETR employs an Expert-wise Router to select visual experts for different detector layers, whereas our PLEM further combines the current FPN-level features with global P2–P5 pyramid context to dynamically allocate visual priors from different depths according to image content and detection scale. This enables hierarchical feature selection to consider both scale-specific requirements and cross-scale context, rather than relying solely on single-level features for routing.

  • Hierarchy-guided dual-granularity relational distillation. Unlike directly aligning teacher and student feature or relational representations, our DSGM assigns appropriate teacher knowledge to different distillation objectives according to the representation characteristics of different depths in vision foundation models. Specifically, shallow-to-middle-layer dense priors guide intra-RoI local spatial relationship learning, while middle-to-deep-layer semantic representations guide inter-RoI global semantic relationship learning. This allows the transfer of spatial structures and semantic relationships to leverage the hierarchical features best suited to their respective objectives.

We further validate this design through Box Boundary F1 and Same-class Pair AUC analyses across DINOv3, DINOv2, CLIP, and RemoteCLIP (see Weakness 2). Moreover, using the same DINOv3 teacher and training protocol, LayerFi-FSOD consistently outperforms Feature L2/Cosine distillation, particularly in low-shot settings (see Weakness 4).

[1] Liu, Changhan, et al. "Don't Need Retraining: A Mixture of DETR and Vision Foundation Models for Cross-Domain Few-Shot Object Detection." NeurIPS, 2025.

AuthorsOpenReview ↗

Response (2/4) to Review AsPM, we remain open to further discussion.

  1. Weaknesses: The core motivation requires stronger validation. The layer-wise analysis primarily relies on inter-patch feature variance and foreground alignment, which do not necessarily establish the claimed differences in spatial structure and semantic information. Additional investigations with different foundation models would help demonstrate the generality of these observations.

Response: We thank the reviewer for this valuable suggestion. The original inter-patch feature variance and foreground alignment analyses provide descriptive evidence of hierarchical representation changes from the perspectives of spatial feature variation and foreground response alignment. To further strengthen the empirical support for our core motivation, we introduce two additional functional evaluations on NWPU VHR-10 and extend the analysis to four vision foundation models: DINOv3 [1], DINOv2 [2], CLIP [3], and RemoteCLIP [4].

First, we employ Box Boundary F1 to evaluate the spatial boundary consistency of features at different layers. Specifically, we use the average patch feature from the central region of each GT box as a query and compute its cosine similarity with all image patches to generate a spatial response map. We then binarize the map using a unified threshold determined on the training set and calculate the F1 score between the response region boundary and the GT box boundary with a tolerance of one patch. This metric reflects the boundary separability between object regions and their surroundings in feature space. The results are as follows:

LayerDINOv3DINOv2CLIPRemoteCLIP
30.54820.71940.64890.6144
60.64780.71760.67920.6689
90.67400.77190.72290.7228
120.69050.71900.75190.7706
150.67310.70800.65480.6820
180.66110.67500.54410.5613
210.65480.63310.53500.5061
240.66750.52340.58700.5557

Second, we employ Same-class Pair AUC to evaluate the class-level semantic discriminability of features at different layers. Object features are extracted using GT RoIAlign and average pooling. We compute cosine similarities between same-class and different-class object pairs across images and calculate ROC-AUC using class-balanced pairs. This metric measures whether same-class objects exhibit higher feature similarity than different-class objects, reflecting the aggregation and discriminability of class-level semantics. The results are as follows:

LayerDINOv3DINOv2CLIPRemoteCLIP
30.81130.68690.79340.7993
60.85030.70800.80610.8063
90.83830.60010.86350.8676
120.84950.66790.90580.9213
150.86730.69860.95250.9715
180.89020.73830.97950.9942
210.92190.86690.97650.9913
240.92360.96610.95210.9764

All four VFMs exhibit a consistent trend in which spatial boundary consistency and class-level semantic discriminability peak at different depths. Specifically, Box Boundary F1 peaks at intermediate layers, whereas Same-class Pair AUC peaks at deeper layers. Although the exact peak layers vary across models, this consistent trend indicates that spatial boundary representations and class-level semantic discriminability exhibit different depth-dependent characteristics. These findings further support our motivation to transfer knowledge using shallow-to-middle-layer dense priors and middle-to-deep-layer semantic priors according to their hierarchical representation characteristics.

To further investigate the applicability of LayerFi-FSOD to different vision foundation models, we replace DINOv3 with DINOv2, CLIP, and RemoteCLIP and evaluate performance under the 5-shot and 10-shot settings on NWPU VHR-10. The results are as follows:

Method5-shot10-shot
SAE-FSDet59.4071.02
LayerFi-FSOD w. RemoteCLIP63.9073.02
LayerFi-FSOD w. DINOv265.4972.54
LayerFi-FSOD w. CLIP65.5072.11
LayerFi-FSOD w. DINOv367.9473.32

The results show that LayerFi-FSOD consistently outperforms the SAE-FSDet baseline with all four vision foundation models, indicating that the proposed hierarchical prior construction and knowledge transfer mechanisms are not dependent on a specific foundation model. Among them, DINOv3 achieves 67.94% and 73.32% mAP50 under the 5-shot and 10-shot settings, outperforming the baseline by 8.54 and 2.30 percentage points, respectively.

[1] Simeoni, Oriane, et al. "DINOv3." TMLR, 2026.

[2] Oquab, Maxime, et al. "DINOv2: Learning Robust Visual Features without Supervision." TMLR, 2024.

[3] Radford, Alec, et al. "Learning Transferable Visual Models from Natural Language Supervision." ICML, 2021.

[4] Liu, Fan, et al. "RemoteCLIP: A Vision Language Foundation Model for Remote Sensing." TGRS, 2024.

AuthorsOpenReview ↗

Response (3/4) to Review AsPM, we remain open to further discussion.

  1. Weaknesses: The experimental improvements lack statistical reliability analysis. Few-shot detection is highly sensitive to training sample selection and random initialization. However, the paper does not report results across multiple random seeds or provide standard deviations. This is particularly important for settings where the improvements over competing methods are relatively small.

Response: We thank the reviewer. To assess reliability, we conduct three runs with independent few-shot sampling seeds for LayerFi-FSOD, MSOCL [1], and SAE-FSDet [2] on DIOR Split 1/2 and NWPU VHR-10. We report novel-class mAP50 (mean ± SD):

DatasetMethod3-shot5-shot10-shot20-shot
DIOR-1MSOCL27.36 ± 2.3329.33 ± 2.6534.43 ± 1.1737.26 ± 1.71
SAE-FSDet29.78 ± 0.8732.40 ± 2.5837.25 ± 0.3241.80 ± 0.61
LayerFi-FSOD31.70 ± 0.6534.06 ± 2.7238.36 ± 0.5742.79 ± 0.58
DIOR-2MSOCL13.05 ± 1.4216.65 ± 2.0915.82 ± 0.1521.66 ± 2.60
SAE-FSDet17.20 ± 0.5720.58 ± 0.8522.64 ± 1.2026.37 ± 0.33
LayerFi-FSOD18.76 ± 0.4721.16 ± 0.8524.77 ± 0.4727.80 ± 0.08
VHR-10MSOCL42.54 ± 3.3953.78 ± 8.9166.68 ± 5.5477.04 ± 2.57
SAE-FSDet53.87 ± 3.5757.73 ± 5.8271.60 ± 3.5881.95 ± 2.90
LayerFi-FSOD59.22 ± 3.7762.92 ± 8.4272.88 ± 4.4383.12 ± 2.75

As shown, LayerFi-FSOD achieves higher mean mAP50 than both MSOCL and SAE-FSDet across all 12 experimental settings, indicating that the performance advantages of our method are not limited to a single random run. Although performance variability remains across different sampling seeds, our method consistently achieves higher mean detection performance across all evaluated settings.

[1] Chen, Jie, et al. "Multiscale Object Contrastive Learning-Derived Few-Shot Object Detection in VHR Imagery." TGRS, 2022.

[2] Liu, Yanxing, et al. "Few-Shot Object Detection in Remote-Sensing Images via Label-Consistent Classifier and Gradual Regression." TGRS, 2024.

  1. Weaknesses: The contribution of individual components could be better justified. The ablation results indicate that local spatial structure distillation contributes substantially to the improvements, while some other components provide relatively modest additional gains. More comparisons against simpler feature or relational distillation baselines would help justify the complexity of the overall framework.

Response: We thank the reviewer for this suggestion. LayerFi-FSOD components have distinct and complementary roles: CAFM and PLEM construct visual priors through global semantic-guided spatial modulation and multi-scale dynamic hierarchical selection, respectively. LSSD distills intra-RoI local spatial relationships, while GSAD transfers inter-RoI global semantic relationships. The original ablation results on NWPU VHR-10 are summarized below:

ComponentVariant3-shot5-shot10-shot20-shot
ADPIMDirect Concatenation56.2861.6372.1784.69
Learnable Routing Weights58.6361.8772.1884.78
Fixed Routing Weights58.0762.4871.8483.98
w/o CAFM63.1162.5672.9785.13
DSGMw/o GSAD62.6464.8371.6585.57
w/o LSSD56.2260.8173.1084.29
LayerFi-FSOD—63.4667.9473.3286.06

Removing LSSD reduces 3-shot and 5-shot performance by 7.24 and 7.13 percentage points, highlighting its importance in extremely low-shot settings. Removing CAFM decreases 5-shot performance by 5.38 points, while removing GSAD reduces 5-shot and 10-shot performance by 3.11 and 1.67 points. Direct concatenation, fixed routing, and globally learnable routing weights also underperform the full model. These results confirm the complementary contributions of visual prior construction, local spatial distillation, and global semantic distillation.

To further evaluate our structural relational distillation against simpler feature alignment, we compare with Feature L2 and Feature Cosine on NWPU VHR-10:

Method3-shot5-shot10-shot20-shot
Feature L256.7262.5272.5484.64
Feature Cosine54.5062.5772.7685.06
LayerFi-FSOD63.4667.9473.3286.06

On NWPU VHR-10, LayerFi-FSOD outperforms both simple feature distillation methods across all four shot settings. Compared with the stronger baseline in each setting, our method achieves improvements of 6.74, 5.37, 0.56, and 1.00 percentage points, respectively, with particularly notable gains under the 3-shot and 5-shot settings. These results indicate that, compared with directly enforcing feature-value or directional consistency between teacher and student, knowledge transfer combining local spatial structures and global semantic relationships can more effectively exploit the hierarchical visual priors of VFMs.

Reviewer_AaodOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LayerFi-FSOD, a distillation framework that distill knowledge from a DINOv3 to a lightweight FSOD model. The contributions are mainly technical. The method consists of two parts: an ADPIM to inject the powerful DINO features to the student model and a DSGM to distill the knowledge from DINO RoI features. Experiments are conducted on DIOR, NWPU VHR-10, showing improvement over previous FSOD methods.

Critical strengths

The technical contribution is significant. It is reasonable to build a distillation pipeline that can transfer the powerful knowledge from a large visual foundation model to the task-specific model.

Critical weaknesses
  1. The paper barely reveals new insight. "Useful visual information is distributed differently across network depths." The low-to-mid layers contains more localization knowledge and the mid-to-high layers contains more semantic knowledge, which is a well-studied knowledge [VFM-DETR]. The paper merely verified this common sense again on DINOv3. The paper therefore contributes technically more than academically. Also, no additional foundation-model inference overhead cannot be treated as a solid contribution either since it is a common sense that knowledge distillation removes the teacher model during inference.

  2. The reported evaluation metric is mAP50 alone. Missing metrics AP@(0.5–0.95) and AP75. In addition, more teacher-student pairs should be verified. The effectiveness of the proposed method is less convincing.

Questions for authors

see weaknesses above

2 comments
AuthorsOpenReview ↗

Response (1/2) to Review Aaod, we remain open to further discussion.

We greatly appreciate the reviewer’s insightful comments and provide our detailed response below. We also welcome any further questions or suggestions.

  1. Weaknesses: The paper barely reveals new insight. "Useful visual information is distributed differently across network depths." The low-to-mid layers contains more localization knowledge and the mid-to-high layers contains more semantic knowledge, which is a well-studied knowledge [VFM-DETR]. The paper merely verified this common sense again on DINOv3. The paper therefore contributes technically more than academically. Also, no additional foundation-model inference overhead cannot be treated as a solid contribution either since it is a common sense that knowledge distillation removes the teacher model during inference.

Response: We thank the reviewer for this valuable comment. We agree that the differences in spatial and semantic representations across layers of vision foundation models have been explored in prior studies. The core contribution of our work is not to rediscover this phenomenon, but to exploit hierarchical representation differences to design visual prior construction and knowledge transfer mechanisms for few-shot remote sensing object detection, enabling effective knowledge transfer from vision foundation models to lightweight detectors. Compared with existing foundation-model-based feature fusion and knowledge distillation methods, the main distinctions are as follows:

  • CLS-guided dense spatial prior enhancement. CAFM exploits CLS–patch semantic similarity from shallow-to-middle layers of vision foundation models for position-dependent spatial modulation, enhancing object responses while preserving spatial structures, rather than relying solely on conventional feature aggregation or region enhancement.
  • Dynamic hierarchical selection for multi-scale detection. PLEM combines current FPN-level features with global P2–P5 pyramid context to dynamically select visual priors from different depths according to image content and detection scale. This enables hierarchical selection to consider both scale-specific requirements and cross-scale context, rather than relying solely on single-level features for routing.
  • Hierarchy-guided dual-granularity relational distillation. DSGM employs shallow-to-middle-layer dense priors for intra-RoI local spatial relationship learning and middle-to-deep-layer semantic priors for inter-RoI global semantic relationship learning, enabling targeted matching between hierarchical knowledge and distillation objectives rather than uniform feature or relational alignment.

We further analyze hierarchical representations across DINOv3, DINOv2, CLIP, and RemoteCLIP using Box Boundary F1 and Same-class Pair AUC. The varying peak layers across models further motivate task-aligned hierarchical knowledge transfer (see AsPM Weakness 2).

To validate the effectiveness of these designs, we conduct additional comparisons with Feature L2 and Feature Cosine on NWPU VHR-10(mAP50, %):

Method3-shot5-shot10-shot20-shot
Feature L256.7262.5272.5484.64
Feature Cosine54.5062.5772.7685.06
LayerFi-FSOD63.4667.9473.3286.06

Under the same DINOv3 teacher and training protocol, LayerFi-FSOD outperforms simple feature distillation methods across all settings, with particularly notable gains under the 3-shot and 5-shot settings, validating the effectiveness of targeted hierarchical knowledge transfer.

Regarding inference overhead, we agree that removing the teacher model is not itself a novel technical contribution. However, as shown in Table 18 of the main paper, some foundation-model-based detectors still retain the foundation model during inference, increasing computational overhead. Our distinction lies in hierarchy-guided knowledge transfer, enabling the student detector to independently exploit learned spatial and semantic knowledge without relying on the foundation model at inference. We further compare inference performance with and without DINOv3 [1] on NWPU VHR-10(mAP50, %):

Method3-shot5-shot10-shot20-shot
LayerFi-FSOD w. DINOv363.0167.1672.8886.33
LayerFi-FSOD w/o DINOv363.4667.9473.3286.06

After removing DINOv3, performance improves by 0.45/0.78/0.44 percentage points under the 3/5/10-shot settings, respectively, while decreasing by only 0.27 percentage points under the 20-shot setting. These results indicate that the student can maintain detection performance without relying on teacher features. Therefore, the main value of our work lies in enabling knowledge transfer tailored to hierarchical representation characteristics, rather than simply removing the teacher model.

[1] Simeoni, Oriane, et al. "DINOv3." TMLR, 2026.

AuthorsOpenReview ↗

Response (2/2) to Review Aaod, we remain open to further discussion.

  1. Weaknesses: The reported evaluation metric is mAP50 alone. Missing metrics AP@(0.5–0.95) and AP75. In addition, more teacher-student pairs should be verified. The effectiveness of the proposed method is less convincing.

Response: We thank the reviewer for this valuable suggestion. To provide a more comprehensive evaluation of the detection performance and generalization capability of LayerFi-FSOD, we conduct additional evaluations under stricter IoU thresholds and comparisons using different vision foundation models as teachers.

We additionally report mAP50, mAP75, and mAP50:95 for both base and novel classes under the 3/5/10/20-shot settings on NWPU VHR-10, and compare our method with MSOCL [1] and SAE-FSDet [2]. For a fair comparison, we re-evaluate MSOCL and SAE-FSDet under the same experimental conditions. The results are as follows:

MetricMethodClass3-shot5-shot10-shot20-shot
mAP50MSOCLNovel47.8352.2566.7277.63
SAE-FSDetNovel58.3159.9370.4084.43
LayerFi-FSODNovel63.4667.9473.3286.06
MSOCLBase83.5185.6184.7382.30
SAE-FSDetBase82.8582.9483.3282.72
LayerFi-FSODBase82.7382.7682.5982.58
mAP75MSOCLNovel14.7515.1723.6231.53
SAE-FSDetNovel19.1018.7628.0737.42
LayerFi-FSODNovel23.4720.3729.5342.56
MSOCLBase50.3050.7952.5352.68
SAE-FSDetBase48.0946.0845.4650.03
LayerFi-FSODBase48.6248.5945.1747.93
mAP50:95MSOCLNovel20.5622.5631.4137.93
SAE-FSDetNovel26.1425.6234.3143.64
LayerFi-FSODNovel29.6230.1236.0646.62
MSOCLBase48.4749.4748.7049.69
SAE-FSDetBase47.9147.8246.2748.75
LayerFi-FSODBase46.6247.2245.1348.68

The results show that LayerFi-FSOD outperforms the competing methods in novel-class mAP75 and mAP50:95 across all four shot settings, demonstrating that its performance advantages persist under stricter IoU thresholds and further validating its effectiveness in improving object localization quality. Meanwhile, base-class performance shows a slight decrease compared with SAE-FSDet in some settings, which may stem from the trade-off between base and novel classes caused by the model's adaptation to novel categories during few-shot fine-tuning [3], [4]. Overall, LayerFi-FSOD substantially improves novel-class detection performance while largely preserving base-class detection capability.

To further validate the applicability of our method to different teacher models, we replace DINOv3 [5] with DINOv2 [6], CLIP [7], and RemoteCLIP [8] while keeping the student detector architecture unchanged, and evaluate performance under the 5-shot and 10-shot settings on NWPU VHR-10. The results are as follows (mAP50, %):

Method5-shot10-shot
SAE-FSDet59.4071.02
LayerFi-FSOD w. RemoteCLIP63.9073.02
LayerFi-FSOD w. DINOv265.4972.54
LayerFi-FSOD w. CLIP65.5072.11
LayerFi-FSOD w. DINOv367.9473.32

The results show that LayerFi-FSOD outperforms the SAE-FSDet baseline with all four vision foundation models. These findings indicate that the proposed hierarchical visual prior construction and knowledge transfer mechanisms are not dependent on a specific vision foundation model, further validating the applicability and effectiveness of our method with different teacher models. Notably, DINOv3 achieves the highest performance among these teachers, improving over SAE-FSDet by 8.54 and 2.30 percentage points at 5-shot and 10-shot, respectively.

[1] Chen, Jie, et al. "Multiscale Object Contrastive Learning-Derived Few-Shot Object Detection in VHR Imagery." TGRS, 2022.

[2] Liu, Yanxing, et al. "Few-Shot Object Detection in Remote-Sensing Images via Label-Consistent Classifier and Gradual Regression." TGRS, 2024.

[3] Zhang, Tianyang, et al. "Generalized Few-Shot Object Detection in Remote Sensing Images." ISPRS, 2023.

[4] Zhu, Zhaohui, et al. "Balancing Attention to Base and Novel Categories for Few-Shot Object Detection in Remote Sensing Imagery." TGRS, 2024.

[5] Simeoni, Oriane, et al. "DINOv3." TMLR, 2026.

[6] Oquab, Maxime, et al. "DINOv2: Learning Robust Visual Features without Supervision." TMLR, 2024.

[7] Radford, Alec, et al. "Learning Transferable Visual Models from Natural Language Supervision." ICML, 2021.

[8] Liu, Fan, et al. "RemoteCLIP: A Vision Language Foundation Model for Remote Sensing." TGRS, 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of Rebuttal and Key Clarifications

Dear Area Chair (AC) and Reviewers, we sincerely thank you for your constructive feedback. We appreciate the reviewers' recognition of several strengths of our work, including the practical relevance and technical contribution of transferring vision foundation model knowledge to a task-specific detector, the clear research motivation and insightful layer-wise representation analysis, the promising detection performance particularly under extremely low-shot settings, and the practical advantage of inference without the vision foundation model. Below, we summarize our key clarifications and additional experiments in response to the reviews, with references to our detailed responses.

1 Methodological Contributions and Motivation (Aaod, AsPM)

  • Targeted Hierarchical Knowledge Transfer: We acknowledge prior evidence of depth-dependent VFM representations, including VFM-DETR [1]. Our contribution is to use these differences for few-shot remote sensing detection without test-time VFMs: CAFM modulates patches via CLS–patch similarity, PLEM dynamically routes low-to-middle-layer priors using both current-level and global FPN context, and DSGM transfers local intra-RoI spatial structures and global inter-RoI semantic relations through depth-aligned distillation objectives.

(See Aaod Weakness 1; AsPM Weakness 1.)

  • Functional and Component-level Validation: Box Boundary F1 measures spatial boundary separability, while Same-class Pair AUC assesses class-level semantic discriminability. Both metrics peak at distinct network depths across DINOv3 [2], DINOv2 [3], CLIP [4], and RemoteCLIP [5]. Ablations and Feature L2/Cosine comparisons support the targeted design, especially at 3/5-shot.

(See AsPM Weaknesses 2 and 4, AsPM Question 1; Aaod Weakness 1.)

2 Cross-Dataset Generalization and Teacher Flexibility (EQ3n, AsPM, Aaod)

  • Layer-selection Generalization: Additional layer-selection experiments on DIOR Split 2 show that assigning shallow-to-middle layers to ADPIM and middle-to-deep layers to GSDM performs best among tested configurations on both DIOR Split 2 and NWPU VHR-10. This supports module-specific layer selection, although results vary by dataset.

(See EQ3n Weakness 2 and Question 1.)

  • Broader Datasets and Teachers: On RSOD [6], LayerFi-FSOD outperforms G-FSDet, MSOCL [7], and SAE-FSDet [8] in all six 3/5/10-shot settings across two category splits. On NWPU VHR-10, replacing DINOv3 with DINOv2, CLIP, or RemoteCLIP also improves over SAE-FSDet [8], indicating teacher flexibility.

(See AsPM Weaknesses 2, 5; Aaod Weakness 2.)

3 Evaluation Completeness and Statistical Reliability (Aaod, AsPM)

  • Stricter IoU and Base-Class Retention: We add base/novel mAP50, mAP75, and mAP50:95 on NWPU VHR-10. Our method leads in novel-class mAP75 and mAP50:95 across all shots, with largely retained base-class performance despite modest decreases on some settings.

(See Aaod Weakness 2; AsPM Weakness 5.)

  • Multi-seed Reliability: Three independent few-shot sampling runs on DIOR Splits 1/2 and NWPU VHR-10 yield higher mean novel mAP50 than MSOCL [7] and SAE-FSDet [8] in all 12 settings, despite seed variability.

(See AsPM Weakness 3 and Question 2.)

4 Knowledge Retention and Computational Efficiency (EQ3n, AsPM, Aaod)

  • Student Knowledge Retention: During training, ADPIM enhances RoI features through gated residual fusion, while DSGM transfers local spatial and global semantic relations via distillation. Backpropagation updates the student parameters; all teacher-dependent branches are removed at inference. Removing DINOv3 changes 3/5/10/20-shot mAP50 by +0.45/+0.78/+0.44/−0.27 points, supporting independent student inference.

(See EQ3n Weakness 3; AsPM Question 3; Aaod Weakness 1.)

  • Training–Inference Trade-off: Relative to SAE-FSDet, fine-tuning requires 3.32–4.20× time and 3.43–3.57× peak memory. These costs are confined to offline training; inference architecture and cost remain unchanged.

(See EQ3n Weakness 1; AsPM Weakness 5.)

References

[1] Liu, Changhan, et al. "Don't Need Retraining: A Mixture of DETR and Vision Foundation Models for Cross-Domain Few-Shot Object Detection." NeurIPS, 2025.

[2] Simeoni, Oriane, et al. "DINOv3." TMLR, 2026.

[3] Oquab, Maxime, et al. "DINOv2: Learning Robust Visual Features without Supervision." TMLR, 2024.

[4] Radford, Alec, et al. "Learning Transferable Visual Models from Natural Language Supervision." ICML, 2021.

[5] Liu, Fan, et al. "RemoteCLIP: A Vision Language Foundation Model for Remote Sensing." TGRS, 2024.

[6] Long, Yang, et al. "Accurate Object Localization in Remote Sensing Images Based on Convolutional Neural Networks." TGRS, 2017.

[7] Chen, Jie, et al. "Multiscale Object Contrastive Learning-Derived Few-Shot Object Detection in VHR Imagery." TGRS, 2022.

[8] Liu, Yanxing, et al. "Few-Shot Object Detection in Remote-Sensing Images via Label-Consistent Classifier and Gradual Regression." TGRS, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.