acceptodds
Withdrawn from ICLR 2027

Diagnose Before You Design: Depth–Scale Dependencies in Foundation-ViT Detectors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Detectors built on plain foundation Vision Transformers (ViTs) must construct multiscale spatial memories from single-resolution backbone features, yet existing interfaces typically predefine how backbone depth is mapped to detector scale without directly measuring the resulting functional dependencies. Rather than repeatedly retraining alternative routing structures, we recast interface analysis as an interventional diagnosis problem: controlled inference-time interventions on a reference detector quantify the functional dependence of existing depth-to-scale paths and spatial memories. Across DINOv2, DINOv3, and distinct detector interfaces, we consistently observe strong and image-specific depth–scale dependencies, while their dominant spatial realization changes with backbone representation and detector topology. Moreover, weak marginal dependence in a fixed model does not imply retraining-time structural redundancy. Controlled factorial experiments reveal that the effect of depth treatment is modulated by spatial memory topology, establishing the two as coupled design variables and motivating a diagnosis-guided redesign workflow. Guided by these measurements, we formulate and validate distinct interface redesign hypotheses across DINOv3- and DINOv2-based detectors that preserve aggregate Full-COCO accuracy while reducing model capacity, demonstrating the practical value of diagnosis-guided interface design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Foundation Models, Object Detection, Vision Transformers, Multiscale Representations, Interventional Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_awcPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Diagnose-Before-Design, a workflow for analyzing and simplifying the interface between plain foundation Vision Transformers and multiscale object detectors. The approach uses inference-time branch zeroing, spatial-memory masking, energy-matched controls, and cross-image permutation to measure functional dependencies in a trained detector. Experiments on DINOv3-S, DINOv2-S/14, and DEIMv2 show that deep semantic features are important, while their dominant spatial routes depend on the backbone and interface topology. Factorial experiments further demonstrate interactions between depth treatment and spatial topology. Guided by these observations, the authors develop redesigns that approximately preserve Full-COCO AP while reducing parameters.

Critical strengths
  1. Useful diagnostic perspective. The paper measures functional reliance on interface paths rather than relying solely on representation similarity or final architecture-level AP.

  2. Complementary intervention controls. Branch zeroing, memory masking, energy matching, and permutation provide converging evidence for image-specific deep-feature dependence.

  3. Insightful depth–topology interaction. The DINOv3 factorial reports interactions of +1.864 Raw AP and +1.395 EMA AP. DEIMv2 provides additional evidence, with a mean interaction of +1.254 Raw AP.

  4. Appropriate qualification of the findings. The authors acknowledge that dependence is checkpoint-specific and that the observed overall accuracy differences should not be presented as established improvements.

Critical weaknesses
  1. The diagnosis-to-design link is only partially demonstrated. The central DINOv3 redesign uses a pre-existing deep-heavy weighting vector whose exact derivation is not recoverable.

  2. Limited comparison with simpler design strategies. The paper does not establish how much diagnostic guidance improves candidate selection over straightforward alternatives such as uniform weighting, deepest-feature-only inputs, or a comparably budgeted heuristic search.

  3. Search-efficiency benefits are not quantified end to end. Linear scaling in inference evaluations is attractive, but total diagnostic cost and training savings relative to a concrete train-and-compare budget are not reported.

  4. The endpoint benefits include trade-offs. DINOv3 reduces GFLOPs by 3.65%, and DINOv2 by 1.44%. Small-object AP decreases in both redesigns. These results support capacity reduction with near-parity aggregate accuracy, with less evidence for uniformly improved detection performance.

Questions for authors
  1. Can you provide a reproducible rule that converts intervention scores into feature weights or architecture choices? How does it perform relative to the inherited deep-heavy vector?

  2. How sensitive is the DINOv3 interaction to alternative deep-heavy coefficients, including simpler hand-specified schedules?

  3. Does the complete DINOv3 four-cell interaction remain positive across additional training seeds?

  4. Under an equal total compute budget, how does diagnosis-guided selection compare with simple heuristic or random candidate selection? Please include the cost of all diagnostic evaluations.

  5. How stable are the dependency rankings across training seeds and training stages?

  6. Can you explain the small-object AP reductions?

Reviewer_e7RdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the interface between plain foundation ViTs (DINOv2, DINOv3) and multiscale detectors: which backbone depths feed which detector scales, and how those features become spatial memories. Instead of training candidate interfaces multiple times, the authors propose an inference-only interventional diagnosis, including source-branch zeroing, spatial-memory masking, energy-matched rescaling, and cross-image permutation on a fixed checkpoint, to measure dependencies in trained detectors.

Across DINOv3-S+MSP, DINOv2-S/14+MSP, and DEIMv2, this paper reports strong, image-specific, highly asymmetric deep-semantic dependence whose spatial realization changes with backbone and detector topology. Factorial retraining shows a crossover interaction between depth treatment and spatial topology, and the maps motivate two redesigns. The resulting interface simplifications achieve approximately unchanged aggregate COCO AP while reducing parameter counts.

Critical strengths
  1. The depth-scale interface is a real bottleneck for foundation ViT detectors, and the paper makes a convincing case that predefined routing is not evidence of functional use. Hence, studying how trained detectors actually use their interfaces is worthwhile. Energy matching, cross-image permutation, and identity-hook checks make the analysis more. The large effects on selected paths provide convincing evidence of dependence in those checkpoints. Meanwhile, the entire diagnostic remains controlled and cheap.

  2. The paper tests architectural interactions comprehensively. The factorial experiments are a substantive strength. In DINOv3 + MSP, the depth treatment changes from slightly detrimental to beneficial after topology simplification. The DEIMv2 interaction is positive in both reported seeds. These results support the narrower conclusion that the effects of the tested depth treatments depend on the surrounding computation graphs.

  3. The resulting simplifications have practical value. DINOv3 reduces total parameters by 16.78% with nearly identical aggregate AP. DINOv2 reduces parameters by 6.63% with a small aggregate AP decrease. Related metrics including FP32 latency, parameter count/GFLOPs are provided.

Critical weaknesses
  1. The paper frames Diagnose-Before-Design as a reusable workflow, but the distinction between the diagnostic protocol, empirical findings, and case-specific design decisions is not sufficiently clear. The protocol is described in Sec. 3.1 with further implementation details in Appendix B, while Sections 3 and 4 interleave diagnostic procedures, observations, and design hypotheses. Consequently, it is difficult to determine which redesign choices follow from explicit diagnostic criteria, which require additional architectural judgment, and which conclusions are supported only by the reported retraining experiments. For example, M3 combines two decisions 1) removing P5 (diagnosis-derived) and 2) applying a pre-existing deep-heavy prior (not diagnosis-derived), but the paper does not state which part of the gain is attributable to which. This matters because the connection between diagnosis and design is central to the paper’s contribution. The paper would benefit from a consolidated description of the workflow that makes these decision points explicit and separates reusable guidance from conclusions specific to the evaluated architectures.

  2. The empirical validation is confined to COCO and DINO-family backbones. Although the inclusion of DINOv2/DINOv3 and two detector interfaces provides useful architectural variation, the generality of the diagnosis-to-design workflow beyond these settings remains uncertain. In particular, the experiments demonstrate successful redesigns in the examined systems but provide limited evidence that the diagnostic findings translate into actionable design guidance for a new setting. The abstract states a consistent principle and system-dependent realization, but the evidence is two backbones (DINOv2-S/14, DINOv3-S) + two interfaces (MSP, DEIMv2) on one dataset. Either temper the claim to 'in the evaluated systems', or add one more family (e.g., a ViT-B/L or a CNN-backbone detector). The paper should clarify its intended scope and distinguish broadly applicable diagnostic procedures from design conclusions supported only by the evaluated configurations.

  3. Appendix D states that the DINOv3 depth-weighting vector predates this study and that its exact derivation cannot be recovered. Freezing it before the factorial runs supports interpreting those runs as a test of a predetermined treatment but does not establish how the diagnostic evidence informed its selection. Although the paper provides qualitative design guidance, it remains unclear how a reader should choose an analogous compensating treatment from a new dependency map. The success of M3, together with the AP reduction from P5 removal alone in M2, validates the tested combination but does not isolate the contribution of diagnostic guidance relative to a pre-existing architectural prior. The uniform/deep-prior controls address aspects of the weighting choice, but do not directly evaluate the overall selection strategy. To the extent that the workflow is claimed to reduce design cost, a comparison with a simple selection heuristic within a common candidate space and matched total budget would strengthen that claim, with diagnostic and training costs reported alongside AP.

Questions for authors

Questions are basically corresponding to the weaknesses above.

  1. Could you provide a step-by-step account of the diagnosis-to-design workflow? Specifically, given a new dependency map, what are the explicit decision rules that produce a redesign candidate, and which steps require architectural judgment that the map does not determine? For M3, which of its two components, removing P5 and applying the pre-existing deep-heavy prior, was selected by the diagnostic criteria, and which was not?

  2. The experiments cover DINOv2-S/14 and DINOv3-S with MSP and DEIMv2 interfaces on COCO. Which conclusions are specific to these configurations, and which are intended to generalize? In particular, please distinguish the applicability of the diagnostic procedure from the transferability of the observed dependency patterns and resulting design guidance. Could one targeted extension, such as a larger ViT/a backbone with non-DINO pretraining/other tasks, test a broader claim? If such evidence is unavailable, please clarify the limits of the claims supported by the current experiments.

  3. What evidence establishes the additional value of diagnostic guidance in selecting redesign candidates? The factorial experiments test the chosen architectural treatments, but do not directly compare how those treatments were selected. To support any claim of more effective or less costly redesign, could you provide one comparison against a simple alternative selection strategy within a common candidate space and total budget, reporting AP, model efficiency, and diagnostic/training costs? Given that the deep-heavy prior predates this study, the comparison should clarify what diagnostic guidance contributes beyond having that prior available?

My assessment would be most affected by clarifying the diagnosis-to-design decisions and substantiating the additional value of diagnostic guidance (Q1 and Q3). Evidence of stability across checkpoints would further strengthen confidence in the conclusions.

Reviewer_imxVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets finding the most efficient and effective mapping between ViT backbones and feature pyramid levels for ViT-based object detectors without brute-forcing all the possible combinations through the costly training but inference-time probing. Study objectives include DINOv2, DINOv3, MSP, and STA. The training-free diagnosis then helps prune the network/connections for an efficient variants. Experiments on COCO show that the proposed method can maintain comparable accuracy while saving the number of parameters and GFLOPs.

Critical strengths
  1. Cheap diagnosis that leads to faster model iteration.
Critical weaknesses
  1. Experiments are not comprehensive enough. This paper only uses COCO for all experiments and diagnosis, making it hard to justify whether the method translate well to other use cases. For example, does the model guided by diagnosis from COCO work from Objects365?
  2. Latency measurement is questionable. Could the authors elaborate on why 3% GFLOPs reduction leads to 20% latency speed up?
  3. The proposed idea has a deep connection with model pruning but the paper doesn't link to any pruning research.
Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.