acceptodds
Under review as a conference paper at ICLR 2027

AnyDepth-DETR/-YOLO: Any-depth object detection with a single network

Abstract

Most object detectors are static, fixed-depth networks optimized for a single operating point, requiring separate models for different deployment scenarios. We present an any-depth detection framework that enables a single network to span a continuous range of accuracy-efficiency trade-offs by controlling depth at inference time without retraining. Our key design principle is stage-wise modularity: each stage's output remains compatible with downstream stages regardless of depth. To this end, each backbone and neck stage is divided into an essential path, which always executes, and a skippable refinement path, preserving the full multi-scale feature hierarchy at every depth configuration. Training such a network requires jointly optimizing many sub-networks, which introduces conflicting gradient signals. We address this via self-distillation between only the two extremes, with prediction- and feature-level alignment losses that enforce stage-wise modularity. Instantiated on RT-DETR and YOLOv12, our full-depth configurations match or surpass their respective state-of-the-art baselines with negligible parameter overhead, while the most efficient configurations achieve up to speedup at a cost of only 2.0 AP, all from a single set of weights.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
object detection, efficient networks, dynamic depths, adaptive inference, latency-accuracy trade-offs, self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_7jdfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents AnyDepth-DETR and AnyDepth-YOLO, which adjust inference depth within a shared detector. Backbone and neck stages are divided into essential and skippable refinement paths, retaining the multi-scale feature hierarchy. The training stagee uses the full and shallow configurations with self-distillation, switchable normalization, detection-specific target alignment, and feature -level alignment. Experiments on COCO with RT-DETR and YOLOv12 demonstrate practical speed and accuracy tradeoff at the endpoints and several intermediate configurations.

Critical strengths

S1. The reported inference gains are demonstrated in two different detector families. In Table 1, AnyDepth-DETR R-101 increases throughput from 78.6 to 125.6 FPS while AP decreases from 54.6 to 52.6. AnyDepth-YOLO L increases throughput from 101.5 to 155.3 FPS with AP changing from 53.9 to 52.1. Those results support these operating points within one shared model, which is useful when deployment budgets changing.

S2. The implementation addresses concrete detection-specific difficulties. Dedicated CSP aggregation handles changes in concatenated feature width, while target-aligned distillation avoids transferring predictions assigned to different objects. Table 3 includes useful architectural and some training ablations, and Table 9 compares feature-alignment granularities. These controls provide evidence that the adaptation involves more than simply skipping blocks.

Critical weaknesses

W1: The contribution relative to AdaptiveDepth [1] is not established clear enough. In [1], it already uses mandatory and skippable stage paths, endpoint self-distillation, and binary skip-aware normalization. Besides, Its Appendix and Table 8 also evaluate adaptive backbones in Faster R-CNN, Mask R-CNN, and RetinaNet on COCO. So characterizing this predecessor as a classification design that is poorly suitable to detection, such saying is too broad. The present extension to neck adaptation and CSP stages can still be valuable. However, Table 3 does not clarify whether its naive distillation configuration can faithfully stand for the earlier detector adaptation, and the Slimmable-YOLO comparison is actually about another different approach. Thus, the incremental contribution is not isolated sufficiently.

[1] Kang, Woochul, and Hyungseop Lee. "Adaptive depth networks with skippable sub-paths." Advances in Neural Information Processing Systems 37 (2024): 33213-33231.

W2: The evidence for arbitrary combinations of stage depths is not so complete as the endpoint results. Table 2 and Table 10 only evaluate some selected component sweeps, while Equation 9 and Table 7 compare the features coming from the two trained endpoints. Global pooled feature agreement and high CKA cannot well establish that fixed downstream modules can use arbitrary mixed-path features in an interchangeable way. Especially, the BN selection depends on the skip flag of the current stage, but mixed configurations will bring in upstream paths that are not encountered together during the endpoint training. The existing intermediate results are encouraging, but the coverage of Figure 3 is not clear. So a more broad mixed-configuration evaluation is needed to substantiate the central composability claim.

Questions for authors

Q1: (From W1.) How does the method compare with a faithful adaptation of AdaptiveDepth to the same detector and training budget? If an existing row in Table 3 serves this purpose, please identify the correspondence and explain which additional gains come from neck adaptation, CSP aggregation, and the revised distillation losses.

Q2: How were the configurations in Figure 3 selected, and was any BN recalibration performed? Could the authors report AP and latency for a representative set of mixed backbone and neck configurations, including unfavorable cases, rather than only an ordered sweep or the frontier?

Q3: What are the exact essential and full computations, including BN placement, for the RepVggBlock encoder stages in Table 4? Section 3.2.1 describes additive residual blocks, whereas the standard RT-DETR RepVggBlock has no identity residual connection. Clarifying any architectural modification would make the method easier to verify.

I currently lean toward weak rejection, close to the acceptance boundary (in my opinion, boarderline, but no such option). The practical acceleration and results across two detector families are promising, but I hope authors can fix my questions. I am willing to raise the rating if author provide convicing response.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7jdf (1/2)

We sincerely thank the reviewer for the detailed and insightful review. The comments on the relation to AdaptiveDepth and on the encoder stages were particularly helpful and led directly to the reframed Introduction and the new analyses in the revision (changes in blue).

W1 / Q1: Contribution relative to AdaptiveDepth. We agree that our original characterization of AdaptiveDepth was too broad. The revised Introduction, Related Work, and Section 3 now credit it explicitly for the components the reviewer lists, and we no longer group it with methods that "transfer poorly" to detection. Our contribution is achieving stage-wise modularity throughout a modern detector (General Response #1).

Below is how Table 2 (Table 3 in the original submission) corresponds to AdaptiveDepth (now stated in Section 4.4).

  • AdaptiveDepth's detection results use an adaptive backbone (pretrained with self-distillation) fine-tuned with ground truth only, with the neck and head static.
  • The "Self-Distillation (Naive)" row applies AdaptiveDepth's training scheme (endpoint self-distillation on outputs, with skip-aware BN) to our architecture. For AD-DETR, whose ResNet backbone follows AdaptiveDepth, it differs mainly in that the neck is also adaptive. For AD-YOLO, whose CSP-style stages AdaptiveDepth does not support, only the training scheme is shared.

The additional gains come from three sources.

  • Neck adaptation (speed range). We trained a backbone-only AD-DETR with naive self-distillation, which approximates a faithful adaptation of AdaptiveDepth to the same detector and training budget (purple curve in Figure 5, Section 4.3). It reaches lower AP at every speed. With the full decoder, its speedup is limited to 1.15×, and decoder early exit could increase it only at a further cost in AP. For YOLOv12, backbone-only adaptation does not apply at all, since its backbone consists of CSP stages.
  • CSP aggregation (AD-YOLO). The switchable aggregator adds +0.3 / +0.4 AP (super / base) over switchable BN alone (Table 2).
  • Revised distillation. Naive self-distillation brings little or no gain (AD-DETR 52.5 / 50.1). Target alignment raises it to 52.9 / 50.3, and feature alignment to 53.3 / 50.6. We also measured why. At convergence, 68% of the query pairs formed by naive self-distillation in AD-DETR pair predictions for different objects (Section 4.4).

W2 / Q2: Arbitrary combinations, Figure 3, and BN recalibration.

  • All configurations. We now evaluate all 384 (AD-DETR R-50) and 256 (AD-YOLO L) sub-networks, including unfavorable ones, with AP and FPS for each (new Section 4.3 and figure; General Response #2). The intermediate sub-networks form a smooth frontier. The configurations well below it mostly spend compute unevenly, e.g., a single decoder layer on top of a largely full-depth backbone.
  • BN recalibration. We do not perform any BN recalibration; every sub-network uses the BN statistics learned during training as-is. As the reviewer notes, each stage's BN is selected by its own skip flag, so mixed configurations combine upstream paths never seen together during training. The full sweep tests exactly this case, and the resulting frontier shows these combinations work without recalibration.
  • Figure 3. The original Figure 3 (now Figure 4) plotted one representative intermediate sub-network per model. The new all-sub-network figure (Figure 5) replaces such hand-picked points with every configuration.
  • Pooled features and CKA. Our CKA (Table 6, formerly Table 7) treats each 4×4 spatial cell as a sample, so it is not purely global. More directly, the all-sub-network evaluation measures interchangeability by the final detection AP of every mixed configuration, which is the property that matters. This is the stage-wise modularity our method is designed to achieve in detectors.
AuthorsOpenReview ↗

Response to Reviewer 7jdf (2/2)

Q3: RepVGG encoder stages. The reviewer is right that the RT-DETR encoder stages are not residual. We added a dedicated description (Section 3.2.1, new Eq. 3). Each encoder stage applies RepVGG blocks g(1),…,g(S)g^{(1)},\dots,g^{(S)} without identity shortcuts on one branch and adds a parallel shortcut branch. Let z(i)=g(i)(z(i−1))z^{(i)} = g^{(i)}(z^{(i-1)}), z(0)=uz^{(0)} = u. Then

xess=h(z(m)+v),xfull=h(R(z(m))+v),R=g(S)∘⋯∘g(m+1),x_{\text{ess}} = h(z^{(m)} + v), \qquad x_{\text{full}} = h(R(z^{(m)}) + v), \qquad R = g^{(S)} \circ \cdots \circ g^{(m+1)}, where u,vu, v are 1×11\times1 projections of the stage input and hh is the output projection, all shared by both paths (S=3S = 3, m=1m = 1; Table 3). As for BN placement, as in all other stage types, switchable BN (skip / no-skip) is placed in the essential-path blocks g(1),…,g(m)g^{(1)},\dots,g^{(m)}, while the refinement blocks use standard BN and the shared projections are unchanged from RT-DETR. Because RR transforms rather than adds to z(m)z^{(m)}, nothing in the architecture keeps the two paths close. This is where the naive joint-train model's CKA degrades most (0.70–0.87, Table 6), and why feature alignment is needed for AD-DETR but not for AD-YOLO (Section 4.4).

We are grateful for the reviewer's suggestions, which made both the positioning and the analysis of the paper considerably stronger. We hope the revision addresses the concerns, and we would welcome a reconsideration of the score if so.

Reviewer_g1MNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem that object detection models typically need to be trained and maintained as multiple fixed-depth models for different computational budgets during deployment, and proposes a unified arbitrary-depth object detection framework, AnyDepth-DETR/YOLO. Its core idea is to use structural design and self-distillation training so that a single shared-weight network can flexibly adjust its computational depth during inference while preserving as much as possible the multi-scale feature representations required for object detection.  The main contributions of the paper include: 

  1. Stage-wise Modularity. Each stage of the backbone network and feature fusion network is divided into an essential path that is always executed and a skippable refinement path, so that different depth configurations can still retain the complete stage hierarchy and multi-scale feature structure. 
  2. Endpoint Self-distillation and Feature Alignment. The full network (super-net) and the most lightweight network (base-net) are jointly trained, using prediction distillation, feature alignment, and specialized handling for DETR query matching and YOLO anchor assignment conflicts to reduce supervision conflicts in shared-weight training. 
  3. Cross-architecture Validation. The method is applied separately to RT-DETR and YOLOv12, and evaluated on COCO val2017 in terms of accuracy, GFLOPs, and FPS under different inference depths.  According to Table 1 of the paper, under the setting of an RTX 4090 and batch size 1, the AP of the full-depth model improves by about 0.1--0.3 percentage points compared with the corresponding baseline; the most lightweight configuration trades a certain accuracy drop for higher throughput. For example, AnyDepth-DETR R-101 increases from 78.6 FPS to 125.6 FPS, with AP decreasing from 54.3 to 52.6; AnyDepth-YOLO L increases from 101.5 FPS to 155.3 FPS, with AP decreasing from 53.9 to 52.1.
Critical strengths
  1. The problem is of significant practical relevance and application value. This paper focuses on the flexible deployment of object detection models under different computational budgets. Conventional detection models typically need to be trained or maintained in different scales for different accuracy and efficiency requirements, which increases the cost of model storage, deployment, and management. By constructing a single network that supports different inference depths, this paper attempts to achieve a flexible trade-off between computational cost and detection performance on the basis of shared model parameters. This research targets real-time object detection and resource-constrained scenarios, and thus has certain practical application value.

  2. The experimental validation is relatively thorough and covers different types of detection architectures. The paper validates the proposed method on RT-DETR and YOLOv12 respectively, and reports detection accuracy, computational cost, and inference speed under different inference configurations. In addition, the paper conducts ablation analyses on key components such as switchable batch normalization, switchable aggregators, endpoint self-distillation, target assignment alignment, and stage-level feature alignment, which helps to understand the impact of different designs on model performance. The experimental results preliminarily show that the method can improve inference efficiency by skipping part of the computation while basically maintaining the detection performance of the full model.

  3. The combination of stage-wise modularity and endpoint self-distillation is somewhat innovative. This paper does not simply reduce the number of network layers or adopt early exit to lower computational cost. Instead, it divides the computation in the backbone and neck into an essential path and a refinement path, and combines endpoint self-distillation with feature alignment to alleviate the feature distribution discrepancy between different inference paths. Moreover, the paper designs corresponding strategies for the different target assignment mechanisms of DETR and YOLO, enabling the method to adapt to two types of architectures with different detection mechanisms. This idea of combining stage-wise modularity, shared-weight training, and architecture-specific distillation strategies is somewhat inspiring, and provides a direction worth exploring for building object detection models that support multiple computational budgets.

Critical weaknesses
  1. The novelty of the method requires clearer delineation. The paper enables object detection at different network depths through stage-wise essential/refinement path decomposition, endpoint self-distillation, and feature alignment. However, the substantive differences between these designs and existing adaptive computation, dynamic-depth networks, and skippable-subnetwork methods remain insufficiently clarified. The authors should better articulate the methodological distinctiveness of the proposed approach and establish its novelty through more targeted comparisons with the most relevant methods.

  2. he contributions of the core mechanisms and the validity of intermediate configurations require further investigation. The proposed method aims to support detection at different computational depths using a unified network. However, the individual contributions of its components and the ability to maintain detection performance across different depth configurations require more thorough analysis. It remains unclear whether the observed gains primarily arise from path decomposition, self-distillation, feature alignment, or their combined effects. Further component-wise ablations and more comprehensive evaluations of intermediate configurations are needed.

  3. The breadth of experimental comparisons is insufficient to fully establish the method's competitiveness. The proposed method is primarily evaluated on two detection architectures, RT-DETR and YOLOv12. However, its competitiveness against a broader range of state-of-the-art object detectors and related dynamic-depth or adaptive-computation methods remains insufficiently established. The authors should include more representative baselines and conduct fair comparisons under, as far as possible, consistent experimental settings and computational budgets to better characterize the trade-off between detection accuracy and inference efficiency.

Questions for authors
  1. Insufficient comparisons with state-of-the-art object detection methods. The proposed method is evaluated on only two detector architectures, RT-DETR and YOLOv12. It remains unclear whether the proposed approach offers consistent advantages over a broader range of state-of-the-art object detectors and related dynamic-depth or adaptive-computation methods. Can the authors provide more comprehensive comparisons with representative advanced methods under comparable computational budgets to better establish the competitiveness of AnyDepth-DETR/YOLO?

  2. Limited evaluation across datasets. The experiments focus primarily on COCO val2017, making it difficult to determine whether the effectiveness of the proposed method generalizes to different datasets, scene distributions, and object-scale distributions. Can the authors evaluate the method on additional object detection benchmarks to demonstrate that its accuracy–efficiency trade-off is not specific to a single dataset?

  3. Insufficient validation of intermediate-depth configurations. A central claim of the paper is that training only the full and minimal network configurations enables reliable performance across intermediate depths. However, the current evaluation provides limited evidence regarding the robustness of these intermediate configurations. Can the authors provide more systematic evaluations across different inference depths and clarify whether the observed performance trends remain consistent across architectures and computational budgets?

  4. Incomplete analysis of practical efficiency and training overhead. The paper emphasizes that a single shared-weight network can replace multiple independently trained models, but the practical benefits of this approach are not fully established by AP, GFLOPs, and FPS alone. Given the additional training computations required by endpoint self-distillation, can the authors provide a more comprehensive comparison of training cost, inference latency, memory consumption, and accuracy–efficiency trade-offs against independently trained fixed-configuration models?

1 comment

We sincerely thank the reviewer for the thoughtful review and for recognizing the practical value of the problem. The questions on component contributions, intermediate configurations, and practical cost led us to new analyses that we believe make the paper considerably more thorough. All changes are marked in blue in the revised PDF.

W1: Novelty relative to existing adaptive methods. We rewrote the Introduction and Related Work to delineate our contribution. Stage-wise modularity is a known principle (AdaptiveDepth, MatFormer), and our contribution is achieving it throughout a modern multi-scale detector, where backbone-only adaptation, CSP-style stages, and conflicting target assignment prevent existing methods from carrying over. To our knowledge, ours are the first detectors whose backbone and neck are both depth-switchable from a single set of weights (General Response #1).

W2: Contribution of each mechanism. Table 2 isolates each component, and the revision adds measurements explaining why each one matters (Section 4.4).

  • Path decomposition. At a fixed decoder depth, the refinement paths account for 2.5–2.6 AP of the super-net/base-net gap, while two extra decoder layers add at most 0.17 AP (Appendix A.3).
  • Switchable BN and aggregator. These decouple the two paths, raising the AD-YOLO base-net by +0.8 and a further +0.4 AP, respectively.
  • Assignment-consistent distillation. Naive self-distillation brings little or no gain because, at convergence, 68% of its query pairs in AD-DETR pair predictions for different objects. Target alignment resolves this.
  • Feature alignment. This is needed for AD-DETR, whose encoder stages have no identity shortcut, but not for AD-YOLO, where the switchable aggregator already keeps the paths aligned.

W3 / Q1: Broader comparisons under comparable budgets. Table 1 now includes the adaptive detector DynamicDet with FPS measured on the same GPU, alongside the static RT-DETR and YOLOv12 families (General Response #3). Appendix A.4 compares against Slimmable Networks adapted to YOLOv12 at matched compute. We chose RT-DETR and YOLOv12 as representatives of the two dominant real-time paradigms (set-based and dense prediction), and our approach is not specific to either.

Q2: Other datasets. We added BDD100K, a driving dataset with a different scene and object-scale distribution (General Response #4, Appendix A.5). The behavior is consistent with COCO. The AnyDepth-DETR base-net outperforms a separately trained model of the same architecture by 1.3 AP, and the frontier of all 384 sub-networks is above the random-path baseline over 81% of the FPS range.

Q3: Intermediate depth configurations. We now evaluate every sub-network, 384 for AnyDepth-DETR and 256 for AnyDepth-YOLO, without any BN recalibration (new Section 4.3, General Response #2). Both architectures show a smooth accuracy–efficiency frontier, and the same holds on BDD100K.

Q4: Training cost, latency, and memory.

  • Training. AnyDepth-DETR (R-50) takes 1.68× the training time of a single static model with nearly the same peak training memory (+3%), slightly less than training the super-net- and base-net-equivalent models separately (General Response #5).
  • Latency. FPS is reported for every configuration on the same GPU at batch size 1 (Table 1, Section 4.3).
  • Model storage. One set of weights serves all configurations, adding less than 0.1% (AnyDepth-DETR) to about 3% (AnyDepth-YOLO) parameters over a single static model, instead of storing a separate model per operating point.

We are grateful for the reviewer's suggestions, which helped us make the evaluation more systematic. We hope the revision addresses the concerns, and we would be glad to answer any further questions.

Reviewer_VeCTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles efficient object detection for varying computational budgets. The paper proposes a single network that can adapt its architectural depth at inference time without retraining, which allows the architecture to be deployed at different levels of accuracy and computational complexity. The proposed network is divided into an "essential path" which learns core feature representations, and a "refinement path", which further refines these representations when additional compute is available. The method also uses self-distillation between the shallowest and deepest configurations to enable intermediate depth configuration usage without additional training. Different combinations of these paths yield different levels of accuracy-efficiency trade-offs. Results demonstrate a 1.6x speedup in FPS and 2.0 AP decrease when using essential paths compared to using both essential and refinement paths within the same network.

Critical strengths
  1. Problem significance: Paper addresses a practical problem of adapting object detectors to varying computational budgets at inference time without retraining.
  2. Method proposed: A interesting single architecture is proposed that allows using different architectural depth configurations for adaptable efficiency.
  3. Evidence: Results on RT-DETR and YOLO demonstrate up to 1.6x FPS speedup with modest accuracy reductions (2.0 AP) compared to full-component network.
Critical weaknesses
  1. Limited comparative evaluation: Most of the evaluation establishes that the proposed method works internally, rather than its competitiveness against the broader set of approaches in the field of efficient object detection. DynamicDet and AdaDet are referenced but not compared with as baselines. Comparisons with compute-matched alternative methods would better contextualize the reported trade-offs. Additionally results are reported only on 1 dataset - COCO.
  2. Intermediate configurations are not sufficiently validated: One of the main claims is recombinations of network paths produce valid sub-networks, but Table 2 mainly evaluates paths progressively being removed, so arbitrary mixed configurations across backbone, neck, and decoder remain largely untested.
  3. Positioning against prior work and motivation: The paper could more clearly position and explain what is specifically novel relative to existing object detection adaptive-computation approaches.
  4. Added training cost not included: The paper mentions in its limitations that its self-distillation procedure approximately doubles training time, however, this cost is not quantified in its reported speedups in Table 1.
  5. Minor terminology clarifications: Terms such as "operating points" and "depth" would benefit from clarification at first mention, specifying that they refer to accuracy–efficiency configurations and architectural depth, respectively.
Questions for authors
  1. Can the authors evaluate arbitrary or randomly sampled path combinations across backbone, neck, and decoder to directly support the claim that any combination yields a valid sub-network?
  2. Can the authors quantify the claimed speedup relative to training, accounting for the roughly 2× training cost for self distillation
  3. What about related work comparisons or baselines beyond matched comparison with YOLO and RT-DETR?
  4. Does the framework generalize beyond COCO? Experiments on other datasets would help establish this
1 comment

We sincerely thank the reviewer for the clear and constructive review. The requests for comparisons with adaptive detectors, for an evaluation beyond COCO, and for an honest accounting of training cost led to new experiments that we believe strengthen the paper considerably. All changes are marked in blue in the revised PDF.

W1 / Q3: Comparison with other approaches. Table 1 now includes DynamicDet, with FPS re-measured on the same RTX 4090 at batch size 1 (General Response #3). Since DynamicDet is built on YOLOv7 with a different training recipe, its AP is not directly comparable to ours, and we note this in the caption. The more fundamental difference is deployment behavior. DynamicDet routes each image to an easy or hard path, so its latency varies from frame to frame, whereas each AnyDepth configuration has a fixed latency, so a per-frame budget can be guaranteed. Like AdaptiveDepth, DynamicDet and AdaDet adapt only the backbone. For AnyDepth-DETR, backbone-only adaptation would cap the speedup at 1.15×, while adapting the neck as well, together with the decoder’s native early exit, raises it to 1.46× (Section 4.2). We also compare with Slimmable Networks adapted to YOLOv12 (Appendix A.4).

W1 / Q4: Generalization beyond COCO. We added BDD100K, a driving dataset (General Response #4, Appendix A.5). The AnyDepth-DETR base-net outperforms a separately trained RT-DETR of the same architecture by 1.3 AP, the super-net is within 0.3 AP of its static counterpart, and across all 384 sub-networks the frontier is above the random-path baseline over 81% of the FPS range.

W2 / Q1: Arbitrary path combinations. We now evaluate all 384 (AnyDepth-DETR R-50) and 256 (AnyDepth-YOLO L) sub-networks, i.e., every combination across backbone, neck, and decoder, without any BN recalibration (new Section 4.3, General Response #2). Although only the two endpoints are trained, the intermediate sub-networks form a smooth accuracy–efficiency frontier, which directly supports the claim that any combination yields a valid sub-network.

W3: Positioning and motivation. We rewrote the Introduction to state what is new relative to prior adaptive-computation work. Stage-wise modularity is a known principle, and our contribution is achieving it throughout a modern multi-scale detector, where backbone-only adaptation, CSP-style stages, and conflicting target assignment prevent existing methods from carrying over (General Response #1).

W4 / Q2: Training cost. Training is a one-time cost, whereas our goal is flexibility at deployment. Profiling with the real COCO training configuration shows that AnyDepth-DETR (R-50) takes 1.68× the training time of a single static model with nearly the same peak memory (+3%). This is slightly less than training the super-net- and base-net-equivalent models separately (0.95× their combined time), while one model provides all 384 sub-networks (General Response #5). We updated the Limitations accordingly.

W5: Terminology. Thank you for pointing this out. We now define both terms at first mention in the Introduction. An operating point is a specific accuracy–efficiency configuration, and depth refers to the number of blocks executed in each stage.

We are grateful for the reviewer's suggestions, which broadened the evaluation of the paper. We hope the revision addresses the concerns, and we would be glad to answer any further questions.

Reviewer_TiYyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AnyDepth-DETR and AnyDepth-YOLO, which allow a single detector to operate at different depths while preserving its multi-scale feature hierarchy. The full and shallow paths are jointly trained using self-distillation, while intermediate configurations can be selected at inference time without retraining. On COCO, AnyDepth-DETR R-101 achieves a 1.60× speedup over its full-depth configuration with a 2.0 AP reduction.

Critical strengths
  1. The stage-wise design preserves the multi-scale features required for object detection while allowing refinement blocks to be skipped.
  2. The framework is demonstrated on two substantially different detector families, RT-DETR and YOLOv12, indicating reasonable architectural applicability.
  3. The paper reports measured throughput alongside AP. At similar throughput, shallow AnyDepth-DETR R-50 achieves 50.6 AP at 165.3 FPS, compared with 48.9 AP at 163.9 FPS for static RT-DETR R-34.
  4. The distillation procedure addresses detection-specific assignment conflicts by aligning DETR queries matched to the same object and filtering conflicting YOLO anchor assignments.
Critical weaknesses
  1. I have reservations about the methodological novelty. AdaptiveDepth already adjusts inference depth by skipping subpaths, while the contribution beyond adapting this principle to multi-scale object detection is not sufficiently distinguished.
  2. The evidence for arbitrary stage combinations is limited. Table 10 evaluates only six of the 254 intermediate YOLO-L configurations, and Table 2 reports a PAN-only DETR configuration even though the submitted encoder uses one shared switch for both FPN and PAN.
  3. Spatially averaged feature alignment may be insufficient to ensure path interchangeability. Equation (9) can produce a small loss even when the two paths respond to the same object at different spatial locations.
  4. It remains unclear whether endpoint training actually alleviates gradient conflicts. Table 3 reports only endpoint AP, so the advantage over random path sampling may arise because the evaluated endpoints are optimized at every iteration.
  5. The YOLO classification-distillation implementation is inconsistent with Equation (11). The equation restricts distillation to anchors assigned by both paths to the same foreground object, whereas the code also distills anchors classified as foreground only by the teacher.
  6. Localization gains cannot be attributed solely to the refinement paths. Table 6 enables the refinement paths while increasing decoder depth from four to six layers, introducing two additional rounds of box refinement.
Questions for authors
  1. How does COCO AP vary across different YOLO-L skipping patterns under comparable computational or latency budgets?
  2. What switch configuration and evaluation code were used to obtain the PAN-only AnyDepth-DETR result in Table 2, given that the submitted encoder uses a shared switch for FPN and PAN?
  3. Which classification-distillation mask was used to produce the YOLO results in Tables 1 and 3, and how does restricting classification distillation to the jointly matched foreground anchors in Equation (11) affect performance?
  4. How does the proposed endpoint-training strategy perform against random path sampling on the same set of unseen intermediate configurations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer TiYy (1/2)

We sincerely thank the reviewer for the exceptionally careful review, including the close reading of our code. Several points (the PAN switch, the distillation masks, and the decoder-depth confound) exposed real inaccuracies in the submission, and fixing them made the paper more precise. All changes are marked in blue in the revised PDF.

W1: Novelty relative to AdaptiveDepth. We agree, and we have reframed the paper accordingly. Stage-wise modularity is a known principle, and our contribution is achieving it throughout a modern multi-scale detector, where backbone-only adaptation, CSP-style stages, and conflicting target assignment prevent existing methods from carrying over (General Response #1).

W2 / Q1: Arbitrary stage combinations and skip patterns at matched latency. We now evaluate all 256 AD-YOLO (L) sub-networks (and all 384 AD-DETR R-50 sub-networks), not 6 of them, with AP and FPS for each (new Section 4.3 and figure, General Response #2). The intermediate sub-networks form a smooth frontier. At comparable latency, skip patterns differ mainly by where compute is removed. For AD-YOLO, skipping refinement paths in the backbone is more efficient than in the neck (32% FPS gain for 1.0 AP vs. 12% for 1.1 AP, Appendix Tables 9 and 10), and the configurations well below the frontier are those that spend compute unevenly, e.g., a full-depth last backbone stage with a skipped neck.

Q2: PAN-only configuration. The reviewer is correct that the submitted code uses a single flag for both FPN and PAN. Because training uses only the two endpoints (all stages skipped or none), a shared flag and separate flags produce identical training. For finer-grained evaluation, we split the flag into independent FPN and PAN flags, which requires no change to the trained weights since each stage has its own switchable BN. The PAN-only row and the 384-configuration sweep (16 backbone × 4 encoder × 6 decoder) were obtained this way. This change affects only evaluation, not training or any reported endpoint result.

W3: Spatially averaged feature alignment. The reviewer is right that Eq. 9 (Eq. 10 in the revision) aligns channel descriptors and does not by itself enforce spatial correspondence. This is a deliberate choice. Per-location alignment (Eq. 13) performs worse (53.1 / 50.3 vs. 53.3 / 50.6, Table 8), because it penalizes the spatial refinements the refinement path is meant to learn. Spatial compatibility is instead checked in two ways. The CKA in Table 6 treats each 4×4 spatial cell as a separate sample, so spatial mismatches would lower it, and it stays at 0.92 or above. More directly, the final detection AP of every mixed configuration shows that later stages accept either path's output.

W4 / Q4: Endpoint training vs. random path sampling on unseen configurations. We evaluated both models on the same 384 / 256 sub-networks (Section 4.3). Our endpoint self-distillation with assignment-consistent targets gives a higher frontier over the full FPS range for AD-YOLO (by 0.1–0.5 AP) and over 68% of it for AD-DETR. On the remaining range, the random-path baseline is ahead by up to 0.6 AP. Counting all sub-networks, ours is better on 65% (AD-YOLO) but only 24.7% (AD-DETR). For AD-DETR, 90% of the random baseline's wins are on dominated sub-networks off both frontiers, i.e., configurations a deployment would not select.

AuthorsOpenReview ↗

Response to Reviewer TiYy (2/2)

W5 / Q3: Classification distillation mask in AD-YOLO. The reviewer is correct, and we thank them for checking the code. Classification and regression KD use different anchor sets, which the original Eq. 11 did not distinguish. Regression KD uses V_reg = (A_super ∩ A_base) ∖ C, since a box target is meaningless at an anchor the base-net treats as background. Classification KD uses V_cls = A_super ∖ C, since the teacher's soft class probabilities remain informative wherever it detects an object. All results in Tables 1 and 2 use these masks, and Section 3.3.2 now states them exactly.

We also measured both conflict types on COCO val2017 with the final checkpoints. Anchors assigned to different objects are rare (about 0.4% of jointly foreground anchors), whereas 13–14% of the anchors distilled by the naive setting are foreground for the super-net but background for the base-net. Restricting classification KD to the jointly matched anchors as in the original Eq. 11 had virtually no effect on AP (difference below 0.1), so we kept V_cls.

W6: Decoder-depth confound in the precision/recall analysis. We thank the reviewer for this catch. We evaluated both paths at both decoder depths (COCO val2017 AP, AD-DETR R-50).

Paths \ Decoder depth46Δ (decoder 4→6)
Essential only (base-net)50.5950.65+0.06
Essential + refinement (super-net)53.1053.27+0.17
Δ (refinement paths)+2.52+2.62

The two extra decoder layers add at most 0.17 AP, whereas the refinement paths add 2.5–2.6 AP at either depth. At matched depth 6, the refinement paths raise AP by 2.6 but AR100 by only 1.5, so the conclusion that they mainly improve the precision of detections rather than recall holds. However, at matched depth the AP75 and AP50 gains are similar, so we removed the original claim that the gain is concentrated at strict IoU thresholds (Appendix A.3).

We are grateful for the reviewer's thorough feedback, which made the paper considerably more accurate. We hope the revision addresses the concerns, and we would be glad to answer any further questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (2/2)

5. Training cost (VeCT, g1MN)

Training is a one-time cost, whereas our goal is flexibility at deployment. We profiled AnyDepth-DETR (R-50) with the real COCO training configuration (4× RTX 4090, DDP, FP32).

Training runsec/iterPeak memory / GPUGPU-hours (72 epochs)
AnyDepth-DETR (both passes)0.2965.84 GB175
Static super-net only (≈ RT-DETR R-50)0.1765.65 GB104
Static base-net only0.1344.04 GB79

AnyDepth-DETR takes 1.68× the training time of a single static model with nearly the same peak memory (+3%), which is slightly less than training the super-net- and base-net-equivalent models separately (0.95× their combined time), while providing all 384 sub-networks. (The static runs use our architecture with a single fixed path, so a vanilla RT-DETR may be slightly faster.) We updated the Limitations accordingly.

6. Corrections

  • Distillation masks for AnyDepth-YOLO (TiYy). Classification and regression KD use different anchor sets, which the original Eq. 11 did not distinguish. Section 3.3.2 now states both masks exactly as used for all reported results.
  • RT-DETR encoder stages (7jdf). These are RepVGG stages without identity shortcuts, not residual stages. Section 3.2.1 now describes them separately.
  • Decoder-depth confound in the precision/recall analysis (TiYy). Decoder depth accounts for only 0.06–0.17 AP of the super-net/base-net gap; the refinement paths account for 2.5–2.6 AP (Appendix A.3).

Summary of revisions

  • Introduction, Abstract, Conclusion. Reframed contribution; new latency-share figure.
  • Related Work. Positioning against AdaptiveDepth, DynamicDet, AdaDet.
  • Section 3. Credits to AdaptiveDepth; new RepVGG encoder description; corrected YOLO distillation masks.
  • Section 4.2. DynamicDet in Table 1; comparison of its input-dependent latency with our fixed per-configuration latency.
  • Section 4.3 (new). Evaluation of all sub-networks against the random-path baseline and a backbone-only variant approximating AdaptiveDepth.
  • Section 4.4. Correspondence of the naive self-distillation setting to AdaptiveDepth; measured assignment-conflict rates.
  • Appendix. BDD100K results; decoder-depth decomposition; component-wise sweeps (moved from the main text).

We are grateful for the reviewers' time and insight, and we would be glad to address any further questions during the discussion period.

AuthorsOpenReview ↗

General Response (1/2)

TL;DR. Figure 5 in the revised PDF addresses two concerns shared by reviewers.

  1. Intermediate sub-networks. Although only two are trained, all 384 (AnyDepth-DETR) and 256 (AnyDepth-YOLO) sub-networks fill a smooth accuracy–efficiency frontier.
  2. Difference from AdaptiveDepth (Kang & Lee, 2024). A backbone-only variant with naive self-distillation, approximating AdaptiveDepth, reaches lower AP at every speed.

We sincerely thank all reviewers for their careful and constructive feedback, which led to new experiments that substantially improved the paper. We have uploaded a revised PDF; all changes are marked in blue, and the Introduction and Section 4.3 were substantially rewritten. Below we address the concerns shared by several reviewers; individual responses follow under each review.

1. Novelty relative to AdaptiveDepth (7jdf, TiYy, g1MN)

In the original submission, we grouped AdaptiveDepth (Kang & Lee, 2024) broadly with other adaptive methods for classification. The revised Introduction now discusses it separately, as an instance of the design principle of stage-wise modularity, which has also been used in language models (LayerDrop, MatFormer). Our contribution is achieving stage-wise modularity throughout a modern multi-scale detector, where existing methods do not carry over for three reasons.

  • Coverage. In RT-DETR the backbone accounts for only 36% of latency (new Figure 1). A backbone-only AD-DETR trained with naive self-distillation, approximating AdaptiveDepth, reaches lower AP at every speed (Figure 5). With the full decoder, its speedup is limited to 1.15×.
  • Stage types. The YOLOv12 backbone and neck consist of CSP-style (concatenation) stages, and the RT-DETR encoder of RepVGG stages without identity shortcuts. AdaptiveDepth's additive residual split applies to neither. We address these with a switchable aggregation layer and with explicit stage-boundary feature alignment.
  • Training signal. The super-net and base-net assign the same query or anchor to different objects. Measured at convergence, 68% of the query pairs formed by naive self-distillation in AD-DETR pair predictions for different objects. AdaptiveDepth's plain output distillation therefore brings little or no gain (Table 2); our assignment-consistent distillation is what makes self-distillation work inside a detector.

The table below positions our work against related adaptive methods.

MethodAdaptsBackboneNeckDecoderPer-input latencyDetector type
Slimmable Networkswidth (global)✓––notwo-stage (backbone only)
AdaptiveDepthdepth✓––notwo-stage / RetinaNet (backbone only)
DynamicDetbackbone route✓––yesYOLOv7
AdaDetearly exit✓––yes–
AnyDepth (ours)depth✓✓✓ (DETR, native exit)no (fixed per config)RT-DETR, YOLOv12

To our knowledge, AnyDepth-DETR/YOLO are the first detectors whose backbone and neck are both depth-switchable from a single set of weights.

2. Evaluation of all sub-networks (all reviewers)

We now evaluate every sub-network, i.e., all 384 of AnyDepth-DETR (R-50) (16 backbone × 4 encoder × 6 decoder) and all 256 of AnyDepth-YOLO (L), using the BN statistics learned in training as-is, with no per-sub-network recalibration (new Section 4.3 and figure).

  • Although only the two endpoints are trained, the intermediate sub-networks form a smooth accuracy–efficiency frontier.
  • Against a compute-matched baseline that trains two random sub-networks per iteration, our frontier is higher over the full FPS range for AnyDepth-YOLO (by 0.1–0.5 AP) and over 68% of it for AnyDepth-DETR.
  • Counting all sub-networks, ours is better on 65% (YOLO) and 24.7% (DETR). For DETR, 90% of the random baseline's wins are on dominated sub-networks off both Pareto frontiers, i.e., configurations a deployment would not select.

3. Comparison with adaptive detectors (VeCT, g1MN)

Table 1 now includes DynamicDet (FPS re-measured on the same RTX 4090 at batch size 1). Beyond accuracy, the key difference is deployment behavior. DynamicDet routes each image to an easy or hard path, so its latency varies from frame to frame, whereas each AnyDepth configuration has a fixed latency, so a per-frame budget can be guaranteed.

4. Generalization beyond COCO (VeCT, g1MN)

We added BDD100K (driving, 70K/10K images) for AnyDepth-DETR (R-50) (Appendix A.5). Its base-net outperforms a separately trained RT-DETR of the same architecture by 1.3 AP (33.3 vs. 32.0), and its super-net is within 0.3 AP of the corresponding static model. Across all 384 sub-networks, its frontier is above the random-path baseline over 81% of the FPS range, by 0.7 AP on average.

Area_Chair_GNpWOpenReview ↗

Please engage in the discussion

Dear Reviewers,

Thank you for your reviews and contributions so far. The authors have now posted their response, and I would like to kindly remind you to review it and engage in the discussion where appropriate.

Please also consider whether the authors' clarifications address any of your concerns and update your assessment if warranted.

Thank you for your time and effort.

Best regards,

Your AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.