- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents AnyDepth-DETR and AnyDepth-YOLO, which adjust inference depth within a shared detector. Backbone and neck stages are divided into essential and skippable refinement paths, retaining the multi-scale feature hierarchy. The training stagee uses the full and shallow configurations with self-distillation, switchable normalization, detection-specific target alignment, and feature -level alignment. Experiments on COCO with RT-DETR and YOLOv12 demonstrate practical speed and accuracy tradeoff at the endpoints and several intermediate configurations.
Critical strengths
S1. The reported inference gains are demonstrated in two different detector families. In Table 1, AnyDepth-DETR R-101 increases throughput from 78.6 to 125.6 FPS while AP decreases from 54.6 to 52.6. AnyDepth-YOLO L increases throughput from 101.5 to 155.3 FPS with AP changing from 53.9 to 52.1. Those results support these operating points within one shared model, which is useful when deployment budgets changing.
S2. The implementation addresses concrete detection-specific difficulties. Dedicated CSP aggregation handles changes in concatenated feature width, while target-aligned distillation avoids transferring predictions assigned to different objects. Table 3 includes useful architectural and some training ablations, and Table 9 compares feature-alignment granularities. These controls provide evidence that the adaptation involves more than simply skipping blocks.
Critical weaknesses
W1: The contribution relative to AdaptiveDepth [1] is not established clear enough. In [1], it already uses mandatory and skippable stage paths, endpoint self-distillation, and binary skip-aware normalization. Besides, Its Appendix and Table 8 also evaluate adaptive backbones in Faster R-CNN, Mask R-CNN, and RetinaNet on COCO. So characterizing this predecessor as a classification design that is poorly suitable to detection, such saying is too broad. The present extension to neck adaptation and CSP stages can still be valuable. However, Table 3 does not clarify whether its naive distillation configuration can faithfully stand for the earlier detector adaptation, and the Slimmable-YOLO comparison is actually about another different approach. Thus, the incremental contribution is not isolated sufficiently.
[1] Kang, Woochul, and Hyungseop Lee. "Adaptive depth networks with skippable sub-paths." Advances in Neural Information Processing Systems 37 (2024): 33213-33231.
W2: The evidence for arbitrary combinations of stage depths is not so complete as the endpoint results. Table 2 and Table 10 only evaluate some selected component sweeps, while Equation 9 and Table 7 compare the features coming from the two trained endpoints. Global pooled feature agreement and high CKA cannot well establish that fixed downstream modules can use arbitrary mixed-path features in an interchangeable way. Especially, the BN selection depends on the skip flag of the current stage, but mixed configurations will bring in upstream paths that are not encountered together during the endpoint training. The existing intermediate results are encouraging, but the coverage of Figure 3 is not clear. So a more broad mixed-configuration evaluation is needed to substantiate the central composability claim.
Questions for authors
Q1: (From W1.) How does the method compare with a faithful adaptation of AdaptiveDepth to the same detector and training budget? If an existing row in Table 3 serves this purpose, please identify the correspondence and explain which additional gains come from neck adaptation, CSP aggregation, and the revised distillation losses.
Q2: How were the configurations in Figure 3 selected, and was any BN recalibration performed? Could the authors report AP and latency for a representative set of mixed backbone and neck configurations, including unfavorable cases, rather than only an ordered sweep or the frontier?
Q3: What are the exact essential and full computations, including BN placement, for the RepVggBlock encoder stages in Table 4? Section 3.2.1 describes additive residual blocks, whereas the standard RT-DETR RepVggBlock has no identity residual connection. Clarifying any architectural modification would make the method easier to verify.
I currently lean toward weak rejection, close to the acceptance boundary (in my opinion, boarderline, but no such option). The practical acceleration and results across two detector families are promising, but I hope authors can fix my questions. I am willing to raise the rating if author provide convicing response.
2 comments
Response to Reviewer 7jdf (2/2)
Q3: RepVGG encoder stages. The reviewer is right that the RT-DETR encoder stages are not residual. We added a dedicated description (Section 3.2.1, new Eq. 3). Each encoder stage applies RepVGG blocks without identity shortcuts on one branch and adds a parallel shortcut branch. Let , . Then
where are projections of the stage input and is the output projection, all shared by both paths (, ; Table 3). As for BN placement, as in all other stage types, switchable BN (skip / no-skip) is placed in the essential-path blocks , while the refinement blocks use standard BN and the shared projections are unchanged from RT-DETR. Because transforms rather than adds to , nothing in the architecture keeps the two paths close. This is where the naive joint-train model's CKA degrades most (0.70–0.87, Table 6), and why feature alignment is needed for AD-DETR but not for AD-YOLO (Section 4.4).
We are grateful for the reviewer's suggestions, which made both the positioning and the analysis of the paper considerably stronger. We hope the revision addresses the concerns, and we would welcome a reconsideration of the score if so.
Response to Reviewer 7jdf (1/2)
We sincerely thank the reviewer for the detailed and insightful review. The comments on the relation to AdaptiveDepth and on the encoder stages were particularly helpful and led directly to the reframed Introduction and the new analyses in the revision (changes in blue).
W1 / Q1: Contribution relative to AdaptiveDepth. We agree that our original characterization of AdaptiveDepth was too broad. The revised Introduction, Related Work, and Section 3 now credit it explicitly for the components the reviewer lists, and we no longer group it with methods that "transfer poorly" to detection. Our contribution is achieving stage-wise modularity throughout a modern detector (General Response #1).
Below is how Table 2 (Table 3 in the original submission) corresponds to AdaptiveDepth (now stated in Section 4.4).
The additional gains come from three sources.
W2 / Q2: Arbitrary combinations, Figure 3, and BN recalibration.