- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Frozen Spatial Consistency Intervention (FSCI). FSCI is a training-free, inference-time framework for frozen DETR-family detectors. Its key idea is that spatial inconsistency under a horizontal flip can serve as a signal for correcting the detector's own small-object predictions. Each image is processed as the original view, its horizontal flip, and four overlapping crops, requiring six forward passes. Flip-view predictions are matched to original predictions by class-constrained Hungarian assignment. Matched boxes are then fused with confidence weights. Crop candidates are assigned to a unique source crop by a nearest-center ownership rule. All candidates are merged through class-aware NMS. FSCI updates no parameters and uses no target-domain annotations. It improves AP in all 15 settings, covering five ResNet-50 DETR variants on COCO val2017, TinyPerson, and VisDrone2019. A mechanistic analysis on COCO includes direction-reversal, random-sign, and budget-matched controls, plus a held-out translation intervention. A boundary analysis shows that instance-level inconsistency does not predict per-instance correction gain.
Critical strengths
-
Valuable problem and practical setting. Small-object detection with frozen DETR detectors is a real and underexplored problem. The training-free, frozen-detector setting matches practical deployment constraints, such as API-only models, license-restricted weights, and no target-domain annotations.
-
Consistent empirical gains. FSCI improves AP in all detector–dataset settings, with all hyperparameters shared across settings. The gains concentrate on small objects, which is where the paper's own diagnosis predicts they should appear.
-
Clear presentation and informative visualization. The paper is well organized and easy to follow. Figure 3 clearly links cross-view attention displacement to correction direction at the instance level. The scale-stratified analyses (Figs. 5–6) make the mechanism easy to follow.
Critical weaknesses
-
Novelty is not established against the cited techniques. The problem itself is valuable. However, flip augmentation and its benefit for small objects are well documented in prior test-time-augmentation work. The flip-fusion step (Eqs. 3–4) is a two-view special case of Weighted Boxes Fusion, and the slicing step with deduplication is close to SAHI-style inference. The paper cites WBF, SAHI, Soft-NMS, and TTA aggregation as the paradigm it improves upon, yet runs none of them. Without these baselines, the claim that gains come from explanation-guided intervention rather than standard multi-view ensembling cannot be verified.
-
Six-fold inference cost is neither reported nor justified. Each image requires six forward passes (original, flip, four crops), but no latency, throughput, or memory numbers appear anywhere. No compute-matched alternative is evaluated, such as a single higher-resolution pass or plain flip TTA with standard aggregation. Training-free does not imply cheap inference, but for a frozen-detector wrapper this overhead must be justified.
-
Baseline detectors are all from 2020–2022. All five are ResNet-50 models predating modern query design. DINO is cited but never evaluated, and RT-DETR is not discussed. The largest gains occur on the weakest baseline, which is also what generic ensembling predicts. Whether the gains persist on modern detectors remains unknown.
-
Writing details. (a) Lines 192–193: the sentence "A lower ρi or a larger ∥∆bi∥ indicates greater sensitivity to spatial perturbations." is duplicated verbatim. (b) Line 116 (Fig. 2): "Counteractual Image" should be "Counterfactual". (c) Line 456 (Fig. 6e legend): the label "CEF" is not defined in the text; please clarify what it refers to. (d) Line 640: the Appendix placeholder was not removed.
Questions for authors
-
All main results compare FSCI only to its own single-view baseline. How does FSCI perform against peer test-time methods at matched compute, e.g., flip TTA with WBF, SAHI-style slicing, or a single higher-resolution pass? How much of the gain is attributable to the explanation-guided gating and ownership rule beyond these standard alternatives?
-
Each image requires six forward passes. What is the latency, throughput, and memory overhead per detector, and how does AP trade off against the number of views? Under what conditions is this preferable to one-time fine-tuning on the public training sets of VisDrone2019 or TinyPerson?
9 comments
Response to Question 2
Thank you for raising these practical deployment questions. We have added a Per-Detector Efficiency and View-Count Trade-offs subsection to the appendix, extending the measurements to all seven detectors and reporting the costs of the one-, two-, five-, and six-view DETR configurations. Per-detector inference costs. All detectors are evaluated on the same 200 COCO val2017 images using an NVIDIA GeForce RTX 3060 Laptop GPU, FP32 precision, batch size one, and sequential view processing. After warmup, we perform CUDA-synchronized pipeline measurements with cyclically rotated method order. Timing includes preprocessing, transfers, inference, decoding, and aggregation, but excludes disk access and image decoding. Single-view/FSCI mean latencies are 87.07/517.43 ms for DETR, 277.48/1658.36 ms for Deformable DETR, 104.69/615.07 ms for Conditional DETR, 110.93/660.20 ms for Anchor-DETR, 72.22/426.23 ms for DAB-DETR, 47.24/292.73 ms for DEIMv2-S, and 53.03/330.45 ms for RT-DETRv3-R50. The appendix reports the corresponding throughput and memory measurements. Across these implementations, FSCI increases mean latency by approximately 5.9–6.2×. We explicitly disclose that Deformable DETR uses a local PyTorch attention fallback, so its absolute runtime does not represent the official optimized CUDA implementation. Peak allocated GPU memory remains unchanged within each detector under sequential execution and CPU aggregation. However, we distinguish allocated from reserved memory: Deformable DETR’s peak reserved memory increases from 1930 to 2564 MiB. These measurements exclude CPU memory, CUDA context, and non-PyTorch allocations and do not characterize parallel multiview execution. Accuracy–view-count trade-offs. On complete COCO val2017, the one-view DETR baseline achieves 41.998 AP at 87.07 ms/image. Two-view counterfactual correction achieves 42.534 AP at 174.16 ms/image, five-view ownership slicing achieves 42.407 AP at 432.33 ms/image, and six-view FSCI achieves 42.729 AP at 517.43 ms/image. Thus, two-view correction provides a lower-cost option, whereas the complete method obtains the highest overall and small-object AP among these configurations. We clarify that these are component-defined operating points, not a pure view-count sweep, because the processing rules also differ. When is frozen inference preferable to fine-tuning? We do not claim that FSCI is generally preferable when supervised target-domain adaptation is feasible. TinyPerson and VisDrone2019 provide public training data, making fine-tuning a meaningful alternative. Our evaluation instead addresses the setting where pretrained weights remain frozen and target-domain annotations are not used. FSCI is relevant when parameter updates or a training workflow are unavailable or undesirable, provided additional inference time is acceptable. When labeled data, training resources, and sustained deployment are available, one-time fine-tuning may offer a better amortized accuracy–cost trade-off.
Response to Weakness 1
Thank you for this criticism. We agree that horizontal flipping, confidence-weighted box averaging, image slicing, and NMS are established techniques. We do not claim novelty for these individual operations. In particular, the matched-pair fusion in Eqs. (3–4) is a restricted form of confidence-weighted box fusion. Our contribution concerns the combined procedure that establishes class-constrained cross-view correspondence, applies supported localization correction, and regulates crop candidates through spatial ownership, together with controlled analyses of these interventions. As detailed in our response to Question 1, the added aggregation and high-resolution controls examine whether standard alternatives reproduce the observed overall benefit. These results support the complete correction and candidate-handling procedure, rather than establishing weighted averaging as a new operator. We have also tightened the abstract and introduction to align the claims with the evidence. The revised abstract describes cross-view matching as exposing localization discrepancies and supporting correction, rather than implying that inconsistency reliably identifies instances that will benefit. It also acknowledges the limitation of instance-level inconsistency as a predictor of correction gains. In the introduction, we replace the categorical distinction from conventional test-time augmentation with a description of our procedure as building on multiview inference. We further distinguish geometric disagreement and a candidate correction direction from a guarantee that correction will improve an individual prediction. These revisions preserve the empirical contribution while avoiding claims that “explanation” independently causes the gains or that cross-view disagreement reliably predicts per-instance improvement.
Response to Question 1
Thank you for requesting these controlled comparisons. We have added Controlled Comparisons with Standard Aggregation and Single-View High-Resolution Control subsections to the appendix, alongside the slicing-based comparison and component analysis. On the complete COCO val2017 set using frozen DETR-R50, we compare methods that reuse the same six source views and retain at most 100 predictions. The controls include class-aware NMS, linear and Gaussian Soft-NMS, and SAHI+flip+WBF with average- and maximum-confidence aggregation. All newly evaluated configurations use an input confidence threshold of 0.01, including a matched-threshold FSCI variant. Gaussian Soft-NMS achieves the highest overall AP among these conventional controls at 39.042, whereas matched-threshold FSCI achieves 42.667 AP, an improvement of 3.625 points. WBF achieves 33.018/35.738 AP with average/maximum confidence. These experiments evaluate WBF under the shared six-view setting rather than flip-only WBF. They match the detector forward-pass budget, source predictions, and output budget, but do not imply identical postprocessing costs. We also evaluate a single high-resolution forward pass whose resolution is selected using training-image latency alone, without validation AP. This control achieves 36.825 AP at 517.06 ms/image, compared with 42.729 AP at 566.96 ms/image for FSCI. Its mean latency is 8.8% lower, which we explicitly acknowledge as an approximate rather than exact runtime match. Regarding attribution, the shared-prediction component analysis provides a more direct comparison: counterfactual correction combined with naive slicing achieves 42.007 AP, while the complete correction and ownership-based candidate-handling procedure achieves 42.729 AP, a difference of 0.722 points. Counterfactual correction alone achieves 42.534 AP, compared with 41.998 AP for the baseline. These differences quantify component contributions under the specified configurations, but are not an additive decomposition of the total gain. In particular, the 0.722-point difference includes the combined ownership and protection procedure and should not be attributed to ownership alone. The direction-reversal, random-intervention, and budget-matched ownership controls further examine whether the prescribed correction direction and spatial assignment matter. Finally, the conventional aggregation controls obtain higher small-object AP but lower medium- and large-object AP than FSCI. Our conclusion is therefore that the proposed procedure provides a better overall accuracy trade-off in these evaluated settings.