- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the sensitivity of Average Precision (AP) to small bounding-box localization errors in object detection. The authors derive closed-form expressions for IoU under translation and corner-anchored scaling and use them to characterize the displacement at which a detection crosses a specified IoU threshold. They further relate dataset-level degradation to the distribution of object sizes.
The theoretical results are validated using controlled perturbations on COCO and LVIS annotations, together with experiments involving pretrained detectors, including Faster R-CNN and YOLOv11. The authors propose size-adaptive IoU thresholds and smoothed AP as potential approaches to reducing size-dependent sensitivity and discontinuities in conventional evaluation.
The central contribution is an analytical characterization of known AP sensitivity phenomena, together with theory-guided modifications to object detection evaluation.
Critical strengths
-
Clear and interpretable theoretical characterization. The paper provides a unified analytical explanation of several localization-sensitivity phenomena, including object-size dependence, directional sensitivity, and differences between IoU thresholds. The connection between individual threshold-crossing distances and dataset-level size distributions is particularly useful because it relates aggregate degradation behavior to measurable dataset properties.
-
Extensive validation of the geometric predictions. The authors evaluate their theoretical predictions using controlled perturbations on COCO and LVIS, as well as real detector outputs. The inclusion of different perturbation types, detector architectures, and Gaussian localization noise strengthens the evidence that the geometric effects are not specific to a single model or dataset. The experiments distinguish idealized absolute prediction accuracy from relative degradation estimates for real detectors.
Critical weaknesses
-
Novelty Relative to Prior Work. The novelty is not sufficiently established relative to existing research. Borji, “Sensitivity of Average Precision to Bounding Box Perturbations,” investigates AP sensitivity using nearly identical experimental protocols, including eight translation directions, 0–10 pixel perturbations, ground-truth boxes as predictions, real detectors, and enlarging/shrinking transformations. Le Jeune and Mokraoui, “Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime,” study scale-adaptive evaluation and derive a mathematically equivalent horizontal IoU inversion in Appendix C.3, for a different purpose. This submission adds general-direction analysis, dataset-level characterization, and additional validation.
-
Insufficient Validation of the Proposed Metric. The proposed size-normalized metric reduces size-dependent AP degradation. This does not necessarily establish better evaluation quality. The paper provides no validation against human localization judgments or downstream task performance and lacks comparisons with relevant alternatives. The proposed metric's detector rankings depend on the chosen localization tolerance. Paper does not establish a principled method for selecting this parameter.
Questions for authors
-
Novelty Relative to Prior Work. What are the main theoretical contributions beyond Borji and Le Jeune and Mokraoui, given the mathematically equivalent horizontal IoU inversion in Appendix C.3 of Le Jeune and Mokraoui?
-
Validity of the Adaptive Threshold. How does Equation (7) handle s <= c, which produces nonpositive thresholds and may accept zero-overlap predictions? Does Corollary 7's direction-uniform guarantee extend to rectangular boxes?
-
Practical Significance. How does the proposed metric compare with alternatives such? Is there evidence that it better reflects human judgments or downstream task performance, beyond reducing size-dependent AP degradation?
3 comments
Part II
PART 2 of 3 — Validity of the adaptive threshold: Eq. (7) when s <= c, and rectangular boxes (Corollary 7)
Eq. (7) and s <= c. You are right: for s <= c the threshold (s-c)/(s+c) is non-positive, and since IoU >= 0 such an object accepts any same-class detection, even a disjoint one. Our code clipped tau at 0, so it did exactly this: 1.7% of val2017 boxes on average over c = 1..10 (5.1% at c = 10; 12% of the small ones) and 19% of LVIS boxes. We now floor it at COCO's loosest threshold, tau(s,c) = max{(s-c)/(s+c), 1/2} (tolerance saturates at s/3), and report the effect: the idealized large-small gap reduction, measured with a real AP evaluator, is 9x (not the previously reported ~20x, which was also a pass-rate statistic); LVIS 8.9x -> 5.2x; on real detectors it changes little (five CNN detectors 5.4x -> 5.9x; ranking agreement with standard mAP over 16 detectors Spearman 0.971 -> 0.974). The un-floored version over-corrected (small objects scored above large ones); the floored version slightly under-corrects them. Appendix B reports tau_min in {0, 0.3, 0.5}.
Corollary 7 (direction-uniform guarantee) does not extend to rectangular boxes. Thank you for catching this: the proof is for squares (W = H = s), but we then applied it with s = sqrt(WH) to all boxes and stated the guarantee without that restriction. We tested it numerically:
- For s = 50 px, c = 5 px, the guaranteed tolerance falls with aspect ratio r: 0.96c at r = 1.5, 0.88c at 2, 0.76c at 3, 0.67c at 4, 0.48c at 8 (the worst direction is no longer 45 degrees).
- On the 36,334 val2017 boxes (median r = 1.67; 36% have r > 2) at c = 10: median guaranteed tolerance 0.94c; 38% of boxes get < 0.9c and 12% get < 0.75c.
- The exact rule for any rectangle is the smallest IoU over all displacements of length c: tau_rect(W,H,c) = min over theta of IoU(c cos theta, c sin theta), a one-dimensional minimization (Lemma 1 plus monotonicity of IoU in |dx|,|dy|). Wherever its floor of 1/2 is inactive (57–90% of boxes for c = 10..3) it guarantees exactly c pixels in every direction (verified numerically: guaranteed tolerance / c = 1.0000).
The corollary is now restated for squares, with a new remark giving these numbers and the exact rectangular rule (Appendix B; a summary is also in Sec. 6 of the main text). We have not used the direction-uniform variant in any experiment (the experiments use the axis-aligned form), and the paper says so.
Part III
PART 3 of 3 — Practical significance: human judgments, downstream performance, alternatives, and the choice of the tolerance
We have no validation against human judgments or downstream performance of our own, and we agree this limits what can be claimed. Rather than leave it at that, we added what can be tested, and one of the results goes against our proposed metric; we report it plainly.
External human evidence (premise only). Le Jeune & Mokraoui's 75-participant study (>3,000 ratings, DOTA images) finds people are more lenient than IoU toward small objects. That supports the premise behind a size-adaptive tolerance. It does not validate our metric (different data and criterion), and their SIoU-vs-IoU correlation gap (Kendall tau 0.701 vs 0.674) is small.
A validity criterion needing no human labels: ranking stability under annotation noise. A metric should not reorder detectors because of irreducible annotation noise. COCO and LVIS annotated the same images independently, so we measured the disagreement on 13,393 matched objects: median 1.1 px per coordinate (90th percentile 7.2 px), growing sub-proportionally with object size (log-log slope 0.62; lower bounds, since matching requires IoU >= 0.5). We then perturbed the ground truth of a 500-image subset with this empirical disagreement (10 draws) and rescored our 16 detectors (Appendix H):
| metric | Kendall tau vs. unperturbed ranking | pairs swapped (of 120) |
|---|---|---|
| standard mAP | 0.993 | 0.4 |
| SIoU-AP (their defaults) | 0.995 | 0.3 |
| AP75 | 0.973 | 1.6 |
| NWD-AP (C = 32) | 0.970 | 1.8 |
| AP50 | 0.937 | 3.8 |
| size-normalized AP (ours) | 0.923 | 4.6 |
The size-normalized metric is the least rank-stable. Starting its tolerance range above the noise floor did not help (c = 5..10: tau = 0.89). So we have no evidence that it evaluates better than standard AP by this criterion, and we now claim it only as a diagnostic of size-dependent localization quality, not a better metric (abstract, Sec. 6 with the ranking-stability table now in the main text as Table 4, Limitations and Conclusion revised accordingly). A by-product that supports the paper's main message: annotation noise moves standard mAP's absolute value by ~11 points (24% relative) while barely touching the ordering.
Choosing the tolerance C. The measured noise gives no principled single C. Annotator disagreement is neither constant in pixels nor proportional to size, so neither a fixed pixel tolerance nor a fixed IoU matches it. Our default C = 10 lies between the 90th and 95th percentile of disagreement, which is a rationale, not a derivation. We continue to recommend reporting rankings at more than one C.
Alternatives. We compared against AP50, AP75, NWD-AP and SIoU-AP (table above), but not against LRP, PDQ or Optimal Correction Cost, which we cite but did not reimplement; a comparison with them, a downstream-task study, and a human study of our own are listed in the Limitations as open.
Part I
PART 1 of 3 — Novelty relative to Borji (2022) and Le Jeune & Mokraoui (2023)
Thank you for the careful and knowledgeable review. Both prior works are real, closely related, and we should have cited them; we have revised the paper accordingly.
Borji (2022), "Sensitivity of Average Precision to Bounding Box Perturbations" (arXiv 2206.10107). This is the closest prior work, and omitting it from the submission was our error, now corrected: it is cited in the Introduction and Related Work and credited as the first measurement of this sensitivity. We deliberately reuse its protocol (ground-truth boxes as predictions, eight translation directions, 0–10 px, one-sided enlarge/shrink) so that results are directly comparable. That work is empirical: it measures AP's sensitivity but has no closed-form analysis, no size-dependence derivation, no dataset-level characterization, no size-adaptive criterion and no implemented remedy. What this submission adds is the explanation and what follows from it:
"We have also sent a confidential note to the AC regarding this reference."
Le Jeune & Mokraoui, "Rethinking IoU for Small Object Detection in Few-Shot Regime" (arXiv 2307.09562). You are right, and we verified it: Appendix C.3 inverts the horizontal IoU of identical squares, C(x) = (w - |x|)/(w + |x|), via g(z) = w(1-z)/(1+z) to derive the distribution of IoU/GIoU under Gaussian shift. That is exactly our axis-aligned Corollary 2, t*(tau, W) = W(1-tau)/(1+tau). We now say so in the paper (Sec. 4 and Related Work) and describe the corollary as elementary algebra, no longer "the central quantitative result". Their purpose differs: a distribution of the criterion under random noise, versus our tolerance (crossing distance) used to explain AP behaviour and to define a pixel-tolerance threshold.
What remains as contribution beyond these two:
We are also happy to scope the paper's claim explicitly: it is primarily an explanation and validation of known AP sensitivity, plus theory-guided diagnostics, not a new theory of IoU.