- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates the issue of fairness in spatial attention ablation experiments for Detection Transformers. The authors point out that when a fixed number of feature cells are removed, small objects, which inherently occupy a smaller spatial area, may undergo a much stronger actual intervention than large objects. Therefore, a fixed number of cells does not necessarily correspond to the same intervention intensity. To address this issue, the paper compares two settings: removing a fixed number of cells and removing a fixed amount of attention mass. Experiments are conducted using DETR-R50 and Conditional DETR-R50. The results show a significant difference between small and large objects when four cells are removed, while this difference is substantially reduced under the equal-mass setting. The paper further presents exploratory analyses across different layers, checkpoints, and detector architectures, including sparse and deformable detectors.
Critical strengths
-
The paper highlights an easily overlooked issue in experimental design: that removing a fixed number of feature cells does not necessarily equate to applying an intervention of equal strength to targets of different scales. This observation is sound and holds methodological significance.
-
The authors provide a comprehensive account of experimental limitations, exploratory analyses, and protocol changes, clearly distinguishing between results that support strong conclusions and those that serve merely as exploratory evidence.
Critical weaknesses
-
The primary experiments utilize earlier architectures such as the original DETR and Conditional DETR, while the subsequent R101 experiments belong to the same model family; however, as the authors themselves acknowledge that the R101 and R50 experiments employed different decoder layers, this portion can only serve as an exploratory extension rather than constituting a rigorous cross-checkpoint confirmation.
-
A significant portion of the text is dedicated to detailing protocol changes, exploratory status, audit limitations, failed gates, post-hoc analyses, and conclusions that could not be supported. While such transparency is commendable, the frequency with which these caveats appear makes the article read more like an audit report of the experiment than a research paper centered on a clear scientific question.
Questions for authors
Can the authors re-verify the core conclusions regarding "fixed-count" versus "equal-mass" using a more modern detection transformer, while maintaining intervention layers and evaluation protocols that are entirely consistent with the main experiments?
Final recommendation
reject. concerns remained.