- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces relational competence, a population-relative characterization of detector behavior based on residual co-observability after exact control of detector coverage and instance observability with a fixed-margin null. It provides a meaningful complement to conventional AP-based evaluation. It shows that similar scalar performance can conceal different relational organization and that, in the tested controlled setting, same-design retrainings remain closer in relational space than cross-design models.
Critical strengths
- By analyzing the behavior of the detector based on instance-level success patterns, a meaningful supplementary perspective is provided for the traditional AP evaluation.
- This paper is well organized and written.
- The paper reports statistically significant residual structure across eight detectors.
Critical weaknesses
-
The practical value of the analysis has not been demonstrated. The paper establishes the existence and reproducibility of relational structure, but does not show whether these findings can guide model ensembling, model selection, or performance improvement. Concrete application experiments would help demonstrate the benefits over conventional performance metrics and simple prediction disagreement analyses.
-
The sources of the residual relational structure remain unclear. Controlling overall coverage and instance observability does not fully account for all factors affecting detection performance. Residual differences may reflect model-specific preferences for object scales, occlusion levels, or categories. Stratified analyses would help explain which detection behaviors the identified relational structure captures.
-
The dataset and evaluation splits are insufficiently described. These details directly affect threshold calibration, statistical testing, and reproducibility and should be fully reported.
-
The characterization of detection behavior is limited. Detection errors involve not only missed objects but also false positives, duplicate detections, localization errors, and confidence ranking. The binary observability representation mainly indicates whether a ground-truth instance is successfully detected at a given threshold and cannot fully capture these differences.
-
The reproducibility and generalization evaluations are limited in scope. The retraining experiments involve only two model designs with three random seeds each, and cross-dataset validation is missing. Evaluating additional model designs, training seeds, and datasets would help establish whether the relational structure remains stable across a broader range of models and data distributions.
Questions for authors
Please refer to weakness.