acceptodds
Under review as a conference paper at ICLR 2027

Beyond Accuracy Rankings: Relational Competence in Object Detection

Abstract

Two object detectors can achieve nearly identical scalar performance while succeeding on different instances. We ask whether these differences are incidental or reflect reproducible structure in how models solve the task. We introduce relational competence, a population-relative characterization of detector behavior based on residual co-observability after exact control of detector coverage and instance observability with a fixed-margin null. Across eight detectors, residual relational structure remains after this marginal control. Independent retrainings also recover design-associated relational neighborhoods: Faster R-CNN and RetinaNet checkpoints matched to within \(0.009\) AP have mean relational distances of \(0.0113\) within design and \(0.1537\) across designs, yielding \(\Delta D=0.1424\) with a 95% bootstrap interval of \([0.09,0.16]\). Under controlled resolution and JPEG degradation, population-relative signature changes are likewise reproducible across retrainings, but a predefined same- versus cross-intervention comparison does not establish intervention-specific directions. These results distinguish scalar performance, relational organization, retraining reproducibility, and response to intervention as separate aspects of model behavior. Relational competence therefore complements conventional evaluation by measuring not only how often models succeed, but how their successes are organized relative to a model population.

Primary area:
interpretability and explainable AI
Keywords:
Object Detection, Relational Competence, Model Evaluation, Model Reliability, Distribution Shift

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CTRPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces relational competence, a population-relative characterization of detector behavior based on residual co-observability after exact control of detector coverage and instance observability with a fixed-margin null. It provides a meaningful complement to conventional AP-based evaluation. It shows that similar scalar performance can conceal different relational organization and that, in the tested controlled setting, same-design retrainings remain closer in relational space than cross-design models.

Critical strengths
  1. By analyzing the behavior of the detector based on instance-level success patterns, a meaningful supplementary perspective is provided for the traditional AP evaluation.
  2. This paper is well organized and written.
  3. The paper reports statistically significant residual structure across eight detectors.
Critical weaknesses
  1. The practical value of the analysis has not been demonstrated. The paper establishes the existence and reproducibility of relational structure, but does not show whether these findings can guide model ensembling, model selection, or performance improvement. Concrete application experiments would help demonstrate the benefits over conventional performance metrics and simple prediction disagreement analyses.

  2. The sources of the residual relational structure remain unclear. Controlling overall coverage and instance observability does not fully account for all factors affecting detection performance. Residual differences may reflect model-specific preferences for object scales, occlusion levels, or categories. Stratified analyses would help explain which detection behaviors the identified relational structure captures.

  3. The dataset and evaluation splits are insufficiently described. These details directly affect threshold calibration, statistical testing, and reproducibility and should be fully reported.

  4. The characterization of detection behavior is limited. Detection errors involve not only missed objects but also false positives, duplicate detections, localization errors, and confidence ranking. The binary observability representation mainly indicates whether a ground-truth instance is successfully detected at a given threshold and cannot fully capture these differences.

  5. The reproducibility and generalization evaluations are limited in scope. The retraining experiments involve only two model designs with three random seeds each, and cross-dataset validation is missing. Evaluating additional model designs, training seeds, and datasets would help establish whether the relational structure remains stable across a broader range of models and data distributions.

Questions for authors

Please refer to weakness.

Reviewer_BH1kOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces relational competence, a method for comparing object detectors beyond traditional accuracy metrics such as Average Precision. Instead of considering only how many objects each model detects, the method examines whether different models successfully detect the same objects. It compares the observed results with randomized results that preserve each detector's total number of detections and the number of detectors that detect each object. These differences are then used to measure similarities between detectors.

The authors first evaluate eight object detectors and find differences in their detection patterns that cannot be explained by these preserved totals alone. They then compare three independently trained Faster R-CNN models with three RetinaNet models, showing that models with the same architecture have more similar detection patterns than models with different architectures, 0.0113 vs. 0.1537. Experiments with reduced image resolution and JPEG compression show reproducible changes in detection patterns, but these changes do not clearly distinguish between the two types of degradation.

Critical strengths
  1. Well-motivated evaluation framework. The paper addresses limitations of accuracy-based evaluation by investigating instance-level relationships. Exact marginal control provides a principled way to identify residual structure beyond raw agreement.

  2. Evidence of reproducibility. Independent retraining exhibits substantially smaller within-design than between-design distances, supported by a positive bootstrap confidence interval. The authors acknowledge that their results depend on the selected reference detectors and that their method does not distinguish between the two tested image degradations.

Critical weaknesses

The fixed-margin statistical component is closely related to existing methods. "Backbone: An R Package for Extracting the Backbone of Bipartite Projections", "Comparing Alternatives to the Fixed Degree Sequence Model for Extracting the Backbone of Bipartite Projections", and "Meta-validation of Bipartite Network Projections". These studies already use fixed-margin null models, Curveball randomization, and pairwise co-occurrence analysis. The use of standardized residuals is well established, as discussed in "Z-scores Unite Pairwise Indices of Ecological Similarity and Association for Binary Data".

The main new contribution appears to be constructing relational signatures to compare detectors. The paper does not compare this approach with existing behavioral similarity measures, such as error consistency "Beyond Accuracy: Quantifying Trial-by-Trial Behaviour of CNNs and Humans by Measuring Error Consistency", disagreement, or classifier-diversity measures used in ensemble learning such as disagreement and double-fault rates. Without these comparisons, it is unclear whether the proposed method offers meaningful advantages over existing approaches.

Questions for authors

Q1. Novelty and baselines: How does the proposed method differ from established fixed-degree-sequence approaches? Can the authors compare relational signatures with error consistency, disagreement, and classifier-diversity measures to demonstrate their incremental value?

Q2. Practical significance: What benefits does relational competence provide beyond existing similarity measures? For example, can it improve ensemble selection, identify complementary detectors, or predict robustness?

Q3. Robustness: What evidence supports generalization beyond the two evaluated architectures?

Reviewer_p7wyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces relational competence to compare object detectors beyond accuracy rankings. For each detector pair, it compares the shared detection count with a random baseline. This baseline preserves how many objects each detector detects and how many detectors detect each object. The comparisons give standardized scores. Each detector’s scores against a fixed set of reference detectors form its signature. Experiments examine eight detectors and repeated training of Faster R-CNN and RetinaNet. The paper further examines signature changes under resolution reduction and JPEG compression. It also tests sensitivity to operating points, IoU thresholds, and reference coordinates.

Critical strengths
  1. The paper addresses an important question in detector evaluation. AP summarizes overall performance, but it does not show which objects each detector successfully detects. Studying these success patterns provides additional information for comparing detectors with similar AP.
  2. The method is clearly organized. Fig. 1 outlines the procedure, and Eqs. (4)–(8) defines the scores, signatures, and distances.
Critical weaknesses
  1. Sec. 5.1 does not clearly state which detectors are used to construct the random baseline. It is unclear whether the six checkpoints are included together or separately with the eight reference detectors. This choice can affect the baseline and the resulting scores. Please specify the detectors included in each calculation. It would also be better to provide the data splits and their sizes, detector configurations, and Curveball update counts before and between retained samples.
  2. Sec. 5.1 compares six checkpoints, but Fig. 3a reports only two calibration AP values. The paper does not explain whether these are averages or results from individual checkpoints. Please report all six calibration AP values so that the stated AP matching can be checked.
  3. The paper does not discuss several closely related works. [1] study pairwise overlap under a fixed-degree null sampled with Curveball. This directly relates to the null model and overlap statistic in Eqs. (4)–(5). [2] report systematic example-level differences between model populations with statistically indistinguishable mean accuracy. [3] show that shared easy and hard examples can mask model differences. It would be better to discuss these works and distinguish the proposed detector signatures from these earlier contributions.
  4. The experiments rely on older detector designs. They only include Faster R-CNN and RetinaNet, introduced in 2015 and 2017, respectively. It would be better to include more recent detector designs.
  5. There are several presentation problems:
    • The comparison rule in Fig. 4b is not clearly explained.
    • “Performance retention” in Table 3 is not defined.

[1] Neal et al., Comparing alternatives to the fixed degree sequence model for extracting the backbone of bipartite projections. Scientific Reports, 2021.

[2] Nguyen et al., Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. ICLR, 2021.

[3] Meding et al., Trivial or impossible—dichotomous data difficulty masks model differences (on ImageNet and beyond). ICLR, 2022.

Questions for authors

Please refer to the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.