acceptodds
Under review as a conference paper at ICLR 2027

Which Checkpoint? Ask the Others

Abstract

Selecting a checkpoint for domain-adaptive object detection requires evaluating target-domain performance without target annotations. Existing label-free evaluation criteria based on confidence, box stability, and prediction consistency characterize observed detections, but do not directly account for false negatives caused by missed objects. Trajectory-based Error Decomposition(TED) addresses this limitation by exploiting relationships between checkpoints, using detections at one checkpoint to expose possible misses at another. Our contributions are as follows. (1) TED performs trajectory-based latent error reconstruction by linking detections into shared object candidates and using their presence, absence, and duplicate detections to reconstruct TP-, FN-, and FP-like trajectories. This label-free reconstruction brings possible misses into checkpoint evaluation. (2) TED turns the resulting reconstruction evidence into a selection criterion through common-mode evidence removal by removing candidates whose presence is invariant at the threshold used to collect them, focusing comparison on candidates whose presence differs across checkpoints at that threshold. In experiments, TED achieves the lowest selection RMSE among the evaluated methods under , , and mAP. Under , TED achieves a selection RMSE of 0.963 AP points, compared with 1.768 for the second-best selector.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unsupervised Checkpoint Selection, Unsupervised Model Selection, Auto Evaluation, Domain Adaptive Object Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mZn8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces label free evaluation of object detectors that can be used for unsupervised model selection in domain adaptive object detection. The proposed method links detections across checkpoints in unlabeled target domain, into so called shared object candidates. The resulting trajectories allow to expose misses and duplicates that detections from a single-checkpoint cannot reveal. The detections are organized into a graph based on confidence and IoU thresholds.

Claimed contributions include

  • A method for identifying true positive, false negative and false positive instances by using detections across different checkpoints during training. This is called trajectory-based label-free error decomposition.
  • An approach for separating bounding box instances into those useful for identifying errors and those useful for ranking checkpoints.
  • Empirical validation with strong results, although thoroughly validating a new method is a minimum requirement to be considered for publication.
Critical strengths
  1. This is a useful method for evaluating different checkpoints of the same method or potentially different object detectors on unlabelled data.

2.The validation shows significant improvements over other baselines and comparable methods.

3.The paper offers some interesting observations such as showing that not the same evidence is needed for reconstructing errors and for ranking checkpoints or that excluding candidates present at every checkpoint reduces RMSE significantly.

Critical weaknesses

1.It would be interesting to see more quantitative evaluations of the proposed error proxies on datasets with GT. Tab 2 and Fig 3 give an idea how aligned the proposed metrics are with GT but it is unclear what impact there is from the errors consistently produced by different checkpoints.

  1. The error proxies are reasonably motivated, but their precise forms are heuristic rather than statistically derived. They would benefit from stronger justification. As such the method can be considered an empirically effective checkpoint-ranking heuristic rather than a quantitatively justified estimator of detection errors.

3.The method uses fixed confidence thresholds for candidate construction. The paper recognizes the dependency on calibration and consistency of confidence scores across checkpoints, it provides some sensitivity results, but does not establish that the ranking is robust to checkpoint-dependent calibration changes.

4.The construction does not inherently require checkpoints of the same model, but the paper only validates selection within a model’s training trajectory. The approach would be more useful if it could rank different models. How much does it rely on the calibration and consistency of the confidence scores within the training trajectory? Could a continuous distribution of confidence for corresponding BB locations be possible to identify candidates that never cross the threshold?

Interesting evaluation would be to retain the candidates detected by every checkpoint and use the ensemble to finetune the model on new data. This would strengthen the use case for this approach.

The main results are in the paper, but several results important to assessing the method are deferred to the appendix, such as cross detector results as evidence for generalization, the number of checkpoints needed, construction choice and building trajectories.

The clarity of the presentation could have been better, requiring multiple passes to understand the details of the method where the main ideas are relatively simple. Clarity issues start in the abstract with ‘candidates whose presence is invariant at the threshold used to collect them’, invariant to what? Candidates are detected across all used thresholds? In paragraph 82-91, many terms are not properly introduced, ‘construction threshold’, ‘difference in presence’ , ‘ratio based scores’, ‘checkpoint rankings’. It continues in the method description.

Questions for authors

Is the proposed definition of error estimates more consistent with GT than other methods?

Is there any theoretical justification in addition to empirical one?

How robust is the checkpoint ranking to calibration changes?

Is ranking of different detectors a possible or dependency on calibration prevents it? What is required for that?

Reviewer_RM2BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses label-free checkpoint selection for domain adaptive object detection. Detections from all 30 saved checkpoints are linked per image and class via IoU union-find into shared candidates. Each candidate is weighted by its recurrence f_j, the fraction of checkpoints detecting it at tau_cand=0.05. For a given checkpoint, a candidate gives TP-like evidence if detected, FN-like evidence if missed, and FP-like evidence for weakly supported or duplicate boxes. Candidates present at every checkpoint (f_j=1) are dropped ("common-mode removal"), and a class-averaged F1-like score picks the checkpoint. On 20 Faster R-CNN setups TED reaches an AP50 oracle-gap RMSE of 0.963, vs 1.768 for the best of 16 baselines. It also wins on 20 YOLOv5/Def-DETR setups. The claim is that other checkpoints expose reachable misses, and that evidence for reconstructing errors differs from evidence for ranking.

Critical strengths
  1. The idea is simple and well motivated. It targets a real blind spot of BoS/PCR/DAS-style criteria: a miss leaves no trace in a single checkpoint (Sec 1, Fig 1).
  2. 16 baselines are integrated into one codebase on identical checkpoint sets, plus two more detector families (Fig 2, Tab 6).
  3. The scope is stated honestly. Objects missed by all checkpoints are unidentifiable (App A), and reachable coverage is reported (Tab 10).
  4. The ablations are useful (Tab 1, 4, 8, 11, 12). Thresholds were fixed a priori, and the non-optimal config is still reported (Tab 5).
Critical weaknesses
  1. Atypical trajectories. 34/40 runs have falling target AP, and 29/40 oracles sit in the first four checkpoints (Tab 13, C.9). Some runs collapse (Def-DETR S2C AT, Fig 5). Simply taking the first checkpoint beats 14 of the other 17 selectors on AP50 (Fig 2). This regime favors a consensus score, and for middle oracles (n=5) TED basically ties DIV (1.834 vs 1.854).

  2. No cross-checkpoint baseline. Every baseline scores one checkpoint in isolation. The natural control is to fuse all checkpoints into pseudo-GT (e.g. WBF, Solovyev et al. 2021) and compute each checkpoint's AP against it, or to use mean pairwise agreement. Without this we can't tell whether the TP/FN/FP decomposition matters or just the consensus. Related agreement-based work is also missing.

  3. The results hinge on one binary rule. F1-like RMSE is 6.105 without removal and 0.743 with it (Tab 4, 8), and the A.2 argument is post hoc. Invariance is also defined at tau_cand. So a candidate with f_j<1 that never crosses tau_det adds the same FN mass at every checkpoint and is kept, although by A.2's own logic it should distort F1 rankings too.

  4. The design was chosen on the evaluation set. The score, removal rule and association were picked from variants run on the same 40 labeled setups. With one seed per setup and RMSE driven by a few tail cases (Fig 4), some gaps may be noise (tau_det 0.4 vs 0.5 gives 1.356 vs 0.743). Please add bootstrap CIs and per-run rank correlation. The mean score-AP correlation is only ~0.57 (C.9).

  5. The score depends on the checkpoint pool, and false positives repeated by most checkpoints earn TP-like credit. Only ~32% of candidates with f in [0.9,1) match GT (Tab 3). This is a fundamental limit and worth stating.

  6. Minor: count-matched random removal gives exactly 6.105 AP50, the same as no removal, while AP75/mAP differ. Please check.

The idea is clean and the gains are large and consistent. However, the benchmark is dominated by degrading runs, the consensus baseline is missing, and the key removal rule is heuristic and was chosen with target labels available.

Questions for authors
  1. Why does target AP fall in 34/40 runs? Do the schedules match the published ALDI/AT settings, and how does the source-only model compare to checkpoint 1?
  2. Did you consider a fused pseudo-GT or pairwise-agreement baseline?
  3. Why define invariance at tau_cand rather than on (z_det, n_det)? Was the latter tried?
  4. Which Tab 1/4 variants were explored before TED was frozen, i.e. with label access?
  5. How much of the Tab 6 margin comes from collapsed runs?
Reviewer_Hck1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper focuses on the problem of checkpoint selection in unsupervised domain adaptive object detection (DAOD). In the absence of ground-truth labels in the target domain, selecting the checkpoint with the best generalization performance from a series of saved training checkpoints remains a critical deployment challenge. Most existing unsupervised evaluation criteria (such as confidence scores, bounding box stability, and prediction consistency) largely evaluate only the prediction results "observed" by the checkpoint itself, failing to directly perceive errors caused by false negatives (FN). A checkpoint might produce highly confident and internally consistent predictions while missing a large number of objects. To address this, the paper proposes a method named TED (Trajectory-based Error Decomposition). Its core observation is that the sequence of checkpoints generated during training can itself serve as a source of unsupervised evaluation evidence. By linking the detection results of different checkpoints on the same target image into shared candidate trajectories, TED leverages the detection results of other checkpoints to account for the false negatives of the current checkpoint.

Critical strengths
  1. Novel and Spot-on Motivation: Previous methods focus exclusively on what a single checkpoint "sees," making them easily misled by high-confidence errors or false-negative blind spots. In contrast, TED cleverly leverages "training dynamics" and a "wisdom-of-the-crowd" approach ("Ask the Others"). By contrasting the disagreements and consensus across multiple checkpoints, it successfully converts hard-to-capture false negatives (FNs) into a quantifiable evaluation metric.
  2. Rigorous Methodology and Ingenious Design: It innovatively introduces a dual-threshold mechanism that balances the supporting role of weak evidence with the strict evaluation of strong detections.It proposes Common-Mode Evidence Removal; both theoretical derivations and experiments demonstrate that removing invariant candidates with fj = 1 effectively suppresses ranking bias caused by the non-linear F1 score, significantly mitigating large tail errors of the algorithm.
  3. Exceptional Generalizability and Comprehensive, Solid Experiments: It proves that TED is not bound to any specific architecture.The baselines for comparison are exceptionally rich and detailed, yielding convincing conclusions.
  4. Outstanding Computational Efficiency and Practical Deployability: TED adopts a "prediction caching" mechanism where all checkpoint predictions need to be cached offline only once. Subsequent graph association, error decomposition, and score calculations are all performed directly on the cached data—eliminating the need to re-run model inference—and requiring no target domain labels.
Critical weaknesses
  1. Core Assumptions Remain Strong: Cross-Checkpoint Consistency Does Not Necessarily Equal Ground-Truth Object Existence. This is, in my view, the most worthy aspect of the paper for in-depth discussion. TED uses the appearance frequency of a candidate across multiple checkpoints as the support weight for object existence. The basic intuition is that if a candidate appears in more checkpoints, it is more likely to correspond to a real object. However, this assumption does not always hold. Checkpoints are not independent observation samples. Model parameters within the same training process are highly correlated, and consecutive checkpoints typically share a large amount of features, prediction biases, and pseudo-label information. Therefore, repeated predictions from multiple checkpoints cannot be simply treated as multiple independent pieces of evidence.

  2. Candidate Association Based on Connected Components May Cause Object Merging and Incorrect Assignment of Duplicate Detections. TED constructs an association graph via class consistency and IoU thresholds, and obtains candidate connected components using a disjoint-set data structure. The paper also acknowledges that this method suffers from a transitivity issue: if A overlaps with B, and B overlaps with C, even if A and C do not satisfy the association threshold, the three detections may still be merged. This issue is particularly concerning in dense scenes.

  3. The Paper Adopts RMSE Relative to the Oracle Optimal Checkpoint for Each Setting as the Primary Evaluation Metric. This metric has a clear advantage: it directly measures how far an unsupervised selection strategy is from the optimal checkpoint. However, RMSE also has limitations. First, squared error assigns higher weights to a few larger selection errors. Consequently, even if a method performs mediocrely across most settings, as long as it avoids a few catastrophic failures, it may still achieve a favorable overall result. Second, RMSE fails to fully reveal the performance distribution across different settings. Two methods may have similar RMSE values, yet one might perform stably across most settings while the other performs exceptionally well in some settings and fails significantly in others.

Questions for authors

Investigate the quantitative relationship between candidate appearance frequency and ground-truth object probability, rather than merely reporting a monotonic trend.

Analyze the impact of checkpoint correlation on candidate support degree, such as comparing the evidence provided by adjacent checkpoints versus distant ones.

Introduce a more robust support degree estimation that accounts for the spatial consistency, class stability, trajectory duration, and independent contributions of different training stages.

Analyze whether TED's scoring systematically biases toward certain checkpoints when erroneous candidates persist.

Incorporate direct evaluations of candidate association quality, such as candidate purity, object merging rate, object fragmentation rate, and duplicate detection assignment accuracy.

Analyze the impact of different IoU thresholds and association algorithms in dense and small-object scenarios.

Compare connected components, cross-checkpoint one-to-one matching, center-distance-based association, and more rigorous multi-object association methods.

In addition to RMSE, it is recommended to report the mean absolute selection error (MAE), the median and high-percentile selection errors, and the proportion of settings where the selection results outperform or underperform the baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.