- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces label free evaluation of object detectors that can be used for unsupervised model selection in domain adaptive object detection. The proposed method links detections across checkpoints in unlabeled target domain, into so called shared object candidates. The resulting trajectories allow to expose misses and duplicates that detections from a single-checkpoint cannot reveal. The detections are organized into a graph based on confidence and IoU thresholds.
Claimed contributions include
- A method for identifying true positive, false negative and false positive instances by using detections across different checkpoints during training. This is called trajectory-based label-free error decomposition.
- An approach for separating bounding box instances into those useful for identifying errors and those useful for ranking checkpoints.
- Empirical validation with strong results, although thoroughly validating a new method is a minimum requirement to be considered for publication.
Critical strengths
- This is a useful method for evaluating different checkpoints of the same method or potentially different object detectors on unlabelled data.
2.The validation shows significant improvements over other baselines and comparable methods.
3.The paper offers some interesting observations such as showing that not the same evidence is needed for reconstructing errors and for ranking checkpoints or that excluding candidates present at every checkpoint reduces RMSE significantly.
Critical weaknesses
1.It would be interesting to see more quantitative evaluations of the proposed error proxies on datasets with GT. Tab 2 and Fig 3 give an idea how aligned the proposed metrics are with GT but it is unclear what impact there is from the errors consistently produced by different checkpoints.
- The error proxies are reasonably motivated, but their precise forms are heuristic rather than statistically derived. They would benefit from stronger justification. As such the method can be considered an empirically effective checkpoint-ranking heuristic rather than a quantitatively justified estimator of detection errors.
3.The method uses fixed confidence thresholds for candidate construction. The paper recognizes the dependency on calibration and consistency of confidence scores across checkpoints, it provides some sensitivity results, but does not establish that the ranking is robust to checkpoint-dependent calibration changes.
4.The construction does not inherently require checkpoints of the same model, but the paper only validates selection within a model’s training trajectory. The approach would be more useful if it could rank different models. How much does it rely on the calibration and consistency of the confidence scores within the training trajectory? Could a continuous distribution of confidence for corresponding BB locations be possible to identify candidates that never cross the threshold?
Interesting evaluation would be to retain the candidates detected by every checkpoint and use the ensemble to finetune the model on new data. This would strengthen the use case for this approach.
The main results are in the paper, but several results important to assessing the method are deferred to the appendix, such as cross detector results as evidence for generalization, the number of checkpoints needed, construction choice and building trajectories.
The clarity of the presentation could have been better, requiring multiple passes to understand the details of the method where the main ideas are relatively simple. Clarity issues start in the abstract with ‘candidates whose presence is invariant at the threshold used to collect them’, invariant to what? Candidates are detected across all used thresholds? In paragraph 82-91, many terms are not properly introduced, ‘construction threshold’, ‘difference in presence’ , ‘ratio based scores’, ‘checkpoint rankings’. It continues in the method description.
Questions for authors
Is the proposed definition of error estimates more consistent with GT than other methods?
Is there any theoretical justification in addition to empirical one?
How robust is the checkpoint ranking to calibration changes?
Is ranking of different detectors a possible or dependency on calibration prevents it? What is required for that?