- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
UA-CP targets reliable endoscopic lesion detection with two stages. (1) Prior-Aware Training (PAT) adds a -weighted, error-adaptive group-norm penalty to the classifier of Faster R-CNN/FCOS, or a prior-weighted term to DINO's Hungarian matching cost, to address class imbalance. (2) Dual Conformal Calibration (DCC) applies RAPS to class probabilities and Bonferroni-corrected CQR to box coordinates of IoU-matched detections, giving marginal label and box coverage and a union-bound joint guarantee (). Validation uses LGLDD (4 classes), Kvasir-Capsule (9 classes) and LDPolypVideo (1 class) with four detector configurations. The paper reports detection utility (AP, recall, F1, FP/image), coverage (label/box/joint), efficiency (set size, MPIW), coverage-level and RAPS-hyperparameter sweeps, and extensive qualitative examples.
Critical strengths
- Relevant problem and broad evaluation. Calibrated category and localization uncertainty for endoscopic detection is clinically meaningful. The paper evaluates three datasets and three detector paradigms, with calibration-replicate variance and coverage/hyperparameter sweeps. This breadth is the main point in the paper's favor, but it does not offset the issues below.
Critical weaknesses
-
Reported coverage contradicts the claimed finite-sample guarantees.
- At on LGLDD (Tab. 2), DCC box coverage is 0.82 (R50), 0.85 (DINO) and 0.90 (FCOS). Joint coverage of 0.80 is below the 0.90 union bound.
- On LDPolypVideo (Tab. 13), three of four detectors give 0.84–0.88. Figs. 6–7 show ~0.6 box coverage at nominal 0.90–0.95 (e.g., UA Faster R-CNN-Rxt101).
- With std ≈0.01 over calibration replicates, these gaps are not finite-sample noise. Either exchangeability is violated (e.g., frame-level splits of videos, multiple correlated detections per image) or the implementation differs from the described method. Alg. 3 uses a max-over-coordinates score at level , whereas Eq. 15 uses per-coordinate .
- The abstract nonetheless claims "valid" uncertainty, and validity is the central claim of a conformal method.
-
Limited novelty, with results that are hard to trust.
- DCC is an off-the-shelf combination of RAPS, CQR and Bonferroni/union bounds. PAT is standard inverse-sqrt-frequency reweighting, with no comparison to long-tailed detection methods (EQL v2, Seesaw, RFS, logit adjustment).
- The two stages are not coupled: PAT does not target conformal efficiency.
- Table 3's FCOS rows are identical to its Rxt101 rows (103.93±2.34, 84.14±2.80). Table 2's Rxt101 DCC row (0.95/0.94/30.58) disagrees with Table 3 (0.90/0.88/84.14).
- GFLOPs change under PAT (Tab. 9: FCOS 188→137, DINO 137→197), although PAT should not alter inference.
- App. A.2 theory is incorrect. Thm A.5 treats the -dependent coefficient as constant. Thm A.6 claims convex stays convex.
Questions for authors
Clarifications (could change my assessment):
- Why does DCC box/joint coverage fall well below the nominal levels (Tab. 2, 13; Figs. 6–7)? Are train/calibration/test splits made at the patient/video level, or at the frame level?
- Which box calibration produced the results: Eq. 15 (per-coordinate ) or Alg. 3 (max score at )? What is ?
- Is Eq. 12 marginal or class-conditional calibration? What is the per-class label coverage for rare classes (stm, can)?
- Please explain the duplicated FCOS/Rxt101 rows in Tab. 3, the Tab. 2 vs Tab. 3 mismatch, and the GFLOP changes in Tab. 9.
Suggestions: 5. Compare PAT with EQL v2/Seesaw/logit adjustment under the same detectors. Report an end-to-end coverage metric that counts missed lesions, since the current guarantee is conditional on GT-matched detections.
The discrepancy between theoretical and empirical coverage can arise from finite-sample variability or departures from exchangeability. Angelopoulos and Bates [1] explain that conformal prediction guarantees marginal coverage rather than coverage on every calibration/test split, while Barber et al. [2] examine the effects of exchangeability violations. In medical applications, Yasar et al. [3] report empirical coverage of 0.85 against a theoretical target of 0.90 under cross-cohort evaluation. These findings provide theoretical and empirical warrants for the plausibility of UA-CP's below-target LGLDD joint coverage of 0.80 for Faster R-CNN–R50 and 0.84 for DINO. The train–calibration–test splits were performed at the frame level, as described in Section 5.1 and Appendix C.2.
Box calibration. The reported results use the coordinate-wise calibration specified in Eq. 15. Algorithm 3 also defines coordinate-wise CQR nonconformity scores and maintains separate residual collections for the four bounding-box coordinates. For each coordinate, calibration computes a finite-sample corrected quantile at level 1−αB/4. As established in Proposition A.3, the union bound guarantees simultaneous bounding-box coverage of at least 1−αB under the stated exchangeability assumptions. The matching threshold is τmatch=0.5; intermediate detector predictions are matched one-to-one with ground-truth lesions, retaining pairs with IoU≥0.5, consistent with the IoU threshold used in AP50 evaluation.
Category calibration. The reported results use marginal RAPS calibration with a single conformal threshold computed from all matched calibration detections, as specified in Algorithm 1 and Proposition A.1. We will clarify this distinction to ensure consistency between the formulation and implementation. Under the LGLDD setting (kreg=2,λ=0.01), the UA detectors achieve empirical matched-lesion label coverage of 94.34%/96.95% (R50), 90.70%/98.05% (RxT101), 96.00%/90.86% (DINO), and 93.35%/94.59% (FCOS) for stm/can, respectively, averaged over ten calibration runs.
The inconsistencies between Tables 2 and 3 are reporting errors: the RxT101-UA uncertainty row in Table 3 mistakenly duplicates the FCOS values, whereas the FCOS row is correct. For LGLDD, the corrected Faster R-CNN–RxT101-UA results achieve 0.98 label coverage, 0.95 box coverage, and 0.94 joint coverage, exceeding the nominal target of 0.90. The reporting errors will be corrected in the revised manuscript.
Regarding Table 9, the reported GFLOPs measure detector forward-pass computation at fixed input resolutions. PAT modifies the training objective, whereas DCC performs conformal calibration and post-processing based on intermediate detector predictions. These operations do not alter detector-forward GFLOPs when the inference architecture remains unchanged.
References
[1] Angelopoulos and Bates. "Conformal Prediction: A Gentle Introduction." Foundations and Trends in Machine Learning, 2023. doi:10.1561/2200000101
[2] Barber R F, Candes E J, Ramdas A, et al. Conformal prediction beyond exchangeability[J]. The Annals of Statistics, 2023, 51(2): 816-845.
[3] Yasar S, Yagin F H, Alzakari S A, et al. A Generalizable and Interpretable Framework for Molecular Subtype Classification of Pancreatic Ductal Adenocarcinoma Integrating Conformal Uncertainty Quantification and Consensus-Based Explainable Artificial Intelligence Across Multiple Cohorts[J]. International Journal of Molecular Sciences, 2026, 27(15): 6989.