- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CARE, a framework for improving AI predictions using expensive oracle feedback while adaptively calibrating when to defer. Its components include a residual correction model that adjusts the original model’s probabilities and a calibrated gate that either accepts the corrected prediction or queries the oracle. Queried labels resolve the current decision and are reused for subsequent correction-model learning and calibration, aiming to reduce total oracle queries while controlling the combined system’s error.
Critical strengths
-
Efficient reuse of expensive labels. The framework reuses oracle feedback for prediction correction and calibration while explicitly accounting for adaptive label selection and model updates. This addresses an important limitation of separating scarce labels into fixed training and calibration sets.
-
Evaluation across multiple domains. Experiments cover driving, hate-speech detection, workplace video hazards, and robotic dispatch, with additional comparisons across eight AI detectors on HateBench. The results demonstrate substantial query savings in several settings.
-
Useful ablations and experimental controls. Comparisons examine alternative correction models, separate versus shared data for learning and calibration, and static versus adaptive operation. The evaluation uses ten paired randomized trials and reports standard errors for several results.
Critical weaknesses
-
The connection between the implemented calibration and the theoretical guarantee needs clarification. The experiments demonstrate useful query savings and generally low observed error. However, Appendix A identifies implementation approximations, including sampled and truncated counterfactual replays and shortened inference iterations. Appendix E.2 describes safeguards for approximate weights and scores, but it is unclear whether the required error bounds and threshold adjustments are enforced in the experiments. This limits my confidence that the reported implementation retains the guarantees of Theorem 1 and Corollary 1. (p. 7; p. 13, “Gap between theory and practice”; p. 29, Appendix E.2)
-
Uncertainty is missing for the paired differences in Table 6. The table reports changes in risk and query rate when switching confidence measures, but does not provide standard errors or confidence intervals for these differences. Although the paper provides uncertainty estimates elsewhere, these omissions make it difficult to assess the reliability of smaller changes and the robustness-to-confidence conclusions. (p. 20, Table 6)
-
The separate active-acquisition mechanism is not evaluated. Section 6 motivates separating active acquisition from safety escalation to improve future query efficiency and avoid a nested calibration dependency. In contrast, Appendix A states that the two coincide in the experiments, and Appendix F states that only safety escalation is implemented. This does not necessarily constitute a contradiction, but it leaves the proposed acquisition strategy’s incremental benefit untested and the handling of the calibration dependency unclear. (p. 7, §6; p. 13; pp. 29–30, Appendix F)
Questions for authors
- Does the experimental implementation retain the guarantees of Theorem 1 and Corollary 1? Specifically, are the approximation-error bounds and conservative threshold adjustments in Appendix E.2 computed and enforced? If not, what evidence characterizes the effect of the approximations on risk control?
- Can the authors provide standard errors or confidence intervals for the paired differences in Table 6, particularly the smaller changes used to support robustness to confidence choice?
- Why does the experimental implementation merge active acquisition with safety escalation, and how is the resulting calibration dependency handled? Could an ablation comparing safety-only escalation with the proposed separate acquisition strategy establish its incremental benefit and computational cost?
3 comments
The rebuttal strengthens the paper, and I am maintaining my rating of 3 (Weak Accept).
Dear Reviewer 4Jrr,
Thank you for acknowledging that our rebuttal has strengthened the paper and thank you again for your positive evaluation of our work. We are happy to answer any further questions you may have.
Response to Reviewer 4Jrr
Dear Reviewer 4Jrr,
We sincerely appreciate your thoughtful review and the opportunity to clarify our contributions. We hope the clarifications below address your concerns, and we have incorporated the feedback into the revision.
We kindly refer the reviewer to the general response for our extended discussion of calibration approximations and how we handle the calibration dependency. We have also added new experiments in Appendix A.4 on the effect of calibration approximations on risk control. These new experiments demonstrate non-trivial effect of the calibration approximations as well as our method's robustness to approximations.
Thanks for the catch! Revised Table 6 now reports standard errors (SE). We originally omitted them to avoid cluttering the table, and Figure 3(d) already shows the uncertainty. Accounting for SE, our GPC rectifier is robust to confidence choice, while PPI and SE variants suffer from low-quality confidence measures, reaffirming our robustness-to-confidence conclusions.
We kindly refer the reviewer to the general response for our new evaluations of the active acquisition module and further clarifications about the calibration dependency. Specifically, we have demonstrated that an active acquisition module improves computational efficiency while delivering comparable risk and cost.