acceptodds
Under review as a conference paper at ICLR 2027

Careful Judge: Safe and Efficient Human–AI Collaborative Decision Making

13 views

Abstract

In human–AI collaborative decision making, human review can prevent unsafe AI decisions, but each human judgment is costly. Treating human intervention as a one-off fallback misses the opportunity to improve future AI decisions, yet adaptively learning from human feedback changes the model and can invalidate previously calibrated safety guardrails. We approach this challenge with CARE—**c**alibrated **a**daptive **r**ectification and **e**scalation—an end-to-end pipeline that combines AI models and human reviewers to guarantee safe, human-aligned decisions, while continuously learning from human feedback to achieve greater automation with fewer human queries. CARE is principled, general, modular, and works with any black-box AI model. We equip it with a rectification module that improves AI decisions by learning the structure of human–AI misalignment. Our novel adaptive calibration module guarantees risk control at every time step for any rectification module. Experiments on four safety-critical real-world datasets spanning language, vision, driving, and robotics demonstrate that CARE achieves human-aligned decisions while reducing human queries by 25–81% relative to baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Human-AI Collaboration, Model Alignment, Adaptive Learning, Calibration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4JrrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CARE, a framework for improving AI predictions using expensive oracle feedback while adaptively calibrating when to defer. Its components include a residual correction model that adjusts the original model’s probabilities and a calibrated gate that either accepts the corrected prediction or queries the oracle. Queried labels resolve the current decision and are reused for subsequent correction-model learning and calibration, aiming to reduce total oracle queries while controlling the combined system’s error.

Critical strengths
  1. Efficient reuse of expensive labels. The framework reuses oracle feedback for prediction correction and calibration while explicitly accounting for adaptive label selection and model updates. This addresses an important limitation of separating scarce labels into fixed training and calibration sets.

  2. Evaluation across multiple domains. Experiments cover driving, hate-speech detection, workplace video hazards, and robotic dispatch, with additional comparisons across eight AI detectors on HateBench. The results demonstrate substantial query savings in several settings.

  3. Useful ablations and experimental controls. Comparisons examine alternative correction models, separate versus shared data for learning and calibration, and static versus adaptive operation. The evaluation uses ten paired randomized trials and reports standard errors for several results.

Critical weaknesses
  1. The connection between the implemented calibration and the theoretical guarantee needs clarification. The experiments demonstrate useful query savings and generally low observed error. However, Appendix A identifies implementation approximations, including sampled and truncated counterfactual replays and shortened inference iterations. Appendix E.2 describes safeguards for approximate weights and scores, but it is unclear whether the required error bounds and threshold adjustments are enforced in the experiments. This limits my confidence that the reported implementation retains the guarantees of Theorem 1 and Corollary 1. (p. 7; p. 13, “Gap between theory and practice”; p. 29, Appendix E.2)

  2. Uncertainty is missing for the paired differences in Table 6. The table reports changes in risk and query rate when switching confidence measures, but does not provide standard errors or confidence intervals for these differences. Although the paper provides uncertainty estimates elsewhere, these omissions make it difficult to assess the reliability of smaller changes and the robustness-to-confidence conclusions. (p. 20, Table 6)

  3. The separate active-acquisition mechanism is not evaluated. Section 6 motivates separating active acquisition from safety escalation to improve future query efficiency and avoid a nested calibration dependency. In contrast, Appendix A states that the two coincide in the experiments, and Appendix F states that only safety escalation is implemented. This does not necessarily constitute a contradiction, but it leaves the proposed acquisition strategy’s incremental benefit untested and the handling of the calibration dependency unclear. (p. 7, §6; p. 13; pp. 29–30, Appendix F)

Questions for authors
  1. Does the experimental implementation retain the guarantees of Theorem 1 and Corollary 1? Specifically, are the approximation-error bounds and conservative threshold adjustments in Appendix E.2 computed and enforced? If not, what evidence characterizes the effect of the approximations on risk control?
  2. Can the authors provide standard errors or confidence intervals for the paired differences in Table 6, particularly the smaller changes used to support robustness to confidence choice?
  3. Why does the experimental implementation merge active acquisition with safety escalation, and how is the resulting calibration dependency handled? Could an ablation comparing safety-only escalation with the proposed separate acquisition strategy establish its incremental benefit and computational cost?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4Jrr

Dear Reviewer 4Jrr,

We sincerely appreciate your thoughtful review and the opportunity to clarify our contributions. We hope the clarifications below address your concerns, and we have incorporated the feedback into the revision.

W1 & Q1. The connection between the implemented calibration and the theoretical guarantee needs clarification.

We kindly refer the reviewer to the general response for our extended discussion of calibration approximations and how we handle the calibration dependency. We have also added new experiments in Appendix A.4 on the effect of calibration approximations on risk control. These new experiments demonstrate non-trivial effect of the calibration approximations as well as our method's robustness to approximations.

W2 & Q2. Uncertainty is missing for the paired differences in Table 6.

Thanks for the catch! Revised Table 6 now reports standard errors (SE). We originally omitted them to avoid cluttering the table, and Figure 3(d) already shows the uncertainty. Accounting for SE, our GPC rectifier is robust to confidence choice, while PPI and SE variants suffer from low-quality confidence measures, reaffirming our robustness-to-confidence conclusions.

W3 & Q3. The separate active-acquisition mechanism is not evaluated.

We kindly refer the reviewer to the general response for our new evaluations of the active acquisition module and further clarifications about the calibration dependency. Specifically, we have demonstrated that an active acquisition module improves computational efficiency while delivering comparable risk and cost.

The rebuttal strengthens the paper, and I am maintaining my rating of 3 (Weak Accept).

Dear Reviewer 4Jrr,

Thank you for acknowledging that our rebuttal has strengthened the paper and thank you again for your positive evaluation of our work. We are happy to answer any further questions you may have.

Reviewer_H7GuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

authors introduce a method called CARE which is an end-to-end pipeline for human-AI collaborative decision making that guarantees user specified error rates while minimizing costly human queries. CARE uses a gaussian process rectifier with a weighted cross-conformal calibration scheme that reuses human feedback to adapt safety thresholds as the model improves.

Critical strengths
  • the evaluation is end-to-end. including ablations, the experiments are thorough and reasonable.
Critical weaknesses
  • missing related literatur e: [1], [2], [3]
  • sec 6 and fig 1 both mention that a seperate acquisition module is needed but in the appendix authors use safety escalation. so one of the four used modules is not tested and the tested model has discrepancies with the presented model
  • only the zero-one loss is used in the experiments. in robotics, false negatives and false positives have different costs and they accompany different meanings/semantics.
  • the paper starts by claiming that they care to solve "human" ai collaboration. even it's mentioned in the title. however, in nuplan, the ai part is an open loop rollout and the human is a closed loop sim. also the setup assumes the oracle is always correct.

[1] Danesh et al. "YRC-Bench: A Benchmark for Learning to Coordinate with Experts." TMLR.

[2] Trinh et al. "Getting by goal misgeneralization with a little help from a mentor." arXiv.

[3] Plaut et al. "Safe Learning Under Irreversible Dynamics via Asking for Help." JMLR.

[4] Plaut et al. "Avoiding catastrophe in online learning by asking for help." ICML.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer H7Gu

Dear Reviewer H7Gu,

Thank you for your review and suggestions. Below, we describe the clarifications and additional experiments included in the revised manuscript.

W1. Missing related literature.

We thank the reviewer for pointing out related literature, which shares the same setting of efficiently seeking human expertise in safety-critical tasks as ours. We have added these references in the related work section.

W2. Missing the acquisition module.

We kindly refer the reviewer to the general response G1 for our further clarifications and new experiments testing the active acquisition module. In the revised manuscript, the clarifications are incorporated in Section 6 and the new experiments are in Appendix A.5.

W3. Only the zero-one loss is used in the experiments.

We fully agree with the reviewer that in robotics false negatives and false positives should be weighted differently in the loss. We would like to offer three layers of discussion on this point. First, our framework is compatible with general user-specified loss functions (Section 1.2). In particular, our risk decomposition (Proposition 1) applies to the Brier score, 0–1 loss, and beyond, and thus the ideas behind the four modules of CARE apply to general loss functions.

Second, our theory readily extends to asymmetric 0–1 loss. Without loss of generality (after normalization and exchanging binary labels), consider L(y^,y)=1y^=1,y=0+a1y^=0,y=1L(\hat{y},y) = \mathbb{1}\\{ \hat{y}=1,y=0 \\} + a\mathbb{1}\\{ \hat{y}=0,y=1 \\} with a∈[0,1]a \in[0,1] (false positives are more dangerous). All modules are identical, except that the model definition Mt=(ft,ct)\mathsf M_t = (f_t,c_t) becomes ft(x)=1pt(x)≥1/(1+a)f_{t}(x) = \mathbb{1}\\{ p_t(x) \ge 1/(1+a) \\} and ct(x)=max⁡pt(x),1−apt(x)c_t (x)= \max\\{ p_t(x), 1-ap_t(x) \\} to match the new risk. Since L(y^,y)≤1y^≠yL(\hat y,y)\le\mathbb{1}\\{\hat y\ne y\\}, Corollary 1 gives E[L(Y^t,Yt)]≤P(Y^t≠Yt)≤ϵ\mathbb{E}[L(\hat Y_t,Y_t)]\le P(\hat Y_t\ne Y_t)\le\epsilon at every fixed time under the same conditions. For unnormalized costs bounded by BB, calibrating at ϵ/B\epsilon/B instead yields asymmetric risk at most ϵ\epsilon.

Third, we added experiments on the robotics dataset ASIMOV in Appendix A.4 testing methods' performance under this asymmetric loss. Our results show that CARE is robust to, and adapts to, asymmetric loss. Please refer to the revised Appendix A.4 for details; we have also updated Section 1.2 to mention this general loss function.

W4. In nuplan, the AI part is an open loop rollout and the human is a closed loop sim.

Using high-fidelity closed-loop simulation as a proxy for real-world human driving is a common practice consistent with the literature. We also incorporated this experiment to showcase the generality of our method, which applies to general asymmetric collaborative decision-making settings. In such settings, "human" refers to an expensive high-fidelity oracle and "AI" refers to an inexpensive low-fidelity model.

In this work, we focus on a noiseless oracle, consistent with the selective classification literature (El-Yaniv & Wiener, 2010). Our setup readily accommodates noisy oracles by incorporating their induced aleatoric risk into the first term of Proposition 1. However, one can see that this additional risk is irreducible and shared across all methods, even the all-human baseline. Since the work focuses on the relative performance of methods compared to the all-human baseline, we reset the risk baseline to zero by considering a noiseless oracle. Considering a noisy oracle leads to a non-zero risk baseline, e.g., curves in Figure 3(a) would be lifted up by a constant amount, rather than starting from zero.

An interesting direction is to consider a third-party noiseless oracle when the human is noisy. Then the baseline is no longer all-human, and AI may have better accuracy in some regions. We leave this for future work.

Thank the authors for the updates. These additions address my first three concerns, and I'm raising my score to 3.

Dear Reviewer H7Gu,

We are glad that our rebuttal has addressed your concerns, and thank you for raising the score.

Reviewer_PHqrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CARE, a framework that learns from human feedback while deciding when an AI prediction needs human review. It uses a Gaussian-process model to correct errors in the original predictor, together with a weighted cross-conformal procedure to calibrate confidence as the model and the labeled sample change. Under the stated assumptions, the method guarantees marginal error control at each fixed time step. Experiments across driving, hate-speech detection, robotic safety, and workplace-hazard detection show reductions in oracle queries compared with static correction and escalation methods.

Critical strengths
  1. The paper takes the statistical consequences of adaptive feedback seriously. Labels collected through selective review cannot simply be treated as an ordinary calibration sample, especially when they also update the predictor.
  2. Accounting for the acquisition history and maintaining evolving held-out models is a promising way to address this problem.
  3. The analyzed distinction between marginal risk control at a fixed time and adaptation to local errors is also useful.
  4. The experiments cover a reasonably broad range of settings.
Critical weaknesses
  1. I do not yet see how the theoretical guarantee carries over to the method used in the experiments. Section 6 separates active acquisition from safety escalation to avoid a nested dependence on calibration. Observing aggregate error below the target is encouraging, but it does not establish the claimed marginal guarantee at each time step. The safety claim also depends on an i.i.d. stream and a correct oracle; those assumptions should remain explicit.
  2. There may be missing an implementation detail, but the small particle counts seem difficult to reconcile with Equation (2).
  3. My third question is whether the implementation uses multiple calibration scores per fold, a different normalization, or some other modification, and, if so, whether the proof covers that version.
Questions for authors
  1. Could you specify the exact calibration statistic used in the experiments?
  2. In particular, how do the particle counts and test-group weight allow nontrivial automation at the smaller tolerances, especially for static CARE?
  3. What bounds are available for the errors introduced by sampled replay and history truncation?
  4. When active acquisition and safety escalation coincide, how do you replay the acquisition policy under counterfactual histories without the nested calibration dependence discussed?
2 comments
AuthorsOpenReview ↗

Response to Reviewer PHqr [1/2]

Dear Reviewer PHqr,

Thank you for your thoughtful review and positive feedback on our work. Below, we provide detailed responses to your questions.

W1.1 & Q3–4. Section 6 separates active acquisition from safety escalation to avoid a nested dependence on calibration. What bounds are available for the approximation errors? How do you replay the acquisition policy when active acquisition and safety escalation coincide?

Our theory covers the double escalation mechanism separating safety escalation and active acquisition. In our main empirical studies, for simplicity, we did not implement a separate active acquisition module and thus active acquisition coincides with safety escalation. During escalation policy replay, we fix the historical safety escalation thresholds. This is a first-order approximation to the nested calibration dependency.

We kindly refer the reviewer to the general response G1 & 2 for our further clarifications on the active acquisition module, approximation errors, and acquisition policy replay. In the revised manuscript, we have incorporated these discussions in Section 6, and included new experiments on the active acquisition module (Appendix A.5) and approximation errors (Appendix A.4).

W1.2. Observing aggregate error below the target does not establish the claimed marginal guarantee at each time step.

In addition to the time-aggregate error plots in the main text, we also plotted rolling-window risk over time for every dataset in Figures 6, 8, 10, and 12 (in Appendix). These risk-over-time plots provide temporal empirical evidence for our risk control guarantee. Since we only run 10 trials, we use rolling windows to reduce noise from individual binary outcomes. Due to the finite-sample nature, the empirical risk may slightly exceed the target ϵ\epsilon at a few time steps, which does not by itself contradict our marginal guarantee.

W1.3. The safety claim also depends on an i.i.d. stream and a correct oracle.

The reviewer correctly identifies our assumptions of an i.i.d. stream and a correct human oracle, which are stated in the problem setup section. The i.i.d. input stream design choice is motivated by our application setup. In all of our experiments, datasets can be collected from any distribution. For example, the nuPlan dataset contains temporally correlated trajectory data points. But the input stream fed into the algorithm is sampled from the dataset in an i.i.d manner. In Appendix A.5, we also test our method on some non-i.i.d. input streams, demonstrating its robustness. We'd also like to highlight that this work already relaxes many assumptions in conformal calibration (Tibshirani et al., 2019; Prinster et al., 2022; Gibbs & Candès, 2021; 2024) and considers a very flexible setting, including offline and online stages, non-stationary data acquisition, and adaptive model updates. We leave more general settings, such as exogenous distribution shifts and noisy human feedback, for future work.

Regarding the data model, we'd also like to remark that even with an i.i.d. input stream, selective escalation produces a non-i.i.d. and non-stationary label stream. Specifically, our data model features three sources of distribution shift: selective escalation, adaptive acquisition, and adaptive model updates. The challenges introduced by these distribution shifts in the conformal calibration setting are acknowledged by prior work, and our work is the first to establish a marginal coverage guarantee at every time step in this setting.

In this work, we focus on a noiseless oracle, consistent with the selective classification literature (El-Yaniv & Wiener, 2010). Our setup readily accommodates noisy oracles by incorporating their induced aleatoric risk into the first term of Proposition 1. However, one can see that this additional risk is irreducible and shared across all methods, even the all-human baseline. Since the work focuses on the relative performance of methods compared to the all-human baseline, we reset the risk baseline to zero by considering a noiseless oracle. Considering a noisy oracle leads to a non-zero risk baseline, e.g., curves in Figure 3(a) would be lifted up by a constant amount, rather than starting from zero.

An interesting direction is to consider a third-party noiseless oracle when the human is noisy. Then the baseline is no longer all-human, and AI may have better accuracy in some regions. We leave this for future work.

AuthorsOpenReview ↗

Response to Reviewer PHqr [2/2]

W2–3 & Q1–2. Missing implementation details. How does the theory cover the implementation?

As the reviewer correctly points out, our implementation in src/care/calibration uses multiple scores per fold. Specifically, given nn samples and KK folds, we maintain KK models (hence KK score functions in Eq. (3)(3)), but evaluate each sample with its held-out fold model to obtain nn scores, giving a nontrivial prediction set (2)(2) by replacing the summation over [K][K] with that over [n][n]. The fold likelihood is shared across its samples, and thus they have the same weights after normalization. As the reviewer points out, in static CARE with uniform weights, a necessary condition for a nontrivial prediction set in Eq. (2)(2) is K/(K+1)≥1−ϵK / (K+1) \ge 1-\epsilon. With nn scores, this condition becomes n/(n+1)≥1−ϵn / (n+1) \ge 1-\epsilon. So the condition can be satisfied by increasing the number of calibration samples without increasing the number of fitted models.

Our implementation with nn scores is consistent with cross-conformal methods (Romano et al., 2020; Barber et al., 2021), and our analysis readily extends to their results (especially for static CARE), which, however, gives an excess miscoverage rate of \min\left\\{\frac{2(1-1/K)}{n/K+1},\frac{1-K/n}{K+1}\right\\}, which is nonzero for K∉{1,n}K\not\in \{ 1,n \}; this is known to be the price of finite-fold correction (Romano et al., 2020, Theorem 2; Barber et al., 2021, Theorem 4). However, we show that this price is paid for the finer granularity by looking into each fold for multiple scores. Our novel proof shows that, by choosing one representative score from each fold, this excess miscoverage rate is gone. Intuitively, each leave-one-fold-out model should only provide one conformity check, and thus KK folds are no different from using KK samples, which should not have any excess miscoverage.

However, the theory-implementation tradeoff is that using KK scores (even when KK is large enough to permit automation) may exhibit lower empirical sample size than using nn scores, resulting in more conservative prediction sets.

In summary, our theory uses one representative score per fold to establish a simpler miscoverage bound without an excess term, while our implementation follows cross-conformal methods by using all samples to construct scores, with shared within-fold score and weight functions. Our theory extends to our implementation by applying the finite-fold correction above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We thank Reviewers PHqr, H7Gu, and 4Jrr for their thoughtful reviews and positive feedback, particularly on the breadth and thoroughness of our experiments (all three reviewers), as well as our careful treatment of adaptive feedback and efficient reuse of labels (Reviewers PHqr and 4Jrr).

We are also grateful for the suggestions for improvement, which we have incorporated into the revised manuscript, with changes highlighted in orange\textcolor{orange}{\text{orange}}. We address each reviewer’s comments in the dedicated responses below. Here, we clarify how the theoretical calibration construction relates to its computational implementation, expanding our existing discussion of limitations in Appendix A (also incorporated in the revised manuscript).

G1. Active acquisition. Manuscript revisions: Section 6 & Appendix A.5

We first clarify the motivations and trade-offs of a separate active acquisition module. Its independence from the safety escalation rule removes the nested calibration dependency during counterfactual policy replay. In addition, a sophisticated lookahead acquisition policy may bring long-term efficiency gains. However, as noted in Section 6 (revised) and Appendix F, it can increase short-term cost as safely escalated queries are not used for calibration or rectifier learning until the next offline checkpoint, obscuring potential longer-term efficiency gains.

We added an active acquisition module ablation on all four datasets in Appendix A.5. It yields broadly similar empirical risk and query complexity, while reducing end-to-end runtime by 11–26%. When using safety-only escalation, the reported experiments keep historical escalation thresholds fixed for escalation replay. This avoids recursion through a one-level approximation. This approximation is equivalent to calibrating an active acquisition rule with a fixed propensity score. Our main benchmark studies use safety-only escalation for simplicity. In these settings, this approximation is empirically safe, efficient, and robust, in the absence of a separate active acquisition module.

We also stress-tested safety-only escalation on pathologically constructed nuPlan data streams (violating the i.i.d. stream setup of our theory and main empirical studies).
In some pathological trials, safety-only escalation stops escalating after an initial period, causing feedback starvation and risk exceedance. In contrast, an active acquisition module is robust to this issue.

G2. Computational approximation. Manuscript revisions: Appendix A.4

Our method involves two sources of approximation, for rectifier learning and for calibration.
Our calibration is model-agnostic, so any computational approximations for rectifier learning are strictly covered by our theory without needing to know their approximation error bounds.
Appendix E.2 further covers approximations for calibration weights and scores when error bounds are available.
As described in Appendix A, these calibration approximations include a truncated replay horizon and simplified replay updates. In the reported experiments, we evaluate these approximations empirically without applying the bound-based correction in Appendix E.2. The observed controlled empirical risk demonstrates our method's robustness against approximation. We have also added new approximation sensitivity experiments in Appendix A.4.

A promising future extension operationalizing Appendix E.2 is a model-agnostic certification of the approximation errors. Bounding the error of each approximation mechanism separately can be expensive and difficult. Instead, we can use additional independent audit particles to obtain high-probability error bounds. The audit particles should maintain exact weights and scores, or their approximations with known error bounds. A concentration argument can then convert their discrepancies from the deployed approximation into high-probability error bounds. The trade-off is the additional computation and storage needed to maintain these audit particles and construct the certificate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.