acceptodds
Under review as a conference paper at ICLR 2027

Estimating and Communicating Correctness in Agent Handoffs

Abstract

A correctness estimate from three sender responses enables lower-loss agent handoffs through a simple cost-aware decision rule. We fit a compact predictor to three token-probability features and communicate its correctness probability with an explanation. The sampling budget, predictor, and decision policy are configurable. The receiver reads only the note and selects a fault or requests specialist investigation. The estimator has useful ranking and lower probability error than raw self-reported confidence. On 125 incidents, its message reduces wrong diagnoses by 5.2 percentage points and increases escalation by 4.8 points relative to no warning. When escalation costs half as much as a wrong diagnosis, a gate on the same estimate lowers loss from 0.408 to 0.360 relative to the recorded probability-message decisions. Two simpler numerical messages do not reproduce this effect. Adding disagreement detail, even a complete twenty-answer tally, has no detectable effect under the original instructions. A separate comparison removes the explicit diagnosis while retaining evidence and votes. Fewer wrong diagnoses and more escalations recur on held-out incidents and with a second sender, with a clear effect in one of three receiver families. We evaluate both changes against the extra cost of escalation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, uncertainty quantification, confidence calibration, selective prediction, agent handoffs, uncertainty communication, abstention, root cause analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Mec9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to communicate diagnostic reliability in agent handoffs. It proposes a lightweight correctness estimator based on three sender responses and token-level probability features, then uses the estimated probability to guide receiver decisions or cost-aware escalation. Experiments show that correctness-probability messages reduce wrong diagnoses, while a direct escalation gate further lowers decision loss.

Critical strengths
  • The paper addresses a practical challenge in multi-agent systems by connecting uncertainty communication with downstream decisions through an explicit cost-sensitive framework.

  • The proposed estimator achieves useful correctness prediction with only three sender responses, reducing sender sampling costs by approximately 85% compared with twenty-response methods.

  • The evaluation includes extensive paired comparisons of reliability estimates, message formats, and decision policies, supplemented by additional senders, receivers, and cross-domain experiments.

Critical weaknesses
  • Appendix C.1–C.2 shows that the three-feature estimator offers no statistically significant improvement over a single token-gap feature or calibrated self-reported confidence. Their Brier scores are 0.200, 0.203, and 0.218, respectively.
  • In Section 4.3 and Appendix K.3, probability messages differ from the baselines not only in numerical estimates but also in source attribution and calibration explanations, making it difficult to isolate the contribution of estimation quality. Moreover, the 5.2-point reduction in wrong diagnoses is accompanied by a 4.8-point increase in escalation, while correct diagnoses increase by only 0.4 points. The observed benefit therefore primarily reflects increased deferral rather than improved diagnostic accuracy.
  • Table 1 reports an AUROC of .762 for the raw token gap versus .756 for the three-feature estimator. Appendix C further shows that gap-only logistic regression achieves comparable Brier scores, while the improvement over calibrated self-reported confidence is statistically insignificant.
Questions for authors

see wealmess.

Reviewer_eY78OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how one language model should communicate a diagnosis and its reliability to another model that must choose a fault or request specialist investigation. It fits logistic regression to three token-probability features from three sender responses, communicates the estimated correctness probability in a handoff note, and evaluates a cost-based escalation gate. A separate experiment removes the explicit diagnosis while keeping the evidence and vote tally fixed.

Evaluation uses 165 RCAEval incidents, including a 125-incident probability-message comparison, plus another sender and transfer checks. Compared with no warning, the probability message reduces wrong diagnoses by 5.2 percentage points and increases escalation by 4.8 points. At an escalation cost of 0.5, the gate reduces loss from 0.408 to 0.360 relative to recorded probability-message decisions. Diagnosis removal has clear support for one receiver family, while the transfer experiments show substantial limits to generalization.

Critical strengths
  1. The probability-message experiment provides useful controlled evidence. It keeps the diagnosis and evidence fixed, matches note lengths, and finds a 5.2-point reduction in wrong diagnoses, with a simultaneous 95% interval of [−8.0, −2.4], alongside 4.8 points more escalation (Table 3). Both main receivers show a reduction relative to no warning, which strengthens the finding beyond a single-model observation (Table 24).

  2. The estimator evaluation and study reporting support the credibility of the results. At three responses, the predictor reaches AUROC 0.756 versus 0.576 for leading vote share, using whole-system holdouts, and it retains useful ranking among unanimously answered incidents (Table 1; Appendix C.4). The authors also clearly report reused controls, failed continuation criteria, and unsuccessful transfer checks, which makes the scope of the evidence easier to judge (Appendices A, E, and I).

Critical weaknesses
  1. The strongest missing baseline is calibrated self-report as a handoff message. Its Brier score is 0.218 versus 0.200 for the proposed estimator, with an inconclusive paired difference, p=.359p=.359, yet the receiver experiment uses raw self-report and changes the explanation alongside the values (Appendices C.2 and K.3). Without communicating calibrated self-report under the same wording, the study does not establish what the token-feature estimator adds to a simpler calibrated-confidence handoff.

  2. The gate comparison gives the policies different cost information. The gate receives numerical cost cc, while the receiver decisions remain fixed under qualitative cost instructions, so the reported 23.5% average loss reduction does not establish an advantage over receivers given the same objective, even though the paper discloses this limitation (Section 6.1; Appendix J.2).

Questions for authors
  1. Does the proposed estimate improve receiver outcomes or gate loss over cross-fitted calibrated self-report when both are communicated using the same explanation?
  2. Does the gate retain its loss advantage when receivers receive the same numerical escalation cost, and how does it compare with a simple gate using a correctness base rate estimated only from training incidents?
  3. Are there any results for the actual learned-probability message or gate on Cloud-OpsBench, beyond the assigned-number experiment, or with a sender given the complete deployment guidance?
Reviewer_tG6wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how reliability information should be estimated and communicated when one agent hands a diagnosis to another. It fits a cross-system correctness predictor from token-probability features of three sender responses and compares probability messages, disagreement summaries, diagnosis removal, and a cost-threshold gate. On RCAEval, the probability message reduces wrong diagnoses relative to no warning, while a direct gate further reduces loss at low and moderate escalation costs. Additional studies characterize receiver dependence and limited transfer to tool-based investigations.

Critical strengths

The paper studies an interesting problem: how uncertainty information should be communicated between agents and whether it can help them make better decisions. I think it is useful that the authors look beyond prediction accuracy and also consider how the receiver responds to different messages. The paper includes several experiments with different uncertainty measures, message formats, and receiver models. The authors also report negative results and limitations, which help provide a more complete picture of the problem.

Critical weaknesses

My main concern is the limited novelty of the proposed method. The core idea is to estimate whether a model’s answer is correct and use this estimate to decide whether to accept the answer or escalate. However, this problem has already been widely studied in uncertainty estimation, selective prediction, and learning to defer. The proposed method mainly combines existing token-level uncertainty features with logistic regression and a simple threshold-based decision rule. The paper does not clearly explain what new technical insight this combination provides. The theoretical analysis also seems extremely limited, as the main decision rule follows directly from classical selective prediction. I think the authors need to better explain what is fundamentally new compared with existing methods.

I am also concerned about the lack of comparison with recent methods. The paper discusses several related works on uncertainty estimation, model routing, and deferral, but the experiments mainly compare against simple baselines such as voting agreement, entropy, and self-reported confidence. There is no direct comparison with more recent approaches such as UCCI, ReDAct, or other uncertainty-aware routing methods. Since the paper claims that its correctness estimates lead to better handoff decisions, I think comparisons with these methods are important. Without them, it is difficult to understand whether the proposed approach actually improves over existing techniques or simply demonstrates something that is already well known.

Another major concern is whether the experiments really support the motivation of improving agent handoffs. The paper mainly studies a simplified setting where one agent provides a diagnosis and another agent either accepts it or escalates. However, the receiver only sees a written note, and the sender is deliberately given incomplete domain knowledge. This creates an artificial situation where the sender is more likely to make mistakes. I am not sure how well this represents real multi-agent systems, where agents may share more information, use tools, or ask follow-up questions. The experiments show that adding confidence information can change receiver decisions, but this does not necessarily demonstrate that the proposed method improves realistic agent collaboration.

I also think the paper does not fully establish the practical value of its main results. For example, the cost-aware gate performs better than the receiver, but this is largely expected because the gate uses an explicit cost threshold while the receiver does not receive the same numerical cost information. More importantly, the paper does not demonstrate that the complete method leads to better outcomes in an actual multi-agent workflow. The improvements are measured mainly through diagnosis accuracy and escalation rates, without evaluating what happens after escalation. Therefore, I think the conclusions about improving agent handoffs are stronger than what the experiments currently demonstrate.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.