- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to communicate diagnostic reliability in agent handoffs. It proposes a lightweight correctness estimator based on three sender responses and token-level probability features, then uses the estimated probability to guide receiver decisions or cost-aware escalation. Experiments show that correctness-probability messages reduce wrong diagnoses, while a direct escalation gate further lowers decision loss.
Critical strengths
-
The paper addresses a practical challenge in multi-agent systems by connecting uncertainty communication with downstream decisions through an explicit cost-sensitive framework.
-
The proposed estimator achieves useful correctness prediction with only three sender responses, reducing sender sampling costs by approximately 85% compared with twenty-response methods.
-
The evaluation includes extensive paired comparisons of reliability estimates, message formats, and decision policies, supplemented by additional senders, receivers, and cross-domain experiments.
Critical weaknesses
- Appendix C.1–C.2 shows that the three-feature estimator offers no statistically significant improvement over a single token-gap feature or calibrated self-reported confidence. Their Brier scores are 0.200, 0.203, and 0.218, respectively.
- In Section 4.3 and Appendix K.3, probability messages differ from the baselines not only in numerical estimates but also in source attribution and calibration explanations, making it difficult to isolate the contribution of estimation quality. Moreover, the 5.2-point reduction in wrong diagnoses is accompanied by a 4.8-point increase in escalation, while correct diagnoses increase by only 0.4 points. The observed benefit therefore primarily reflects increased deferral rather than improved diagnostic accuracy.
- Table 1 reports an AUROC of .762 for the raw token gap versus .756 for the three-feature estimator. Appendix C further shows that gap-only logistic regression achieves comparable Brier scores, while the improvement over calibrated self-reported confidence is statistically insignificant.
Questions for authors
see wealmess.