- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether calibrated AI confidence remains informative in delegation when agents can influence users’ decisions. It models the “Confidence Game,” a repeated signaling game where agents report confidence, users choose delegation, and outcomes are observed only under delegation. Theory shows strategic incentives can make truthful reporting unsustainable, especially with short-term gains. LLM experiments find confidence overstatement even when success probabilities are known. this reduces signal informativeness and undermines delegation benefits.
Critical strengths
-
The paper provides a reframing of confidence reporting in agentic AI as a strategic signaling problem rather than merely a calibration problem. The combination of hidden ability and honesty, endogenous monitoring, and reputation dynamics yields sharp equilibrium predictions about truthful reporting, confidence inflation, and under-reporting.
-
The empirical evaluation explicitly provides ground-truth success probabilities, helping distinguish strategic distortion from ordinary calibration errors, systematically explores the theoretical state space, examines protocol variations, and observes similar behavior in real-task settings. The behavioral and welfare analyses further investigate the mechanism and quantify the cost of reduced signal informativeness.
Critical weaknesses
-
All empirical experiments use a single model, gpt-oss-120b, under a fixed prompting and decoding configuration. Without cross-model or cross-capability evaluation, it remains unclear whether the observed strategic confidence distortion generalizes beyond the evaluated model and experimental setup.
-
The toy prompt explicitly instructs the model to behave strategically and maximize its payoff, potentially inducing the behavior being measured. Moreover, the substantial sensitivity to elicitation protocol raises questions about whether the observed distortion reflects a robust strategic tendency or a protocol-specific response. The real-task experiment also does not directly test the theory's comparative-static predictions.
Questions for authors
-
How can the authors disentangle strategic confidence distortion from behavior induced by the elicitation prompt? In particular, does confidence inflation persist under a neutral prompt and when the delegation-related payoff is removed?
-
How should the substantial sensitivity to elicitation protocol be interpreted? Which empirical conclusions remain robust across protocols, and should the magnitude of the observed distortion be regarded as a stable behavioral property?
-
To what extent do the real-task results validate the theoretical comparative-static predictions? The theoretical model makes predictions about how reporting behavior changes with parameters, but these comparative statics are not directly estimated in the real-task setting.
2 comments
Response to Reviewer zUdq (part 2 of 2)
Protocol Sensitivity (Q2)
This is an important question, and we have added a table to the paper with every protocol and prompt variant (Appendix O). The appendix now opens by separating the conclusions that hold across the five elicitation protocols from those that vary. In short, the direction of the distortion and its welfare consequences are robust; its magnitude is not.
- Holds under all five protocols: almost no under-reporting (the high-report rate on easy draws is at least 0.98); majority inflation on hard draws at 75 to 100 of the 100 states; and a naive welfare loss of 68% to 85% of the gains from trade (under full second-round inflation, our upper bound).
- Varies with the protocol: the level of inflation (from 0.56 to 0.86) and responsiveness to parameters (the user's beliefs and the agent's patience).
The paper now reports the level as a range, 0.56 to 0.86. The main protocol, which we report throughout, is its low end, so our headline misreporting and welfare figures sit at the conservative end.
Real-task evaluation (Q3)
The reviewer is right that the real-task experiment does not estimate the comparative statics in the user's beliefs and the agent's patience. Those require a known success probability crossed with a full grid of beliefs, which is the controlled setting of Section 5, where we estimate them. The real task instead tests the theory's qualitative predictions when the success probability is the model's own uncertainty. Holding the model's earlier answer fixed, so that only the report can move, it passes the three we can test:
- Inflation concentrates where the model's own confidence is low. The report exceeds the model's stated estimate by 0.40 to 0.55 when that estimate is below 0.6, and by at most 0.13 above it.
- It rises with the fee. On hard questions, mean confidence is 0.63 when the fee is paid regardless and 0.91 when it depends on delegation (0.30 under the plain prompt).
- The report loses information. The share of wrong answers reported at 0.9 or above rises from 24% to 66%.
The revision states this scope in the real-task section. Further details are in our response to Reviewer pFpn (part 2) where we describe new real-task experiments and their results.
Response to Reviewer zUdq (part 1 of 2)
We thank the reviewer for their thoughtful feedback. We respond to their comments below and detail the corresponding revisions made to the manuscript.
Prompt-induced behavior (W2, Q1)
Owing to this comment and similar feedback from other reviewers, we ran several new experiments which further substantiate that the observed behavior is indeed a strategic response to the incentives of the Confidence Game. We have rebuilt Section 5 around these new results and other reviewer feedback.
We first wish to clarify the question Section 5 is designed to answer. A user who delegates to an AI agent wants its confidence to tell her when it is likely to succeed, but this need not be the objective of the agent, which is set by its training and deployment. A revenue-motivated provider who is paid per token might seek to drive delegation regardless of success. This misalignment can also arise without intent as tuning on human feedback has been shown to induce sycophantic, manipulative and deceptive behavior (Sharma et al., 2025; Williams et al., 2025). Either way the agent then has a strategic incentive to manipulate its confidence reports.
Our theory asks what an agent with such an objective would do. Our experiments ask whether a current LLM, once given that objective, can and does act on it. The claim is not that deployed assistants necessarily manipulate confidence. It is that a model with such an objective (given or learned) has the capability to manipulate its reports strategically. The reviewer’s comment then asks whether the resulting behavior is driven by the incentives or the prompt wording itself.
To test this, we ran three new groups of prompts:
The first group tests whether the behavior is solely role-playing, the second tests whether a model distorts reports under implied incentives but without explicit instruction, and the third provides a control to test whether the model is truly responding to the underlying incentives.
We see that the rate of over-reporting across the first group of prompt variants is at least 49% (similar to our originally reported rate of 56%). This drops to just 8.5% when the fee is paid regardless of delegation, indicating that it is indeed a response to the incentive structure. Furthermore, significant over-reporting persists even when the model is not instructed to maximize any payoff. The structure the theory predicts also survives without the label: inflation still rises with the user’s trust, and about nine in ten low reports on easy tasks occur where trust in honesty is below 1/2, the only region where Thm 4.4 permits under-reporting.
Cross-model evaluations (W1)
We agree that only evaluating one model was a limitation, and we have added two models and varied the decoding temperature for gpt-oss-120b (Section 5 and Table 1; Appendix O.3). Below we report the over-reporting rates across these experiments.
The GLM models inflate more even without the incentive, but tying the fee to delegation still adds 42 to 49 points in every model. GLM-5.3 also responds to patience and trust about as strongly as gpt-oss-120b. Temperature leaves the level of inflation unchanged (57% at the default 0.7). Incentive-driven inflation thus appears in every model and decoding setting we evaluated.