acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Trust Appropriately? Evaluating Calibrated Reliance on Imperfect Partners

Abstract

Large language models increasingly collaborate with humans and peer AI agents whose contributions may be incomplete, inconsistent, or incorrect. Effective collaboration requires calibrated trust, whereby decisions to rely on, verify, or assist a partner adapt rationally as empirical evidence accumulates. We investigate whether models translate partner evidence into appropriate reliance policies. We introduce a benchmark of 324 scenarios across 23 collaboration categories. Each scenario involves three sequential decisions with explicit transition dynamics, feedback emissions, and cost schedules, permitting exact analytical evaluation of choices against their expected consequences over the remaining interaction. Evaluating seven models alongside 36 controlled-history probes, we dissociate evidence sensitivity from normatively appropriate action. While the strongest frontier model approaches near-optimal decisions (96.9% all-stage accuracy), other models exhibit recurring pathologies: procuring excessive safeguards, purchasing verification tests with zero expected value of information, prioritizing immediate savings over cooperative learning, and misallocating assistance to suboptimal channels. Supplying explicit posterior distributions resolves certain errors while introducing new failure modes. Calibrated trust in autonomous agents critically demands coupling empirical evidence with forward-looking decision utility.

Primary area:
datasets and benchmarks
Keywords:
Trust Calibration, Human-AI Collaboration, Calibrated Reliance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JdmsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces a benchmark testing LLM’s decision making in collaborative settings where collaborators may be unreliable. Seven LLMs were tested, and GPT 5.6 Sol performed near perfect while the rest had varied performance. Several failure modes in agent decision making are identified, including incurring costs to obtain useless information, and myopic behaviors.

Critical strengths

The focus of this work is both interesting and relevant to how LLMs are increasingly used in the real world. The benchmark design admits a well-defined optimal strategy to which LLM behavior can be compared. The results show meaningful differences between LLMs and interesting behavioral failure modes. The explanation of the benchmark tasks is clear and comprehensive.

Critical weaknesses

Ultimately, the benchmark tasks come down to solving well-defined math problems because prior probabilities are provided to the LLM, and a cost function whose expectation is to be minimized is also provided. Do the authors know whether the LLMs solved these math problems when making their decisions? This may be invisible for the closed-weight models with hidden reasoning, though the near perfect performance of GPT 5.6 Sol suggests it likely was solving these math problems. The authors should be able to report whether the open-weight models were making decisions by solving these math problems.

One reason that the paper’s topic is interesting is that, in the real world, humans/agents must make decisions where prior probabilities are unknown. Humans learn heuristics for handling these situations. The fact that prior probabilities are given in this evaluation weakens the real-world relevance of the benchmark. An alternative benchmark design could be to create longer-horizon tasks where agents are not supplied the prior probabilities, but can learn these probabilities through multiple interactions with their collaborator.

Some terms are not clearly defined. For example, “blueprint-macro,” “switch relation,” and “stability relation”.

The results section is very difficult to parse and relies on terminology that isn’t clearly defined. The writing patterns in the results section are similar to what I would expect from LLM-generated writing.

Several of the figure have very small text.

Questions for authors

I am interested in the authors' thoughts about what can be learned about LLM trust-based behavior and decision making from benchmark tasks that reduce to math problems. Perhaps we can learn things only for the LLMs that do not use math to make their decisions? Could the benchmark be redesigned to avoid this issue?

3 comments

Thank you for your thoughtful and constructive review. We respond to each point below.

[W1, Q1] What can we learn when models use mathematical reasoning?

The open-weight models do calculate. DeepSeek and Qwen return their reasoning text, and in the traces we read both update the partner's probability and compare expected costs. After a partner success, for example, both compute the probability of high capability as 0.585/0.69≈0.84780.585/0.69\approx0.8478, compare the remaining action costs, and choose advice_only. We read a fixed, hash-ordered sample and did not count how often this happens across all responses (Appendix C.1).

What we score is the decision about the partner, however the model reaches it. The benchmark asks how a model judges a partner's capability from evidence, and whether it then relies on, verifies, or assists that partner to the right degree. Research on collaboration treats both as part of trust calibration. Stating the probabilities and costs is what lets us say exactly what the right degree is.

The tasks are also more than arithmetic, for two reasons. First, wording matters. When the same decision trees are rewritten without partner roles, first-decision accuracy rises by 8.7 percentage points for Sonnet (95% interval [0.5, 17.8]) and 7.1 for Qwen [0.8, 14.1], although the calculation is identical in both versions (Section 5.5; Appendix D). Second, models that calculate still choose badly. At 48 histories we ask for the judgment and the action in separate queries, three times each, and we also run a matched condition that gives the full correct posterior. DeepSeek and Qwen, the two models whose calculations we can see, choose suboptimally in 41 of 116 and 29 of 80 cases where their reported judgment is accurate. Across six models the rate is 17%–38%; Sol has 2 of 141. Giving the correct posterior helps Gemini and Sonnet, but errors remain. So the benchmark is informative for models that calculate as well. The judgment comes from a separate query and does not show what the model believed when it acted (Section 5.1).

ModelSuboptimal / accurate-judgment pairsOptimal actions: original → full posterior
GPT-6 Luna22/132120 → 125
GPT-5.6 Sol2/141141 → 141
Gemini 3.8 Flash29/137112 → 126
Claude Sonnet 518/4896 → 109
Claude Haiku 4.513/41103 → 102
DeepSeek V4.1 Flash41/11676 → 74
Qwen3.8 27B29/8076 → 65

[W2, Q1] Can models learn to collaborate without numerical reliability priors?

We implemented your proposed extension as 50-round interactions without numerical partner reliability priors. Models can use partners cost-effectively in some environments, but cumulative costs vary with both the environment and the learner.

Models select two-person teams from four partners or take over the work. Costs and outcome rules are stated, but partner success rates and their generating prior are not supplied. Only the chosen team's joint outcome is observed. The environments contain similarly reliable partners (Close), two strong partners among weaker alternatives (Leader), or uniformly less reliable partners (Safe). We compare realized costs with fixed takeover and with learning algorithms that receive the same task information and feedback.

Model / policyCloseLeaderSafe
GPT-5.6 Sol96.0988.38126.65
Gemini 3.8 Flash100.29101.58124.48
Claude Sonnet 594.40107.02136.97
Claude Haiku 4.593.85105.34124.60
GPT-6 Luna91.6686.74138.78
DeepSeek V4.1 Flash94.7593.25122.88
Fixed takeover100.00100.00100.00
Thompson sampling93.24106.41163.16
Structured Thompson sampling93.63109.22118.13

Values are actual 50-round costs, lower is better. Each environment has three partner instances and two outcome seeds, so each model runs six trajectories. The six reported models complete all trajectories; Qwen has no complete trajectory, and no cost is imputed. Stochastic baselines average 20 algorithm seeds within each partner-instance and environment-seed combination. Additional greedy and UCB results appear in Appendix E.

Whether the models or the algorithms are cheaper depends on the environment. In Close, both Thompson sampling variants have lower mean costs than five of the six models. In Leader, all six models are cheaper than structured Thompson sampling and five are cheaper than ordinary Thompson sampling. In Safe, all six are cheaper than ordinary Thompson sampling but costlier than its structured variant, which infers individual abilities from team outcomes. Each model saves costs relative to takeover in at least one Leader instance and exceeds it in another.

Takeover is the best fixed policy in Safe, but learners are not told this in advance. Costs above takeover include exploration, so they do not by themselves show poor learning. The algorithms, which get the same information, give a second point of comparison. These are cost comparisons on nine partner instances; with no prior specified, there is no unique Bayes-optimal policy to compare against. We also added a 12-round task where partner types are hidden but the prior is given, so the exact optimum can still be computed (Section 5.6; Appendix E).

We have also made revisions to improve the clarity of the presentation, figures and terminology. If any points remain unclear, we would be happy to provide further clarification.

Thank you for these improvements. I will increase my score to a 3.

Reviewer_F1d3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a benchmark, which tests whether LLMs minimize expected total cost when using, testing, or helping imperfect partners. This paper has 324 scenarios that specify partner probabilities, feedback, and costs for three sequential decisions. This allows exact calculation of optimal actions. There are seven models tested on these scenarios, paired histories with different evidence, and 24 scenarios supplying updated partner probabilities.

Critical strengths

Some interesting insights are provided in this paper. For example, the paper finds models that react to evidence in the right direction yet choose overly costly safeguards. This suggests that trust evaluations could score responsiveness and decision quality separately.

The benchmark scores reliance on partners against the action that minimizes expected cost across sequential decisions. This gives researchers an explicit standard for deciding whether an LLM's reliance is appropriate.

Critical weaknesses
  1. There may be some claim-evidence misalignment. Some key arguments and insights are not well-supported. For example,
    • The title and conclusions present the results as evidence about how LLMs trust partners. But it seems that each prompt states the partner's probabilities and costs. So the tasks may mainly test expected-cost calculation and the trust conclusions may not hold. It is worth noting that the prompt in Appendix B.1 lists every probability and fee and asks the model to "Minimize total expected fees and failure losses", and there is no control removes the collaboration framing.
    • Sections 5.3 to 6 attribute unnecessary testing and missed opportunities for useful feedback to reasoning deficits. But fixed preferences for testing or immediate savings could produce these patterns. Thus the explanations may not hold. In all seven Table 10 cases, testing is optimal. So always testing would score seven of seven, as Claude Sonnet 5 does while buying unnecessary tests in 29 of 32 cases (in Table 8). Errors in Figures 3b and 3c coincide with using both partners to reduce risk or helping the weaker partner.
  2. The generalizability of the experiment results and insights are questionable. The benchmark spans 324 scenarios and 23 collaboration categories. But all 300 released core prompts use the same partner parameters. So findings may hold for one reliability setting only. It is worth noting that every released core prompt gives the same initial partner probabilities (0.65 and 0.50), success rates, and probabilities of reliability changes. Only the 24 specialized scenarios differ. One partner is always more reliable. And in the released trees, always helping that partner is optimal in most initial decisions to help or test. It is suggested that the authors may consider varying priors and success rates across scenarios, or stating this restriction and testing a second parameter set.
  3. The experiments could be more solid. For example, in this paper, each decision is sampled once with an unspecified temperature, and the intervals estimate only variation across scenarios. So per-model claims may reflect run-to-run variation, including that supplying updated partner probabilities introduces "new failure modes". The claim about updated probabilities is based on 10 worsened decisions across seven models, all on the 24 specialized scenarios (See Table 12), while the number of fully correct scenarios rises for most models. It is suggested that the authors may consider rerunning the diagnostic subsets several times and reporting how often decisions differ between two runs on identical inputs would give a reference rate for Table 12.
Questions for authors
  1. How do the models perform when a subset of the same decision trees is presented with neutral labels? Do the error types of Sections 5.2 to 5.4 persist?
  2. Do the error patterns of Sections 5.2 to 5.4 persist when the partner priors and success rates differ from the single set used in all 300 core scenarios?
  3. When the baseline condition is run twice on identical inputs, how many decisions change? How does that rate compare with the counts in Table 12?
2 comments

Thank you for the detailed and specific suggestions. We ran the controls you proposed and respond to each point below.

[W1a, Q1] Do the error patterns persist under neutral wording?

Yes. Unnecessary testing and myopic choices persist under neutral role wording, even where overall accuracy improves. Assistance errors are more sensitive to presentation. We evaluated 116 source inputs spanning all 74 blueprints in three versions. The original preserves the narrative and action labels; the action-coded version retains the collaboration setting but recodes the labels; the neutral version additionally replaces role words with sources, sensors, or components. The decision trees and initial information states are matched, before model choices create different subsequent histories.

Relative to the original, neutral first-decision accuracy rises by 8.7% for Sonnet (95% interval [0.5, 17.8]) and 7.1 for Qwen [0.8, 14.1]. Haiku and Luna show similar gains with intervals spanning zero; Sol, Gemini, and DeepSeek change by less than 3 points. Sonnet scores 76.3%, 76.7%, and 85.0% across the original, action-coded, and neutral versions, suggesting sensitivity to role wording beyond action recoding. Under neutral wording, myopic choices remain in all seven models (2.2%–13.7%) and unnecessary testing in six (2.4%–18.7%). Sonnet's myopic-choice rate stays at 11.1%, while its wrong-target assistance rate falls from 16.7% to 4.2%. Appendix D gives the valid/planned response counts.

The table reports all seven models. Each cell lists the error rates (%) in the order Original / Action-coded / Neutral.

ModelUnnecessary testingMyopic choiceWrong assistance target
GPT-6 Luna4.9 / 12.2 / 2.414.8 / 11.5 / 13.74.2 / 0.0 / 5.6
GPT-5.6 Sol0.0 / 0.0 / 0.04.4 / 4.1 / 2.24.2 / 0.0 / 4.2
Gemini 3.8 Flash16.3 / 18.7 / 15.44.1 / 6.7 / 4.40.0 / 16.7 / 12.5
Claude Sonnet 516.3 / 18.7 / 18.711.1 / 8.5 / 11.116.7 / 16.7 / 4.2
Claude Haiku 4.510.6 / 8.1 / 4.915.2 / 18.5 / 12.212.5 / 8.3 / 0.0
DeepSeek V4.1 Flash7.3 / 0.0 / 2.66.7 / 3.0 / 4.80.0 / 0.0 / 4.5
Qwen3.8 27B9.9 / 2.6 / 8.17.3 / 10.3 / 8.30.0 / 0.0 / 0.0

[W2, Q2] Do the error patterns persist when partner reliability changes?

Yes. Unnecessary testing and myopic choices recur beyond the original reliability setting, while assistance errors depend more on the configuration. We evaluated the same 116 sources under three reserved, seed-defined configurations varying both high-type priors and conditional success probabilities. Initial marginal success probabilities for P/Q are approximately 0.487/0.672, 0.435/0.602, and 0.706/0.540. The first two reverse the original reliability ordering; configurations were not filtered for model performance or action balance.

Each cell below lists error rates (%) for Configuration 1 / Configuration 2 / Configuration 3, using the same averaging and error definitions as the wording comparison.

ModelUnnecessary testingMyopic choiceWrong assistance target
GPT-6 Luna2.3 / 2.3 / 5.47.0 / 4.8 / 6.94.2 / 4.2 / 4.2
GPT-5.6 Sol0.0 / 0.0 / 0.00.0 / 2.0 / 1.60.0 / 0.0 / 0.0
Gemini 3.8 Flash4.5 / 4.5 / 8.16.0 / 3.1 / 4.94.2 / 4.2 / 20.8
Claude Sonnet 52.3 / 2.3 / 8.124.3 / 19.7 / 9.80.0 / 0.0 / 8.3
Claude Haiku 4.54.5 / 2.3 / 5.425.7 / 13.6 / 24.50.0 / 0.0 / 8.3
DeepSeek V4.1 Flash0.0 / 0.0 / 0.06.7 / 0.0 / 5.20.0 / 0.0 / 0.0
Qwen3.8 27B0.0 / 2.4 / 1.86.8 / 9.8 / 14.50.0 / 0.0 / 0.0

As the table shows, Sonnet makes myopic choices in all three configurations but chooses the wrong assistance target only in the third. Gemini tests unnecessarily in all three (4.5%, 4.5%, and 8.1%). Optimal actions and eligible error opportunities are recomputed for each configuration, so changes in error rates also reflect changes in which tasks present an opportunity for that error (Section 5.5, Appendix D).

[W1b, W2] Can fixed preferences explain the reported behavior?

Simple fixed rules cannot explain all observed choices, although adaptation remains incomplete. The seven original cases all favor auditing, so always auditing passes them. Our 23 low/high-fee pairs instead require auditing at low fees and skipping at high fees. Always auditing passes 0/23 complete pairs, versus 13/23 for Sonnet, 18/23 for Gemini, and 4/23 for Haiku. Haiku audits in 8/23 low-fee and 7/23 high-fee cases, showing limited adjustment.

Across 80 assistance inputs from 20 blueprints, we vary both reliability ordering and the optimal recipient. Always helping P, Q, the weaker partner, or the stronger partner each scores 40/80; choosing the largest immediate reliability gain scores 51/80. Sonnet and Haiku each score 73/80 (Appendix C.3). These controls do not identify the cause of individual errors. In the examples using both partners, the choices sacrifice future information value, but we cannot separate risk-reduction preferences from failure to consider future feedback. We therefore define the error categories by action values without attributing a unique internal cause (Sections 5.2–5.4).

[W3, Q3] How does posterior support compare with ordinary response variation?

Identical inputs produce different actions in 3.0%–10.7% of baseline comparisons. Posterior effects vary across models, with no clear overall deterioration.

We repeated both conditions three times at the 89 original histories from 24 specialized scenarios. An action change can preserve correctness, so “excess worsening” subtracts baseline correct-to-incorrect variation from support-associated worsening. Accuracy change and excess worsening are in percentage points.

ModelNatural action change (%)Accuracy change [95% CI]Excess worsening [95% CI]
GPT-5.6 Sol3.00+1.50 [−0.36, +3.88]−1.12 [−2.84, +0.37]
Gemini 3.8 Flash3.00−2.62 [−5.99, 0.00]+2.00 [0.00, +5.01]
Claude Sonnet 58.99−0.76 [−3.90, +2.16]−0.63 [−2.55, +1.40]
Claude Haiku 4.58.24−5.62 [−9.68, −1.22]+4.00 [+0.11, +8.21]
GPT-6 Luna6.74+1.50 [−2.48, +5.88]+0.12 [−3.16, +3.66]
DeepSeek V4.1 Flash5.24+1.70 [−2.22, +5.94]−0.13 [−2.72, +2.56]
Qwen3.8 27B10.73+2.62 [−0.41, +5.49]−2.04 [−4.61, +0.60]

The table uses valid outputs. Intervals resample source scenarios, retaining related histories and generations together. Haiku's intervals exclude zero for both declining accuracy and excess worsening. The seven-model mean accuracy change is −0.24 percentage points [−1.32, +0.81], and excess worsening is +0.31 [−0.62, +1.28]. Both summary intervals also include zero when invalid answers count as incorrect (Appendix C.2).

Across two collections of the 324-scenario core, decision accuracy differs by 0.2–5% per model. Settings differ, so this descriptive comparison complements the matched repeats (Appendix B). Core results use one generation per input and intervals across blueprints. Section 4 and Appendix B report the settings; current requests specify medium reasoning and a 32,768-token output limit, with temperature left at provider defaults.

We have also made revisions to improve the clarity of the presentation. If any points remain unclear, we would be happy to provide further clarification.

Reviewer_KKL9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at actions taken by agents in a multi-agent system setting and builds a benchmark of 324 scenarios that they test. They ask is current LLMs can act as rational agents. The core of the paper is looking at LLMs responses to reliability vs optimal allocation of decisions to rely on, verify, or assist. They show that while GPT-6 sol is near optimal in allocations several of the weaker models are lacking.

Critical strengths

I appreciate the creation of this benchmark in an interesting and thoughtful way. The idea of having an optimal sequence of actions to bechmark and quantify LLM agent behavior is important and timely. I especially like the distinction between epistemic calibration and decision calibration which I think can be even further highlighted.

While the paper isn't the first to look at the AI collaboration from an optimality point of view nor the first to show that LLMs fail to appropriately calibrate trust, they do provide an extremely elegant framework for studying the behavior and in section 5 do a nice job at teasing this apart.

Critical weaknesses

I think that this is an interesting idea and benchmark, but my only concern is that it is "solved." I'd like the authors to think about how to expand the benchmark to see if there's cases where GPT-6 Sol is not in fact optimal in its behavior.

Suggested references to include and contextualize your work Lui et al CostBench and Yang et al, "LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off". Especially Yang et al is quite similar and missing.

I found the paper dense and hard to get through despite being quite interested in the work. There's too much of a focus on detail rather than the main story of the paper. I think that the

It is important to explicitly state (possibly even in the abstract) that the authors adopt expected-cost-optimal adaptation as the normative standard. In human collaborations this is often not the case. This makes me question construct validity -- is this actually trust that you are measuring. The authors motivate their paper by Lee & See's definition of trust, but they are pretty clear that trust is the attitude towards the agents capability to support ones goals, so I would argue no ... it is decision calibration. More generally, I still can't tell if the benchmark measures trust rather than general planning.

Questions for authors

Supplying explicit posterior distributions created new errors, can you help explain a bit more why.

Suggested references to include and contextualize your work Lui et al CostBench and Yang et al, "LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off" Especially Yang et al is quite similar and missing.

I'd really like to be convinced that the benchmark is really measuring trust. One option might be to frame the counterpart as a stochastic process to see if behavior changes, but there's probably other designs that are better.

Is your regret g actually measuring cumulative cost? Is it possible to see how different these models are more long run than let's say a Thompson Sampling approach?

3 comments

Thank you for the thoughtful review and for recognizing the value of this work. We answer each point below and make the corresponding changes in the paper:

[W1, Q4b] Is the benchmark ‘solved’? Where does Sol fall short?

We agree that the core leaves little headroom for Sol, which acts optimally throughout 311 of 324 scenarios. We therefore added two longer learning settings to look for its limits. Sol remains near-optimal when partner types are hidden but the prior is disclosed. Without numerical priors, its mean cost exceeds that of structured Thompson sampling in all three Safe instances.

In the 12-round setting, unknown partner types are drawn from a public prior, allowing an exact Bayes-optimal reference. Sol's cumulative action gaps (0 is optimal) are 0.013 with team feedback and 0.169 with individual feedback, compared with 3.23 and 2.59 for Thompson sampling and 4.64 and 6.12 for posterior-mean greedy choice.

In the 50-round setting, agents learn without numerical reliability priors by selecting two-person teams from four partners or taking over the work. The environments contain similarly reliable partners (Close), two strong partners among weaker alternatives (Leader), or uniformly less reliable partners (Safe).

Model / policyCloseLeaderSafe
GPT-6 Luna91.6686.74138.78
GPT-5.6 Sol96.0988.38126.65
Gemini 3.8 Flash100.29101.58124.48
Claude Sonnet 594.40107.02136.97
Claude Haiku 4.593.85105.34124.60
DeepSeek V4.1 Flash94.7593.25122.88
Fixed takeover100.00100.00100.00
Thompson sampling93.24106.41163.16
Structured Thompson sampling93.63109.22118.13

Entries are mean total task costs (lower is better) over three partner instances and two outcome seeds. Six models complete all trajectories. Stochastic baselines average 20 algorithm seeds and have the same task information and feedback access.

Relative to ordinary Thompson sampling, one of six models has a lower mean cost in Close, five in Leader, and all six in Safe. Relative to structured Thompson sampling, which uses the team structure to infer individual abilities, all six have lower mean costs in Leader and higher mean costs in Safe. In each of the three Safe instances, Sol's cost is 26–27% above fixed takeover, versus 17–19% for structured Thompson sampling. Exploration costs are not errors in themselves, and exceeding takeover does not alone show that a learning policy is unreasonable; none of these baselines is an optimal policy. Without a specified prior, these are empirical cost comparisons on the evaluated instances (Section 5.6; Appendix E).

[W4, Q3] What the benchmark measures

The benchmark measures two kinds of observable, trust-related behavior: the judgments models report about imperfect partners, and the decisions they make to rely on, verify, or assist them. Inspired by trust research, we distinguish reported judgments about partners from the calibration of reliance behavior. We do not measure trust as a psychological attitude. Our normative criterion is minimum expected total cost under the stated task objective (Section 3.1), rather than a universal standard for human collaboration. The two measures can come apart. At 48 histories we ask for the judgment and the action in separate queries, three times each. We also run a matched condition that gives the model the full correct posterior before it acts:

ModelSuboptimal / accurate-judgment pairsOptimal actions: original → full posterior
GPT-6 Luna22/132120 → 125
GPT-5.6 Sol2/141141 → 141
Gemini 3.8 Flash29/137112 → 126
Claude Sonnet 518/4896 → 109
Claude Haiku 4.513/41103 → 102
DeepSeek V4.1 Flash41/11676 → 74
Qwen3.8 27B29/8076 → 65

The first column counts suboptimal actions among cases where the reported judgment is accurate (total-variation error ≤ 0.02). Action counts are out of 144 planned responses per condition, and invalid responses receive no credit. Intervals and full counts appear in Appendix C.1.

For six models, 17%–38% of actions are suboptimal even when the reported judgment is accurate; Sol has 2 of 141. Giving the correct posterior helps Gemini and Sonnet but does not remove their errors. The judgments come from a separate query, so they do not show what the model believed when it chose the action (Section 5.1).

Assisted and unaided success can justify different capability judgments, while a fee change can alter the appropriate action without changing reliability. Across 36 matched history pairs, models match the expected pattern of switching or maintaining their action in 94.4%–100% of evaluations, but choose optimally in both histories in 82.4%–100% (Appendix C.1). Matching the expected response pattern therefore does not guarantee appropriate choices.

Following your suggestion, we compared three versions of the same decision trees: the original, one that only recodes the action labels, and a role-neutral one in which partners become sources, sensors, or components. From original to neutral, first-decision accuracy rises by 8.7 percentage points for Sonnet (95% interval [0.5, 17.8]) and 7.1 for Qwen [0.8, 14.1]. Haiku and Luna show similar gains with intervals that include zero; Sol, Gemini, and DeepSeek change by less than 3 points. Sonnet scores 76.3%, 76.7%, and 85.0% across the three versions, suggesting sensitivity to role wording beyond action recoding. Some errors persist under neutral role wording: Sonnet's myopic-choice rate is 11.1% in both the original and neutral versions (Appendix D).

[W2, Q2] Compare with current research.

We have discussed and added references of CostBench and Yang et al. CostBench studies cost-optimal tool planning under failures and changing costs, while Yang et al. investigate whether predicted failure risks justify the additional costs of different collaboration protocols. In contrast, we study how evidence about individual partners should guide sequential decisions to rely, verify, or assist.

These decisions have downstream consequences. Verification provides information about partner reliability, while assistance can improve future performance. For tasks with known generative probabilities, we compute exact reference policies that account for these effects using only information available at each decision point. Our framework enables controlled diagnosis by varying evidence provenance and relevance across matched histories and independently eliciting reliability judgments and action choices. This allows us to identify when agents misjudge partners, fail to act on accurate assessments, or make suboptimal verification and assistance decisions. Thus, our contribution extends cost-aware collaboration to the evaluation of evidence-grounded, calibrated reliance.

[Q4a] Does the action gap measure cumulative cost?

Yes. The expected sum of the unnormalized action gaps equals the excess expected total cost relative to the optimal policy.

At each decision, gt=Qt⋆(ht,at)−Vt⋆(ht)g_t=Q_t^{\star}(h_t,a_t)-V_t^{\star}(h_t) measures the additional expected cost of the chosen action under optimal continuation. For trajectories generated by policy π\pi,

=V^{\pi}(h_1)-V^{\star}(h_1).$$ This equality holds in expectation, rather than for every realized trajectory. Normalized regret rescales the gaps by the available action-value range and therefore does not retain this cost identity. Section 3.4 states the relationship, and Appendix B.5 provides the derivation. The 50-round comparison above reports realized total task costs directly. We have also made revisions to improve the clarity of the presentation. If any points remain unclear, we would be happy to provide further clarification.

Comments on the paper

1 comment

We thank all reviewers for thoughtful and constructive reviews. We first summarize our contributions and then explain how the revision addresses the main concerns.

Contributions

  • C1. A benchmark for testing whether LLMs can be relied on in collaboration with imperfect partners. The benchmark gives researchers an way to evaluate a capability that matters before LLMs are used in real collaboration: deciding when to rely on, verify, or assist a partner. Across 324 collaboration scenarios, reference policies show whether each decision minimizes expected total cost and what a deviation costs, including its effect on what the agent learns and how the partner performs later (KKL9, F1d3, Jdms).
  • C2. Evaluations that separate assessing a partner from acting on that assessment. We use controlled histories and separately elicited judgments and actions to show whether a costly choice comes with an inaccurate reported judgment or occurs despite an accurate one, a distinction that an overall score cannot make (KKL9, F1d3).
  • C3. Specific evidence of where current models fall short. In our evaluation, most models repeatedly buy unnecessary tests and give up useful feedback for immediate savings, while excessive safeguards and misdirected assistance are concentrated in fewer models. These errors give concrete targets for improving LLMs' collaboration ability (F1d3, Jdms).
  • C4. Longer tasks that test whether LLMs can learn how far to rely on a partner. In practice, agents are rarely told how reliable their partners are. We therefore added tasks in which they must learn this from outcomes over 12 or 50 rounds. The strongest model stays near-optimal when the prior over partner types is given, but can cost more than a structured learning algorithm when no numerical prior is given. A high score on fully specified tasks therefore does not show that an agent will collaborate cost-effectively when it must learn how reliable its partners are (added in response to KKL9 and Jdms).

Main concerns and revisions

What does the benchmark measure? [KKL9 W4/Q3; F1d3 W1a; Jdms W1/Q1] We measure the observable bahavior of trust calibration: how models judge imperfect partners, and how well they decide when to rely on, verify, or assist them. This is the observable side of trust calibration, not trust as an attitude. We focus on these behaviors because they can be scored directly and they determine the cost and outcome of collaboration. The tasks state probabilities and costs so that appropriate reliance has a precise definition and every decision has an exact reference. Calculation is part of the skill being tested, because an agent must work through the numbers to turn evidence about a partner into a good decision (Sections 3.1 and 3.4).

Do the findings generalize? [F1d3 W2/Q1/Q2] The most common errors are not tied to one task description or one set of partner parameters. We re-evaluated tasks under different wording and new partner reliability settings, including settings that change which partner is more reliable. Unnecessary testing and myopic choices recur in most models across these conditions, while assistance errors are more sensitive to them (Section 5.5; Appendix D).

Can simple strategies explain the results? [F1d3 W1b] Not entirely. Always auditing or always helping the same partner can look effective when the right answer never changes, so we added tasks where it does: auditing pays off at a low fee but not at a high fee, and the partner worth helping varies across tasks. Always auditing gets none of the 23 fee pairs fully right, while models get 4 to 20. Simple assistance rules get 40 to 51 of 80 tasks right, while models get 55 to 78. Models therefore do more than follow a fixed habit, although some adapt only partly (Sections 5.2 and 5.4; Appendix C.3).

Is the benchmark solved? [KKL9 W1/Q4b; Jdms W2] Not across models or settings. The strongest model is close to the ceiling when partner priors are given, but the other models still have substantial room to improve on the core tasks. We also added longer tasks in which partner reliability must be learned from experience. Without numerical priors, no model beats a structured learning algorithm in every evaluated environment, and costs depend on how reliable the partners are. The benchmark therefore remains useful for diagnosing current models and for testing how well they learn through collaboration (Section 5.6; Appendix E).

We have also revised the writing, figures, and terminology for clarity. If any points remain unclear, we would be happy to clarify them and further improve the presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.