- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work introduces a benchmark testing LLM’s decision making in collaborative settings where collaborators may be unreliable. Seven LLMs were tested, and GPT 5.6 Sol performed near perfect while the rest had varied performance. Several failure modes in agent decision making are identified, including incurring costs to obtain useless information, and myopic behaviors.
Critical strengths
The focus of this work is both interesting and relevant to how LLMs are increasingly used in the real world. The benchmark design admits a well-defined optimal strategy to which LLM behavior can be compared. The results show meaningful differences between LLMs and interesting behavioral failure modes. The explanation of the benchmark tasks is clear and comprehensive.
Critical weaknesses
Ultimately, the benchmark tasks come down to solving well-defined math problems because prior probabilities are provided to the LLM, and a cost function whose expectation is to be minimized is also provided. Do the authors know whether the LLMs solved these math problems when making their decisions? This may be invisible for the closed-weight models with hidden reasoning, though the near perfect performance of GPT 5.6 Sol suggests it likely was solving these math problems. The authors should be able to report whether the open-weight models were making decisions by solving these math problems.
One reason that the paper’s topic is interesting is that, in the real world, humans/agents must make decisions where prior probabilities are unknown. Humans learn heuristics for handling these situations. The fact that prior probabilities are given in this evaluation weakens the real-world relevance of the benchmark. An alternative benchmark design could be to create longer-horizon tasks where agents are not supplied the prior probabilities, but can learn these probabilities through multiple interactions with their collaborator.
Some terms are not clearly defined. For example, “blueprint-macro,” “switch relation,” and “stability relation”.
The results section is very difficult to parse and relies on terminology that isn’t clearly defined. The writing patterns in the results section are similar to what I would expect from LLM-generated writing.
Several of the figure have very small text.
Questions for authors
I am interested in the authors' thoughts about what can be learned about LLM trust-based behavior and decision making from benchmark tasks that reduce to math problems. Perhaps we can learn things only for the LLMs that do not use math to make their decisions? Could the benchmark be redesigned to avoid this issue?
3 comments
[W2, Q1] Can models learn to collaborate without numerical reliability priors?
We implemented your proposed extension as 50-round interactions without numerical partner reliability priors. Models can use partners cost-effectively in some environments, but cumulative costs vary with both the environment and the learner.
Models select two-person teams from four partners or take over the work. Costs and outcome rules are stated, but partner success rates and their generating prior are not supplied. Only the chosen team's joint outcome is observed. The environments contain similarly reliable partners (Close), two strong partners among weaker alternatives (Leader), or uniformly less reliable partners (Safe). We compare realized costs with fixed takeover and with learning algorithms that receive the same task information and feedback.
| Model / policy | Close | Leader | Safe |
|---|---|---|---|
| GPT-5.6 Sol | 96.09 | 88.38 | 126.65 |
| Gemini 3.8 Flash | 100.29 | 101.58 | 124.48 |
| Claude Sonnet 5 | 94.40 | 107.02 | 136.97 |
| Claude Haiku 4.5 | 93.85 | 105.34 | 124.60 |
| GPT-6 Luna | 91.66 | 86.74 | 138.78 |
| DeepSeek V4.1 Flash | 94.75 | 93.25 | 122.88 |
| Fixed takeover | 100.00 | 100.00 | 100.00 |
| Thompson sampling | 93.24 | 106.41 | 163.16 |
| Structured Thompson sampling | 93.63 | 109.22 | 118.13 |
Values are actual 50-round costs, lower is better. Each environment has three partner instances and two outcome seeds, so each model runs six trajectories. The six reported models complete all trajectories; Qwen has no complete trajectory, and no cost is imputed. Stochastic baselines average 20 algorithm seeds within each partner-instance and environment-seed combination. Additional greedy and UCB results appear in Appendix E.
Whether the models or the algorithms are cheaper depends on the environment. In Close, both Thompson sampling variants have lower mean costs than five of the six models. In Leader, all six models are cheaper than structured Thompson sampling and five are cheaper than ordinary Thompson sampling. In Safe, all six are cheaper than ordinary Thompson sampling but costlier than its structured variant, which infers individual abilities from team outcomes. Each model saves costs relative to takeover in at least one Leader instance and exceeds it in another.
Takeover is the best fixed policy in Safe, but learners are not told this in advance. Costs above takeover include exploration, so they do not by themselves show poor learning. The algorithms, which get the same information, give a second point of comparison. These are cost comparisons on nine partner instances; with no prior specified, there is no unique Bayes-optimal policy to compare against. We also added a 12-round task where partner types are hidden but the prior is given, so the exact optimum can still be computed (Section 5.6; Appendix E).
We have also made revisions to improve the clarity of the presentation, figures and terminology. If any points remain unclear, we would be happy to provide further clarification.
Thank you for these improvements. I will increase my score to a 3.
Thank you for your thoughtful and constructive review. We respond to each point below.
[W1, Q1] What can we learn when models use mathematical reasoning?
The open-weight models do calculate. DeepSeek and Qwen return their reasoning text, and in the traces we read both update the partner's probability and compare expected costs. After a partner success, for example, both compute the probability of high capability as 0.585/0.69≈0.8478, compare the remaining action costs, and choose
advice_only. We read a fixed, hash-ordered sample and did not count how often this happens across all responses (Appendix C.1).What we score is the decision about the partner, however the model reaches it. The benchmark asks how a model judges a partner's capability from evidence, and whether it then relies on, verifies, or assists that partner to the right degree. Research on collaboration treats both as part of trust calibration. Stating the probabilities and costs is what lets us say exactly what the right degree is.
The tasks are also more than arithmetic, for two reasons. First, wording matters. When the same decision trees are rewritten without partner roles, first-decision accuracy rises by 8.7 percentage points for Sonnet (95% interval [0.5, 17.8]) and 7.1 for Qwen [0.8, 14.1], although the calculation is identical in both versions (Section 5.5; Appendix D). Second, models that calculate still choose badly. At 48 histories we ask for the judgment and the action in separate queries, three times each, and we also run a matched condition that gives the full correct posterior. DeepSeek and Qwen, the two models whose calculations we can see, choose suboptimally in 41 of 116 and 29 of 80 cases where their reported judgment is accurate. Across six models the rate is 17%–38%; Sol has 2 of 141. Giving the correct posterior helps Gemini and Sonnet, but errors remain. So the benchmark is informative for models that calculate as well. The judgment comes from a separate query and does not show what the model believed when it acted (Section 5.1).