acceptodds
Under review as a conference paper at ICLR 2027

AmbiguityBench: Reasoning Modes Bidirectionally Shift Ambiguity Sensitivity in Frontier LLMs

Abstract

We introduce AmbiguityBench, a diagnostic benchmark for evaluating how large language models behave under ambiguity (unknown probabilities) versus risk (known probabilities), and use it to audit reasoning-enabled configurations across providers. Four probe families—Ellsberg variants, matched Allais risk controls, and tool-selection scenarios—are administered under direct, chain-of-thought, and ambiguity-aware prompting to four base/reasoning pairs from Google, DeepSeek, and Anthropic, plus an open-weight check. Across 23,560 trials, unprompted (direct) elicitation yields large, provider-dependent shifts: Claude Sonnet 4.6 thinking reduces the Ambiguity Premium from +0.396 to +0.080 (delta = +0.316, q_FDR = 0.002; +0.344 at matched temperature, p < 0.0001), while DeepSeek-R1 increases it (delta = -0.111, q = 0.002). Both Gemini generations are small or null. These pair differences largely disappear under chain-of-thought: CoT on Claude base already matches thinking-mode AP, and all four CoT pair tests are non-significant after FDR (q > 0.68). Matched Allais controls remain flat (8/8 cells q > 0.29). Ambiguity sensitivity is therefore a real, model-specific behavioral property, but bidirectionality is a fact about unprompted choice, not a general law of reasoning APIs. We release probes, data, and code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evaluation and benchmarking, Large language models, Decision making, Reasoning, Safety and alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KzyZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AmbiguityBench, an empirical work measuring how language models behave under ambiguity and risk. It compares 4 model families (DeepSeek, Gemini 2.5, Gemini 3, Claude Sonnet 4.6) on 4 task families and claims the ambiguity premium (AP) is model dependent and disappears under CoT. No explanation for the described phenomena is given as the paper is simply an empirical account.

Critical strengths

None

Critical weaknesses
  • Lack of explanation/investigation of the phenomena. I believe a scientific paper should go further than demonstrating some empiric phenomena in some LLMs, and make an effort to explain it. The limitation truthfully states "We do not explain why Claude and DeepSeek move in opposite directions when unprompted" but precisely these explanations would be what would make an empirical account interesting and useful contribution.
  • It's difficult to clearly understand the results on the different task families. It seems like only Family A is discussed in depth, and the lack of the other results doesn't make it clear whether this is results that persist across tasks for models, or whether it is an artifact of a particular task. I found the results section poorly structured and difficult to read.
  • The scope of the study is minimal and there are no clear generalizable findings/learnings.
Questions for authors
  • The DeepSeek API says that thinking mode doesn't support the temperature setting (https://api-docs.deepseek.com/guides/thinking_mode/). How did you prompt the model at different temperatures using the API?
  • There are significant parse failures for DS-R1 (Table 3), how did you treat these cases?
  • How exactly are the models prompted, and what does ambiguity-aware prompting look like?
Reviewer_WhuGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Summary

In this work, the authors introduce AmbiguityBench, a benchmark for examining LLM choices under ambiguity. This benchmark includes four probe families: two-urn Ellsberg tasks, three-color Ellsberg tasks, Allais risk controls, and tool-selection scenarios. The authors evaluate four base-reasoning model pairs under direct, chain-of-thought, and ambiguity-aware prompting, along with an additional open-weight model.

The main finding is that differences between base and reasoning models depend on how they are prompted. Under direct prompting, Claude’s thinking mode substantially reduces its preference for the known-probability option, while DeepSeek-R1 shows a stronger preference than V3. These differences become smaller under chain-of-thought prompting. A matched-temperature ablation supports the Claude finding, while the Allais comparisons show no statistically significant differences within the model pairs. The paper suggests evaluating ambiguity-related behavior under the model configurations and prompting conditions used in practice.

Critical strengths

Strengths

Generally, I think this paper addresses an important problem setting: how do the LLM agents cope with the ambiguity via the construction of a comprehensive benchmark across different models. I would like to list the following strengths I find valuable in this work.

  1. Motivating examples. This work starts with the Ellsberg Two-Urn Paradox to illustrate the main motivation. As humans would prefer certainty over ambiguity, it's natural to evaluate LLMs' performance over these settings.

  2. The paper includes useful controls and appropriately limits several claims. The Claude temperature ablation addresses an obvious confound and retains a substantial difference at matched temperature. The authors also explicitly treat tool selection as exploratory, distinguish the open-weight check from a reasoning-pair replication, and acknowledge the complications in the three-color task. These choices improve the credibility of the narrower empirical contribution.

Critical weaknesses

Despite the above strengths, I still find some critical weaknesses in this work.

Weaknesses

  1. Unclear problem formulation. Although Section 2 introduces ambiguity through specific examples of paradoxes, the formulation remains insufficiently clear. The authors define the Ambiguity Premium using mathematical notation, but do not adequately explain how this metric captures ambiguity aversion.

  2. Insufficiently justified experimental settings. Section 4 lists four probe families with different numbers of tasks, but does not explain the reasons for these differences. The authors also directly refer to src/probes/*.py in the main text without clearly introducing the associated codebase. Although the anonymized repository is linked later, this reference is confusing without sufficient context.

  3. Unclear presentation. In Section 4.2, the authors cite a reference for the DeepSeek series but do not provide corresponding references for the other model families. In Section 5, the presentation places considerable emphasis on numerical results, such as confidence intervals and p-values, without always making the general observations sufficiently clear. This makes it difficult for readers to follow the main findings and understand what the benchmark reveals about LLM behavior.

Questions for authors

Questions

I have some questions for the authors. And I hope them could address them.

  1. Do the Claude and DeepSeek effects persist on the 50–50 subset of Family A and separately under each option-label assignment? What neutral-choice baseline is justified for the full mixture of known probabilities?

  2. How were the numbers of base instances in the four probe families chosen, and how might these differences affect the statistical comparisons? Ideally the numbers should be the same across different families as a fair choice.

  3. Under chain-of-thought prompting, the base/reasoning differences are no longer statistically significant. Is there sufficient evidence to conclude that these differences disappear, or could smaller effects remain? I think further experiment should be done to further validate these claims.

Reviewer_eqsdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AmbiguityBench, which uses Ellsberg-style tasks with an Allais-style risk control to test whether reasoning changes LLM ambiguity aversion across several base/reasoning model pairs. Under direct prompting, reasoning shifts ambiguity aversion in opposite directions for different providers, but these differences disappear under chain-of-thought prompting and do not appear on the risk control.

Critical strengths

The chain-of-thought condition sharpens the claim. Showing that the pair differences vanish under chain-of-thought prompting, and that prompting Claude base to reason already reproduces the thinking-mode behavior, turns a loose claim into a precise one about default policies.

Clustered bootstrap inference, multiple-comparison correction, label reversal, a temperature ablation, full reporting of parse failures, and released data make the benchmark easy to reuse on new models.

Critical weaknesses

The Allais control is not well matched. In the Allais items, the certain option has lower expected value than the gamble, so this family measures certainty preference, whereas the balanced urn items compare options of equal expected value. The Allais paradox is also defined by inconsistency across a pair of choices, which a single-choice measure does not capture. A null result on this control therefore does not establish that the urn-task shift is specific to ambiguity. A control in the same urn format, differing only in whether the composition is stated, would be more convincing.

The measure may reflect expected-value computation, and the headline rests on few contrasts. Some urn items use unbalanced known compositions, where the two urns differ in expected value under a symmetric prior. A model that computes expected value more carefully when reasoning would move away from the known urn without any change in ambiguity attitude, and the paper does not report results by composition. In addition, the bidirectional claim rests on only two significant pairs, which involve different interventions: a thinking toggle on the same model for Claude, and a separately post-trained model for R1.

Questions for authors
  1. Could you report the urn-task results separately for balanced and unbalanced known compositions? Does the Claude reduction hold on the balanced items alone?

  2. Do any Allais items have equal expected value between the certain and risky options? Why was a same-format urn control not used?

  3. The per-model confidence intervals appear to treat rephrasings as independent, while the pair tests cluster by base instance. Could you report clustered intervals for each model as well?

Comments on the paper

2 comments
Area_Chair_NZeEOpenReview ↗

Please engage with the authors responses

Dear Reviewers,

This is a polite reminder to engage with the authors responses to your comments.

Thank you again for your time and effort in reviewing.

Kind regards,

Your ICLR Area Chair

Area_Chair_NZeEOpenReview ↗

Last 24 hours of the reviewer discussion period

Dear Authors and Reviewers

Just a reminder we are soon to be entering the last 24 hours of the reviewer discussion period.

Authors: If you have not yet responded to the reviews and intend to please do so ASAP. Please note reviewers will most likely not have time to respond to any late rebuttal comments so please make sure any comments are clear and not in need of further clarification.

Reviewers: If the authors have responded to your reviews and you have not yet responded please do so, even if it is a one line response acknowledging you have read the rebuttal.

Thanks for all your time, and I hope you all learnt something interesting during the process.

Kind regards,

Your ICLR Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.