- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces AmbiguityBench, an empirical work measuring how language models behave under ambiguity and risk. It compares 4 model families (DeepSeek, Gemini 2.5, Gemini 3, Claude Sonnet 4.6) on 4 task families and claims the ambiguity premium (AP) is model dependent and disappears under CoT. No explanation for the described phenomena is given as the paper is simply an empirical account.
Critical strengths
None
Critical weaknesses
- Lack of explanation/investigation of the phenomena. I believe a scientific paper should go further than demonstrating some empiric phenomena in some LLMs, and make an effort to explain it. The limitation truthfully states "We do not explain why Claude and DeepSeek move in opposite directions when unprompted" but precisely these explanations would be what would make an empirical account interesting and useful contribution.
- It's difficult to clearly understand the results on the different task families. It seems like only Family A is discussed in depth, and the lack of the other results doesn't make it clear whether this is results that persist across tasks for models, or whether it is an artifact of a particular task. I found the results section poorly structured and difficult to read.
- The scope of the study is minimal and there are no clear generalizable findings/learnings.
Questions for authors
- The DeepSeek API says that thinking mode doesn't support the temperature setting (https://api-docs.deepseek.com/guides/thinking_mode/). How did you prompt the model at different temperatures using the API?
- There are significant parse failures for DS-R1 (Table 3), how did you treat these cases?
- How exactly are the models prompted, and what does ambiguity-aware prompting look like?
Please engage with the authors responses
Dear Reviewers,
This is a polite reminder to engage with the authors responses to your comments.
Thank you again for your time and effort in reviewing.
Kind regards,
Your ICLR Area Chair