acceptodds
Under review as a conference paper at ICLR 2027

Ask, Relax, or Act? Evaluating Actionable Indeterminacy in LLM Preference Reasoning

Abstract

An LLM agent can recognize uncertainty yet still choose the wrong next step: asking when action is already justified, or seeking clarification when the constraints must change. We formalize actionable indeterminacy: act when an accepted action is shared across all admissible preferences or objectives, clarify when each possibility is feasible but no action is shared, and propose a minimum-cost permitted constraint repair when the request is infeasible. We construct a solver-grounded benchmark spanning object allocation, meeting scheduling, apartment choice, and stable matching. Matched pairs retain the same source while changing whether intervention is necessary, and evaluation separates decision correctness, matched-pair reliability, and fully correct responses. Our findings reveal a recurring difficulty in recognizing when intervention is unnecessary: models can identify situations requiring clarification or repair yet still intervene when a justified action already exists. Correct decision labels also fail to guarantee usable actions, questions, or repairs. Crucially, response requirements shape not only how decisions are expressed but also which decisions are made. Making the required content explicit substantially improves fully correct responses and can change intervention decisions, even when outputs are already parseable. These findings highlight that reliable agency requires more than recognizing uncertainty: it requires intervening only when necessary and translating the chosen next step into a verifiable response.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Preference Reasoning, Actionable Indeterminacy

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HrgoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a verifiable benchmark to measure whether AI models are good at handling uncertainty in tasks. However, it is extremely unclear and has questionable soundness since the benchmark construction and validity is not clearly explained. Because of this, I give a clear rejection.

Critical strengths
  • Interesting Idea. It is interesting to attempt to evaluate models' abilities in resolving uncertain scenarios in a deterministic way. I believe this is much needed given the literature on ambiguity.
Critical weaknesses
  • Benchmark construction is unsound. Since the paper proposes a benchmark for understanding how models act under uncertainty, it would be incredibly helpful if the authors spent more time convincing me that their benchmark construction is correct. Instead, I am given, if interpreted generously, the benchmark construction in less than 1 page which essentially just says "we constructed the benchmark." I cannot tell how accurate your benchmark is (e.g., whether it exhibits task, construct, outcome validity if you do not discuss its quality and quality control). Rather, because the entire paper is unclear and seems AI-generated (e.g., the entire paper refers to the benchmark as "V3", there are random requirements never introduced that are referenced like "the first three also cover H2 and H5" and I have no idea what H2 and H5 are since they're given a very vague reference in Section 2.2, a lot of jargon is introduced, lots of colons, very information dense but also glib writing, etc. What does "descriptive independence reference of 85.28%" mean? When was the reference introduced?). Based on this, I am left to believe that the benchmark and paper was one-shotted, which makes me doubt its soundness. It would be extremely helpful to spend more time discussing how the benchmark was created, how you ensured data quality, and what an example concretely looks like in the main text. As a result, all of your experiments are hard to interpret. In fact, even the appendix is not clear on how the benchmark was constructed.
  • Lack of Uncertainty Quantification: The paper's main result figures (Figures 2 and 3) rely on differentiating between very small percentage point differences. It is hard to know whether the differences exhibited between models are statistically significant and whether the difference between preference, objective, and constraint problems are significant. Since the paper is a benchmark paper, it is incredibly important to quantify uncertainty.
Questions for authors
  • Can you please elaborate on how your benchmark was constructed? Please be very clear about each step and justify validity and quality (see critical weaknesses).
  • Can you include at least 1 qualitative example of a benchmark task, the models' actions in that task, and how it is scored?
Reviewer_edHvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

A benchmark that measures what the agent will do when facing with incomplete context. Design detailed patterns on the proposed benchmark, including 4320 tasks。 Findings are:

  1. Model can be over-intervene.
  2. A good context helps with the results.
Critical strengths
  1. Question is interesting.
  2. Evaluation and experiments are solid.
Critical weaknesses
  1. The value of the benchmark should be reconsidered. The experiments just modify the input context and the acc of GPT Astra raise from 50.79% to 94.40%. This experiments shows that the frontier model can have a pretty good results on the benchmark which reduces the value of the benchmark.
Questions for authors

The change of the context can influence a lot on the result. Can you do ablations on the components of the context, such as the structure/content/guidance?

1 comment

Thanks for the clarification. I will maintain my original score.

Reviewer_RKizOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether LLMs can recognize situations where multiple permissible actions exist under a set of constraints and objectives, which they refer to as "actionable indeterminacy". The authors consider domains such as object allocation, meeting scheduling, and stable matching, where ambiguity in preferences leads to ambiguity in what the next optimal action is (supposedly while running a decision procedure). The paper claims that LLMs can accurately detect uncertainty (situations where "clarification" or "repair" is required) but still "intervene" where valid actions are present (what "intervention" really means is not clear to me after multiple attempts to understand). Additional results explore how response format changes (e.g. requiring a JSON format) improves performance.

Critical strengths

I find no clear strengths in this work.

Critical weaknesses

Writing: The way the paper is currently written makes it impossible to understand. The authors have clearly used AI to heavily edit or even entirely write the paper. While that is not a problem in itself, it seems like the authors have not considered how the paper reads. It almost seems like the authors have not even read the paper themselves before submitting, otherwise some very obvious issues at least might have been addressed. Terms/symbols are used before or without being defined, the language is extremely convoluted, and important details are left to the reader's assumption. Multiple examples of each of these issues are provided in the "Questions" section below (and that is only the a subset of the issues identified until page 3).

Not only does this make it extremely frustrating to read this paper, it is also not possible to properly evaluate the merit of this paper given the lack of clarity. If this is something other authors do not experience (without passing the paper into an AI model to parse it), I am willing to update my judgement.

Novelty: This paper seems to be too similar to "Preference Reasoning Under Indeterminacy in Large Language Models" [Hosseini et. al., 2026], with the problem (indeterminacy arising from underspecified preferences), elements of the teaser figure, the domains considered (object allocation, stable matching), models used (latest versions for GPT, Claude Sonnet, Gemini Pro, and OSS), and the colors/layouts of various graphs being extremely similar. "Actionable indeterminacy" seems like a repackaging of the "epistemic indeterminacy" studied by Hosseini et. al., which even if not the same definition measures the same (at best, analogous) capabilities in LLMs.

Questions for authors
  1. Could you please clarify the following: a. What is the “pivotal question” in line 52 b. What is the “action/intervention contrast” in line 53? 
 c. What is the “six-task group” in line 73? d. What do you mean by “preference and objective uncertainty” in line 74? These have not been defined at this point. The definitions provided in the figure are not possible to understand on their own. e. What are “matched pairs” (used everywhere)? The sentence in the abstract does not make it clear what this means. 
 f. 
What is “V3 evaluation” (line 90)? 
 g. What are “matched local features” (line 102)?

  2. From the descriptions of the problem domains given in 148-152, it is impossible to understand what actions and objectives are. The section in the appendix (although the reader shouldn’t have to go through this to understand something so central to the design) is equally dense and depends on a lot of things not defined until that point in the main body. What order of reading does one need to follow to understand the details (and examples) of the terms and symbols defined in Section 2.1?

  3. What faculty of LLMs does actionable indeterminacy test that has not been tested before (either as "epistemic indeterminacy" or something else)? I am not asking you to explain how the definition of the former is different from previously studied constructs, but I am asking you what aspect of LLMs would make influence their performance on your benchmark that is different from what has already been documented (e.g. lack of abstention, failure to flag indeterminacy in preferences)? In other words, what new property of LLMs is this work helping me learn?

1 comment
Reviewer_RKizOpenReview ↗

The responses do not address my concerns

I have gone through the authors's responses and this does not address my concerns. It took the authors two comments to clarify 5-6 terms/expressions that are not defined in the first 3 pages of the submission itself. This just demonstrates that the paper is not possible for a human reader to understand in this form. I am in agreement with reviwer Hrgo, and unless the authors completely re-write the paper with minimal use of AI, I shall maintain my score.

Additionally, it seems like the authors have not attempted to answer my question/concern about novelty (Q3), and have instead pasted the responses they prepared for a different reviewer. I would strongly recommend the authors to be more careful about what they are writing or submitting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.