acceptodds
Under review as a conference paper at ICLR 2027

Preference Reasoning under Indeterminacy in Large Language Models

Abstract

As large language models evolve into decision-making agents, the ability to reason over preferences becomes fundamental to alignment, coordination, and collective intelligence. Yet, unlike standard benchmarks, real-world preference reasoning is inherently indeterminate: information may be incomplete, and valid solutions may not exist. We argue that indeterminacy, rather than correctness alone, is a central challenge for AI reasoning. We formalize this challenge along two axes, (i) epistemic indeterminacy, arising from incomplete, partial, or expressive preferences, and (ii) structural indeterminacy, arising from the non-existence of solutions under standard social choice concepts. Across a hierarchy of tasks, we show that language models systematically fail to distinguish between determined and undetermined instances, exhibiting miscalibrated reasoning even in verification settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Reasoning under Uncertainty, Indeterminacy, Algorithmic Reasoning, Ranked Preferences

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HsEZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether LLMs recognize when a preference question has no determined answer. This happens either because incomplete preferences do not give a clear answer, or because a solution concept has no solution. The main contribution is a benchmark of determined and undetermined instances over house allocation and stable matching problems, on which four models rarely abstain and often fill gaps with their own systematic assumptions. However, part of the reported failure comes from how the tasks are prompted and scored rather than from the models’ reasoning. In addition, several existing studies have observed similar abstention failures in LLMs, which somewhat limits the novelty and impact of this work.

Critical strengths
  1. The question of whether LLMs notice when preferences are not sufficient to determine an answer is interesting and well motivated. It is important for studying agentic behaviors, LLM-based preference elicitation, and AI-mediated decision making processes.
  2. The paper is generally well-written. Figure 1 gives a very helpful overview of the task taxonomy for general audiences who are not familiar with this type of problem.
Critical weaknesses
  1. The claimed gap seems to mix reasoning with unstated conventions and prompt format. For example, the prompts never explain what an unlisted item means. However, in its instance generation, the paper also claims that it follows what it calls the standard convention that unranked alternatives are unacceptable (Appendix E.2.1), under which an agent who lists a but not b should prefer a to b. Yet in the aggregative task, an agent preferring a to b is instead scored as an error (Figure 1, Table 8). Thus, it is plausible that the models are simply adopting standard economic conventions rather than exhibiting a reasoning failure.
  2. The free-flow prompts also directly demand a number or a name with no abstention option. Prior work has well established that post-trained LLMs are heavily biased toward providing direct answers rather than abstaining when prompted this way (e.g., Kalai et al., 2025). Indeed, as shown in Table 15, simply adding an abstention option enables all models to achieve 100% accuracy, suggesting that the observed failure is largely an artifact of prompt framing.
Questions for authors
  1. How would the results change if the prompts state how unranked items should be read?
  2. How would the findings change if recognition of indeterminacy is measured separately from the final answer, for example by asking whether the answer is uniquely determined?
Reviewer_LWgwOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This is an empirical research paper that studies the preference reasoning capability of large language model under indeterminacy. The authors introduce some interesting problems from social sciences, particularly economics, where models (especially in agentic settings) have to do implicit or explicit inference, comparison and aggregation of of potentially conflicting preferences. They highlight two types of indeterminacy. The first is epistemic indeterminacy which is due to incomplete preference information which does not allow for a proper inference. The second is structural indeterminacy which is due to the interaction between the preference structure and solutions like the case of ties. Table 1 captures all of the types of indeterminacy cases considered. The paper has constructed a preference reasoning benchmark of 3360 prompts. Using this dataset, four frontier models have been tested. Extensive evaluation (presented in the main paper and the supplementary material) show that these models perform poorly on undetermined questions with a variety of reasoning errors. This is for epistemic indeterminacy. For structural indeterminacy prompts as well, the performance is poor, especially for larger problem instances. The authors then try to propose improving the performance by refinement by feedback as well as allowing for reasoning with code execution. both of these do help but in a limited manner for smaller problem instances that is unlikely to scale with problem size.

Critical strengths

(1) The paper introduces a new problem area for the benchmarking and evaluation of LLMs. The problem has been introduced in a compelling manner and extremely well presented even for a non-expert in the area. The problem highlighted is real and it is very interesting intellectually.

(2) The benchmarking evaluation done on four frontier models is systematic and rigorous. The conclusions drawn about the preference reasoning capabilities of LLMs in this setting are therefore quite convincing.

Critical weaknesses

(1) The main weakness of this paper is provided by the authors themselves. In lines 089-092, the authors state that all tasks in the benchmark "are computable in polynomial time using well-established combinatorial algorithms".

If this is true, then why should one use LLMs for this purpose?

It is very well known that LLMs are poor at doing many such tasks. for example, consider arithmetic computations. LLMs perform well on small problem sizes but this performance does not scale at all. However, there are excellent deterministic calculator algorithms are available. Hence for an agentic AI setting, the model harness invokes the external calculator for any consequential calculation to be performed. A similar approach can easily be adopted for this problem.

(2) As far as the benchmark dataset goes, it is not clear whether 3360 prompts are sufficient for testing purposes. Also, it is not clear whether the problems in the dataset are a fair representation of the problem distribution in the real world.

Questions for authors

(1) While there is a lot of material from economics and social sciences which makes the paper eminently readable, there is not enough justification for the suggested use of the model as highlighted in the first weakness. It will be good to argue why LLMs will be used as described in the paper (and not invoke verifiable tools for the purpose).

(2) Please explicitly state the exact "incremental" contribution of this paper. Especially when compared to previous literature.

1 comment

Hi Reviewer LWgw,

Thank you once again for your valuable questions, particularly regarding motivation and scope of our work. It has helped us add clarity to both. As the rebuttal period is concluding tomorrow, we wanted to follow up to ensure our responses have adequately addressed the your concerns. Please let us know if there are any remaining questions or areas where further clarification would be helpful. We appreciate the time and effort you've dedicated to reviewing our work.

Thank you!

Reviewer_sK1VOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how language models handle prompts that pose unanswerable questions, whether due to lack of information or due to social choice-theoretic impossibilities. It finds that models often do not express concerns over the unanswerability of the question, which the authors read as being unable to distinguish answerable from unanswerable questions.

Critical strengths

The authors have made significant efforts to vary the conditions in an attempt to test the robustness/generalizability of the results, which made it easier to discuss the claims made in the paper in a substantive manner.

Critical weaknesses

As the authors noted, framing effects in the prompt have very large effects on the results. My read is that the authors believe that different framings lead to different levels of ability for the model to discovery the unanswerability of a question. I suspect the relevant mechanism is different, where the model may interpret experimenter intent differently when framing changes; for example, when without the "if known" suffix, it seems more likely for the model to view the question as a brain teaser that resists common-sense reasoning, even if it's able to realize that the problem is unanswerable based on common-sense reasoning. If this explanation is true (that the seeming failure to reason under indeterminacy is a result of misinterpreting experimenter intent), the generalizability of this paper's results to real-world indeterminacy will be limited, as misinterpreting real-life cases as brain teasers is much less likely.

Questions for authors

Could you share randomly sampled test transcripts with the reasoning trace or CoT of models, to rule out the intent misinterpretation explanation?

2 comments

Hi Reviewer sK1V,

Thank you once again for your valuable questions, particularly regarding potential alternative interpretation of our tasks. Your feedback was very helpful for rule out a potential confounder in our results. As the rebuttal period is concluding tomorrow, we wanted to follow up to ensure our responses have adequately addressed the your concerns. Please let us know if there are any remaining questions or areas where further clarification would be helpful. We appreciate the time and effort you've dedicated to reviewing our work.

Thank you!

Dear authors,

Thank you for the follow-up results and for the clarification on your claim; I think the new results you showed gives preliminary evidence against the misinterpretation hypothesis. I encourage you to discuss it prominently in your upcoming revisions.

I have increased my score accordingly.

Best and thank you

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.