- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether LLMs recognize when a preference question has no determined answer. This happens either because incomplete preferences do not give a clear answer, or because a solution concept has no solution. The main contribution is a benchmark of determined and undetermined instances over house allocation and stable matching problems, on which four models rarely abstain and often fill gaps with their own systematic assumptions. However, part of the reported failure comes from how the tasks are prompted and scored rather than from the models’ reasoning. In addition, several existing studies have observed similar abstention failures in LLMs, which somewhat limits the novelty and impact of this work.
Critical strengths
- The question of whether LLMs notice when preferences are not sufficient to determine an answer is interesting and well motivated. It is important for studying agentic behaviors, LLM-based preference elicitation, and AI-mediated decision making processes.
- The paper is generally well-written. Figure 1 gives a very helpful overview of the task taxonomy for general audiences who are not familiar with this type of problem.
Critical weaknesses
- The claimed gap seems to mix reasoning with unstated conventions and prompt format. For example, the prompts never explain what an unlisted item means. However, in its instance generation, the paper also claims that it follows what it calls the standard convention that unranked alternatives are unacceptable (Appendix E.2.1), under which an agent who lists a but not b should prefer a to b. Yet in the aggregative task, an agent preferring a to b is instead scored as an error (Figure 1, Table 8). Thus, it is plausible that the models are simply adopting standard economic conventions rather than exhibiting a reasoning failure.
- The free-flow prompts also directly demand a number or a name with no abstention option. Prior work has well established that post-trained LLMs are heavily biased toward providing direct answers rather than abstaining when prompted this way (e.g., Kalai et al., 2025). Indeed, as shown in Table 15, simply adding an abstention option enables all models to achieve 100% accuracy, suggesting that the observed failure is largely an artifact of prompt framing.
Questions for authors
- How would the results change if the prompts state how unranked items should be read?
- How would the findings change if recognition of indeterminacy is measured separately from the final answer, for example by asking whether the answer is uniquely determined?
Follow-up
Hi Reviewer LWgw,
Thank you once again for your valuable questions, particularly regarding motivation and scope of our work. It has helped us add clarity to both. As the rebuttal period is concluding tomorrow, we wanted to follow up to ensure our responses have adequately addressed the your concerns. Please let us know if there are any remaining questions or areas where further clarification would be helpful. We appreciate the time and effort you've dedicated to reviewing our work.
Thank you!