- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors address the issue of when should LLMs abstain from giving direct answers. Existing work does not appear to have an agreed upon way of when and why a direct answer is unwarranted nor do they appear to have a way to evaluate what an alternative response should be. The authors evaluate whether a response meets four necessary conditions for direct answerability: well-definedness, knowledge support, criterion invariance, and system capability. Each failed condition specifies a criterion that a response should not assume. Then they go on to construct a diagnostic benchmark of thousands of manually screened questions and evaluate seven models. The results seem to reveal substantial differences across failure layers: violations are most frequent for underspecified questions and inaccessible information.
Critical strengths
The paper certainly addresses an important practical problem. When should an LLM refrain from giving a direct answer and what alternative responses might be more appropriate. The four proposed criteria offer a structured framework for distinguishing the reasons for the LLM to not give a direct answer. The benchmark tests on several models is a useful empirical contribution. The differences across the models may give further insight into their strengths and limitations.
Critical weaknesses
There is however limited conceptual and methodological novelty in the paper.
-
On what basis are the four criteria selected? Is there any theoretical justification or what guarantees do we have that these criteria exhaustively capture all possible underlying reasons that the LLM should not provide a direct answer? Do they form a "minimal" set of reasons? If yes, in what sense?
-
The four conditions basically formalize quite intuitive observations regarding when a question should not receive a direct answer. Beyond existing fairly well understood intuitive notions of ambiguity, epistemic uncertainty, insufficient access to information, and system limitations, I am not sure there is any fundamental new understanding that this work offers.
-
There is absolutely no theoretical analysis or a formal framework that explores how these criteria are related to each other, their inter-dependence (or are they "independent" of each other), any implication from one to the other, etc. What the authors offer is a simple taxonomy that is not at all grounded or characterized in theory.
-
What the empirical studies validate is how the models behave w.r.t. the four criteria. If the criteria are different, how would this behavior change?
Questions for authors
-
Can the authors provide a theoretical basis for the selected conditions? How do we know that they are necessary for direct answers to be unwarranted? Are they sufficient? What alternate set of criteria might work?
-
Are these four conditions independent, or can one identify any dependency among them? Would one imply another? Could there be an overlap? I do not believe that meaningful answers to these questions could be provided without addressing item 1 above.
-
How do the empirical conclusions made depend on the selected criteria? How robust are these empirical observations?
Response to W1: Boundary Cases
Response to Boundary Cases: Ambiguity and Multiple-Condition Failures
We thank the reviewer for highlighting the insufficient discussion of boundary cases. We have revised the Method section to clarify the distinctions among layers and how our framework handles overlapping failure reasons.
1. L1 vs. L3: Well-definedness vs. Criterion Invariance.
L1 concerns whether a request defines a sufficiently clear answer space, whereas L3 concerns whether an unconditional answer depends on unspecified evaluation criteria.
Consider "What is the best restaurant in the city?" Without identifying the city, the request fails L1. Even after specifying the city, "best" may still depend on unspecified criteria such as price or food quality (L3). Under our ordered framework, L1 is the primary diagnostic category. The primary response boundary is that the model must not assume a particular city has been specified, although it may provide a conditional answer under an explicitly stated assumption about the city.
2. L2 vs. L4: Knowledge Support vs. System Capability.
L2 concerns whether available world knowledge supports an answer, independently of the answering model. L4 concerns whether a particular model can obtain and provide that answer.
For example, "When did Einstein win the Nobel Prize in Literature?" fails L2 because it rests on a false premise. In contrast, "What is the main image on NASA's homepage right now?" fails L4 for a model without web access: the relevant information exists, but the model cannot retrieve it.
3. Multiple-Condition Failures and Real-World Applicability.
We now explicitly clarify that the four layers are not assumed to be mutually exclusive. The first failed check determines the primary diagnostic category and its corresponding response boundary. For real-world requests involving multiple failures, a more comprehensive evaluation would need to consider additional boundaries, potentially combining them after resolving earlier ambiguities.
Our current benchmark intentionally retains single-failure cases to isolate the evaluation of individual response boundaries. We acknowledge that systematically evaluating multiple simultaneous failures remains outside its current scope.
These revisions clarify that our hierarchy provides a consistent rule for identifying the primary failure reason. We hope the reviewer will read the revised Introduction and Method sections, and we welcome any further questions or concerns.