acceptodds
Under review as a conference paper at ICLR 2027

The Other Side of Answering: How Language Models Respond Beyond Direct Answers

Abstract

Large language model (LLM) abstention studies when models should answer a question and how they should respond when a direct answer is unwarranted. Existing work typically evaluates whether models follow prescribed actions for predefined problem types, leaving no unified account of why direct answering is unwarranted or how alternative valid responses should be evaluated. We instead evaluate whether a response exceeds what the question supports. We formulate four ordered necessary conditions for direct answerability: well-definedness, knowledge support, criterion invariance, and system capability. Each failed condition specifies what a response must not assume or claim. Guided by this framework, we construct a diagnostic benchmark of 3,474 manually screened questions and evaluate seven models. Our results reveal substantial differences across failure layers, with violations most frequent for underspecified questions and inaccessible information. Boundary adherence and helpfulness emerge as related but distinct abilities. Ordinary CoT has inconsistent effects, whereas an explicit four-condition check reduces violations across all layers, especially from 26.5% to 11.8% for well-definedness and from 29.7% to 11.3% for system capability, without reducing helpfulness.

Primary area:
interpretability and explainable AI
Keywords:
LLM reliability, abstention, uncertainty quantification, answerability, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hTsMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors address the issue of when should LLMs abstain from giving direct answers. Existing work does not appear to have an agreed upon way of when and why a direct answer is unwarranted nor do they appear to have a way to evaluate what an alternative response should be. The authors evaluate whether a response meets four necessary conditions for direct answerability: well-definedness, knowledge support, criterion invariance, and system capability. Each failed condition specifies a criterion that a response should not assume. Then they go on to construct a diagnostic benchmark of thousands of manually screened questions and evaluate seven models. The results seem to reveal substantial differences across failure layers: violations are most frequent for underspecified questions and inaccessible information.

Critical strengths

The paper certainly addresses an important practical problem. When should an LLM refrain from giving a direct answer and what alternative responses might be more appropriate. The four proposed criteria offer a structured framework for distinguishing the reasons for the LLM to not give a direct answer. The benchmark tests on several models is a useful empirical contribution. The differences across the models may give further insight into their strengths and limitations.

Critical weaknesses

There is however limited conceptual and methodological novelty in the paper.

  1. On what basis are the four criteria selected? Is there any theoretical justification or what guarantees do we have that these criteria exhaustively capture all possible underlying reasons that the LLM should not provide a direct answer? Do they form a "minimal" set of reasons? If yes, in what sense?

  2. The four conditions basically formalize quite intuitive observations regarding when a question should not receive a direct answer. Beyond existing fairly well understood intuitive notions of ambiguity, epistemic uncertainty, insufficient access to information, and system limitations, I am not sure there is any fundamental new understanding that this work offers.

  3. There is absolutely no theoretical analysis or a formal framework that explores how these criteria are related to each other, their inter-dependence (or are they "independent" of each other), any implication from one to the other, etc. What the authors offer is a simple taxonomy that is not at all grounded or characterized in theory.

  4. What the empirical studies validate is how the models behave w.r.t. the four criteria. If the criteria are different, how would this behavior change?

Questions for authors
  1. Can the authors provide a theoretical basis for the selected conditions? How do we know that they are necessary for direct answers to be unwarranted? Are they sufficient? What alternate set of criteria might work?

  2. Are these four conditions independent, or can one identify any dependency among them? Would one imply another? Could there be an overlap? I do not believe that meaningful answers to these questions could be provided without addressing item 1 above.

  3. How do the empirical conclusions made depend on the selected criteria? How robust are these empirical observations?

Reviewer_fsmtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates scenarios where large language models (LLMs) need to abstain or are unsuitable to provide direct answers. Existing research on LLM abstention mostly targets predefined question types and evaluates whether models produce designated responses, lacking a unified analytical framework. This work proposes a four-tier framework of necessary conditions that defines the prerequisites for a valid direct answer: L1 clear definability, L2 knowledge support, L3 invariance of evaluation criteria, and L4 system accessibility. If any tier fails to be satisfied, the model should not deliver an unconditional deterministic answer.

Critical strengths

It points out that compliance is not equivalent to simply replying “I do not know”. Under the premise of compliance, a variety of high-value responses are allowed, including clarification, conditional deduction, and premise revision, which corrects the oversimplified evaluation logic adopted by many benchmark datasets.

For failures at each tier, it explores which response strategies can substantially improve response utility.

Critical weaknesses

The four tiers are formally defined, yet borderline ambiguous cases between tiers are not thoroughly discussed.

The paper only evaluates the performance of the full four-condition prompt in combination and lacks ablation studies that test L1 alone, L4 alone, or the removal of individual tiers. As a result, it cannot identify which condition contributes the most performance gain, nor can it verify whether redundant conditions exist.

Although the authors mention that general refusal rates do not rise, no dedicated metric is designed to quantify over-refusal, i.e., the phenomenon where the model declines to answer entirely even though a partial answer would have been feasible.

Questions for authors

The four-tier framework adopts sequential evaluation, where a tier is assessed only if the preceding tier passes. In real-world settings, user queries may suffer from both ambiguous definitions at L1 and missing evaluation criteria at L3. How can this framework be applied to evaluate such mixed queries?

The dataset consists entirely of manually curated single-failure samples. What modifications are required if the evaluation method proposed in this paper is directly deployed on real user queries?

5 comments
AuthorsOpenReview ↗

Response to W1: Boundary Cases

Response to Boundary Cases: Ambiguity and Multiple-Condition Failures

We thank the reviewer for highlighting the insufficient discussion of boundary cases. We have revised the Method section to clarify the distinctions among layers and how our framework handles overlapping failure reasons.

1. L1 vs. L3: Well-definedness vs. Criterion Invariance.

L1 concerns whether a request defines a sufficiently clear answer space, whereas L3 concerns whether an unconditional answer depends on unspecified evaluation criteria.

Consider "What is the best restaurant in the city?" Without identifying the city, the request fails L1. Even after specifying the city, "best" may still depend on unspecified criteria such as price or food quality (L3). Under our ordered framework, L1 is the primary diagnostic category. The primary response boundary is that the model must not assume a particular city has been specified, although it may provide a conditional answer under an explicitly stated assumption about the city.

2. L2 vs. L4: Knowledge Support vs. System Capability.

L2 concerns whether available world knowledge supports an answer, independently of the answering model. L4 concerns whether a particular model can obtain and provide that answer.

For example, "When did Einstein win the Nobel Prize in Literature?" fails L2 because it rests on a false premise. In contrast, "What is the main image on NASA's homepage right now?" fails L4 for a model without web access: the relevant information exists, but the model cannot retrieve it.

3. Multiple-Condition Failures and Real-World Applicability.

We now explicitly clarify that the four layers are not assumed to be mutually exclusive. The first failed check determines the primary diagnostic category and its corresponding response boundary. For real-world requests involving multiple failures, a more comprehensive evaluation would need to consider additional boundaries, potentially combining them after resolving earlier ambiguities.

Our current benchmark intentionally retains single-failure cases to isolate the evaluation of individual response boundaries. We acknowledge that systematically evaluating multiple simultaneous failures remains outside its current scope.

These revisions clarify that our hierarchy provides a consistent rule for identifying the primary failure reason. We hope the reviewer will read the revised Introduction and Method sections, and we welcome any further questions or concerns.

AuthorsOpenReview ↗

Response to Q1

Response to Q1: Queries Involving Multiple Failure Conditions

First, we clarify that our framework follows a hierarchical diagnostic order: if a request fails L1, subsequent conditions such as L3 are not evaluated, because L1 already explains why a direct answer is unwarranted. However, real-world requests may involve additional constraints that become apparent after earlier failures are resolved.

For example, "What is the most beautiful place in ?" fails L1 because the question is incomplete. Even after completing it as "What is the most beautiful place in Europe?", the question still involves L3 because "most beautiful" depends on unspecified aesthetic criteria. Our current framework assigns the original request to L1, without further evaluating L3. A natural extension is to combine the corresponding response boundaries: the model must neither silently assume the missing location (L1) nor present a subjective judgment as an unconditional conclusion (L3). Any response respecting both constraints should be considered compliant, regardless of its specific strategy.

This illustrates an advantage over question-type-based taxonomies. Traditional approaches might classify the request as ambiguous or subjective, but combining these categories and their associated response strategies does not directly establish which responses are permissible. In contrast, our framework specifies what a response must not assume or claim, allowing multiple response constraints to be combined without prescribing fixed response strategies. Although our current benchmark focuses on the primary failure condition and has not empirically validated such combinations, this boundary-oriented formulation provides a natural direction for extending evaluation to mixed queries.

AuthorsOpenReview ↗

Response to W2: Ablation Study

Response to W2: Ablation Study

Our additional ablation results reveal condition-specific specialization. All results are averaged across six models on AnswerBoundaryBench, where Only-Li denotes a prompt that checks only condition Li, and all reported differences are relative to the full four-condition prompt. On their respective layers, Only-L1 and Only-L4 improve boundary adherence by 3.34 and 2.34 percentage points, respectively, while the differences for L2 and L3 are not significant. However, across all four layers, Only-L1, Only-L2, and Only-L3 reduce boundary adherence by 1.97, 7.60, and 4.69 percentage points, respectively.

These results suggest that the conditions are complementary: individual checks can be more effective for their corresponding failure types, while the full prompt performs better across heterogeneous questions. Importantly, RQ5 aims not to identify the optimal prompt, but to investigate whether models can recognize when direct answers are unwarranted and apply this understanding when responding. Our findings suggest that models possess some recognition ability but do not consistently apply it in their actual responses.

Reviewer_AGGQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a four-layer hierarchy for assessing whether LLMs should provide direct answers, covering question clarity, knowledge support, criterion invariance, and system capability. Based on this framework, the authors construct AnswerBoundaryBench with 3,474 questions and evaluate seven LLMs on boundary adherence, helpfulness, and response strategies. They further show that an explicit four-condition check reduces boundary violations compared with standard CoT prompting.

Critical strengths
  • The paper distinguishes boundary adherence from helpfulness, allowing valid conditional or corrective responses rather than requiring a fixed refusal strategy.
  • The four-layer hierarchy provides an interpretable organization of different reasons why direct answering may be unwarranted.
  • The evaluation covers seven models and includes cross-judge comparisons and human validation. The four-condition check also produces substantial reductions in boundary violations.
Critical weaknesses
  • The four layers substantially overlap with existing taxonomies in CoCoNot and AbstentionBench, which already cover similar failure types and support responses beyond simple refusal. The paper does not sufficiently demonstrate the added value of its hierarchical formulation or boundary-based evaluation.
  • The boundaries between L1 and L3 (missing information vs. unspecified criteria) and L2 and L4 (unsupported knowledge vs. inaccessible information) can be ambiguous. Moreover, the benchmark excludes questions that fail multiple conditions, leaving the applicability of the ordered hierarchy to such cases unclear.
  • The benchmark contains only questions that do not warrant direct answers. Without answerable or contrastive examples, it is unclear whether the proposed intervention increases unnecessary abstention or compromises normal answering performance.
Questions for authors
  • What concrete diagnostic advantage does the proposed hierarchy offer over existing taxonomies such as CoCoNot?
  • How does the four-condition check affect performance on directly answerable questions, particularly the over-abstention rate?
1 comment
AuthorsOpenReview ↗

Response to W2: Condition Ambiguity and Multiple-Condition Failures

Response to W2: Condition Ambiguity and Multiple-Condition Failures

We thank the reviewer for raising concerns about ambiguity and overlapping conditions. We have revised the Introduction and Method to better explain the motivation and design of our hierarchical classification framework.

First, our framework aims to ensure consistent classification and evaluation.

Existing taxonomies allow a request to belong to multiple categories, potentially leading to different evaluation criteria. Our framework instead classifies requests by why they do not warrant direct answers, with each category corresponding to a common response boundary. The ordered hierarchy assigns each request to its first failed check, providing a consistent rule for determining its primary diagnostic category.

Second, the four layers have distinct diagnostic roles and a natural ordering.

We examine the request, its possible answers, and finally the answering model.

  • L1 vs. L3: L1 examines whether the request defines a clear question, whereas L3 examines whether an answer to an already well-defined question depends on unspecified criteria. For example, "What is the most beautiful place in ?" fails L1 because the location is missing. Once specified as "Europe," the question may still fail L3 because "most beautiful" depends on unspecified preferences.
  • L2 vs. L4: L2 examines whether a knowledge-supported answer exists, independently of the answering model. L4 concerns whether a particular model has the knowledge or access required to provide it. Thus, an unsupported premise and an answer that exists but is inaccessible to the model represent different reasons for non-direct answerability.

Third, we acknowledge that multiple conditions may fail simultaneously.

Our hierarchy does not assume that the four layers are mutually exclusive. Instead, the first failed check determines the primary diagnostic category, without implying that later constraints cannot also apply. Our current benchmark focuses on requests with a clearly identifiable primary boundary and does not systematically evaluate combinations of multiple failed conditions. We have clarified this scope in the revised manuscript and leave the systematic evaluation of combined boundaries to future work.

Importantly, we no longer present the four layers as a minimal or exhaustive set of theoretically necessary conditions, but as an ordered diagnostic framework designed to support consistent classification and boundary-based evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.