acceptodds
Under review as a conference paper at ICLR 2027

Training-Free and Generation-Free Answerability Estimation in Foundation Models

Abstract

Foundation models, including large language models (LLMs) and vision-language models (VLMs), answer some queries correctly and fail on others. We study this through the concept of answerability: whether a model can correctly answer a given query. Estimating answerability before generation enables adaptive inference, such as model routing, retrieval, or abstention. Most existing methods either estimate answerability after generation or rely on task-specific data to construct an estimator. We consider a training-free and generation-free setting, in which the estimate requires neither parameter optimisation nor answer generation. We propose Answerability Direction Probing (ADP), guided by a decomposition of answerability into input adequacy, knowledge boundary, and reasoning capability. ADP constructs answerability-related concept directions from vocabulary tokens in the unembedding space and probes representations across layers, using a Jacobian lens for intermediate-layer alignment. The score is obtained from a single forward pass. With a single configuration shared across LLMs and VLMs, ADP outperforms the compared training-free and generation-free methods across multiple model families and diverse tasks. In a model-cascading case study, an ADP-based cascade matches the accuracy of always using the larger model while invoking it less often.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Answerability estimation, Large language models, Vision-language models, Representation probing, Uncertainty estimation, Model cascading

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_STjGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article introduces Answerability Direction Probing (ADP), a method for estimating a model's answerability without training or generating answers. The authors break down answerability into three factors: input sufficiency, knowledge boundaries, and reasoning ability, and use vocabulary embeddings to construct semantic directions, evaluating them by aligning hidden states across layers with a Jacobian Lens. Experiments cover 12 LLMs/VLMs and 5 datasets, confirming the method's effectiveness and showing its potential for use in model cascades.

Critical strengths
  • The method is practically useful. ADP can estimate answerability before generation without extra training or answer generation, providing an effective scheme for model routing and adaptive reasoning.

  • Experimental validation is solid. It consistently achieves overall performance gains across multiple LLMs and VLMs, and the core design has been verified through directional replacement, robustness analysis, and ablation experiments.

Critical weaknesses
  • The three-factor decomposition lacks direct validation. The final method only uses the combined directions of Challenge and Feasibility, rather than directly modeling the three factors proposed. The existing correlation analysis isn’t enough to prove that these three factors can effectively explain different types of answer failures.

  • The actual efficiency gains aren’t very clear. Although the method only requires one online forward pass, there’s still extra overhead from multi-layer Jacobian precomputation and projection.

  • The model cascade experiments also didn’t report end-to-end latency or real computation costs.

Questions for authors
  • Is it possible to evaluate ADP separately for input insufficiency, knowledge gaps, and reasoning failures, to verify the actual effect of the three-factor decomposition?

  • Could u also add the precomputation cost of the Jacobian, online inference latency, and the actual computational cost savings of model cascading?

3 comments
AuthorsOpenReview ↗

Response (1/3)

Dear Reviewer STjG,

Thank you for your detailed comments and helpful feedback. We appreciate the time and effort you invested in reviewing our paper. We address each point below and will incorporate the new results and clarifications into the revised manuscript.

W1: Three-part Decomposition and Factor-level Validation

Answer:

The three-part decomposition provides theoretical guidance for constructing answerability-related concepts. A factor is a theoretical source of failure, a concept is a specific semantic attribute, and a concept direction reads out a representational signal associated with that attribute rather than measuring the factor as a whole. Factors and concepts do not have a one-to-one correspondence: one factor may be associated with multiple concepts, and one concept may be related to multiple factors. The five factor-level directions are specific concept directions constructed with the corresponding factors as theoretical starting points. Each focuses on a particular concept mainly relevant to the factor that motivated its construction. They are used for concept-level analysis rather than as direct measures of the three factors. ADP aims to estimate overall answerability rather than output three separate factor-level scores.

The main experiments use two integrative directions, Challenge and Feasibility, as a theoretically motivated general-purpose default configuration. They were designed from the outset to account for multiple factors and to provide complementary perspectives: Challenge takes the task perspective, whereas Feasibility takes the perspective of the current model–input pair. Challenge describes the overall demands of a query, including the adequacy of the input and the knowledge and reasoning it requires. Feasibility describes how feasible it is for the current model to answer the current input correctly. It primarily reflects Knowledge Boundary and Reasoning Capability, while also being affected by Input Adequacy. The two directions do not correspond one-to-one to the three factors, nor are they a direct mathematical combination of the three factors.

Figure 3 shows that the factor-level directions exhibit selective but overlapping responses to factor-relevant variables. The main experiments support the effectiveness of Challenge + Feasibility for estimating overall answerability. These results support the intended interpretations of the directions, but are insufficient to establish that the three factors have been directly validated or that they can separately explain different types of answer failures.

ADP does not currently identify which of the three factors is responsible for a given failure. Failure sources are not mutually exclusive, and a single error may involve multiple factors. Moreover, because factors and concepts do not have a one-to-one correspondence, a single ADP score cannot uniquely attribute a failure to a particular factor. In principle, the joint distribution and response patterns across a larger set of concept directions may reflect the relative strengths of the three factors and help attribute failures to individual factors; we leave this for future work.

AuthorsOpenReview ↗

Response (2/3)

W2: Jacobian Precomputation and Online Estimation Cost

Answer:

The Jacobian lens outperforms the logit lens across all six combinations of self-assessment and generation prompts, with gains of 0.16–0.57 pp. This indicates a small but consistent improvement in AUROC (see Reviewer kkDw, W3, for details). To address the additional computational cost, we report online estimation cost separately from the offline storage and precomputation costs of the Jacobian.

The online costs are broken down as follows:

Table 1. Online Cost of ADP Computation and 2B Self-assessment

ComponentLatency (ms/query) ↓Latency Relative to 2B Inference (%)GFLOPs/query ↓GFLOPs Relative to 2B Inference (%)
ADP Computation2.890.15%0.180.01%
2B Self-assessment34.631.76%992.6470.35%
2B Inference1963.79100.00%1411.07100.00%

Table 1 shows that the online overhead of ADP projection is small. The 2B self-assessment requires only a single forward pass, without token-by-token generation; its GFLOPs mainly come from the image input tokens of the VLM.

The offline precomputation and storage costs of the Jacobian are reported below. All times were measured on a single RTX PRO 6000:

Table 2. Jacobian Storage and Precomputation Time (1× RTX PRO 6000)

ModelModel Size (GB)Jacobian Storage (GB)Jacobian / Model Size (%)Jacobian Precomputation Time (1× RTX PRO 6000)
Qwen3-VL-2B4.260.173.9412 min
Qwen3-VL-4B8.880.343.8422 min
Qwen3-VL-8B17.530.874.9855 min
Qwen3-VL-32B66.722.463.693 h 37 min
Gemma-3-4B8.600.333.8121 min
Gemma-3-12B24.381.034.231 h 22 min
Llama-3.2-1B2.470.093.737 min
Llama-3.2-3B6.430.385.8721 min
Phi-4-mini7.670.435.6625 min
Phi-429.321.525.191 h 38 min
Qwen3-0.6B1.500.042.795 min
Qwen3-8B16.380.875.3356 min

Table 2 shows that, across the 12 models, Jacobian storage ranges from 0.04 to 2.46 GB, accounting for approximately 2.79%–5.87% of model size. Jacobian precomputation takes between 5 minutes and 3 hours 37 minutes. This is a one-time offline construction cost per model, separate from the online estimation cost for each query.

AuthorsOpenReview ↗

Response (3/3)

W3: End-to-end Cost of the Model Cascade

Answer:

We randomly sampled 300 questions from MathVista and measured the full end-to-end latency and GFLOPs of 2B Only, ADP Cascade, and 8B Only.

Table 3. End-to-end Cascade Cost on 300 MathVista Samples

MethodEnd-to-end Latency (ms/query) ↓Latency Change Relative to 8B Only (%)GFLOPs/query ↓GFLOPs Change Relative to 8B Only (%)Accuracy (%) ↑Accuracy Difference (Current Method − 8B Only, pp)
2B Only1963.79−40.34%1411.07−76.51%52.70−15.21 pp
ADP Cascade2998.14−8.91%5497.26−8.51%67.57−0.34 pp
8B Only3291.550.00%6008.320.00%67.910.00 pp

Table 3 shows that, compared with using 8B directly, ADP Cascade reduces end-to-end latency by 8.91% and GFLOPs by 8.51%, with an accuracy decrease of only 0.34 pp. This result includes both the 2B forward pass and ADP computation. For text-only inputs, or in settings where the model uses a higher thinking level and generates longer answers, the cascade is expected to save more computation.

Q1: Factor-specific Evaluation

Answer:

Comprehensive instance-level annotations covering all three factors are not available, but our tasks include settings in which one factor dominates or is manipulated. PopQA failures mainly reflect missing knowledge: ADP reaches 75.87 macro AUROC and is best on all six LLMs. GSM8K gives all numbers in the question, so failures mainly reflect reasoning: ADP reaches 62.76 against 63.74 for PE, whose advantage comes largely from a question-length shortcut; after controlling for length, ADP reaches 61.02 against 56.41 for PE (see our response to Reviewer kkDw, W4). On DocVQA, controlled blur manipulates input adequacy with the question and model fixed (Figure 3). In addition, the LLM judge used for labelling (Appendix B) first checks input adequacy and marks inputs that cannot determine the answer (6.1% of OK-VQA, 1.2% of MathVista and 0.1% of DocVQA). These are dataset-level and controlled analyses rather than instance-level attribution, which, as discussed in W1, we leave to future work.

Q2: Precomputation, Online Cost, and Cascade Savings

Answer:

Jacobian storage and precomputation, together with online estimation costs, are reported in W2. The cascade's end-to-end latency and computational savings are reported in W3.

Reviewer_kkDwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Answerability Direction Probing (ADP), a training-free and generation-free method that estimates whether an LLM or VLM can correctly answer a query. It projects hidden states, obtained from a single forward pass on a self-assessment prompt and aligned across layers with the Jacobian lens, onto an “answerability direction” built from unembedding vectors of contrastive seed words (e.g., difficult vs. simple, unable vs. able). Across 12 models and 5 datasets, ADP outperforms likelihood-based and Yes/No-based training-free baselines in AUROC and PRR, and the authors also demonstrate its use for model cascading.

Critical strengths
  • The method requires neither training nor task data, and applies uniformly to both LLMs and VLMs without modality-specific changes.
  • The analysis in Figure 3, which relates individual directions to factor-relevant variables, is interesting and supports the intended interpretation of the directions.
  • The appendices are thorough: control, related and random directions, seed-word robustness, random-word corruption, layer and position analyses, and bootstrap confidence intervals.
  • The Limitations section is fair and openly acknowledges where the method underperforms.
Critical weaknesses
  • Appendix A states that 25 directions are used, including the Yes/No and positive/negative reference directions, which are “listed for completeness”. However, Table 4 contains only 23 directions, and results for positive/negative are not reported anywhere. Appendix G also states that “the 25 directions fall into four classes”, although the four classes contain 23 directions.
  • The three-factor decomposition (Input Adequacy, Knowledge Boundary, Reasoning Capability) is presented as a contribution, but it does not participate in the final method. The factor-level directions are used for analysis only, and combining all seven directions (72.40) performs worse than the two integrative ones (73.75).
  • The benefit of the Jacobian lens is unclear. Replacing it with the logit lens reduces AUROC by only 0.40 (Table 9), while the Jacobians require offline computation and storage of a d×d matrix per probed layer.
  • On GSM8K, ADP is outperformed by PE on three of six models. On Llama-3.2-1B, ADP is also worse than ADP(Yes). This suggests the method captures the Reasoning Capability factor poorly.
  • The baselines are weak. The likelihood-based scores are computed on the raw query without the self-assessment prompt. No trained probe is included as a reference, so it is unclear how much performance is lost by being training-free.
Questions for authors
  • How correlated are the individual directions? Could the authors report cosine similarities between the concept directions and correlations between their per-sample projections?
  • Why are exactly the last 75% of layers used (lines 309–310)? How does performance change with the size of the probed span?
  • How were the weights λ in Eq. (5) chosen? Since the concept directions are not normalised before combination, the effective weights depend on their norms. Could the authors report these norms and results with normalised directions?
  • On which data was the final configuration selected (the direction set, layer span and probing position)? What do the superscripts H and L in SaHS_a^H and SaLS_a^L stand for?
9 comments
AuthorsOpenReview ↗

Response (1/9)

Dear Reviewer kkDw,

Thank you for your detailed comments and helpful feedback. We appreciate the time and effort you invested in reviewing our paper. We address each point below and will incorporate the new results and clarifications into the revised manuscript.

W1: Direction Count and Reference Directions

Answer:

The description in Appendix A was unclear and has now been corrected. The full set of 25 directions consists of the 23 directions in the four categories listed in Table 4 and two additional reference directions (Yes/No and Positive/Negative). Appendix G has also been corrected: its four classes include only the 23 directions in those four categories and exclude the two reference directions.

We report ADP(Yes) rather than ADP(Positive) because ADP(Yes) corresponds more directly to the P(Yes) measure in Chen et al. (2026) and the Yes/No readout used by IC. The table below reports the performance of ADP(Positive).

Table 1. Macro AUROC of ADP, IC, ADP(Yes), and ADP(Positive) (%)

Evaluation GroupADP (%) ↑IC (%) ↑ADP(Yes) (%) ↑ADP(Positive) (%) ↑Δ (ADP(Positive) − ADP(Yes), pp)
VLM-1873.7567.8170.6871.51+0.83
LLM-1269.3259.7965.5663.60−1.96
All-3071.9864.6068.6368.34−0.29

W2: Three-part Decomposition and Integrative Directions

Answer:

The three-part decomposition informs ADP's final method by providing the theoretical basis for constructing its concepts. A factor describes a theoretical source of failure, a concept captures a specific semantic attribute, and a concept direction reads out the signal associated with that attribute. Thus, factor-level directions capture specific concepts rather than producing three separate factor scores. Factors and concepts do not have a one-to-one correspondence: one factor may be associated with multiple concepts, and one concept may be related to multiple factors. Therefore, selecting one concept direction from each of the three factors would still not mean that the factors are being measured separately.

The main experiments use two integrative directions: Challenge and Feasibility. Challenge captures the overall demands of a query, including the adequacy of the input and the knowledge and reasoning it requires. Feasibility captures how feasible a correct answer is for the current model–input pair. It primarily reflects Knowledge Boundary and Reasoning Capability, while also being affected by Input Adequacy. These two directions account for multiple potential sources of failure and estimate overall answerability from complementary perspectives; they do not correspond one-to-one to the three factors, nor are they a direct mathematical combination of those factors.

The direction-count experiment shows that mean AUROC increases with the number of directions, but with diminishing marginal gains. This is an average trend across combinations and does not mean that every specific combination improves as directions are added. The per-sample signals of different concepts may overlap; adding directions does not necessarily introduce independent information and may instead reinforce shared signals and change their relative contributions within the combination. Therefore, it is not contradictory that the equal-weight combination of seven directions achieves an AUROC of 72.40, below the 73.75 achieved by Challenge + Feasibility. We use the latter as a theoretically motivated general-purpose default configuration, chosen for its complementary perspectives and simplicity, without claiming that it is empirically optimal.

AuthorsOpenReview ↗

Response (2/9)

W3: Jacobian lens and logit lens

Answer:

In the original configuration, the Jacobian lens outperforms the logit lens by 0.40 pp. To test whether this gain occurs only under one prompt setting, we further compare six combinations of self-assessment and generation prompts.

Table 2. VLM-18 Macro AUROC with the Jacobian lens and logit lens under different prompt combinations (%)

Self-assessment PromptGeneration PromptJacobian lens Macro AUROC (%) ↑Logit lens Macro AUROC (%) ↑Δ (Jacobian lens − logit lens, pp)
ConciseOriginal74.2573.67+0.57
ConciseAlternative75.1274.59+0.53
OriginalOriginal73.7573.35+0.40
OriginalAlternative74.7874.42+0.36
DetailedOriginal73.1072.81+0.29
DetailedAlternative73.8573.70+0.16

Table 2 shows that the Jacobian lens outperforms the logit lens in all six combinations, with gains ranging from 0.16 to 0.57 pp. The gains are modest, but their direction is consistent across all six settings, indicating that Jacobian-lens alignment provides a small but stable improvement.

The additional costs of the Jacobian lens are offline precomputation and Jacobian storage:

Table 3. Jacobian Storage and Precomputation Time (1× RTX PRO 6000)

ModelModel Size (GB)Jacobian Storage (GB)Jacobian / Model Size (%)Jacobian Precomputation Time (1× RTX PRO 6000)
Qwen3-VL-2B4.260.173.9412 min
Qwen3-VL-4B8.880.343.8422 min
Qwen3-VL-8B17.530.874.9855 min
Qwen3-VL-32B66.722.463.693 h 37 min
Gemma-3-4B8.600.333.8121 min
Gemma-3-12B24.381.034.231 h 22 min
Llama-3.2-1B2.470.093.737 min
Llama-3.2-3B6.430.385.8721 min
Phi-4-mini7.670.435.6625 min
Phi-429.321.525.191 h 38 min
Qwen3-0.6B1.500.042.795 min
Qwen3-8B16.380.875.3356 min

Table 3 shows that, across the 12 models, Jacobian storage ranges from 0.04 to 2.46 GB, and precomputation takes from 5 minutes to 3 hours and 37 minutes. All measurements were obtained on a single RTX PRO 6000. The Jacobian lens trades a modest amount of offline computation and storage for a small but stable AUROC improvement; the logit lens remains an option when deployment simplicity is more important.

AuthorsOpenReview ↗

Response (3/9)

W4: GSM8K and Token-length Shortcut

Answer:

In the original GSM8K results, PE outperforms ADP on three of the six models. One possible explanation is that GSM8K question length may correlate with the number of conditions and reasoning steps required, allowing length itself to act as a shortcut for predicting correctness.

Following the setting of Chen et al. (2026) and to facilitate comparison, we compute adapted PE without length normalisation. Consequently, the PE scores retain a clear length signal: the per-sample Spearman correlation between PE and negative token length is 0.86. Using negative token length alone as a score also achieves a Macro AUROC of 62.15. To examine whether the original PE performance is affected by this length signal, we divide questions into 30 token-length quantile bins using each model's own tokenizer, and compute a pair-count-weighted AUROC using only positive–negative sample pairs within the same bin. Table 4 reports the per-model results.

Table 4. Token-length Analysis for Each GSM8K Model (30 Bins, AUROC, %)

ModelPE AUROC (%) ↑−Token Length AUROC (%) ↑Spearman ρ (PE, −Token Length)PE AUROC after 30-bin Matching (%) ↑Δ (PE Matched − PE Original, pp)ADP AUROC after 30-bin Matching (%) ↑Δ (ADP Matched − PE Matched, pp)
l1B63.4862.620.8954.14−9.3454.39+0.25
l3B64.9363.360.8857.31−7.6257.51+0.20
p4B64.7962.350.8457.64−7.1566.06+8.42
p14B61.2456.970.8560.06−1.1871.75+11.69
Q06B68.4267.370.8857.23−11.1958.97+1.74
Q8B59.5960.210.8352.11−7.4857.44+5.33
Macro 663.7462.150.8656.41−7.3361.02+4.61

The per-model results in Table 4 show that, after 30-bin matching, PE AUROC is lower than its original AUROC on all six models. At the same time, ADP's within-bin AUROC is higher than PE's on all six models.

Reviewer_tqbXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ADP, a method to estimate whether an LLM or VLM will answer a question correctly before generating its response. ADP reads the model's internal representations and scores them using directions constructed from contrasting words, such as "difficult–simple" and "unable–able." Experiments across 12 models and five datasets show that the method generally distinguishes correct from incorrect responses better than the compared baselines. The authors also demonstrate its use in deciding when to pass a question from a smaller model to a larger one based on a low predicted answerability score.

Critical strengths

1) The research question has clear practical relevance.

Estimating whether a model will answer correctly before generating its response could support abstention, retrieval, and model routing. I find this particularly interesting because it could avoid unnecessary generation and reduce computational costs. The applicability to both LLMs and VLMs further increases its potential usefulness.

2) The experiments provide encouraging evidence across different settings.

The evaluation covers multiple model families and tasks. I also appreciate the comparison with ADP(Yes), which uses the same procedure but replaces the proposed probing direction with a Yes/No direction. This helps show that the proposed direction contributes to the reported improvements.

Critical weaknesses

1) I found the connection between the three answerability factors and the actual method difficult to understand.

The distinction between Input Adequacy, Knowledge Boundary, and Reasoning Capability is not sufficiently clear to me. Figure 1 provides examples, but I would appreciate a more direct explanation of how the authors distinguish missing knowledge from a failure to reason over the available information. A wrong answer alone does not necessarily tell us which of these problems occurred.

I also found the connection between these factors and the chosen directions difficult to follow. For example, why does Certainty represent reasoning capability? A model can be confident and still reason incorrectly. Furthermore, the final method uses Challenge and Feasibility rather than separate directions for the three factors. I would like the authors to explain this transition more clearly and distinguish what they assume from what their experiments actually establish.

The choice of directions is important, as shown by the difference between ADP and ADP(Yes). I appreciate the synonym analysis in Appendix F, which shows stable results when changing the Challenge word groups on VLMs while keeping Feasibility fixed. However, I would like to understand whether this stability also holds when changing the Feasibility groups and when evaluating LLMs. The issue is not that manually chosen words are necessarily inappropriate, but that readers need to understand how much the conclusions depend on these choices.

The same question applies to the self-assessment prompt. The paper uses a fixed formulation, but I could not find an experiment testing alternative ways of asking whether the model can answer correctly. Since ADP reads the representations produced under this prompt, changes in its wording could affect the scores. It would be useful to check whether the main improvements persist across several equivalent formulations.

Finally, the answerability labels come from whether the model answers correctly under a particular generation prompt and greedy decoding. This measures success under that configuration, rather than necessarily establishing whether the model can answer the question. I would appreciate a clearer discussion of this distinction and of whether changing the answer-generation prompt changes the findings.

2) I would like to understand more clearly what happens when ADP is used to make an actual decision.

The ranking results are encouraging, but they do not fully answer the practical question of when we should trust the model. Among questions classified as answerable, how often does the model actually answer correctly? Conversely, among those classified as unanswerable, how often would it have answered correctly anyway? Compared with the baselines, does ADP more often accept questions that lead to incorrect answers, or reject questions that could have been answered correctly?

These two errors have different consequences. Accepting an incorrect answer reduces reliability, while rejecting a correct one can cause unnecessary abstention or additional computation. I would therefore like to see these outcomes at clearly specified thresholds and comparable operating points, rather than relying only on AUROC and PRR. It is also unclear how a user should choose the threshold without labeled examples from the task. In particular, was the threshold matching the larger model’s accuracy in the cascading experiment selected using the same data on which that accuracy is reported, or a separate calibration set?

The computational benefit also needs to be quantified more directly. In the cascading experiment, 85.4% of inputs are still sent to the larger model, while every input first requires a forward pass through the smaller model and the ADP computation. This may still provide a saving, but reducing the number of large-model calls does not tell us how much the complete system saves. Reporting end-to-end latency or computational cost, together with the offline cost and storage requirements of the Jacobian mappings, would make the practical benefit clearer.

Questions for authors
  • Could you explain more intuitively how Input Adequacy, Knowledge Boundary, and Reasoning Capability are distinguished when several factors could explain the same failure? Why is Certainty specifically associated with reasoning capability?
  • How sensitive is ADP's performance to changes in the self-assessment prompt, the answer-generation prompt, or the reasoning configuration? Is the intended prediction explicitly conditional on the answer-generation configuration?
  • At a specified decision threshold, how often does ADP incorrectly accept an unanswerable question or reject an answerable one? How do these outcomes compare with the baselines at matched coverage or error levels?
  • How was the operating point matching the larger model's accuracy selected in the cascading experiment? Would selecting a threshold for new inputs require a separate calibration set?
  • What is the end-to-end cost of the cascade compared with directly using the larger model, including the smaller model’s forward pass and ADP computation? Please distinguish the offline Jacobian construction cost from the online estimation cost.
  • How would ADP work in multiple-choice settings where the question must be interpreted together with the candidate options? Would the self-assessment input include the complete option set? I would be interested in whether its score captures changes in answerability when the question remains fixed but the options are perturbed, as studied in "ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks?" (EACL 2026). A related application concerns medical abstention: could ADP help identify when a model should abstain rather than select a potentially incorrect answer under uncertainty, as investigated in "LLMs (Almost) Never Abstain Under Medical Uncertainty" (ACL 2026)? I suggest discussing these works as relevant context for option-sensitive answerability and abstention in high-stakes settings. Evaluation in these settings would be an interesting future direction.
9 comments
AuthorsOpenReview ↗

Response (1/8)

Dear Reviewer tqbX,

Thank you for your detailed comments and helpful feedback. We appreciate the time and effort you invested in reviewing our paper. We address each point below and will incorporate the new results and clarifications into the revised manuscript.

W1.1: Three-part Decomposition

Answer:

The three-part decomposition further refines the classical distinction between aleatoric and epistemic uncertainty from the perspective of failure sources. Input Adequacy is conceptually related to input-side aleatoric uncertainty: if the current input lacks necessary information, evidence, or clarity, even a model with the relevant knowledge and capabilities cannot determine the answer from that input alone. Knowledge Boundary and Reasoning Capability further distinguish two aspects of model-side epistemic uncertainty: the former concerns whether the model has the factual knowledge required to answer the question, whereas the latter concerns its intrinsic ability to process, integrate, and reason over available information.

Knowledge Boundary refers to the model's lack of the factual knowledge required to answer a question. For example, a model may lack facts about a particular public figure or be unaware of a specific theorem. Such knowledge gaps can often be addressed by supplying the missing knowledge through RAG or knowledge injection.

Reasoning Capability refers to the model's intrinsic ability to process and integrate available information and carry out the required reasoning. If a model has access to the relevant knowledge but lacks arithmetic ability, or cannot maintain coherence over a long reasoning chain, this reflects a limitation in Reasoning Capability. Unlike a factual knowledge gap, such a limitation is generally difficult to overcome simply by adding information to the input. It usually requires improving the model's own capabilities through pre-training or post-training, although tool use can also assist with computation or reasoning.

We agree that a specific failure source cannot be identified from a wrong answer alone. The three factors are not mutually exclusive: a single failure may involve insufficient input information, missing knowledge, and inadequate reasoning capability at the same time. Accordingly, the three-part decomposition describes possible sources of failure; it is not intended to assign every incorrect answer to a single factor.

W1.2: From Factors to Concepts and the Final Directions

Answer:

The three-part decomposition provides theoretical guidance for constructing answerability-related concepts. A factor describes a broad source of failure, whereas a concept captures a more specific semantic attribute. A concept direction reads out a signal associated with that attribute from the model's representations; it does not measure the factor as a whole. Factors and concepts do not have a one-to-one correspondence: one factor may be associated with multiple concepts, and one concept may be related to multiple factors. For example, Certainty is a concept constructed from the perspective of Reasoning Capability. The certainty-related signal encoded in the model's representation during pre-generation self-assessment reflects the model's certainty about the outcome of its own processing, but it does not directly measure actual reasoning capability; the model may still be confident but wrong.

ADP aims to estimate overall answerability rather than output three separate scores for the three factors. The five factor-level directions capture specific concepts and support concept-level analysis; therefore, selecting one concept direction for each factor would still not amount to measuring the factors themselves. The main configuration uses two integrative directions, designed from the outset to account for multiple potential sources of failure. Challenge captures the overall demands of a query, including the adequacy of the input and the knowledge and reasoning it requires. Feasibility captures how feasible a correct answer is for the current model–input pair. It primarily reflects Knowledge Boundary and Reasoning Capability, while also being affected by Input Adequacy. The two directions provide complementary, potentially overlapping perspectives on overall answerability; they do not correspond one-to-one to the three factors, nor are they a direct mathematical combination of those factors. We use Challenge + Feasibility as a compact, theoretically motivated general-purpose default configuration, without claiming that it is empirically optimal.

Figure 3 shows that the directions exhibit selective but overlapping responses to factor-relevant variables. The main experiments further support the effectiveness of Challenge + Feasibility for estimating overall answerability. These results support the intended interpretations of the directions, but do not establish that the three factors have been directly validated.

AuthorsOpenReview ↗

Response (2/8)

W1.3: Seed-word Robustness

Answer:

We added three seed-word replacement settings: C-only replaces Challenge alone, F-only replaces Feasibility alone, and C+F replaces both directions. We evaluated all 225 combinations for each of C-only and F-only, and all 50,625 cross-side pairings for C+F. The additional seed candidates for Feasibility were: Unanswerable (incompetent, unrealistic); Answerable (achievable, manageable).

Table 1. Seed-word Replacement Results for Challenge and Feasibility (Macro AUROC, %)

Replacement SettingEvaluation GroupOriginal ADP Macro AUROC (%) ↑Mean ± SD Across Candidate Combinations (%) ↑Normalised Rank Position of Original Configuration (%) ↓Δ (Original ADP − Candidate Mean, pp)
C-onlyVLM-1873.7573.80 ± 0.1067.1%−0.05
C-onlyLLM-1269.3169.30 ± 0.3348.9%+0.01
C-onlyAll-3071.9772.00 ± 0.1558.7%−0.03
F-onlyVLM-1873.7573.70 ± 0.1133.3%+0.05
F-onlyLLM-1269.3169.55 ± 0.2483.1%−0.23
F-onlyAll-3071.9772.04 ± 0.1072.9%−0.06
C+FVLM-1873.7573.67 ± 0.1833.8%+0.08
C+FLLM-1269.3169.46 ± 0.4064.2%−0.14
C+FAll-3071.9771.98 ± 0.1754.2%−0.01

Table 1 shows that, on All-30, the mean AUROCs across candidate combinations for C-only, F-only, and C+F are 72.00, 72.04, and 71.98, respectively, compared with 71.97 for the original configuration. Overall, the differences are small. When both directions are replaced in C+F, the original configuration differs from the candidate-combination mean by only −0.01 pp. The normalised rank position is calculated as the rank of the original configuration divided by the total number of candidate combinations; a lower value indicates a better relative rank. It describes relative ranking only, not statistical significance or global optimality. These results show that the default configuration was not selected by searching over candidate combinations; rather, it was chosen as a natural, concise setting based on theoretical motivation.

AuthorsOpenReview ↗

Response (3/8)

W1.4: Self-assessment Prompt Robustness

Answer:

We used two semantically equivalent self-assessment prompts:

Concise:

Original question: {question}
Can you answer the original question correctly?
Reply with Yes or No.
Judge:

Detailed:

Original question: {question}
Consider whether you can understand the question, access the necessary information, and carry out any reasoning needed. Then judge whether you can provide a fully correct answer to the original question.
Reply with Yes or No.
Judge:

Table 2. Self-assessment Prompt Robustness (All-30 Macro AUROC, %)

MethodOriginal Self-assessment Prompt (%) ↑Concise Prompt (%) ↑Δ (Concise − Original, pp)Detailed Prompt (%) ↑Δ (Detailed − Original, pp)
P(Yes)63.0864.05+0.9762.53−0.55
IC64.6064.32−0.2863.13−1.47
ADP(Yes)68.6367.83−0.8167.01−1.62
ADP71.9872.17+0.1970.83−1.15

Table 2 shows that ADP achieves the highest AUROC under all three settings—Original, Concise, and Detailed—with scores of 71.98%, 72.17%, and 70.83%, respectively. Relative to Original, Concise improves AUROC by 0.19 pp, whereas Detailed reduces it by 1.15 pp. Even under Detailed, ADP remains 3.82 pp higher than ADP(Yes). In this section, all four likelihood-based baselines (Max-logp, PE, Min-K, and PPL) are computed on the raw query, without the Concise or Detailed self-assessment prompts; their scores therefore remain unchanged.

W1.5: Answer-generation Prompt Robustness

Answer:

We used the following alternative answer-generation prompt:

For the question below, provide only your answer, without any reasoning, justification, or additional text.
Keep your response as brief as possible.
Question: {question}
Answer:

The ADP score is computed before answer generation, so changing the answer-generation prompt does not affect it. Only the generated answers and the resulting correctness labels change; the labels are specific to the generation configuration used for evaluation. ADP aims to estimate the model's intrinsic answerability for the given input, rather than its generation success under that configuration.

Table 3. Answer-generation Prompt Robustness (All-30 Macro AUROC, %)

MethodOriginal Generation Prompt (%) ↑Alternative Generation Prompt (%) ↑Δ (Alternative − Original, pp)
Max-logp53.3153.35+0.04
PE56.9157.17+0.26
Min-K55.9256.09+0.17
PPL55.6555.48−0.17
P(Yes)63.0863.06−0.03
IC64.6064.99+0.39
ADP(Yes)68.6368.97+0.33
ADP71.9872.70+0.73

Table 3 shows that ADP achieves the highest AUROC under both generation prompts, increasing from 71.98% to 72.70%, a gain of 0.73 pp. The changes for the other methods are also small, ranging from −0.17 to +0.39 pp. Overall, changing the answer-generation prompt does not alter the relative performance of the methods, and ADP remains the best-performing method.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. Reviewers found the problem practically relevant for abstention, retrieval and model routing (tqbX, STjG), noted that ADP requires neither training nor task data and applies uniformly to LLMs and VLMs (kkDw), and considered the experimental validation solid (STjG) and the appendices thorough (kkDw). We summarise the main concerns and our responses below; details are given in the individual replies.

1. Role of the three-part decomposition (tqbX, kkDw, STjG)

The decomposition describes possible sources of answerability failure. The three factors are not mutually exclusive, and the decomposition is not intended to assign every incorrect answer to a single factor. It guides the construction of the directions: the five factor-level directions capture specific concepts and support concept-level analysis, while the two integrative directions used in the main configuration, Challenge and Feasibility, were designed from the outset to account for multiple factors. Figure 3 shows that the directions exhibit selective but overlapping responses to factor-relevant variables in settings where one factor dominates (PopQA, GSM8K) or is manipulated (blurred DocVQA). These are dataset-level and controlled analyses; instance-level attribution is left to future work.

2. Robustness of the configuration (tqbX, kkDw)

  • Seed words: across all 225 replacements of Challenge, all 225 replacements of Feasibility and all 50,625 joint replacements, the mean macro AUROC over all 30 model–task combinations (All-30) is 72.00, 72.04 and 71.98, compared with 71.97 for the original configuration.
  • Prompts: under two alternative self-assessment prompts, ADP remains the best method, with its AUROC changing by +0.19 and −1.15 pp relative to the original prompt. Because the ADP score is computed before generation, changing the answer-generation prompt leaves the score unchanged and affects only the generated answers and correctness labels; under an alternative answer-generation prompt, ADP is again the best method, with its AUROC increasing by 0.73 pp. On MathVista and GSM8K, ADP also outperforms the other self-assessment baselines both with and without CoT.
  • Jacobian lens: it outperforms the logit lens under all six pairings of self-assessment and generation prompts (+0.16 to +0.57 pp).

3. Decision-level utility and cost (tqbX, kkDw, STjG)

  • At matched coverage (50%, 70% and 90%), ADP achieves the highest accepted accuracy and the lowest unnecessary rejection rate; under accepted error rate bounds of 5%, 10% and 20%, it achieves the highest coverage.
  • The cascade operating point in the paper was illustrative. Over 20 random 50:50 calibration/test splits, with the threshold selected on the calibration half only, the cascade reaches a mean test accuracy of 69.00%, compared with 69.20% for 8B Only, while 16.84% of questions do not require the 8B model.
  • On 300 randomly sampled MathVista questions, the cascade reduces end-to-end latency by 8.91% and GFLOPs by 8.51% relative to 8B Only, with an accuracy decrease of 0.34 pp. ADP computation itself takes 2.89 ms/query, 0.15% of the latency of 2B inference.
  • Jacobian precomputation is a one-time offline cost per model: across the 12 models, 0.04 to 2.46 GB of storage (2.79% to 5.87% of model size) and 5 minutes to 3 hours 37 minutes on a single RTX PRO 6000.

4. Baselines and GSM8K (kkDw)

  • PE exceeds ADP on GSM8K largely because of a token-length shortcut. After controlling for token length, ADP outperforms all other methods on GSM8K (61.02 vs. 56.41 for PE).
  • Computing the likelihood-based baselines with the self-assessment prompt changes their All-30 macro AUROC by only −0.68 to +0.69 pp.
  • On held-out test sets, a supervised three-layer MLP probe trained on task samples and correctness labels reaches 75.01 All-30 macro AUROC, 4.77 pp above ADP. This quantifies the cost of avoiding target-task training data.

5. Revised PDF

Content changes are marked in blue. Appendix A now states that the 25 directions consist of the 23 directions in the four classes and two reference directions, Yes/No and Positive/Negative; Appendix G now states that the four classes contain only these 23 directions (kkDw). As suggested by Reviewer tqbX, the Limitations paragraph now discusses multiple-choice settings and medical abstention, citing Cocchieri et al. (2026a; 2026b). All additional results reported in our responses will be incorporated into the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.