- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This article introduces Answerability Direction Probing (ADP), a method for estimating a model's answerability without training or generating answers. The authors break down answerability into three factors: input sufficiency, knowledge boundaries, and reasoning ability, and use vocabulary embeddings to construct semantic directions, evaluating them by aligning hidden states across layers with a Jacobian Lens. Experiments cover 12 LLMs/VLMs and 5 datasets, confirming the method's effectiveness and showing its potential for use in model cascades.
Critical strengths
-
The method is practically useful. ADP can estimate answerability before generation without extra training or answer generation, providing an effective scheme for model routing and adaptive reasoning.
-
Experimental validation is solid. It consistently achieves overall performance gains across multiple LLMs and VLMs, and the core design has been verified through directional replacement, robustness analysis, and ablation experiments.
Critical weaknesses
-
The three-factor decomposition lacks direct validation. The final method only uses the combined directions of Challenge and Feasibility, rather than directly modeling the three factors proposed. The existing correlation analysis isn’t enough to prove that these three factors can effectively explain different types of answer failures.
-
The actual efficiency gains aren’t very clear. Although the method only requires one online forward pass, there’s still extra overhead from multi-layer Jacobian precomputation and projection.
-
The model cascade experiments also didn’t report end-to-end latency or real computation costs.
Questions for authors
-
Is it possible to evaluate ADP separately for input insufficiency, knowledge gaps, and reasoning failures, to verify the actual effect of the three-factor decomposition?
-
Could u also add the precomputation cost of the Jacobian, online inference latency, and the actual computational cost savings of model cascading?
3 comments
Response (2/3)
W2: Jacobian Precomputation and Online Estimation Cost
Answer:
The Jacobian lens outperforms the logit lens across all six combinations of self-assessment and generation prompts, with gains of 0.16–0.57 pp. This indicates a small but consistent improvement in AUROC (see Reviewer kkDw, W3, for details). To address the additional computational cost, we report online estimation cost separately from the offline storage and precomputation costs of the Jacobian.
The online costs are broken down as follows:
Table 1. Online Cost of ADP Computation and 2B Self-assessment
| Component | Latency (ms/query) ↓ | Latency Relative to 2B Inference (%) | GFLOPs/query ↓ | GFLOPs Relative to 2B Inference (%) |
|---|---|---|---|---|
| ADP Computation | 2.89 | 0.15% | 0.18 | 0.01% |
| 2B Self-assessment | 34.63 | 1.76% | 992.64 | 70.35% |
| 2B Inference | 1963.79 | 100.00% | 1411.07 | 100.00% |
Table 1 shows that the online overhead of ADP projection is small. The 2B self-assessment requires only a single forward pass, without token-by-token generation; its GFLOPs mainly come from the image input tokens of the VLM.
The offline precomputation and storage costs of the Jacobian are reported below. All times were measured on a single RTX PRO 6000:
Table 2. Jacobian Storage and Precomputation Time (1× RTX PRO 6000)
| Model | Model Size (GB) | Jacobian Storage (GB) | Jacobian / Model Size (%) | Jacobian Precomputation Time (1× RTX PRO 6000) |
|---|---|---|---|---|
| Qwen3-VL-2B | 4.26 | 0.17 | 3.94 | 12 min |
| Qwen3-VL-4B | 8.88 | 0.34 | 3.84 | 22 min |
| Qwen3-VL-8B | 17.53 | 0.87 | 4.98 | 55 min |
| Qwen3-VL-32B | 66.72 | 2.46 | 3.69 | 3 h 37 min |
| Gemma-3-4B | 8.60 | 0.33 | 3.81 | 21 min |
| Gemma-3-12B | 24.38 | 1.03 | 4.23 | 1 h 22 min |
| Llama-3.2-1B | 2.47 | 0.09 | 3.73 | 7 min |
| Llama-3.2-3B | 6.43 | 0.38 | 5.87 | 21 min |
| Phi-4-mini | 7.67 | 0.43 | 5.66 | 25 min |
| Phi-4 | 29.32 | 1.52 | 5.19 | 1 h 38 min |
| Qwen3-0.6B | 1.50 | 0.04 | 2.79 | 5 min |
| Qwen3-8B | 16.38 | 0.87 | 5.33 | 56 min |
Table 2 shows that, across the 12 models, Jacobian storage ranges from 0.04 to 2.46 GB, accounting for approximately 2.79%–5.87% of model size. Jacobian precomputation takes between 5 minutes and 3 hours 37 minutes. This is a one-time offline construction cost per model, separate from the online estimation cost for each query.
Response (3/3)
W3: End-to-end Cost of the Model Cascade
Answer:
We randomly sampled 300 questions from MathVista and measured the full end-to-end latency and GFLOPs of 2B Only, ADP Cascade, and 8B Only.
Table 3. End-to-end Cascade Cost on 300 MathVista Samples
| Method | End-to-end Latency (ms/query) ↓ | Latency Change Relative to 8B Only (%) | GFLOPs/query ↓ | GFLOPs Change Relative to 8B Only (%) | Accuracy (%) ↑ | Accuracy Difference (Current Method − 8B Only, pp) |
|---|---|---|---|---|---|---|
| 2B Only | 1963.79 | −40.34% | 1411.07 | −76.51% | 52.70 | −15.21 pp |
| ADP Cascade | 2998.14 | −8.91% | 5497.26 | −8.51% | 67.57 | −0.34 pp |
| 8B Only | 3291.55 | 0.00% | 6008.32 | 0.00% | 67.91 | 0.00 pp |
Table 3 shows that, compared with using 8B directly, ADP Cascade reduces end-to-end latency by 8.91% and GFLOPs by 8.51%, with an accuracy decrease of only 0.34 pp. This result includes both the 2B forward pass and ADP computation. For text-only inputs, or in settings where the model uses a higher thinking level and generates longer answers, the cascade is expected to save more computation.
Q1: Factor-specific Evaluation
Answer:
Comprehensive instance-level annotations covering all three factors are not available, but our tasks include settings in which one factor dominates or is manipulated. PopQA failures mainly reflect missing knowledge: ADP reaches 75.87 macro AUROC and is best on all six LLMs. GSM8K gives all numbers in the question, so failures mainly reflect reasoning: ADP reaches 62.76 against 63.74 for PE, whose advantage comes largely from a question-length shortcut; after controlling for length, ADP reaches 61.02 against 56.41 for PE (see our response to Reviewer kkDw, W4). On DocVQA, controlled blur manipulates input adequacy with the question and model fixed (Figure 3). In addition, the LLM judge used for labelling (Appendix B) first checks input adequacy and marks inputs that cannot determine the answer (6.1% of OK-VQA, 1.2% of MathVista and 0.1% of DocVQA). These are dataset-level and controlled analyses rather than instance-level attribution, which, as discussed in W1, we leave to future work.
Q2: Precomputation, Online Cost, and Cascade Savings
Answer:
Jacobian storage and precomputation, together with online estimation costs, are reported in W2. The cascade's end-to-end latency and computational savings are reported in W3.
Response (1/3)
Dear Reviewer STjG,
Thank you for your detailed comments and helpful feedback. We appreciate the time and effort you invested in reviewing our paper. We address each point below and will incorporate the new results and clarifications into the revised manuscript.
W1: Three-part Decomposition and Factor-level Validation
Answer:
The three-part decomposition provides theoretical guidance for constructing answerability-related concepts. A factor is a theoretical source of failure, a concept is a specific semantic attribute, and a concept direction reads out a representational signal associated with that attribute rather than measuring the factor as a whole. Factors and concepts do not have a one-to-one correspondence: one factor may be associated with multiple concepts, and one concept may be related to multiple factors. The five factor-level directions are specific concept directions constructed with the corresponding factors as theoretical starting points. Each focuses on a particular concept mainly relevant to the factor that motivated its construction. They are used for concept-level analysis rather than as direct measures of the three factors. ADP aims to estimate overall answerability rather than output three separate factor-level scores.
The main experiments use two integrative directions, Challenge and Feasibility, as a theoretically motivated general-purpose default configuration. They were designed from the outset to account for multiple factors and to provide complementary perspectives: Challenge takes the task perspective, whereas Feasibility takes the perspective of the current model–input pair. Challenge describes the overall demands of a query, including the adequacy of the input and the knowledge and reasoning it requires. Feasibility describes how feasible it is for the current model to answer the current input correctly. It primarily reflects Knowledge Boundary and Reasoning Capability, while also being affected by Input Adequacy. The two directions do not correspond one-to-one to the three factors, nor are they a direct mathematical combination of the three factors.
Figure 3 shows that the factor-level directions exhibit selective but overlapping responses to factor-relevant variables. The main experiments support the effectiveness of Challenge + Feasibility for estimating overall answerability. These results support the intended interpretations of the directions, but are insufficient to establish that the three factors have been directly validated or that they can separately explain different types of answer failures.
ADP does not currently identify which of the three factors is responsible for a given failure. Failure sources are not mutually exclusive, and a single error may involve multiple factors. Moreover, because factors and concepts do not have a one-to-one correspondence, a single ADP score cannot uniquely attribute a failure to a particular factor. In principle, the joint distribution and response patterns across a larger set of concept directions may reflect the relative strengths of the three factors and help attribute failures to individual factors; we leave this for future work.