acceptodds
Under review as a conference paper at ICLR 2027

Understanding the Knowledge–Behavior Gap in Language Models through Answer Measurement

14 views

Abstract

The knowledge-behavior gap refers to the discrepancy between the answers a language model's internal representations support and the answers it actually gives. Prior work identifies this gap by comparing what probes recover from hidden states with what models answer, and shows that probing can partly mitigate it. However, how the model's answering process loses internal answer information remains unclear. We develop a measurement theory of answer support, the total probability mass of continuations expressing the same answer. Answering estimates this support through probability readings and commits to the answer with the highest reading. Two measurement properties are key: the measurement unit, which determines whether readings concern complete answers or individual expressions, and the measurement scale, which determines whether they share a common reference. Mismatch can distort support ordering, preventing supported answers from becoming commitments. Accordingly, we construct native measurements that vary in measurement unit and scale. We further prove that, when answer support is measurable from complete-answer representations, a shared map places candidates formed along separate generation paths on a common answer-level scale. Across four frozen models, aligning either property improves answering accuracy and robustness to answer rewording; aligning both yields the highest accuracy throughout and the lowest flip rate in most settings. Supervised by the model's own support ordering, shared linear maps improve native measurements in nearly all settings and generalize even when trained only on support relations among incorrect answers. Used as generation feedback, the learned readings also improve free-form answers. Measuring answer support and converting it into commitments thus offers a route to understanding and narrowing the knowledge-behavior gap.

Primary area:
interpretability and explainable AI
Keywords:
large language models, knowledge-behavior gap, answer measurement, measurement theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iCxXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work aims to provide a framework of formalism for scoring sequential text-model’s answers to questions, both in terms of semantic equivalence and score scale.

Critical strengths
  1. The idea of extending the scoring to free generation is interesting
  2. The figures and tables are very clear
Critical weaknesses
  1. The primary motivation provided for this work is to find an explanation for the well-recorded gap in accuracy when the LLM is used to generate an answer compared to when an answer is probed from the hidden state more directly. The conclusion of the work seems to be that if the list of clustered answer candidates is provided then there is no gap, which is somewhat in contradiction to the literature which mostly shows that probes trained on exactly such a representation (A_c) can improve performance.
  2. There are several incoherences in the flow of the methodology. Equation (6) is never really used or referred to afterwards. The measurement scale criterion is meaningless: as it is written, if I apply some arbitrary increasing function T to the log-scores then I achieve common scale. I can’t see why this is necessary if I already have an order preserving measurement (eq. 5). We never see a function notated T again in the main text. The authors chose to refer to the probability of an answer cluster P_q(A) as “answer support”, which is confusing since the “support” induced by a distribution P(X) typically means the set of values that X can take for which P(X) is defined. Equation 1, in particular the left expression for u_q = … is impossible to understand as the # notation is not defined. For this equation, an entire probability textbook is cited.
  3. Novelty is weak: both the notion of candidate-ordering and semantic clustering have already been established. A proof for the existence of a probe (Proposition 3.1) is not really necessary in my view.
  4. Impact: the claimed contributions are not particularly impactful:
    • “we develop a measurement theory […]”: the proposed measurement theory does not provide any real new insight I
    • “Through four native […]”: in plain language, the authors make this contribution by demonstrating that including both the set of answer choices and prompting to avoid answer equivalence issues improves performance, something that is both evident and well-known
    • “We develop a shared-map construction […]”: in plain language, the authors implement a probe and show that it can improve on bad prompting strategies (although not enough to beat good prompting strategies).

Based on these weaknesses, I am rating the paper a clear rejection.

Questions for authors
  1. Figure 1: for T_s, do you count commitment flips between two semantically equivalent answers?
Reviewer_AyZfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates the gap between internal answer support and model behavior and develops a theoretical framework of answer measurement. Experiments evaluate measurement interfaces and shared linear readouts for multiple-choice answering, rewording robustness, and guided free-form generation.

Critical strengths
  1. The paper proposed a question concerning the gap between internal answer support and model behavior.
  2. The paper proposed a mathematical framework formulating this gap and provides a principled approach to analyzing how measurement units and comparison scales can distort answer selection.
Critical weaknesses
  1. Gap between the theoretical quantity and its experimental measurement. The theory defines answer support by aggregating the probability mass of all continuations expressing the same answer. However, the main MCQ experiments primarily use one expression per candidate without establishing that label probabilities preserve the original support ordering. Evaluating multiple realizations of every candidate and comparing interface scores against independently estimated answer support would provide more direct evidence for the proposed account.

  2. Unverified assumptions underlying the mechanistic interpretation. The theoretical guarantees rely on a separable label-measurement assumption and the measurability of answer support from candidate representations, neither of which is directly validated. Behavioral gains do not establish support recovery, and AC-supervised maps cannot independently validate AC as a support measure. At minimum, the paper should assess the relevant assumptions and demonstrate that the improvements correspond to better recovery of independently estimated answer-support orderings.

Questions for authors

Please find the weakness above.

Reviewer_TmTGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper develops a theoretic framework for formalizing and measuring the knowledge-behavior gap in LLMs, a well-known issue when information that a language model internally possesses does not make into its final answer. The authors provide probabilistic formalization of the answer support and characterize the model's answering procedure as a measurement of this support. Based on it, they construct "four native measurements", proper alignment of which should improve models' accuracy and stability to rewording. The authors conduct experiments on several pre-trained LLMs on MCQA tasks and provide empirical evidence for their arguments. They further demonstrate that a shared linear map learned from frozen representations can be used to modestly improve free-form answering on questions from GSM8K and MATH500.

Critical strengths

S1) The theoretical part of this work is quite interesting; it provides a well-developed mathematical background.

Critical weaknesses

W1) The practical significance of the results is rather limited.

  • This work considers primarily MCQA setup (with addition of short-form math questions from MATH500/GSM8K in section 5.3, which is also reformulated in MCQA-similar format by repetitive sampling). Since modern top-performing LLMs leverage the reasoning to enhance their capabilities, the setup with immediate forcing of the answer limits the applicability of this paper's findings. The proposed approach (map-guided generation) in its current form is very hard to implement in cases when the answer cannot be expressed in a short form (or when the entire solution as "path to the answer" matters).

  • The motivation behind "four native measurements" and their connection to the theoretical background developed in Section 3 are not clear from the text. Why exactly the balanced ACA_C is used as the ground-truth ordering for training the shared map, but not other measures? In ACA_C and TST_S setups model receives different amount of information in the input, therefore gap AC−TSA_C - T_S referenced in Table 2 mixes up measurement alignment and information form extra context, and is also contaminated with random guessing chance probability.

  • Overall, the claim of "narrowing the knowledge-behavior gap" from the abstract is not supported by any clear quantitative indicator.

W2) The experimental part of this work is very limited.

  • Only 300-500 questions from each dataset (MMLU/ARC-C/GSM8K) were used, not the whole datasets. It is unclear why this downsampling was performed, and how well these subsets represent whole datasets.

  • Very small range of models (only 4, and 2 of them are just the different sized versions of Qwen2.5) was covered in experiments, and only base models -- no Thinking- or Instruct-tuned models were used.

  • None of the experiments provide comparison against existing competitive "knowledge-gap-closing" methods (e.g., Self-Consistency, or some MCQA-bias-mitigation approaches like PRIDE (Large Language Models are not Robust Multiple Choice Selectors, Zheng et al., ICLR 2024)).

  • Results of the free-form answer generation experiment are given only for two models, and, in half of the cases, the accuracy of the map-guided solution is within standard deviation from the performance of the initial model.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.