acceptodds
Under review as a conference paper at ICLR 2027

Attention as a Markov Random Field

Abstract

Reliably detecting when large language models (LLMs) hallucinate remains an open challenge. Existing uncertainty estimators either require costly multi-sample generation or supervised calibration data, limiting their practical deployment. We introduce the Attention Bethe Score (ABS), a zero-shot, single-pass uncertainty metric grounded in variational inference over probabilistic graphical models. Our key insight is to reinterpret the attention mechanism of a frozen Transformer as the structure of a Markov Random Field: token positions become nodes, attention weights define edges with attention-modulated Potts couplings, and hidden-state projections yield unary potentials. Running loopy belief propagation on this graph, we compute the Bethe free energy as a principled measure of internal representational frustration-the degree to which strongly attended positions hold inconsistent beliefs. We prove that, under a contraction condition satisfied in typical operating regimes, the Bethe free energy monotonically tracks projected frustration, and we validate this relationship empirically (Spearman p > 0.85). Across eight benchmarks spanning open-domain QA, fact verification, hallucination detection, and summarization, ABS achieves aggregate AUROC of 0.788 (LLaMA-3.1-8B) and 0.806 (Qwen-2.5- 7B), surpassing all baselines including multi-sample Semantic Entropy and supervised Internal Probes-while adding fewer than four milliseconds and less than three percent computational overhead to a standard forward pass.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Hallucination Detection, Uncertainty Estimation, Markov Random Fields, Bethe Free Energy, Belief Propagation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_kdo8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that 1) hallucinations are due to "representational frustration" (attention-weighted disagreement between directions). 2) We can track this using the Bethe free energy of an MRF representation of the attention. 3) Therefore, we can use that BFE (or actually, a weighted combination of layer wise BFEs) as a UQ metric. They show that it gives good AUROC and ECE (when translated to a probability)

My main concern is that, the results don't really agree with this story. The attention-weighted disagreement can be directly tracked, and it does worse than their score. So - what is the ABS adding that the attention-weighted disagreement misses, and why?

My secondary concern is that it isn't clear how the correctness score for the AUROC is calculated. Santilli et al (2025) show that many ways of calculating correctness are correlated with response length, leading to errors in ranking methods. I would expect that the ABS score would also be highly dependent on length (longer q/a -> more opportunities for disagreement), so I would like to verify how the AUROC is calculated (it isn't specified in the appendix).

Third, a conceptual question. You are treating the attention as a Markov random field, but it isn't -- it's a DAG. Why throw away the direction/causal structure? Isn't this likely to explain away directional disagreement?

Finally, I think the paper could benefit from some qualitative analysis, and potentially also some datasets with "different" types of uncertainty (ambiguity, math reasoning). It would be really nice to see visual examples of the sort of disagreement/"frustration" this method is designed to detect. The reason I suggest datasets with different types of uncertainty, is it's not clear to me that trivia/recall type questions would naturally have this sort of frustration. TruthfulQA (which is included in the appendix but not as part of the main multi-model comparison) seems more likely to have it (as the questions have somewhat dual meanings), as do ambiguous questions.

While I have some concerns and would like to better understand what is going on - the method seems to do well. I'd be open to raising my score if I had some stronger insight into why ABS works better than raw frustration, and if I could see how the AUROC is calculated.

Santilli, Andrea, et al. "Revisiting uncertainty quantification evaluation in language models: Spurious interactions with response length bias results." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2025.

Critical strengths
  • The method appears to perform better than other UQ methods on a variety of QA tasks.
  • AFAIK the idea of "frustration" is new and interesting to me.
Critical weaknesses
  • Unclear how AUROC is calculated, which may introduce confounding in results.
  • Unclear conceptually why ABS outperforms raw frustration
Questions for authors
  • How is correctness calculated for AUROC?
  • Why do you believe ABS outperforms raw frustration?
  • Would you be willing to add any qualitative analyses or examples of frustration?
Reviewer_BKSxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the Attention Bethe Score (ABS), a zero-shot uncertainty metric for large language model responses that requires no training or calibration labels. It treats token positions as nodes in a Markov random field, uses attention weights to define edges, and derives unary potentials from hidden-state projections. Loopy belief propagation then yields the Bethe free energy, which measures internal representational frustration between strongly connected positions. The authors report aggregate AUROC scores of 0.788 and 0.806 across five core benchmarks, surpassing all nine baselines while adding less than 3% wall-clock overhead.

Critical strengths
  1. Modeling attention as a Markov random field and using the Bethe free energy to capture inconsistencies between representations is an insightful approach.

  2. The raw ABS score requires neither learned parameters nor labeled calibration data, making the proposed approach appealing as a zero-shot uncertainty measure.

  3. The evaluation covers two models and five core benchmarks. The ablation studies also help clarify the contributions of the projection, Potts couplings, and belief propagation.

Critical weaknesses
  1. I am concerned that Lemma 1 does not follow from its assumptions. The projected frustration index sums the contributions from individual edges, while the pairwise energy weights each contribution by its edge-specific rho_{ij}. A larger unweighted sum need not produce a larger weighted sum. Since Proposition 2 uses this step, the current proof does not establish the bound in Eq. (17).

  2. The contraction bound in Assumption 2 is difficult to reconcile with the damped update in Eq. (8). The update retains a fraction 1-alpha of the previous message; with alpha=0.5, half of that message remains. It is therefore unclear how the stated update yields a contraction coefficient of approximately 1.5 × 10^{-4}. This matters because that coefficient is used to justify running only three BP iterations.

These two issues are the main reasons for my rejection score, as they are fundamental to the validity of the paper’s theoretical claims. If the authors can resolve both with a corrected proof or clearly stated additional assumptions, I would raise my score.

In addition, I have a few questions about the experiments.

  1. I could not reconcile the “Reasoning” result in Table 5 with its definition in Appendix C.5. The appendix describes this stratum as the incorrect subset of GSM8K, in which the model’s final numeric answer differs from the gold answer. As written, the subset contains no correct responses, so its AUROC is undefined. The authors should explain which correct responses were included and how this stratum was constructed.

  2. The efficiency analysis needs a clearer accounting of attention extraction. Algorithm 1 extracts attention matrices and hidden states through forward hooks, but the peak additional-memory estimate in Eq. (19) lists only unary potentials, messages, and the projection matrix. Does Table 7 include the cost of producing and extracting the attention matrices? The authors should also clarify how its “forward-pass latency” relates to the stated greedy-decoding baseline.

Questions for authors

See Critical Weaknesses.

Reviewer_GUTqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the Attention Bethe Score, a zero-shot uncertainty estimator for detecting incorrect LLM responses. ABS constructs an undirected Markov Random Field from attention weights, defines unary potentials using randomly projected hidden states, and uses attention-modulated Potts couplings to encode representational disagreement. Experiments on five core benchmarks using LLaMA-3.1-8B and Qwen-2.5-7B report aggregate AUROC values of 0.788 and 0.806, respectively, outperforming nine baselines. The paper also presents a theoretical connection between free energy and representational frustration and reports inference overhead below 3%. The idea is clean and efficient, but novelty, theory, and empirical coverage need strengthening.

Critical strengths

1.The proposed graphical-model perspective is interesting. Reinterpreting attention as the edge structure of an MRF and using the Bethe free energy as an uncertainty signal is a creative formulation that goes beyond standard token-probability or attention-entropy heuristics. 2.The reported improvements are consistent across five core benchmarks and two model families. ABS improves aggregate AUROC over Semantic Entropy by 1.8 percentage points on LLaMA and 1.9 points on Qwen.

Critical weaknesses

1.The central hypothesis and theoretical justification are insufficiently established. High representational frustration need not correspond to factual errors, and factual errors need not exhibit high frustration. The theory mainly relates ABS to the proposed frustration index rather than to hallucination, while several modeling choices remain heuristic. 2.The evaluation protocol and label construction are unclear. The paper does not clearly distinguish between fixed annotated responses and newly generated outputs. The construction of the GSM8K evaluation set is also unclear, particularly the positive and negative examples used for AUROC. Potential label noise in TriviaQA, NQ-Open, XSum, and HaluEval should also be acknowledged.

Questions for authors
  1. Does Lemma 1 require componentwise increases in edge frustration? If so, how does this change Proposition 2 and its applicability to comparisons between responses?
  2. Are HaluEval and XSum evaluated on fixed annotated candidates or on newly generated responses? If responses are newly generated, how are they labeled? Please also specify which correct and incorrect examples are included in the GSM8K reasoning-stratum AUROC evaluation. Please also report performance by error type, including memorized misinformation, reasoning errors, and context-faithfulness errors.
  3. Please clarify whether Platt scaling uses labeled validation data. If so, please separate the zero-shot ABS score and abstention rule from the supervised calibration procedure used for ECE. If not, please explain how the Platt scaling parameters are estimated. Please also report the number of validation examples used for Platt scaling and the sensitivity of ECE to the calibration-set size.
  4. Please clarify the exact inference pipeline. Is ABS computed during generation from saved activations, or by an additional forward pass over the completed prompt–response sequence? Does the reported latency include attention extraction, attention-buffer memory, and any loss of FlashAttention or KV-cache efficiency? Please also report the attention backend, batch size, and generated response length used in Table 7.
  5. For QA tasks, please analyze the impact of computing ABS only on response tokens. How does performance vary with answer length, particularly for one-token or very short answers? Does including prompt-response edges improve or hurt performance?
  6. Please provide more details for the multi-sample baselines, including the number of samples, temperature, semantic clustering method, and total compute budget. Please also clarify the training setup and feature representation used for the Internal Probe. Comparisons with simpler graph- or attention-based scores, such as graph Laplacian energy, attention-weighted cosine distance, head-specific disagreement, or hidden-state variance, would help isolate the contribution of loopy BP and Bethe free energy.
  7. Please resolve the aggregate-reporting inconsistency in ablation tables. If XSum is excluded from some aggregate AUROC values, this should be clearly stated. Please also provide confidence intervals for the main AUROC/AUPRC results, especially for smaller datasets such as XSum.
Reviewer_Zg7nOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper reintrepretates transformer attention as Markov Random Field and thereby enables uncertainty quantification for LLM in zero-shot, single-pass manner. Specifically, this work treats the token positions as nodes, attention weights as edges with Potts coupling, and hidden state as unary potentials. Following the graph construction, loopy belief propagation is run on the obtained graph, and results in the Bethe free energy. This energy values is treated as LLM uncertainty and utilized for application such as hallucination detection. The core point of this work is that when highly attended token position shows strong inconsisentency, the model is uncertain. This work conducts evaluation across 8 different benchmarks and yield consistent improvement over 9 baseline methods.

Critical strengths
  1. Thorough evaluation. This work conducts comprehensive evaluation across 8 benchmarks and on 2 models, and achieves superior performance over 9 baseline methods. It also presents necessary ablation studies and quantitative analysis to show the postive effect of it proposed designs.
  2. Solid theoretical foundation. This work treats attention structure as Markov Random Field, and adopt mechanism of MRF to achieve proper quantification of LLM uncertainty. The defination and derivation process is rigor and reasonable, which provides adequate support for the provided methods.
Critical weaknesses
  1. Unavoidable limiation. The proposed method requires direct access to model weight. As a result, it can not quantify uncertainty for any closed-source models. In contrast, method such as Semantic Entropy provides a clean, unified manner for estimating uncertainty for both closed-source and open-source models. This drawback greatly constrains the value of the proposed method, since most emphasis is put on closed-source models and studying security of their reponses.
  2. Need better figure illustration. Fig. 1 presents the overview of the proposed pipeline, but it is quite hard to follow and understand clearly. No matter how complex the proposed theory, there should be simple and straightforward way to present it clearly. The author could add a concise figure before original Fig. 1 to present the core pipeline of proposed method clearly, and then utilize original Fig. 1 to present corresponding details.
  3. Lack of qualitative analysis. The author is recommeneded to provide more qualitative analysis to show the effectiveness, internal process of the proposed method, such as qualitative cases of successful hallucination detection, concise illustration of constructed graph, visulization of internal process, etc.
Questions for authors

See Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.