acceptodds
Withdrawn from ICLR 2027

Beyond Atomic Claims: Probabilistic Logical Coherence for Long-Form LLM Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-form factuality assessment typically follows a decompose-then-verify paradigm: a response is decomposed into atomic claims, each claim is verified against evidence, and the resulting judgments are aggregated into a final score. While effective, this approach evaluates claims in isolation and cannot assess whether they form a coherent argument. We formalize logical coherence as a property of the joint distribution over claim truth values induced by a Markov random field whose factors encode discourse relations expressed within the response. To bridge discourse and inference, we propose a two-level taxonomy that maps twelve PDTB discourse senses onto five inferential couplings. We also introduce LoCoBench, a benchmark that holds claims fixed while varying only their relational structure. Using gold relation graphs, our logical coherence scores satisfy of 202 ordering constraints, compared with – when relations are extracted automatically, identifying relation extraction as the primary bottleneck. Across various baselines, our method is the only one that both orders responses by coherence and remains invariant under meaning-preserving edits. Furthermore, it localizes incoherence through claim-level posteriors rather than just detecting it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
logical coherence, language models, probabilistic inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jrbMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new way to evaluate text coherence by separating it from factuality. Instead of just checking if individual claims in a response are true or false, the authors model the logical relations between those claims. They use a probabilistic graphical model (a Markov Random Field) where claims are nodes and discourse relations (like entailment, contradiction, etc.) are edges with specific mathematical rules. They also build a synthetic benchmark called LOCOBENCH, which creates "ladders" of responses where the claims stay the same but the logical structure changes (e.g., resolving a contradiction or shuffling sentences). The results show that when given a correct graph of relations, their model works well and ranks responses properly. However, when they try to automatically extract these relations from the text using LLMs, performance drops significantly, showing that extracting relations is currently the main bottleneck.

Critical strengths
  1. Separating Factuality from Coherence: The paper does a great job of showing that these are two different things. Modeling coherence as a graph of relations between claims is a smart way to move beyond just counting true/false facts.

  2. The LOCOBENCH Benchmark: Building "ladders" where only the logical structure changes (while keeping the text content identical) is a very clean way to test these metrics. The human study (even with its small size) helps validate that these ladders make sense to real readers.

Critical weaknesses
  1. The End-to-End System Doesn't Work Well: The paper essentially proves its own current pipeline is not ready for real use. When the authors use LLMs to automatically extract the relations from the text, the performance tanks, and the model fails all the invariance tests (meaning it changes its score even when the text's meaning hasn't changed). This makes the practical contribution limited.

  2. The model ignores sentence order: The human study shows that readers really care about sentence order and flow (anaphora). The model completely ignores this because it only looks at the relation graph. While mathematically elegant, calling this perfect "invariance" a success is misleading—it's actually a limitation because humans don't see it that way.

Questions for authors
  1. The paper shows that LLM-based relation extraction is the main bottleneck. Do you have any ideas on how to make the model more robust to noisy or missing relations, so it doesn't completely fail on end-to-end tasks?

  2. Since your human study shows that shuffling sentences breaks anaphora (and humans hate this), how could you incorporate sentence order or discourse flow into your graph without losing the clean mathematical properties of the MRF?

  3. You mention that removing factuality priors increases the model's discriminative power but lowers interpretability. Which setting would you actually recommend for a real-world evaluation pipeline?

Reviewer_AMB5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a probabilistic framework for evaluating the logical coherence of long-form LLM responses. The authors utilize atomic claims as binary variables in a pairwise Markov random field, and pairwise potentials encoding discourse relations between claims. They present a two-level taxonomy mapping 12 discourse senses to 5 inferential coupling types. The paper develops several logical coherence scores, and proposes a pipeline that incorporates external factuality estimates as claim-level priors. The paper also introduces LoCoBench, a synthetic benchmark comprising 70 responses organized into 14 coherence ladders. Overall, the paper presents an interesting structural approach to coherence evaluation, but concerns are on the benchmark validity, and practical effectiveness of the end-to-end method.

Critical strengths
  1. A meaningful problem formulation with interpretable probabilistic structure. The paper targets a genuine limitation of atomic-claim factuality evaluation: individually verifiable statements can participate in inconsistent or unsupported arguments. Representing claim dependencies through an explicit probabilistic graphical model is a principled and potentially useful direction. The distinction between discourse senses and inferential couplings is also conceptually appealing, and posterior marginals provide more interpretable diagnostics than a single holistic LLM-judge score.

  2. Controlled evaluation that separates relation extraction from downstream inference. LoCoBench provides a useful experimental design for investigating the effect of relational structure while controlling claim content. In particular, the separation between gold and automatically extracted relation graphs makes it possible to identify important pipeline failure modes. The reported degradation under mined relations is informative and highlights a practically relevant research challenge.

Critical weaknesses
  1. The stated monotonicity property does not hold in general. R3 requires that removing a conflict cannot decrease coherence, and Table 6 claims that LCSmm satisfies it. Consider priors π = (0.6, 0.6, 0.8), a CONTRADICTION relation from a1 to a2 with strength 0.9, and an EXCLUSIVE relation between a2 and a3 with strength 0.99. Using the paper’s potentials and exact enumeration, LCSmm is approximately 0.55739. Removing the CONTRADICTION relation lowers it to 0.53636. All three priors exceed 0.5. Appendix D, Lemma 1 also claims that the MAP-world mass Zmin cannot increase with conflict strength. But the CONTRADICTION potential for assignment (1,0) is p, which increases with p. With priors (0.9, 0.01), Zmin rises from 0.4455 at p = 0.5 to 0.8019 at p = 0.9. These claims help justify the scores, so they need correction or clearly stated conditions under which they hold.

  2. The evaluation does not establish reliable end-to-end coherence assessment. Constraint satisfaction drops from 82.7% with gold graphs to 34.7–37.1% with extracted relations, and every extracted-graph configuration scores 0/20 on the invariance tests. Since relation extraction is essential to the proposed application, this substantially limits the practical result. Some benchmark assumptions are also questionable. In the ORDER study, none of the eight human judgments considered the reordered responses equivalent. Annotators reported broken references and unnatural information order. Perfect model invariance on these examples therefore does not clearly indicate better coherence assessment. The 250 natural responses have no human coherence annotations, so differences between numerical scores say little about their correctness. The small human comparison is also inconclusive: the better extracted-graph LCSmm configuration agrees with the majority on 7/12 evaluable pairs, versus 8/12 for the direct LLM judge.

Questions for authors

N/A

Reviewer_5QrKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The main observation motivating this paper is that each atomic fact in an output can be true, but that does not guarantee the logical coherence of the output as a whole. Therefore, they argue for a specialized approach for evaluating coherence. It formalizes logical coherence as a property of the joint distribution over claim truth values induced by a Markov random field whose factors encode discourse relations. It also introduces LOCOBENCH, a benchmark that holds claims fixed while varying only their relational structure.

Critical strengths
  1. The main idea behind the paper is sound -- each atomic fact in an output can be true, but that does not guarantee the logical coherence of the output as a whole. However, I am not convinced that the main experiments establish the performance superiority of the proposed approach.
Critical weaknesses
  1. While I agree with the main assertion in strength 1, it is worth questioning how often, in a realistic dataset, does the aggregated individual factuality contradict the overall factuality. This would help establish if the complexity introduced in this methodology is justified by the potential improvements.

  2. The construction of LocoBench is extremely confusing. The benchmark description introduces a lot of terminology (ladders, rungs, etc.) that is disconnected with the rest of the paper. The resulting dataset if quite small (70 datapoints).

  3. Most crucially, the results in table 1 show that the proposed method only outperforms baselines when usnig the gold relation structure. However, when mining this structure (as is realistic), it does not consistently outperform the baselines.

  4. I am not convinced by the experiments in Table 2. Instead of measuring the correctness of factuality or coherence labels produced by the model, it measures mean, sd, and d (number of distinct numerical scores) for the outputs. The paper states "Since these datasets do not provide coherence labels, we evaluate a necessary condition for usefulness: the ability of a measure to assign distinct scores to different responses." This is measured by d. However, just because a model produces distinct coherence scores does not guarantee that it produces correct scores. A random number generator is a simple baseline that would beat all coherence models, under this metric.

  5. The paper reports inter-annotator agreement based on annotation of only 14 responses. The paper does not report any confidence interval.

  6. The paper lists that existing factuality benchmarks do not contain the necessary coherence labels to support evaluation of their method. However, there does exists coherence datasets that can be used to compare their method against: See https://arxiv.org/abs/2205.09641. Overall, evauation over a 70 datapoint LocoBench dataset is insufficient to support the central claims.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.