- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a new way to evaluate text coherence by separating it from factuality. Instead of just checking if individual claims in a response are true or false, the authors model the logical relations between those claims. They use a probabilistic graphical model (a Markov Random Field) where claims are nodes and discourse relations (like entailment, contradiction, etc.) are edges with specific mathematical rules. They also build a synthetic benchmark called LOCOBENCH, which creates "ladders" of responses where the claims stay the same but the logical structure changes (e.g., resolving a contradiction or shuffling sentences). The results show that when given a correct graph of relations, their model works well and ranks responses properly. However, when they try to automatically extract these relations from the text using LLMs, performance drops significantly, showing that extracting relations is currently the main bottleneck.
Critical strengths
-
Separating Factuality from Coherence: The paper does a great job of showing that these are two different things. Modeling coherence as a graph of relations between claims is a smart way to move beyond just counting true/false facts.
-
The LOCOBENCH Benchmark: Building "ladders" where only the logical structure changes (while keeping the text content identical) is a very clean way to test these metrics. The human study (even with its small size) helps validate that these ladders make sense to real readers.
Critical weaknesses
-
The End-to-End System Doesn't Work Well: The paper essentially proves its own current pipeline is not ready for real use. When the authors use LLMs to automatically extract the relations from the text, the performance tanks, and the model fails all the invariance tests (meaning it changes its score even when the text's meaning hasn't changed). This makes the practical contribution limited.
-
The model ignores sentence order: The human study shows that readers really care about sentence order and flow (anaphora). The model completely ignores this because it only looks at the relation graph. While mathematically elegant, calling this perfect "invariance" a success is misleading—it's actually a limitation because humans don't see it that way.
Questions for authors
-
The paper shows that LLM-based relation extraction is the main bottleneck. Do you have any ideas on how to make the model more robust to noisy or missing relations, so it doesn't completely fail on end-to-end tasks?
-
Since your human study shows that shuffling sentences breaks anaphora (and humans hate this), how could you incorporate sentence order or discourse flow into your graph without losing the clean mathematical properties of the MRF?
-
You mention that removing factuality priors increases the model's discriminative power but lowers interpretability. Which setting would you actually recommend for a real-world evaluation pipeline?