- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a method for detecting incorrect claims in LLM-generated text while controlling the false negative rate (FNR). The main idea is to improve existing factuality scores by using NLI relationships between claims, such as entailment, contradiction, and neutrality. The authors first define a posterior score based on an independent Bernoulli working model and then use star approximation and belief propagation to reduce computational cost. They apply conformal risk control (CRC) to provide finite-sample marginal FNR guarantees. Experiments on four datasets show that the proposed method can retain more correct claims while improving detection accuracy compared to existing methods.
Critical strengths
S1. The paper addresses an important and practical problem.
Detecting incorrect claims while retaining as much correct information as possible is an important problem. Since FNR can be trivially controlled by removing all claims, I think it is natural to consider retention alongside FNR control.
S2. The idea of NLI-based score refinement is intuitive.
The idea of updating factuality scores using logical relationships between claims, rather than evaluating each claim independently, is interesting. Formulating this idea through a probabilistic model and considering multiple approximation methods are also meaningful aspects of the work.
Critical weaknesses
W1. The theoretical explanation connecting NLI refinement to retention improvement is insufficient.
- While the intuition behind updating factuality scores using NLI relationships is understandable, it is unclear whether these updates actually produce better scores and lead to improved retention.
- The FNR guarantee in Theorem 1 holds regardless of the quality of the scoring function under appropriate conditions. Therefore, I believe the main contribution of this paper lies in improving score quality and retention through NLI information. However, Theorem 2 only provides an upper bound for the star posterior and does not directly explain these improvements.
- Of course, it would be difficult to expect a guarantee that NLI always improves performance. Nevertheless, analyzing the conditions under which improvements can be expected would help clarify the contribution. For example, Noh et al. (2026), although working in a different setting, analyze the relationship between score estimation error and the retention gap. I wonder whether a similar connection could be established between the accuracy of NLI information, the effect of refinement, and retention in this paper.
W2. The experimental contribution of the NLI component is not sufficiently demonstrated.
- Although several scoring methods are compared, it is difficult to understand which NLI relationships actually contribute to performance improvements. For example, the retention of the Full posterior increases from 0.3500 to 0.4792 on the authors' constructed dataset (Table 6), whereas the improvement on WiCE is much smaller, from 0.2023 to 0.2093. There is insufficient analysis explaining why these differences occur.
- It would be helpful to report the proportions of entailment, contradiction, and neutrality relationships, as well as NLI prediction accuracy, for each dataset. I also think ablation studies comparing performance when NLI relationships are removed or made inaccurate are needed.
W3. The experimental setting and sensitivity analysis are insufficient.
- Section 3.1 defines a training/calibration split, but Appendix A.6 only reports a Train/Test split. Since pretrained VERA and DeBERTa-v3 models are used, it is unclear how the training data are specifically used and how calibration is performed.
- Another limitation is that all main experiments are conducted at . Comparing the FNR–retention trade-off across different values of would help clarify the method's practical utility.
- In addition, since the method uses NLI relationships between all pairs of claims and the computational complexity of the full posterior is , I think an analysis of performance and computational cost as a function of the number of claims is necessary.
Questions for authors
Q1. Can the authors theoretically characterize the conditions under which NLI-based posterior updates improve score quality or retention? In particular, how does the accuracy of NLI relationships affect the final performance?
Q2. What are the distributions and prediction accuracies of NLI relationships across the datasets? Is it possible to analyze performance according to relationship type or NLI prediction error?
Q3. Could the authors compare the FNR–retention trade-off across different values of (e.g., )? In particular, I wonder whether the improvements persist at smaller values of .
Q4. How do posterior approximation accuracy, retention, and computation time change with ? What values of and padding/truncation settings were used in practice?
Q5. How exactly are the training, calibration, and test splits constructed? If pretrained scorers are used, what is the role of the training stage in Algorithm 1?
Q6. Appendix A.6 mentions modifications to BTProp. How do these changes affect performance compared with the original method?
Suggestions:
- It would be helpful to organize the experimental comparisons according to their objectives. Existing methods can be categorized into conformal factuality (Mohri & Hashimoto, 2024; Cherian et al., 2024), NLI-based refinement (Hou et al., 2025; Marinescu et al., 2025; Rubin-Toles et al., 2025), and standalone factuality scoring. Evaluating scoring quality using AUROC/AUPRC and evaluating FNR–retention after applying CRC would help clarify the contribution. However, comparisons should account for differences in the guarantees provided and the information used by each method.
- NLI ablation and scalability analyses are needed. Comparisons with neutral-only, corrupted NLI, and oracle NLI settings, along with runtime analysis as a function of , would be useful.
- The notation and practical details could be clarified. It would be helpful to provide more explanation of the transition from to in Section 3.1, the effects of claim ordering and truncation, and how flagged claims are used for filtering the actual text.
References
[1] Mohri, C. and Hashimoto, T. (2024). Language Models with Conformal Factuality Guarantees. ICML.
[2] Cherian, J. J., Gibbs, I., and Candès, E. J. (2024). Large Language Model Validity via Enhanced Conformal Prediction Methods. NeurIPS.
[3] Noh, K., Lee, S., Kim, I., and Song, K. (2026). Multi-LLM Adaptive Conformal Inference for Reliable LLM Response. ICLR.
[4] Hou, B., Zhang, Y., Andreas, J., and Chang, S. (2025). A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagation. NAACL.
[5] Marinescu, R. et al. (2025). FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models. Findings of EMNLP.
[6] Rubin-Toles, M. et al. (2025). Conformal Language Model Reasoning with Coherent Factuality. ICLR 2025.