acceptodds
Under review as a conference paper at ICLR 2027

Training Fact-Checkers with Reinforcement Learning

Abstract

Large language models (LLMs) often generate content that contains factual errors, hindering their reliability. To detect these factual errors, we propose FactCheckRL, a new reinforcement learning (RL) based paradigm for training search augmented fact checking agents. With FactCheckRL we train fact checking agents that can flexibly reason and search the web to find factual errors in LLM responses. On two human-verified evaluation sets, FactCheckRL substantially outperforms prompt-based fact-checking pipelines including SAFE, VeriScore, and FaStFact at recovering factual errors identified by human annotators. We further propose to use FactCheckRL as a factuality reward alongside general helpfulness reward models in online RL, showing that this can substantially reduce the amount of factual errors without degrading overall response usefulness. Using this approach, we are able to post-train a model that is more factual than state-of-the-art open models while maintaining comparable helpfulness. We will open-source our code, training data, and models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Fact Checking, Hallucinations, Language models, Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KTthOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes, FactCheckRL, a RL-trained agentic fact checker, which is equipped with a search tool to check for factual errors in a given response. It is trained to reason and issue search queries adaptively, and is shown to significantly outperform existing fact check methods such as SAFE, VeriScore, and FaStFact. FactCheckRL is further used as a factuality reward in post-training Nemotron using an improved variant of Chen et al., (2025), which significantly improves the model's factuality while not regressing on helpfulness.

Critical strengths
  • The paper proposes a sensible approach to a practical and important problem, which achieves strong empirical performance. Training an agentic model to issue search queries itself for fact checking allows the model more flexibility than previous works which follows the fixed decompose-search-verify pipeline. FactCheckRL's fact-checking performance beats other baselines in both in-distribution (their own held-out test set) and out-of-distribution evaluations (test set from FaStFact).

  • When used as a factuality reward during post-training, FactCheckRL is highly effective. It identifies an important flaw in the previous work (Chen et al., 2025) which used linear combination of helpfulness and factuality rewards. The argument that geometric or harmonic combinations make more sense as it penalizes the model for low score on either front is sound and proven by empirical results.

  • The fact-checking data, which the authors promise to release, would be valuable to the research community.

Critical weaknesses
  • Only one factuality benchmark (LongFact) and one helpfulness guardrail (ArenaHard) are reported. LongFact is close to saturation where the contradiction rate is only 3-6% for non-trained open models (and even lower for frontier models). It would strengthen the results if results are reported on more long-form factuality benchmarks. For example, the previous work cited by this paper (Chen et al., 2025) reported several other datasets alongside LongFact.

  • The paper considers two search corpora, both using offline closed sets of documents (20M and 200M docs, respectively). To show the versatility and real-world utility of the approach, it would be great to also consider the open web as the search corpus using free or commercial search engines. The latter is more realistic and challenging: There are many facts not present in offline corpora such as Wikipedia, so the ability to search the Internet allows the fact checker to verify a much wider range of claims. On the other hand, the Internet is much more noisy, so retrieving a evidence doc does not mean that the source is trustworthy. The model needs to reason about the retrieved evidence, potentially resolving conflicts, before returning the final verdict. This is in fact somewhere the proposed RL approach may shine even more, so giving the model a proper web search tool could strength this work meaningfully.

Questions for authors

Q1: Is the response-level 5-way numerical score the final score used in training / eval? Are the individual errors only used by the model itself to derive the 5-way score, and discarded afterwards?

Reviewer_3xXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FactCheckRL, a reinforcement-learning approach for training agents to identify factual errors in LLM-generated responses through iterative web search. The agent produces descriptions of errors, and its reward uses an LLM judge to semantically match predictions against annotated errors, penalizing unmatched and repeated predictions. Training data are constructed by injecting controlled factual perturbations into responses judged correct, followed by filtering and verification. On human-verified evaluations, Qwen3.5-27B achieves a row-level F1 of 0.8327 and micro-F1 of 0.7439; on FaStFact, its micro-F1 is 0.5443, compared with 0.2820 for the same-backbone baseline. The paper also uses fact-checking as a reward for downstream response generation, combining it with a helpfulness reward through multiplicative formulations. The resulting models show improved factuality while maintaining helpfulness on the reported evaluations. The approach is promising, although the reliability of its judge-based reward and generalization beyond controlled errors need further investigation.

Critical strengths

The paper addresses an important problem: identifying specific factual errors in long-form LLM responses rather than assigning a single factuality score. Combining iterative search with structured error descriptions provides a practical approach to evidence-based fact-checking.

The reported results are encouraging. Qwen3.5-27B reaches 0.8327 row-level F1 and 0.7439 micro-F1, while its FaStFact micro-F1 of 0.5443 substantially exceeds the reported same-backbone baseline of 0.2820.

The paper also demonstrates a downstream application by using the fact-checker as a factuality reward for response generation. The comparison of linear and multiplicative reward combinations, including the use of a harmonic formulation, provides useful evidence about balancing factuality and helpfulness. The production-scale results make the work relevant beyond the standalone fact-checking task.

Critical weaknesses

The training data are constructed by injecting controlled factual errors into otherwise factual responses. Although the authors evaluate on both their own held-out set and a refined FaStFact test set, these evaluations remain limited in size and scope. Broader testing on naturally occurring factual errors would provide stronger evidence that the approach generalizes beyond the error types represented in the training pipeline.

The reward and evaluation metrics rely on an LLM judge to match predicted error descriptions against human-verified annotations. While the reference annotations are verified, errors or systematic biases in semantic matching could still affect training and evaluation. Direct validation of judge decisions against independent human judgments would strengthen confidence in the reported F1 scores.

For downstream generation, the factuality and helpfulness results rely substantially on model-based evaluation, and the LongFact assessment uses a fixed subset of 200 prompts. It remains unclear how robust the gains are across broader prompt distributions and independent evaluators. Finally, because the trained fact-checker is itself used as a reward signal, a more direct investigation of whether the policy can exploit weaknesses in that reward would strengthen the evidence that the downstream improvements reflect genuine factuality gains.

Questions for authors
  1. How closely do the automatically generated training errors resemble naturally occurring factual errors, and how does performance change on errors outside the perturbation pipeline?

  2. What is the agreement between the LLM judge and independent human judgments when matching predicted errors to reference errors? Are there systematic types of errors the judge misses?

  3. How sensitive are the reported F1 scores to the semantic-matching criteria and the treatment of unmatched or repeated predictions?

  4. During downstream reinforcement learning, how do the authors test whether the policy exploits weaknesses in the fact-checking reward instead of becoming more factually reliable?

  5. Are the gains consistent across multiple training runs and broader evaluation sets? Can the authors report uncertainty estimates or statistical tests for the key comparisons?

  6. How much do retrieval quality and search strategy contribute to performance, given the reported differences between lexical and semantic search indexes?

Reviewer_bd8ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FactCheckRL is a method that employs reinforcement learning to train a retrieval agent to identify factual errors within responses. Training data is constructed by injecting errors into human-verified responses, and the F1 score, calculated between the predicted set of errors and the ground-truth set is used as a reward signal. Across two backbone models and two evaluation datasets, this approach demonstrates superior error-detection performance compared to existing prompt-based methods. Furthermore, by incorporating the severity of the detected errors (as assessed by the verifier) ​​into a "helpfulness" reward, the generative model can be fine-tuned, leading to improved factual accuracy on the LongFact benchmark.

Critical strengths
  1. Improvements in error detection performance are consistent across both backbones and external datasets. For the Qwen-based model, the Micro-F1 score increases from 0.4708 to 0.7439 when compared to FaStFact (using the same backbone) in self-evaluation, and from 0.2820 to 0.5443 in external FaStFact evaluation. Similar improvements are observed for the Nano variant. These results are compelling, as they demonstrate that the findings are not limited to a specific model or a proprietary, self-constructed dataset.

  2. Experiments were also conducted to explore using the trained verifier to improve generative models. When compared to an RLHF-only approach, starting from the same Lightning SFT model the inclusion of a factuality reward reduces the "Contradicted" rate on the LongFact benchmark from 8.61% to 3.49%. Meanwhile, the overall Arena-Hard scores remain comparable, at 29.0% and 30.0%, respectively. This suggests that enhancing the performance of the verifier can effectively contribute to the training of generative models.

Critical weaknesses
  1. Lack of Reliability Verification for Automated Evaluators: The accuracy of the LLM-based evaluators used to identify errors has not been sufficiently verified. In particular, it remains unclear whether the low error rates reported by SAFE and VeriScore reflect genuine performance or are instead the result of false positives or classification issues during the evaluation process. Furthermore, the criteria for categorizing a "lack of supporting evidence" as a specific type of error are not clearly defined.

  2. Uncertainty Regarding the Preservation of Information Content Amidst Factual Improvements: LongFact does not provide data on the total number of claims, response length, or the rate of refusal to answer; consequently, it is difficult to determine whether improvements in factual accuracy are merely a byproduct of a reduction in the overall volume of information. Moreover, the "usefulness" scores from Arena-Hard are insufficient to conclude that factual accuracy has been enhanced while maintaining the original level of information content.

Questions for authors
  1. Has there been any human verification of the error matching and count aggregation conducted while keeping the method names hidden? Additionally, please explain the rules used to convert "insufficient evidence" determinations from SAFE and VeriScore into a list of errors.

  2. Is it possible to compare the total number of claims, the number of supported claims, the response length, and the refusal rate based on the existing outputs from LongFact? I would also like to verify whether the information fidelity of the responses is maintained when the same questions are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.