acceptodds
Under review as a conference paper at ICLR 2027

PolitiBench: A Structured Evidence Benchmark for Post-Training LLM in Fact-Checking

17 views

Abstract

Automated fact-checking with large language models (LLMs) is dominated by frontier closed-source models, and prior attempts to specialize smaller models have not closed the gap yet. We ask whether the articles written by professional fact-checkers can serve as post-training data that teaches smaller LLMs to fact- check. We collect, process and convert 26,101 real-world PolitiFact fact-checking data into training instances that pair a contextualized claim with the fact-checker’s analysis, stripped of verdict-revealing sentences, and with evidence extracted from fact-checking articles and organized into supporting facts, weakening facts, and missing context. Our post-training of Qwen3-4B and Qwen3.5-9B, with Group Relative Policy Optimization (GRPO) and our proposed reward function, raises six-class verdict accuracy over the base checkpoints by 5.6% and 7.4%, respec- tively, with particularly large accuracy gains on difficult ambiguous classes (e.g., Half True from 6.51% to 44.79% and Mostly True from 11.02% to 29.02%). In the setting that mirrors retrieval-augmented fact-checking in practical deployment, where only the claim and formatted evidence are available, the post-trained 9B model exceeds the best among the 5 frontier models, GPT-5.6 Luna, by 1.3%. When sufficient evidence is available, our post-trained models improve the accu- racy of their base models by 7.1% and 5.8%, respectively, once again reaching the same level as GPT-5.6 Luna. The gains transfer to 11 external fact-checking eval- uation cases, where our post-trained models outperform their base checkpoints in 80% of cases, and match or surpass frontier models on 8 of the 11 benchmarks under zero-shot settings. This verifies that the post-training on our datasets boosts the general fact-checking ability of small models. We release PolitiBench, the processed corpus, as a resource for both evaluating and specializing LLMs for evidence-grounded fact-checking.

Primary area:
datasets and benchmarks
Keywords:
benchmark, fact checking, post training, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ifXoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark called PolitiBench, and uses it to post-train smaller open-weight LLMs via GRPO to improve their multi-class fact-checking accuracy across both in-domain and external zero-shot benchmarks, achieving higher performance compared to the baselines.

Critical strengths
  1. The benchmark construction process looks reliable and can be a good resource for future work.
Critical weaknesses
  1. Inconsistent descriptions across the paper
  • In the Abstract and Introduction, the authors claim that fact-checking data is converted into training instances "stripped of verdict-revealing sentences." However, the main results section states that the primary methods (U-RL and U-RLF) are trained by "retaining any sentences, regardless of whether it might reveal the final verdict." The authors should address the inconsistency and make the takeaways of the paper coherent.
  • The Introduction states that "the full article with structured evidence outperforms..." other variants. However, according to the main results, U-RL (which does not use structured evidence) actually outperforms U-RLF in most settings.
  • In the Introduction, the authors state that "both models reach GPT-5.6 Luna's level," whereas the results show that the models are only close to, rather than reaching, GPT-5.6 Luna's level; a statistical significance test is required to justify the claim that they reach the same level. Additionally, the Introduction's claim of "particularly large accuracy gains on difficult ambiguous classes" appears to refer to class "recall" gains rather than "accuracy" gains.
  1. Lack of simpler baselines and lack of method justification
  • It is unclear why the authors trained the models directly with resource-intensive GRPO without considering SFT baselines on the provided models, or fine-tuning specialized fact-checking models on the curated data to directly output a class label. Without comparing against SFT, it is unclear whether the added computational cost of GRPO is justified.
  • Although the models are trained to generate explanations (<analysis>) during GRPO, the quality, reliability, and trustworthiness of these explanations are never formally evaluated during downstream inference. Moreover, it remains unclear whether including <analysis> generation during training actually benefits final prediction performance, or if the model would perform worse without it.
  1. Unclear data splits and checkpoint selection
  • It is unclear whether the authors used a three-way train/dev/test split or only a two-way train/dev or train/test split. Table 5 (lines 303–304) suggests only a train/dev split, which implies that the best checkpoint may have been both selected and evaluated on the same dev set, even though a test set is mentioned elsewhere in the paper. If the best checkpoint was chosen based on the same subset used for final evaluation, the results would overfit the benchmark, making the comparison against zero-shot models unconvincing.
  • Throughout the paper, different numbers are reported for the training, dev, and test sets, and their sums also vary from section to section, making the exact dataset statistics very confusing to follow.
Questions for authors

Please address the issues above.

5 comments
AuthorsOpenReview ↗

Reply part 1

Q1. Consistency of claims. (1) How can the claims about verdict filtering and the benefits of structured evidence be reconciled with the use of unfiltered training articles and U-RL’s generally stronger performance than U-RLF? (2) Do the results statistically support frontier-model parity, and should the reported improvements on ambiguous classes be described as gains in class recall?

R1:

(1)

Verdict Filtering. We would like to first clarify that the verdict filtering is only conducted in inference for fair evaluation and comparison, not for training. The reason is that the sentences with verdict labels also contains the expert’s synthesis, which is important to train the models to learn to build meaningful connections from context to true verdict. For example, for Mike Pence’s claim of “record investments in education,” the article reports that spending set a nominal record but fell below earlier levels after inflation adjustment. The unfiltered training input includes:

Pence’s statement is partially accurate but leaves out important details, so we rate it Half True.

The filtered version retains the spending facts but deletes this assessment. The model therefore loses an explicit example of how a missing qualification changes the appropriate truthfulness rating. U-RL and U-RLF preserve such assessments during training, while inference inputs exclude verdict cues.

Structured evidence. Our main findings distinguish expert analysis for training from structured evidence for inference. U-RL versus U-RLF compares training inputs, whereas adding formatted evidence evaluates its inference-time benefit. Adding formatted evidence to Claim + Article increases U-RL performance from 51.67 / 45.18 to 59.38 / 52.60 for 4B model, and from 53.10 / 46.26 to 62.26 / 56.44 for 9B (Accuracy / Balanced Accuracy, %). Training with additional formatted evidence yields smaller gains in this setting: U-RLF reaches 59.80 / 54.43 for 4B and 62.28 / 56.68 for 9B. Thus, the strongest benefit comes from providing structured evidence at inference, while its additional training benefit depends on the evaluation setting, especially when the training format aligns perfectly with the evaluation format (see column 4 in table 2). We will revise the introduction to clarify this distinction.

(2) Yes, accuracy computed within each ground-truth class is mathematically equivalent to that class’s recall. We will describe these improvements as gains in class recall. Across three seeds, 9B U-RL achieves 62.26 ± 0.38 / 56.44 ± 0.27, and U-RLF achieves 62.28 ± 0.51 / 56.68 ± 0.46 under Claim + Article + Formatted Evidence (Accuracy / Balanced Accuracy, %, mean ± standard deviation). In the same setting, GPT-5.6 Luna achieves 62.63 ± 0.64 / 57.35 ± 0.75 across three runs. Both post-trained models have mean scores close to Luna’s, and their mean ± 1 SD ranges overlap with Luna’s for both metrics. Therefore, our results support our claims that

AuthorsOpenReview ↗

Reply part 2

Q2. Training method. Does GRPO provide sufficient improvements over SFT or direct label prediction to justify its additional computational cost? Does generating explanations during training improve final verdict accuracy, and how reliable are the explanations produced during inference?

R2:

ModelClaim onlyClaim + ArticleAug. ClaimFormatted Ev. + ArticleAug. Claim + Formatted Ev.
4B U-RL31.13 ± 1.31 / 24.89 ± 1.4151.67 ± 0.79 / 45.18 ± 0.9229.38 ± 2.29 / 26.13 ± 1.1159.38 ± 1.12 / 52.60 ± 1.5446.41 ± 1.27 / 37.76 ± 2.11
4B U-RLF30.66 ± 1.34 / 25.75 ± 1.2649.74 ± 0.85 / 43.98 ± 1.0030.02 ± 1.32 / 26.86 ± 1.8959.80 ± 0.88 / 54.43 ± 1.3146.09 ± 0.57 / 38.44 ± 0.84
9B U-RL35.86 ± 0.77 / 28.11 ± 0.8253.10 ± 0.49 / 46.26 ± 0.4736.08 ± 0.56 / 28.78 ± 0.5662.26 ± 0.38 / 56.44 ± 0.2751.86 ± 0.48 / 43.59 ± 0.42
9B U-RLF34.97 ± 0.52 / 25.44 ± 0.5050.02 ± 0.42 / 42.87 ± 0.4436.13 ± 0.47 / 29.12 ± 0.5362.28 ± 0.51 / 56.68 ± 0.4648.24 ± 0.62 / 40.10 ± 0.57
4B SFT32.64 ± 0.47 / 21.18 ± 0.6045.57 ± 0.86 / 36.40 ± 1.0730.46 ± 1.18 / 20.52 ± 1.2651.68 ± 0.64 / 43.96 ± 0.8141.66 ± 0.73 / 31.08 ± 0.95
4B SFT-F32.49 ± 0.82 / 21.06 ± 0.7345.11 ± 0.58 / 36.43 ± 0.7631.26 ± 0.97 / 21.78 ± 1.1852.36 ± 0.43 / 44.52 ± 0.6241.96 ± 1.12 / 31.46 ± 1.29
9B SFT30.65 ± 0.66 / 24.91 ± 0.7947.37 ± 1.08 / 41.89 ± 1.2232.26 ± 0.47 / 26.39 ± 0.6455.43 ± 0.91 / 49.05 ± 1.0443.84 ± 0.73 / 36.73 ± 0.88
9B SFT-F30.15 ± 0.92 / 25.17 ± 1.0745.49 ± 0.61 / 40.50 ± 0.7532.11 ± 1.26 / 26.99 ± 1.3856.32 ± 0.52 / 50.27 ± 0.6845.61 ± 0.84 / 39.02 ± 0.97

GRPO mostly achieves better performances, especially in terms of balanced accuracy across the reported settings. Balanced accuracy gives equal weight to each verdict class, making it more informative about performance across the imbalanced label distribution. GRPO has higher reported scores in all three evidence-provided settings. Under Augmented Claim + Formatted Evidence, U-RL exceeds SFT by 4.75 / 6.68 percentage points for 4B and 8.02 / 6.86 points for 9B in accuracy / balanced accuracy. The pattern is strongest when evidence is available (as the ability that the models are trained is to reason the fact out of the given evidence); 4B claim-only accuracy remains higher with SFT. We ran the experiments with three random seeds (0, 42, and 100). Requiring the model to generate explanations to improve its reasoning performance, the authors manually reviewed the model’s explanations for a subset of 300 examples and found them to be coherent, especially when formatted evidence was provided.

Q3. Evaluation protocol. What are the exact train/dev/test splits and sample counts, and how can the inconsistent dataset statistics across the paper be reconciled? Was checkpoint selection performed on a validation set separate from the set used for final evaluation?

R3:

Thanks for raising this question. We would clarify that there were many cases where few examples were removed due to insufficient data. To make this clear, during preliminary experiments for the construction of the dataset, a 2100 balanced sample across 6 verdicts were used to select the optimal values of k (Maximum number of facts retrieved for each category) and experiment with the different input settings (Original Claim / Augmented Claim). During our RL training, the dataset is split into 21,118 examples of training data, 2,346 examples of development data, and 2,607 examples of test data. Resulting in a total of 26,071 valid examples in the entire benchmark.

For clarification, table 5 refers to the preliminary experiments for the construction of the dataset, this set is not used during the RL training. The data between the preliminary experiments and RL training are not disjoint. Both the train and dev sets were used during training and the checkpoint was selected based on the best dev set. We will clarify the settings in our revised version.

AuthorsOpenReview ↗

Follow up with the reviewer ifXo

Dear Reviewer ifXo,

As the author response period draws to a close, we would be grateful if you could review our responses and let us know whether any concerns remain. If our clarifications have addressed your concerns, we would appreciate it if you could reconsider your assessment accordingly.

Thank you again for your time and thoughtful review.

Best regards,
The Authors

Reviewer_q2UzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PolitiBench is a fact-checking dataset that utilizes 26,101 articles from PolitiFact to provide claims, article analyses, and structured evidence. It supplements the context of each claim and categorizes evidence into supporting facts, refuting facts, and missing context. Using this dataset, 4B and 9B parameter models were trained via GRPO, and their performance was evaluated based on PolitiFact’s six-tier classification system as well as on external datasets.

Critical strengths
  1. The paper provides both specific claims and granular evidence. By going beyond simple true/false classifications, distinguishing between partial support and missing context, it proves useful for examining the nuanced veracity of political claims.

  2. Improvements over the base model are evident across various evaluation conditions. For instance, in PolitiFact evaluations, where claims and structured evidence are provided, the accuracy of the 9B U-RL model increases from 43.96% to 51.40%; in Snopes evaluations, it rises from 50.50% to 60.50%. The fact that performance also improves on certain external datasets underscores the practical value of this approach.

Critical weaknesses
  1. Lack of verification regarding the causes of performance improvement: U-RL includes clues to the correct answer within its training data; when these are removed, the F-RL model’s performance falls below that of the baseline model. Furthermore, U-RLF, which incorporates structured evidence, also underperforms compared to U-RL. Consequently, it remains unclear whether the expert-driven reasoning and the structuring of evidence actually contribute to performance gains.

  2. Insufficient validation of benchmark quality: There has been no independent human verification of the reformulated claims or the evidence extracted by the model. Issues such as shifts in meaning, omissions of evidence, misclassification, and the potential persistence of clues pointing to the correct answer have not been assessed, making it difficult to gauge the reliability of the benchmark.

Questions for authors
  1. Assessment of Training Effects: What are the results when F-RL is evaluated under the ‘Augmented Claim + Formatted Evidence’ condition (as shown in Table 2)? Furthermore, how can one distinguish between the impact of retaining the ground-truth sentences and the impact of learning from expert analysis and the structuring of evidence?

  2. Assessment of Data Quality: Have there been independent checks on a random sample regarding the preservation of the original claim’s meaning, the fidelity of the extracted evidence to the source article, the accuracy of category assignments, and the potential leakage of ground-truth information? Please verify whether the primary performance differences persist even when cases containing identified errors are excluded.

5 comments
AuthorsOpenReview ↗

Reply part 1

Q1. Source of the gains. How does F-RL perform under Table 2’s Augmented Claim + Formatted Evidence condition, and how much improvement survives removal of verdict cues? What evidence establishes the contribution of expert analysis and structured evidence, given that U-RL retains verdict cues and generally outperforms U-RLF, which also includes structured evidence?

R1:

We would like to first clarify our contributions concern expert analysis for training and structured evidence for inference. Expert analysis provides the reasoning context for learning verdict prediction, while structured evidence presents relevant facts explicitly at inference. The filtered-training results across all input settings are:

Model/InputClaim onlyClaim + ArticleAug. ClaimClaim + Formatted Ev. + ArticleAug. Claim + Formatted Ev.
4B Filtered RL31.30 / 20.2248.64 / 39.6631.80 / 22.4457.12 / 49.3644.07 / 32.69
9B Filtered RL30.42 / 24.8345.38 / 37.9023.86 / 23.8454.09 / 47.3543.92 / 34.69

Values are Accuracy / Balanced Accuracy (%). Under Augmented Claim + Formatted Evidence, 4B F-RL improves over Base from 42.08 / 31.18 to 44.07 / 32.69, retaining gains of 1.99 / 1.51 percentage points after filtering. For 9B, performance changes from 43.96 / 36.75 to 43.92 / 34.69, remaining near Base accuracy with lower balanced accuracy. The cross-model comparison is metric-dependent: in this setting, 9B F-RL has nearly identical accuracy to 4B and 2.00 points higher balanced accuracy. The two backbones therefore differ in their distribution of class-level performance.

Why removal of verdict cues in training data can cause damage to post-training Filtering also changes the available expert analysis: verdict-bearing sentences may combine a conclusion with the evidence weighting and qualifications that explain it. Their removal can discard useful training context alongside verdict cues, therefore make model hard to learn to reason the connection between the context and the corresponding true verdict.

The inference-time benefit of structured evidence remains clear with filtered training. Holding the checkpoint fixed, adding structured evidence to the augmented claim improves 4B F-RL from 31.80 / 22.44 to 44.07 / 32.69, and 9B F-RL from 23.86 / 23.84 to 43.92 / 34.69. Moreover, adding structured evidence to Claim + Article likewise improves 4B F-RL from 48.64 / 39.66 to 57.12 / 49.36, and 9B F-RL from 45.38 / 37.90 to 54.09 / 47.35. Those ablation studies demonstrate the benefit of supplying structured evidence at inference independently of whether it was beneficial as an additional training input. For all models, providing formatted evidence results in better prediction accuracy.

Q2. Benchmark quality. Can an independent human audit verify that rewritten claims preserve their original meaning, extracted evidence faithfully represents the source, evidence categories are correct, and verdict cues have been removed? Do the main performance differences persist after excluding examples with identified errors?

R2:

We agree that preserving the claim semantics, ensuring evidence fidelity, and preventing ground-truth leakage are essential for reliable evaluation. As what we mentioned in our paper, we constructed our prompts to minimize these said errors through few shot inferencing and such. For both claim augmentation and evidence retrieval, we both implemented a series of constraints through prompting with examples and explicit rules. Throughout the development of our benchmark, we conducted multiple rounds of testing and iterative refinement of the components during data-processing with the explicit goal of minimizing these potential sources of error. In particular the claim preserves the original claim’s meaning, and the evidence extraction component was structured to distinguish supporting facts, weakening facts, and missing context. These components and their output structures were repeatedly evaluated and refined during development to improve consistency and reduce potential inaccuracies.

In terms of verdict leakage, we have done multiple precautions and methods in removing the verdict leakage and have done audits across the entire dataset with frontier AI. Which resulted in no verdict leakage.

AuthorsOpenReview ↗

Reply part 2

R2, continued:

To further address your concerns, we conducted an additional experiment using 300 examples sampled from the test set with human audition. This audit was done across 3 people, in searching for these errors. With the result of the audit, we then ran an experiment over 283 samples, removing examples with errors from the 300. The result of this experiment can be found in the table below. The results show that the main performance of our proposed methods persist after excluding examples with identified errors.

CheckScopeClear Problems
Claim changed in meaning113 reworded claims4 (1.3% of 300)
Evidence fidelity1,037 facts4 (99.6% faithful)
Evidence category1,037 facts10 (99.0% correct)
ModelAccuracy (%)
4B article39.75 ± 2.25
4B article + formatted41.52 ± 2.25
9B article41.87 ± 1.75
9B article + formatted42.58 ± 0.25
GPT5.6-LUNA42.40

Accuracy across 2 seeds, reported as mean ± sample standard deviation where multiple seeds are available.

AuthorsOpenReview ↗

Follow up with the reviewer q2Uz

Dear Reviewer q2Uz,

As the author response period draws to a close, we would be grateful if you could review our responses and let us know whether any concerns remain. If our clarifications have addressed your concerns, we would appreciate it if you could reconsider your assessment accordingly.

Thank you again for your time and thoughtful review.

Best regards,
The Authors

Reviewer_1dxiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PolitiBench turns 26,101 PolitiFact articles (2007–2026) into training data. Each claim gets context added, verdict-revealing sentences are removed, and the article’s evidence is sorted into supporting facts, weakening facts and missing context. The authors post-train Qwen3-4B and Qwen3.5-9B with GRPO for six-class verdict prediction, in a single-shot setting with the evidence given in the prompt. The models gain 5.6–7.4 points of accuracy over their base versions, and the 9B model edges past GPT-5.6 Luna by about 1.3 points, but only when given the augmented claim plus the extracted evidence. Results on 10 external benchmarks are mixed: they match or beat the best reference model on accuracy in 7 of 11 settings (the paper says 8) and lose clearly on Snopes, RealFactBench and FACTors.

Critical strengths

S1: A large, recent, well-structured resource. PolitiBench releases 26,101 PolitiFact fact-checks from 2007 to June 2026, with six-way verdicts, claims rewritten to stand on their own, evidence organised into supporting facts, weakening facts and missing context, and code. That makes it about 5× the size of AVeriTeC. Part of the data post-dates the training cutoffs of current models, which reduces contamination. It is also useful as training data: it improves the base models in 35 of 44 external comparisons and sharply raises recall on hard classes (Half True: 6.5% → 44.8%).

Critical weaknesses

W1: The headline setting is unrealistic, and the gains are confounded with verdict leakage. All evaluation is single-shot verdict prediction without retrieval, and the authors assume “the retrieval pipeline works perfectly.” Yet the headline setting, which they say “mirrors… practical deployment,” gives the model evidence that GPT-5.6 Luna extracted from the PolitiFact article itself, a document written after the verdict to justify it. There is no experiment with a real retriever and no agentic or web-search baseline. Without evidence, the post-trained models trail the frontier models (claim only: −1.27 / −3.72).

The reported U-RL models are trained on inputs that keep verdict-revealing sentences, which contradicts the abstract. The leakage-filtered variant performs substantially worse (Appendix D.1: 54.09% vs 62.56% for 9B with article + evidence). The authors concede that “removing data leakage from the training dataset leads to a decrease in model performance.” The filtered variant is never evaluated in the headline setting.

So it is unclear whether the models learned to verify claims or to read verdict cues. This also undermines the claim that fact-checkers’ reasoning is “the best signal,” because the variant that wins the ablation is the one trained with leakage.

The paper’s motivation is that small models can stand in for frontier fact-checkers in practice. That isn’t supported if the advantage depends on oracle evidence and training with leakage.

W2: The “matches or beats frontier models” claim rests on small, overstated margins.

In-domain: the 9B model leads by only +1.30 / +0.66 on about 2.6k test claims, roughly one standard error by my estimate. There is a single seed and no confidence intervals, and in every other PolitiFact input setting the post-trained models tie or lose.

this is the paper’s main selling point, and as reported it is within noise in-domain and mixed out of domain.

Questions for authors

1. Leakage: Can you report the leakage-free model (F-RL) in the augmented claim + evidence setting? If it still matches GPT-5.6 Luna, my concern about leakage (W1) is largely resolved. If not, the headline claim doesn’t hold.

2. SciFact / ClimateCheck prompt: Appendix E says this prompt was developed after looking at test errors, and that the original prompt caused regressions. What are the results with the original prompt? These datasets account for 2 of the 3 wins on both metrics.

4 comments
AuthorsOpenReview ↗

Reply Part 1

Q1. How does the filtered 9B model (F-RL) perform under Table 2’s Augmented Claim + Formatted Evidence condition? Does it still match GPT-5.6 Luna after verdict-revealing sentences are removed from the training inputs?

R1:

Clarification We would like to first clarify that the verdict-revealing sentences are only used for training, instead of inference. The reason that we keep verdict-revealing sentences for training is that filtering removes more than verdict labels when the same sentence also contains the expert’s synthesis. For example, a training claim states that Mike Pence made “record investments in education.” The article establishes record spending in nominal dollars but lower spending than earlier years after adjusting for inflation. The unfiltered input includes:

Pence’s statement is partially accurate but leaves out important details, so we rate it Half True.

The filtered input deletes this entire sentence, removing the explicit connection between partial factual support, omitted context, and the graded verdict. Such loss of expert synthesis impairs the fact-checking reasoning ability of the trained models, and is a possible contributor to the filtering penalty. Therefore, we expect the training without verdict-revealing sentences would be worse than keeping the verdict-revealing sentences, and this aligns with what we get from experiments (as shown in the table below).

ModelClaim onlyClaim + ArticleAug. ClaimFormatted Ev. + ArticleAug. Claim + Formatted Ev.
9B Base30.49 / 23.8147.99 / 41.4832.68 / 26.0656.92 / 50.9043.96 / 36.75
9B Filtered RL30.42 / 24.8345.38 / 37.9023.86 / 23.8454.09 / 47.3543.92 / 34.69
GPT-5.6 Luna36.94 / 31.7054.70 / 49.1140.90 / 37.6962.63 / 57.3550.15 / 41.79

In the above table for post-training results with removed verdict-revealing sentences, each cell reports Accuracy / Balanced Accuracy (%). Under Augmented Claim + Formatted Evidence, 4B retains gains of 1.99 / 1.51 points over Base, while 9B changes by −0.04 / −2.06 points. The cross-model ordering is metric-dependent: 9B F-RL is only 0.15 points below 4B in accuracy and 2.00 points above it in balanced accuracy. Under Augmented Claim + Formatted Evidence, 9B F-RL achieves 43.92 / 34.69, compared with GPT-5.6 Luna’s 50.15 / 41.79, a gap of 6.23 / 7.10 percentage points. As we expected, without verdict-revealing sentences in the training data, the 9B model cannot achieve the GPT-5.6 Luna level of performance.

Q2. Retrieval realism. Do the performance gains persist when evidence is obtained through a real retrieval pipeline? How does the model compare with agentic or web-search fact-checking baselines under this setting?

R2:

Thanks for pointing this out and we would like to clarify that the model ability that we are trying to improve in this paper is the ability to reason the true fact out of the given evidence. Therefore, the study of the performance with real retrieval pipeline is out of the scope of our paper. Also, when we compare the model performance, we feed the same evidence to each model for inference, which can guarantee fair comparisons. Also, web search can add an additional problem, as PolitiBench is constructed from publicly available PolitiFact website, the web search can directly find the correct answer there. This would undermine the value of our post-training and the comparisons. So to isolate the fair comparison of fact-checking reasoning ability, we didn't include a real retrieval pipeline.

We agree that a real retrieval pipeline is important for the deployment of a real fact checking system. Therefore, in the future, we will include web search into our pipeline as you suggest.

AuthorsOpenReview ↗

Reply part 2

Q3. (1) Are the small performance advantages over frontier models statistically reliable, given the single training seed and absence of confidence intervals? (2) How can the authors reconcile their claim of matching or outperforming frontier models in 8 of 11 external settings with the reviewer’s count of 7 of 11 on accuracy?

R3:

(1) We train our models with three different seeds on PolitiBench. Under Claim + Article + Formatted Evidence, 9B U-RL achieves 62.26 ± 0.38% accuracy and 56.44 ± 0.27% balanced accuracy; U-RLF achieves 62.28 ± 0.51% and 56.68 ± 0.46%, respectively (mean ± standard deviation). Under the same input condition, the frontier model GPT-5.6 Luna achieves 62.63% accuracy and 57.35% balanced accuracy. From the above results we can see that, our post-trained small models can achieve the GPT-5.6 Luna level performance, with no significant statistical differences. This is consistent with what we claim in our paper.

(2) According to our standard, on a held-out dataset, we claim our post-training models match or outperform frontier models when the best of our post-training models at least does not significantly underperform the best of the frontier models. Here are the full results with repeated evaluations with 3 different seeds.

DatasetBest post-trained resultBest of frontier models
Snopes (C)34.70 ± 2.95 / 34.70 ± 2.9550.80 / 50.80
Snopes (C+E)60.50 ± 3.03 / 60.50 ± 3.0362.90 / 62.90
ClaimReview 2025Q458.60 ± 3.29 / 40.47 ± 3.7848.72 / 50.32
SciFact83.78 ± 3.92 / 84.92 ± 3.8882.89 / 83.09
ClimateCheck70.54 ± 2.05 / 70.37 ± 2.2066.91 / 63.16
RealFactBench72.14 ± 3.32 / 65.80 ± 3.7184.00 / 83.98
SciClaimHunt84.30 ± 2.25 / 78.84 ± 2.9781.70 / 69.54
QuanTemp++62.85 ± 2.51 / 49.93 ± 2.7862.92 / 50.80
AVeriTeC dev86.30 ± 3.22 / 70.15 ± 5.5583.79 / 71.53
TSVer73.98 ± 7.75 / 52.24 ± 8.9573.17 / 58.74
FACTors 2024+29.75 ± 3.17 / 25.93 ± 3.0837.50 / 34.96

As we can see, the best of our models match or outperform frontier models on 8 of 11 held-out datasets, which is consistent with our claims in the paper. We will clarify our standard explicitly in the revised paper.

Q4. What are the SciFact and ClimateCheck results with the original prompt, before it was revised after inspecting test errors? How much do the reported improvements depend on those revisions?

R4:

DatasetNPrompt9B U-RL9B RL + formatted evidence
SciFact339Original79.06 / 80.0479.06 / 80.33
SciFact339Modified82.01 / 82.9483.78 / 84.92
ClimateCheck1,904Original63.18 / 68.2962.61 / 66.51
ClimateCheck1,904Modified65.81 / 70.0667.02 / 70.37

Thanks for raising this question. We show the comparison of 9B base model with 9B U-RL and 9B RL +formatted evidence under original prompt and modified prompt. The revised prompt improves 9B U-RL by 2.95 / 2.90 points on SciFact and 2.63 / 1.77 on ClimateCheck; the corresponding gains for 9B RL + formatted evidence are 4.72 / 4.59 and 4.41 / 3.86.

Reason for prompt revision: We would like to clarify that the main goal of the tests on SciFact and ClimateCheck is to compare the general fact-checking ability of our post-trained with the frontier models. And for fair comparison, we try to eliminate all the other covariants in the benchmarks besides fact checking ability, e.g. general language understanding. However, we find that the verdict labels in SciFact and ClimateCheck have serious mismatch with the training data from PolitiBench, and therefore, a correct answer will also depend on the model’s general language understanding ability on the meaning of the labels in SciFact and ClimateCheck, which is harder for small models than frontier models. Therefore, to isolate fact-checking ability as the main task to compare, we revise the prompt for our post-trained model, so that it can first understand the label meaning for SciFact and ClimateCheck and then do reasoning.

To be more specific about how the revision addresses a task mismatch: PolitiFact training assigns a degree of truthfulness, with no NEI target, whereas SciFact and ClimateCheck require NEI when the supplied abstract cannot establish or refute a claim. Although the original prompt defined NEI, some outputs still made a truthfulness judgment using information beyond the abstract. The revision explicitly reinforces the evidence boundary and the NEI decision rule.

AuthorsOpenReview ↗

Reply part 3

R4, continued:

For example, SciFact 294:10874408 claims that crossover hot spots are absent from gene promoters in S. cerevisiae. The abstract does not establish their location relative to promoters. Originally, 9B U-RL invoked “general biological knowledge” to judge the claim inaccurate and output REFUTED. With the revised prompt, the same checkpoint recognized the missing evidence and correctly output NEI. The revision was motivated by test-error inspection and selected through a separate development comparison; we therefore retain both conditions and describe the follow-up as exploratory.

AuthorsOpenReview ↗

Follow up with the reviewer 1dxi

Dear Reviewer 1dxi,

As the author response period draws to a close, we would be grateful if you could review our responses and let us know whether any concerns remain. If our clarifications have addressed your concerns, we would appreciate it if you could reconsider your assessment accordingly.

Thank you again for your time and thoughtful review.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.