- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark called PolitiBench, and uses it to post-train smaller open-weight LLMs via GRPO to improve their multi-class fact-checking accuracy across both in-domain and external zero-shot benchmarks, achieving higher performance compared to the baselines.
Critical strengths
- The benchmark construction process looks reliable and can be a good resource for future work.
Critical weaknesses
- Inconsistent descriptions across the paper
- In the Abstract and Introduction, the authors claim that fact-checking data is converted into training instances "stripped of verdict-revealing sentences." However, the main results section states that the primary methods (U-RL and U-RLF) are trained by "retaining any sentences, regardless of whether it might reveal the final verdict." The authors should address the inconsistency and make the takeaways of the paper coherent.
- The Introduction states that "the full article with structured evidence outperforms..." other variants. However, according to the main results, U-RL (which does not use structured evidence) actually outperforms U-RLF in most settings.
- In the Introduction, the authors state that "both models reach GPT-5.6 Luna's level," whereas the results show that the models are only close to, rather than reaching, GPT-5.6 Luna's level; a statistical significance test is required to justify the claim that they reach the same level. Additionally, the Introduction's claim of "particularly large accuracy gains on difficult ambiguous classes" appears to refer to class "recall" gains rather than "accuracy" gains.
- Lack of simpler baselines and lack of method justification
- It is unclear why the authors trained the models directly with resource-intensive GRPO without considering SFT baselines on the provided models, or fine-tuning specialized fact-checking models on the curated data to directly output a class label. Without comparing against SFT, it is unclear whether the added computational cost of GRPO is justified.
- Although the models are trained to generate explanations (<analysis>) during GRPO, the quality, reliability, and trustworthiness of these explanations are never formally evaluated during downstream inference. Moreover, it remains unclear whether including <analysis> generation during training actually benefits final prediction performance, or if the model would perform worse without it.
- Unclear data splits and checkpoint selection
- It is unclear whether the authors used a three-way train/dev/test split or only a two-way train/dev or train/test split. Table 5 (lines 303–304) suggests only a train/dev split, which implies that the best checkpoint may have been both selected and evaluated on the same dev set, even though a test set is mentioned elsewhere in the paper. If the best checkpoint was chosen based on the same subset used for final evaluation, the results would overfit the benchmark, making the comparison against zero-shot models unconvincing.
- Throughout the paper, different numbers are reported for the training, dev, and test sets, and their sums also vary from section to section, making the exact dataset statistics very confusing to follow.
Questions for authors
Please address the issues above.
5 comments
Reply part 2
Q2. Training method. Does GRPO provide sufficient improvements over SFT or direct label prediction to justify its additional computational cost? Does generating explanations during training improve final verdict accuracy, and how reliable are the explanations produced during inference?
R2:
| Model | Claim only | Claim + Article | Aug. Claim | Formatted Ev. + Article | Aug. Claim + Formatted Ev. |
|---|---|---|---|---|---|
| 4B U-RL | 31.13 ± 1.31 / 24.89 ± 1.41 | 51.67 ± 0.79 / 45.18 ± 0.92 | 29.38 ± 2.29 / 26.13 ± 1.11 | 59.38 ± 1.12 / 52.60 ± 1.54 | 46.41 ± 1.27 / 37.76 ± 2.11 |
| 4B U-RLF | 30.66 ± 1.34 / 25.75 ± 1.26 | 49.74 ± 0.85 / 43.98 ± 1.00 | 30.02 ± 1.32 / 26.86 ± 1.89 | 59.80 ± 0.88 / 54.43 ± 1.31 | 46.09 ± 0.57 / 38.44 ± 0.84 |
| 9B U-RL | 35.86 ± 0.77 / 28.11 ± 0.82 | 53.10 ± 0.49 / 46.26 ± 0.47 | 36.08 ± 0.56 / 28.78 ± 0.56 | 62.26 ± 0.38 / 56.44 ± 0.27 | 51.86 ± 0.48 / 43.59 ± 0.42 |
| 9B U-RLF | 34.97 ± 0.52 / 25.44 ± 0.50 | 50.02 ± 0.42 / 42.87 ± 0.44 | 36.13 ± 0.47 / 29.12 ± 0.53 | 62.28 ± 0.51 / 56.68 ± 0.46 | 48.24 ± 0.62 / 40.10 ± 0.57 |
| 4B SFT | 32.64 ± 0.47 / 21.18 ± 0.60 | 45.57 ± 0.86 / 36.40 ± 1.07 | 30.46 ± 1.18 / 20.52 ± 1.26 | 51.68 ± 0.64 / 43.96 ± 0.81 | 41.66 ± 0.73 / 31.08 ± 0.95 |
| 4B SFT-F | 32.49 ± 0.82 / 21.06 ± 0.73 | 45.11 ± 0.58 / 36.43 ± 0.76 | 31.26 ± 0.97 / 21.78 ± 1.18 | 52.36 ± 0.43 / 44.52 ± 0.62 | 41.96 ± 1.12 / 31.46 ± 1.29 |
| 9B SFT | 30.65 ± 0.66 / 24.91 ± 0.79 | 47.37 ± 1.08 / 41.89 ± 1.22 | 32.26 ± 0.47 / 26.39 ± 0.64 | 55.43 ± 0.91 / 49.05 ± 1.04 | 43.84 ± 0.73 / 36.73 ± 0.88 |
| 9B SFT-F | 30.15 ± 0.92 / 25.17 ± 1.07 | 45.49 ± 0.61 / 40.50 ± 0.75 | 32.11 ± 1.26 / 26.99 ± 1.38 | 56.32 ± 0.52 / 50.27 ± 0.68 | 45.61 ± 0.84 / 39.02 ± 0.97 |
GRPO mostly achieves better performances, especially in terms of balanced accuracy across the reported settings. Balanced accuracy gives equal weight to each verdict class, making it more informative about performance across the imbalanced label distribution. GRPO has higher reported scores in all three evidence-provided settings. Under Augmented Claim + Formatted Evidence, U-RL exceeds SFT by 4.75 / 6.68 percentage points for 4B and 8.02 / 6.86 points for 9B in accuracy / balanced accuracy. The pattern is strongest when evidence is available (as the ability that the models are trained is to reason the fact out of the given evidence); 4B claim-only accuracy remains higher with SFT. We ran the experiments with three random seeds (0, 42, and 100). Requiring the model to generate explanations to improve its reasoning performance, the authors manually reviewed the model’s explanations for a subset of 300 examples and found them to be coherent, especially when formatted evidence was provided.
Q3. Evaluation protocol. What are the exact train/dev/test splits and sample counts, and how can the inconsistent dataset statistics across the paper be reconciled? Was checkpoint selection performed on a validation set separate from the set used for final evaluation?
R3:
Thanks for raising this question. We would clarify that there were many cases where few examples were removed due to insufficient data. To make this clear, during preliminary experiments for the construction of the dataset, a 2100 balanced sample across 6 verdicts were used to select the optimal values of k (Maximum number of facts retrieved for each category) and experiment with the different input settings (Original Claim / Augmented Claim). During our RL training, the dataset is split into 21,118 examples of training data, 2,346 examples of development data, and 2,607 examples of test data. Resulting in a total of 26,071 valid examples in the entire benchmark.
For clarification, table 5 refers to the preliminary experiments for the construction of the dataset, this set is not used during the RL training. The data between the preliminary experiments and RL training are not disjoint. Both the train and dev sets were used during training and the checkpoint was selected based on the best dev set. We will clarify the settings in our revised version.
Follow up with the reviewer ifXo
Dear Reviewer ifXo,
As the author response period draws to a close, we would be grateful if you could review our responses and let us know whether any concerns remain. If our clarifications have addressed your concerns, we would appreciate it if you could reconsider your assessment accordingly.
Thank you again for your time and thoughtful review.
Best regards,
The Authors
Reply part 1
Q1. Consistency of claims. (1) How can the claims about verdict filtering and the benefits of structured evidence be reconciled with the use of unfiltered training articles and U-RL’s generally stronger performance than U-RLF? (2) Do the results statistically support frontier-model parity, and should the reported improvements on ambiguous classes be described as gains in class recall?
R1:
(1)
Verdict Filtering. We would like to first clarify that the verdict filtering is only conducted in inference for fair evaluation and comparison, not for training. The reason is that the sentences with verdict labels also contains the expert’s synthesis, which is important to train the models to learn to build meaningful connections from context to true verdict. For example, for Mike Pence’s claim of “record investments in education,” the article reports that spending set a nominal record but fell below earlier levels after inflation adjustment. The unfiltered training input includes:
The filtered version retains the spending facts but deletes this assessment. The model therefore loses an explicit example of how a missing qualification changes the appropriate truthfulness rating. U-RL and U-RLF preserve such assessments during training, while inference inputs exclude verdict cues.
Structured evidence. Our main findings distinguish expert analysis for training from structured evidence for inference. U-RL versus U-RLF compares training inputs, whereas adding formatted evidence evaluates its inference-time benefit. Adding formatted evidence to Claim + Article increases U-RL performance from 51.67 / 45.18 to 59.38 / 52.60 for 4B model, and from 53.10 / 46.26 to 62.26 / 56.44 for 9B (Accuracy / Balanced Accuracy, %). Training with additional formatted evidence yields smaller gains in this setting: U-RLF reaches 59.80 / 54.43 for 4B and 62.28 / 56.68 for 9B. Thus, the strongest benefit comes from providing structured evidence at inference, while its additional training benefit depends on the evaluation setting, especially when the training format aligns perfectly with the evaluation format (see column 4 in table 2). We will revise the introduction to clarify this distinction.
(2) Yes, accuracy computed within each ground-truth class is mathematically equivalent to that class’s recall. We will describe these improvements as gains in class recall. Across three seeds, 9B U-RL achieves 62.26 ± 0.38 / 56.44 ± 0.27, and U-RLF achieves 62.28 ± 0.51 / 56.68 ± 0.46 under Claim + Article + Formatted Evidence (Accuracy / Balanced Accuracy, %, mean ± standard deviation). In the same setting, GPT-5.6 Luna achieves 62.63 ± 0.64 / 57.35 ± 0.75 across three runs. Both post-trained models have mean scores close to Luna’s, and their mean ± 1 SD ranges overlap with Luna’s for both metrics. Therefore, our results support our claims that