- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new, light-weight reward for improving the factuality of an LLM via RL. As factuality lacks cheap, verifiable rewards, traditional methods often rely on model-based rewards such as LLM-as-a-judge. This work, instead, proposes to use occurrence statistics, grounded in a reference corpus (Wikipedia in this work), of the (subject, object) pair of each sentence in an LLM response (CoT + answer) as a proxy reward on the factuality of that sentence. This surprisingly simple and efficient reward, is shown to achieve superior performance than a number of recent papers on multiple short-form factual question answering benchmarks.
Critical strengths
- The fact that a cheap reward like corpus co-occurrence outperforms a number of LLM-based rewards is an interesting and surprising finding. It significantly speeds up factuality RL training without sacrificing accuracy.
Critical weaknesses
-
Acknowledged by the authors, the co-occurrence reward is inherently noisy, which may penalize rare facts or reward false claims involving co-occurring entities. Examining the factuality of a sentence (against a reference corpus) involves much more than looking at how many times the subject and object appear together. Therefore, this approach is only suitable as a cheaper and inferior proxy to LLM-based rewards when the latter is cost-prohibitive, not as a replacement.
-
As a result, the value of the proposed reward gets down to the efficiency gain vs. the accuracy compromise. The evaluation setup (datasets, baselines, etc.) is hence critical in assessing the performance of the approach (and the paper itself). The experimental setup is defensible and covers a number of recent baselines on multiple factual QA benchmarks, but there are a few improvements that may strengthen the results.
- It would be interesting to see a more apple-to-apple comparison of the proposed reward with more expensive LLM-based rewards. For example, given the same response sentence, compare with three baselines: i) a frontier agentic model (with access to a search engine for verifying factuality), ii) a frontier agentic model (with instruction to only search on Wikipedia to control the reference corpus), and iii) a SoTA RAG model (combining a SoTA open-weight LLM with a SoTA retriever). Everything else should stay the same as the proposed method, the only difference being a more expensive reward that uses an LLM to judge the factuality of a sentence. The performance drop shown in this experiment is, IMHO, the more meaningful indicator of the effectiveness of the proposed approach.
- All the datasets used in this work are short-form factual QA benchmarks, where the answer is no more than a few words. (Exact match is used to check for accuracy.) However, this approach computes sentence-level reward, and can be applied to long-form factuality benchmarks in theory. (see https://arxiv.org/abs/2508.05618, which is already cited by this work.) It would be interesting to see how / if this approach works well on long-form benchmarks.
Questions for authors
-
FSPO pairs each training prompt with the evidence doc. Is that also the case in your implementation (or do you use retrieval)?
-
For RLFH, is the judge given access to the reference corpus (via RAG) or a search engine? Or is it purely relying on the judge's parametric knowledge to check for factuality? (As you mentioned on Line 432, I think agentic LLMs, which can search and verify against external evidence, should be used for factuality judge, to avoid circular reasoning.)
-
For KnowRL, you mentioned "Our GRPO implementation uses GPT-4o-mini, DeBERTa-v3 NLI, and a Wikipedia knowledge base for reward computation". Is a retriever used to retrieve Wikipedia passages / documents for GPT-4o? If so, which one?
-
For equation 2, why use multiple thresholds instead of making it more continuous? Wouldn't the former introduce additional hyper-parameters to tune?
2 comments
Follow-up on the Revised PDF
Thank you again for your thoughtful suggestions. Your comments are reflected in the revised PDF (changes in blue) as follows:
- Positioning and stronger verifiers (W1, W2(a)): §7 positions CorVer as a cheaper proxy for, not a replacement of, strong LLM verifiers; Appendix D.7 estimates the cost of frontier and agentic verifiers.
- Accuracy versus efficiency (W2): Table 2 adds reward-only timing on identical responses.
- Long-form benchmarks (W2(b)): §4.4 and Appendix D.5 report FActScore Biography results under FActScore-NPM and VeriScore.
- Baseline details (Q1–Q3): Appendix A.1.1 describes the FSPO, RLFH, and KnowRL verifiers; §5.3 now notes that evidence-grounded judges reduce the circularity concern.
- Binned counts (Q4): Appendix C explains the binning rationale.
We would be glad to discuss any remaining questions.
Response to Reviewer A3Q8
We thank the reviewer for the thoughtful suggestions.
W1: We agree with this positioning: CorVer is a cheaper proxy, not a replacement for a strong verifier. A frontier LLM verifier, especially an agentic one with search, would give more accurate rewards and likely a better policy; our baselines show that CorVer beats existing LLM-judge rewards, not the strongest possible judge.
Strong verifiers are prohibitively expensive. Even the moderate, non-thinking judges of RLFH, FoRAG and KnowRL take 8.5–12.9 hours to score the 4,800 responses of one Qwen3-8B run, versus about 8 minutes for CorVer (W2). Their time is mostly judge token generation (81–95%), so it grows with output length: a frontier judge at high reasoning effort emits far more tokens per call, and an agentic one adds search rounds: about a month per run (W2(a)), plus API fees, for every model, seed and baseline.
The noise is measured and bounded. From our 700-sentence audit (App. B) and a manual check of 196 zero-count sentences (2.04% extraction errors), a zero-count penalty is wrong with probability P(E∣Z)+P(C∣Z)(1−P(E∣Z))=2.04%+24%×97.96%≈25.6%, so about 74% of penalties are right, and 81% of sentences with counts of at least 20 are correct; each sentence reward lies in [−0.2, 0.1], alongside the answer-correctness reward. A judge thus improves on CorVer only if it is clearly more accurate than this, as frontier verifiers likely are, at far higher cost. We will state this positioning explicitly in the paper.
W2: Macro accuracy (Qwen3-4B/8B mean, Table 18) and reward time on the same 1,000 saved responses, including remote-verifier waiting:
W2(a): CorVer targets closed-book QA: the corpus only computes training rewards, and the policy answers from the question alone (Sec. 1). Within this setting, the question is interesting and one we considered: it measures CorVer's gap to a near-ground-truth upper bound, which is hard to obtain. No dataset labels reasoning sentences, so only strong verifiers can serve; and verifiers like (i) and (ii) approach the upper bound, so we fully expect CorVer to fall short of them.
Such training also far exceeds a research budget. One Qwen3-8B run has 4,800 responses, roughly 70,000 sentences (363 tokens per response, about 25 per sentence). Three search-and-reasoning rounds per sentence mean about 210,000 billed search queries and as many frontier-model calls per run; at our measured 12 s per remote request without thinking, a lower bound for frontier models, this is about a month of sequential judge time versus 8 minutes for CorVer. We will discuss this as future work.
W2(b): We evaluate our Llama-3.1-8B policies, trained only on short-form QA, on the 183 FActScore Biography entities used by Chen et al. (2025), with FActScore-NPM and VeriScore (Exa search; Dtl = supported claims per biography; one training seed):
Without long-form training, CorVer gives the most supported content at the highest precision under both evaluators; VeriScore with Parallel search gives the same ranking. CorVer also improves multi-hop QA (Table 4).
Q1: Yes: FSPO uses the evidence document paired with each training question, without retrieval. The policy sees only the question; HHEM-2.1 scores reasoning sentences against 384-token chunks (stride 320), marking a sentence supported if its maximum entailment probability exceeds 0.5.
Q2: The RLFH judge receives the question, the paired evidence and the generated sentences, splits them into atomic claims, and labels each supported, contradicted or unverifiable against the evidence; it has no retrieval or search tool. Evidence grounding reduces reliance on parametric knowledge, and hence the circularity noted in Line 432, though shared model biases remain.
Q3: Yes, gtr-t5-large, but its passages go to an NLI verifier, not GPT-4o-mini: GPT-4o-mini extracts atomic facts, gtr-t5-large retrieves five passages per fact from the title-matched Wikipedia document, and DeBERTa-v3-base-mnli-fever-anli checks support (entailment threshold 0.3).
Q4: Count magnitude reflects entity popularity and coverage, not calibrated confidence, so we map counts 0, 1–4, 5–19 and ≥20 to −0.2, −0.1, 0 and +0.1: fluctuations within a bin leave the reward unchanged, and frequent pairs get no extra bonus. The two cutoffs are hyperparameters, but a continuous mapping would also need a transformation and scale, and policy gradients do not need differentiability in the count. The bins coarsely track Figure 1 (24% vs 81% correct, W1).