acceptodds
Under review as a conference paper at ICLR 2027

Beyond Math and Code: Lightweight Corpus-Grounded Process Rewards for Factual Question Answering

Abstract

Process supervision during reinforcement learning (RL) post-training matters in factual question answering (QA) because responses receiving positive response-level rewards can still contain sentence-level factual errors. Unlike math and code, factual QA lacks inexpensive programmatic checks, making reward computation a bottleneck. Existing methods rely on neural verifiers to score individual sentences, requiring extensive model inference as RL repeats these checks across many sampled responses at every update. We therefore introduce CorVer (Corpus Verify), a lightweight training-time approach to process supervision that derives sentence-level rewards from corpus co-occurrence statistics. A 0.5B extractor identifies subject–object pairs, indexed corpus queries supply their co-occurrence counts, and the resulting sentence rewards are assigned to the corresponding tokens for RL. On Qwen3-4B and Qwen3-8B, CorVer reduces mean complete training time by 5.5–10.4× relative to the four factuality-RL baselines. Across the four models evaluated against these baselines, CorVer achieves the highest accuracy in 17 of 20 model–benchmark settings. CorVer outperforms the unmodified models in all 30 standard factual-QA settings (six models from three families across five benchmarks) and remains effective on two additional multi-hop QA datasets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
factual question answering, reinforcement learning, process supervision, large language models, corpus co-occurrence, reward design

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A3Q8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new, light-weight reward for improving the factuality of an LLM via RL. As factuality lacks cheap, verifiable rewards, traditional methods often rely on model-based rewards such as LLM-as-a-judge. This work, instead, proposes to use occurrence statistics, grounded in a reference corpus (Wikipedia in this work), of the (subject, object) pair of each sentence in an LLM response (CoT + answer) as a proxy reward on the factuality of that sentence. This surprisingly simple and efficient reward, is shown to achieve superior performance than a number of recent papers on multiple short-form factual question answering benchmarks.

Critical strengths
  • The fact that a cheap reward like corpus co-occurrence outperforms a number of LLM-based rewards is an interesting and surprising finding. It significantly speeds up factuality RL training without sacrificing accuracy.
Critical weaknesses
  • Acknowledged by the authors, the co-occurrence reward is inherently noisy, which may penalize rare facts or reward false claims involving co-occurring entities. Examining the factuality of a sentence (against a reference corpus) involves much more than looking at how many times the subject and object appear together. Therefore, this approach is only suitable as a cheaper and inferior proxy to LLM-based rewards when the latter is cost-prohibitive, not as a replacement.

  • As a result, the value of the proposed reward gets down to the efficiency gain vs. the accuracy compromise. The evaluation setup (datasets, baselines, etc.) is hence critical in assessing the performance of the approach (and the paper itself). The experimental setup is defensible and covers a number of recent baselines on multiple factual QA benchmarks, but there are a few improvements that may strengthen the results.

    • It would be interesting to see a more apple-to-apple comparison of the proposed reward with more expensive LLM-based rewards. For example, given the same response sentence, compare with three baselines: i) a frontier agentic model (with access to a search engine for verifying factuality), ii) a frontier agentic model (with instruction to only search on Wikipedia to control the reference corpus), and iii) a SoTA RAG model (combining a SoTA open-weight LLM with a SoTA retriever). Everything else should stay the same as the proposed method, the only difference being a more expensive reward that uses an LLM to judge the factuality of a sentence. The performance drop shown in this experiment is, IMHO, the more meaningful indicator of the effectiveness of the proposed approach.
    • All the datasets used in this work are short-form factual QA benchmarks, where the answer is no more than a few words. (Exact match is used to check for accuracy.) However, this approach computes sentence-level reward, and can be applied to long-form factuality benchmarks in theory. (see https://arxiv.org/abs/2508.05618, which is already cited by this work.) It would be interesting to see how / if this approach works well on long-form benchmarks.
Questions for authors
  • FSPO pairs each training prompt with the evidence doc. Is that also the case in your implementation (or do you use retrieval)?

  • For RLFH, is the judge given access to the reference corpus (via RAG) or a search engine? Or is it purely relying on the judge's parametric knowledge to check for factuality? (As you mentioned on Line 432, I think agentic LLMs, which can search and verify against external evidence, should be used for factuality judge, to avoid circular reasoning.)

  • For KnowRL, you mentioned "Our GRPO implementation uses GPT-4o-mini, DeBERTa-v3 NLI, and a Wikipedia knowledge base for reward computation". Is a retriever used to retrieve Wikipedia passages / documents for GPT-4o? If so, which one?

  • For equation 2, why use multiple thresholds instead of making it more continuous? Wouldn't the former introduce additional hyper-parameters to tune?

2 comments
AuthorsOpenReview ↗

Response to Reviewer A3Q8

We thank the reviewer for the thoughtful suggestions.

W1: We agree with this positioning: CorVer is a cheaper proxy, not a replacement for a strong verifier. A frontier LLM verifier, especially an agentic one with search, would give more accurate rewards and likely a better policy; our baselines show that CorVer beats existing LLM-judge rewards, not the strongest possible judge.

Strong verifiers are prohibitively expensive. Even the moderate, non-thinking judges of RLFH, FoRAG and KnowRL take 8.5–12.9 hours to score the 4,800 responses of one Qwen3-8B run, versus about 8 minutes for CorVer (W2). Their time is mostly judge token generation (81–95%), so it grows with output length: a frontier judge at high reasoning effort emits far more tokens per call, and an agentic one adds search rounds: about a month per run (W2(a)), plus API fees, for every model, seed and baseline.

The noise is measured and bounded. From our 700-sentence audit (App. B) and a manual check of 196 zero-count sentences (2.04% extraction errors), a zero-count penalty is wrong with probability P(E∣Z)+P(C∣Z) (1−P(E∣Z))=2.04%+24%×97.96%≈25.6%P(E\mid Z)+P(C\mid Z)\,(1-P(E\mid Z))=2.04\%+24\%\times97.96\%\approx25.6\%, so about 74% of penalties are right, and 81% of sentences with counts of at least 20 are correct; each sentence reward lies in [−0.2, 0.1], alongside the answer-correctness reward. A judge thus improves on CorVer only if it is clearly more accurate than this, as frontier verifiers likely are, at far higher cost. We will state this positioning explicitly in the paper.

W2: Macro accuracy (Qwen3-4B/8B mean, Table 18) and reward time on the same 1,000 saved responses, including remote-verifier waiting:

MethodMacro (%)Reward time (s)Reward ratio
CorVer35.451021.00×
FSPO31.974414.32×
RLFH28.846,34462.20×
FoRAG28.766,50563.77×
KnowRL25.649,69395.03×

W2(a): CorVer targets closed-book QA: the corpus only computes training rewards, and the policy answers from the question alone (Sec. 1). Within this setting, the question is interesting and one we considered: it measures CorVer's gap to a near-ground-truth upper bound, which is hard to obtain. No dataset labels reasoning sentences, so only strong verifiers can serve; and verifiers like (i) and (ii) approach the upper bound, so we fully expect CorVer to fall short of them.

Such training also far exceeds a research budget. One Qwen3-8B run has 4,800 responses, roughly 70,000 sentences (363 tokens per response, about 25 per sentence). Three search-and-reasoning rounds per sentence mean about 210,000 billed search queries and as many frontier-model calls per run; at our measured 12 s per remote request without thinking, a lower bound for frontier models, this is about a month of sequential judge time versus 8 minutes for CorVer. We will discuss this as future work.

W2(b): We evaluate our Llama-3.1-8B policies, trained only on short-form QA, on the 183 FActScore Biography entities used by Chen et al. (2025), with FActScore-NPM and VeriScore (Exa search; Dtl = supported claims per biography; one training seed):

MethodNPM Prec. (%)NPM Supported/bioVeriScore Prec. (%)VeriScore Dtl
Raw88.4913.4230.108.72
FSPO88.2513.4231.079.15
CorVer89.0514.3331.339.21

Without long-form training, CorVer gives the most supported content at the highest precision under both evaluators; VeriScore with Parallel search gives the same ranking. CorVer also improves multi-hop QA (Table 4).

Q1: Yes: FSPO uses the evidence document paired with each training question, without retrieval. The policy sees only the question; HHEM-2.1 scores reasoning sentences against 384-token chunks (stride 320), marking a sentence supported if its maximum entailment probability exceeds 0.5.

Q2: The RLFH judge receives the question, the paired evidence and the generated sentences, splits them into atomic claims, and labels each supported, contradicted or unverifiable against the evidence; it has no retrieval or search tool. Evidence grounding reduces reliance on parametric knowledge, and hence the circularity noted in Line 432, though shared model biases remain.

Q3: Yes, gtr-t5-large, but its passages go to an NLI verifier, not GPT-4o-mini: GPT-4o-mini extracts atomic facts, gtr-t5-large retrieves five passages per fact from the title-matched Wikipedia document, and DeBERTa-v3-base-mnli-fever-anli checks support (entailment threshold 0.3).

Q4: Count magnitude reflects entity popularity and coverage, not calibrated confidence, so we map counts 0, 1–4, 5–19 and ≥20 to −0.2, −0.1, 0 and +0.1: fluctuations within a bin leave the reward unchanged, and frequent pairs get no extra bonus. The two cutoffs are hyperparameters, but a continuous mapping would also need a transformation and scale, and policy gradients do not need differentiability in the count. The bins coarsely track Figure 1 (24% vs 81% correct, W1).

AuthorsOpenReview ↗

Follow-up on the Revised PDF

Thank you again for your thoughtful suggestions. Your comments are reflected in the revised PDF (changes in blue) as follows:

  • Positioning and stronger verifiers (W1, W2(a)): §7 positions CorVer as a cheaper proxy for, not a replacement of, strong LLM verifiers; Appendix D.7 estimates the cost of frontier and agentic verifiers.
  • Accuracy versus efficiency (W2): Table 2 adds reward-only timing on identical responses.
  • Long-form benchmarks (W2(b)): §4.4 and Appendix D.5 report FActScore Biography results under FActScore-NPM and VeriScore.
  • Baseline details (Q1–Q3): Appendix A.1.1 describes the FSPO, RLFH, and KnowRL verifiers; §5.3 now notes that evidence-grounded judges reduce the circularity concern.
  • Binned counts (Q4): Appendix C explains the binning rationale.

We would be glad to discuss any remaining questions.

Reviewer_Lut8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CorVer, which converts entity co-occurrence counts into sentence-level rewards for factual QA. These rewards are assigned to corresponding tokens and combined with answer correctness and format rewards for RL training. Experiments report improved QA accuracy across several models and shorter training time.

Critical strengths
  1. The paper is easy to follow, and the reward construction is clearly described.
  2. The ablations help assessing the roles of corpus rewards, answer rewards, and token-level reward assignment in the tested setting.
Critical weaknesses
  1. Appendix D.5 clarifies that the timings compare complete training configurations. However, without generated token counts and separate timings for generation, scoring, updates, and API waiting, the contribution of each component to the speedup remains unquantified.
  2. The reasoning factuality evaluation compares only Raw and CorVer on one model and 1000 HotpotQA questions, leaving generalization across models and comparison with factuality-RL baselines untested. The default corpus reward discards the relation field. Although Table 4 reports multi-hop QA gains and Appendix C.2 studies RelCheck, these evaluations do not isolate improvements in relational reasoning from improved entity association.
Questions for authors
  1. Please report rollout token counts and component timings, including a scoring comparison with local FSPO/HHEM-2.1 on identical responses.
  2. Does the reasoning factuality improvement hold on additional models?
  3. How does CorVer compare with FSPO on the same reasoning factuality evaluation?
  4. For zero-count sentences, how often do extraction errors, surface-form differences, or missing corpus coverage explain the result? Are reward assignments stable across query windows and corpus snapshots?
  5. Does performance improve when entities are held fixed but relations, negation, or temporal conditions change?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Lut8

We thank the reviewer for the detailed comments.

W1/Q1: Our original logs did not separate these components, so we reran complete Qwen3-8B training with component timers (100 updates and 4,800 responses per run, one H100 NVL; mean over two seeds):

StageCorVer (s)FSPO (s)
Generation4,48322,528
Scoring6412,243
Updates1,0452,034
Saving125235
API waiting00
Total6,29427,040

The speedup comes from generation and reward. For generation, CorVer uses vLLM for rollouts, whereas our FSPO setting uses Transformers generation; CorVer still generates 32% more tokens (1,743,752 vs 1,324,020). For the reward, CorVer's scoring is much faster than FSPO's (641 s vs 2,243 s), and faster still than LLM-judge rewards. To hold generation fixed, we score the same 1,000 saved responses with each reward, including remote-verifier waiting (FoRAG's verifier is accessed remotely in this test):

MethodRemote waiting (%)Reward time (s)Reward ratioEnd-to-end ratio (Table 18)
CorVer01021.00×1.00×
FSPO (local HHEM-2.1)04414.32×5.52×
RLFH99.26,34462.20×10.41×
FoRAG99.56,50563.77×7.76×
KnowRL92.99,69395.03×8.77×

In sampled remote requests, network transfer takes only 1.4% of the time; server-side processing takes 98.6%, mostly the judge's token generation (81–95%), so remote waiting reflects judge inference rather than network latency.

W2/Q2/Q3: We extend Table 3 to four models and FSPO, with three training seeds and one scoring pipeline; FSPO and CorVer report mean ± SD (Table 3 is one of the CorVer seeds, hence slight differences), and Raw is a single run.

ModelMethodSupported/resp ↑Precision (%) ↑Unsupported/resp ↓
Llama-3.2-3BRaw1.29283.730.251
FSPO0.824 ± 0.02683.00 ± 1.530.169 ± 0.018
CorVer1.426 ± 0.01784.70 ± 1.440.258 ± 0.030
Qwen3-8BRaw7.17387.780.999
FSPO6.431 ± 0.20689.05 ± 0.080.791 ± 0.019
CorVer9.104 ± 0.02792.10 ± 0.160.781 ± 0.017
Qwen3-4BRaw6.06686.260.966
FSPO5.463 ± 0.03488.53 ± 0.110.708 ± 0.007
CorVer7.851 ± 0.14491.86 ± 0.170.695 ± 0.008
Llama-3.1-8BRaw1.77291.670.161
FSPO0.987 ± 0.03193.88 ± 0.550.064 ± 0.006
CorVer2.975 ± 0.05196.06 ± 0.210.122 ± 0.005

The gains hold across models: CorVer raises supported content and precision over Raw on all four and lowers unsupported content on three. Against FSPO, CorVer has more supported facts and higher precision on all four; FSPO has fewer unsupported facts on the Llama models but less supported content than even Raw. CorVer also leads on long-form FActScore Biography (our response to Reviewer A3Q8, W2(b)).

Q4: A zero count is a noisy negative signal, not proof of falsity. Among all 196 zero-count sentences (ZZ) in 1,000 early Qwen3-8B training responses, manual checks give extraction errors (EE) P(E∣Z)=4/196=2.04%P(E\mid Z)=4/196=2.04\% and surface-form differences P(S∣Z)=36/196=18.37%P(S\mid Z)=36/196=18.37\%; our 700-sentence audit (Figure 1, App. B) gives P(C∣Z)=24%P(C\mid Z)=24\% for correct claims CC. Counting every extraction error as a false penalty, and since extraction errors are unrelated to correctness,

P(false penalty∣Z)=P(E∣Z)+P(C∣Z) (1−P(E∣Z))=2.04%+23.51%≈25.6%.P(\text{false penalty}\mid Z)=P(E\mid Z)+P(C\mid Z)\,\bigl(1-P(E\mid Z)\bigr)=2.04\%+23.51\%\approx25.6\%.

This total does not depend on how often surface-form cases are correct.

Replaying 5,774 queries from 1,000 fixed responses with only the window changed:

Window (tokens)Zero count (%)Reward agreement with 1,000 (%)
5005.3392.81
1,000 (default)3.39100.00
2,0002.7495.90

Since rewards use four coarse count bins, count shifts across corpus snapshots should rarely change the bin, as the window replay suggests; a cross-snapshot replay is left for future work, and a larger pre-training corpus remains fast to query (Liu et al., 2024).

W2/Q5: Checking every relation would need a sentence-level NLI or LLM verifier, the cost CorVer avoids (4.3–95× faster reward, W1/Q1). Instead, CorVer improves both outcome and process: macro accuracy exceeds FSPO's on all four models (Table 1), and reasoning contains more supported facts at higher precision (W2/Q2/Q3), indicating a stronger reasoning chain rather than only entity association. Omitting relations is deliberate: adding relation terms lowers TriviaQA accuracy from 66.19% to 61.34% (App. C.2). On 300 true–false pairs with fixed entities (100 each for relation, negation and time; closed-book, greedy), accuracy (%) over the 600 sentences is:

ModelRawFSPOCorVer
Qwen3-8B50.6751.44 ± 0.4251.83 ± 0.29
Llama-3.2-3B42.3343.67 ± 1.4943.50 ± 1.24

Entity association cannot separate a pair, yet CorVer improves over Raw on both models (+1.16 and +1.17pp), on par with FSPO, whose NLI reward is relation-aware. The gains are small for all methods, and we will note relational reasoning as a limitation.

AuthorsOpenReview ↗

Follow-up on the Revised PDF

Thank you again for your detailed review. Your comments are reflected in the revised PDF (changes in blue) as follows:

  • Component timings (W1/Q1): Table 2 adds reward-only timing on identical responses, including local FSPO/HHEM-2.1; Appendix D.7 gives stage-wise timings with token counts.
  • Additional models and FSPO (W2/Q2/Q3): §4.4 and Table 3 report reasoning factuality on four models against FSPO with three training seeds.
  • Zero-count sentences and stability (Q4): §5.3 and Appendix B.1 give the extraction-error and surface-form analysis and the query-window replay; §7 notes corpus snapshots as future work.
  • Fixed entities with changed relations, negation, or time (Q5): Appendix C.3 describes the 300-pair diagnostic, and §7 states relational reasoning as a limitation.

We would be glad to clarify anything further before the discussion period ends.

Reviewer_QajbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CorVer, a lightweight training-time process reward for factual question answering. A small extractor identifies subject–object pairs, corpus co-occurrence counts provide noisy sentence-level signals, and these signals are aligned to generated tokens and combined with answer-correctness rewards for reinforcement learning. The experiments cover six models, five standard QA benchmarks, additional multi-hop tasks, factuality-RL baselines, and reward ablations. The main contribution is a practical accuracy–training-cost tradeoff; co-occurrence is appropriately treated as weak supervision rather than a certificate of factual truth.

Critical strengths
  1. The practical training contribution has broad empirical support. CorVer improves over the unmodified models across all 30 reported standard QA settings, and the comparisons with factuality-RL baselines and the reward ablations support the usefulness of the corpus signal and token alignment (Tables 1 and 5). The complete-training comparison with local FSPO is particularly relevant to the efficiency claim. These results support a useful low-cost alternative for factual-QA post-training.

  2. The paper makes a serious effort to test alternative explanations. It includes baseline retuning, one-way answer matching, semantic rescoring, and additional triplet/relation-word controls (Appendices C and D). The semantic checks still favor CorVer over FSPO. Although their coverage is limited, these checks strengthen the central result and the paper explicitly acknowledges the limitations of co-occurrence.

Critical weaknesses
  1. Evidence for improved reasoning factuality is much narrower than the final-answer evidence. Table 3 evaluates one model on one setting and shows more supported content and a modest increase in factual precision; it does not establish fewer unsupported facts per response. Entity co-occurrence also cannot distinguish a correct relation from an incorrect relation between the same entities. This limits the process-factuality interpretation, although it does not invalidate the narrower weak-supervision contribution.

  2. Measurement robustness and training variability are not yet established at the breadth of the headline results. The semantic rescoring and one-way matching checks are valuable but concentrated on one model and limited subsets, while training uses one seed per configuration. Small margins are therefore difficult to assess. The reported speedups should also remain explicitly tied to the complete training configurations, whose generation and update workloads differ, rather than be interpreted as equal-work verifier speedups. These limitations temper my recommendation but do not outweigh the practical contribution.

Questions for authors
  1. Clarification: For Table 3, can you report paired uncertainty for supported content, factual precision, and unsupported facts per response, and clarify which process-factuality claims these measurements support?

  2. Evidence that could strengthen the recommendation: Do the main gains over FSPO persist under broader semantic scoring, especially on SimpleQA and TruthfulQA, and across additional training seeds for a representative model?

  3. Clarification: Can you summarize the workload differences in the complete-training timing comparison alongside the headline speedups, including rollout quantities, update schedules, and remote-verifier waiting time?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Qajb

We thank the reviewer for the constructive comments.

W1/Q1: We extend the Table 3 setting to four models with three training seeds; all methods are scored by the same pipeline. FSPO and CorVer report mean ± SD over seeds; for Llama-3.2-3B, Table 3 is one of the three CorVer seeds, so the mean differs slightly. Raw involves no training and is reported from a single run on the same questions, so the paired difference CorVer − Raw has the same SD as CorVer.

ModelMethodSupported/resp ↑Precision (%) ↑Unsupported/resp ↓
Llama-3.2-3BRaw1.29283.730.251
FSPO0.824 ± 0.02683.00 ± 1.530.169 ± 0.018
CorVer1.426 ± 0.01784.70 ± 1.440.258 ± 0.030
Qwen3-8BRaw7.17387.780.999
FSPO6.431 ± 0.20689.05 ± 0.080.791 ± 0.019
CorVer9.104 ± 0.02792.10 ± 0.160.781 ± 0.017
Qwen3-4BRaw6.06686.260.966
FSPO5.463 ± 0.03488.53 ± 0.110.708 ± 0.007
CorVer7.851 ± 0.14491.86 ± 0.170.695 ± 0.008
Llama-3.1-8BRaw1.77291.670.161
FSPO0.987 ± 0.03193.88 ± 0.550.064 ± 0.006
CorVer2.975 ± 0.05196.06 ± 0.210.122 ± 0.005

These results support three claims: CorVer (i) increases supported content on all four models; (ii) attains the highest precision on all four, although the Llama-3.2-3B gain is small; and (iii) relative to Raw, reduces unsupported content on three models and keeps it comparable on Llama-3.2-3B. FSPO has fewer unsupported facts on the two Llama models, but its supported content falls below Raw on all four models. We therefore do not claim fewer unsupported facts than every baseline, and will narrow L324–325 accordingly. On long-form FActScore Biography, without long-form training, CorVer also gives the most supported facts at the highest precision under both FActScore-NPM and VeriScore (our response to Reviewer A3Q8, W2(b)).

Co-occurrence does not verify relations, and omitting them is a deliberate choice. Following QuCo-RAG (Min et al., 2025, §3.3), we query only head and tail entities because relational predicates vary lexically, whereas named entities are stable; adding relation terms lowers Llama-3.2-3B TriviaQA accuracy from 66.19% to 61.34% (App. C.2). The corpus reward is a step-level weak signal combined with the answer-correctness reward, which penalizes relation errors that change the answer; the two are complementary. The evaluator above also scores complete atomic claims, including relations, independently of co-occurrence. On fixed-entity true–false pairs, CorVer still improves slightly over Raw (our response to Reviewer Lut8, W2/Q5).

W2/Q2: We evaluate Qwen3-8B on the full SimpleQA (4,326) and TruthfulQA (817) sets with the Table 1 lexical rule and a semantic judge (Qwen3.8-Flash-Next-FP8). Values are mean ± SD over three training seeds. Accuracy (%):

MethodTruthfulQA (lexical)TruthfulQA (semantic)SimpleQA (lexical)SimpleQA (semantic)
Raw42.7242.843.013.31
FSPO48.88 ± 1.2750.06 ± 1.443.94 ± 0.074.23 ± 0.10
CorVer51.73 ± 0.1456.71 ± 0.395.19 ± 0.284.94 ± 0.15

CorVer remains ahead of FSPO in every seed under both rules on both datasets.

W2/Q3: Workload of the Table 18 runs (one H100 80GB), with reward-only time measured by scoring the same 1,000 saved responses with each reward, including remote-verifier waiting (RLFH, KnowRL, and FoRAG, whose verifier is accessed remotely in this test):

MethodResponses (4B / 8B)UpdatesRemote waiting (%)Reward time, 1,000 responses (s)Reward ratioEnd-to-end ratio
CorVer6,400 / 4,800100 GRPO01021.00×1.00×
FSPO6,400 / 4,800100 GRPO04414.32×5.52×
RLFH6,400 / 4,800100 PPO × 4 epochs99.26,34462.20×10.41×
FoRAG6,400 / 400100 PPO × 4 epochs99.56,50563.77×7.76×
KnowRL6,400 / 4,800100 GRPO92.99,69395.03×8.77×

The end-to-end ratio is complete training time (initialization, generation, rewards, updates, saving) over both models relative to CorVer. Qwen3-8B FoRAG uses its more accurate 400-response allocation (App. D.3). We agree that end-to-end ratios are not verifier speedups: CorVer uses vLLM for rollout generation, whereas our FSPO setting uses Transformers generation, so these ratios also reflect generation speed. The reward-only column isolates reward computation at equal work. In sampled remote requests, network transfer takes only 1.4% of the time on average and server-side processing 98.6%, mostly the judge's token generation (81–95%), so remote waiting reflects judge inference rather than network latency. New Qwen3-8B runs on an H100 NVL (two seeds) give 4.30× (6,294 s vs 27,040 s for FSPO; stage-wise breakdown in our response to Reviewer Lut8, W1/Q1).

We will report these workload details and the reward-only timing next to the headline speedups.

AuthorsOpenReview ↗

Follow-up on the Revised PDF

Thank you again for your review. Your comments are reflected in the revised PDF (changes in blue) as follows:

  • Reasoning factuality (W1/Q1): §4.4 and Table 3 extend the evaluation to four models and FSPO with three training seeds; the claims are narrowed accordingly, and §4.4 adds long-form biography transfer (Appendix D.5).
  • Relations (W1): §7 states relational reasoning as a limitation, supported by the fixed-entity diagnostic in Appendix C.3.
  • Semantic scoring and seeds (W2/Q2): §4.1 and Appendix D.4 report full-set SimpleQA and TruthfulQA under the lexical rule and a semantic judge with three training seeds.
  • Workload and speedups (W2/Q3): the Abstract and §4.3 tie the speedups to complete training configurations; Table 2 adds reward-only timing, and Appendices D.6–D.7 give the workload and stage-wise breakdown.

We would be glad to clarify anything further before the discussion period ends.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their feedback. We have uploaded a revised PDF in which all changes appear in blue; the main text uses the additional page. Table, appendix, and line numbers cited in our individual responses refer to the original submission.

Main text

  • Speedups and workload (Abstract, contributions, §4.3, Table 2). Complete-training speedups are now explicitly tied to full training configurations. Table 2 adds reward-only time on the same 1,000 saved responses (4.3–95× faster), and §4.3 notes the vLLM versus Transformers generation difference. (Reviewer Qajb W2/Q3; Reviewer Lut8 W1/Q1; Reviewer A3Q8 W2)
  • Reasoning factuality (§4.4, Table 3). Extended to four models and FSPO with three training seeds; the claims are narrowed to what these measurements support (former L324–325). We add long-form FActScore Biography transfer under FActScore-NPM and VeriScore. (Reviewer Qajb W1/Q1; Reviewer Lut8 W2/Q2/Q3; Reviewer A3Q8 W2(b))
  • Answer-scoring robustness (§4.1). Full SimpleQA and TruthfulQA with three training seeds, under the lexical rule and a semantic judge. (Reviewer Qajb W2/Q2)
  • Reward noise and circularity (§5.3). Zero-count error analysis; the circularity discussion now acknowledges evidence-grounded judges. (Reviewer Lut8 Q4; Reviewer A3Q8 Q2)
  • Limitations (§7). Corpus snapshots, relational reasoning, and the positioning of CorVer as a cheaper proxy for, not a replacement of, strong LLM verifiers. (Reviewer Lut8 Q4/Q5; Reviewer A3Q8 W1/W2(a))

Appendix

  • A.1.1: implementation details of FSPO, RLFH, and KnowRL (Reviewer A3Q8 Q1–Q3); we also clarify that FoRAG's verifier runs on a remote endpoint, with matching edits in D.6 and §4.3 (reported timings unchanged).
  • B.1: zero-count error decomposition and query-window replay. (Reviewer Lut8 Q4)
  • C: rationale for binned counts (Reviewer A3Q8 Q4); C.3: fixed-entity relation, negation, and time pairs. (Reviewer Lut8 Q5)
  • D.4: full-set SimpleQA/TruthfulQA results (Reviewer Qajb W2/Q2); D.5: long-form biography results (Reviewer A3Q8 W2(b)).
  • D.7: reward-only timing, stage-wise H100 NVL breakdown with token counts, and cost estimates for stronger verifiers. (Reviewer Qajb Q3; Reviewer Lut8 W1/Q1; Reviewer A3Q8 W2(a))

AI use statement. Updated to disclose the AI checks used to verify the small diagnostic set in Appendix C.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.