acceptodds
Under review as a conference paper at ICLR 2027

DRAFT: Disentangling Retrieval and Abstention via Factual Post-Training

Abstract

Factual hallucination in large language models (LLMs) manifests along two axes: the model may fail to a fact it has been taught, or it may an answer for a fact it was never taught. Reducing hallucination therefore requires two complementary skills: accurate retrieval and appropriate abstention. The standard approach is post-training, where supervised fine-tuning (SFT) embeds knowledge and reinforcement learning (RL) aligns behavior, yet how these stages interact for factuality is not well understood. We present a systematic study under a controlled setup that probes a model's knowledge boundary and isolates SFT and RL effects on retrieval and abstention within one pipeline. We find: RL with binary rewards reliably teaches abstention by enforcing a designated answer/abstain boundary, and the form of SFT data determines what RL can subsequently improve. Together, these findings reveal that RL acts as elicitation over SFT-encoded representations rather than acquiring new knowledge. SFT data construction determines what the model can know, while RL determines how reliably it retrieves or abstains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Reinforcement Learning, Factuality, Knowledge Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PzduOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the distinct roles of supervised fine-tuning (SFT) and reinforcement learning (RL) in factual knowledge retrieval and abstention during LLM post-training. The authors introduce a controlled experimental framework based on SimpleQA and Wikipedia. They use Pass@32 to identify facts that the base model cannot reliably answer and construct the RichQA dataset to systematically analyze the effects of SFT data formats and RL reward designs.

Critical strengths
  1. The paper distinguishes factual hallucinations into two dimensions: knowledge retrieval failures and inappropriate answering. It systematically investigates the roles of SFT and RL within a unified framework. By controlling knowledge boundaries, training data formats, and reward mechanisms, the experiments provide valuable insights into the factors influencing factual post-training. This mechanistic perspective is more informative than simply proposing another training method.

  2. The paper shows that higher post-SFT accuracy does not necessarily lead to larger RL gains, while atomic knowledge representations can increase the relative improvement from RL even when they reduce SFT accuracy. Furthermore, different reward mechanisms produce substantially different abstention behaviors. These findings offer useful guidance for factual training data construction and post-training strategy design.

Critical weaknesses
  1. The paper argues that RL primarily elicits knowledge already encoded during SFT rather than acquiring new knowledge. However, this conclusion is mainly inferred from accuracy changes, training and validation curves, and RL gains under different SFT data formats. The experiments do not directly establish whether the corresponding facts are already encoded in the model's internal representations. Therefore, the current evidence supports a behavioral interpretation but is insufficient to establish a strict mechanistic distinction between knowledge encoding and knowledge retrieval.

  2. The main experiments focus on SimpleQA and the derived RichQA dataset, leaving the applicability of the findings to more complex knowledge types and practical post-training scenarios insufficiently explored. Moreover, Table 2 shows that although atomic decomposition increases the relative gains from RL, all clean decomposition settings achieve lower final accuracy than the original RichQA baseline. This limits the practical advantage of atomic knowledge representations and highlights the need to distinguish mechanistic findings from actual end-to-end performance improvements.

Questions for authors
  1. Could the authors provide more direct evidence that RL improves the retrieval of already encoded knowledge rather than merely improving adaptation to specific question-answering formats? For example, could experiments involving facts excluded from SFT, alternative question formulations, or knowledge probing help distinguish knowledge acquisition from knowledge retrieval?

  2. Both Table 2 and the cross-model experiments in the appendix show that clean decomposition settings yield larger RL gains but still underperform the RichQA baseline in final accuracy. Could the authors further explain this phenomenon and investigate whether combining paraphrasing augmentation with atomic decomposition could improve both final accuracy and RL effectiveness?

  3. The abstention experiments construct knowledge boundaries by randomly assigning 50% of the facts as abstention targets. Could the authors clarify how these artificially designated abstention boundaries correspond to facts that the model genuinely does not know? To what extent does this experimental setup support the claim that the model learns to recognize its own knowledge boundaries?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Pzdu (1/3)

Thank you for the thoughtful review and for recognizing the value of separating retrieval failures from inappropriate answering, and the guidance our findings give for constructing factual training data. Below we add more direct evidence for the elicitation claim, explain the lower final accuracy of clean decomposition, and clarify the abstention boundary.

Terms. Held-out questions (and held-out F1) are the 550 SimpleQA questions used for our main evaluation, as in the paper. Their facts are taught during SFT, as RichQA passages, but the questions themselves are excluded from RL training. Never-seen facts are a new set of 243 facts excluded from both SFT and RL.

W1, Q1: More direct evidence that RL retrieves stored knowledge

Our evidence is behavioral: we measure what the model answers, not its internal representations. Within that scope, we ran the three tests the reviewer suggests. All three point the same way: RL improves the retrieval of facts that SFT put into the model, and it does not improve retrieval of facts that SFT did not.

1. Facts excluded from SFT. We test this in two ways. First, we built a set of 243 never-seen facts: SimpleQA facts that appear in no SFT or RL data, that the base model never answers correctly in 32 attempts, and whose answer string appears nowhere in the training text. None of our models answers a single one of them correctly, before or after RL (0.0% correct for the SFT, binary RL, and ternary RL models). This indicates that RL does not teach facts that were not seen during SFT. Second, we ran the same RL directly on the base model, with no SFT. Held-out F1 rises by only 0.2 (vs. +4.0 after RichQA SFT). Even on the facts RL itself trains on, F1 rises only from 0.0 to 6.6, compared with 41.0 to 54.2 (+13.2) when RL starts from RichQA SFT. RL alone therefore adds almost no new facts, even when it is trained on them directly.

2. Alternative question formulations. SFT teaches each fact as a context-rich RichQA passage, while evaluation asks the original short SimpleQA question, which appears in neither SFT nor RL training. The RL gain is therefore measured on a different format and on questions RL never sees. RL does train on SimpleQA-style questions for other facts, so it could in principle improve only the answer format. However, the base-model RL-only experiment rules this out: it receives the same format training, yet gains only +0.2 F1. The gain requires the facts to have been introduced by SFT.

3. Knowledge probing. For each held-out question, we sample 16 answers from the post-SFT model and call the fact stored if at least one is correct. We then take the questions the post-SFT model answers incorrectly (with greedy decoding), which are the ones RL could fix, and look at the proportion of these questions that are stored. Then, we look at the questions RL fixes and ask how many were stored. If RL fixed facts at random, the proportion of stored facts it fixes would match the proportion of stored facts among all incorrectly answered questions.

SFT substrateRL F1 gainStored, among all facts wrong after SFTStored, among the facts RL fixed
RichQA (30 epochs)+3.028% (85/304)73% (19/26)
Clean mixture p=25% (20 epochs)+11.320% (79/390)56% (42/75)
Clean mixture p=25% (2 epochs)+0.715% (78/511)76% (13/17)

RL F1 gains are from this re-evaluation, graded with gpt-6-luna (the paper reports +4.0 for RichQA with gpt-4o-mini). We use gpt-6-luna because gpt-4o-mini is no longer available to us. On the paper's own saved outputs for Table 6, the two judges give the same grade on 97.3% of items, and the new judge is slightly stricter on correctness.

In every run, the facts RL fixes are far more likely to be stored than a typical incorrectly answered fact (73% vs. 28%, 56% vs. 20%, and 76% vs. 15%, i.e., 2.6-5 times the base rate). RL therefore selectively improves the facts that SFT already made retrievable. The remaining fixes in the 20-epoch run may be facts stored too weakly to appear in 16 samples. The choice of 16 samples does not drive this result: using fewer samples (k = 1, 2, 4, 8) lowers both shares together, and for the two larger runs the fixed facts remain 2.2-2.8 times more likely to be stored at every k.

Together, these results show that RL gains disappear when SFT did not introduce the facts, hold across a change of question format, and concentrate on facts SFT already made retrievable. We will state our claim in these behavioral terms (RL improves retrieval of facts made accessible by SFT) and avoid wording that implies a claim about internal representations.

AuthorsOpenReview ↗

Response to Reviewer Pzdu (2/3)

W2, Q2: Scope, and the lower final accuracy of clean decomposition

Other models. The main patterns reproduce on a larger model and a second model family:

BackboneRichQA: post-SFT → post-RLClean mixture (p=25%): post-SFT → post-RL
Qwen2.5-3B-Instruct43.3 → 47.3 (+4.0)29.1 → 40.4 (+11.3)
Qwen2.5-7B-Instruct49.2 → 53.2 (+4.0)36.4 → 49.5 (+13.1)
Llama-3.1-8B-Instruct50.8 → 53.1 (+2.3)30.0 → 47.2 (+17.2)

The full set of SFT conditions also reproduces on both larger backbones. Each table is sorted by post-SFT F1:

7B SFT conditionPost-SFT F1Post-RL F1Δ\Delta
Q+A paraphrase (k=3)59.960.3+0.5
Decomposition, unfiltered59.566.2+6.8
RichQA (baseline)49.253.2+4.0
Clean mixture (p=25%)36.449.5+13.1
Clean mixture (p=50%)31.744.8+13.1
Clean decomposition22.927.3+4.4
Clean mixture (p=100%)20.832.5+11.7
Llama-3.1-8B SFT conditionPost-SFT F1Post-RL F1Δ\Delta
Q+A paraphrase (k=3)60.464.6+4.2
Decomposition, unfiltered54.160.2+6.2
RichQA (baseline)50.853.1+2.3
Clean mixture (p=25%)30.047.2+17.2
Clean mixture (p=50%)23.342.7+19.4
Clean mixture (p=100%)16.441.6+25.1
Clean decomposition14.820.6+5.7

Q+A paraphrasing reaches the highest post-SFT F1 but gains less from RL, while the clean mixtures start lower and gain the most. This is the same pattern as at 3B.

Why SimpleQA, and how far the findings apply. We study atomic, single-hop facts on purpose. They are the basic unit that more complex factual tasks, such as multi-hop questions and long-form answers, are built from, and they let us control exactly which facts the model knows before training. SimpleQA (Wei et al., 2024) is the standard benchmark for this setting. Its 4,326 short, fact-seeking questions cover a wide range of topics (most commonly science and technology, politics, and art), and its grading labels each answer as Correct, Incorrect, or Not Attempted, which lets us measure retrieval and abstention on the same questions. It also remains in active use for frontier models: Google DeepMind's SimpleQA Verified (Haas et al., 2025), a cleaned version of SimpleQA, is used to compare frontier models such as Gemini 2.5 Pro and GPT-5 on a public leaderboard. Our findings therefore concern a factuality problem that is widely measured in practice, not an isolated synthetic task.

The setup also mirrors how knowledge is added in practical post-training, where new facts usually arrive as documents rather than as the exact questions users will ask. Our SFT data (RichQA) presents each fact as a context-rich passage built from its Wikipedia evidence, while evaluation uses the original short SimpleQA question, which the model never trains on. Our results therefore measure whether a model can retrieve a fact in a form different from the one it learned it in.

The main patterns are also not specific to one model: the RichQA and clean-mixture results reproduce on Qwen2.5-7B and Llama-3.1-8B (tables above). In the revision, we will state the scope explicitly: our claims concern atomic factual retrieval and abstention, and extending the analysis to multi-hop and long-form knowledge is a natural next step.

Why clean decomposition ends lower. Our goal is to analyze which factual data forms work and which do not, not to propose decomposition as a better recipe. That clean decomposition ends lower is therefore a finding rather than a failure, and it shows how nuanced factual accuracy is. On 3B, the clean mixture (p=25%) starts much lower after SFT than RichQA (29.1 vs. 43.3), gains nearly three times as much from RL (+11.3 vs. +4.0), and still ends lower (40.4 vs. 47.3). So the three quantities one might optimize, post-SFT accuracy, RL gain, and final accuracy, do not move together: post-SFT accuracy does not predict how much RL can add, and a large RL gain does not imply the best final model. A training intervention therefore cannot be judged by its effect on one stage alone. In the revision, we will state clearly which of our findings concern what limits RL and which concern what produces the best final model.

AuthorsOpenReview ↗

Response to Reviewer Pzdu (3/3)

Our per-question probing (W1) shows where the difference comes from. We call a fact stored if at least one of 16 post-SFT samples answers it correctly. After RichQA SFT, 60% of all 550 held-out facts are stored, compared with 43% after the clean mixture, so the clean mixture's ceiling is lower and it ends lower. Yet the two models have a similar number of facts that are stored but answered incorrectly with greedy decoding (85 vs. 79), and RL turns these into correct answers more than twice as often after the clean mixture (53% vs. 22%). The clean mixture therefore stores fewer facts, but RL draws out the facts it does store much more effectively.

Combining paraphrasing with decomposition. Our clean mixtures already combine two forms of the same facts: each fact appears as one context-rich RichQA passage together with atomic QA pairs. They gain the most from RL but do not surpass RichQA's final accuracy (40.4 vs. 47.3 on 3B). The combination the reviewer proposes, Q+A paraphrasing augmentation together with clean decomposition, would add more varied phrasings of each fact on top of this. We agree it is a natural way to try to get both a high final accuracy and a large RL gain, and we see it as a promising direction for future work.

Q3: How the random 50/50 abstention split relates to facts the model truly does not know

The random split is applied before training, as a controlled intervention. All facts start in the same pool that the base model cannot answer (Pass@32 = 0), so the two groups are matched in difficulty. For answer-target facts, SFT teaches the RichQA facts. For abstention-target facts, the facts are left out and replaced by a refusal. So the split decides which facts enter the model's knowledge during post-training, and randomization removes differences in topic and difficulty between the groups.

So "knowledge boundary" in our paper means the boundary created by post-training. We will define it this way, and remove wording suggesting the model discovers its own boundary from nothing.

To test the real-world case, we built a set of 243 never-seen facts: SimpleQA facts that the base model never answers correctly in 32 tries, that appear in no SFT or RL data, and whose answer string appears nowhere in the training text. No model answers any of them correctly. Abstention on them rises from 37.0% for the base model and 54.7% after SFT to 71.6% after binary RL and 87.2% after ternary RL (gpt-6-luna judge), and wrong answers fall from 44.4% after SFT to 28.4% after binary RL. So the learned abstention reaches facts outside the designed split, which the model genuinely does not know.

Reviewer_uZsSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how knowledge is injected into and retrieved from an LLM, and how to teach the model to abstain, using SFT and RL.

Critical strengths

An informative experiment shows that RL gain and final accuracy can point in opposite directions.

Critical weaknesses
  1. The main conclusion, that RL only draws out knowledge SFT put in, has two simpler explanations that the paper does not rule out.
  • RL may be repairing damage from SFT. SFT trains for 20 epochs on facts the model "didn't know". Prior work (Gekhman et al., 2024) shows that this increases hallucination, and here it also substantially hurts general ability: MMLU drops from 64.4 to 38.5. The SFT mixtures that gain more from RL start 13–34 points lower after SFT and still end below plain RichQA. "RL fixes what SFT broke" may also explain this.
  • Some "new" facts were already known. A fact counts as new if the base model gets it wrong in 32 tries. The paper states that under 1% are actually known, but its own appendix shows that 64 tries catch another 4.0% (App. A.1.1). Part of the gain may therefore come from what the base model already knew, not from SFT.
  1. The abstention result largely repeats what SFT and earlier work already achieve.
  • SFT does most of the work. After SFT, the model already abstains far more on the questions labeled "I don't know" (a 57.8-point gap), and RL barely widens the gap (60.4). RL raises abstention on both kinds of questions, including ones the model should answer.
  • The reward comparison repeats TruthRL. The binary and ternary rewards are effectively TruthRL's, and TruthRL found the same trade-off: the ternary reward cuts hallucinations but also cuts correct answers. Scored the way TruthRL scores it, the ternary reward wins here too.
  • Teaching "I don't know" with SFT is also known (R-Tuning; Gekhman et al.).
Questions for authors
  1. Fitting new facts by SFT to convergence is known to increase hallucination (Gekhman et al., 2024), while stopping early mostly avoids learning them. Could the authors explain why they chose to train to convergence?
  2. Does the learned abstention generalize to facts the model truly does not know?
  3. What findings does this paper offer beyond existing work (Gekhman et al., TruthRL, R-Tuning)?
8 comments
AuthorsOpenReview ↗

Response to Reviewer uZsS (1/5)

Thank you for the detailed review and for highlighting that RL gain and final accuracy can point in opposite directions. Below we test the two alternative explanations you raise, clarify what RL adds to abstention, and position our reward relative to TruthRL.

Terms. Held-out questions (and held-out F1) are the 550 SimpleQA questions used for our main evaluation, as in the paper. Their facts are taught during SFT, as RichQA passages, but the questions themselves are excluded from RL training. Never-seen facts are a new set of 243 facts excluded from both SFT and RL.

W1: RL may be repairing damage from SFT

Recovery from SFT damage. The alternative explanation proposed is that RL's gains reflect recovery of abilities degraded by SFT, rather than retrieval of facts learned during SFT. Following the paper, we use MMLU as our measure of general capability (the base model scores 64.4). If RL's gains came mainly from repairing SFT damage, we would expect two things: a model more degraded by SFT should benefit more from RL, and gains in RL F1 performance should be accompanied by a recovery in MMLU. We test both predictions on RichQA and the clean mixture (p=25%), which start from the same base model and use the same RL procedure.

SFT dataMMLU after SFTMMLU after RLHeld-out F1 after SFTHeld-out F1 after RLRL gain (F1)
RichQA38.545.643.347.3+4.0
Clean mixture (p=25%)47.743.429.140.4+11.3

Neither prediction holds. The clean mixture is less degraded by SFT than RichQA (MMLU 47.7 vs. 38.5), so we should see a smaller RL gain; instead, it gains nearly three times as much on held-out facts (+11.3 vs. +4.0 F1). Moreover, RL lowers the clean mixture's MMLU from 47.7 to 43.4 while raising its held-out F1 by 11.3 points, so the largest RL gain occurs while general capability declines rather than recovers.

Varying SFT length. The comparison above changes the data format as well as the amount of SFT damage. To hold the data fixed, we also train the clean mixture for 2, 10, and 20 SFT epochs (each with its own full learning-rate schedule) and apply the same RL to each.

SFT epochs (p=25%)MMLU after SFTMMLU after RLHeld-out F1 after SFTHeld-out F1 after RLRL gain (F1)
256.658.85.96.6+0.7
1053.053.825.437.3+11.9
20 (paper)47.743.429.140.4+11.3

The 10-epoch model is less degraded than the 20-epoch model (MMLU 53.0 vs. 47.7), yet RL gives it a comparable gain (+11.9 vs. +11.3 F1), and its MMLU stays nearly unchanged during RL (53.0 → 53.8). The 2-epoch model is the least degraded, but it has learned few of the facts (held-out F1 5.9 after SFT), and RL gains almost nothing (+0.7). The RL gain therefore follows how many facts SFT taught, not how much SFT degraded the model.

Factual recall outside the training data. MMLU measures general capability, but not open-ended factual recall, which is the task RL trains on. RL could therefore repair the model's ability to answer factual questions without this appearing in MMLU. To test this, we evaluate the RichQA models on PopQA and TriviaQA, which are open-ended factual QA datasets. Their facts appear in neither the SFT nor the RL data, so any RL gain on them would have to come from repair.

BenchmarkBaseAfter RichQA SFTAfter RL
PopQA18.79.310.6
TriviaQA56.519.521.6

SFT lowers PopQA from 18.7 to 9.3 and TriviaQA from 56.5 to 19.5, and RL recovers only 1.3 and 2.1 points of these drops. RL therefore does little to repair factual recall in general; its gains are concentrated on the held-out facts that were taught in SFT.

RL depends on the facts taught in SFT. Finally, we run the same RL directly on the base model, with no SFT. Held-out F1 changes from 0.0 to 0.2, compared with +4.0 after RichQA SFT. Since the held-out facts never appear in RL training, RL alone cannot produce the gain; it requires the facts the model learned during SFT.

AuthorsOpenReview ↗

Response to Reviewer uZsS (2/5)

RL fixes the facts SFT stored, not random ones. If RL were mainly repairing general damage from SFT, its fixes should be spread over incorrectly answered questions regardless of what SFT stored. We test this with per-question probing. For each held-out question, we sample 16 answers from the post-SFT model (temperature 1.0, top-p 0.9) and call the fact stored if at least one is correct. We then take the questions the post-SFT model answers incorrectly (with greedy decoding), which are the ones RL could fix, and ask: among the questions RL fixes, how many were stored? If RL fixed facts at random, this share would match the share of stored facts among all incorrectly answered questions.

SFT substrateRL F1 gainStored, among all facts wrong after SFTStored, among the facts RL fixed
RichQA (30 epochs)+3.028% (85/304)73% (19/26)
Clean mixture p=25% (20 epochs)+11.320% (79/390)56% (42/75)
Clean mixture p=25% (2 epochs)+0.715% (78/511)76% (13/17)

RL F1 gains are from this re-evaluation, graded with gpt-6-luna (the paper reports +4.0 for RichQA with gpt-4o-mini).

In every run, the facts RL fixes are far more likely to be stored than a typical incorrectly answered fact (73% vs. 28%, 56% vs. 20%, and 76% vs. 15%, i.e., 2.6-5 times the base rate). The choice of 16 samples does not drive this result: using fewer samples (k = 1, 2, 4, 8) lowers both shares together, and for the two larger runs the fixed facts remain 2.2-2.8 times more likely to be stored at every k. This is the opposite of what general repair predicts: RL selectively improves the facts that SFT already made retrievable.

Together, these results indicate that repair of SFT damage explains little of RL's gain on held-out facts. The gain does not track the amount of SFT damage, the largest gains occur without recovery of general capability, RL recovers little factual recall outside the training data, the gain requires the facts taught in SFT, and RL's fixes concentrate on the facts SFT stored.

The gain is reliable. Four random seeds of the RichQA RL run give post-RL F1 of 47.9 ± 0.33, a gain of +3.15 ± 0.34. Four RL algorithms (GRPO, Reinforce++, SPIN-DPO, DAPO) all reach 46.6-47.3.

The clean mixtures still end below RichQA in final accuracy. Our claim concerns what limits RL, not which mixture gives the best final model, and we will state this distinction clearly in the revision.

W1: Some "new" facts may already be known

We reconstructed RichQA on the stricter Pass@64 split and reran the matched SFT and RL experiment:

SettingPost-SFT F1Post-RL F1Δ\Delta
Pass@32 (paper)43.347.3+4.0
Pass@64 (stricter)45.948.0+2.1

The RL gain stays positive with the stricter filter, and the final F1 is about the same. We also re-graded the saved Pass@32 samples with a stronger judge: only 3 of 200 facts labeled unknown were re-labeled as known (1.5%). So the gain is not explained by facts the base model already knew.

AuthorsOpenReview ↗

Response to Reviewer uZsS (3/5)

W2: The abstention result largely repeats SFT and prior work

SFT does most of the work. SFT does create most of the separation between answer and abstention targets on the trained facts, and RL's added value shows most clearly on never-seen facts, which are excluded from both SFT and RL. On the 243 never-seen facts (see Q2 below), binary RL raises abstention from 54.7% to 71.6% and lowers hallucination from 44.4% to 28.4%. So RL's contribution is largest exactly where SFT alone generalizes worst. We also agree that RL raises abstention on answer targets (20% → 29%), and we will report this cost clearly.

Relation to TruthRL's trade-off. We agree that our ternary reward is TruthRL's, and that by TruthRL's truthfulness score, which counts a wrong answer as worse than a missed correct one, the ternary reward also scores higher here. Our new analysis shows what lies behind this score. The ternary model answers far fewer answer targets than the binary model (40% vs. 71%), so its fewer wrong answers could come either from better judgment or simply from answering less. To tell these apart, we compare the two models at matched coverage, i.e., when both answer the same fraction of answer targets.

To match coverage, we make each model more cautious after training. For each question, we sample 8 answers and measure the model's confidence as the number that agree with its greedy answer (0 to 8). The model answers only when its confidence reaches a threshold and abstains otherwise. For each model, we choose the thresholds that yield roughly 20%, 30%, and 40% coverage on the answer targets.

Matched coverageBinary RLTernary RL
~20%21.0% / 85.7% / 99.6%19.9% / 84.9% / 99.2%
~30%28.1% / 81.3% / 98.9%29.2% / 71.8% / 98.1%
~40%38.6% / 74.8% / 97.8%40.4% / 61.1% / 97.3%

Each cell reports coverage on answer targets / precision on answer targets (correct answers / attempted answers) / abstention rate on abstention targets. The ternary model at ~40% coverage is evaluated as trained, without a threshold.

At the same coverage, binary RL is 13.7 points more precise at ~40% and 9.5 points at ~30%, and similar at ~20% (85.7% vs. 84.9%), while both models abstain on 97-100% of abstention targets at every level. So the ternary reward's lower error rate comes from answering fewer questions, not from answering or abstaining more accurately. This is not visible in TruthRL's evaluation, which does not compare rewards at matched coverage, and whose binary baseline (+1 correct, -1 otherwise) gives no credit for abstention. Our binary reward instead rewards abstention on facts labeled as abstention targets, and in this setting it keeps more correct answers at the same answer rate with the same abstention quality.

Q1: Why train SFT to convergence?

We originally trained SFT to convergence to establish a consistent setting in which factual learning had largely plateaued before applying RL. Held-out F1 during SFT shows that the model continues learning facts until approximately epoch 20:

SFT epoch~2~5~7~12~2130
Qwen2.5-3B, RichQA, held-out F110.823.334.241.846.245.1

Logged during a separate 30-epoch RichQA run, so values differ from the 43.3 in the main comparison.

However, convergence is not necessarily required for substantial RL improvements. To investigate this, we additionally trained clean-mixture models with shorter SFT schedules (2, 10, and 20 epochs), each followed by binary RL:

SFT epochsPost-SFT MMLUPost-RL MMLUPost-SFT F1Post-RL F1RL gain
256.658.85.96.6+0.7
1053.053.825.437.3+11.9
2047.743.429.140.4+11.3

At two epochs, factual retrieval and subsequent RL improvements remain limited. By ten epochs, however, RL achieves a substantial +11.9-point improvement, comparable to the 20-epoch model (+11.3), while retaining higher MMLU accuracy (53.0 vs. 47.7). These results demonstrate that while sufficient factual learning during SFT is important, convergence is not necessary for substantial downstream RL gains. Our original convergence-based setup enables systematic comparison across SFT configurations, and the new experiments show that the central SFT–RL interaction persists under shorter training schedules.

Reviewer_SLtZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper studies an important problem: improving factual recall while avoiding incorrect answers through appropriate abstention. Evaluating these behaviors separately is useful.
  2. The controlled framework enables comparisons across SFT data formats and RL objectives. Evaluation questions are excluded from RL training, allowing the authors to study transfer to facts previously supplied through SFT.
Critical strengths
  1. The paper studies an important problem: improving factual recall while avoiding incorrect answers through appropriate abstention. Evaluating these behaviors separately is useful.

  2. The controlled framework enables comparisons across SFT data formats and RL objectives. Evaluation questions are excluded from RL training, allowing the authors to study transfer to facts previously supplied through SFT.

Critical weaknesses
  1. The contribution relative to prior work is insufficiently established. Prior work already demonstrates improvements in factual reliability and abstention through SFT and RL. Examples include R-Tuning [1], SaySelf [2], TruthRL[3], and Binary RAR [4]. The potentially distinctive contribution appears to be the interaction between SFT data format and subsequent RL, rather than the effectiveness of either training stage.
  2. The evidence does not establish the proposed division between knowledge acquisition and elicitation. Exposure to a fact during SFT does not demonstrate that the model successfully encoded it.
  3. The reward comparison does not establish a generally better abstention strategy. Penalizing incorrect answers more heavily than refusals changes the intended accuracy–abstention tradeoff.

[1] Zhang et al. (2023) R-Tuning: Instructing Large Language Models to Say `I Don't Know' [2] Xu et al. (2024) SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales [3] Wei et al. (2025) TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning [4] Chen et al. (2025) Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations

Questions for authors
  1. What specific finding goes beyond prior supervised and RL approaches to factuality and abstention?
  2. What evidence distinguishes elicitation of SFT-encoded knowledge from alternative explanations, including recovery from SFT-induced degradation? Can the data-format comparison control for starting performance, training-token exposure, and factual coverage?
  3. Does improved abstention transfer to questions whose facts were excluded entirely from both factual and refusal supervision?
  4. Does the binary-reward advantage persist at matched answer coverage or error cost, and relative to confidence-based selective answering?
8 comments
AuthorsOpenReview ↗

Response to Reviewer SLtZ (1/6)

Thank you for the careful review and for recognizing the importance of the problem and the value of a controlled framework. Below we clarify our contribution relative to prior work, sharpen our claims where you point out they need more support, and add new experiments on transfer to excluded facts and on recovery from SFT damage.

Terms. Held-out questions (and held-out F1) are the 550 SimpleQA questions used for our main evaluation, as in the paper. Their facts are taught during SFT, as RichQA passages, but the questions themselves are excluded from RL training. Never-seen facts are a new set of 243 facts excluded from both SFT and RL.

W1, Q1: Contribution relative to prior work

Prior work establishes that SFT and RL can each improve factuality and abstention. Our contribution builds on this and asks a different question: how the two stages interact, and specifically how the factual data used in SFT determines what later RL can and cannot improve.

Novel setting and foundation. We place factual retrieval and abstention inside one controlled post-training pipeline, so that interventions can be compared under the same initial knowledge boundary, factual corpus, training stack, and held-out evaluation. Beyond supporting the present comparisons, this testbed provides a reusable foundation for future studies of how knowledge acquisition, retrieval, and abstention interact across post-training stages.

Novel question. The cited studies and ours ask different questions. Prior work mainly studies SFT for abstention (R-Tuning), confidence expression (SaySelf), or an RL reward for truthfulness (TruthRL, Binary RAR). We instead ask how the factual representation created during SFT constrains what subsequent RL can elicit. Starting from Pass@32-filtered real-world facts, introducing them through context-rich SFT, evaluating on a different atomic form, and varying both the SFT substrate and the RL objective make this interaction directly observable.

Novel findings.

  1. The form of the SFT data, not post-SFT accuracy, determines how much RL can add. On 3B, RichQA SFT reaches the highest post-SFT F1 (43.3) but RL adds only +4.0. Mixing RichQA with atomic single-fact QA pairs (after removing any pair that restates an evaluation question) gives lower post-SFT F1 (27.3-29.1) but larger RL gains (+5.7 to +11.3). Conversely, Q+A paraphrasing gives the highest post-SFT F1 but much smaller RL gains than the mixtures (+0.5 on 7B, +4.2 on Llama-3.1-8B). The same pattern holds on 7B (+11.7 to +13.1 for the mixtures vs. +4.0 for RichQA) and Llama-3.1-8B (+17.2 to +25.1 vs. +2.3).
  2. The RL algorithm matters much less than the SFT data. Starting from the same SFT model, GRPO, Reinforce++, SPIN-DPO, and DAPO all reach 46.6-47.3 held-out F1, even though F1 on the RL training facts keeps rising. This suggests the held-out gain is limited by what SFT stored, not by the choice of RL algorithm.
  3. With a fixed answer/abstain label per fact, rewarding abstention over wrong answers costs correct answers. The ternary reward, which scores abstaining above a wrong answer, lowers answer-target correctness from 39.2% after SFT to 24.6%, while the binary reward lowers it only to 35.8% (gpt-6-luna judge). At matched coverage, binary RL is also more precise (see W3/Q4). Aggregate hallucination metrics hide this cost.

We will lead with these interaction results, and add the comparison table below to the related work.

AuthorsOpenReview ↗

Response to Reviewer SLtZ (2/6)

The cited works differ from ours as follows:

WorkWhat it doesWhat our work adds
R-TuningFinds the questions a base model cannot answer and fine-tunes it to refuse them (refusal-aware SFT). Shows that this refusal ability transfers to out-of-domain datasets.We do not claim novelty on SFT-based abstention or its transfer. We study what RL adds after such SFT, including on the 243 never-seen facts (see Q3 below).
SaySelfSFT on self-reflective rationales built from inconsistencies across sampled answers, followed by RL with a calibration reward, so the model states fine-grained confidence. Evaluated mainly by calibration error.SaySelf teaches the model to report its confidence. We study whether the model answers or abstains, and how the SFT data form limits what RL can then change. We evaluate correctness and abstention directly rather than calibration.
TruthRLGRPO with a ternary reward (+1 correct, 0 abstain, -1 wrong). The reward does not depend on whether the model knows the fact. Its binary baseline (+1 correct, -1 otherwise) scores abstention the same as a wrong answer.Our ternary reward is TruthRL's. Our binary reward is not: it gives 1.0 for abstaining on facts labeled as abstention targets. With such per-fact labels, the ternary reward lowers answer-target correctness (39.2% to 24.6%, vs. 35.8% for binary) and, at matched coverage, is less precise than binary (W3/Q4). We also vary the SFT data before RL, which TruthRL does not.
Binary RAROnline RL with a reward of 1 only when the whole output is factually correct, as verified against retrieved documents, and 0 otherwise. Reduces hallucination without hurting instruction following, math, or code, and learns to say "I don't know" in short-form QA.Binary RAR applies RL to knowledge the model already has from pre-training. We control which facts are introduced through SFT and in what form, and measure how this changes what RL can recover.
Gekhman et al.A controlled closed-book QA study of SFT: new facts are learned more slowly than known ones, and once learned they linearly increase hallucination. Concludes that fine-tuning mostly teaches models to use knowledge they already have.Our RichQA SFT, trained to convergence, shows a similar cost (MMLU 64.4 to 38.5). We then ask how RL interacts with this SFT substrate: RL gains almost nothing without SFT (+0.2 F1), and the facts it fixes are 2.6-5 times more likely to have been stored after SFT than a typical incorrectly answered fact (see W2 below). This extends their view from SFT to RL.

We will add this table to the related work section, and move the SFT-RL interaction results to the front of the paper.

AuthorsOpenReview ↗

Response to Reviewer SLtZ (3/6)

W2: Exposure to a fact during SFT does not show the model stored it

This is an important distinction: a fact being in the SFT data does not by itself prove that the model learned it. Controlling exposure is still central to our design, because it is what lets us give each fact a known label (answer or abstain) and ask whether RL can draw out facts the model was shown. But whether a given fact was actually stored has to be measured, not assumed.

We will make our wording precise: we will say "facts exposed during SFT" when referring to the training data, and state clearly that elicitation is a claim about the model's answers, not about its internal representations.

We add two new pieces of evidence that measure, rather than assume, what SFT stored:

RL needs the facts that SFT put in. We ran the same RL as the paper (680 steps, binary reward, same split) directly on the base model, with no SFT:

Starting modelEval setBefore RL F1After RL F1Change
Base model (no SFT)Held-out facts (550, never rewarded in RL)0.00.2+0.2
Base model (no SFT)RL training facts (440)0.06.6+6.6
RichQA SFT (paper)Held-out facts (550)43.347.3+4.0
RichQA SFT (paper)RL training facts (440)41.054.2+13.2

Without SFT, RL learns only a few of the exact facts it is rewarded on (+6.6, vs. +13.2 after SFT), and this does not carry over to held-out facts (+0.2, vs. +4.0 after SFT). With SFT, the same RL improves held-out facts that RL never saw.

Which facts does RL fix? For each held-out question, we sample 16 answers from the post-SFT model (temperature 1.0, top-p 0.9) and call the fact stored if at least one is correct. We then take the questions the post-SFT model answers incorrectly (with greedy decoding), which are the ones RL could fix, and ask: among the questions RL fixes, how many were stored? If RL fixed facts at random, this share would match the share of stored facts among all incorrectly answered questions.

SFT substrateRL F1 gainStored, among all facts wrong after SFTStored, among the facts RL fixed
RichQA (30 epochs)+3.028% (85/304)73% (19/26)
Clean mixture p=25% (20 epochs)+11.320% (79/390)56% (42/75)
Clean mixture p=25% (2 epochs)+0.715% (78/511)76% (13/17)

RL F1 gains are from this re-evaluation, graded with gpt-6-luna (the paper reports +4.0 for RichQA with gpt-4o-mini).

In every run, the facts RL fixes are far more likely to be stored than a typical incorrectly answered fact (73% vs. 28%, 56% vs. 20%, and 76% vs. 15%, i.e., 2.6-5 times the base rate). The choice of 16 samples does not drive this result: using fewer samples (k = 1, 2, 4, 8) lowers both shares together, and for the two larger runs the fixed facts remain 2.2-2.8 times more likely to be stored at every k. RL therefore selectively improves the facts that SFT already made retrievable, which is direct per-question evidence for our claim at the level of the model's answers.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revision

We thank all reviewers for the careful reviews. We appreciate the recognition that the problem is important and that evaluating retrieval and abstention separately is useful (SLtZ), that RL gain and final accuracy can point in opposite directions (uZsS), and that the controlled framework gives a mechanistic view that is more informative than proposing another training method (Pzdu). The main concerns, and how we address them, are:

  1. Is RL only repairing damage from SFT? Our new results point against this. (a) The clean mixture (p=25%) loses less general ability in SFT than RichQA (MMLU 47.7 vs. 38.5), yet it gets the larger RL gain on held-out facts (+11.3 vs. +4.0 F1). RL then lowers its MMLU (47.7 → 43.4). So the largest RL gain happens while general ability goes down, not while it is being repaired. We further show that a 10-epoch clean-mixture model retains higher MMLU than the 20-epoch model (53.0 vs. 47.7), yet achieves a comparable RL gain (+11.9 vs. +11.3 F1), with almost no MMLU recovery. (b) The same RL run directly on the base model, with no SFT, gains almost nothing on held-out facts (F1 0.0 → 0.2). RL needs the facts that SFT put in. (Reviewer uZsS, W1; Reviewer SLtZ, Q2.)

  2. Does RL draw out facts that SFT stored, or something else? We probed each held-out question by sampling 16 post-SFT answers, and called a fact stored if at least one was correct. Among the questions the post-SFT model answers incorrectly, only 15-28% are stored. Yet among the questions RL then fixes, 56-76% were stored, 2.6-5 times the base rate, in all three runs we tested. This holds at every sample count we tested (1, 2, 4, 8, and 16). RL therefore selectively improves the facts SFT made retrievable, which is neither random improvement nor general repair. (Reviewer SLtZ, W2; Reviewer uZsS, W1; Reviewer Pzdu, W1 and Q1.)

  3. Does learned abstention generalize to facts the model truly does not know? Yes. We built a set of 243 facts that appear in no SFT or RL data and whose answers appear nowhere in the training text. On these never-seen facts, binary RL raises abstention from 54.7% (after SFT) to 71.6%. (Reviewer SLtZ, Q3; Reviewer uZsS, Q2; Reviewer Pzdu, Q3.)

  4. Novelty relative to prior work. Our contribution is the interaction between the two stages: how the form of SFT data determines what later RL can improve. Prior work shows that SFT or RL can each improve factuality; we show how they depend on each other. We now lead with these interaction results in the abstract and introduction, and have added a comparison table with R-Tuning, SaySelf, TruthRL, Binary RAR, and Gekhman et al. (Reviewer SLtZ, W1 and Q1; Reviewer uZsS, W2 and Q3.)

  5. The reward comparison. Our results reveal that there is an important tradeoff between factual retrieval and abstention under different reward designs. At matched answer coverage (~40%), binary RL achieves substantially higher precision than ternary RL (74.8% vs. 61.1%), showing that the difference is not solely explained by refusal rates. Both models abstain on about 97% of abstention targets at this coverage, so the ternary reward's fewer wrong answers come from answering fewer questions, not from better abstention. With a fixed answer/abstain label per fact, the binary reward keeps more correct answers at the same answer rate. (Reviewer SLtZ, W3 and Q4; Reviewer uZsS, W2.)

Terms. Held-out questions (and held-out F1) are the 550 SimpleQA questions used for our main evaluation, as in the paper. Their facts are taught during SFT, as RichQA passages, but the questions themselves are excluded from RL training. Never-seen facts are a new set of 243 facts excluded from both SFT and RL.

A note on the judge. The judge used in the paper (gpt-4o-mini) is no longer available to us, so all new numbers use gpt-6-luna. To check that this does not change our results, we re-graded the paper's own saved outputs for Table 6 with the new judge. The two judges give the same grade on 97.3% of items, and the new judge is slightly stricter on correctness, so it does not inflate gains.

The detailed evidence appears in our response to each reviewer.

Changes to the paper. We have uploaded a revised PDF, with new or changed text in teal:

  • Abstract and introduction: we now lead with the interaction between SFT data form and RL.
  • Wording: our claims are stated in behavioral terms, and the knowledge boundary is defined as the boundary created by post-training.
  • New appendix section, "Additional Experiments": RL without SFT, per-question probing, tests of SFT damage repair, abstention on never-seen facts, the matched-coverage reward comparison, and controls for SFT data form.
  • Related work: a comparison table with the closest prior work.
  • Limitations: the scope of our claims, and the distinction between what limits RL and what produces the best final model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.