- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the distinct roles of supervised fine-tuning (SFT) and reinforcement learning (RL) in factual knowledge retrieval and abstention during LLM post-training. The authors introduce a controlled experimental framework based on SimpleQA and Wikipedia. They use Pass@32 to identify facts that the base model cannot reliably answer and construct the RichQA dataset to systematically analyze the effects of SFT data formats and RL reward designs.
Critical strengths
-
The paper distinguishes factual hallucinations into two dimensions: knowledge retrieval failures and inappropriate answering. It systematically investigates the roles of SFT and RL within a unified framework. By controlling knowledge boundaries, training data formats, and reward mechanisms, the experiments provide valuable insights into the factors influencing factual post-training. This mechanistic perspective is more informative than simply proposing another training method.
-
The paper shows that higher post-SFT accuracy does not necessarily lead to larger RL gains, while atomic knowledge representations can increase the relative improvement from RL even when they reduce SFT accuracy. Furthermore, different reward mechanisms produce substantially different abstention behaviors. These findings offer useful guidance for factual training data construction and post-training strategy design.
Critical weaknesses
-
The paper argues that RL primarily elicits knowledge already encoded during SFT rather than acquiring new knowledge. However, this conclusion is mainly inferred from accuracy changes, training and validation curves, and RL gains under different SFT data formats. The experiments do not directly establish whether the corresponding facts are already encoded in the model's internal representations. Therefore, the current evidence supports a behavioral interpretation but is insufficient to establish a strict mechanistic distinction between knowledge encoding and knowledge retrieval.
-
The main experiments focus on SimpleQA and the derived RichQA dataset, leaving the applicability of the findings to more complex knowledge types and practical post-training scenarios insufficiently explored. Moreover, Table 2 shows that although atomic decomposition increases the relative gains from RL, all clean decomposition settings achieve lower final accuracy than the original RichQA baseline. This limits the practical advantage of atomic knowledge representations and highlights the need to distinguish mechanistic findings from actual end-to-end performance improvements.
Questions for authors
-
Could the authors provide more direct evidence that RL improves the retrieval of already encoded knowledge rather than merely improving adaptation to specific question-answering formats? For example, could experiments involving facts excluded from SFT, alternative question formulations, or knowledge probing help distinguish knowledge acquisition from knowledge retrieval?
-
Both Table 2 and the cross-model experiments in the appendix show that clean decomposition settings yield larger RL gains but still underperform the RichQA baseline in final accuracy. Could the authors further explain this phenomenon and investigate whether combining paraphrasing augmentation with atomic decomposition could improve both final accuracy and RL effectiveness?
-
The abstention experiments construct knowledge boundaries by randomly assigning 50% of the facts as abstention targets. Could the authors clarify how these artificially designated abstention boundaries correspond to facts that the model genuinely does not know? To what extent does this experimental setup support the claim that the model learns to recognize its own knowledge boundaries?
3 comments
Response to Reviewer Pzdu (2/3)
W2, Q2: Scope, and the lower final accuracy of clean decomposition
Other models. The main patterns reproduce on a larger model and a second model family:
| Backbone | RichQA: post-SFT → post-RL | Clean mixture (p=25%): post-SFT → post-RL |
|---|---|---|
| Qwen2.5-3B-Instruct | 43.3 → 47.3 (+4.0) | 29.1 → 40.4 (+11.3) |
| Qwen2.5-7B-Instruct | 49.2 → 53.2 (+4.0) | 36.4 → 49.5 (+13.1) |
| Llama-3.1-8B-Instruct | 50.8 → 53.1 (+2.3) | 30.0 → 47.2 (+17.2) |
The full set of SFT conditions also reproduces on both larger backbones. Each table is sorted by post-SFT F1:
| 7B SFT condition | Post-SFT F1 | Post-RL F1 | |
|---|---|---|---|
| Q+A paraphrase (k=3) | 59.9 | 60.3 | +0.5 |
| Decomposition, unfiltered | 59.5 | 66.2 | +6.8 |
| RichQA (baseline) | 49.2 | 53.2 | +4.0 |
| Clean mixture (p=25%) | 36.4 | 49.5 | +13.1 |
| Clean mixture (p=50%) | 31.7 | 44.8 | +13.1 |
| Clean decomposition | 22.9 | 27.3 | +4.4 |
| Clean mixture (p=100%) | 20.8 | 32.5 | +11.7 |
| Llama-3.1-8B SFT condition | Post-SFT F1 | Post-RL F1 | |
|---|---|---|---|
| Q+A paraphrase (k=3) | 60.4 | 64.6 | +4.2 |
| Decomposition, unfiltered | 54.1 | 60.2 | +6.2 |
| RichQA (baseline) | 50.8 | 53.1 | +2.3 |
| Clean mixture (p=25%) | 30.0 | 47.2 | +17.2 |
| Clean mixture (p=50%) | 23.3 | 42.7 | +19.4 |
| Clean mixture (p=100%) | 16.4 | 41.6 | +25.1 |
| Clean decomposition | 14.8 | 20.6 | +5.7 |
Q+A paraphrasing reaches the highest post-SFT F1 but gains less from RL, while the clean mixtures start lower and gain the most. This is the same pattern as at 3B.
Why SimpleQA, and how far the findings apply. We study atomic, single-hop facts on purpose. They are the basic unit that more complex factual tasks, such as multi-hop questions and long-form answers, are built from, and they let us control exactly which facts the model knows before training. SimpleQA (Wei et al., 2024) is the standard benchmark for this setting. Its 4,326 short, fact-seeking questions cover a wide range of topics (most commonly science and technology, politics, and art), and its grading labels each answer as Correct, Incorrect, or Not Attempted, which lets us measure retrieval and abstention on the same questions. It also remains in active use for frontier models: Google DeepMind's SimpleQA Verified (Haas et al., 2025), a cleaned version of SimpleQA, is used to compare frontier models such as Gemini 2.5 Pro and GPT-5 on a public leaderboard. Our findings therefore concern a factuality problem that is widely measured in practice, not an isolated synthetic task.
The setup also mirrors how knowledge is added in practical post-training, where new facts usually arrive as documents rather than as the exact questions users will ask. Our SFT data (RichQA) presents each fact as a context-rich passage built from its Wikipedia evidence, while evaluation uses the original short SimpleQA question, which the model never trains on. Our results therefore measure whether a model can retrieve a fact in a form different from the one it learned it in.
The main patterns are also not specific to one model: the RichQA and clean-mixture results reproduce on Qwen2.5-7B and Llama-3.1-8B (tables above). In the revision, we will state the scope explicitly: our claims concern atomic factual retrieval and abstention, and extending the analysis to multi-hop and long-form knowledge is a natural next step.
Why clean decomposition ends lower. Our goal is to analyze which factual data forms work and which do not, not to propose decomposition as a better recipe. That clean decomposition ends lower is therefore a finding rather than a failure, and it shows how nuanced factual accuracy is. On 3B, the clean mixture (p=25%) starts much lower after SFT than RichQA (29.1 vs. 43.3), gains nearly three times as much from RL (+11.3 vs. +4.0), and still ends lower (40.4 vs. 47.3). So the three quantities one might optimize, post-SFT accuracy, RL gain, and final accuracy, do not move together: post-SFT accuracy does not predict how much RL can add, and a large RL gain does not imply the best final model. A training intervention therefore cannot be judged by its effect on one stage alone. In the revision, we will state clearly which of our findings concern what limits RL and which concern what produces the best final model.
Response to Reviewer Pzdu (3/3)
Our per-question probing (W1) shows where the difference comes from. We call a fact stored if at least one of 16 post-SFT samples answers it correctly. After RichQA SFT, 60% of all 550 held-out facts are stored, compared with 43% after the clean mixture, so the clean mixture's ceiling is lower and it ends lower. Yet the two models have a similar number of facts that are stored but answered incorrectly with greedy decoding (85 vs. 79), and RL turns these into correct answers more than twice as often after the clean mixture (53% vs. 22%). The clean mixture therefore stores fewer facts, but RL draws out the facts it does store much more effectively.
Combining paraphrasing with decomposition. Our clean mixtures already combine two forms of the same facts: each fact appears as one context-rich RichQA passage together with atomic QA pairs. They gain the most from RL but do not surpass RichQA's final accuracy (40.4 vs. 47.3 on 3B). The combination the reviewer proposes, Q+A paraphrasing augmentation together with clean decomposition, would add more varied phrasings of each fact on top of this. We agree it is a natural way to try to get both a high final accuracy and a large RL gain, and we see it as a promising direction for future work.
Q3: How the random 50/50 abstention split relates to facts the model truly does not know
The random split is applied before training, as a controlled intervention. All facts start in the same pool that the base model cannot answer (Pass@32 = 0), so the two groups are matched in difficulty. For answer-target facts, SFT teaches the RichQA facts. For abstention-target facts, the facts are left out and replaced by a refusal. So the split decides which facts enter the model's knowledge during post-training, and randomization removes differences in topic and difficulty between the groups.
So "knowledge boundary" in our paper means the boundary created by post-training. We will define it this way, and remove wording suggesting the model discovers its own boundary from nothing.
To test the real-world case, we built a set of 243 never-seen facts: SimpleQA facts that the base model never answers correctly in 32 tries, that appear in no SFT or RL data, and whose answer string appears nowhere in the training text. No model answers any of them correctly. Abstention on them rises from 37.0% for the base model and 54.7% after SFT to 71.6% after binary RL and 87.2% after ternary RL (gpt-6-luna judge), and wrong answers fall from 44.4% after SFT to 28.4% after binary RL. So the learned abstention reaches facts outside the designed split, which the model genuinely does not know.
Response to Reviewer Pzdu (1/3)
Thank you for the thoughtful review and for recognizing the value of separating retrieval failures from inappropriate answering, and the guidance our findings give for constructing factual training data. Below we add more direct evidence for the elicitation claim, explain the lower final accuracy of clean decomposition, and clarify the abstention boundary.
Terms. Held-out questions (and held-out F1) are the 550 SimpleQA questions used for our main evaluation, as in the paper. Their facts are taught during SFT, as RichQA passages, but the questions themselves are excluded from RL training. Never-seen facts are a new set of 243 facts excluded from both SFT and RL.
W1, Q1: More direct evidence that RL retrieves stored knowledge
Our evidence is behavioral: we measure what the model answers, not its internal representations. Within that scope, we ran the three tests the reviewer suggests. All three point the same way: RL improves the retrieval of facts that SFT put into the model, and it does not improve retrieval of facts that SFT did not.
1. Facts excluded from SFT. We test this in two ways. First, we built a set of 243 never-seen facts: SimpleQA facts that appear in no SFT or RL data, that the base model never answers correctly in 32 attempts, and whose answer string appears nowhere in the training text. None of our models answers a single one of them correctly, before or after RL (0.0% correct for the SFT, binary RL, and ternary RL models). This indicates that RL does not teach facts that were not seen during SFT. Second, we ran the same RL directly on the base model, with no SFT. Held-out F1 rises by only 0.2 (vs. +4.0 after RichQA SFT). Even on the facts RL itself trains on, F1 rises only from 0.0 to 6.6, compared with 41.0 to 54.2 (+13.2) when RL starts from RichQA SFT. RL alone therefore adds almost no new facts, even when it is trained on them directly.
2. Alternative question formulations. SFT teaches each fact as a context-rich RichQA passage, while evaluation asks the original short SimpleQA question, which appears in neither SFT nor RL training. The RL gain is therefore measured on a different format and on questions RL never sees. RL does train on SimpleQA-style questions for other facts, so it could in principle improve only the answer format. However, the base-model RL-only experiment rules this out: it receives the same format training, yet gains only +0.2 F1. The gain requires the facts to have been introduced by SFT.
3. Knowledge probing. For each held-out question, we sample 16 answers from the post-SFT model and call the fact stored if at least one is correct. We then take the questions the post-SFT model answers incorrectly (with greedy decoding), which are the ones RL could fix, and look at the proportion of these questions that are stored. Then, we look at the questions RL fixes and ask how many were stored. If RL fixed facts at random, the proportion of stored facts it fixes would match the proportion of stored facts among all incorrectly answered questions.
RL F1 gains are from this re-evaluation, graded with gpt-6-luna (the paper reports +4.0 for RichQA with gpt-4o-mini). We use gpt-6-luna because gpt-4o-mini is no longer available to us. On the paper's own saved outputs for Table 6, the two judges give the same grade on 97.3% of items, and the new judge is slightly stricter on correctness.
In every run, the facts RL fixes are far more likely to be stored than a typical incorrectly answered fact (73% vs. 28%, 56% vs. 20%, and 76% vs. 15%, i.e., 2.6-5 times the base rate). RL therefore selectively improves the facts that SFT already made retrievable. The remaining fixes in the 20-epoch run may be facts stored too weakly to appear in 16 samples. The choice of 16 samples does not drive this result: using fewer samples (k = 1, 2, 4, 8) lowers both shares together, and for the two larger runs the fixed facts remain 2.2-2.8 times more likely to be stored at every k.
Together, these results show that RL gains disappear when SFT did not introduce the facts, hold across a change of question format, and concentrate on facts SFT already made retrievable. We will state our claim in these behavioral terms (RL improves retrieval of facts made accessible by SFT) and avoid wording that implies a claim about internal representations.