acceptodds
Under review as a conference paper at ICLR 2027

When Easy Contexts Teach Less: Contextual Answerability-Guided Data Reformulation for LLM Training

Abstract

The growing scarcity of high-quality natural corpora has made data reformulation and synthesis increasingly important for large language model training. However, existing document reformulation methods largely rely on human-designed heuristics, while identifying useful strategies typically requires expensive corpus generation and train-and-evaluate comparisons. In this work, we investigate whether feedback from the model itself can predict the knowledge-learning utility of reformulated data before training. We introduce Contextual Answerability, which measures how readily a model can recover a target fact when a reformulated document is provided as context. Interestingly, we uncover a consistent and counter-intuitive relationship: **reformulations from which the model can more readily recover factual knowledge yield smaller improvements in closed-book factual recall after training.** In other words, higher Contextual Answerability indicates lower knowledge learnability. Our gradient analysis suggests that readily answerable reformulations favor context-dependent processing, whereas less answerable reformulations provide comparatively stronger signals for parametric knowledge acquisition. Building on this analysis, we propose **Carve**, a self-evolving framework that uses Contextual Answerability as feedback to iteratively explore and refine effective corpus-level reformulation strategies under a factual-fidelity constraint. Experiments show that the reformulation selected by Carve outperforms the original corpus and reformulations without feedback during continuous training, reaching a score comparable to the final baselines with **27% fewer training tokens**. The same reformulation also transfers to pretraining from scratch across multiple model scales, where it consistently achieves the highest average performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Data Synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qrQBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors evaluate the "contextual answerability" of corpus reformulations, showing that samples where the answer can be easily derived from context are less effective during training. They then apply this metric in a loop to iteratively develop better reformulation strategies, showing improved downstream performance across common evaluation benchmarks for OLMo-3 models.

Critical strengths

The reported correlations are strong, and the equal-document-coverage control provides additional support for the finding.

The authors evaluate both training from scratch and continued pretraining with reasonable token budgets. The comparisons appear well matched, and the selected reformulation achieves the highest average performance across the evaluated settings.

Critical weaknesses

The baselines are limited to raw text and two random reformulations. A simple repetition/paraphrasing baseline and CA-based selection without evolution would help establish what the proposed framework adds.

The evidence comes from one model family and one generator–target combination on Wikipedia. Transfer across model sizes is encouraging, but the generality of CA as a predictor remains unclear.

The gradient analysis uses contextual answer-prediction loss rather than the actual training objective. It is suggestive, but does not establish that lower CA leads to greater factual storage in MLPs.

Questions for authors

How is the factuality verifier implemented, and what fidelity scores do the selected reformulation and random baselines achieve? Does verification also check for unsupported additions by the reformulating model?

Is CA computed over all extracted QA pairs or a subset? Please clarify the QA extraction procedure, scoring prompt, and handling of long documents.

Could you specify the benchmark prompting and scoring protocols, particularly for SimpleQA and PopQA, and provide the numerical continued-pretraining results?

7 comments
AuthorsOpenReview ↗

Response to Reviewer qrQB (Part 1/3)

Thank you for your careful review and constructive suggestions! We will try our best to address all your concerns. Please let us know if you still have further concerns, or if you are not satisfied with the current responses, so that we can further update the response ASAP.

W1: Additional baselines

W1: “The baselines are limited to raw text and two random reformulations. A simple repetition/paraphrasing baseline and CA-based selection without evolution would help establish what the proposed framework adds.”

Thank you for your comments. We follow your suggestion to add baselines in two categories:

  1. Simple repetition/paraphrasing baselines. We add two baselines: (1) Simple Repeat: Repeat the same rewrite twice. This rewrite restates each fact from the original document using straightforward, conventional sentence structures, consistent with how Dual-Track first presents each fact in the manuscript. (2) Simple paraphrasing: First state each fact using the same rewrite as in Simple Repeat, then state it a second time using the Nemotron-CC paraphrasing recipe [c1].
  2. CA-based selection without evolution. Under the same search budget, we select the strategy with the lowest CA (CA-rank1) that passes factual fidelity validation from the independently generated candidate pool without evolution (Lines 359–361).

The results show that:

  1. The strategy selected through CA-guided evolution outperforms simple repetition and paraphrasing on most benchmarks and in average performance, under the same training token budget and factual exposure. This indicates that its advantage does not come simply from increasing factual exposure: using CA to guide how the second occurrence of a fact is organized and expressed also matters.
  2. CA-guided evolution achieves better average performance than CA-based selection without evolution, indicating that iterative CA-guided search helps identify more effective reformulations under the same search budget.

370M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.40290.24780.43100.35650.24980.64040.12610.09750.3190
Simple paraphrasing0.39930.24550.49000.35990.26020.64200.12200.08630.3257
CA-rank10.41200.23880.48700.36240.26450.64420.12130.09250.3278
CARVE0.41660.24550.51100.36260.25470.64640.12960.08930.3320

670M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.42570.24110.52700.38830.25860.66210.13890.09850.3425
Simple paraphrasing0.42130.24330.51200.38810.25960.66490.13790.09220.3399
CA-rank10.42630.23440.54000.39330.25650.66160.13450.09930.3432
CARVE0.42300.25450.54200.39570.26080.66700.13770.10580.3483

760M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.43440.28130.56700.40940.26320.67360.13790.10080.3585
Simple paraphrasing0.44240.24550.51900.41170.26190.67740.13610.10130.3494
CA-rank10.42550.26120.54800.40810.26290.66920.14530.10490.3531
CARVE0.45140.28790.57100.41660.26580.66320.14560.10060.3628

7B continued pretraining

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.83150.30360.74700.74120.34300.79050.23510.22400.5270
Simple paraphrasing0.83330.27460.73900.73980.35230.78510.23770.22720.5236
CA-rank10.82920.29690.73800.74310.34870.79220.24390.22510.5271
CARVE0.83330.31030.75300.74460.35810.78670.23930.22940.5318
AuthorsOpenReview ↗

Response to Reviewer qrQB (Part 2/3)

W2: Generality of CA evidence

W2: “The evidence comes from one model family and one generator–target combination on Wikipedia. Transfer across model sizes is encouraging, but the generality of CA as a predictor remains unclear.”

Thank you for your comments. Following your suggestion, we add two LLMs to further test the generality of the CA evidence: 1) Llama-3.1-8B from another model family, and 2) OLMo-3-32B, a larger model.

ModelPearson rrPearson ppSpearman ρ\rhoSpearman pp
Llama-3.1-8B−0.64961.938×10−31.938\times10^{-3}−0.69177.280×10−47.280\times10^{-4}
OLMo-3-32B−0.70325.434×10−45.434\times10^{-4}−0.68378.879×10−48.879\times10^{-4}

The results show that CA remains strongly and significantly negatively correlated with post-training knowledge recall for both Llama-3.1-8B and OLMo-3-32B. This further supports the generality of the CA finding across model families and larger model sizes.


W3: Gradient analysis using the training objective

W3: “The gradient analysis uses contextual answer-prediction loss rather than the actual training objective. It is suggestive, but does not establish that lower CA leads to greater factual storage in MLPs.”

This is a good point! Following your suggestion, we conduct an additional experiment using the model's actual pretraining loss for gradient analysis.

For a rewritten document z=(z1,…,zT)z=(z_1,\ldots,z_T), the pretraining loss is LLM(z;θ)=−1T∑t=1Tlog⁡pθ(zt∣z<t).\mathcal{L}_{\mathrm{LM}}(z;\theta) = -\frac{1}{T}\sum_{t=1}^{T}\log p_\theta(z_t\mid z_{<t}).

Pearson rrPearson ppSpearman ρ\rhoSpearman pp
−0.74091.860×10−41.860\times10^{-4}−0.78054.910×10−54.910\times10^{-5}

The results show that the MLP-to-attention gradient ratio remains strongly and significantly negatively correlated with CA when gradients are computed using the actual training objective. This supports our interpretation that lower CA may place relatively greater optimization pressure on MLP modules associated with factual storage, providing a possible explanation for the counterintuitive negative relationship between CA and factual recall.


Q1: Factuality verification

Q1: “How is the factuality verifier implemented, and what fidelity scores do the selected reformulation and random baselines achieve? Does verification also check for unsupported additions by the reformulating model?”

  1. For Fidelity score, we check whether the answer strings from the original QA pairs are still present in the rewritten text, to prevent strategies from obtaining lower CA simply by deleting the answers. The implementation is straightforward: we lowercase the answer and text, normalize punctuation, collapse whitespace, and perform substring matching. We provide the fidelity scores below
StrategyFidelity score (answer-string retention)
The selected reformulation85.30%
Random baseline 185.81%
Random baseline 285.69%
  1. Yes, the framework check for unsupported additions. After each reformulation round, the strategy analyzer samples rewritten texts for faithfulness checks, examining whether they introduce unsupported additions or alter the meaning of the original text. It uses these results to guide the direction of strategy refinement in the next round.
AuthorsOpenReview ↗

Response to Reviewer qrQB (Part 3/3)

Q2: QA extraction, scoring prompt, and long documents

Q2: “Is CA computed over all extracted QA pairs or a subset? Please clarify the QA extraction procedure, scoring prompt, and handling of long documents.”

Thank you for your questions. We clarify these details below and plan to add them to the manuscript.

1. QA coverage. CA is computed over all retained QA pairs after filtering and deduplication.

2. QA extraction procedure.

  • Fact extraction: Qwen3.6-35B-A3B extracts atomic facts from the text, while retaining the corresponding source context.
  • QA generation and quality control: We use Qwen3.6-35B-A3B to generate natural, unambiguous questions and answers from the extracted facts and their source context, requiring the answers to be supported by the source. The same LLM is used for quality review to repair or remove problematic QA pairs, and finally remove duplicates to form the final QA set.

3. Scoring prompt. The prompt used to compute CA is:

{rewritten_document}
Question: {question}

4. Handling long documents. We preprocess long documents by splitting the full text into consecutive, non-overlapping chunks before QA extraction and CA scoring, using newline boundaries where possible.


Q3: Evaluation protocols and continued-pretraining results

Q3: “Could you specify the benchmark prompting and scoring protocols, particularly for SimpleQA and PopQA, and provide the numerical continued-pretraining results?”

For PopQA, we follow the original evaluation implementation of Mallen et al. [c2]. For SimpleQA, we use the grader prompt and grading criteria from the official simple-evals implementation of Wei et al. [c3]. Our prompting, sampling, and scoring settings are detailed below.

SimpleQA.

  1. Benchmark prompting. We use a five-shot completion prompt in the following format:

    Question: {example_question_1}
    Answer: {example_answer_1}
    
    ... [five demonstrations in total] ...
    
    Question: {test_question}
    Answer:
    
  2. Scoring protocols. We use Qwen2.5-7B-Instruct to classify each response as CORRECT, INCORRECT, or NOT_ATTEMPTED based on the question, reference answer, and model response. Grading uses semantic correctness rather than exact string matching.

PopQA.

  1. Benchmark prompting. We evaluate all 14,267 questions using a 15-shot Q: … A: completion prompt.
  2. Scoring protocols. We use answer-alias inclusion matching: a question counts as correct if any reference answer alias appears in the prediction under the case variants specified in the original implementation.

Numerical continued-pretraining results.

MethodARC-ChallengeARC-EasyGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Raw Wikipedia0.83190.82820.26560.75100.74320.34470.78560.23650.22860.5573
Random-10.83280.84220.26790.74200.74030.35290.79000.23560.22370.5586
Random-20.83620.84760.26560.75000.74310.34140.78620.23280.23150.5594
CARVE0.83620.83040.31030.75300.74460.35810.78670.23930.22940.5653

References

[c1] Su, Dan, et al. "Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset." ACL 2025.

[c2] Mallen, Alex, et al. "When not to trust language models: Investigating effectiveness of parametric and non-parametric memories." ACL 2023.

[c3] Wei, Jason, et al. "Measuring short-form factuality in large language models." arXiv preprint arXiv:2411.04368 (2024).

Reviewer_dwbPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper defines Contextual Answerability (CA): the length-normalized log-likelihood a frozen model assigns to a fact’s answer when a rewritten document and the question are in context. Across 20 rewriting strategies for Wikipedia (base model OLMo-3-7B), CA correlates negatively with closed-book SimpleQA after continued training (Pearson −0.851, Spearman −0.872). A gradient analysis associates lower CA with a higher MLP-to-attention gradient ratio. CARVE uses CA, subject to a factual-fidelity constraint, to drive an LLM-based search over rewriting strategies. The selected strategy, Dual-Track Factual Redundancy, improves the benchmark average in 50B-token continued training (matching baselines with 27% fewer tokens) and in from-scratch pretraining at 370M to 760M parameters.

Critical strengths

An interesting and at first glance counterintuitive relationship. The paper introduces Contextual Answerability, a novel and cheap, training-free measure computed from forward passes of a frozen model, and reports that it relates inversely to how much a rewritten corpus improves closed-book recall after training. Easier-to-answer text is the less efficient training signal, which runs against the natural expectation, and Figure 6 checks it in an equal-item setting. If it holds more broadly, it targets a real bottleneck, since train-and-evaluate search over data transformations is expensive. A sensible guard against a degenerate solution. The fidelity constraint blocks the trivial route of lowering CA by deleting or obscuring the answer, and Figure 4 shows the search trajectory openly, including the low-fidelity failures.

Critical weaknesses
  1. The evidence is narrow. The central evidence rests on a narrow experimental base. The inversion rests on 20 strategies, one base model (OLMo-3-7B), one corpus (English Wikipedia), one fact type (long-tail entity-attribute facts), one main metric, and one training run per strategy. The from-scratch results in Table 1 show that the selected corpus transfers to smaller models, but they do not test the CA-recall relationship itself.

  2. CA may be confounded with the base model’s prior knowledge. CA has no no-context baseline, so it mixes how readable a fact is in the rewrite with how well the model already knows it. OLMo’s pretraining data very likely includes Wikipedia, so many of the rewritten facts may already have been seen. The observed gains may therefore come from the more unorthodox rephrasings presenting familiar facts in a new formulation, which improves the extractability of knowledge the model already stores, as in Allen-Zhu & Li (2023), and not from acquiring new facts. I suggest testing the inversion on a fictional dataset such as TOFU (Maini et al., 2024), where the base model cannot have seen the facts, so CA reflects only how readable they are in the rewrite, and recall can be measured on the exact facts that were trained.

  3. The distilled strategy closely resembles a known recipe, so it is unclear whether the improvement is due to CA. The strategy distilled by CARVE, Dual-Track Factual Redundancy, states each fact twice, first directly and then in a syntactically different form. This resembles the multiplicity and paraphrase augmentation of Allen-Zhu & Li (2023). The gains may thus come from this repetition and rephrasing. The baselines do not address this: the two random strategies are single draws from the same search distribution, and there is no paraphrase-based or repetition-based baseline to compare against.

  4. The mechanistic interpretation is only loosely supported. Section 2.3 and Figure 2 suggest, as a possible explanation, that low-CA text places relatively more gradient on the MLPs, which are associated with factual storage. However, the gradients are those of the answer-prediction loss on the base model with the document in context, which is not the loss used when training on the rewritten documents (next-token loss over the text, with no question). It is therefore unclear whether the analysis says anything about what training on these corpora emphasizes. The ratio also varies by only about 3% across strategies, and no result connects it to recall.

Questions for authors

Does the negative relationship between CA and recall hold when CA is replaced by the context gain, log p(a|doc,q) − log p(a|q), computed per fact within each strategy?

Could you explain the recall metric and what exactly is plotted in Figures 1a and 6? I assume it is the gain U(r) from Eq. 1, which would mean the improvement is only about 3 percentage points of SimpleQA, but the captions describe “recall performance after training,” so please confirm. Figure 1a has no numeric axes, and the SimpleQA values in Table 1 are much larger than those in Figure 6.

4 comments
AuthorsOpenReview ↗

Response to Reviewer dwbP (Part 1/4)

Thank you for your careful review and constructive suggestions! We will try our best to address all your concerns. Please let us know if you still have further concerns, or if you are not satisfied with the current responses, so that we can further update the response ASAP.


W1: Generality of the CA–recall relationship

W1: “The evidence is narrow. The central evidence rests on a narrow experimental base. The inversion rests on 20 strategies, one base model (OLMo-3-7B), one corpus (English Wikipedia), one fact type (long-tail entity–attribute facts), one main metric, and one training run per strategy. The from-scratch results in Table 1 show that the selected corpus transfers to smaller models, but they do not test the CA–recall relationship itself.”

Thank you for your comments. Following your suggestions, we conduct additional experiments from 4 aspects to further examine the generality of the CA evidence.

1. Additional model families and sizes. We add two LLMs: 1) Llama-3.1-8B from another model family, and 2) OLMo-3-32B, a larger size model.

ModelPearson rrPearson ppSpearman ρ\rhoSpearman pp
Llama-3.1-8B−0.64961.938×10−31.938\times10^{-3}−0.69177.280×10−47.280\times10^{-4}
OLMo-3-32B−0.70325.434×10−45.434\times10^{-4}−0.68378.879×10−48.879\times10^{-4}

The results show that CA remains strongly and significantly negatively correlated with post-training knowledge recall for both Llama-3.1-8B and OLMo-3-32B. This further supports the generality of the CA finding across model families and larger model sizes.

2. Additional factual QA benchmarks. We also evaluate the CA–recall relationship on three benchmarks with different question sources and formulations:

  • TriviaQA [c1] contains questions written by trivia enthusiasts, including complex, compositional questions, extending the evaluation to quiz-style general knowledge.
  • EntityQuestions [c2] contains Wikidata-based entity–relation questions with systematic question patterns, providing a complementary evaluation of entity facts beyond the particular long-tail question distribution in SimpleQA.
  • NQ Open [c3, c4] derives from real Google search queries, extending the evaluation to naturally occurring user questions and information needs.
DatasetPearson rrPearson ppSpearman ρ\rhoSpearman pp
TriviaQA−0.81151.409×10−51.409\times10^{-5}−0.84213.217×10−63.217\times10^{-6}
EntityQuestions−0.68728.156×10−48.156\times10^{-4}−0.70684.940×10−44.940\times10^{-4}
NQ Open−0.74061.881×10−41.881\times10^{-4}−0.77146.813×10−56.813\times10^{-5}

The negative relationship remains statistically significant across all three benchmarks, supporting its generality beyond SimpleQA across these different question distributions.

3. Additional training seeds. We train each of the 20 original reformulation strategies with two additional seeds. Including the original run, each strategy is therefore evaluated over three training runs.

Training runPearson rrPearson ppSpearman ρ\rhoSpearman pp
Additional run 1−0.87245.309×10−75.309\times10^{-7}−0.80931.547×10−51.547\times10^{-5}
Additional run 2−0.86161.062×10−61.062\times10^{-6}−0.82667.032×10−67.032\times10^{-6}

4. The CA–recall relationship for smaller models. We also test this relationship for the smaller models in Table 1.

ModelPearson rrPearson ppSpearman ρ\rhoSpearman pp
370M−0.82129.107×10−69.107\times10^{-6}−0.75771.090×10−41.090\times10^{-4}
670M−0.63282.750×10−32.750\times10^{-3}−0.72582.920×10−42.920\times10^{-4}
760M−0.78164.706×10−54.706\times10^{-5}−0.62833.011×10−33.011\times10^{-3}

The negative CA–recall relationship also remains strong and statistically significant across additional training seeds and smaller model sizes.

Thank you for these suggestions, we hope these additional experiments could further strengthen the evidence for the generality of the CA–recall relationship.

AuthorsOpenReview ↗

Response to Reviewer dwbP (Part 2/4)

W2 / Q1: Prior knowledge and context gain

W2: “CA may be confounded with the base model’s prior knowledge. CA has no no-context baseline, so it mixes how readable a fact is in the rewrite with how well the model already knows it.” … “I suggest testing the inversion on a fictional dataset such as TOFU (Maini et al., 2024), where the base model cannot have seen the facts, so CA reflects only how readable they are in the rewrite, and recall can be measured on the exact facts that were trained.”

Q1: “Does the negative relationship between CA and recall hold when CA is replaced by the context gain, log p(a|doc,q) − log p(a|q), computed per fact within each strategy?”

Thank you for your constructive comments!

  • We would like to first clarify that, the model's possible prior knowledge does not negate the predictive value of CA in our experiments. Our primary empirical claim is that CA measured before training predicts the improvement in closed-book factual recall induced by a reformulation. Such improvement may reflect both learning new facts and improving access to existing knowledge, so the predictive claim does not require all training facts to be previously unseen.
  • However, we also agree that it is important and interesting to test CA while minimizing the base model's prior knowledge. Following your suggestions, we test the correlation on the fictional dataset TOFU and examine how replacing CA with context gain affects the correlation.

1. Fictional facts from TOFU.

We use the biographies and accompanying QA pairs for all fictional authors in TOFU [c5]. Specifically, we apply the same 20 reformulation strategies from the original manuscript (Lines 162–170) to each biography, and train the same OLMo-3-7B base model separately on each rewritten corpus. Both pre-training CA computation and post-training evaluation use TOFU's original QA pairs. Other experimental settings remain the same as in the original manuscript.

The correlations between CA and knowledge recall on TOFU are:

Pearson rrPearson ppSpearman ρ\rhoSpearman pp
−0.74621.579×10−41.579\times10^{-4}−0.83195.437×10−65.437\times10^{-6}

Results show that, CA remains strongly and significantly negatively correlated with post-training recall on TOFU's fictional facts. Since the evaluation directly measures closed-book recall of the facts included in training, this result supports the relevance of CA to learning new facts, beyond improving access to familiar Wikipedia knowledge.

2. Context gain.

The Pearson correlation remains −0.8510, and the Spearman correlation remains −0.8720. Note that, since we use the same base model, QA pairs, and answer-length normalization for all strategies, averaging the per-fact context gain therefore subtracts the same constant from each strategy's CA: the mean log-probability of the reference answers without the document. Subtracting this constant only shifts the horizontal axis and leaves both correlation coefficients unchanged. Thus, our correlation finding is unchanged by subtracting the no-context baseline.

Overall, The TOFU experiment extends the negative relationship to fictional facts, while the context-gain analysis shows that subtracting the no-context baseline leaves the original correlation unchanged. Thank you for helping us further investigate the behavior of CA.

AuthorsOpenReview ↗

Response to Reviewer dwbP (Part 3/4)

W3: Repetition and paraphrasing baselines

W3: “The distilled strategy closely resembles a known recipe, so it is unclear whether the improvement is due to CA. The strategy distilled by CARVE, Dual-Track Factual Redundancy, states each fact twice, first directly and then in a syntactically different form. This resembles the multiplicity and paraphrase augmentation of Allen-Zhu & Li (2023). The gains may thus come from this repetition and rephrasing. The baselines do not address this: the two random strategies are single draws from the same search distribution, and there is no paraphrase-based or repetition-based baseline to compare against.”

Thank you for your comments. Following your suggestion, we add baselines in two categories:

  1. Simple repetition/paraphrasing baselines. We add two baselines: (1) Simple Repeat: Repeat the same rewrite twice. This rewrite restates each fact from the original document using straightforward, conventional sentence structures, consistent with how Dual-Track first presents each fact in the manuscript. (2) Simple paraphrasing: First state each fact using the same rewrite as in Simple Repeat, then state it a second time using the Nemotron-CC paraphrasing recipe [c6].
  2. CA-based selection without evolution. Under the same search budget, we select the strategy with the lowest CA (noted as CA-rank1) that passes factual fidelity validation from the independently generated candidate pool without evolution (Lines 359–361).

The results show that:

  1. The strategy selected through CA-guided evolution outperforms simple repetition and paraphrasing on most benchmarks and in average performance, under the same training token budget and factual exposure. This indicates that its advantage does not come simply from increasing factual exposure: using CA to guide how the second occurrence of a fact is organized and expressed also matters.
  2. CA-guided evolution achieves better average performance than CA-based selection without evolution, indicating that iterative CA-guided search helps identify more effective reformulations under the same search budget.

370M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.40290.24780.43100.35650.24980.64040.12610.09750.3190
Simple paraphrasing0.39930.24550.49000.35990.26020.64200.12200.08630.3257
CA-rank10.41200.23880.48700.36240.26450.64420.12130.09250.3278
CARVE0.41660.24550.51100.36260.25470.64640.12960.08930.3320

670M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.42570.24110.52700.38830.25860.66210.13890.09850.3425
Simple paraphrasing0.42130.24330.51200.38810.25960.66490.13790.09220.3399
CA-rank10.42630.23440.54000.39330.25650.66160.13450.09930.3432
CARVE0.42300.25450.54200.39570.26080.66700.13770.10580.3483

760M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.43440.28130.56700.40940.26320.67360.13790.10080.3585
Simple paraphrasing0.44240.24550.51900.41170.26190.67740.13610.10130.3494
CA-rank10.42550.26120.54800.40810.26290.66920.14530.10490.3531
CARVE0.45140.28790.57100.41660.26580.66320.14560.10060.3628

7B continued pretraining

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.83150.30360.74700.74120.34300.79050.23510.22400.5270
Simple paraphrasing0.83330.27460.73900.73980.35230.78510.23770.22720.5236
CA-rank10.82920.29690.73800.74310.34870.79220.24390.22510.5271
CARVE0.83330.31030.75300.74460.35810.78670.23930.22940.5318
AuthorsOpenReview ↗

Response to Reviewer dwbP (Part 4/4)

W4: Gradient analysis

W4: "The mechanistic interpretation is only loosely supported." ... "However, the gradients are those of the answer-prediction loss on the base model with the document in context, which is not the loss used when training on the rewritten documents (next-token loss over the text, with no question). It is therefore unclear whether the analysis says anything about what training on these corpora emphasizes. The ratio also varies by only about 3% across strategies, and no result connects it to recall."

This is a good point!

  1. Following your suggestion, we conduct an additional experiment using the model's actual pretraining loss for gradient analysis.

For a rewritten document z=(z1,…,zT)z=(z_1,\ldots,z_T), the pretraining loss is LLM(z;θ)=−1T∑t=1Tlog⁡pθ(zt∣z<t).\mathcal{L}_{\mathrm{LM}}(z;\theta) = -\frac{1}{T}\sum_{t=1}^{T}\log p_\theta(z_t\mid z_{<t}).

Pearson rrPearson ppSpearman ρ\rhoSpearman pp
−0.74091.860×10−41.860\times10^{-4}−0.78054.910×10−54.910\times10^{-5}

The results show that the MLP-to-attention gradient ratio remains strongly and significantly negatively correlated with CA when gradients are computed using the actual pretraining objective. This supports our interpretation, and providing a possible explanation insight for the counterintuitive negative relationship between CA and factual recall.

2. The relationship between the gradient ratio and factual recall. We also measure the correlation between the MLP-to-attention gradient ratio and factual recall:

Pearson rrPearson ppSpearman ρ\rhoSpearman pp
+0.75751.099×10−41.099\times10^{-4}+0.73532.206×10−42.206\times10^{-4}

The ratio is positively and significantly correlated with factual recall, i.e., strategies with a larger MLP-to-attention gradient ratio tend to achieve better post-training recall. This positive correlation is consistent with our observations that CA is negatively correlated with both recall and the gradient ratio.

3. Variation in the gradient ratio across strategies. Although the variation in the original gradient ratio is small, its negative correlation with CA is statistically significant. In addition, the gradient ratio computed using the actual pretraining loss ranges from 1.4611 to 1.8978 across strategies, corresponding to a 29.89% variation relative to the minimum.

Together, the gradient analyses based on the actual pretraining loss provide further evidence consistent with our interpretation that lower-CA reformulations may place relatively greater optimization pressure on MLPs associated with factual storage.

Q2: Recall metrics in the figures and Table 1

Q2: “Could you explain the recall metric and what exactly is plotted in Figures 1a and 6? I assume it is the gain U(r) from Eq. 1, which would mean the improvement is only about 3 percentage points of SimpleQA, but the captions describe ‘recall performance after training,’ so please confirm. Figure 1a has no numeric axes, and the SimpleQA values in Table 1 are much larger than those in Figure 6.”

We apologize for the confusion.

  1. Figures 1a and 6 show the model's absolute post-training SimpleQA pass@1, rather than the gain. We use pass@1 in the CA analysis to examine whether CA predicts the reliability of recalling a fact in a single sampled attempt.
  2. The SimpleQA values in Table 1 report pass@32. Here, we evaluate whether the model can produce a correct answer within a fixed budget of 32 sampled attempts, using the pass@kk evaluation following [c7, c8]. This complements pass@1 by measuring successful recall across multiple attempts. Pass@32 counts a question as correct if at least one of the 32 sampled answers is correct, which explains its higher values relative to pass@1. The two metrics address different evaluation goals, so the values in Table 1 and Figures 1a and 6 are not directly comparable.

References

[c1] Joshi, Mandar, et al. "Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension." ACL 2017.

[c2] Sciavolino, Christopher, et al. "Simple entity-centric questions challenge dense retrievers." EMNLP 2021.

[c3] Kwiatkowski, Tom, et al. "Natural questions: a benchmark for question answering research." TACL 2019.

[c4] Lee, Kenton, Ming-Wei Chang, and Kristina Toutanova. "Latent retrieval for weakly supervised open domain question answering." ACL 2019.

[c5] Maini, Pratyush, et al. "Tofu: A task of fictitious unlearning for llms." COLM 2024.

[c6] Su, Dan, et al. "Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset." ACL 2025.

[c7] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

[c8] Zhou, Jiang, et al. "WRAP++: Web Discovery Amplified Pretraining." EMNLP 2026.

Reviewer_48S8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the utility of data reformulation for LLM training can be predicted without expensive train-and-evaluate procedures. Experiments on continued training and pretraining from scratch show improvements over raw Wikipedia and randomly selected reformulations, including a reported 27% reduction in training tokens needed to reach comparable performance.

Critical strengths

1. An interesting and potentially useful empirical insight.

The inverse relationship between contextual answerability and knowledge learnability is counterintuitive and provides a clear, memorable insight. The strong correlations across 20 strategies, together with the additional equal-item experiment, offer meaningful evidence that CA may serve as a useful training-free signal for evaluating data reformulations.

2. Encouraging results across training settings.

CARVE improves average benchmark performance in both continued training and pretraining from scratch across multiple model scales. The comparisons use matched training-token budgets and background data, suggesting that the discovered reformulation has practical value beyond a single training configuration.

Critical weaknesses

1. Insufficient evidence that CA-guided evolution itself provides the claimed benefit.

The main experiments compare CARVE against raw Wikipedia and only two randomly selected reformulation strategies. However, CARVE selects the best candidate through a multi-round search, while the random baselines are not similarly selected for quality. This selection asymmetry makes it difficult to attribute the improvements specifically to CA-guided evolution rather than simply finding a stronger reformulation. Moreover, the paper lacks comparisons with competitive existing reformulation methods or a best-of-budget unguided search. Consequently, the central methodological claim remains insufficiently supported.

2. The improvements may primarily arise from factual redundancy rather than the proposed CA-based explanation.

The final selected strategy, Dual-Track Factual Redundancy, explicitly repeats each fact using different syntactic realizations. Such repetition and paraphrasing may independently improve factual learning, regardless of CA. The paper does not include controlled comparisons against simple factual repetition or paraphrasing strategies, making it unclear whether the improvements reflect the proposed CA-based principle or established benefits of repeated and diverse factual exposure. This alternative explanation directly weakens the interpretation of the main results.

Questions for authors
  1. Does CA-guided evolution outperform a matched-budget unguided search? In particular, how does CARVE compare against the lowest-CA, fidelity-valid strategy selected from an equally sized pool of candidates generated without evolutionary feedback? This comparison would help establish whether iterative CA-guided evolution contributes beyond candidate selection.

  2. Can the authors disentangle the effects of CA from factual redundancy and paraphrasing? How does the selected Dual-Track strategy compare against simpler repetition or paraphrasing baselines under matched token budgets and comparable factual exposure? Reporting both CA and downstream factual recall would help determine whether the proposed explanation accounts for the observed improvements.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 48S8 (Part 1/3)

Thank you for your careful review and constructive suggestions! We will try our best to address all your concerns. Please let us know if you still have further concerns, or if you are not satisfied with the current responses, so that we can further update the response ASAP.

W1.1 / Q1: Matched-budget unguided search

W1.1: “Insufficient evidence that CA-guided evolution itself provides the claimed benefit. The main experiments compare CARVE against raw Wikipedia and only two randomly selected reformulation strategies.”

Q1: “Does CA-guided evolution outperform a matched-budget unguided search? In particular, how does CARVE compare against the lowest-CA, fidelity-valid strategy selected from an equally sized pool of candidates generated without evolutionary feedback? This comparison would help establish whether iterative CA-guided evolution contributes beyond candidate selection.”

Thank you for your suggestion. We follow your instructions to add CA-based selection without evolution as a matched-budget unguided search baseline. Specifically, under the same search budget, we select the lowest-CA strategy that passes factual fidelity validation from the independently generated candidate pool without evolutionary feedback (Lines 359–361).

The results show that CA-guided evolution achieves better average performance than CA-based selection without evolution. This indicates that iterative CA-guided search helps identify more effective reformulations under the same search budget. In the table, CA-rank1 denotes the lowest-CA strategy selected without evolution.

370M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
CA-rank10.41200.23880.48700.36240.26450.64420.12130.09250.3278
CARVE0.41660.24550.51100.36260.25470.64640.12960.08930.3320

670M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
CA-rank10.42630.23440.54000.39330.25650.66160.13450.09930.3432
CARVE0.42300.25450.54200.39570.26080.66700.13770.10580.3483

760M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
CA-rank10.42550.26120.54800.40810.26290.66920.14530.10490.3531
CARVE0.45140.28790.57100.41660.26580.66320.14560.10060.3628

7B continued pretraining

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
CA-rank10.82920.29690.73800.74310.34870.79220.24390.22510.5271
CARVE0.83330.31030.75300.74460.35810.78670.23930.22940.5318
AuthorsOpenReview ↗

Response to Reviewer 48S8 (Part 2/3)

W1.2: Comparisons with existing reformulation methods

W1.2: “Moreover, the paper lacks comparisons with competitive existing reformulation methods or a best-of-budget unguided search. Consequently, the central methodological claim remains insufficiently supported.”

Thank you for your suggestions. We follow your instructions to add four competitive existing reformulation methods: WRAP [c2], Nemotron-CC [c1], Active Reading [c3], and WRAP++ [c4]. Note that, these methods typically combine training data generated with multiple reformulation strategies or styles. Prior work shows that diverse reformulations can improve training performance [c2, c1]. Thus, to make the comparison fairer, we also add CARVE-mix, which mixes corpora generated by the four strategies with the lowest CA scores in our framework.

Experiments across multiple model sizes show that CARVE-mix outperforms all four existing methods on average in every setting. CARVE-mix achieves the best average performance for 370M and 670M from-scratch training and 7B continued pretraining, while the single-strategy CARVE (Dual-Track) achieves the best average performance at 760M. Beyond average performance, either CARVE or CARVE-mix achieves superior results on most individual benchmarks.

370M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
WRAP0.41590.23880.49100.36150.25160.65130.12800.09430.3291
Nemotron-CC0.40410.25220.51000.36120.25360.64090.14720.09100.3325
Active Reading0.40810.23880.48600.35110.25940.65400.13580.10210.3294
WRAP++0.40580.25450.47300.36000.26310.63870.13260.09410.3277
CARVE0.41660.24550.51100.36260.25470.64640.12960.08930.3320
CARVE-mix0.40740.28350.54100.36730.25980.65890.13750.09260.3435

670M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
WRAP0.41820.24780.46800.39120.25750.66700.14190.09840.3363
Nemotron-CC0.41420.22100.50100.38510.25900.66050.14210.10480.3360
Active Reading0.41710.26790.49900.38500.26190.66050.13150.09860.3402
WRAP++0.41910.26560.52200.38740.25740.65940.12770.09170.3413
CARVE0.42300.25450.54200.39570.26080.66700.13770.10580.3483
CARVE-mix0.41940.23880.57300.38430.26290.66870.14580.10070.3492

760M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
WRAP0.43420.26560.53100.40870.26450.67740.14650.10500.3541
Nemotron-CC0.43550.28130.54100.41400.25600.67460.14720.10390.3567
Active Reading0.43060.27900.55600.41380.25810.66540.14350.10750.3567
WRAP++0.43320.25890.52800.40980.25910.66920.14440.09420.3496
CARVE0.45140.28790.57100.41660.26580.66320.14560.10060.3628
CARVE-mix0.43000.26340.55800.41130.26610.66810.15300.10530.3569

7B continued pretraining

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
WRAP0.82990.29020.75100.73790.34580.77750.24210.22540.5250
Nemotron-CC0.82930.28570.76000.74080.35960.78560.23260.22530.5274
Active Reading0.83180.28570.74200.74280.36490.78400.23540.22130.5260
WRAP++0.83310.28790.74100.74210.36550.78180.24460.22840.5281
CARVE0.83330.31030.75300.74460.35810.78670.23930.22940.5318
CARVE-mix0.83600.29460.76200.74250.36720.79050.23650.23140.5326
AuthorsOpenReview ↗

Response to Reviewer 48S8 (Part 3/3)

W2 / Q2: Factual repetition and paraphrasing

W2: "The improvements may primarily arise from factual redundancy rather than the proposed CA-based explanation." ... "The paper does not include controlled comparisons against simple factual repetition or paraphrasing strategies, making it unclear whether the improvements reflect the proposed CA-based principle or established benefits of repeated and diverse factual exposure."

Q2: "Can the authors disentangle the effects of CA from factual redundancy and paraphrasing? How does the selected Dual-Track strategy compare against simpler repetition or paraphrasing baselines under matched token budgets and comparable factual exposure? Reporting both CA and downstream factual recall would help determine whether the proposed explanation accounts for the observed improvements."

Thank you for your comments. Following your suggestion, we add two baselines:

  1. Simple Repeat: Repeat the same rewrite twice. This rewrite is consistent with how Dual-Track first presents each fact in the manuscript.
  2. Simple paraphrasing: First state each fact using the same rewrite as in Simple Repeat, then state it a second time using the Nemotron-CC paraphrasing recipe [c1].
  • The results show that the strategy selected through CA-guided evolution outperforms simple repetition and paraphrasing on most benchmarks and in average performance, under the same training token budget and factual exposure. This indicates that its advantage does not come simply from increasing factual exposure: using CA to guide how the second occurrence of a fact is organized and expressed also matters.
  • In addition, we compute the CA scores of all three strategies to examine whether CA distinguishes different ways of repeating facts. The scores are −1.65683 for CARVE (Dual-Track), −1.34658 for Simple Repeat, and −1.16028 for Simple paraphrasing. Although all three strategies increase the occurrences of each fact, they receive different CA scores from the model.

370M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.40290.24780.43100.35650.24980.64040.12610.09750.3190
Simple paraphrasing0.39930.24550.49000.35990.26020.64200.12200.08630.3257
CARVE0.41660.24550.51100.36260.25470.64640.12960.08930.3320

670M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.42570.24110.52700.38830.25860.66210.13890.09850.3425
Simple paraphrasing0.42130.24330.51200.38810.25960.66490.13790.09220.3399
CARVE0.42300.25450.54200.39570.26080.66700.13770.10580.3483

760M from-scratch training

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.43440.28130.56700.40940.26320.67360.13790.10080.3585
Simple paraphrasing0.44240.24550.51900.41170.26190.67740.13610.10130.3494
CARVE0.45140.28790.57100.41660.26580.66320.14560.10060.3628

7B continued pretraining

MethodARCGPQAPubMedQAHellaSwagAGIEvalPIQASimpleQAPopQAAvg.
Simple Repeat0.83150.30360.74700.74120.34300.79050.23510.22400.5270
Simple paraphrasing0.83330.27460.73900.73980.35230.78510.23770.22720.5236
CARVE0.83330.31030.75300.74460.35810.78670.23930.22940.5318

References

[c1] Su, Dan, et al. "Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset." ACL 2025.

[c2] Maini, Pratyush, et al. "Rephrasing the web: A recipe for compute and data-efficient language modeling." ACL 2024.

[c3] Lin, Jessy, et al. "Learning facts at scale with active reading." ICLR 2026.

[c4] Zhou, Jiang, et al. "WRAP++: Web Discovery Amplified Pretraining." EMNLP 2026.

Dear Reviewer 48S8,

Thank you again for taking the time to review our paper and for your constructive suggestions!

Following your feedback, we follow your instructions conduct extensive additional experiments and revise the manuscript accordingly. Your comments help us strengthen the empirical evidence and improve the clarity and quality of our paper.

As the discussion period draws to a close, we would be grateful for any feedback on whether our responses adequately address your concerns. We would also appreciate the opportunity to clarify any remaining questions or address any additional concerns before the discussion period ends.

Sincerely, the authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Responses and Manuscript Update

Dear Reviewers and Area Chair,

Thank you for your careful reviews and constructive suggestions! We appreciate the time and effort you devote to evaluating our work.

We are grateful for the reviewers’ recognition of our empirical finding as “An interesting and potentially useful empirical insight” (48S8), Contextual Answerability (CA) as “a novel and cheap, training-free measure computed from forward passes of a frozen model” (dwbP), and the observation that “the selected reformulation achieves the highest average performance across the evaluated settings” (qrQB).

Following your suggestions, we conduct additional experiments and clarify the implementation and evaluation details. We summarize the main responses below.

  1. Additional baselines and the contribution of CA-guided evolution. To examine whether the gains come from factual repetition, paraphrasing, or candidate selection, we add Simple Repeat, Simple Paraphrasing, and CA-based selection without evolution. The last baseline selects the lowest-CA, fidelity-valid strategy from a matched-budget unguided candidate pool. Under matched training token budgets, CARVE achieves higher average performance than these baselines across model scales and training settings. The repetition controls further suggest that CARVE’s CA-guided evolution helps identify effective ways to organize and express repeated facts, even when the repetition count is held fixed. The comparison with selection without evolution also supports the usefulness of iterative CA-guided search.

  2. Generality of the CA–recall relationship and the influence of prior knowledge. Following your suggestions, we extend the analysis in the following settings.

    • Additional model families and sizes. We add Llama-3.1-8B from another model family and the larger OLMo-3-32B.
    • Additional factual QA benchmarks. We evaluate TriviaQA, EntityQuestions, and NQ Open, covering different question sources and formulations.
    • Additional training seeds. We train each reformulation strategy with two additional seeds to examine the consistency of the relationship across runs.
    • Smaller models. We directly test the CA–recall relationship on the 370M, 670M, and 760M models used in the from-scratch experiments.
    • Fictional facts. We reformulate TOFU’s fictional biographies and use its original QA pairs to evaluate closed-book recall of the trained facts, addressing the concern about the base model’s prior knowledge.

    The negative CA–recall relationship remains statistically significant across these additional settings, including fictional facts.

  3. Gradient analysis using the actual training objective. We recompute the MLP-to-attention gradient ratio using the actual pretraining loss and examine its relationship with CA. The ratio remains strongly and significantly negatively correlated with CA under the actual pretraining objective. These observations provide further evidence consistent with our interpretation that lower-CA reformulations may place relatively greater optimization pressure on MLP modules associated with factual storage.

  4. Comparisons with existing reformulation methods. We add WRAP, Nemotron-CC, Active Reading, and WRAP++. Since these methods typically combine multiple reformulation strategies or styles, we also introduce CARVE-mix, which mixes the four fidelity-valid strategies with the lowest CA scores. CARVE-mix outperforms all four existing methods on average in every evaluated setting, supporting the effectiveness of CA-guided reformulation under multi-strategy mixing.

  5. Implementation details and evaluation protocols. We clarify the factuality verifier and the strategy analyzer’s faithfulness checks, QA extraction and coverage, scoring prompts, and long-document processing. We also clarify the recall metrics and benchmark prompting and scoring protocols, and provide the full mid-training results.

Thank you again for helping us improve the quality and clarity of the paper. We have revised and re-uploaded the manuscript according to your suggestions, with the revisions marked in blue. We hope the additional experiments and clarifications address your concerns, and we welcome any further questions or suggestions.

Sincerely,

the authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.