- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors evaluate the "contextual answerability" of corpus reformulations, showing that samples where the answer can be easily derived from context are less effective during training. They then apply this metric in a loop to iteratively develop better reformulation strategies, showing improved downstream performance across common evaluation benchmarks for OLMo-3 models.
Critical strengths
The reported correlations are strong, and the equal-document-coverage control provides additional support for the finding.
The authors evaluate both training from scratch and continued pretraining with reasonable token budgets. The comparisons appear well matched, and the selected reformulation achieves the highest average performance across the evaluated settings.
Critical weaknesses
The baselines are limited to raw text and two random reformulations. A simple repetition/paraphrasing baseline and CA-based selection without evolution would help establish what the proposed framework adds.
The evidence comes from one model family and one generator–target combination on Wikipedia. Transfer across model sizes is encouraging, but the generality of CA as a predictor remains unclear.
The gradient analysis uses contextual answer-prediction loss rather than the actual training objective. It is suggestive, but does not establish that lower CA leads to greater factual storage in MLPs.
Questions for authors
How is the factuality verifier implemented, and what fidelity scores do the selected reformulation and random baselines achieve? Does verification also check for unsupported additions by the reformulating model?
Is CA computed over all extracted QA pairs or a subset? Please clarify the QA extraction procedure, scoring prompt, and handling of long documents.
Could you specify the benchmark prompting and scoring protocols, particularly for SimpleQA and PopQA, and provide the numerical continued-pretraining results?
7 comments
Response to Reviewer qrQB (Part 2/3)
W2: Generality of CA evidence
W2: “The evidence comes from one model family and one generator–target combination on Wikipedia. Transfer across model sizes is encouraging, but the generality of CA as a predictor remains unclear.”
Thank you for your comments. Following your suggestion, we add two LLMs to further test the generality of the CA evidence: 1) Llama-3.1-8B from another model family, and 2) OLMo-3-32B, a larger model.
| Model | Pearson | Pearson | Spearman | Spearman |
|---|---|---|---|---|
| Llama-3.1-8B | −0.6496 | −0.6917 | ||
| OLMo-3-32B | −0.7032 | −0.6837 |
The results show that CA remains strongly and significantly negatively correlated with post-training knowledge recall for both Llama-3.1-8B and OLMo-3-32B. This further supports the generality of the CA finding across model families and larger model sizes.
W3: Gradient analysis using the training objective
W3: “The gradient analysis uses contextual answer-prediction loss rather than the actual training objective. It is suggestive, but does not establish that lower CA leads to greater factual storage in MLPs.”
This is a good point! Following your suggestion, we conduct an additional experiment using the model's actual pretraining loss for gradient analysis.
For a rewritten document , the pretraining loss is
| Pearson | Pearson | Spearman | Spearman |
|---|---|---|---|
| −0.7409 | −0.7805 |
The results show that the MLP-to-attention gradient ratio remains strongly and significantly negatively correlated with CA when gradients are computed using the actual training objective. This supports our interpretation that lower CA may place relatively greater optimization pressure on MLP modules associated with factual storage, providing a possible explanation for the counterintuitive negative relationship between CA and factual recall.
Q1: Factuality verification
Q1: “How is the factuality verifier implemented, and what fidelity scores do the selected reformulation and random baselines achieve? Does verification also check for unsupported additions by the reformulating model?”
- For Fidelity score, we check whether the answer strings from the original QA pairs are still present in the rewritten text, to prevent strategies from obtaining lower CA simply by deleting the answers. The implementation is straightforward: we lowercase the answer and text, normalize punctuation, collapse whitespace, and perform substring matching. We provide the fidelity scores below
| Strategy | Fidelity score (answer-string retention) |
|---|---|
| The selected reformulation | 85.30% |
| Random baseline 1 | 85.81% |
| Random baseline 2 | 85.69% |
- Yes, the framework check for unsupported additions. After each reformulation round, the strategy analyzer samples rewritten texts for faithfulness checks, examining whether they introduce unsupported additions or alter the meaning of the original text. It uses these results to guide the direction of strategy refinement in the next round.
Response to Reviewer qrQB (Part 3/3)
Q2: QA extraction, scoring prompt, and long documents
Q2: “Is CA computed over all extracted QA pairs or a subset? Please clarify the QA extraction procedure, scoring prompt, and handling of long documents.”
Thank you for your questions. We clarify these details below and plan to add them to the manuscript.
1. QA coverage. CA is computed over all retained QA pairs after filtering and deduplication.
2. QA extraction procedure.
- Fact extraction: Qwen3.6-35B-A3B extracts atomic facts from the text, while retaining the corresponding source context.
- QA generation and quality control: We use Qwen3.6-35B-A3B to generate natural, unambiguous questions and answers from the extracted facts and their source context, requiring the answers to be supported by the source. The same LLM is used for quality review to repair or remove problematic QA pairs, and finally remove duplicates to form the final QA set.
3. Scoring prompt. The prompt used to compute CA is:
{rewritten_document}
Question: {question}
4. Handling long documents. We preprocess long documents by splitting the full text into consecutive, non-overlapping chunks before QA extraction and CA scoring, using newline boundaries where possible.
Q3: Evaluation protocols and continued-pretraining results
Q3: “Could you specify the benchmark prompting and scoring protocols, particularly for SimpleQA and PopQA, and provide the numerical continued-pretraining results?”
For PopQA, we follow the original evaluation implementation of Mallen et al. [c2]. For SimpleQA, we use the grader prompt and grading criteria from the official simple-evals implementation of Wei et al. [c3]. Our prompting, sampling, and scoring settings are detailed below.
SimpleQA.
-
Benchmark prompting. We use a five-shot completion prompt in the following format:
Question: {example_question_1} Answer: {example_answer_1} ... [five demonstrations in total] ... Question: {test_question} Answer: -
Scoring protocols. We use Qwen2.5-7B-Instruct to classify each response as CORRECT, INCORRECT, or NOT_ATTEMPTED based on the question, reference answer, and model response. Grading uses semantic correctness rather than exact string matching.
PopQA.
- Benchmark prompting. We evaluate all 14,267 questions using a 15-shot
Q: … A:completion prompt. - Scoring protocols. We use answer-alias inclusion matching: a question counts as correct if any reference answer alias appears in the prediction under the case variants specified in the original implementation.
Numerical continued-pretraining results.
| Method | ARC-Challenge | ARC-Easy | GPQA | PubMedQA | HellaSwag | AGIEval | PIQA | SimpleQA | PopQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Raw Wikipedia | 0.8319 | 0.8282 | 0.2656 | 0.7510 | 0.7432 | 0.3447 | 0.7856 | 0.2365 | 0.2286 | 0.5573 |
| Random-1 | 0.8328 | 0.8422 | 0.2679 | 0.7420 | 0.7403 | 0.3529 | 0.7900 | 0.2356 | 0.2237 | 0.5586 |
| Random-2 | 0.8362 | 0.8476 | 0.2656 | 0.7500 | 0.7431 | 0.3414 | 0.7862 | 0.2328 | 0.2315 | 0.5594 |
| CARVE | 0.8362 | 0.8304 | 0.3103 | 0.7530 | 0.7446 | 0.3581 | 0.7867 | 0.2393 | 0.2294 | 0.5653 |
References
[c1] Su, Dan, et al. "Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset." ACL 2025.
[c2] Mallen, Alex, et al. "When not to trust language models: Investigating effectiveness of parametric and non-parametric memories." ACL 2023.
[c3] Wei, Jason, et al. "Measuring short-form factuality in large language models." arXiv preprint arXiv:2411.04368 (2024).
Response to Reviewer qrQB (Part 1/3)
Thank you for your careful review and constructive suggestions! We will try our best to address all your concerns. Please let us know if you still have further concerns, or if you are not satisfied with the current responses, so that we can further update the response ASAP.
W1: Additional baselines
Thank you for your comments. We follow your suggestion to add baselines in two categories:
The results show that:
370M from-scratch training
670M from-scratch training
760M from-scratch training
7B continued pretraining