- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether distributing a fixed number of fact exposures across multiple linguistic expressions improves factual recall compared with repeatedly presenting the same expression. The authors control fact identities, exposure counts, answer occurrences, tokenizer-level answer exposure, training budgets, and corpus-level template frequencies. In synthetic experiments, diverse expression allocation improves forward exact match by 28.75 percentage points across ten seeds for Qwen3-0.6B. The effect also appears across other models, synthetic worlds, and fact inventories, though its magnitude varies. Natural-source experiments provide more mixed evidence: containment improves in four of five domains for both model sizes, while answer-token NLL favors diverse allocation in seven of ten domain-model settings. Mechanism experiments are also mixed, with relation-discrimination gains appearing more consistently for Llama than Qwen. Overall, the paper presents controlled evidence that expression diversity can improve factual callability in synthetic settings, but its generality and underlying mechanism remain unresolved.
Critical strengths
The paper isolates a specific training-data design choice while carefully controlling several potential confounds, including fact frequency, answer exposure, and template frequency. This makes the central comparison more informative than simply comparing datasets with different repetition rates.
The synthetic experiments include multiple seeds, worlds, model sizes, and fact inventories. The authors also distinguish the primary causal evidence from the more exploratory natural-source results rather than treating all experiments as equally conclusive.
The paper examines multiple outcomes, including exact match, containment, and answer-token NLL, and investigates potential mechanisms instead of relying exclusively on aggregate performance. Its reporting of mixed and negative findings, including the lack of a universal mechanism, makes the conclusions more appropriately bounded.
Critical weaknesses
The strongest evidence comes from controlled synthetic settings. Although the five-domain natural-source experiments show encouraging containment results, they use pipeline-verified rewrites, approximate token-budget matching, and weaker controls. Moreover, the separate matched source-QA confirmation misses its prespecified criterion. These results therefore provide suggestive evidence of transfer rather than a clean causal confirmation under realistic training conditions.
The demonstrated effect concerns factual callability on particular probes, not necessarily broader factual generalization across contexts. The answer-token NLL results are useful diagnostics but do not independently establish retrieval, storage, or the mechanism responsible for the behavioral improvements.
The proposed relation-preservation explanation also receives mixed support: relation-discrimination gains are consistent across the tested Llama worlds but inconsistent for Qwen. The results therefore do not establish a universal mechanism. Finally, inference based on five synthetic worlds has limited statistical resolution, and the experiments do not establish a general scaling law across model sizes or training regimes.
Questions for authors
-
Does the advantage persist when facts are tested using paraphrases and query formats that are held out from training, rather than the expression patterns represented in the training data?
-
Can the authors provide a stronger natural-source comparison with better-matched expressions and the planned source-QA confirmation?
-
How should the differences among strict exact match, answer containment, and answer-token NLL be interpreted? Do these measures support the same conclusion?
-
Given the mixed relation-discrimination results, what evidence supports relation preservation as the mechanism, rather than another effect of expression diversity?
-
How robust are the statistical conclusions with only five synthetic worlds, and can the main findings be replicated under realistic continued-pretraining conditions?