acceptodds
Under review as a conference paper at ICLR 2027

Beyond Fact Frequency: Controlled Evidence on Expression Allocation and Factual Callability

Abstract

Fact frequency does not specify how a fixed fact’s exposure is distributed across linguistic expressions. We isolate this within-fact expression-allocation estimand by matching fact identity, exposure count, answer occurrence, tokenizer and train- ing budgets, and corpus-level template marginals. In controlled synthetic settings, distributing fixed exposure across faithful expressions instead of repeating one sub- stantially improves held-out Forward callability: +28.75 percentage points across ten Qwen3-0.6B training seeds, with replication across independently generated worlds from the same programmed generator, larger Qwen and Llama models, natural held-out wording, and a same-family Qwen3 persistence test through the largest tested 8B checkpoint. We then apply the same intervention to a five-domain natural-source panel at two Qwen sizes, where generated behavior and teacher- forced target-answer likelihood are heterogeneous across domains and model sizes, with strict exact match remaining at floor. Dedicated stress tests do not support a universal relation-preservation mechanism. Thus, in our controlled construction, equal fact frequency need not imply behaviorally equivalent training exposure, while natural-data, bidirectional-transfer, scaling-law, and internal-mechanism claims remain bounded.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model pretraining, training data, data quality, factual knowledge, data diversity, controlled experiments

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AUq8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether distributing a fixed number of fact exposures across multiple linguistic expressions improves factual recall compared with repeatedly presenting the same expression. The authors control fact identities, exposure counts, answer occurrences, tokenizer-level answer exposure, training budgets, and corpus-level template frequencies. In synthetic experiments, diverse expression allocation improves forward exact match by 28.75 percentage points across ten seeds for Qwen3-0.6B. The effect also appears across other models, synthetic worlds, and fact inventories, though its magnitude varies. Natural-source experiments provide more mixed evidence: containment improves in four of five domains for both model sizes, while answer-token NLL favors diverse allocation in seven of ten domain-model settings. Mechanism experiments are also mixed, with relation-discrimination gains appearing more consistently for Llama than Qwen. Overall, the paper presents controlled evidence that expression diversity can improve factual callability in synthetic settings, but its generality and underlying mechanism remain unresolved.

Critical strengths

The paper isolates a specific training-data design choice while carefully controlling several potential confounds, including fact frequency, answer exposure, and template frequency. This makes the central comparison more informative than simply comparing datasets with different repetition rates.

The synthetic experiments include multiple seeds, worlds, model sizes, and fact inventories. The authors also distinguish the primary causal evidence from the more exploratory natural-source results rather than treating all experiments as equally conclusive.

The paper examines multiple outcomes, including exact match, containment, and answer-token NLL, and investigates potential mechanisms instead of relying exclusively on aggregate performance. Its reporting of mixed and negative findings, including the lack of a universal mechanism, makes the conclusions more appropriately bounded.

Critical weaknesses

The strongest evidence comes from controlled synthetic settings. Although the five-domain natural-source experiments show encouraging containment results, they use pipeline-verified rewrites, approximate token-budget matching, and weaker controls. Moreover, the separate matched source-QA confirmation misses its prespecified criterion. These results therefore provide suggestive evidence of transfer rather than a clean causal confirmation under realistic training conditions.

The demonstrated effect concerns factual callability on particular probes, not necessarily broader factual generalization across contexts. The answer-token NLL results are useful diagnostics but do not independently establish retrieval, storage, or the mechanism responsible for the behavioral improvements.

The proposed relation-preservation explanation also receives mixed support: relation-discrimination gains are consistent across the tested Llama worlds but inconsistent for Qwen. The results therefore do not establish a universal mechanism. Finally, inference based on five synthetic worlds has limited statistical resolution, and the experiments do not establish a general scaling law across model sizes or training regimes.

Questions for authors
  1. Does the advantage persist when facts are tested using paraphrases and query formats that are held out from training, rather than the expression patterns represented in the training data?

  2. Can the authors provide a stronger natural-source comparison with better-matched expressions and the planned source-QA confirmation?

  3. How should the differences among strict exact match, answer containment, and answer-token NLL be interpreted? Do these measures support the same conclusion?

  4. Given the mixed relation-discrimination results, what evidence supports relation preservation as the mechanism, rather than another effect of expression diversity?

  5. How robust are the statistical conclusions with only five synthetic worlds, and can the main findings be replicated under realistic continued-pretraining conditions?

Reviewer_vVVrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper tests whether allocating fixed fact exposure across several expressions improves held-out factual recall. It matches fact and answer exposure, training and token budgets, and global template frequencies. A 28.75-percentage-point Forward accuracy gain across ten Qwen3-0.6B seeds is supported by world, wording and model replications through 8B. Natural-source and mechanism tests give mixed results, bounding the finding's scope.

Critical strengths
  1. A credible controlled result. The study separates exposure amount from within-fact allocation, with replications across seeds, worlds and model configurations (§§3–4). It also reports failed extensions and keeps synthetic, natural-source and mechanism results separate.
Critical weaknesses
  1. The additional scientific insight remains limited. Prior work already connects contextual or format diversity to generalization (Facts in Stats; FormatMix). This paper's stricter per-fact controls are a useful advance, but the positive conclusion remains concentrated in one synthetic construction. The heterogeneous natural-source results and mechanism tests do not establish a comparably clear extension. My concern is the significance of this bounded contribution, not the validity of the controlled effect.
Questions for authors
  1. Which alternative explanation left open by the closest work is ruled out by the new controls, and how does that change our understanding of factual learning?
  2. Which existing result provides the strongest evidence beyond the shared synthetic generator, given the weaker natural-source controls and mixed endpoints?
Reviewer_PpESOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address whether distributing a fact’s fixed exposure across different expressions changes its subsequent accessibility in language models. The authors note that “Fact frequency does not specify how a fixed fact’s exposure is distributed across linguistic expressions.”

Critical strengths
  1. The synthetic comparison controls several common explanations for gains from data diversity. The authors state that “Fact identities, answer occurrences, optimization and tokenizer-measured budgets, and corpus-level template marginals are matched.” (page 1), making the estimated effect more interpretable than an unmatched comparison of differently sized corpora.

  2. The statistical design avoids treating many probes from the same trained model as independent replications. The statement that “The primary independent unit is a training seed or generated world, never an individual probe.” (page 1), together with seed- and world-level replication, supports a more defensible assessment of the controlled effect.

Critical weaknesses
  1. The natural-source containment metric may reward mentions of an answer without establishing a correct response. It counts cases where “the fully normalized canonical answer occurs as a contiguous character substring of the fully normalized response.” (page 5), which can include negated answers or responses listing conflicting alternatives. I recommend human-validated semantic correctness alongside containment, with audits of negation and multiple-answer cases, to determine whether the reported gains reflect usable factual knowledge.

  2. The design identifies a bundled expression-allocation effect but does not isolate generalization from increased surface-form coverage. The paper notes that “the bundled intervention does not isolate lexical, syntactic, interface, or interaction components” (page 9), leaving uncertainty about which changes produce the benefit even with globally balanced templates. I recommend a matched factorial study varying lexical, syntactic, and interface diversity separately while controlling overlap with held-out probes, to clarify the mechanism and improve transferability.

Questions for authors

Please see the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.