acceptodds
Under review as a conference paper at ICLR 2027

Understanding the Role of Reasoning Data in Pretraining

Abstract

Reasoning data expose models to multi-step reasoning, but a reasoning trace may reveal only a subset of the underlying facts it composes. How should pretraining balance atomic facts with reasoning traces? We study this question by constructing a setting in which every fact provides an atomic training signal, while reasoning traces contain only a subset of the facts they compose, enabling us to measure generalization to unseen transitions. For a linear transformer, we derive the weight assigned to atomic training signals that minimizes held-transition squared error and show that its leading-order optimum increases with reasoning-chain length and decreases with the fraction of facts held out from reasoning traces. These trends persist in a full transformer layer with an MLP, softmax attention, and cross-entropy loss, as well as in a realistic mathematical reasoning setting. Together, our results provide a principled view of how reasoning-chain structure should shape the allocation of training data during pretraining.

Primary area:
generative models
Keywords:
reasoning, language model, transformer theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_R7zDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows when to mix in reasoning data in pretraining data and what the right balance is between atomic facts and reasoning data. They start
with experiments in a toy State Tracking setting, using a one-layer
transformer. Figure 1 nicely shows their finding that both ends of the
spectrum, having only atomic facts and only reasoning data, lead to poor OOD generalization. It also shows that the more transitions are excluded from
reasoning training, the lower the model's OOD accuracy gets, while ID
accuracy generally remains higher. They also consider different lengths of
reasoning chains.

To provide a theoretical explanation for the optimal balance between atomic facts and reasoning data, they simplify the model to a linear-attention
model. The optimal atomic-loss weight depends on reasoning-chain length and the coverage of facts in reasoning traces, increasing with chain length and decreasing with the held-transition fraction.

After setting the theoretical foundation, they are experimenting with
natural language data, for which they train Olmo-3 150M models using
FineMath -
a general mathematical text dataset - and MathLake - a mathematical
reasoning
dataset with problems of varying difficulty, which require different numbers of output tokens to be solved. Also, here they first test the ratio of text data to reasoning data, then examine reasoning-data coverage and problem
difficulty, and finally fix the ratio to examine when to introduce reasoning data. And their main finding is that an 80/20 split between text and
reasoning data performs best in their experiments, as measured by GSM8K
solution cross-entropy. They also compare math problems with varying
difficulty and find no benefit to introducing reasoning data later in
training; therefore, they recommend mixing in reasoning data right from the beginning.

Critical strengths
  1. Controlled experiments investigating the balance between atomic facts and reasoning data.

The authors introduce a well-controlled state-tracking setting in which they can independently vary the ratio of atomic facts to reasoning data, the
length of reasoning chains, and the fraction of transitions excluded from
reasoning training. I particularly liked Figure 1, which nicely illustrates how both types of supervision contribute to OOD generalization. This is a
strength because it allows the authors to investigate the trade-off
systematically rather than only comparing a few fixed data mixtures.

  1. Combining theoretical analysis with experiments on mathematical
    pretraining.

In addition to the synthetic experiments, the authors derive a theoretical
characterization of the optimal balance using a simplified linear-attention model. They then investigate whether similar trends can be observed when
pretraining OLMo-3 150M models on mathematical data. I think this
combination
is valuable because it provides both a theoretical perspective on the
problem and experimental evidence beyond the synthetic setting.

Critical weaknesses
  1. It is unclear whether the model fails to learn heldout transitions or fails to generalize compositionally.

One concern I have with Figure 1 is that the authors show a large difference between ID and OOD accuracy, but do not separately report whether the model has successfully learned the heldout transitions through atomic supervision.
This makes it difficult to determine whether the model struggles to compose transitions it already knows or whether it has simply not learned those
transitions sufficiently well. I think evaluating atomic accuracy on heldout
transitions would help clarify this distinction and strengthen the
conclusions about compositional generalization.

  1. The connection between the synthetic setting and mathematical pretraining remains limited.

While I appreciate that the authors extend their experiments to mathematical pretraining, the two settings are quite different. FineMath is used as a
proxy for atomic data, although it contains general mathematical text rather than isolated facts. Furthermore, the 80/20 mixture is evaluated using GSM8K solution cross-entropy rather than problem-solving accuracy, and changing
the
reasoning-data ratio also changes how frequently MathLake examples are
repeated. This makes it difficult to determine how directly the findings
from the synthetic setting translate to realistic pretraining and how
broadly the optimal mixture can be generalized.

Questions for authors

In Def. 2, you state that r2,...,rL belong to S, although S contains
entity–relation pairs, while the formal definition samples all relation
sequences uniformly. Could you clarify which sampling procedure was actually used in the experiments?

Could you report atomic accuracy on heldout transitions? This would help
distinguish whether the model fails to learn these transitions or struggles to compose them despite knowing them.

To what extent could the increased repetition of MathLake examples explain
the optimal 80/20 mixture? Also, do the authors have results on actual GSM8K problem-solving accuracy rather than only solution cross-entropy?

3 comments
AuthorsOpenReview ↗

Rebuttal By Authors (1/1)

We thank the reviewer for their highly constructive and insightful feedback on our work!

W1/Q2: It is unclear whether the model fails to learn heldout transitions or fails to generalize compositionally.

A1: Thank you for suggesting this diagnostic; the distinction is central to our mechanism. We now report atomic accuracy on held-out transitions in Appendix B.1, Figure 4, overlaid on the same panels as Figure 1. At q=10q = 10\\% and 3030\\%, held-out transitions are recalled almost perfectly in isolation across most mixtures, yet OOD accuracy remains clearly lower for longer chains. In these settings the model struggles to compose transitions it already knows, which is the “unseen use of a known fact” that our setting is designed to isolate. At q=50q = 50\\%, atomic recall is itself incomplete, so the OOD gap there reflects both insufficiently learned transitions and failure to compose them. This matches the structure of the theory, where the gain on held transitions is a product of fact learning, driven by atomic data, and routing at later steps, driven by reasoning data.

W2/Q4: The connection between the synthetic setting and mathematical pretraining remains limited.

A2: Thank you for raising the question! We strengthen this connection through new controlled natural-language experiments and clarify the interpretation of the mathematical-pretraining results.

New controlled natural-language experiments. In Section 4.1, we add a multi-hop reasoning task in which atomic statements and reasoning traces are constructed from the same underlying knowledge world, providing a controlled counterpart to FineMath’s “general mathematical text rather than isolated facts.” We explicitly control reasoning chain length and which facts appear in reasoning supervision, while keeping all facts available through atomic supervision. We report generated-answer accuracy across three training seeds. The changes in the preferred mixture agree in direction with the predicted dependence on reasoning length and factual coverage, though the curves are noisy. These experiments provide a controlled test in natural language, while the FineMath–MathLake experiments provide complementary evidence in a more realistic setting.

Solution cross-entropy rather than problem-solving accuracy. Cross-entropy is a standard metric for studying pretraining and resource allocation, including the scaling-law analysis like [1-2]. More recently, Olmix [3], developed for Olmo 3, evaluates data mixtures using normalized negative log-likelihood over reference responses, including mathematical responses. Our GSM8K solution cross-entropy therefore provides a meaningful measure of predictive quality under the pretraining objective. In zero-shot evaluation without fine-tuning, the model solves too few GSM8K problems for answer accuracy to reliably distinguish between mixtures. We agree that it does not directly establish improved problem-solving accuracy, which motivates reporting generated-answer accuracy in the new controlled experiments.

[1] Scaling Laws for Neural Language Models.

[2] Training Compute-Optimal Large Language Models.

[3] Olmix: A Framework for Data Mixing Throughout LM Development.

Q1: Could you clarify which sampling procedure was actually used in the experiments?

A3: In the theory (Definition 2), atomic examples sample e0e_0 and all relations independently and uniformly. In the Figure 1 experiments (Appendix B.1), the supervised pair (e0,r1)(e_0, r_1) is uniform over the full fact table, including held pairs, while each later relation is sampled uniformly from the seen outgoing relations of the current state.

In both cases the later relations are never supervised in atomic examples; they only align the atomic format with the reasoning format, so that the first prediction sits at the same position with the same number of candidate relations. The difference does not affect the leading-order result: the atomic loss enters the leading term of Theorem 1 only through (e0,r1)(e_0, r_1), which is uniform under both procedures, and the later-role routing is trained by reasoning examples alone. Empirically, the new controlled experiment in Section 4.1 uses atomic documents that state a single fact with no trailing relations, and shows the same trends.

Q3: To what extent could the increased repetition of MathLake examples explain the optimal 80/20 mixture?

A4: We thank the reviewer for raising this question! Our theory does not predict the location of the optimum in this setting. Its claim concerns how the preferred mixture shifts with coverage and reasoning length. We describe 80/20 as the best observed mixture for this model, corpus pair, budget, and metric.

We hope that our clarifications have addressed your concerns. Please let us know if there is anything else we can clarify!

Thank you for the detailed rebuttal and the additional experiments! I appreciate that the authors have addressed several of my concerns and strengthened the paper with the additional results. However, my overall assessment is that even with these improvements, the paper is not a "clear accept", so I will maintain my rating.

We thank the reviewer for the thoughtful discussion and for recognizing that the additional results strengthen the paper! We further clarify the significance of our contribution here. We hope this clarification, together with the additional evidence, will encourage reconsideration of the overall assessment.

Reviewer_zgQAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to balance atomic facts and multistep reasoning traces during pretraining. It introduces a controlled state tracking setting in which all state transitions are available through atomic supervision, but only some appear in reasoning traces. This setting allows the authors to test whether a model can use facts observed in isolation in reasoning contexts it has not encountered before. The paper combines experiments with a single-layer Transformer containing attention and an MLP, a theoretical analysis of a scalar linear attention model, and mathematical pretraining experiments using Olmo-3 150M on mixtures of FineMath and MathLake. To leading order, the theory predicts that the optimal weight on atomic supervision increases with reasoning length and decreases with the fraction of state transitions absent from reasoning traces. The experiments show benefits from mixing the two data sources in moderate proportions and examine how the amount, difficulty, and timing of reasoning data affect the results.

Critical strengths
  1. An interesting perspective on the balance and complementary roles of pretraining data. The paper distinguishes atomic facts from reasoning traces and studies how the two jointly affect compositional transfer. This perspective helps clarify the relationship between fact learning and reasoning training and provides a useful framework for studying pretraining data mixtures.
  2. A clear theoretical prediction supported by complementary experiments. Under its stated assumptions, the linear attention analysis gives a closed-form prediction for the optimal mixture rather than just an empirical trend. The experiments with softmax attention and an MLP, along with the mathematical pretraining experiments, broaden the study and provide evidence that combining the two data sources can outperform using either alone.
Critical weaknesses
  1. The natural language evaluation is too limited to establish the claimed gains in practical reasoning. The reported natural language experiments use only one 150M-parameter model and measure cross-entropy and perplexity on GSM8K reference solutions. They report neither generated answer accuracy nor variation across training seeds. These metrics measure how well the model fits reference solutions, not whether it solves problems more accurately. The robustness of the optimal mixture and its applicability to other model scales remain untested. This substantially limits the empirical support for general recommendations about pretraining for reasoning.

  2. The experiments on real data do not adequately distinguish the proposed mechanism from alternative explanations. FineMath and MathLake are not a matched pair of atomic facts and reasoning traces. In the coverage experiment, reducing the available MathLake pool while keeping the reasoning token budget fixed necessarily increases data repetition. The difficulty experiment uses problem difficulty as a proxy for reasoning length. These interventions are informative, but they do not isolate the effects of factual coverage or reasoning chain length as the controlled experiments do. Given the overfitting observed when using MathLake alone, changes in the optimal mixture could partly reflect data repetition and distributional differences rather than the mechanism described by the theory. The connection between the theoretical result and the experiments on real data remains suggestive rather than conclusive.

Questions for authors
  1. Interpretation of the coverage effect.
    In Figure 2a, smaller reasoning data pools are reused more often when the reasoning token budget is fixed. What evidence distinguishes the proposed coverage effect from overfitting caused by repeated use of the same data? Please clarify how far the comparison between this experiment and Theorem 1 holds, and whether additional controls or existing training curves support this interpretation.

  2. Scope of the “without loss of generality” assumption in Theorem 1.
    Theorem 1 assumes M=NM=N and $ \pi_r(e)=1+((e+r-2)\bmod N) $ “without loss of generality” (W.L.O.G.). However, the proof appears to rely substantively on this complete cyclic shift structure. Specifically, Eq. (15) establishes the exact balance identities

\qquad \frac{1}{N}\sum_r y_{r,e}^{2}=\nu,$$ which are then used to eliminate correlations between distinct target values in Eqs. (13) through (17). These identities do not generally hold for an arbitrary collection of permutations, including the random permutation setting with $M=8,N=64$ in Figure 1. Could you justify this reduction or state explicitly the structural assumptions required for the result to hold? More importantly, when these balance identities hold only approximately, can you characterize the additional correlation terms and explain whether they preserve the predicted dependence of the optimal mixture on $L$ and $q$? This clarification would establish how directly Theorem 1 explains the reported experimental results.
4 comments
AuthorsOpenReview ↗

Rebuttal By Authors (1/2)

We thank the reviewer for their highly constructive and insightful feedback on our work!

W1: The reported natural language experiments use only one 150M-parameter model and measure cross-entropy and perplexity on GSM8K reference solutions. They report neither generated answer accuracy nor variation across training seeds. These metrics measure how well the model fits reference solutions, not whether it solves problems more accurately. The robustness of the optimal mixture and its applicability to other model scales remain untested.

A1: We thank the reviewer for this important question.

Scope of the claim. Our main contribution is the controlled setting and the theoretical mechanism it exposes; the natural-language experiments test whether the predicted trend directions persist beyond the theorem’s assumptions. We have revised the abstract, contributions, and Section 4 so that they claim consistency with the predicted trends.

New controlled experiment on a different 12M scale. We added a controlled natural-language experiment (Section 4.1, Figure 2) that reports generated final-answer accuracy with mean and standard deviation over three training seeds. Its atomic corpus states one fact per document, reasoning examples compose the same facts, and coverage and hop count are varied independently with the number of reasoning examples held fixed. Interior mixtures beat both pure-source endpoints in every panel, and the trends in hop count and coverage agree in direction with Theorem 1, though the curves are noisy and the near-best regions are wide.

Why cross-entropy and a 150M model in Section 4.2. This section checks plausibility on real mathematical data. A 150M model serves that purpose while keeping full from-scratch sweeps feasible: each 10B-token run takes approximately 14 GPU-hours on H100, and Section 4.2 comprises 35 runs (490 GPU-hours). This scale is comparable to related theoretical work [4]. We report cross-entropy over the gold solution instead of generated-answer accuracy because a 150M base model, evaluated zero-shot without fine-tuning, solves too few GSM8K problems for accuracy to separate mixtures; likelihood-based evaluation is standard for pretraining studies [1–3].

Robustness of optimal mixture. We do not claim a stable optimum: the gap between 10% and 20% MathLake is 0.005, and we will describe 20% as the best observed setting only. What Table 1 supports is that interior mixtures (1.613 and 1.608) beat pure FineMath (1.785) and pure MathLake (7.513).

[1] Scaling Laws for Neural Language Models.

[2] Training Compute-Optimal Large Language Models.

[3] Olmix: A Framework for Data Mixing Throughout LM Development.

[4] The Power of Power Law: Asymmetry Enables Compositional Reasoning, ICML 2026.

W2: The experiments on real data do not adequately distinguish the proposed mechanism from alternative explanations.

A2: Thank you for highlighting the need to strengthen the “connection between the theoretical result and the experiments on real data.” We address this through new controlled natural-language experiments and clarify the complementary roles of our experimental settings.

New controlled natural-language experiments directly test factual coverage and reasoning chain length. In the new Section 4.1, we introduce a BIO-style multi-hop reasoning task in which atomic statements and reasoning traces form a “matched pair” constructed from the same underlying knowledge world. Every setting uses nested holdout sets, and the same question-and-solution templates. Every fact remains available through atomic supervision, while we explicitly control which facts appear in reasoning traces. We also vary the number of reasoning hops directly, removing the need to use “problem difficulty as a proxy for reasoning length.” We report final-answer accuracy across three training seeds. Interior mixtures outperform both pure-data endpoints, and the changes in the preferred mixture with factual coverage and reasoning length remain consistent with the theoretical predictions. This provides a direct bridge between the theoretical setting and natural-language training.

Increased repetition is inherent to reducing coverage under a fixed reasoning budget, and the resulting concentration of supervision is already represented in our theoretical training distribution through the 1/(pL)1/(pL) term. Moreover, if increased repetition primarily degraded performance, one would expect smaller pools to favor a lower MathLake share. Figure 3a shows the opposite direction: the smallest pool performs best at the highest tested MathLake share (30%), whereas the full pool favors 10%. This observation also runs counter to a simple explanation based on repetition-driven degradation.

(To be continued)

AuthorsOpenReview ↗

Rebuttal By Authors (2/2)

The real-data experiments provide complementary evidence in a less controlled setting. FineMath and MathLake serve as practical analogues of atomic knowledge and reasoning supervision, and difficulty serves as a proxy for reasoning length. We agree that these interventions do not isolate every factor as precisely as the controlled experiments. We therefore position Section 4.2 as evidence that the predicted qualitative trends also appear under realistic corpora. Together, the controlled state-tracking experiments, the new controlled natural-language multi-hop experiments, and the real-math experiments support the mechanism at progressively more realistic levels, with the first two providing the direct tests of factual coverage and reasoning chain length.

Q1: In Figure 2a, smaller reasoning data pools are reused more often when the reasoning token budget is fixed. What evidence distinguishes the proposed coverage effect from overfitting caused by repeated use of the same data? Please clarify how far the comparison between this experiment and Theorem 1 holds, and whether additional controls or existing training curves support this interpretation.

A3: We thank the reviewer for raising this question.

Evidence against overfitting as the explanation. We agree that smaller pools are reused more. One might expect that overfitting from reuse would favor a smaller MathLake share when the pool is small. Figure 3a (Figure 2a in the original submission) shows the opposite: the 25% pool performs best at the largest MathLake share, its most repeated setting, and the full pool at the smallest.

How far the comparison with Theorem 1 holds. The pools are nested (lines 482–483), so increasing p only adds documents and coverage increases monotonically at a fixed reasoning-token budget. This matches the theorem’s comparison qualitatively: lower coverage concentrates the same amount of reasoning supervision on a smaller support. The analogy the limit that coverage in the theorem is over transitions, whereas here it is over documents. We therefore regard this experiment as suggestive; the fully controlled test of coverage is now in Section 4.1.

Q2: Scope of the “without loss of generality” assumption in Theorem 1.

A4: We thank the reviewer for this important question!

Structural assumptions. We agree and have removed the “without loss of generality” wording. The revised Theorem 1 is stated under an explicit Assumption 1 (balanced world): M=NM = N, and for every pair of entities exactly one relation maps one to the other. Balance means the current entity alone carries no information about the next one, so the only way to reduce error is to use the fact itself. It removes a shortcut, i.e. without balance, a model can cut error by guessing an entity’s typical successor while ignoring the relation. Real models learn such frequency priors quickly, and they are a separate matter from learning facts. Balance leaves only the fact-specific signal, which is what the theorem studies. Cyclic shifts are one instance. The proof uses only two consequences of this assumption: the per-entity balance identities (Eq. 9) and the bijection between relation tuples and state paths.

When balance holds approximately. Exact balance is a simplifying assumption; our analysis also accommodates balance that holds only approximately. Define

\qquad \kappa=\frac1N\sum_{e=1}^N\mu_e^2, \qquad \nu=\frac{N^2-1}{3N^2}.$$ Exact balance gives $\mu_e=0$ for every entity. For independently sampled uniform permutations, exact balance need not hold, but $\mathbb E[\kappa]=\nu/M$, and, with probability at least $1-\delta$, $$\kappa=O\left(\frac{\log(2/\delta)}{M}\right).$$ Thus, the relevant aggregate imbalance concentrates near zero as the number of relations grows. Under the same random held-set model, these concentration estimates replace the exact balance cancellations in the proof, with the resulting errors incorporated into the approximation bounds. The same finite-time analysis therefore continues to apply under approximate balance. **Implication for Figure 1.** Figure 1 uses softmax attention, an MLP, cross-entropy. Theorem 1 therefore explains Figure 1 qualitatively, through the direction of the trends and the mechanism behind them, and not quantitatively. We hope that our clarifications have addressed your concerns. Please let us know if there is anything else we can clarify!

Dear Reviewer,

Thank you again for your insightful feedback on our work. We have posted a detailed response addressing your points, including the theoretical generalizability and the connection between theory and experiments. We further provide a brief summary of the problem significance and our contributions here.

Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time. If our responses address your concerns, we kindly request an updated score.

Best regards,

The Authors

Your detailed rebuttal and supplementary experiments are much appreciated. I am also very grateful that the authors responded to several of my concerns and strengthened the paper’s persuasiveness with additional results. Even so, these improvements do not change my overall assessment: the paper cannot yet be regarded as “fully accepted,” and I will therefore maintain my original score.

Reviewer_HAZ5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper separates atomic fact learning from compositional training through a state-tracking task and studies how reasoning-chain length and factual coverage affect the optimal data mixture. It provides an asymptotic analysis of a linear-attention model and examines related trends using a single-layer Transformer and mathematical-text pretraining experiments.

Critical strengths

Evaluating transitions seen individually but never within reasoning traces provides a useful design for studying compositional transfer. The paper offers an explicit mixture formula, detailed proofs, and experiments including both pure-source endpoints, with variability across three training seeds reported for the synthetic experiments.

Critical weaknesses

1 The theory has limited explanatory scope for practical training: the theorem requires complete cyclic shifts, M=N, and a special initialization, and these assumptions cannot be described as “without loss of generality” without justification. At a fixed training horizon as N grows, the proved error improvement is only O(N⁻⁴), leaving unclear whether the optimal mixture describes training regimes with substantial learning. 2 The natural-language experiments do not adequately isolate the theoretical factors: FineMath is not a controlled collection of atomic facts, reducing the MathLake pool also increases repetition, and difficulty splits change both content and length. Evaluation reports only solution-sequence cross-entropy and perplexity on GSM8K for one 150M model, without generated-answer accuracy or uncertainty across seeds; the cross-entropy advantage of 20% over 10% MathLake is only 0.005, which does not establish a stable optimum or improved reasoning. 3 The connection between loss weights, example ratios, and actual training budgets remains unestablished: the theory fixes gradient-flow time, synthetic experiments fix update counts, and natural-language experiments fix token counts, leaving the claimed fixed-budget allocation law without a unified validation. Moreover, the formula predicts that greater coverage p increases the atomic weight, contrary to the introduction’s interpretation that broader coverage makes reasoning data relatively more valuable.

Questions for authors

1 Why can the complete cyclic-shift assumption be made “without loss of generality,” and does the mixture law hold for other transition structures, initializations, and training regimes with substantial error reduction? 2 Can you report generated-answer accuracy and results across multiple seeds, and isolate coverage and reasoning length using controls matched for repetition, content, and difficulty? 3 Can you clarify how mixture ratios map to token and computational budgets across the three settings and resolve the opposite directions of the coverage effect in the formula and its verbal interpretation?

3 comments
AuthorsOpenReview ↗

Rebuttal By Authors (1/2)

We thank the reviewer for their highly constructive and insightful feedback on our work!

W1/Q1: The theorem requires complete cyclic shifts, M=N, and a special initialization, and these assumptions cannot be described as “without loss of generality” without justification. At a fixed training horizon as N grows, the proved error improvement is only O(N⁻⁴), leaving unclear whether the optimal mixture describes training regimes with substantial learning.

A1: Thank you for pointing out these two limitations. We clarify both the assumptions and the training regime covered by the theorem.

Transition structure. Theorem 1 is stated under Assumption 1 (balanced worlds); cyclic shifts are one instance, not the assumption itself. The proof uses only two consequences of Assumption 1: per-entity balance (Eq. 9) and the bijection between relation tuples and state paths. Balance is a centering condition (a uniformly chosen relation leads to a uniformly distributed next state), and M=NM = N follows prior state-tracking theory [1]. Balance means the current entity alone carries no information about the next one, so the only way to reduce error is to use the fact itself. It removes a shortcut, i.e. without balance, a model can cut error by guessing an entity’s typical successor while ignoring the relation. Real models learn such frequency priors quickly, and they are a separate matter from learning facts. Balance leaves only the fact-specific signal, which is what the theorem studies.

Initialization. Structured initializations are standard in analyses of transformer dynamics [2, 3]. Ours encodes a blank slate: B(0)=0B(0) = 0 means no facts are known, and C(0)=E11C(0) = E_{11} means later-role routing is untrained, with one nonzero entry to leave the saddle point. 

Training regime. We agree that the theorem characterizes early learning and does not establish the same optimum after substantial error reduction. Limited-budget analysis matters because large-scale pretraining is budget-limited: models are trained for roughly one pass and do not fully converge, so data mixtures are compared at fixed token budgets. When all interior mixtures share an optimum like our theory setting does, asymptotic analysis doesn’t meaningfully distinguishes one mixture by another because all of them will be near perfect.

The mechanism is general. The interior optimum follows from the data design, not from the model: held facts are supervised only by atomic examples, and later-step composition only by reasoning traces. In our model, pure atomic training gives later-role routing zero gradient at every horizon. The leading gain g(β)=β(1−β)[β+(1−β)/(pL)]g(\beta) = \beta(1−\beta)[\beta + (1−\beta)/(pL)] reflects this. Atomic data teach the held fact (β\beta), reasoning data teach routing (1−β1−\beta), and routing learning depends on how well in-trace facts are known, to which reasoning contributes at rate 1/(pL)1/(pL). Longer chains or wider coverage weaken that contribution, shifting the optimum toward atomic data. We therefore claim the mechanism and the direction of the trends, which Figures 1–3 support beyond the theorem’s assumptions, and present β(pL)\beta(pL) in Eq. 10 as one solvable instance, not a quantitative prescription.

[1] On the Emergence of Implicit Curriculum in RLVR Learning Dynamics, ICML 2026.

[2] How Transformers Learn Causal Structure with Gradient Descent, ICML 2024.

[3] Transformers Provably Learn Algorithmic Solutions for Graph Connectivity, But Only with the Right Data, ICML 2026.

W2/Q2: FineMath is not a controlled collection of atomic facts, reducing the MathLake pool also increases repetition, and difficulty splits change both content and length. Evaluation reports only solution-sequence cross-entropy and perplexity on GSM8K for one 150M model, without generated-answer accuracy or uncertainty across seeds; the cross-entropy advantage of 20% over 10% MathLake is only 0.005, which does not establish a stable optimum or improved reasoning.

A2: We thank the reviewer for asking this insightful question.

Controlled isolation of coverage and length. We added a controlled natural-language experiment (Section 4.1, Figure 2) designed around these concerns. The atomic corpus states one person–relation fact per document, and reasoning examples compose the same facts. Coverage and length are varied independently: holdout sets are nested across qq, hop count HH varies over the same fact table and templates. We report generated final-answer accuracy with mean and standard deviation over three seeds. Interior mixtures beat both endpoints in every panel, and the trends in HH and qq agree in direction with Theorem 1.

Role of Section 4.2. We agree it is less controlled; it tests whether analogous trends appear with real mathematical text.

(To be continued)

AuthorsOpenReview ↗

Rebuttal By Authors (2/2)

Metrics, scale, and seeds. Our main contribution is the controlled setting and the theoretical mechanism it exposes; Section 4.2 is meant to check whether the predicted trend directions are plausible on real mathematical data, not to establish optimal mixtures for pretraining at scale. A 150M model serves that purpose while keeping full from-scratch sweeps feasible: each 10B-token run takes approximately 14 GPU-hours on H100, and Section 4.2 comprises 35 runs (490 GPU-hours). This scale is comparable to related theoretical work [4]. We report cross-entropy over the gold solution instead of generated-answer accuracy because a 150M base model, evaluated zero-shot without fine-tuning, solves too few GSM8K problems for accuracy to separate mixtures; likelihood-based evaluation is standard for pretraining studies [1–3]. We have reported the bootstrap evaluation variance following [5].

The 0.005 gap. We do not claim a stable optimum at 20%. Table 1 shows that interior mixtures (1.613 and 1.608) beat pure FineMath (1.785) and pure MathLake (7.513). We have described 20% as the best observed setting, not an established optimum.

[1] Scaling Laws for Neural Language Models.

[2] Training Compute-Optimal Large Language Models.

[3] Olmix: A Framework for Data Mixing Throughout LM Development.

[4] The Power of Power Law: Asymmetry Enables Compositional Reasoning, ICML 2026.

[5] s1: Simple test-time scaling, EMNLP 2025.

W3/Q3: The connection between loss weights, example ratios, and actual training budgets remains unestablished: the theory fixes gradient-flow time, synthetic experiments fix update counts, and natural-language experiments fix token counts, leaving the claimed fixed-budget allocation law without a unified validation. Moreover, the formula predicts that greater coverage p increases the atomic weight, contrary to the introduction’s interpretation that broader coverage makes reasoning data relatively more valuable.

A3: We thank the reviewer for this important question!

Loss weights, example ratios, and budgets. In the synthetic experiments, the loss is averaged over supervised tokens. An atomic example contributes one supervised token, whereas a reasoning example contributes LL. Thus, an atomic example ratio α\alpha corresponds to

β=αα+L(1−α),\beta=\frac{\alpha}{\alpha+L(1-\alpha)},

which is also the atomic share of the supervised-token budget. The map is increasing in α\alpha for fixed LL, and its inverse is increasing in both β\beta and LL. Consequently, the predicted trends with reasoning length and factual coverage are preserved across these parameterizations and are qualitatively supported by the three experimental settings in Figures 1–3.

Direction of the coverage effect. The theorem’s direction is the correct one: broader coverage (larger p) increases the optimal atomic weight. We have revised the introduction to match.

We hope that our clarifications have addressed your concerns. Please let us know if there is anything else we can clarify!

Dear Reviewer,

Thank you again for your insightful feedback on our work. We have posted a detailed response addressing your points, including the theoretical generalizability and the connection between theory and experiments. We further provide a brief summary of the problem significance and our contributions here.

Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time. If our responses address your concerns, we kindly request an updated score.

Best regards,

The Authors

Comments on the paper

1 comment

We would like to briefly highlight the key contributions and implications of our work:

Why the problem matters. Reasoning data are increasingly incorporated into pretraining, yet a fundamental question remains: how should a fixed data budget be divided between learning individual facts and learning to reason with them? Today, this choice is largely guided by trial and error, with each experiment potentially requiring an entire pretraining run. Principled guidance could therefore have significant practical value.

What we discover. Our findings reveal a surprising insight: as reasoning becomes more complex, allocating more data to individual facts can actually be more beneficial than providing additional reasoning traces. This challenges the conventional intuition that harder reasoning requires more reasoning data. Instead, the optimal mixture depends on the structure and coverage of the reasoning task.

Our findings are well-grounded. We theoretically establish this result for a linear-attention model under explicitly stated assumptions, deriving a closed-form expression for the optimal mixture and explaining the mechanism behind this counterintuitive behavior. We further validate the predicted trends through experiments on symbolic state tracking, controlled natural-language multi-hop reasoning, and pretraining on real mathematical data.

We believe our findings offer valuable insights into more effective and data-efficient foundation-model training and will benefit the broader community. We hope the reviewer will consider further supporting our work, and would be happy to address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.