- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper shows when to mix in reasoning data in pretraining data and what
the right balance is between atomic facts and reasoning data. They start
with experiments in a toy State Tracking setting, using a one-layer
transformer. Figure 1 nicely shows their finding that both ends of the
spectrum, having only atomic facts and only reasoning data, lead to poor OOD
generalization. It also shows that the more transitions are excluded from
reasoning training, the lower the model's OOD accuracy gets, while ID
accuracy generally remains higher. They also consider different lengths of
reasoning chains.
To provide a theoretical explanation for the optimal balance between atomic
facts and reasoning data, they simplify the model to a linear-attention
model. The optimal atomic-loss weight depends on reasoning-chain length and
the coverage of facts in reasoning traces, increasing with chain length and
decreasing with the held-transition fraction.
After setting the theoretical foundation, they are experimenting with
natural language data, for which they train Olmo-3 150M models using
FineMath -
a general mathematical text dataset - and MathLake - a mathematical
reasoning
dataset with problems of varying difficulty, which require different numbers
of output tokens to be solved. Also, here they first test the ratio of text
data to reasoning data, then examine reasoning-data coverage and problem
difficulty, and finally fix the ratio to examine when to introduce reasoning
data. And their main finding is that an 80/20 split between text and
reasoning data performs best in their experiments, as measured by GSM8K
solution cross-entropy. They also compare math problems with varying
difficulty and find no benefit to introducing reasoning data later in
training; therefore, they recommend mixing in reasoning data right from the
beginning.
Critical strengths
- Controlled experiments investigating the balance between atomic facts and reasoning data.
The authors introduce a well-controlled state-tracking setting in which they
can independently vary the ratio of atomic facts to reasoning data, the
length of reasoning chains, and the fraction of transitions excluded from
reasoning training. I particularly liked Figure 1, which nicely illustrates
how both types of supervision contribute to OOD generalization. This is a
strength because it allows the authors to investigate the trade-off
systematically rather than only comparing a few fixed data mixtures.
- Combining theoretical analysis with experiments on mathematical
pretraining.
In addition to the synthetic experiments, the authors derive a theoretical
characterization of the optimal balance using a simplified linear-attention
model. They then investigate whether similar trends can be observed when
pretraining OLMo-3 150M models on mathematical data. I think this
combination
is valuable because it provides both a theoretical perspective on the
problem and experimental evidence beyond the synthetic setting.
Critical weaknesses
- It is unclear whether the model fails to learn heldout transitions or fails to generalize compositionally.
One concern I have with Figure 1 is that the authors show a large difference
between ID and OOD accuracy, but do not separately report whether the model
has successfully learned the heldout transitions through atomic supervision.
This makes it difficult to determine whether the model struggles to compose
transitions it already knows or whether it has simply not learned those
transitions sufficiently well. I think evaluating atomic accuracy on heldout
transitions would help clarify this distinction and strengthen the
conclusions about compositional generalization.
- The connection between the synthetic setting and mathematical pretraining remains limited.
While I appreciate that the authors extend their experiments to mathematical
pretraining, the two settings are quite different. FineMath is used as a
proxy for atomic data, although it contains general mathematical text rather
than isolated facts. Furthermore, the 80/20 mixture is evaluated using GSM8K
solution cross-entropy rather than problem-solving accuracy, and changing
the
reasoning-data ratio also changes how frequently MathLake examples are
repeated. This makes it difficult to determine how directly the findings
from the synthetic setting translate to realistic pretraining and how
broadly the optimal mixture can be generalized.
Questions for authors
In Def. 2, you state that r2,...,rL belong to S, although S contains
entity–relation pairs, while the formal definition samples all relation
sequences uniformly. Could you clarify which sampling procedure was actually
used in the experiments?
Could you report atomic accuracy on heldout transitions? This would help
distinguish whether the model fails to learn these transitions or struggles
to compose them despite knowing them.
To what extent could the increased repetition of MathLake examples explain
the optimal 80/20 mixture? Also, do the authors have results on actual GSM8K
problem-solving accuracy rather than only solution cross-entropy?
3 comments
Thank you for the detailed rebuttal and the additional experiments! I appreciate that the authors have addressed several of my concerns and strengthened the paper with the additional results. However, my overall assessment is that even with these improvements, the paper is not a "clear accept", so I will maintain my rating.
We thank the reviewer for the thoughtful discussion and for recognizing that the additional results strengthen the paper! We further clarify the significance of our contribution here. We hope this clarification, together with the additional evidence, will encourage reconsideration of the overall assessment.
Rebuttal By Authors (1/1)
We thank the reviewer for their highly constructive and insightful feedback on our work!
A1: Thank you for suggesting this diagnostic; the distinction is central to our mechanism. We now report atomic accuracy on held-out transitions in Appendix B.1, Figure 4, overlaid on the same panels as Figure 1. At q=10 and 30, held-out transitions are recalled almost perfectly in isolation across most mixtures, yet OOD accuracy remains clearly lower for longer chains. In these settings the model struggles to compose transitions it already knows, which is the “unseen use of a known fact” that our setting is designed to isolate. At q=50, atomic recall is itself incomplete, so the OOD gap there reflects both insufficiently learned transitions and failure to compose them. This matches the structure of the theory, where the gain on held transitions is a product of fact learning, driven by atomic data, and routing at later steps, driven by reasoning data.
A2: Thank you for raising the question! We strengthen this connection through new controlled natural-language experiments and clarify the interpretation of the mathematical-pretraining results.
New controlled natural-language experiments. In Section 4.1, we add a multi-hop reasoning task in which atomic statements and reasoning traces are constructed from the same underlying knowledge world, providing a controlled counterpart to FineMath’s “general mathematical text rather than isolated facts.” We explicitly control reasoning chain length and which facts appear in reasoning supervision, while keeping all facts available through atomic supervision. We report generated-answer accuracy across three training seeds. The changes in the preferred mixture agree in direction with the predicted dependence on reasoning length and factual coverage, though the curves are noisy. These experiments provide a controlled test in natural language, while the FineMath–MathLake experiments provide complementary evidence in a more realistic setting.
Solution cross-entropy rather than problem-solving accuracy. Cross-entropy is a standard metric for studying pretraining and resource allocation, including the scaling-law analysis like [1-2]. More recently, Olmix [3], developed for Olmo 3, evaluates data mixtures using normalized negative log-likelihood over reference responses, including mathematical responses. Our GSM8K solution cross-entropy therefore provides a meaningful measure of predictive quality under the pretraining objective. In zero-shot evaluation without fine-tuning, the model solves too few GSM8K problems for answer accuracy to reliably distinguish between mixtures. We agree that it does not directly establish improved problem-solving accuracy, which motivates reporting generated-answer accuracy in the new controlled experiments.
[1] Scaling Laws for Neural Language Models.
[2] Training Compute-Optimal Large Language Models.
[3] Olmix: A Framework for Data Mixing Throughout LM Development.
A3: In the theory (Definition 2), atomic examples sample e0 and all relations independently and uniformly. In the Figure 1 experiments (Appendix B.1), the supervised pair (e0,r1) is uniform over the full fact table, including held pairs, while each later relation is sampled uniformly from the seen outgoing relations of the current state.
In both cases the later relations are never supervised in atomic examples; they only align the atomic format with the reasoning format, so that the first prediction sits at the same position with the same number of candidate relations. The difference does not affect the leading-order result: the atomic loss enters the leading term of Theorem 1 only through (e0,r1), which is uniform under both procedures, and the later-role routing is trained by reasoning examples alone. Empirically, the new controlled experiment in Section 4.1 uses atomic documents that state a single fact with no trailing relations, and shows the same trends.
A4: We thank the reviewer for raising this question! Our theory does not predict the location of the optimum in this setting. Its claim concerns how the preferred mixture shifts with coverage and reasoning length. We describe 80/20 as the best observed mixture for this model, corpus pair, budget, and metric.
We hope that our clarifications have addressed your concerns. Please let us know if there is anything else we can clarify!