acceptodds
Under review as a conference paper at ICLR 2027

Recall, Reason, Regress: How Reasoning Data Mixtures Shape Generalization in LLMs

Abstract

How does increasing the share of reasoning supervision change an LLM’s ability to use what it knows? We investigate this question in a controlled relational reasoning framework that separates fact recall, within-pattern generalization, and cross-pattern transfer. Across four model sizes (4B–32B) and 15 nonzero data mixtures, we vary the number of reasoning demonstrations under a fixed training-row budget while holding the demonstrated composition patterns constant. We identify three empirical regimes: Recall, Reason, and Regress. Knowledge-only training supports perfect fact and rule recall but fails to produce correct composite answers. Sparse reasoning demonstrations bridge this gap, enabling generalization to new instances and undemonstrated compositions, with within-pattern errors decreasing approximately as a power law. Yet further increasing supervision can reverse cross-pattern gains while within-pattern accuracy remains near saturation: doubling the reasoning-data share from 6.4% to 12.8% reduces cross-pattern accuracy by 8.82–20.31 percentage points across all four models, with within-pattern changes of at most 0.16 points. This decline persists when independently probed prerequisites remain accessible; pattern-specific shifts in generated solutions are consistent with specialization to demonstrated patterns. At a common training endpoint, model scale raises observed peak transfer from 55.94% at 4B to 96.74% at 32B, but the grid-maximizing mixture does not vary monotonically with model size. These findings make cross-pattern transfer an essential objective for selecting reasoning mixtures: continued gains on demonstrated patterns can conceal a narrowing of compositional generalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reasoning Supervision, Compositional Generalization, Data Mixtures, Knowledge Utilization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VudtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how reasoning data mixtures affect LLM generalization and shows that increasing reasoning supervision over fixed patterns can improve within-pattern generalization but eventually degrade cross-pattern transfer.

Critical strengths
  1. The motivation of this paper reasonable and practically relevant. Understanding how the proportion of reasoning supervision affects a model's ability to utilize acquired knowledge and generalize is indeed an important question in model training.
  2. The paper adopts a relatively simple and clear controlled experimental setting which makes the experimental design and results easier for readers to understand.
Critical weaknesses
  1. The scope of the experimental setting is somewhat limited. The experiments are primarily conducted on synthetic map-based tasks with a fixed map size. It remains unclear whether the findings generalize to different levels of task complexity (e.g., map sizes and reasoning depths) or other types of synthetic reasoning tasks.

  2. The characterization and explanation of the three regimes could be further strengthened. Although the authors describe Recall, Reason, and Regress as empirical regimes, the conditions under which these regimes emerge, their transition behaviors, and the underlying mechanisms remain insufficiently understood. More systematic analysis would help clarify these observations.

  3. The practical implications of the findings for model training remain somewhat limited. While the authors emphasize the importance of reasoning data mixtures and suggest selecting mixtures and checkpoints based on cross-pattern validation performance, more concrete guidance or approach on how to determine appropriate mixture proportions across different models or tasks would strengthen the practical value of the work.

  4. The robustness of some experimental observations could be further examined. For example, the curves in Figures 1 and 3 exhibit considerable fluctuations, and the trends are not always consistent across different models and reasoning patterns. Further analysis of these fluctuations along with additional experiments using multiple seeds to assess the stability of the findings would further help establish the reliability of the findings.

  5. "Keywords" appear below the abstract, and I am not sure whether this complies with the formatting requirements.

Questions for authors

see Weakness.

1 comment

(2/2)

Responses to Weakness 3, Weakness 4, and Weakness 5: Practical guidance, robustness, and metadata

Weakness 3 -- practical training guidance.

We now frame the procedure as an empirically motivated post-knowledge-training workflow rather than a validated pretraining recipe. It recommends held-out-pattern validation, a coarse multi-seed proxy sweep, monitoring a parser-defined template-capture diagnostic, and refinement only near the selected ratio. The full replay is retrospective on the 8x8 fixed3 cohort; the 10x10 result is an illustrative check, not an independent strategy replay. Since the current selection reuses composition items, a prospective study should reserve maps before search and test cross-scale transfer.

Weakness 4 -- robustness and seed variability.

The low-dose gain occurs in all fixed3 and larger-world seeds; the medium-to-high contrast is negative in 3/3 fixed3 seeds, 4/5 joint5 seeds, and 3/3 larger-world seeds. Appendix D includes the complete final-epoch tables, individual seed curves, epoch-wise trajectories, and the paired bootstrap protocol. These are exploratory seed-level summaries: the larger-world frozen criterion is not fully met and individual curves can rebound. The results support a reproducible regime-level tendency, not universal pointwise monotonicity.

Replicated resultfixed3joint5independent 10x10
Seeds353
Negative medium-to-high contrasts3/34/53/3
High-minus-medium AUA_U-15.29 pp-18.53 pp-17.75 pp
High-ratio reboundNo required reboundR=5120R=5120 close to R=2560R=2560Rebound at R=4114R=4114

Weakness 5 -- Keywords formatting.

We removed the nonstandard Keywords line from the anonymous front matter so that the submission follows the conference template. This formatting change does not affect the scientific content.

Revision location: Section 5, lines 593--609; Section 6, lines 682--705; Appendix C, lines 46--64; Appendix D, lines 66--158; manuscript front matter.

Reviewer_TnrrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the effect of different data mixtures on compositional generalization in LLMs. The experiments are based on a synthetic environment, which is a navigation task over an 8x8 grid with named residents and their relative positions to each other (e.g., Alice Hall’s north neighbor is Peter Martinez). An atomic fact is a triple of resident name, direction, and the corresponding neighbor name in that direction. A rule is a sequence of directions; for example, a type-A rule is north→east. There are 2 fine-tuning stages: (1) stage 1 fine-tunes on facts and rules using a question-answering format, and then stage 2 fine-tunes on a mixture of demos, rules, and facts. The dataset for stage 1 is fixed, while datasets in stage 2 vary by the ratio of demo samples. There are several findings on how the ratio of solution demonstration and model scale affects the resulting compositional generalization.

Critical strengths
  • The main research question is interesting, and the proposed experimental setting is useful for separating factuality recall, rules, and rule composition.
  • Experiments are conducted on different model sizes, from 4B to 32B parameters.
Critical weaknesses
  • The paper requires significant improvement in terms of presentation. In Section 2, it’s unclear what a demo sample and a fact sample are. The format of prompts and responses is also not provided. This significantly reduces reproducibility and makes the paper difficult to follow.
  • There is a lack of discussion of how the synthetic setting relates to general reasoning and practical fine-tuning scenarios. Some settings seem to be arbitrary. For example, the range of demonstration ratio (R) is set from 0.04% to 12.8% without a clear motivation, making it difficult to evaluate the significance of the findings.
  • The novelty of the findings is also limited. For instance, [1] demonstrates that compositional generalization is limited when fine-tuning on mathematical reasoning data. Other findings, such as “sparse demonstrations enable generalization,” are difficult to interpret due to unclear presentation in Section 2.

[1] Sun, Y, Hu, S, Zhou, G, Zheng, K, Hajishirzi, H, Dziri, N & Song, D 2025, OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization

Questions for authors

Please refer to the weaknesses.

Reviewer_xj4YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies an interesting problem: how the data mixture of original facts, rules, and compositional questions influences the model's capability of solving unseen questions with parametric knowledge—specifically, how it influences within-pattern generalization (using rules seen in training data's compositional questions) and cross-pattern generalization (using rules excluded from the training data's compositional questions).

The authors created a dataset of 32 8×8 maps and 8 rules for different direction combinations. They first train the model on a mixture of facts and rules to acquire knowledge, followed by a second stage of training where the model additionally learns to solve compositional questions. They observe that while mixing a small amount of compositional questions can already help models achieve within-pattern generalization, their cross-pattern generalization is unstable and shows a regression behavior—with the increase in the number of compositional questions included, the models first improve their cross-pattern generalization, but then the generalization gets worse with more compositional questions.

The authors further analyze the performance of models and find that the models can successfully recall the rules correctly but fail to use these rules consistently. Moreover, although a model of a larger scale typically achieves better peak cross-pattern generalization, the performance is not stable with different ratios of questions, meaning that larger models can generalize worse than smaller models.

Critical strengths
  1. Meaningful research question: in general, understanding how to make models learn composable and generalizable parametric knowledge is a meaningful direction. As previous works have shown, mixing in questions is helpful for this, making a clear understanding of the optimal mixture timely and useful.

  2. The empirical framework is clean, setting clear boundaries between factual recall, within-pattern generalization, and cross-pattern generalization. This framework could be adopted by future work to study relevant phenomena.

Critical weaknesses
  1. I don't think the results can fully support the main claim of a three-stage learning process. Cross-pattern generalization fluctuates throughout, instead of showing a consistent pattern of first increasing and then declining. I would suggest that the authors report results for different mixture ratios across more seeds to address this gap.

  2. It is unclear whether the two-stage training process differs in behavior from regular single-stage training. The paper does not provide a comparison, making it harder to relate the results to real-world training.

  3. The setup is not fully clear from the paper: the paper describes the idea and major parameters for constructing the data. However, the details of the data generation process, training process, and hyperparameter selection are unclear. In particular, the lack of exact data formats, such as templates for verbalization and reasoning steps, makes some analyses hard to follow. The authors could consider including more details in the appendix and, perhaps more importantly, adding a figure that details the data format and generation process in the main text. The paper cited for data construction does not use the same setup as this paper either.

  4. The paper could benefit from a more thorough error analysis. For example, the paper has shown that models can correctly recall the rules—however, how exactly do the models fail to produce the correct answer afterward? Do they fail to follow the directions specified by the rules, such as going south instead of north, or do they follow the correct direction but end up with the wrong neighbor? Also, what is the error rate at each step? Understanding this could help me better understand why the models fail to generalize and propose methods to address this failure.

  5. Because of the four issues I have mentioned above, it is hard to identify a concrete takeaway for model pretraining. Indeed, the data mixture can influence generalization; however, this has been shown before. The interesting question is how to choose the mixture effectively—and I do not think the current evidence makes our understanding substantially deeper.

Questions for authors
  1. Can you add more details about the experiments and validate the consistency of the results?

  2. Would the results remain the same with single-stage training?

  3. What do you think is the difference between ICL and parametric knowledge here? I am asking because, if the model successfully decodes the correct direction combination for a rule, then the remaining task is to follow this instruction in context while retrieving the relevant facts from parametric memory. Would providing these facts in context make the task substantially easier? I am curious about the potential gap between these two settings.

2 comments

(2/3)

Response to Weakness 3: Experimental setup and input-output format

We agree that the original presentation did not make the supervisory roles and evaluation split sufficiently concrete. The revised Section 2 now specifies 16 demonstration maps and 16 target maps with disjoint residents; Stage 1 teaches all atomic facts and all eight rules; worked execution demonstrations cover A--D only on demonstration maps; A--D are within-pattern tests, while E--H are undemonstrated compositions on target maps. It also gives the 20,000-row Stage-2 budget, learning-rate and decoding settings, and the fact that target-map composite questions are withheld from training.

ComponentRevised specification
Fact recordMap, resident, direction, and neighbor; Stage 1 covers both demonstration and target maps
Rule recordType-A--H direction definitions; all eight rules are taught in Stage 1
Execution recordRule declaration, intermediate residents/directions, and final answer; demonstrations cover A--D only
Evaluation splitA--D: new instances of shown patterns; E/F: unseen two-hop patterns; G/H: unseen three-hop patterns
Data split16 demonstration maps / 16 target maps with disjoint resident identities

Appendix C, Figure C1, and the literal records give the complete input-output format, generation counts, training configuration, and scoring rules. We also clarify that the comparison fixes the row budget, factual coverage, and demonstrated pattern set, while fact replay, rule exposure, instance diversity, and supervised-token composition co-vary with RR. The citation language now describes a controlled synthetic relational-world construction rather than attributing our grid construction to prior work.

Response to Weakness 4: Execution-error analysis

We replaced the preliminary overlapping diagnostic with a mutually exclusive first-discrepancy partition. For every wrong answer, the first visible discrepancy is assigned to one category; percentages use all wrong answers in the corresponding group as the denominator.

Cohort / groupWrong answersNo parsedDirectionNeighborStop at correct prefix
fixed3 G/H16,1770.78%29.11%13.14%56.78%
joint5 G/H31,3611.55%38.49%10.24%49.64%
independent 10x10 G/H16,4910.03%26.81%15.82%56.85%

The complete partitions are Tables F1--F3. The knowledge-gated step-risk tables further show why the third hop is fragile: for G/H hop 3, the exact-hop rate is 44.73% in joint5 and 43.14% in the independent world, with 42.14% and 42.48% missing-current-hop rates, respectively. Parsed edges can remain valid map edges even when the declared or executed program is wrong. This supports a behavioral separation between local edge retrieval and rule selection/execution, but it does not establish faithful internal execution or a unique causal mechanism. Appendix F states the parser limitations, and Appendix G provides verbatim examples.

Revision location: Section 2, lines 67--208; Section 4, lines 370--532; Appendix C, lines 9--64; Appendix F, Tables F1--F5 and lines 217--343; Appendix G, lines 344--362.

(3/3)

Response to Weakness 5: Practical mixture-ratio selection and Question 3: In-context versus parametric knowledge

We agree that demonstrated-pattern accuracy alone is not a sufficient selection objective. The new retrospective replay is reproduced below.

StrategyRunsSelected RRSelection detailThree-seed mean AUA_U
Full grid x 3 seeds51640three-seed mean69.7%
Full grid x 1 seed17160 / 80 / 640seed-dependent42.8% / 53.4% / 69.7%
Coarse grid x 3 seeds15640three-seed mean69.7%

The independent 10x10 cohort is an illustrative peak check, not a second strategy replay: its three-seed mean favors approximately the same 3.2% share, while individual seed peaks vary. Because the current replay reuses composition items for selection and evaluation, we present it as an empirically motivated retrospective procedure. A prospective workflow should reserve maps before ratio search and validate transfer across model scales.

The new ICL control is also reproduced here. ICL-P1 supplies rules, A--D worked examples, and query-specific facts; ICL-P2 adds 20 same-map distractor facts.

Model stateClosed-book fact/ruleICL-P1 AS/AUA_S/A_UICL-P2 AS/AUA_S/A_U
Base model0 / 099.97 / 99.7983.48 / 77.95
Stage-1 model100 / 10015.43 / 15.311.69 / 1.94

For completeness, the high-share prompt-intervention comparison reports the following cross-pattern AUA_U values (means over five checkpoints per mixture):

RRP0P1P2P3P4
128042.5639.1825.809.0132.24
256037.5841.6631.7713.3035.15

This shows that the fully specified in-context task is accessible, but it is not a facts-only ablation and is not interchangeable with parametric knowledge. The P0--P4 interventions change prompt format, information order, and distractor content, so we interpret them as prompt-sensitivity controls rather than isolated retrieval tests. Table C1 and Table E1 are reproduced above; the complete P0--P4 comparison, including ASA_S and standard deviations, is Table E2.

Revision location: Section 4, lines 381--391; Section 5, lines 534--609; Appendix C, Table C1 and lines 46--64; Appendix E, Tables E1--E2 and lines 159--195.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.