- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores how reasoning data mixtures affect LLM generalization and shows that increasing reasoning supervision over fixed patterns can improve within-pattern generalization but eventually degrade cross-pattern transfer.
Critical strengths
- The motivation of this paper reasonable and practically relevant. Understanding how the proportion of reasoning supervision affects a model's ability to utilize acquired knowledge and generalize is indeed an important question in model training.
- The paper adopts a relatively simple and clear controlled experimental setting which makes the experimental design and results easier for readers to understand.
Critical weaknesses
-
The scope of the experimental setting is somewhat limited. The experiments are primarily conducted on synthetic map-based tasks with a fixed map size. It remains unclear whether the findings generalize to different levels of task complexity (e.g., map sizes and reasoning depths) or other types of synthetic reasoning tasks.
-
The characterization and explanation of the three regimes could be further strengthened. Although the authors describe Recall, Reason, and Regress as empirical regimes, the conditions under which these regimes emerge, their transition behaviors, and the underlying mechanisms remain insufficiently understood. More systematic analysis would help clarify these observations.
-
The practical implications of the findings for model training remain somewhat limited. While the authors emphasize the importance of reasoning data mixtures and suggest selecting mixtures and checkpoints based on cross-pattern validation performance, more concrete guidance or approach on how to determine appropriate mixture proportions across different models or tasks would strengthen the practical value of the work.
-
The robustness of some experimental observations could be further examined. For example, the curves in Figures 1 and 3 exhibit considerable fluctuations, and the trends are not always consistent across different models and reasoning patterns. Further analysis of these fluctuations along with additional experiments using multiple seeds to assess the stability of the findings would further help establish the reliability of the findings.
-
"Keywords" appear below the abstract, and I am not sure whether this complies with the formatting requirements.
Questions for authors
see Weakness.
(2/2)
Responses to Weakness 3, Weakness 4, and Weakness 5: Practical guidance, robustness, and metadata
Weakness 3 -- practical training guidance.
We now frame the procedure as an empirically motivated post-knowledge-training workflow rather than a validated pretraining recipe. It recommends held-out-pattern validation, a coarse multi-seed proxy sweep, monitoring a parser-defined template-capture diagnostic, and refinement only near the selected ratio. The full replay is retrospective on the 8x8 fixed3 cohort; the 10x10 result is an illustrative check, not an independent strategy replay. Since the current selection reuses composition items, a prospective study should reserve maps before search and test cross-scale transfer.
Weakness 4 -- robustness and seed variability.
The low-dose gain occurs in all fixed3 and larger-world seeds; the medium-to-high contrast is negative in 3/3 fixed3 seeds, 4/5 joint5 seeds, and 3/3 larger-world seeds. Appendix D includes the complete final-epoch tables, individual seed curves, epoch-wise trajectories, and the paired bootstrap protocol. These are exploratory seed-level summaries: the larger-world frozen criterion is not fully met and individual curves can rebound. The results support a reproducible regime-level tendency, not universal pointwise monotonicity.
Weakness 5 -- Keywords formatting.
We removed the nonstandard Keywords line from the anonymous front matter so that the submission follows the conference template. This formatting change does not affect the scientific content.
Revision location: Section 5, lines 593--609; Section 6, lines 682--705; Appendix C, lines 46--64; Appendix D, lines 66--158; manuscript front matter.