acceptodds
Under review as a conference paper at ICLR 2027

Standard Data Scaling Analyses Obscure Loss of Important Information in Math Reasoning

Abstract

In deep learning, increasing dataset size has been shown to improve the performance of deep neural networks. However, it is unclear if adding more data negatively impacts certain abilities learned from existing data, obscured by an overall net increase in performance. Understanding this is especially important in the current large language model era, where data scarcity has become a pressing issue. We discover that when performing fine-tuning on mathematical reasoning tasks, adding more training data causes the model to incorrectly answer a large portion of previously correctly answered test samples. This remains true even with popular test-time scaling techniques, which can iron out inconsistencies in model predictions. To better understand this phenomenon, we show that large language models fine-tuned on the same data learn very different functions across different random seeds, exhibiting extremely high predictive multiplicity. This work contains novel insights that encourage practitioners to investigate if additional data is yielding equitable benefits.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Math reasoning, Data scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NenbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies sample-level effects of scaling training data during LLM fine-tuning on math reasoning. While overall test accuracy rises with more data, the authors find that 10–15% of test samples that were correctly answered with smaller training sets end up incorrect as data grows. This pattern holds across supervised fine-tuning, reinforcement learning, multiple model sizes, PEFT variants, and even test-time majority voting.

The authors show this is not driven by conflicting training samples. Instead, models trained on identical data with different random seeds learn very different functions, achieving near-identical overall accuracy but sharing little overlap in correctly answered samples. They link this to predictive multiplicity / the Rashomon effect, attributing it to a mismatch between standard next-token loss and the demands of mathematical reasoning. Ablations identify sample order and LoRA dropout as two concrete sources of run-to-run variation.

Critical strengths
  1. The core observation is verified across supervised fine-tuning, reinforcement learning, multiple model families and sizes, PEFT and full-parameter training, and test-time majority voting.
  2. Virtually all existing scaling studies only track overall accuracy, which hides the fact that adding data can degrade performance on a non-trivial subset of samples. This paper calls out this gap and demonstrates that aggregate scaling metrics can obscure important sample-level outcomes.
Critical weaknesses
  1. Predictive multiplicity across random seeds and the Rashomon effect are already well documented.
  2. The paper attributes the effect to a mismatch between next-token loss and reasoning goals, but this is presented as a plausible narrative rather than rigorously tested. There is no detailed analysis of why math reasoning specifically produces such large Rashomon sets, or what training dynamics drive previously correct samples to flip wrong as data grows.
  3. All main experiments use math reasoning datasets, with only one small image classification control. It is unknown whether the same pattern appears in other reasoning-heavy tasks like code generation or logical deduction, or if it is specific to math fine-tuning properties.
  4. The work is purely observational: it documents the problem but proposes no mitigation strategies, nor offers practical guidance for adjusting training pipelines to reduce sample-level churn.
Questions for authors
  1. All main experiments are on math reasoning, with only one CIFAR-10 classification control. Have you tested whether this pattern of newly incorrect samples appears in other reasoning-heavy tasks like code generation or multi-hop QA?
  2. The paper proposes a mismatch between next-token loss and reasoning objectives drives the effect. Are there additional analyses or ablations that more directly support this mechanism, rather than just being consistent with it?
  3. The work focuses entirely on documenting the phenomenon. Did you experiment with any training strategies that might reduce the rate of samples flipping from correct to incorrect as data scales? If so, how effective were they?
Reviewer_8YFDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main idea is that by adding more data in reasoning (math) post-training the overall performance on the test set improves but some problems that were solved correctly before, the model gets wrong. The authors fine-tuned different models (Gemma, LLama) with SFT and with RL (Qwen) on math datasets and in all these setups and on different math benchmarks they show that with bigger dataset size number of newly incorrectly solved problems increases as well as the performance. They hypothesize that SFT objective (predicting next token) is different from the training objective (solving a math task) contrary to supervised image classification. So the model learns some reasoning patterns that help to minimize the loss but those patterns are not tied to correct solutions.

Critical strengths
  • Authors use different datasets, base models, training setups (SFT and RL) and different evaluation benchmarks and protocols (greedy and majority voting) to prove their point.
  • If it is indeed the case that LLMs with SFT and RL learn different functions on the same dataset and therefore have similar scores but different problem-level correctness - this will be a very interesting finding.
Critical weaknesses
  • It might be that even though the models trained on the same dataset with different seeds solve different tasks, their p@N is the same i.e. with multiple samples per prompt these models solve the same problems in total: p@N > p@1 and p@N ≈ Union(M1, M2, ...) (union of the solved problems by different models with different seeds).
Questions for authors
  • what if the pass@N (i.e. how many answers out of N samples were correct) was used and not just majority voting (what is the most frequent answer out of N). Does this number change with more data? It might be that the upper bound (union) of different subset models is p@N so SFT just sharpens the distribution and the p@1 goes up while p@N stays the same?
Reviewer_f9kQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on the problem that post-training LLMs with additional data can cause large language models to incorrectly answer a substantial portion of test samples that they previously answered correctly. This issue is generally obscured by a net increase in overall test performance.

The paper validates this phenomenon across various base models and datasets, for both SFT and RL. It also finds that the samples that change from previously correct to newly incorrect vary substantially across different random seeds. The paper explains this large discrepancy by pointing out that there is a significant mismatch between the desired outcome and the loss functions used to train the model.

Critical strengths

None

Critical weaknesses

The explanation of why “additional data can cause large language models to incorrectly answer a substantial portion of test samples that they previously answered correctly” is not convincing. The authors explain this by stating that “there is a significant mismatch between the desired outcome and the loss functions used to train the model,” but provide no direct validation. Two critical unresolved questions remain:

  1. SFT does have a mismatch between training and inference, but RL largely closes this gap and still exhibits this phenomenon. For RL, the argument that “this training paradigm is then paired with the rich and complex task of mathematical reasoning where it is difficult to find invariant reasoning rules across samples that generalize well, along with the fact that there are many strategies that can be used to correctly solve a problem” does not seem sufficient to explain the phenomenon.

  2. The authors mention that the newly incorrect samples depend heavily on the random seed and also vary across different base models. How can the effect of such noise and model-dependent variation be disentangled from the claimed underlying mechanism?

Questions for authors
  1. For a fixed training dataset, different base models have substantially different sets of samples that become newly incorrect. It is not clear why this observation supports the conclusion that the models are “unable to make complete use of their training data.” If different base models instead failed on the same set of samples, would that imply that they are making complete use of the training data?

  2. The paper shows an interesting contrast: in simple classification tasks, scaling the training data does not cause previously correct samples to become incorrect. This suggests that some property of mathematical reasoning is important. Can the authors investigate more deeply what specifically causes this difference and provide more direct evidence for the proposed explanation, rather than attributing it broadly to the complexity of mathematical reasoning?

  3. Why is the model fine-tuned from scratch for each larger subset instead of continually fine-tuning as more data is added? Does the same phenomenon hold under continual fine-tuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.