acceptodds
Withdrawn from ICLR 2027

Generalization Before Memorization: How Early Objective Prioritization Shapes Subsequent Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) both learn abstract patterns that generalize to unseen examples and memorize specific knowledge that cannot be inferred from patterns. Although models typically learn both objectives concurrently, we identify a fundamental learning order asymmetry: prioritizing generalization first leads to stronger subsequent generalization, whereas prioritizing memorization first can make generalization harder to acquire. Across 54 model–dataset configurations spanning six architectures and nine tasks, generalization-first training improves generalization performance in 74.1% of cases while preserving memorization. Conversely, a memorization-first regime frequently prevents models from developing skills beyond memorizing data. To further investigate this finding, we conduct an analysis of intermediate neural representations across training, and isolate specific sub-networks responsible for this behavioral divergence. Our findings demonstrate that training order governs optimization trajectories, highlighting the need to prioritize generalization early in LLM training schedules.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
memorization, generalization, training dynamics, objective prioritization, language models, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_14Y7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how training on structured or unstructured data - encouraging model generalization and memorization respectively - affects subsequent learning from a mixture of both data. The authors find that memorization hinders subsequent generalization from the full data mixture across a wide variety of models and tasks, and suggest that this is a curriculum effect. A mechanistic dependence between the circuits underlying memorization and generalization is suggested through ablation experiments.

Overall, the paper purports to characterize a phenomenon for which a mechanistic account could contribute to the growing understanding of the interaction between memorization/generalization. However, the paper should be rejected because 1) significant details of the experiment design are left out, hindering any assessment of the results and 2) questions of the methods as presented cast doubt on the presented results.

Critical strengths

None

Critical weaknesses

The most significant omissions relate to the following:

  • Basic details of the training setting, including the objective function and the sampling scheme for mixed GEN/MEM training (151), are unspecified. Experiment results are later presented without specification of relevant parameter settings, such as loss mixture weight (Figure 5).
  • The main mechanistic analysis involves identifying subsets of neurons which are causally implicated in generalization performance. The method to identify these neuron subsets is never explained throughout the main text or appendices; instead, the method is described only in reference (161). As a result, the implementation as detailed in the appendices refers to parameters (1235), “hard-concrete” gates (1217), and “analyzed coordinates” (1281) that are never defined or specified.
  • The performance of models trained under different curricula is compared in terms of “wins” and “ties” (233), without any description of the applied criteria. It is important that this criteria, whatever it may be, involve an assessment of significance of the observed performance gap. The corresponding data (Figure 3) do not include error bars.
  • Furthermore, the main experiment investigating the interactions between memorization and generalization involve the ablation of the generalization-implicated neurons, but the ablation method is never specified across the main text or appendices (194, 355).

Major questions of method and interpretation include:

  • The authors claim that the suppression of generalization after memorization is a result of the initial memorization curriculum. However, the effect size as presented in the main text appears small: only 4 more models showed impaired generalization performance when memorization is trained first rather than interleaved throughout (Table 1).
  • The method to derive the GEN-implicated neuron family selects the subset which maximally degrades GEN accuracy. As a result, a family is always identified even if none meaningfully degrade accuracy. It would be more principled to choose a threshold degradation, and consider all families which meet this threshold as GEN-associated.
  • Results are provided from ablation of the GEN-family for the MemMix model (Fig. 4). However, the method for identifying the GEN-family states that this is done at the final GEN-only checkpoint (354), which should not exist for the MemMix model. If ablations are done in the MemMix model using a neuron family identified in the separately trained GenMix, this presents a major methodological error.
  • The experiments ablating the GEN-associated family (357) are difficult to interpret without comparison to a null experiment, perhaps one which ablates a random family.

Minor comments:

  • It would be helpful to justify the decision to train the PureMix setting for 2K iterations (153). I could see an argument for matching the Mix training iterations across the 3 curricula.

  • The MEM-target construction uses a very particular (200 - 1998) range to sample the target from (Table A.2). How was this range chosen? Why not shuffle the targets instead?

  • What is the motivation for the choice of phase length K (Table B.2) for each model?

  • A_GEN and A_MEM are referenced without definition (141).

  • The main text refers to a series of analyses without providing any description of their design, including the measured quantities (204).

  • The second question (249) is posed as unanswered, yet it seems to have been already resolved at this point by the PureMix and GenMix training results.

  • The results for the loss-weighting experiment are discussed without any reference to the data provided in Table 2 (268).

  • Why are microbatches reported for each model (Table B.3)? This seems like an implementation detail that has no impact on the scientific results.

Questions for authors

Introduction

  • It may be simpler to state that memorization has unstructured prompt-target associations (039)
  • Could be clearer to call the three settings Gen-Mix, Mem-Mix, and Mix
  • “Higher” (055) with respect to what performance metric? The percentage comparison feels unnecessary here

Related work

  • Run on sentence (108)
  • Doesn’t continual learning involve studying interleaved curricula? (119)

Experimental setup

  • What are A_GEN (138) and A_MEM (141)?
  • Are GEN and MEM sampled equally, or randomly? (151)
  • What is the loss function?
  • Need to include at minimum a conceptual summary of the referenced neuron family identification technique (150)
  • How is ablation of the neuron families performed? (194)
  • The method to derive the “GEN-associated family” selects the neuron subset which maximally degrades GEN accuracy. As a result, a family could be identified as GEN-associated even if it only marginally degrades accuracy. It would be more principled to choose a threshold significant degradation, and consider all families which meet this threshold as GEN-associated.
  • “Phase-1” and “Phase-2” are never defined (206)
  • The three follow-up analyses (204) need, at minimum, conceptual summaries in the main text of the method and recorded quantities

Figure 2

  • “Subnetwork” in panel 2 is misspelled
  • What is the meaning of the half-filled gates in panel 2?

Table 1

  • This table seems unnecessary, since it contains only 2 relevant values

Results

  • What is the “win” and “tie” criteria (230)? It is important that these measures be designed to detect only significant differences in final accuracy
  • PureMix results are repeatedly qualitatively summarized but never provided in the main text (239)
  • It would be helpful to move the 6 accuracy values and confidence intervals to a table, rather than listing in the main text (242). In addition, data needs to be shown to evidence the purported good memorization performance (246).
  • The second question has already been answered by the PureMix and GenMix training results (249)
  • It would be helpful to justify the decision to train the PureMix setting for 2K iterations. I could see an argument for matching the Mix training iterations across the 3 curricula.
  • The results for the loss-weighting experiment are discussed without any reference to the data provided in Table 2 (268)
  • What is the “fixed confirmation set” (355)?
  • The experiments ablating the GEN-associated family could benefit by a comparison to a null experiment, perhaps one which ablates a random family (357)
  • It would be helpful to justify why the authors chose to constrain activations rather than weights (368)
  • It seems odd that the anchor loss is provided (Eq. 2) when the general loss function used is never specified
  • The authors never provide an interpretation of the activation anchoring results (430)

Discussion

  • The analyses are done on a variety of model and task settings, but any effect of these factors on the reported phenomenon are never shown or discussed

Appendices

  • Why is A_GEN never detailed, while A_MEM is (Eq. A.1)? One could write the accuracy more generally and cover both simultaneously.
  • The MEM-target construction seems to use a very particular (200 - 1998) range to sample the target from. How was this range computed? Why not shuffle the targets instead?
  • The discussion of the MEM evaluation seems unnecessary and overly verbose (781)
  • What is the motivation for the choice of phase length K (Table B.2) for each model?
  • Why are microbatches reported for each model (Table B.3)? If typical gradient accumulation is used, this is an implementation detail that has no impact on the scientific results
  • Eq. B.3 is redundant since the memorization accuracy A_MEM was previously defined
  • Why are the differences in accuracy not reported in a table? (1126)
  • The explanation of the family construction provides settings for parameter values (βHC\beta_{HC}, γ\gamma, ξ\xi) that are never defined because the “low-complexity subnetwork-probing framework” is never described
  • The fixed reference scale (Eq. E.2) computed as part of the activation-anchor intervention is a reasonable part of the method design, but never motivated
  • How is the “calibration set” formed? (1376)
  • Table E.4 references units of “pp” which are never defined.

Figure 3

  • The overlapping labels near the origin are unreadable
  • It would be helpful to provide error bars for these results to help assess significance
  • Comparing the final GEN accuracies across task, model size, training method, etc may reveal interesting dependencies

Figure 4

  • The figure claims to ablate the GEN-family for the MemMix model. However, the method for identifying the GEN-family states that this is done at the final GEN-only checkpoint, which does not exist for the MemMix model. This discrepancy needs clarifying.

Figure 5

  • How is drift measured?
  • What are the λanchor\lambda_{\text{anchor}} values used to generate these data?

Table 2

  • It could be more informative to plot the two accuracies as functions of α\alpha for the three model/task pairs
Reviewer_PM7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effect of asymmetry in learning order of generalization and memorization in subsequent learning. The authors design two tasks, GEN, a structural rule learning task and, MEM, a task with fixed prompted-target associations. They compare model behaviours in GenMix (training first with GEN and then GEN + MEM) and MemMix (training first with MEM and then GEN + MEM), taking baseline of PureMix (GEN+MEM training). Across model-dataset evaluations, the paper claims 1) GenMix leads to better generalization, 2) MemMix leads to worse generalization if not trained only with GEN afterwards and 3) functional evidence based on activation drifts.

Critical strengths
  1. The authors cover a broad model and tasks spanning 54 combinations, including different model states (pretraining vs. finetuning).
  2. The paper does not only provide evaluations but also provides further analysis. Functional subnetwork analysis is interesting.
Critical weaknesses
  1. Construction of GEN/MEM split is not described in details and it's unclear if careful control is done other than random splits including frequency of tokens, operations for arithmetic tasks etc.

  2. Looking at Figure 3, there seems to be a systematic difference between small vs. big models, which is also correlated with the choice of pretraining vs finetuning. The authors do not consider this difference.

  3. I found some interpretation and description of results unclear/overclaiming - e.g. line 319 states "Together with strong recovery of GEN at α=0\alpha = 0)", but "recovery" is not the right term when we are only fully training with GEN with α=0\alpha=0.

  4. It's unclear if the model was trained to convergence for all the stages; is there a figure for the learning curves for each stage?

Questions for authors
  1. Could you explain the differences that we see in figure 3? e.g. bigger models (Qwen 9B) doesn't show much difference in GenMix/MemMix, which might suggest that the meaningful between two settings we see is only plausible in either small model or pretraining.

  2. How is it different from forgetting studied in continual learning? The results in table also reads that generalization task is more easily forgettable when there is less optimization pressure (due to distributed pressure with memorization task) than the memorization task.

Reviewer_UMQ2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors run a series of data ordering and mixing experiments to determine the effect of memorization and generalization in training. They explicitly construct memorization and generalization objectives using deliberately incorrect targets and rule-consistent targets, respectively. The authors find that memorizing first does not prevent a model from subsequently learning a general rule, but MEM-first training yields lower generalization than GEN-first training when both are followed by mixed training. The authors replicate their experiments across tasks and models. Finally, the authors run identify a subnetwork which contributes most strongly to performance on data that requires generalization, ablate this subnetwork, and find that it also hurts memorization after mixed training.

Critical strengths
  • The empirical investigation has significant breadth of settings and models. The authors test a wide variety of tasks and models, and present their results in a coherent and thorough manner. The authors perform reasonable ablations and present all results clearly to the reader. I appreciate the authors' transparency and care taken toward presentation.
Critical weaknesses
  • Data order is confounded with exposure. GenMix, PureMix, and MemMix see different numbers of examples of GEN examples throughout training. The main experiment (Figure 3) does not isolate the effect of data ordering from total GEN exposure. The controlled continuations do not resolve this confounder. The controlled continuations do establish that GEN remains learnable after MEM-first training, but do not resolve this confound.
  • The MEM data is more aptly described as random label noise. The MEM prompts and GEN prompts are similar (~from the same distribution), but MEM prompts have the answer replaced with something incorrect. Therefore, the MEM examples actually contradict the general rule the model might learn from GEN examples. This does not reflect typical LLM memorization, where the model must memorize facts with no pattern but which aren't contradictory or necessarily inconsistent.
Questions for authors
  1. Does extending mixed training allow MemMix to close the gap with GenMix? Could you plot the training trajectories against the number of cumulative GEN examples?
  2. Given that MEM examples use deliberately incorrect labels, how do your findings extend or differ from prior work on learning with label noise and random labels, particularly Arpit et al.’s "A Closer Look at Memorization in Deep Networks?"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.