- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates how training on structured or unstructured data - encouraging model generalization and memorization respectively - affects subsequent learning from a mixture of both data. The authors find that memorization hinders subsequent generalization from the full data mixture across a wide variety of models and tasks, and suggest that this is a curriculum effect. A mechanistic dependence between the circuits underlying memorization and generalization is suggested through ablation experiments.
Overall, the paper purports to characterize a phenomenon for which a mechanistic account could contribute to the growing understanding of the interaction between memorization/generalization. However, the paper should be rejected because 1) significant details of the experiment design are left out, hindering any assessment of the results and 2) questions of the methods as presented cast doubt on the presented results.
Critical strengths
None
Critical weaknesses
The most significant omissions relate to the following:
- Basic details of the training setting, including the objective function and the sampling scheme for mixed GEN/MEM training (151), are unspecified. Experiment results are later presented without specification of relevant parameter settings, such as loss mixture weight (Figure 5).
- The main mechanistic analysis involves identifying subsets of neurons which are causally implicated in generalization performance. The method to identify these neuron subsets is never explained throughout the main text or appendices; instead, the method is described only in reference (161). As a result, the implementation as detailed in the appendices refers to parameters (1235), “hard-concrete” gates (1217), and “analyzed coordinates” (1281) that are never defined or specified.
- The performance of models trained under different curricula is compared in terms of “wins” and “ties” (233), without any description of the applied criteria. It is important that this criteria, whatever it may be, involve an assessment of significance of the observed performance gap. The corresponding data (Figure 3) do not include error bars.
- Furthermore, the main experiment investigating the interactions between memorization and generalization involve the ablation of the generalization-implicated neurons, but the ablation method is never specified across the main text or appendices (194, 355).
Major questions of method and interpretation include:
- The authors claim that the suppression of generalization after memorization is a result of the initial memorization curriculum. However, the effect size as presented in the main text appears small: only 4 more models showed impaired generalization performance when memorization is trained first rather than interleaved throughout (Table 1).
- The method to derive the GEN-implicated neuron family selects the subset which maximally degrades GEN accuracy. As a result, a family is always identified even if none meaningfully degrade accuracy. It would be more principled to choose a threshold degradation, and consider all families which meet this threshold as GEN-associated.
- Results are provided from ablation of the GEN-family for the MemMix model (Fig. 4). However, the method for identifying the GEN-family states that this is done at the final GEN-only checkpoint (354), which should not exist for the MemMix model. If ablations are done in the MemMix model using a neuron family identified in the separately trained GenMix, this presents a major methodological error.
- The experiments ablating the GEN-associated family (357) are difficult to interpret without comparison to a null experiment, perhaps one which ablates a random family.
Minor comments:
-
It would be helpful to justify the decision to train the PureMix setting for 2K iterations (153). I could see an argument for matching the Mix training iterations across the 3 curricula.
-
The MEM-target construction uses a very particular (200 - 1998) range to sample the target from (Table A.2). How was this range chosen? Why not shuffle the targets instead?
-
What is the motivation for the choice of phase length K (Table B.2) for each model?
-
A_GEN and A_MEM are referenced without definition (141).
-
The main text refers to a series of analyses without providing any description of their design, including the measured quantities (204).
-
The second question (249) is posed as unanswered, yet it seems to have been already resolved at this point by the PureMix and GenMix training results.
-
The results for the loss-weighting experiment are discussed without any reference to the data provided in Table 2 (268).
-
Why are microbatches reported for each model (Table B.3)? This seems like an implementation detail that has no impact on the scientific results.
Questions for authors
Introduction
- It may be simpler to state that memorization has unstructured prompt-target associations (039)
- Could be clearer to call the three settings Gen-Mix, Mem-Mix, and Mix
- “Higher” (055) with respect to what performance metric? The percentage comparison feels unnecessary here
Related work
- Run on sentence (108)
- Doesn’t continual learning involve studying interleaved curricula? (119)
Experimental setup
- What are A_GEN (138) and A_MEM (141)?
- Are GEN and MEM sampled equally, or randomly? (151)
- What is the loss function?
- Need to include at minimum a conceptual summary of the referenced neuron family identification technique (150)
- How is ablation of the neuron families performed? (194)
- The method to derive the “GEN-associated family” selects the neuron subset which maximally degrades GEN accuracy. As a result, a family could be identified as GEN-associated even if it only marginally degrades accuracy. It would be more principled to choose a threshold significant degradation, and consider all families which meet this threshold as GEN-associated.
- “Phase-1” and “Phase-2” are never defined (206)
- The three follow-up analyses (204) need, at minimum, conceptual summaries in the main text of the method and recorded quantities
Figure 2
- “Subnetwork” in panel 2 is misspelled
- What is the meaning of the half-filled gates in panel 2?
Table 1
- This table seems unnecessary, since it contains only 2 relevant values
Results
- What is the “win” and “tie” criteria (230)? It is important that these measures be designed to detect only significant differences in final accuracy
- PureMix results are repeatedly qualitatively summarized but never provided in the main text (239)
- It would be helpful to move the 6 accuracy values and confidence intervals to a table, rather than listing in the main text (242). In addition, data needs to be shown to evidence the purported good memorization performance (246).
- The second question has already been answered by the PureMix and GenMix training results (249)
- It would be helpful to justify the decision to train the PureMix setting for 2K iterations. I could see an argument for matching the Mix training iterations across the 3 curricula.
- The results for the loss-weighting experiment are discussed without any reference to the data provided in Table 2 (268)
- What is the “fixed confirmation set” (355)?
- The experiments ablating the GEN-associated family could benefit by a comparison to a null experiment, perhaps one which ablates a random family (357)
- It would be helpful to justify why the authors chose to constrain activations rather than weights (368)
- It seems odd that the anchor loss is provided (Eq. 2) when the general loss function used is never specified
- The authors never provide an interpretation of the activation anchoring results (430)
Discussion
- The analyses are done on a variety of model and task settings, but any effect of these factors on the reported phenomenon are never shown or discussed
Appendices
- Why is A_GEN never detailed, while A_MEM is (Eq. A.1)? One could write the accuracy more generally and cover both simultaneously.
- The MEM-target construction seems to use a very particular (200 - 1998) range to sample the target from. How was this range computed? Why not shuffle the targets instead?
- The discussion of the MEM evaluation seems unnecessary and overly verbose (781)
- What is the motivation for the choice of phase length K (Table B.2) for each model?
- Why are microbatches reported for each model (Table B.3)? If typical gradient accumulation is used, this is an implementation detail that has no impact on the scientific results
- Eq. B.3 is redundant since the memorization accuracy A_MEM was previously defined
- Why are the differences in accuracy not reported in a table? (1126)
- The explanation of the family construction provides settings for parameter values (, , ) that are never defined because the “low-complexity subnetwork-probing framework” is never described
- The fixed reference scale (Eq. E.2) computed as part of the activation-anchor intervention is a reasonable part of the method design, but never motivated
- How is the “calibration set” formed? (1376)
- Table E.4 references units of “pp” which are never defined.
Figure 3
- The overlapping labels near the origin are unreadable
- It would be helpful to provide error bars for these results to help assess significance
- Comparing the final GEN accuracies across task, model size, training method, etc may reveal interesting dependencies
Figure 4
- The figure claims to ablate the GEN-family for the MemMix model. However, the method for identifying the GEN-family states that this is done at the final GEN-only checkpoint, which does not exist for the MemMix model. This discrepancy needs clarifying.
Figure 5
- How is drift measured?
- What are the values used to generate these data?
Table 2
- It could be more informative to plot the two accuracies as functions of for the three model/task pairs