- Rating
- 3
- Confidence
- 2(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how initial training on the abstract stack-manipulation task STACK, before standard language pretraining, affects the capabilities that models subsequently acquire. It also examines how the properties of the abstract data and the timing of exposure influence transfer.
Critical strengths
1.The paper conducts an extensive set of targeted control experiments to investigate the conditions for positive transfer. The QUEUE and STACKRAND variants examine the role of the abstract task’s structure, while weight interventions test whether weight magnitudes alone are sufficient to explain the benefits. The experiments take the study beyond a simple performance comparison, providing concrete evidence and practical guidance on which types of abstract data are beneficial and when they should be introduced.
2.The paper also provides empirical evidence that abstract pretraining can improve specific capabilities without necessarily improving language-modeling loss. These findings suggest that relying solely on perplexity may overlook the effects of abstract pretraining on specific capabilities, providing a useful basis for evaluating and selecting abstract data.
Critical weaknesses
The scope of some conclusions about training timing needs further clarification. The abstract and Discussion state that mixing abstract data with natural language is less effective than an initial dedicated phase. However, Section 6.1 describes inserting a separate STACK training phase at different points during language pretraining. This is not equivalent to continuously mixing abstract and natural-language data. The authors should clarify what “mixing” refers to and align the scope of their conclusions with the experimental settings.
Questions for authors
See weakness.
2 comments
My concern is addressed by the authors. I will maintain my positive score and raise my confidence rating.
W: Scope of the timing conclusions
We agree that 'mixing' has two possible meanings.
Mixing phases (current experiments). In Section 6.1 we insert the STACK phase at different points during language pretraining. This evaluates the timing of the language/STACK data without mixing their respective training signals.
Within-batch mixing (new experiments). We added experiments for the suggested setting (also suggested by Reviewer 3Zao) where language and STACK data are mixed within batches. Using the same (and same amount of) STACK data spread overall pretraining batches, the mean gain in F1 is only +0.14 (within the evaluation noise), compared with +3.22 for a dedicated initial phase. See the full results in response to Reviewer 3Zao, Q3.
We are updating Section 6 with the new experiments and with a clearer distinction between these two 'mixing' settings.