acceptodds
Under review as a conference paper at ICLR 2027

Lasting Effects of Abstract Pretraining Beyond Perplexity

Abstract

Language models are typically pretrained from randomly initialized weights. Recent work challenges this convention, showing that a brief warm-up on abstract, algorithmically generated data can provide a useful starting point for subsequent learning from natural language. In this paper, we show that, in small language models, such a warm-up improves the acquisition of downstream capabilities in ways that are not reflected in language-modeling perplexity. Our warm-up uses an abstract stack-manipulation task in which models process sequences of push and pop operations and predict the remaining contents. Although it uses only approximately 1% as many tokens as the language-pretraining phase that follows, it improves multi-hop question answering after fine-tuning by up to 3.9 F1 points on MuSiQue, with additional gains on HotpotQA and 2WikiMultihopQA, despite comparable language-modeling perplexity. Controlled experiments further show that the warm-up substantially accelerates the acquisition of deeper reasoning chains. We then explore what drives this transfer. First, it depends on the computation learned: replacing the stack's last-in-first-out rule with a matched first-in-first-out rule eliminates the downstream advantage. Second, transfer is selective: gains concentrate on dependent reasoning chains, with no consistent benefit for tasks that merge or compare independent lookups. Third, timing matters: exposure before language pretraining produces the largest gains, during pretraining yields smaller gains, and afterwards has no improvement. Moreover, the early advantage persists through billions of subsequent language tokens. Together, these results show that early training on abstract algorithmic data can shape the capabilities that language models later acquire, producing benefits that language-modeling perplexity does not capture.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Procedural pretraining, Multi-hop reasoning, Training order, Inductive biases, Language models, Initialization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Q9xvOpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how initial training on the abstract stack-manipulation task STACK, before standard language pretraining, affects the capabilities that models subsequently acquire. It also examines how the properties of the abstract data and the timing of exposure influence transfer.

Critical strengths

1.The paper conducts an extensive set of targeted control experiments to investigate the conditions for positive transfer. The QUEUE and STACKRAND variants examine the role of the abstract task’s structure, while weight interventions test whether weight magnitudes alone are sufficient to explain the benefits. The experiments take the study beyond a simple performance comparison, providing concrete evidence and practical guidance on which types of abstract data are beneficial and when they should be introduced.

2.The paper also provides empirical evidence that abstract pretraining can improve specific capabilities without necessarily improving language-modeling loss. These findings suggest that relying solely on perplexity may overlook the effects of abstract pretraining on specific capabilities, providing a useful basis for evaluating and selecting abstract data.

Critical weaknesses

The scope of some conclusions about training timing needs further clarification. The abstract and Discussion state that mixing abstract data with natural language is less effective than an initial dedicated phase. However, Section 6.1 describes inserting a separate STACK training phase at different points during language pretraining. This is not equivalent to continuously mixing abstract and natural-language data. The authors should clarify what “mixing” refers to and align the scope of their conclusions with the experimental settings.

Questions for authors

See weakness.

2 comments

W: Scope of the timing conclusions

We agree that 'mixing' has two possible meanings.

  1. Mixing phases (current experiments). In Section 6.1 we insert the STACK phase at different points during language pretraining. This evaluates the timing of the language/STACK data without mixing their respective training signals.

  2. Within-batch mixing (new experiments). We added experiments for the suggested setting (also suggested by Reviewer 3Zao) where language and STACK data are mixed within batches. Using the same (and same amount of) STACK data spread overall pretraining batches, the mean gain in F1 is only +0.14 (within the evaluation noise), compared with +3.22 for a dedicated initial phase. See the full results in response to Reviewer 3Zao, Q3.

We are updating Section 6 with the new experiments and with a clearer distinction between these two 'mixing' settings.

My concern is addressed by the authors. I will maintain my positive score and raise my confidence rating.

Reviewer_3ZaoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how brief pretraining on abstract, algorithmically generated data affects the capabilities that language models subsequently acquire. The main approach trains models on STACK, followed by natural-language pretraining and task-specific supervised fine-tuning. Experiments with SmolLM-135M and SmolLM-360M show improvements on multi-hop question answering. Controlled experiments further examines different abstract-task structure and training order, finding that initial STACK produces the strongest benefits.

Critical strengths

The paper demonstrates that STACK training before natural-language pretraining can improve downstream performance on multi-hop reasoning tasks, even when language-modeling perplexity does not improve.

Critical weaknesses

The paper does not sufficiently explain how different abstract tasks influence the model’s subsequent learning. In particular, the mechanisms underlying the performance differences between STACK, STACKRAND, and QUEUE remain unclear. This limits the support for the paper’s interpretation of which properties of abstract data drive downstream improvements.

Questions for authors
  1. What useful learning signal does STACKRAND provide? Since the final stack contents are randomly permuted, their output order is unpredictable. Further analysis would help clarify what the model learns from this task.
  2. Why do both STACK and STACKRAND outperform QUEUE? Both stack and queue manipulation require tracking and updating an internal state. What properties of stack manipulation make it more beneficial for downstream multi-hop reasoning, and why does this advantage persist when the output order is randomized?
  3. Could you evaluate mixing STACK and natural-language data within every pretraining batch? How would this approach compare with a dedicated initial STACK training phase in terms of downstream performance?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer 3Zao

Thank you for the thoughtful questions. We performed additional experiments that help explain the source of improvements from different forms of abstract data.


W1/Q2: Why STACK/STACKRAND outperform QUEUE

All tasks require state tracking, but only STACK/RANDSTACK (LIFO) require resolving dependencies between operations. Under FIFO (QUEUE), a 'pop' always removes the oldest element, so the final contents are simply the last nn pushed symbols where nn=(number of 'push') - (number of 'pop'). Thus QUEUE can be solved by counting and copying a suffix of the input. Under LIFO however, a pop removes the most recent unpopped push, and each update depends on the state left by all preceding ones. The need to track chains of updates mirrors the sequential compositions in the QAs where we see the most improvements (Section 4.2).

The improvements also persist when the output is shuffled, because shuffling changes only the readout: STACKRAND must still resolve the LIFO pairing (and it does learn this, see Q1).

We ran two new control experiments on DEPO that support this explanation (see details in our response to Reviewer U4ti, W1/Q1):

  • QUEUERAND (FIFO updates, shuffled readout) reaches a mean accuracy of 52.2% vs. 78.0% for STACKRAND. So LIFO still helps when the readout is matched.
  • DYCK (which requires the same LIFO matching) reaches 82.3% while its non-nested control DYCK-SHUFFLE (which can be tracked with counters) reaches 32.4%.

Conclusion: What makes STACK beneficial is the LIFO matching and not state tracking in general. QUEUE also tracks a state, but it is simple enough to solve by counting. We will add this explanation and the new control experiments to Section 5.1 and clarify/restructure the text around the matched pairs of tasks.

Q1: What does STACKRAND learn?

STACKRAND teaches the same LIFO computation as STACK, and the shuffling only removes the ordered readout.

The set of output symbols is predictable, even if their exact order is not. So the optimal prediction at each output position is a uniform distribution over the "surviving" symbols not yet emitted (and EOS once all have been emitted). The task therefore requires the model to track a state just like STACK does.

We verified that models trained on STACKRAND learn this correctly by evaluating the models from our DEPO experiments on 6k held-out sequences. The top prediction is a valid continuation 92.2% of the time.

Shuffling removes only the ordered readout. An ordered readout requires following a precised ordered chain. Consistent with this, STACKRAND retains most of STACK's benefits on HotpotQA and 2WikiMH, whose questions are mostly two-hop, but is worse on MuSiQue which requires up to four hops. Similarly on DEPO, the accuracy of STACKRAND models falls from 85.7% at one hop to 72.9% at four, whereas STACK models are near-perfect at every hop count (Fig. 7).

Conclusion: STACKRAND teaches the same LIFO-matching as STACK, but not the ordered readout. This explains why it retains much of STACK's benefit but falls short on longer chains. We will add this explanation to Section 5.1.

Q3: Mixing STACK into every pretraining batch

Mixing gives no QA benefit, even though the model learns STACK well. The benefits require a dedicated STACK phase.

We performed the suggested experiments. The 'mixed' model receives the same number of STACK examples as the initial phase (320k examples, 80M tokens), spread over all pretraining batches (23 per batch). Everything else is unchanged (same loss masking, optimizer, and language budget; SmolLM-135M; average over 3 fine-tuning seeds).

MuSiQueHotpotQA2WikiMHMean F1Mean ΔFineWeb-Edu val. loss
Language only37.5341.3550.1743.02–2.9144
Initial STACK42.3844.6051.7346.24+3.222.9122
Mixed STACK37.3741.6550.4543.16+0.142.9141

The mixing gives almost no benefit with +0.14 mean F1 (within the evaluation noise) compared to +3.22 for a dedicated initial phase. The language-modeling loss is nearly identical for all three models. We verified that the model does learn the STACK task correctly: it achieves 99.5% token accuracy on stack sequences (even though the STACK tokens only make up ~0.26% of supervised tokens per step).

Conclusion: Learning the STACK task is not sufficient, and the benefits require a dedicated initial phase. We propose to add this additional evidence to the experiments of Section 6 that already support this major finding of the paper.

Thanks for your comment. I will keep my positive score.

Reviewer_U4tiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper studies how brief pretraining on a synthetic stack task affects later learning. SmolLM models show improved multi-hop question answering after language pretraining and task-specific fine-tuning, despite similar or worse language-modeling loss. Gains concentrate on sequential reasoning chains. Controlled experiments also show faster learning on a synthetic multi-hop task. Queue and shuffled-output variants test the role of task structure. Training-order experiments favor initial exposure, and the QA advantage persists across subsequent language-pretraining checkpoints. The study covers 135M and 360M language models and smaller controlled models.

Critical strengths

The separation between language loss and downstream learning is the clearest result. On SmolLM-135M, STACK raises MuSiQue F1 from 38.10 to 41.99 despite similar language-modeling loss. The question-type breakdown and DEPO learning curves provide useful evidence beyond an aggregate QA gain. The tuned language-only baseline and multiple pretraining and fine-tuning seeds at 135M also make the comparison more credible.

Critical weaknesses
  • The STACK–QUEUE comparison shows that the training target matters, but leaves the source of transfer unclear.
  • The timing experiments establish an ordering effect, but do not isolate why early exposure works better. Later insertion leaves less subsequent language training to consolidate useful structure or recover from interference. The learning-rate sweep in Table 6 is helpful, but does not resolve this distinction.
Questions for authors
  • How well are STACK and QUEUE solved before transfer? Since STACKRAND retains much of the gain, does LIFO still help when FIFO uses the same shuffled readout?

  • Can you vary the insertion point and the amount of subsequent language training separately, with controls matched for total language tokens?

  • Does the QA gap persist with longer, appropriately tuned fine-tuning, or does the language-only baseline catch up?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer U4ti

Thank you for the careful review and precise questions. We ran new experiments addressing each of them.


W1/Q1: Source of transfer and STACK/QUEUE comparison

Both tasks are learned before transfer, and LIFO still helps when the readout is matched.

How well are the tasks solved? Before language pretraining, held-out token-level accuracy is 97.7% for STACK and 99.5% for QUEUE (SmolLM-135M, mean of 3 seeds). QUEUE is solved accurately, so its lack of transfer is not due to under-training.

What differs between them? Under FIFO, pops always remove the oldest element, so the final contents are always the last (#push − #pop) symbols pushed: QUEUE reduces to counting and copying a suffix of the input. Under LIFO, each pop removes the most recent unpopped push, so which symbols survive depends on how each pop pairs with an earlier push, and each update depends on the state left by all preceding ones. This chain of dependent updates mirrors the sequential compositions where our QA gains concentrate (Section 4.2).

Does LIFO help with a matched readout? Yes. We ran QUEUERAND (FIFO updates, shuffled readout), the direct counterpart of STACKRAND. We also include a second matched pair from our appendix: DYCK, which requires the same LIFO matching (each closing bracket must match the most recent unclosed one), and DYCK-SHUFFLE, a non-nested control that can be tracked with counters. Mean final DEPO accuracy (%) over 1–4 hops after direct transfer:

Matched pairTask with LIFOControl without LIFOGap
STACKRAND / QUEUERAND (same shuffled readout)78.052.2+25.8
DYCK / DYCK-SHUFFLE (nested / non-nested)82.332.4+49.9

In both pairs, the task that requires LIFO performs far better; for STACKRAND vs. QUEUERAND, the gap is 23–28 points at every hop count. The same holds with ordered readout (STACK vs. QUEUE, Fig. 7).

Conclusion: The source of transfer is the LIFO rule, i.e. resolving which 'push' is removed by each 'pop', rather than readout order or surface statistics (which STACK and QUEUE share). The ordered readout adds a further benefit on top of LIFO (STACK vs. STACKRAND), which we analyse in our response to Reviewer 3Zao (Q1). We will add QUEUERAND, the DYCK comparison and this explanation to Section 5.1.

W2/Q2: Separating insertion point from subsequent language training

We ran this experiment, and it supports the reviewer's explanation: with language training after STACK held fixed, the insertion point has little effect. But our claims remain valid: under a fixed budget, the practical recommendation is unchanged since placing STACK first maximizes the training that follows.

We fix language training after STACK at 1.35B tokens and vary only the insertion point. Each model is compared with a language-only baseline trained on the same total number of language tokens, with its own learning-rate schedule. Other settings follow Section 6.1 (SmolLM-135M, three fine-tuning seeds).

RunLanguage before STACKLanguage after STACKTotal languageMean ΔF1
A01.35B1.35B+1.28
B0.67B1.35B2.02B+1.45
C1.35B1.35B2.69B+1.33

'Mean ΔF1' is the F1 of the STACK model minus F1 of its token-matched baseline, averaged over MuSiQue, HotpotQA and 2WikiMH.

STACK improves QA in every setting, but the gain is similar across insertion points. Fig. 9 compared insertion points at a fixed total budget, so STACK first was followed by 2.69B language tokens and STACK at 50% by only 1.35B (+2.4 vs. +1.2 F1; run C repeats the 50% setting and gives a similar +1.33). Since the insertion point by itself has little effect, this difference is largely explained by the additional language training after STACK. The same account explains Table 6: STACK placed after all language training has no language training after it, and none of the learning rates tested yields a gain. A related experiment for Reviewer 3Zao (Q3) shows that spreading the same STACK data across every batch gives no gain, so a dedicated phase is also needed.

Conclusion: The benefit comes from a dedicated STACK phase followed by enough language training, not from timing per se. Under a fixed budget, the practical recommendation is unchanged, since placing STACK first maximizes the training that follows. We propose to update the abstract, Section 6 and the discussion to make this clear, and add the new experiment to Section 6.1.

Q3: Tuned/longer fine-tuning

The benefits are strongest in low-data settings but do not vanish with multiple epochs of fine-tuning.

We ran additional experiments on different hardware (averaged over 3 pretraining × 3 fine-tuning seeds), so the 1-epoch numbers differ slightly from the paper.

Learning rate (1 epoch). The paper's learning rate (3e-4) gives the best baseline F1 on all three benchmarks among {1e-4, 3e-4, 1e-3}, so the baseline is not under-tuned. STACK remains ahead at every learning rate and benchmark (mean Δ +2.1 to +2.8).

Longer fine-tuning (LR 3e-4). Each cell contains the F1 for Random / STACK (and Δ). 'Mean Δ' is the average over the three benchmarks.

MuSiQueHotpotQA2WikiMHMean Δ
1 epoch38.3 / 41.9 (+3.6)41.5 / 43.6 (+2.1)50.3 / 51.2 (+0.9)+2.2
2 epochs39.8 / 42.2 (+2.4)45.3 / 47.2 (+1.9)52.7 / 53.4 (+0.7)+1.7
3 epochs40.0 / 41.7 (+1.7)46.6 / 48.2 (+1.6)54.1 / 54.1 (0.0)+1.1

The mean Δ narrows from +2.2 to +1.1 F1 with more epochs but stays positive. Comparing each model at its best setting, STACK still leads by +2.2 F1 on MuSiQue and +1.6 on HotpotQA, and on MuSiQue, STACK after one epoch (41.9) exceeds the baseline after three (40.0). 2WikiMH shows the smallest gain throughout, which is explained by the fact that nearly half of its questions are comparison/bridge-comparison types, where STACK gives no benefit as analyzed in the paper (Fig. 3).

Less fine-tuning data (1 epoch). STACK leads in all settings from 1k to 20k training examples.

Training examplesMuSiQueHotpotQA2WikiMH
1k+2.6+3.5+2.3
2k+3.3+3.1+4.7
5k+3.4+3.3+2.9
10k+2.9+2.8+1.6
20k+3.6+2.1+0.9

Conclusion: STACK makes multi-hop fine-tuning more sample-efficient, and longer/tuned fine-tuning narrows the gap but does not close it. We propose to include these additional results and observations to the paper.

Reviewer_U4tiOpenReview ↗

reply

Thank you for your rebuttal. I will keep my original score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision Upload

We thank the reviewers for their valuable feedback, which has helped us improve the manuscript. We uploaded a revision with new results and clarifications (in blue in the PDF):

  • (Pages 7, 14, 15, 20) New results showing that LIFO matching is the source of transfer: a FIFO control with the same shuffled readout as StackRand, and the nested/non-nested Dyck pair.
  • (Pages 8, 20) New results separating the insertion point of the abstract phase from the amount of subsequent language training.
  • (Pages 8, 14, 15, 20) New results with abstract data mixed into every pretraining batch, showing that a dedicated phase is required.
  • (Pages 4, 17, 21) New results showing that the gains persist with tuned and longer fine-tuning, and are largest with less fine-tuning data.
  • (Pages 1, 2, 3, 9, 10) Clarified conclusions about scheduling (a dedicated phase, no mixing, placed first) and the distinction between inserted phases and within-batch mixing.
  • (Page 4) Clarification of what StackRand learns.

We remain available in case of any further requests.

Dear Reviewers,

Please read the authors' response. Please evaluate whether your concerns are addressed or if anything needs further attention. Please engage with the authors while we have time.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.