acceptodds
Withdrawn from ICLR 2027

Brief Treatments Select Long-Run Function in Post-LN Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Can a 25-update treatment change a model's function after thousands of unmodified updates? In an unstable Post-LN decoder trained without warmup, we modify only updates 4–28 of paired runs sharing an exact parent and then resume native AdamW through update 5000. On six complete master seeds, a scaled ascending learning-rate ramp is healthy on 6/6, while an unscaled ascending ramp and a norm-preserving directional intervention are each healthy on 5/6 and fail on different seeds; untreated ACTUAL, an exact-dose constant profile, and a release-continuous V0 profile are healthy on 0/6. Exact-dose reverse and half-LR controls also fail on the original seeds. Together with a successful release-shock control, these comparisons show that the tested outcomes separate strongly by within-window temporal organization beyond cumulative dose, release continuity, and sustained release-step magnitude. Transfer is recipe-dependent: at eight layers, 25-, 50-, and 100-update ramps are healthy on 0/3, 2/3, and 3/3 seeds; at 107M parameters, tested 25-, 100-, and 200-update ramps are 0/3, while Warmup500 remains more robust. At two lower-LR release states, parameter state and complete AdamW state each suffice for healthy continuation, whereas isolated moments do not. Release diagnostics show distinct organizations for healthy directional and scalar routes. These are tested-point results, not a universal warmup law or an early-commitment period.

Primary area:
optimization
Keywords:
training dynamics, optimization, Transformers, Post-LN, learning-rate warmup, path dependence, persistent learning states, AdamW

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_13zMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how brief early-training interventions affect long-term outcomes in unstable Post-LN Transformers. The authors show that modifying only 25 updates can prevent functional collapse after 5,000 steps and further examine learning-rate schedules, transfer across model configurations, and optimizer-state effects.

Critical strengths
  • The authors explore several useful controls and report negative results and limitations quite transparently.
  • I appreciate the controlled experimental design, particularly the paired runs that share the same initial state and differ only during a short intervention window.
Critical weaknesses
  1. I found the paper difficult to follow despite careful reading. The many internal experiment identifiers (R5, R6, R8, R9, etc.) make it hard to track the results. More importantly, I struggled to see how the three contributions connect. For example Sec. 6 reports very different effective ranks for healthy RAMP and NEUTRAL runs, but it remains unclear what these diagnostics tell us about why either intervention works.

  2. The novelty and broader significance are not entirely clear to me. [1] already studied how large early updates and parameter rotations affect warmup in GPT training, while [2] investigated how warmup leads to more stable optimization. The successful ramps here seem consistent with these findings. Given that the short interventions fail in several larger configurations while Warmup500 remains more reliable, I am not sure what additional insight this work establishes.

  3. Some key mechanisms remain unresolved. In Sec. 4, the equal-dose UP, CONSTANT, and REVERSE schedules have very different initial learning rates, so their outcomes may simply reflect the importance of suppressing the earliest updates. In Sec. 5.2, the failed m-only transplant has a much larger update proposal norm than the healthy full-state transplant (14.20 vs. 0.59). As the authors acknowledged that this makes it difficult to distinguish insufficient moment information from instability caused by mismatched optimizer states.

[1] Kosson et al. "Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training" NeurIPS 2024
[2] Kalra & Barkeshli. "Why Warmup the Learning Rate? Underlying Mechanisms and Improvements" NeurIPS 2024
Questions for authors

Q1. Could the authors compare schedules with similar initial update magnitudes but different later ordering, to clarify whether temporal organization matters beyond early-step suppression?

Q2. Would the m-only transplant still fail if its initial continuation update were norm-matched or clipped?

Q3. What do the authors see as the main new insight beyond existing warmup research (e.g. two papers indicated in weaknesses), particularly given the limited transfer and largely descriptive release-state analysis?

Reviewer_8xFWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether a 25-step treatment early in training (updates 4 to 28) prevents degeneration in an unstable Post-LN Transformer trained with AdamW. The authors evaluate 24M, 30M, and 107M parameter networks (6–8 layers) trained on WikiText for 5000 steps, consuming roughly 20M tokens overall (batch size 32, sequence length 129). They compare different treatments for steps 4 to 28, such as short warmup, using half the learning rate, intermittent annealing, and or directional updates to the baseline using normal 500 step warmup. While several 25-step treatments reduce the rate of collapse across seeds in the smallest setup, they fail to transfer reliably to the 107M scale, where standard warmup remains the most stable option.

Critical strengths
  • The authors explore several treatments during training in a structured experimental setup.
  • The authors run their experiments across multiple random seeds and honestly report where their treatments fail.
Critical weaknesses
  • The motivation for studying this specific setup is unclear, particularly why the authors selected an unstable Post-LN architecture, which is not commonly used in practice. There is no motivation why exactly the 25-step window from update 4 to 28 is selected as well. Additionally, the evaluation only checks whether models pass a binary non-degeneration threshold rather than comparing the overall performance or at least test perplexity.
  • The work provides limited novel insights for practical pre-training. In practice, much larger models and batch sizes are used. Prior work has established that batch size strongly impacts early optimization dynamics (e.g., via critical batch size and early gradient correlation), and modern standard practices already account for warmup and annealing dynamics in model training. Furthermore, training for 10 epochs on a small WikiText corpus is an unrealistic token distribution for modern pre-training, and the 25-step treatments already fail on all 3/3 seeds at the 107M scale.
  • The experimental protocol is unnecessarily complicated: it is not clear why the authors split runs into "original" seeds and "confirmation" seeds rather than treating all seeds equally.
  • The write-up is not self-contained. Key methods, metrics (such as NLL, D_16, C, and NEUTRAL), and experimental “cells” (e.g., R8) are not well defined. Furthermore, the plots consume nearly two pages to display a handful of data points without standard deviations or training curves, and the text is cluttered with numbers reported to 9 and more decimal places, which exceed float32 precision and makes the paper noisy to read.
Questions for authors
  • What motivates the exact 25-step window (updates 4 to 28) and the use of a Post-LN architecture over standard Pre-LN models?
Reviewer_n4ruOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper tests whether brief changes early in training can prevent collapse in Post-LN language models. The authors change the update direction or learning rate during updates 4–28, then resume standard AdamW training through update 5000. Both methods can prevent collapse in a small model on WikiText-2, but success depends on model size, learning rate, and treatment duration. Controls show that the sum of learning rates alone does not explain the results. At two tested checkpoints, transferring either treated parameters or both AdamW moments enables successful training; transferring either moment alone does not. Standard 500-update warmup remains more reliable across the tested settings.

Critical strengths

None

Critical weaknesses

The paper’s main message is difficult to identify. Numerous experiment labels, repeated caveats, and execution details obscure the scientific argument. Key evidence is scattered across the main text and supplement. The reader must reconstruct which comparisons support each claim and what they add to existing warmup results.

The contribution is also narrow: all experiments use one corpus, and the tested brief treatments fail at 107M parameters while standard warmup succeeds. The controls do not separate smaller initial steps from ramp shape or duration. Likewise, first-moment-only transfer may fail because of incompatible moment scales. These gaps leave the main results without a clear explanation.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.