- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies linear weight interpolation between pairs of independently trained weight-tied (looped) 35M-parameter language models It measures the midpoint barrier before and after permutation alignment that respects weight tying, at the trained depth
Critical strengths
-
The joint alignment / endpoint-quality / runtime-depth protocol exposes a real pitfall. The paper shows that a relative barrier can "vanish" at deeper runtime simply because the reference endpoints collapse. This is a concrete, easily overlooked artifact for anyone merging looped / weight-tied models, and the recommendation to always report absolute endpoint quality and depth alongside the barrier is well supported and actionable.
-
The measurements are unusually carefully controlled and honestly reported. Four matcher families, norm-matched random twins, direction-vs-orthogonal subspace perturbations at identical dose, a dense non-tied control, a second-corpus replication, pair-cluster bootstrap intervals, and negative transfer results on public Pythia pairs are all included.
Critical weaknesses
-
Narrow scope and weak statistical footing for the headline claim. All controlled results come from one 35M weight-tied family trained for 5k steps / 0.25B tokens (endpoint PPL 55-80, i.e., far from converged), and six of the fifteen pairs reuse four checkpoints, so there are far fewer than fifteen independent units.
-
After the best of four alignments the midpoint is still 12-24x worse than the endpoints, which is roughly what a random perturbation of the same norm produces (twins reach ~1.3x the aligned log-barrier), so alignment does not yield usable merges here and the "residual" is not given a mechanistic interpretation. The repairs are evaluated on only one or two pairs each; plain continuation beats distillation at trained depth and at matched compute, so the distillation result reduces to a deep-runtime-stability advantage on a single pair family.
Questions for authors
-
Why does permutation alignment remove 1-4 dex on the 35M looped pairs but <=5% on Pythia? Is this attributable to weight tying, to the short training budget (0.25B tokens), or to the Pythia confounds (data/dedup, unpublished seeds)? The dense 18-layer control (naive 1.83 dex, residual 1.29 dex) seems to show a much smaller shell than the looped pairs - does the shell shrink with training budget or with depth of the dense model? A dense pair with truly independent seeds at the same scale would resolve this.
-
Given the [0.07, 2.77] dex interval on the naive recipe contrast, do the authors have additional independent seed pairs per recipe (rather than pairs reusing checkpoints) that would tighten this? Without it, is the claim better phrased as "after alignment, both recipes sit on the same residual" rather than "alignment explains the recipe contrast"?
-
The norm-matched twin reaches ~1.3x the aligned log-barrier, i.e., the aligned midpoint is somewhat better than a random same-norm perturbation but still unusable. Is there any evidence that the achieved alignment finds structure (e.g., better than the twin on the depth-stable pairs after continued training, or lower loss on specific token classes), or is the practical conclusion that alignment is not useful for merging these models?
-
Please state which perplexity estimator each headline number uses in one place (e.g., a single column in Table 1 on one basis), or justify why two estimators are needed at all.