acceptodds
Under review as a conference paper at ICLR 2027

Permutation Alignment and Depth in Weight-Tied Language Models

Abstract

Averaging the weights of two independently trained language models can yield a far worse model, even after their hidden units are permuted into correspondence. We ask how much of this damage permutation alignment removes, and how to interpret the remaining damage in weight-tied (looped) models, whose depth can vary at runtime. We measure fifteen pairs of 35M-parameter looped models at 16 and 192 loops, before and after a permutation alignment that respects the shared weights, recording the quality of the averaged (midpoint) model and both endpoints. Before alignment, the median barrier (the ratio of midpoint to better-endpoint perplexity) is 1.8 dex (base-10 log units) higher for eight pairs trained mostly with random loop counts than for six trained at a fixed 16 loops; after alignment the gap is under 0.05 dex. A shared residual remains: with the best of four alignment methods, midpoint perplexity is at least 12 times the better endpoint's on every pair. Equal-norm random perturbations reproduce its typical size; perturbations along the aligned weight difference's leading directions cause more damage than orthogonal ones on every pair tested. At 192 loops, endpoint quality sets the barrier: it falls where endpoints become unusable and persists between depth-stable ones. Continuing training from the aligned midpoint for 30% of the original budget removes over 90% of its residual on two depth-stable pairs, and distilling a probability-mixture teacher keeps deep-runtime degradation below 0.05 dex, versus 0.40 dex for plain continuation. Relative barriers should thus be reported with absolute endpoint quality and runtime depth.

Primary area:
foundation or frontier models, including LLMs
Keywords:
weight-tied language models, permutation alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BfA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies linear weight interpolation between pairs of independently trained weight-tied (looped) 35M-parameter language models It measures the midpoint barrier before and after permutation alignment that respects weight tying, at the trained depth

Critical strengths
  1. The joint alignment / endpoint-quality / runtime-depth protocol exposes a real pitfall. The paper shows that a relative barrier can "vanish" at deeper runtime simply because the reference endpoints collapse. This is a concrete, easily overlooked artifact for anyone merging looped / weight-tied models, and the recommendation to always report absolute endpoint quality and depth alongside the barrier is well supported and actionable.

  2. The measurements are unusually carefully controlled and honestly reported. Four matcher families, norm-matched random twins, direction-vs-orthogonal subspace perturbations at identical dose, a dense non-tied control, a second-corpus replication, pair-cluster bootstrap intervals, and negative transfer results on public Pythia pairs are all included.

Critical weaknesses
  1. Narrow scope and weak statistical footing for the headline claim. All controlled results come from one 35M weight-tied family trained for 5k steps / 0.25B tokens (endpoint PPL 55-80, i.e., far from converged), and six of the fifteen pairs reuse four checkpoints, so there are far fewer than fifteen independent units.

  2. After the best of four alignments the midpoint is still 12-24x worse than the endpoints, which is roughly what a random perturbation of the same norm produces (twins reach ~1.3x the aligned log-barrier), so alignment does not yield usable merges here and the "residual" is not given a mechanistic interpretation. The repairs are evaluated on only one or two pairs each; plain continuation beats distillation at trained depth and at matched compute, so the distillation result reduces to a deep-runtime-stability advantage on a single pair family.

Questions for authors
  1. Why does permutation alignment remove 1-4 dex on the 35M looped pairs but <=5% on Pythia? Is this attributable to weight tying, to the short training budget (0.25B tokens), or to the Pythia confounds (data/dedup, unpublished seeds)? The dense 18-layer control (naive 1.83 dex, residual 1.29 dex) seems to show a much smaller shell than the looped pairs - does the shell shrink with training budget or with depth of the dense model? A dense pair with truly independent seeds at the same scale would resolve this.

  2. Given the [0.07, 2.77] dex interval on the naive recipe contrast, do the authors have additional independent seed pairs per recipe (rather than pairs reusing checkpoints) that would tighten this? Without it, is the claim better phrased as "after alignment, both recipes sit on the same residual" rather than "alignment explains the recipe contrast"?

  3. The norm-matched twin reaches ~1.3x the aligned log-barrier, i.e., the aligned midpoint is somewhat better than a random same-norm perturbation but still unusable. Is there any evidence that the achieved alignment finds structure (e.g., better than the twin on the depth-stable pairs after continued training, or lower loss on specific token classes), or is the practical conclusion that alignment is not useful for merging these models?

  4. Please state which perplexity estimator each headline number uses in one place (e.g., a single column in Table 1 on one basis), or justify why two estimators are needed at all.

Reviewer_SDSSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines weight interpolation between independently trained looped language models. It separates damage removed by permutation matching from the remaining midpoint damage, and evaluates both quantities alongside endpoint quality and runtime depth. In the primary 35M-model population, matching largely removes the contrast between training recipes, while a sizable residual remains. Norm- and direction-controlled perturbations characterize this residual. Further experiments examine continued training, probability-mixture distillation, and public Pythia pairs.

Critical strengths
  1. The joint treatment of alignment and endpoint degradation prevents a misleading interpretation of deep-runtime barriers.
  2. The perturbation study goes beyond a simple interpolation plot. Tensorwise norm matching, leading-versus-orthogonal subspaces, and random-frame controls test distinct explanations for the residual.
Critical weaknesses
  1. The analysis is more descriptive than explanatory. The strongest observations come from one small weight-tied model family. The public-model experiments reveal an important unresolved difference: the best tested matchers remove only about 2.36%, 5.00%, and 0% of the log-barrier on the 70M, 160M, and 410M Pythia pairs, and the direction contrast reverses on the larger pairs (Appendix R, pp. 41–44). The authors explicitly acknowledge this family dependence, so these results do not contradict the scoped claim.
  2. The repair experiments add limited weight to the contribution. Midpoint continuation is measured mainly on two depth-stable pairs, and the naive midpoint quickly approaches the aligned one after further training (Section 5.2).
Questions for authors

Do the existing experiments suggest a specific property that explains the contrasting alignment and direction results between the tied models and Pythia? A testable explanation of that difference would substantially strengthen the contribution.

Reviewer_3JLyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper empirically studies weight interpolation between independently trained weight-tied language models, examining how permutation alignment, runtime depth, and endpoint quality affect the measured midpoint loss. Across fifteen pairs of approximately 35M-parameter models, alignment largely removes the observed barrier difference between training-recipe groups, but the best tested matcher still leaves midpoint perplexity roughly 12-24 times that of the better endpoint on fourteen pairs. Deeper evaluation shows that smaller relative barriers can reflect endpoint deterioration rather than useful merged models. The study supplements these observations with alternative matchers, norm- and direction-controlled perturbations, width and training-budget comparisons, and public Pythia pairs. Continued training substantially recovers midpoint performance, while endpoint-initialized distillation provides a limited, budget-dependent depth-stability benefit. I understand the main contribution as an empirical characterization and an argument for reporting achieved alignment, absolute model quality, and runtime depth together, rather than a demonstrated practical parameter-merging method.

Critical strengths
  • Useful evaluation perspective. Section 4 and Figure 3 jointly report relative barriers and absolute model quality, showing that a smaller barrier can result from endpoint deterioration rather than a useful merged model. At 192 loops, the anchor pairs have a median better-endpoint perplexity of approximately 3.3e8 despite a negative median aligned barrier.

  • Targeted diagnostic controls. Section 3.2 compares perturbations with matched per-tensor norms across the displacement's leading subspace, its orthogonal complement, and a random subspace of equal rank. These comparisons help separate perturbation magnitude from directional effects, with median damage differences of approximately 1.14 and 1.27 dex against the latter two controls within the tested population.

  • Extensive and inspectable empirical records. The supplement provides configurations, training logs, per-window measurements, and analysis code that permit recomputation of key statistics. It also retains unfavorable results, including failed seed replication, limited Pythia alignment benefits, and the loss of distillation's advantage at longer budgets, allowing readers to assess the scope of the findings.

Critical weaknesses
  • The main comparison is not the one described in the paper. The abstract, Section 2, and Figure 1 describe the anchor models as trained at a fixed depth of 16. However, the released configurations and training code sample depths uniformly from 1 to 16, and all 400 logged depth entries across the four anchor runs match that sampler. The experiment therefore compares different training-depth distributions, not fixed-depth versus sampled-depth training. This changes the meaning of a central result and also undermines the interpretation of the depth-jitter control.

  • It is unclear what important new understanding the experiments provide. Most qualitative results fit existing expectations: independently trained parameters may not line up, permutation matching may leave substantial damage, perturbation direction matters, and a relative barrier can improve when its reference models deteriorate. Some specific observations are interesting, but the paper does not turn them into a clear explanation or a reliable prediction beyond the tested cases. The repair experiments do not fill this gap. Training the midpoint for another 30% of the original budget restores much of its quality, but does not establish an advantage over continuing a parent. Distillation starts from an intact parent, so its limited benefit does not demonstrate the value of parameter averaging. An empirical paper need not produce a deployable method, but it should make clear which important uncertainty it resolves.

  • The confidence intervals support less than the paper claims. The recipe-gap bootstrap takes the absolute difference between group medians. Keeping the sign with exactly the same resamples changes the naive-gap interval from approximately [0.072, 2.767] to [-0.320, 2.767] dex. The positive lower bound therefore cannot establish that the signed group difference is positive. Separately, the interval behind the claim that at least 83% of the contrast is explained keeps the original gap fixed in the denominator, leaving out its uncertainty. These issues matter particularly because the groups contain few independent training runs and several pairs share checkpoints.

  • The additional validation does not establish the claimed transfer and repair results. The public Pythia original/deduplicated pairs share identical published initial-weight files at all three tested sizes, so they do not test transfer to independently initialized public models. The stitching experiments have a separate problem: the maps are fitted and evaluated on the same windows. The reported roughly 90% and 95% recovery therefore shows performance on the fitting data, without establishing recovery on held-out data. The paper's separate cross-fit check concerns permutation matching and does not resolve this stitching issue.

Questions for authors

I'll suggest the authors to spend more time to think about how essentially this paper shall be reworked to some extent. But given what you currently deliver, if you want to continue to improve it, these two directions might help:

  1. Can you reconcile the claimed fixed-depth anchors with the released training records? All 400 logged depth entries across the four anchor runs match uniform sampling from 1 to 16. Please identify the training setup actually used for the reported checkpoints. If the released records are correct, revise the affected claims and add a genuinely fixed-16 control with multiple seeds, evaluated using the same alignment and depth protocol. This would clarify whether the main result concerns depth randomization or differences in training-depth support.

  2. What specific explanation does the paper establish beyond the expected failure of averaging independently trained models? Please select one central finding and state two plausible explanations that predict different outcomes. Then add a focused experiment that distinguishes them using fresh training seeds and evaluation data separate from any matching or fitting data. For example, does the near-disappearance of the group gap reflect learned feature correspondence, or would comparably selected random permutations remove much of it? A clear answer would strengthen the paper more than additional descriptive probes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.