acceptodds
Withdrawn from ICLR 2027

How Parameter Correspondence Shapes Cross-layer Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cross-layer optimization couples updates from different Transformer layers and therefore requires a correspondence between their parameters. We introduce bounded depth tempering (BDT), which reshapes AdamW proposals across depth by attenuating high-energy depthwise modes while preserving their common-depth component and pooled centered norm. We compare BDT with correspondence-altered controls matched in common component, centered norm, and angular departure from AdamW at each control branch’s own state. In controlled 124M language models, BDT lowers validation loss relative to both AdamW and these matched controls, showing that correspondence-derived direction carries useful information beyond these measures of update strength. Pretrained layer copying then gives a known relationship between descendants of the same block. Preserving that ancestry retains the benefit, while breaking the ancestry match loses it. Ancestry alone is not sufficient. True descendant pairing outperforms false pairing when update differences are amplified, and the ordering reverses under suppression. We then separate magnitude allocation across ancestry-defined sectors from direction within them. Crossed controls identify magnitude allocation as the more consequential difference. Under a prospectively specified shared window-smoothed direction construction, full-BDT allocations outperform structured-rule allocations, with the ordering reproduced on fresh parents. Relative to fixed ancestry amplification, retained diagnostics show that BDT assigns a larger share of centered proposal energy to descendant differences early in training and a smaller share late. Together, these results establish correspondence, operation, and magnitude allocation as distinct design choices in depth-coupled optimization.

Primary area:
optimization
Keywords:
deep learning optimization, parameter correspondence, Transformer optimization, layer-wise optimization, optimizer geometry, parameter reuse, permutation symmetries, pretrained model growth

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sdKHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Bounded Depth Tempering (BDT), an AdamW-based optimizer that adapts updates across Transformer layers using depthwise statistics. Experiments on 124M language models and pretrained layer-copying settings show improvements over AdamW and suggest that parameter correspondence and update-magnitude allocation affect optimization, although advantages over Gradient Smoothing and generalization beyond copied-layer settings remain unestablished.

Critical strengths
  1. Careful controls for isolating correspondence effects. The matched-direction comparison controls the common-depth component, pooled centered norm, and angular departure from AdamW.

  2. A meaningful mechanistic decomposition supported by prospective confirmation. The amplification–suppression crossover shows that the utility of ancestry depends on how updates are transformed.

  3. A clean demonstration that cross-layer coupling depends on correspondence.

Critical weaknesses
  1. Unclear advantage over the closest baseline. The equal-search comparison with Gradient Smoothing does not establish a statistically significant advantage for BDT (mean difference: −0.000185 nats/token; 95% CI: [−0.003497, +0.003127]).

  2. Unproven practical efficiency. BDT has higher measured full-step latency than both Gradient Smoothing and native fused AdamW, yet the paper does not establish a compensating time-to-quality or end-to-end training-cost advantage.

  3. Incomplete isolation of the correspondence mechanism. The matched-direction control does not match layerwise update-norm allocation or temporal direction consistency, and the reported extreme amplification of scrambled residuals raises concerns about whether the observed performance gap can be attributed specifically to correspondence-derived direction.

  4. Limited generalization beyond copied-layer initialization. BDT provides nearly no improvement for independently initialized FFNs, while its larger gains occur with cloned FFNs; moreover, the longer-horizon adjusted confidence interval for preserved BDT versus AdamW includes zero, leaving the robustness of the benefit insufficiently established.

Questions for authors
  1. Is BDT intended primarily for pretrained model growth and copied-layer initialization, or as a general improvement to cross-layer optimization? How do the authors reconcile the broader framing with the near-zero effect under independent FFN initialization (Table 22)?

  2. Does BDT offer a meaningful quality–cost advantage over Standard-Proj Gradient Smoothing, given the inconclusive equal-search loss comparison and the higher measured full-step latency? A compute-matched comparison or time-to-fixed-validation-loss analysis would materially affect my assessment.

  3. Incremental value of spectral estimation. The fixed (\rho=2) pair-amplification rule comes within (0.000057) nats/token of combined BDT on two development parents (Table 34). Can the authors demonstrate a setting in which BDT's data-dependent operator reliably outperforms this simpler rule, particularly across different copying layouts?

  4. Long-horizon persistence. In the 15K-update study, the adjusted preserved-versus-AdamW interval includes zero, although the ancestry-preservation contrast remains significant. Can the authors provide intermediate trajectories for these existing policies and, if claiming that BDT's own optimization benefit persists, add a canonical-BDT comparison at the longer horizon?

1 comment
AuthorsOpenReview ↗

Response to Reviewer sdKH

Hello, thank you for taking the time to read and review my paper. I think your assessment is fair and aligns with my evaluation of my paper.

Q1 & Q2: The practical benefit of BDT is limited. It is mainly useful in copied-layer initialization, and the quality comparison to Gradient Smoothing was a minimal difference in the experiments run prior to the submission of the paper. A compute-matched, time-to-target comparison needed more GPU time than I had available. I now realize that I was exploring the mechanism of parameter correspondence shaping cross-layer optimization and providing an optimization result alongside it (which was not convincing enough), and this paper did not frame the real focus clearly enough.

Q3: I tested this, and a simple rule did better. In the paper, the fixed rule used rho = 2 because it roughly matches BDT’s own effective coefficient, which is why the two came out to be so close in Table 34. However, I tuned rho on two development parents and tests it on four fresh parents: rho = 2.5 beat combined BDT by -0.00086 nats/token, 95% CI [-0.00092, -0.00080] on all four parents, and this led me to test rho = 3 which improved further, suggesting the optimal value may be even larger. A schedule following BDT’s own effective coefficient performed about the same as BDT. This suggests that the amplification level of paired-layer differences is what matters, and that a fixed rule captures it better than the online spectral estimate.

Q4: At 15k updates, the ancestry contrast persists, but the longer-horizon study didn’t include a canonical BDT arm, so it doesn’t show whether BDT’s own benefit persists. Combining the insights I have gained from these reviews, I plan to modify BDT, replace the adaptive estimator with a tuned fixed rule, and evaluate it against AdamW at the longer horizon, reporting intermediate trajectories as well.

However, considering the current results, I don’t think the paper is ready in its current state, so I will be withdrawing it and revising it with your comments in mind. Once again, thank you so much for taking the time to review and advise my paper.

Reviewer_rtCWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces bounded depth tempering (BDT), which applies an adaptive spectral transformation to centered AdamW updates across Transformer layers while preserving the common component and pooled centered norm. Through matched correspondence controls, pretrained layer copying, pairing interventions, and crossed sector-budget/direction experiments, it studies which cross-layer relationships make the transformation useful. The main findings are that preserving copied-layer ancestry helps, the effect depends on whether differences are amplified or suppressed, and sector magnitude allocation explains most of the gap to a structured comparator in the tested setting.

Critical strengths
  1. The experimental controls are careful. Matching the common component, centered norm, and angular departure goes beyond a routine optimizer comparison, while balanced chart reassignment directly tests ancestry. Separate fresh-parent confirmations, including the allocation comparison in Appendix AA, strengthen the evidence.

  2. The distinction between correspondence, operation, and allocation is informative. The paper separates development from confirmation and reports negative or inconclusive outcomes, making its empirical conclusions more credible.

Critical weaknesses
  1. The practical contribution remains narrow. Appendix K reports very small gains for ordinarily and independently initialized Dense models, whereas the much larger cloned-FFN gains do not close the gap to independent initialization. The equal-search standalone Gradient Smoothing comparison in Appendix L does not establish an advantage. There is a larger-model experiment, but it uses only 98.3M training exposures; the longer-horizon fresh-parent comparison also does not establish superiority to AdamW under its specified adjusted interval (Table 38). These results support an interesting account of copied-layer correspondence more strongly than a broadly useful optimizer improvement.

  2. The allocation experiments do not establish the necessity of the adaptive spectral machinery. The principal structured comparator uses fixed amplification rho = 2, and a simple amplification rule already captures much of the gain in the development comparisons. Appendix Z acknowledges that another constant or predetermined schedule could obtain similar quality. The remaining allocation benefit is small, so an equally tuned simpler policy is important for assessing whether the online estimator adds enough value to justify its complexity and measured computational cost.

Questions for authors
  1. Can an equally tuned fixed amplification coefficient or predetermined phase/projection schedule recover BDT's allocation advantage on fresh parents?

  2. Can the strict, equal-search protocol include standalone Gradient Smoothing and longer ordinary independently initialized Dense training, reporting both quality and measured time to a common target? The existing comparisons leave these practical benefits unresolved.

  3. How much of the ancestry contrast remains when individual-layer norm allocation is additionally controlled, given the wider reassigned layerwise norm ratios in Appendix M? The unforced controls in Appendix Q are helpful, but do not fully answer this question.

1 comment
AuthorsOpenReview ↗

Response to Reviewer rtCW

Hello, first of all, thank you for taking the time to read and review my paper. I think your criticisms are fair and generally aligned with my own evaluation of the paper, and your questions helped me clarify the next direction I should take from here.

Q1: I tested both. A tuned fixed coefficient (rho selected on two development parents, tested on four fresh parents) matched and exceeded combined BDT: rho = 2.5 gave -0.00086 nats/token, 95% CI [-0.00092, -0.00080], 4/4 parents. This led me to test rho = 3, and it improved further, suggesting that the optimal rho may be even larger, but I have yet to test larger values. A predetermined schedule that follows BDT’s own effective coefficient performed about the same as BDT (+0.00008). This suggests that amplification level of paired-layer differences is the key ingredient of BDT’s allocation and it can actually be set by a fixed rule.

Q2: I have not run this yet. An equal-search, time-to-target comparison across several methods and a longer horizon requires more GPU time than I had available, so the practical benefit remains unresolved. In hindsight, I was juggling between a) exploring mechanics in parameter correspondence shaping cross layer optimization and b) finding a strong practical benefit, and resulted in neither being convincing enough. This is a major takeaway I got from the reviews, and I will keep this in mind when fixing the paper.

Q3: I haven’t tested this yet. The per-layer norms were left unmatched on purpose, because BDT itself redistributes update size across layers and I treated that as part of the effect being tested. Given the Q1 result though, this layerwise allocation may be the mechanism rather than a side effect, so I agree that a norm-matched control is needed to tell the two apart, and I’ll add it in the revision.

In light of everything, I think the current state of the paper does not meet the quality of the conference, so I will be withdrawing this paper. However, I have gained insight on how to move forward with this paper, and I will look to revise and improve it. Once again, thank you so much for taking the time to review and advise my paper.

Reviewer_zzTeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a modification to AdamW from the perspective of cross-layer optimization, where interactions between layers are used to produce a method that the authors call bounded depth tempering (BDT). Several experimental results are provided, which the authors claim to establish the main claims.

Critical strengths

I cannot identify any critical strengths in the current manuscript. It is possible that there are genuine contributions, but the poor quality of the writing makes it difficult to comprehend what this paper is trying to say.

Critical weaknesses

I have absolutely no idea what large parts of this paper are trying to say. As stated the AI disclosure, generative AI was heavily used in the preparation of this work, and the current manuscript has the characteristics of poor-quality AI-generated writing. I would focus on revising Sections 2 to 5, particularly improving the presentation of the main method with justifications of the specific design choices, as well as clearer discussion of what the experimental results mean.

Questions for authors

Could you rewrite the text to be more human readable, so that reviewers can accurately assess the strengths and weaknesses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.