- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Bounded Depth Tempering (BDT), an AdamW-based optimizer that adapts updates across Transformer layers using depthwise statistics. Experiments on 124M language models and pretrained layer-copying settings show improvements over AdamW and suggest that parameter correspondence and update-magnitude allocation affect optimization, although advantages over Gradient Smoothing and generalization beyond copied-layer settings remain unestablished.
Critical strengths
-
Careful controls for isolating correspondence effects. The matched-direction comparison controls the common-depth component, pooled centered norm, and angular departure from AdamW.
-
A meaningful mechanistic decomposition supported by prospective confirmation. The amplification–suppression crossover shows that the utility of ancestry depends on how updates are transformed.
-
A clean demonstration that cross-layer coupling depends on correspondence.
Critical weaknesses
-
Unclear advantage over the closest baseline. The equal-search comparison with Gradient Smoothing does not establish a statistically significant advantage for BDT (mean difference: −0.000185 nats/token; 95% CI: [−0.003497, +0.003127]).
-
Unproven practical efficiency. BDT has higher measured full-step latency than both Gradient Smoothing and native fused AdamW, yet the paper does not establish a compensating time-to-quality or end-to-end training-cost advantage.
-
Incomplete isolation of the correspondence mechanism. The matched-direction control does not match layerwise update-norm allocation or temporal direction consistency, and the reported extreme amplification of scrambled residuals raises concerns about whether the observed performance gap can be attributed specifically to correspondence-derived direction.
-
Limited generalization beyond copied-layer initialization. BDT provides nearly no improvement for independently initialized FFNs, while its larger gains occur with cloned FFNs; moreover, the longer-horizon adjusted confidence interval for preserved BDT versus AdamW includes zero, leaving the robustness of the benefit insufficiently established.
Questions for authors
-
Is BDT intended primarily for pretrained model growth and copied-layer initialization, or as a general improvement to cross-layer optimization? How do the authors reconcile the broader framing with the near-zero effect under independent FFN initialization (Table 22)?
-
Does BDT offer a meaningful quality–cost advantage over Standard-Proj Gradient Smoothing, given the inconclusive equal-search loss comparison and the higher measured full-step latency? A compute-matched comparison or time-to-fixed-validation-loss analysis would materially affect my assessment.
-
Incremental value of spectral estimation. The fixed (\rho=2) pair-amplification rule comes within (0.000057) nats/token of combined BDT on two development parents (Table 34). Can the authors demonstrate a setting in which BDT's data-dependent operator reliably outperforms this simpler rule, particularly across different copying layouts?
-
Long-horizon persistence. In the 15K-update study, the adjusted preserved-versus-AdamW interval includes zero, although the ancestry-preservation contrast remains significant. Can the authors provide intermediate trajectories for these existing policies and, if claiming that BDT's own optimization benefit persists, add a canonical-BDT comparison at the longer horizon?
Response to Reviewer sdKH
Hello, thank you for taking the time to read and review my paper. I think your assessment is fair and aligns with my evaluation of my paper.
Q1 & Q2: The practical benefit of BDT is limited. It is mainly useful in copied-layer initialization, and the quality comparison to Gradient Smoothing was a minimal difference in the experiments run prior to the submission of the paper. A compute-matched, time-to-target comparison needed more GPU time than I had available. I now realize that I was exploring the mechanism of parameter correspondence shaping cross-layer optimization and providing an optimization result alongside it (which was not convincing enough), and this paper did not frame the real focus clearly enough.
Q3: I tested this, and a simple rule did better. In the paper, the fixed rule used rho = 2 because it roughly matches BDT’s own effective coefficient, which is why the two came out to be so close in Table 34. However, I tuned rho on two development parents and tests it on four fresh parents: rho = 2.5 beat combined BDT by -0.00086 nats/token, 95% CI [-0.00092, -0.00080] on all four parents, and this led me to test rho = 3 which improved further, suggesting the optimal value may be even larger. A schedule following BDT’s own effective coefficient performed about the same as BDT. This suggests that the amplification level of paired-layer differences is what matters, and that a fixed rule captures it better than the online spectral estimate.
Q4: At 15k updates, the ancestry contrast persists, but the longer-horizon study didn’t include a canonical BDT arm, so it doesn’t show whether BDT’s own benefit persists. Combining the insights I have gained from these reviews, I plan to modify BDT, replace the adaptive estimator with a tuned fixed rule, and evaluate it against AdamW at the longer horizon, reporting intermediate trajectories as well.
However, considering the current results, I don’t think the paper is ready in its current state, so I will be withdrawing it and revising it with your comments in mind. Once again, thank you so much for taking the time to review and advise my paper.