acceptodds
Under review as a conference paper at ICLR 2027

Thermodynamic Weight Decay: Global vs. Block-wise Control around Criticality

Abstract

Weight decay is typically applied using a fixed coefficient whose selection requires task-specific tuning. Inspired by dissipative structures in nonequilibrium thermodynamics, we propose Thermodynamic Weight Decay (TWD), a dynamic weight-decay method that views learning as a balance between energy influx and dissipation. TWD partitions a network into architectural parameter blocks, such as embedding, attention, and feed-forward modules, and treats the squared weight norm of each block as its internal energy. It sets a reference energy for each block according to its architecture and role, and drives the block energy toward a scaled target. On modular addition and multiplication tasks, TWD accelerates grokking by approximately 60-fold compared with training without control, while matching the performance of tuned static weight decay. Compared with global control, which constrains only the network-wide total energy, block-wise control prevents distortions in the energy allocation across blocks and accelerates grokking by approximately 20-fold. On Fashion-MNIST and CIFAR-10, TWD reduces test loss and the generalization gap while largely maintaining test accuracy. A broad sweep of the target-energy scale reveals four empirical regimes. Lowering the target initially accelerates grokking, but eventually weakens post-onset stability and robustness across random seeds; at still lower energy, grokking is substantially delayed or absent. Conversely, high energy leads to lazy feature learning, while the reference scale provides a balance between plasticity and stability. Experiments on GPT-2 (124M) show that block-wise control remains feasible at this scale, while also revealing that suitable energy targets depend on both parameter type and architecture. These results indicate that both the level and allocation of weight energy strongly affect plasticity and stability during learning, and may provide an operational indicator of criticality.

Primary area:
optimization
Keywords:
dynamic weight decay, block-wise norm control, criticality, grokking, feature learning, adaptive regularization, neural network optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jcjFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Thermodynamic Weight Decay (TWD), which chooses the weight-decay coefficient dynamically instead of fixing it. Inspired by nonequilibrium thermodynamics, it treats the squared weight norm of each functional block (embedding, attention, feed-forward, etc.) as "internal energy". At every step, it solves for the weight-decay coefficient that keeps that energy at a target, which is a hand-designed, architecture-based "reference energy" times a scale factor s (default 1). The coefficient can be negative, so energy can be added as well as removed. The authors report faster grokking on modular arithmetic, and lower test loss on Fashion-MNIST and CIFAR-10 at similar accuracy. Block-wise control works better than controlling only the total norm. A GPT-2 (124M) run shows the method can be used at that scale. Sweeping s, they describe four empirical "regimes" of learning behavior.

Critical strengths
  • The proposed method is described clearly and in detail: the full algorithm is given (App. A.1), with explicit formulas for each optimizer (App. A.3). The experimental settings are also explained in detail.
  • The experiments are broad: modular addition and multiplication, Fashion-MNIST, CIFAR-10 and GPT-2 pretraining, plus several diagnostics (CKA, Jacobian spectra, control statistics).
  • The comparison between global and block-wise control is a clean ablation: the total target energy is identical, and only the per-block allocation differs.
Critical weaknesses
  1. The motivation is weak. The paper introduces the "thermodynamic weight decay" idea right after the related work, without first establishing the problem it solves. The idea is interesting, but it's unclear what concrete problem with standard weight decay the method addresses, beyond avoiding a search over the coefficient. And as noted in Weakness 3, TWD still needs its own per-architecture design.

  2. The thermodynamic interpretation is not rigorous. It offers an interesting view of the method, but it is an analogy rather than a derivation. The authors admit this: "This thermodynamic correspondence is not asserted as a strict physical identity" (L.154). Moreover, terms such as "dissipative structure", "criticality", and "subcritical/supercritical" are labels for empirical patterns rather than tested physical claims. The four regimes in Table 1 come from a sweep over s on modular arithmetic, and the authors note that no single Jacobian criterion characterizes them (Sec. 5.2).

  3. The choice of reference/target energy lacks a strong rationale. The reference energies uirefu_i^{ref} come from initialization-style formulas (Kaiming, Xavier, N(0, σ²)), but they are selected and corrected by hand for each architecture and parameter role (App. A.2, Tables A.1–A.2), and they are not derived from the thermodynamic view. Some specific concerns:

    • The reference does not match the actual initialization. In App. B.5 (Table B.8), using the actual initial norm as the reference gives a much slower grokking onset on modular addition (800 vs. 448 epochs), yet the paper doesn't explain why the hand-designed reference is better.
    • The authors acknowledge that performance "depends strongly on the architecture and on the design of the functional blocks and reference energies" (Sec. 5.2). This shifts the tuning burden from the weight-decay coefficient to the block and reference design rather than removing it.
  4. It's not convincing that the method works better than standard weight decay. Fast grokking can also be obtained with a suitable fixed weight decay. Please clarify what the additional procedure buys over normal weight decay. In the paper's own results:

    • On CIFAR-10, final test accuracy is essentially the same across all methods (0.91–0.92), and no weight decay gives the highest accuracy (Sec. 4.2).
    • The GPT-2 result is a single seed, and the authors explicitly do not claim an improvement (Sec. 4.3).
  5. The CIFAR-10 baseline looks non-standard. ResNet-18 trained with SGD (lr 0.1) on CIFAR-10 usually reaches about 95% test accuracy with the standard weight decay of 5×10⁻⁴. Here all methods reach only 0.91–0.92, wd = 5×10⁻⁴ gives 0.837 (Table B.5), and Table B.2 does not mention a learning-rate schedule. This weakens the "suppressing overfitting" claim, which rests mainly on test loss.

Questions for authors
  • How are the reference energies uirefu_i^{ref} determined? Is the design based on a thermodynamic argument, or purely on initialization heuristics? How should practitioners design them for a new architecture?
  • What initialization does each model use, and how far is the initial energy from the target at s = 1?
Reviewer_Rg1eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose new weight decay variant name Thermodynamic Weight Decay (TWD), which is inpirsed by thermodynamic rule of nonequilibrium system. TWD dynamically adjusts decay strength across different parts (blocks) of neural networks, to move weights towards the equal-energy condition. TWD is evaluated on three experimental settings: modular addtion (grokking), image classification (CIFAR-10), and language learning (GPT-2 pretraining). Block-wise TWD surpasses vanilla weight decay in the modular addtion and language learning experiment.

Critical strengths

None

Critical weaknesses
  1. Limited novelty
  • The idea of adaptively adjusting weight decay strength is largely explored in previous works [1-4]. The paper does not describes limitation of those similar existing methods nor how PWD solves it.
  • It is unable to find advantage of TWD compare to existing method or new lessons obtained through experiment in the paper.
  1. The experiments are not sound. It is hard to assess effectiveness of the proposed method
  • The paper does not use similar existing methods as baseline [1-6], and only compares TWD with vanilla WD. It is unable to judge effectivness of the proposed method unless appropriate baselines are included.
  • In Figure 3, no control (no weight decay) baseline is largely better than tuned static baseline. In most of the case weight decay gives slight improvement or similar performance if it is tuned properly. I am very suspicious of whether the proper search space is used for the tuning.
  1. The paper is very poorly written and hard to understand
  • Introduction section does not provides any existing observations and limitation of previous works, or challenges this paper aims to solve. It is extremely short and does not reflect any novelty or significance of the work.
  • Important technical details are not sufficiently presented in the main text, and placed in appendix (e.g. L183, L197). It is very hard to follow the method section due to insuffucient explanation.
  • Justification on the proposed method is very weak. The motvation based on thermodynamic law is not sufficient to justify every details of the proposed method. Especially the method section only delivers details of the method rather than design principles or reason of the design choice.
  • Overall, I believe the paper should largely be updated to reflect novelty and significance of the research, improve clarity, and include sufficient justification of the method.

References

[1] Franke, Jörg K., et al. "Improving deep learning optimization through constrained parameter regularization." Advances in Neural Information Processing Systems 37 (2024): 8984-9025.

[2] He, Di, et al. "Alphadecay: Module-wise weight decay for heavy-tailed balancing in llms." Advances in Neural Information Processing Systems 38 (2026): 29745-29768.

[3] Ghiasi, Mohammad Amin, Ali Shafahi, and Reza Ardekani. "Improving robustness with adaptive weight decay." Advances in Neural Information Processing Systems 36 (2023): 79067-79080.

[4] Nakamura, Kensuke, and Byung-Woo Hong. "Adaptive weight decay for deep neural networks." IEEE Access 7 (2019): 118857-118865.

[5] Xie, Zeke, et al. "On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective." Advances in Neural Information Processing Systems 36 (2023): 1208-1228.

[6] Chen, Lizhang, et al. "Cautious weight decay." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. What is novelty of this work compared to existing works?
  2. Does PWD is better than similar existing methods? 2-1. If yes, then why it is better than them?
  3. Could the authors clarify the baselines are tuned in proper search space?
  4. Could the authors provided enough justification on the proposed method?
Reviewer_PeGgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Thermodynamic Weight Decay (TWD), a closed loop dynamic weight-decay method inspired by nonequilibrium thermodynamics which adaptively adjusts weight-decay coefficients to maintain architecture-aware target energies. Experimental results show that TWD substantially accelerates grokking, reduces test loss and generalization gaps on image classification tasks. The author further varifies the usefulness of TWD on a larger scale model like GPT -2. Finally, the authors identify four empirical learning regimes, by varying the target-energy scale.

Critical strengths
  1. The empirical evidence is strong, clear and broad. Block-wise TWD substantially accelerates grokking compared with no weight decay and global TWD. These results demonstrate the value of TWD in accelerating grokking. The authors evaluate TWD across several architectures and tasks, which further strengthen the usefulness of the proposed method.
  2. With the rapid development of AI, understanding the underlying principles of AI models is increasingly important. I appreciate the authors' efforts to understand deep learning from a statistical physics perspective.
Critical weaknesses
  1. The practical value of TWD remains unclear. Although TWD accelerates grokking, it does not consistently improve model performance and can even underperform the baseline without TWD, as shown in Fig. 3. Considering its additional computational cost, its overall advantage is questionable. Moreover, reliably observing grokking often requires carefully designed and specific experimental settings, further raising concerns about TWD's applicability to more general benchmarks and experimental settings.
  2. While I appreciate the authors' effort that try to understand the problem from thermodynamic and statistical physics perspective, the paper does not appear to provide substantive insights from statistical physics. The thermodynamic interpretation seems largely analogical, without a theoretical connection between the proposed equations and thermodynamic principles. Furthermore, the definitions and claims regarding various forms of criticality in Figs. 5–6 and Table 1 lack rigor and sufficient supporting evidence, making their scientific significance unclear.The proposed critical band is mainly identified through empirical learning performance. The paper provides no clear evidence of a phase transition, bifurcation, or universal critical signature.
  3. Appendix C.3 shows that post update norm projection achieves nearly identical results to TWD. This raises questions about the unique benefit of the proposed dynamic control mechanism over simpler norm-control methods, further weakening the paper's contribution.
Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.