- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Thermodynamic Weight Decay (TWD), which chooses the weight-decay coefficient dynamically instead of fixing it. Inspired by nonequilibrium thermodynamics, it treats the squared weight norm of each functional block (embedding, attention, feed-forward, etc.) as "internal energy". At every step, it solves for the weight-decay coefficient that keeps that energy at a target, which is a hand-designed, architecture-based "reference energy" times a scale factor s (default 1). The coefficient can be negative, so energy can be added as well as removed. The authors report faster grokking on modular arithmetic, and lower test loss on Fashion-MNIST and CIFAR-10 at similar accuracy. Block-wise control works better than controlling only the total norm. A GPT-2 (124M) run shows the method can be used at that scale. Sweeping s, they describe four empirical "regimes" of learning behavior.
Critical strengths
- The proposed method is described clearly and in detail: the full algorithm is given (App. A.1), with explicit formulas for each optimizer (App. A.3). The experimental settings are also explained in detail.
- The experiments are broad: modular addition and multiplication, Fashion-MNIST, CIFAR-10 and GPT-2 pretraining, plus several diagnostics (CKA, Jacobian spectra, control statistics).
- The comparison between global and block-wise control is a clean ablation: the total target energy is identical, and only the per-block allocation differs.
Critical weaknesses
-
The motivation is weak. The paper introduces the "thermodynamic weight decay" idea right after the related work, without first establishing the problem it solves. The idea is interesting, but it's unclear what concrete problem with standard weight decay the method addresses, beyond avoiding a search over the coefficient. And as noted in Weakness 3, TWD still needs its own per-architecture design.
-
The thermodynamic interpretation is not rigorous. It offers an interesting view of the method, but it is an analogy rather than a derivation. The authors admit this: "This thermodynamic correspondence is not asserted as a strict physical identity" (L.154). Moreover, terms such as "dissipative structure", "criticality", and "subcritical/supercritical" are labels for empirical patterns rather than tested physical claims. The four regimes in Table 1 come from a sweep over s on modular arithmetic, and the authors note that no single Jacobian criterion characterizes them (Sec. 5.2).
-
The choice of reference/target energy lacks a strong rationale. The reference energies come from initialization-style formulas (Kaiming, Xavier, N(0, σ²)), but they are selected and corrected by hand for each architecture and parameter role (App. A.2, Tables A.1–A.2), and they are not derived from the thermodynamic view. Some specific concerns:
- The reference does not match the actual initialization. In App. B.5 (Table B.8), using the actual initial norm as the reference gives a much slower grokking onset on modular addition (800 vs. 448 epochs), yet the paper doesn't explain why the hand-designed reference is better.
- The authors acknowledge that performance "depends strongly on the architecture and on the design of the functional blocks and reference energies" (Sec. 5.2). This shifts the tuning burden from the weight-decay coefficient to the block and reference design rather than removing it.
-
It's not convincing that the method works better than standard weight decay. Fast grokking can also be obtained with a suitable fixed weight decay. Please clarify what the additional procedure buys over normal weight decay. In the paper's own results:
- On CIFAR-10, final test accuracy is essentially the same across all methods (0.91–0.92), and no weight decay gives the highest accuracy (Sec. 4.2).
- The GPT-2 result is a single seed, and the authors explicitly do not claim an improvement (Sec. 4.3).
-
The CIFAR-10 baseline looks non-standard. ResNet-18 trained with SGD (lr 0.1) on CIFAR-10 usually reaches about 95% test accuracy with the standard weight decay of 5×10⁻⁴. Here all methods reach only 0.91–0.92, wd = 5×10⁻⁴ gives 0.837 (Table B.5), and Table B.2 does not mention a learning-rate schedule. This weakens the "suppressing overfitting" claim, which rests mainly on test loss.
Questions for authors
- How are the reference energies determined? Is the design based on a thermodynamic argument, or purely on initialization heuristics? How should practitioners design them for a new architecture?
- What initialization does each model use, and how far is the initial energy from the target at s = 1?