- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Motivated by existing theoretical and empirical works showing that training overparameterized models with weight decay induces grokking, this paper explores a straightforward question that how would strengthening/weakening weight decay during the plateau stage of grokking affect the generalization delay. The authors use the AdamW optimizer, which separate the effect of weight decay and the gradient computation. The authors implement large-scale intervention-sweep experiments on three concrete algorithmic tasks and observe that the temporary weight decay change early in the plateau stage has vague effects, while in a later stage, a large increase of weight decay advances generalization more and a large decrease delays generalization more. They also compare the perturbed and baseline generalization models and find that in a regime where regularization governs optimization, generalization solution selection becomes more restricted, thus matching the existing theoretical results. Overall, the contributions of this paper are mainly on the empirical side.
Critical strengths
-
The empirical observation that towards the end of the test accuracy plateau, temporary interventions on the weight decay induce ordered generalization time change, is a bit interesting. Although no theoretical/analytical explanation has been provided in the paper, it might be of interests for future works.
-
The experimental design as well as the intervention framework are clean. Also, the experiments are thorough and carefully controlled such that the results are convincing.
Critical weaknesses
I think the contribution of the paper is somewhat limited. Weight decay induces grokking has been extensively explored both theoretically and empirically. While existing works usually focus on small models such as linear models or shallow neural networks, this paper also just empirically tests three small selected algorithmic grokking tasks with clean plateau phenomenon (also discussed by the authors in Section 4). Does the conclusion observed in the three tasks generalize to more general learning scenarios? Could the intervention design be generalized to scenarios where no explicit plateau exists? I found it hard for me to come up with a broader significance of this work. I understand that an empirical paper could merit acceptance without any theorem proved or algorithm designed, but it should convey important messages to guide future works.
Questions for authors
From my understanding, one of the most interesting results in the paper is that, on the three small algorithmic tasks, weight decay interventions tend to have ordered effects when approaching the late stage of plateau. I think the results could be more meaningful if it is general and widely applicable. For example, would you like to test this observation on some general deep learning problems such as training overparameterized two-layer networks using SGD? (Since prior works have demonstrated grokking in two-layer ReLU networks trained on modular addition using full-batch gradient descent.)
Another minor question: how would the magnitude of the intervention duration affect the generalization delay?