Is Grokking Observable from Representation Dynamics?

Prior work has linked grokking to the evolution of hidden representation geometry, yet it remains unclear whether representation geometry can provide a reliable observable of grokking during learning. To address this question, we introduce Mint, a label-free statistic that quantifies pairwise representation overlap, i.e., the overlap of hidden features, and identify geometry-to-behaviour assumptions under which representation overlap serves as a reliable observable of grokking. Although these assumptions need not hold in general, Mint nevertheless captures informative changes in representation overlap throughout training. We further investigate Mint-penalized training as a means of reducing representation overlap. Our experiments show that penalizing the loss by Mint within a small range can advance the onset of generalization in practice. Together, our results clarify the conditions under which representation overlap can serve as an observable of grokking. We also identify representation overlap as a practical target for influencing generalization onset, and establish Mint as an informative descriptor of representation dynamics throughout learning.