The Geometry of Fast and Slow Weights In A Shared Address Space

Fast weights have proven ubiquitous in memory systems of current state-space models (SSMs), but only recently have we seen advances in their theoretical understanding and in how they may relate to canonical forms of recurrent neural networks. Here, we provide a theoretical bridge between a classical connectionist view of recurrent weights as fixed connectivity, and one of fast weights, closer to modern ML literature. We use a structural connectivity approach to derive a low-rank common projection space for both slow and fast weights, relating each to mean and variance modes. In doing so, we provide a geometric and dynamical interpretation of fast weights from the perspective of a common presynaptic or address space, which reflects hard-wired pathways of inputs, and distinct subspaces for postsynaptic or content spaces. We find that delta-rule corrections, operating within the slow weight's subspace, reduce interference compared to a vanilla fast-weight update. This provides a framework for analyzing a division of labor between in-context or test-time learning, stored in fast weights, and more long-term functional attributes. We use this framework to design an SSM with competitive results in within-task learning and analyze the contributions of fast and slow modes.