BEYOND 4D FFNS: HESSIAN POTENTIAL MLP FOR MEMORY AND LANGUAGE MODELING

Position-wise feed-forward networks (FFNs) dominate Transformer cost, yet remain standard two-layer MLPs expanding to width $4d$ before linear projection. We introduce the Hessian Feed-Forward Network (H-FFN), $\text{H-FFN}(x)={\nabla }^2\Psi (x)x={\sum }_j{\lambda }_j(x)(u_j^{\top }x)u_j$, expressing the sublayer as a sum of rank-one directional units derived from a scalar potential. For associative retrieval, a coaxial Memory-Track employs multi-scale contextual gating; for language modeling, Decoupled H-FFN separates content, routing, and emission. Under matched training budgets, Memory-Track reaches $96.5\pm 0.7\%$ Needle-in-a-Haystack accuracy versus $75.7\pm 5.8\%$ for a matched MLP, and Decoupled H-FFN achieves $1.06$–$1.38\times$ perplexity gains across causal language modeling benchmarks ($1.17\times$ on WikiText-103; $\approx 1.15\times$ on FineWeb-Edu). In transfer, analytic GELU matching improves GPT-2 fine-tuning ($1.07\times$) and accelerates ViT-B/16 convergence ($+2.4\%$ top-1 with $3\times$ fewer steps).