QUILL: In-Weight Computation with MLPs

Language models continue to improve across mathematics, programming, and long-context reasoning tasks. Nevertheless, elementary arithmetic remains unreliable. Existing remedies use additional reasoning tokens or dedicated numerical executors. We instead study $\text{in-weight computation}$: encoding numerical functions directly in ordinary MLP weights for forward-pass execution, with intermediate values retained as neural activations. We introduce $\text{QUILL}(\text{QU}asi-\text{I}nterpo\text{L}ant\text{L}ayers)$, which constructs single-hidden-layer MLP approximations using quasi-interpolation theory. For univariate analytic functions, QUILL achieves geometric error decay to a finite-precision floor, with $O(\log (1/\epsilon ))$ width and working precision for target error $\epsilon$. The construction also reveals conditions under which gradient descent is slow to acquire the hidden-layer scales supporting high precision. We extend QUILL to multivariate polynomial programs by composing square modules into multipliers. For bounded computations in the realizable precision regime, the resulting QUILL circuits attain output error $\epsilon$ with $O(\log (2+\kappa /\epsilon ))$ nonlinear neurons per multiplication, where \(\kappa\) bounds the amplification of local approximation errors. Tested polynomial programs, including dot products and determinants, achieve sampled relative $L^2$ errors below $6.5\times 10^{-14}$ in FP64. Additional primitives support non-polynomial functions, fixed-step numerical algorithms, and bounded discrete programs. As a proof of concept, we install a width-80 QUILL squaring module in three frozen language models. With engineered routing and input/output handling, the installed pathway returns correct answers in all $2,250$ evaluations across three-, five-, and seven-digit inputs.