acceptodds
Under review as a conference paper at ICLR 2027

QUILL: In-Weight Computation with MLPs

Abstract

Language models continue to improve across mathematics, programming, and long-context reasoning tasks. Nevertheless, elementary arithmetic remains unreliable. Existing remedies use additional reasoning tokens or dedicated numerical executors. We instead study : encoding numerical functions directly in ordinary MLP weights for forward-pass execution, with intermediate values retained as neural activations. We introduce , which constructs single-hidden-layer MLP approximations using quasi-interpolation theory. For univariate analytic functions, QUILL achieves geometric error decay to a finite-precision floor, with width and working precision for target error . The construction also reveals conditions under which gradient descent is slow to acquire the hidden-layer scales supporting high precision. We extend QUILL to multivariate polynomial programs by composing square modules into multipliers. For bounded computations in the realizable precision regime, the resulting QUILL circuits attain output error with nonlinear neurons per multiplication, where \(\kappa\) bounds the amplification of local approximation errors. Tested polynomial programs, including dot products and determinants, achieve sampled relative errors below in FP64. Additional primitives support non-polynomial functions, fixed-step numerical algorithms, and bounded discrete programs. As a proof of concept, we install a width-80 QUILL squaring module in three frozen language models. With engineered routing and input/output handling, the installed pathway returns correct answers in all evaluations across three-, five-, and seven-digit inputs.

Primary area:
learning theory
Keywords:
in-weight arithmetic, quasi-interpolation, MLP construction, multilayer perceptrons, high-precision computation, neural function approximation, arithmetic circuits, numerical algorithms, neural network compilation, discrete computation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JmZ3OpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose a method for encoding bounded numerical functions in ordinary MLP weights. The proposed method can assist LM to achieve better accuracy with lower token cost in numerical functions computation compared to traditional solutions.

Critical strengths
  1. The proposed method can assist LM to achieve better accuracy with lower token cost in numerical function computations compared to using caculate, direct prompt and Chain of Thought.
  2. This method enables the MLP component to achieve a lower L2L^2 error whilst reducing the scale of hidden neurons.
Critical weaknesses

The proposed method encodes the target bounded numerical function into the weights of a standard MLP, which appears to imply that QUILL must prepare specific MLP weights for different target numerical functions. Although table 3 has demonstrated that the proposed approach can improve accuracy and reduce token cost in x2x^2 computation, the sacrifice of flexibility in general use cases does not convince me to use QUILL to replace a calculator.

Questions for authors

I appreciate the author's efforts in this work and would kindly ask authors to clarify the following questions:

  1. Does QUILL require to prepare different MLP weights for different numerical functions?
  2. If so, please clarify in which use cases it is worthwhile to sacrifice flexibility in exchange for improved accuracy and reduced token cost.
7 comments

edit nov 16: mistakenly duplicated response; please ignore this comment

We thank the reviewer for recognizing QUILL's numerical accuracy, compact MLP construction, and reduced token cost in our language-model experiments. We appreciate the reviewer's questions about flexibility and practical applications, and would like to clarify how the LLM experiments relate to the paper's broader contributions before addressing them.

1. Clarifying QUILL's contributions

QUILL's primary contribution is a constructive framework for reliable numerical computation in ordinary MLP weights. Our goal is to understand what numerical computations can be explicitly implemented in standard neural-network architectures, rather than to develop a replacement for conventional calculators. The LLM experiments in Section 5 provide an initial proof of concept for accessing these capabilities within pretrained models, and our shared response further discusses the broader motivation for studying in-weight computation.

The theoretical contribution extends beyond approximating individual numerical functions. For the stated analytic target class, QUILL jointly establishes geometric convergence, numerical realizability in standard floating-point arithmetic, and ordinary single-hidden-layer MLP form (Theorem 3.1; Table 1). Our experiments achieve relative L2L^2 errors near 10−1410^{-14} in FP64. We also establish how reusable primitives can be composed into bounded polynomial arithmetic circuits, with end-to-end accuracy and resource guarantees (Theorem 4.1).

The construction also provides a reference for studying numerical capabilities learned by neural networks. Prior work finds that pretrained language models perform arithmetic through collections of MLP heuristics [1]. Our controlled shallow-MLP experiments and analysis investigate why gradient-based training can struggle to recover the high precision attainable through explicit construction (Theorems 3.2–3.3). This is a separate scientific motivation for studying numerical computation in ordinary MLPs.

We agree that conventional calculators provide efficient, flexible general-purpose numerical evaluation. However, QUILL is not intended to replace them. Its primary value lies in establishing a constructive framework for precise computation within ordinary neural-network weights, with explicit size and error guarantees.

AuthorsOpenReview ↗

2. Responses to the reviewer's questions

The construction, composition guarantees, and training analysis above form the core of QUILL’s contributions to understanding and improving numerical computation in MLPs. We now address the reviewer’s questions about the flexibility of function-specific weights and the practical settings in which these constructions are useful.

Q1. Does QUILL require different weights for different numerical functions?

Yes. QUILL constructs weights for a prescribed primitive and domain. These primitives can be reused across inputs and assembled into different programs without fitting each complete function from scratch. For example, squaring supplies multiplication (Equation (10)), supporting dot products, determinants, and other polynomial computations. Theorem 4.1 establishes end-to-end accuracy and network-resource guarantees for bounded polynomial arithmetic circuits assembled from these primitives.

Our expanded experiments further demonstrate construction of larger continuous and discrete computations, including heat-equation solution evaluation and Boolean sorting (Table R1). Constructed QUILL achieves a maximum absolute error of 2.32×10−122.32\times10^{-12} on the heat-equation task, compared with 5.06×10−35.06\times10^{-3} for the trained baseline. For Boolean sorting, QUILL produces exact outputs on all 4,096 evaluated cases, whereas end-to-end training of the same architecture produces none under the tested optimization settings.

Table R1. Accuracy of constructed and end-to-end-trained numerical and discrete programs.

ComputationConstructed QUILLSame architecture, trained
Heat-equation solution evaluationMaximum absolute error 2.32×10−122.32\times10^{-12}Median maximum absolute error 5.06×10−35.06\times10^{-3}
Boolean sorting4,096 / 4,096 exact0 / 4,096 exact

Q2. When is this tradeoff worthwhile?

We agree that conventional calculators offer efficient, flexible general-purpose numerical evaluation. One setting where function-specific weights may be worthwhile is when a model repeatedly invokes prescribed numerical operations, since the token overhead of external tool calls can accumulate across many computations. QUILL modules could instead be accessed directly through the model's hidden representations, without generating explicit tool-call tokens. Our learned-routing experiments provide an initial demonstration of this approach, achieving 98.9–100% exact accuracy with 3.353.35–3.76×3.76\times fewer generated tokens than native tool calling (Table R2 in our common response).

The potential applications also extend beyond the operations of conventional calculators. As demonstrated in Section 4, constructed primitives can be composed into matrix operations, iterative numerical algorithms, sorting, and shortest-path computations. These results illustrate how larger computational programs can be implemented directly in standard neural architectures, rather than relying on external numerical executors.

These experiments provide an initial proof of concept for accessing reliable in-weight computation within pretrained models. More general integration remains an open research direction [5], but QUILL's construction and composition guarantees do not depend on resolving that challenge.

References

[1] Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics. International Conference on Learning Representations (ICLR), 2025. arXiv:2410.21272. ICLR Proceedings

[2] Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding Factual Recall in Transformers via Associative Memories. International Conference on Learning Representations (ICLR), 2025. arXiv:2412.06538. ICLR Proceedings

[3] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems, 35, 2022. arXiv:2202.05262. NeurIPS Proceedings

[4] Roberto Garcia, Jerry Liu, Ronny Junkins, Sabri Eyuboglu, Atri Rudra, and Christopher Ré. MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers. arXiv preprint arXiv:2607.10034, 2026. arXiv

[5] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks. International Conference on Learning Representations (ICLR), 2024. arXiv:2304.01665.

(Edit, Nov. 16: Added references to missing citations. The authors sincerely apologize for any confusion this caused.)

Thank you to the authors for their response. However, I remain unconvinced by the existing results and arguments.

The claim that "These primitives can be reused across inputs and assembled into different programs without fitting each complete function from scratch" is not sufficiently supported by the evaluations and justifications provided in the current manuscript. In particular, what is the token overhead associated with having an LM adaptively assemble these primitives?

Furthermore, the statement that "QUILL is worthwhile when a model repeatedly invokes prescribed numerical operations" does not sufficiently justify the practical advantages of QUILL over existing alternatives. Would QUILL be more token-efficient and accurate than using a calculator or specialised software for high-precision numerical computations? The manuscript also lacks a discussion and evaluation of how numerical functions are decomposed into primitives, making it difficult to assess whether QUILL offers efficiency advantages over conventional calculation tools.

Therefore, I maintain my rating of Weak Reject.

Reviewer_WgoCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes to extend MLPs with function blocks computing exact functions like x+y, x^2, c*x, which in principle can improve accuracy of the neural networks (especially LLMs) on these tasks.

Critical strengths
  • Extending neural networks with exact mathematical functions in interesting, because it can not only improve the accuracy, which was primary motivation of authors, but also interpretability.
  • The integration of the proposed module inside the LLM is indeed interesting.
Critical weaknesses
  • I do not understand, what is the motivation of approximating known algebraic functions by MLPs in Equation (3).
  • The demonstrated method works because authors have selected easy functions, like addition, multiplication, and square on a bounded interval. For this choice, the gradient is finite and everything converges. For more important functions, like division, this does not hold and everything becomes way more complicated. For example authors in [1] aims to extend neural networks with exact multiplication and division and the resulting unit is very unstable to train. Critically, this instability is not caused by chosen representation, but by the nature of function to be approximated.
  • Line 393 states: "QUILL can approximate bounded primitives such as 1/x, sin(x), and 1/√x". To my knowledge, 1/x is unbounded around zero, which causes enormous issues when QUILL would be trained as a part of NN. Similarly 1/√x has unbounded derivative.
  • Statement "QUILL can approximate bounded primitives" is obvious from the universal approximation theorem due to its form in Equation (3).
  • The selected experimental case, x^2 is very simple and naive. The method in [1] can solve this with ease. It is trivial.

[1] Heim, Niklas, Tomas Pevny, and Vasek Smidl. "Neural power units." Advances in Neural Information Processing Systems 33 (2020): 6573-6583.

Questions for authors

Generally all my questions are in critical weakness.

Reviewer_whqgOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether exact numerical computation can be built directly into the weights of ordinary neural networks, instead of relying on chain-of-thought or external tools. It proposes QUILL, a recipe for setting the weights of a one-hidden-layer tanhtanh network so that it computes a chosen smooth one-variable function (for example x2x^2, 1x\frac{1}{x}, exp) to near the limit of 64-bit floating point.

Critical strengths
  • The paper provides a clearer understanding of why high-precision computation is difficult for neural networks. In particular, it shows a tradeoff between network width, weight magnitude, and numerical precision, and how a one-hidden-layer construction can balance all three.
  • The paper proposes a single-layer construction that is very accurate, uses small weights, and works with standard computer precision.
  • The handling of rounding errors is thorough and well validated.
  • The experiments convincingly show errors around 10−1410^{-14} across many functions.
  • The observation that accurate small networks exist but normal training does not find them is interesting.
Critical weaknesses
  • The following claims in the paper are stronger than what the experimental results demonstrate.

    • Training.\textbf{Training.} The paper says it proves that gradient descent cannot learn the right network fast enough. The proof only holds under conditions checked experimentally for limited training periods, and stronger training methods are not tested.
    • Theory versus experiments.\textbf{Theory versus experiments.} The experiments use a different fitting method from the one analyzed in the theory.
  • The LLM demo doesn’t show what the paper suggests. The input number is read by hand-written code, and the answer is written back by hand-written code. The module also runs in FP64 while the model runs in BF16. So the language model is not doing any of the computation. Rather, it works more like a calculator plugged into the forward pass. Because of this, the comparison with CoT and calculator use in Table 3 is not fair.

  • Precision is a problem for LLMs. The paper’s own Figure 2c shows that accuracy drops a lot at lower precision. FP64 precision itself only holds about 15–16 digits, so it is not clear if the proposed method would work beyond 7-digit numbers.

  • It is not clear if the neural network form is needed for one-variable functions. Standard methods like Chebyshev approximation already reach this accuracy easily. The main reason given for wanting it as a network is the LLM use case, which isn’t really shown.

  • The paper states it proves gradient descent can’t learn these networks, but the result depends on conditions that are only checked experimentally for limited training runs. Some of the difficulty may simply be that the fitting problem is numerically hard. Michaud et al. (2023) trained networks close to machine precision with better optimizers, and that paper isn’t cited.

  • The paper argues construction is necessary because training fails, but only tests GD and Adam. Stronger optimizers such as L-BFGS, used by Michaud et al. (2023) to reach near machine precision, should be compared.

Questions for authors
  • What real benefit does the neural-network form give compared with standard high-precision methods run on a computer?
  • In the language-model demo, what is the worst-case error over all 3-, 5- and 7-digit inputs? What exactly does the hand-written input reader do, and does it differ between the three models?
  • How well does the module work at the language model’s own precision (16-bits)?
  • Can the method handle numbers longer than 7 digits?
  • How do stronger training methods, such as L-BFGS and the methods of Michaud et al., compare with the proposed construction?

Comments on the paper

5 comments
AuthorsOpenReview ↗

W3, W5, Q1. Motivation for MLPs and compositional computation

(edit Nov 16: fixed references. The authors apologize for any confusion.)

Q1. Motivation. We are interested in realizing computations in ordinary MLPs to enable reliable numerical computation within the same architecture used by pretrained LLMs, rather than relying solely on imperfect learned arithmetic heuristics [9], specialized numerical architectures, or external tools.

Toward this goal, QUILL demonstrates that ordinary MLP weights can implement not only high-precision functions, but also composed computations including matrix operations, sorting, and shortest-path algorithms (discussed below). Moreover, our construction is compatible with activation functions used across different LLM families. This provides a foundation for reliable, interpretable in-weight computation and the broader direction of weight-space programming (WSP).

LLM integration. We appreciate the reviewer's concerns that the hard-coded invocation and operand extraction in our original proof-of-concept limits our LLM integration demo. To address this, we introduce two additional experiments, described in our shared response: native tool calling, in which the LLM determines when to invoke QUILL and supplies its operands, and learned internal routing, which selects invocation and operands directly from frozen LLM representations without generating explicit tool-call tokens. These experiments provide stronger evidence that constructed numerical capabilities can be accessed within pretrained models, beyond the engineered interface in our original demonstration.

W3, W5. Computational breadth and comparison with conventional methods. We thank the reviewer for highlighting established high-precision numerical methods, including Chebyshev approximation and conventional solvers. We agree that these methods are effective for numerical evaluation; our objective is to establish comparable precision in ordinary MLP weights, together with guarantees on numerical realizability and composition.

Importantly, QUILL's scope extends beyond the operations of conventional calculators. Section 4 establishes end-to-end accuracy and network-resource guarantees for compiling bounded polynomial arithmetic circuits from reusable QUILL primitives. This enables multivariate computations such as matrix operations and determinants. With additional constructed primitives, we also demonstrate iterative numerical algorithms, including Newton iteration and quadrature, and discrete programs such as sorting and shortest paths. These results illustrate how entire computational programs, rather than just scalar function approximations, can be implemented in ordinary neural-network weights.

W6, W7 To demonstrate the distinction between constructing these computations and learning them end-to-end, we introduce additional experiments comparing QUILL against training of the same composed architectures. Across determinants, matrix inversion, normal CDF evaluation, and heat-equation solution evaluation, constructed programs attain maximum errors between 4.25×10−134.25\times10^{-13} and 8.48×10−118.48\times10^{-11}, compared with 1.15×10−31.15\times10^{-3} to 8.78×10−38.78\times10^{-3} for the tested Adam→L-BFGS training procedure (Table R5).

Table R5. Constructed programs versus end-to-end training of the same architectures.

ComputationParametersConstructed maximum errorTrained maximum error (Adam→L-BFGS)
2×22\times2 determinant6414.25×10−134.25\times10^{-13}5.42×10−35.42\times10^{-3}
2×22\times2 matrix inverse9392.53×10−122.53\times10^{-12}6.56×10−36.56\times10^{-3}
Normal CDF7308.48×10−118.48\times10^{-11}1.15×10−31.15\times10^{-3}
Heat-equation solution1,3975.27×10−125.27\times10^{-12}8.78×10−38.78\times10^{-3}

W3 / Q3. Worst-case error and hard-coded components. QUILL provably achieves 100% accuracy on the supported 3-, 5-, and 7-digit domains. To further validate, we evaluate all 9,090,900 positive 3-, 5- and 7-digit integer with 100% accuracy (Table R6).

Table R6. Exhaustive numerical accuracy of the original QUILL squaring module.

Input lengthInputs evaluatedExact QUILL outputsWorst-case integer error
3 digits900900/9000
5 digits90,00090,000/90,0000
7 digits9,000,0009,000,000/9,000,0000
Overall9,090,9009,090,900/9,090,9000

While QUILL produces the correct integer result throughout these domains, errors can still arise during LLM invocation, operand selection, or answer generation. We evaluate the LLM integration methods on 750 prompts per model (250 each for 3-, 5-, and 7-digit squaring), with the results shown in Table R2.

Table R7. LLM integration accuracy and generated-token cost on 3-, 5- and 7-digits.

ModelMethod3 digits5 digits7 digitsOverallMean generated tokens
Qwen2.5-7BModel alone†79.2%7.2%4.8%30.4%9.87
Qwen2.5-7BChain of thought†79.2%28.4%9.6%39.1%299.07
Qwen2.5-7BNative calculator tool100%100%100%100%35.89
Qwen2.5-7BNative QUILL tool100%100%100%100%35.89
Qwen2.5-7BQUILL + learned router100%99.6%97.2%98.9%9.54
Llama-3.1-8BModel alone†23.6%2.0%0.4%8.7%5.50
Llama-3.1-8BChain of thought†35.2%4.8%0.4%13.5%375.33
Llama-3.1-8BNative calculator tool96.0%98.4%99.2%97.9%32.91
Llama-3.1-8BNative QUILL tool96.0%98.4%99.2%97.9%32.91
Llama-3.1-8BQUILL + learned router100%100%100%100%9.82

† Model-alone and chain-of-thought results use the original single-template evaluation. Native-tool and learned-router results use the newer five-family evaluation (direct questions, paraphrases, word problems, numerical distractors, and comma-formatted inputs). All conditions use 250 examples per digit length.

W2, W3 / Q4, Q5. Beyond seven digits and BF16. To support operands longer than seven digits, we build a circuit: we compose multiple applications of the same bounded QUILL primitive, analogous to how arbitrary-precision libraries combine fixed-width operations. We evaluate 50 examples each at 16, 20, and 32 digits, achieving 100% exact accuracy across all 150 tested cases.

We can also support operands of arbitrary length in bf16 by composing similar circuits. BF16 weights and activations, the composed circuits likewise achieve 100% exact accuracy across 150 evaluated cases at 16, 20, and 32 digits. The same primitive weights are reused across all operand lengths, so longer operands do not require constructing new function-specific weights, although the number of operations increases.

References:

[1] Eric J. Michaud, Ziming Liu, and Max Tegmark. Precision Machine Learning. Entropy, 25(1):175, 2023. https://doi.org/10.3390/e25010175

[2] Yongji Wang et al. Discovery of Unstable Singularities. arXiv:2509.14185, 2025. https://arxiv.org/abs/2509.14185

[3] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. 2nd edition, Springer, 2006. https://doi.org/10.1007/978-0-387-40065-5

[4] Dong C. Liu and Jorge Nocedal. On the Limited Memory BFGS Method for Large Scale Optimization. Mathematical Programming, 45:503–528, 1989. https://doi.org/10.1007/BF01589116

[5] Elham Kiyani, Khemraj Shukla, Jorge F. Urbán, Jérôme Darbon, and George Em Karniadakis. Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks. Computer Methods in Applied Mechanics and Engineering, 446:118308, 2025. https://doi.org/10.1016/j.cma.2025.118308

[6] Keller Jordan. Muon: An Optimizer for Hidden Layers in Neural Networks. 2024. https://kellerjordan.github.io/posts/muon/

[7] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: Improving and Stabilizing Shampoo Using Adam. NeurIPS 2024 Workshop on Optimization for Machine Learning, 2024. https://arxiv.org/abs/2409.11321

[8] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR), 2015. https://arxiv.org/abs/1412.6980

[9] Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics. International Conference on Learning Representations (ICLR), 2025. https://arxiv.org/abs/2410.21272

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.