- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper propose a method for encoding bounded numerical functions in ordinary MLP weights. The proposed method can assist LM to achieve better accuracy with lower token cost in numerical functions computation compared to traditional solutions.
Critical strengths
- The proposed method can assist LM to achieve better accuracy with lower token cost in numerical function computations compared to using caculate, direct prompt and Chain of Thought.
- This method enables the MLP component to achieve a lower error whilst reducing the scale of hidden neurons.
Critical weaknesses
The proposed method encodes the target bounded numerical function into the weights of a standard MLP, which appears to imply that QUILL must prepare specific MLP weights for different target numerical functions. Although table 3 has demonstrated that the proposed approach can improve accuracy and reduce token cost in computation, the sacrifice of flexibility in general use cases does not convince me to use QUILL to replace a calculator.
Questions for authors
I appreciate the author's efforts in this work and would kindly ask authors to clarify the following questions:
- Does QUILL require to prepare different MLP weights for different numerical functions?
- If so, please clarify in which use cases it is worthwhile to sacrifice flexibility in exchange for improved accuracy and reduced token cost.
7 comments
2. Responses to the reviewer's questions
The construction, composition guarantees, and training analysis above form the core of QUILL’s contributions to understanding and improving numerical computation in MLPs. We now address the reviewer’s questions about the flexibility of function-specific weights and the practical settings in which these constructions are useful.
Q1. Does QUILL require different weights for different numerical functions?
Yes. QUILL constructs weights for a prescribed primitive and domain. These primitives can be reused across inputs and assembled into different programs without fitting each complete function from scratch. For example, squaring supplies multiplication (Equation (10)), supporting dot products, determinants, and other polynomial computations. Theorem 4.1 establishes end-to-end accuracy and network-resource guarantees for bounded polynomial arithmetic circuits assembled from these primitives.
Our expanded experiments further demonstrate construction of larger continuous and discrete computations, including heat-equation solution evaluation and Boolean sorting (Table R1). Constructed QUILL achieves a maximum absolute error of on the heat-equation task, compared with for the trained baseline. For Boolean sorting, QUILL produces exact outputs on all 4,096 evaluated cases, whereas end-to-end training of the same architecture produces none under the tested optimization settings.
Table R1. Accuracy of constructed and end-to-end-trained numerical and discrete programs.
| Computation | Constructed QUILL | Same architecture, trained |
|---|---|---|
| Heat-equation solution evaluation | Maximum absolute error | Median maximum absolute error |
| Boolean sorting | 4,096 / 4,096 exact | 0 / 4,096 exact |
Q2. When is this tradeoff worthwhile?
We agree that conventional calculators offer efficient, flexible general-purpose numerical evaluation. One setting where function-specific weights may be worthwhile is when a model repeatedly invokes prescribed numerical operations, since the token overhead of external tool calls can accumulate across many computations. QUILL modules could instead be accessed directly through the model's hidden representations, without generating explicit tool-call tokens. Our learned-routing experiments provide an initial demonstration of this approach, achieving 98.9–100% exact accuracy with – fewer generated tokens than native tool calling (Table R2 in our common response).
The potential applications also extend beyond the operations of conventional calculators. As demonstrated in Section 4, constructed primitives can be composed into matrix operations, iterative numerical algorithms, sorting, and shortest-path computations. These results illustrate how larger computational programs can be implemented directly in standard neural architectures, rather than relying on external numerical executors.
These experiments provide an initial proof of concept for accessing reliable in-weight computation within pretrained models. More general integration remains an open research direction [5], but QUILL's construction and composition guarantees do not depend on resolving that challenge.
References
[1] Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics. International Conference on Learning Representations (ICLR), 2025. arXiv:2410.21272. ICLR Proceedings
[2] Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding Factual Recall in Transformers via Associative Memories. International Conference on Learning Representations (ICLR), 2025. arXiv:2412.06538. ICLR Proceedings
[3] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems, 35, 2022. arXiv:2202.05262. NeurIPS Proceedings
[4] Roberto Garcia, Jerry Liu, Ronny Junkins, Sabri Eyuboglu, Atri Rudra, and Christopher Ré. MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers. arXiv preprint arXiv:2607.10034, 2026. arXiv
[5] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks. International Conference on Learning Representations (ICLR), 2024. arXiv:2304.01665.
(Edit, Nov. 16: Added references to missing citations. The authors sincerely apologize for any confusion this caused.)
Thank you to the authors for their response. However, I remain unconvinced by the existing results and arguments.
The claim that "These primitives can be reused across inputs and assembled into different programs without fitting each complete function from scratch" is not sufficiently supported by the evaluations and justifications provided in the current manuscript. In particular, what is the token overhead associated with having an LM adaptively assemble these primitives?
Furthermore, the statement that "QUILL is worthwhile when a model repeatedly invokes prescribed numerical operations" does not sufficiently justify the practical advantages of QUILL over existing alternatives. Would QUILL be more token-efficient and accurate than using a calculator or specialised software for high-precision numerical computations? The manuscript also lacks a discussion and evaluation of how numerical functions are decomposed into primitives, making it difficult to assess whether QUILL offers efficiency advantages over conventional calculation tools.
Therefore, I maintain my rating of Weak Reject.
edit nov 16: mistakenly duplicated response; please ignore this comment
We thank the reviewer for recognizing QUILL's numerical accuracy, compact MLP construction, and reduced token cost in our language-model experiments. We appreciate the reviewer's questions about flexibility and practical applications, and would like to clarify how the LLM experiments relate to the paper's broader contributions before addressing them.
1. Clarifying QUILL's contributions
QUILL's primary contribution is a constructive framework for reliable numerical computation in ordinary MLP weights. Our goal is to understand what numerical computations can be explicitly implemented in standard neural-network architectures, rather than to develop a replacement for conventional calculators. The LLM experiments in Section 5 provide an initial proof of concept for accessing these capabilities within pretrained models, and our shared response further discusses the broader motivation for studying in-weight computation.
The theoretical contribution extends beyond approximating individual numerical functions. For the stated analytic target class, QUILL jointly establishes geometric convergence, numerical realizability in standard floating-point arithmetic, and ordinary single-hidden-layer MLP form (Theorem 3.1; Table 1). Our experiments achieve relative L2 errors near 10−14 in FP64. We also establish how reusable primitives can be composed into bounded polynomial arithmetic circuits, with end-to-end accuracy and resource guarantees (Theorem 4.1).
The construction also provides a reference for studying numerical capabilities learned by neural networks. Prior work finds that pretrained language models perform arithmetic through collections of MLP heuristics [1]. Our controlled shallow-MLP experiments and analysis investigate why gradient-based training can struggle to recover the high precision attainable through explicit construction (Theorems 3.2–3.3). This is a separate scientific motivation for studying numerical computation in ordinary MLPs.
We agree that conventional calculators provide efficient, flexible general-purpose numerical evaluation. However, QUILL is not intended to replace them. Its primary value lies in establishing a constructive framework for precise computation within ordinary neural-network weights, with explicit size and error guarantees.