- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies whether reduced-precision execution preserves the intended computation of compiled transformers. It introduces CRAFT, a compiler for LU-based triangular solves, together with a certified-exact lookup oracle for auditing address precision. CRAFT allows the authors to distinguish incorrect address selection from errors in the selected values and subsequent arithmetic. Experiments show that protecting the address path with higher precision while keeping the value path in low precision can restore selector correctness, although value-path errors remain.
Critical strengths
My background is in compilers and floating-point semantics, not compiled transformers or mechanistic interpretability, so I may not be the best person to judge the broader importance of these results and significance of the problem that the paper aims to solve. It seems natural to me that converting real numbers to floating point and using lower precision can cause errors when selecting addresses. Even when some computations use higher precision, rounding errors can build up and still produce value errors. However, I appreciate that the paper provides concrete experiments showing this clearly. I also like its clear diagnosis-and-repair approach, useful experimental controls, and appropriately limited claims.
Critical weaknesses
It is unclear how well the approach applies beyond the programs studied. The paper mainly studies fixed-matrix triangular solves (LU-triangular-solve) with a fixed execution order and a specific quadratic addressing method, as mentioned in Section 6. The paper does mention these limits, but it is still hard to tell which parts of the auditing method would work for other compiled programs, such as programs with input-dependent branches or loops whose number of iterations depends on the input. With these, it'll be challenging to provide such a lookup oracle since the compiler has no ability to predict the bound of a loop and no ability to totally unroll a loop. It would help to clearly separate the assumptions needed for the general auditing method from those that are specific to CRAFT. This would make it easier to understand how broadly the approach can be used.
The explanation could be easier to follow. As someone outside the compiled-transformer field, I found it hard to understand how the source program, the transformer operations produced by the compiler, and the certified lookup oracle connect to one another. A small, complete example would help to explain how Tracr compiles an RASP program, such as showing how one source-level operation is compiled, how lower precision changes the result, and how the oracle finds the resulting error.
Details about the oracle's arithmetic could be clearer. The oracle mentions at Lines 180–186 describe a float64 calculation that compares the winning margin with a rounding-error limit. If the result is unclear, the oracle uses Fraction instead. Could the authors explain the assumptions behind the float64 calculation and why? In particular, what rounding mode is used, and does the two-term dot product use separate operations or fused multiply-add? It would also help to say whether overflow and underflow are impossible by design or handled by the oracle. These details will influence the computed scores and margins.
Questions for authors
See the weakness part.