acceptodds
Under review as a conference paper at ICLR 2027

Precision Audits for Compiled Transformers with Source-Level Ground Truth

Abstract

Compiled transformers provide source-level ground truth for a numerical/compiler audit: does reduced-precision execution still compute the intended program? In CRAFT, an LU-triangular-solve compiler, a certified-exact lookup oracle reveals that low precision can preserve finite outputs and hard attention while fetching the wrong source value. Across 142 executing programs and 359,202 emitted rows, no row fails above 17 tested arithmetic bits. Full bf16 breaks 105/142 programs; protecting the address path at every tested width of 18 bits or above restores selector correctness while leaving value error. The arithmetic-only sufficient rule is approximately bits; storage rounding adds a separate term. A microbenchmark reaches and shows two additional bits per address doubling. Binary codes and direct gather delimit this representation-specific effect. Native Apple-GPU fp32-address/bf16-value runs also restore selectors on three frontier programs. In a public Tracr case, official ACDC recovers the full known graph despite int8 errors on 232/256 inputs; additional ACDC and trained controls remain correct. These results support source-level numerical auditing of compiled artifacts, without establishing universal fragility, compression benefits, or generalization to trained transformers.

Primary area:
interpretability and explainable AI
Keywords:
compiled transformers, numerical stability, mixed-precision arithmetic, mechanistic interpretability, floating-point arithmetic, source-level ground truth

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NJZdOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether reduced-precision execution preserves the intended computation of compiled transformers. It introduces CRAFT, a compiler for LU-based triangular solves, together with a certified-exact lookup oracle for auditing address precision. CRAFT allows the authors to distinguish incorrect address selection from errors in the selected values and subsequent arithmetic. Experiments show that protecting the address path with higher precision while keeping the value path in low precision can restore selector correctness, although value-path errors remain.

Critical strengths

My background is in compilers and floating-point semantics, not compiled transformers or mechanistic interpretability, so I may not be the best person to judge the broader importance of these results and significance of the problem that the paper aims to solve. It seems natural to me that converting real numbers to floating point and using lower precision can cause errors when selecting addresses. Even when some computations use higher precision, rounding errors can build up and still produce value errors. However, I appreciate that the paper provides concrete experiments showing this clearly. I also like its clear diagnosis-and-repair approach, useful experimental controls, and appropriately limited claims.

Critical weaknesses

It is unclear how well the approach applies beyond the programs studied. The paper mainly studies fixed-matrix triangular solves (LU-triangular-solve) with a fixed execution order and a specific quadratic addressing method, as mentioned in Section 6. The paper does mention these limits, but it is still hard to tell which parts of the auditing method would work for other compiled programs, such as programs with input-dependent branches or loops whose number of iterations depends on the input. With these, it'll be challenging to provide such a lookup oracle since the compiler has no ability to predict the bound of a loop and no ability to totally unroll a loop. It would help to clearly separate the assumptions needed for the general auditing method from those that are specific to CRAFT. This would make it easier to understand how broadly the approach can be used.

The explanation could be easier to follow. As someone outside the compiled-transformer field, I found it hard to understand how the source program, the transformer operations produced by the compiler, and the certified lookup oracle connect to one another. A small, complete example would help to explain how Tracr compiles an RASP program, such as showing how one source-level operation is compiled, how lower precision changes the result, and how the oracle finds the resulting error.

Details about the oracle's arithmetic could be clearer. The oracle mentions at Lines 180–186 describe a float64 calculation that compares the winning margin with a rounding-error limit. If the result is unclear, the oracle uses Fraction instead. Could the authors explain the assumptions behind the float64 calculation and why? In particular, what rounding mode is used, and does the two-term dot product use separate operations or fused multiply-add? It would also help to say whether overflow and underflow are impossible by design or handled by the oracle. These details will influence the computed scores and margins.

Questions for authors

See the weakness part.

Reviewer_FFeMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines whether reduced-precision execution preserves the intended lookup operations of compiled transformers. Using CRAFT, the authors define a certified lookup oracle, derive a sufficient precision bound for quadratic addressing, and audit the effects of low precision, quantization, and pruning. They show that finite outputs and sharp attention can coexist with incorrect lookups. Protecting the address path restores selector correctness but does not eliminate numerical errors in values. Binary position codes and direct gather provide alternative implementations. The paper also uses Jacobi, Tracr, ALTA, trained-model controls, serialized checkpoints, and native GPU execution to test the precision findings and their limitations.

Critical strengths

This paper uses a certified lookup oracle to check whether attention selects the correct source position. This is more direct than relying only on output accuracy or attention patterns, and helps identify numerical errors that common metrics may miss. They provide both theoretical analysis and extensive experiments to support the precision bound.

Critical weaknesses

The main precision bound is specific to CRAFT's quadratic addressing scheme. Binary codes and direct gather avoid this issue in the tested settings, while trained models show different behavior. It is still unclear how well the findings apply to other compiled or trained transformers.

The practical benefit of the precision repair is not so clear. Uniform fp32 already preserves correct selectors, while the mixed-precision checkpoint uses more memory than fp32. Numerical errors also remain in the value path, and the paper does not show a clear advantage in memory or runtime.

Some findings about evaluation metrics seem relatively expected. Sharp attention does not necessarily mean correct selection, and ACDC focuses on circuit structure rather than numerical correctness. The experiments provide useful examples, but it is unclear how much new insight they add beyond these known limitations.

Questions for authors
  1. How does the certified oracle verify that the exact score winner matches the intended source-level lookup index?

  2. Can the proposed precision repair provide any memory or runtime advantage over uniform FP32 while preserving selector correctness?

  3. Can the Tracr score-path repair work without using scores from a clean shadow model?

Reviewer_5w6WOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

NOTE: The paper is very hard to follow: many abbreviations and internal script names (p26, p37, p54, …; int8T/int8C; pr10 nz) are never defined, the central testbed CRAFT is neither cited nor described, key terms (“address island”, “selector-hit signature”, “false green”, “live prefix”) are used before they are explained, and the main text is extremely dense while deferring essential definitions to 22 appendices.

Based on the paper, this is what I understand as the paper summary: The paper is a numerical audit of compiled transformers: given a program compiled into weights with known operation labels, it measures how many significand bits are needed to keep the emitted attention addresses (hard-argmax lookups) correct, and shows which standard diagnostics stay “green” once they are not.

Critical strengths

Reproducibility effort

Critical weaknesses
  1. The paper is extremely dense and defensively written: undefined abbreviations and internal script names (p26, p37, p54, …), terms used before they are defined (“address island”, “selector-hit signature”, “false green”), and essential definitions deferred to 22 appendices.

  2. CRAFT is never described or attributed. Every result is a property of this one compiler, yet it is neither cited nor listed as a contribution. It is unclear whether CRAFT is introduced in this paper, is the authors’ prior work (no reference given), or is an unpublished framework — and whether it relates to the craft layer of Tracr, which shares the name.

  3. Narrow object of study. The testbed is a hard-attention toy compiler for ≤44-variable triangular solves on 154 synthetic netlists, by the authors’ own account “five orders of magnitude slower than a library call.” The paper’s own Tracr/ALTA/trained controls show the bit law does not transfer beyond this gadget, leaving the auditing protocol as the main takeaway.

Questions for authors

I do not have question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.