acceptodds
Under review as a conference paper at ICLR 2027

What Accuracy Hides In Learned Transformer Programs

Abstract

Transformer Programs (TP) (Friedman et al., 2023) constrain a Transformer’s architecture so that its learned computation can be extracted as readable Python code. TP expose learned computations, but token accuracy can hide incorrect complete answers. In this work, we examine how program inspection provides insights beyond accuracy scores and helps guide training. We first evaluated TP on nine algorithmic tasks. We compared token accuracy with accuracy on complete output sequences and checked for failures missed by sampled tests. Further, we assessed training data improvements informed by program inspection and failure analysis, finding gains in sequence accuracy on several tasks. We examined a counting task where a program achieved 100% accuracy on sampled test sequences but failed on another input using the symbols and lengths allowed during training. The program counted correctly, but the required answer was missing from its output labels. After adding training examples with larger counts, the retrained models passed exhaustive evaluation over the bounded domain. In the frequency ranking task, inspection showed how correct counts contributed to incorrect answers in programs trained before and after changes to the training data. Finally, we studied constraints imposed by TP training and operations. The TP training process can lead to accuracy losses late in training. Our reference constructions also show how composing the restricted TP operations can use many layers or large lookup tables.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, mechanistic interpretability, transformer programs, formal verification, program extraction, interpretability by construction, SMT solving, algorithmic tasks, RASP, correctness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DUsGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigate an interesting topic with transformer programming, with three claimed contributions: a. they found that accuracy is not reliable as even an accuracy (token & sentence) is high, it does not mean the program has been learned; b. they add data to guide this training; c. they take histogram as an example.

Critical strengths
  1. They have trained the model and evaluated it on 9 tasks, which builds the benchmark of the TP explainability.
  2. They provide useful insight from case study that it could be due to the classifier available labels and wrong logic when submitting the rank after correct frequency calculation.
Critical weaknesses
  1. Motivation: The paper directly jumped into the TP error analysis, with very little related work discussion, the inspiration of the work, and contribution over previous work, which is the core issue of the paper writing and hinder the understanding of the paper. There are a lot to be compared such as: https://arxiv.org/abs/2301.05062v3; https://arxiv.org/abs/2301.05062

  2. Novelty: I understand that it would be hard to argue against novelty but essentially this work is providing insights from analysis perspective. The first contribution is quite intuitive as a large number of explainable AI work stated; the second contribution is simply the data augmentation skills without neither a. large dataset creation as contribution nor b. out-of-domain test results. You are creating dataset specifically for an error you spotted, and it should work; the third contribution is not summarised as general findings but only comes out of limited experiments. The validation only N<8. It provides limited scalable insights for future work.

  3. Experiments: a. all the tasks are math related or small tasks where the introduction and scope statements did not limit them, and the original TP paper has entity recognition etc deep learning tasks b. the non-attributed improvements: the paper did a lot improvements for example on frequency but we do not know where the gains come from c. it should discuss whether the failure modes found in one task would be generalised to another; d. it should prove at least your findings is a general case but not summarised from a single case.

  4. Language: There are a lot of academic writing confused description such as line 23: After adding training examples with larger counts'', what means larger counts (solely from abstract context here?), line 42: Having the code available allows us to understand the underlying computations'' - underlying computations?

Overall, I do not think it has significant findings in its current form and it is out of the scope of minor revision to be accepted in this round, so I recommend clear reject.

Questions for authors

See as above.

6 comments

Thank you for recognizing the evaluation across nine tasks and the diagnostic value of the histogram and frequency ranking examples. We appreciate your assessment of these case studies. We will distinguish their specific findings from claims about broader applicability or recovery of complete algorithms.

Motivation

  • Thank you for pointing out that the motivation and relationship to prior work need clearer development before the error analysis.
  • Tracr is relevant to this framing. It compiles supplied RASP programs into Transformers with known computational structure, whereas TP learns from examples and exposes the resulting computation as a program. Our study examines whether those learned programs satisfy their task specifications and what inspection reveals when they fail.
  • We will expand the discussion of RASP, Tracr, the original TP paper and subsequent work. The revised introduction will explain what our analysis adds, rather than presenting access to executable code or program based debugging as new contributions.

Novelty

  • We agree that familiar limitations of accuracy are not, by themselves, novel findings. Our contribution needs to be stated through the specific evidence obtained from learned TP artifacts.
  • We do not propose a new data augmentation technique or claim a dataset contribution. The repair experiments examine whether changes motivated by observed failures produce correctness beyond the motivating examples, within a defined bounded domain.
  • The outcome is not uniformly positive. Histogram repair succeeds across five hard neural models in the reported bounded evaluation, while sorting enrichment can remove targeted failures and increase the overall failure count. We will give both outcomes appropriate prominence.
  • We agree that these results do not establish generalization beyond the checked domains. Some checks extend through total length eight inclusive, but the broader limitation of small verification domains remains.
  • We will sharpen the takeaways from the training and reference construction experiments without presenting them as universal constraints. In particular, the costs of our constructions are not proven architectural lower bounds.
Reviewer_CWzuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates transformer programs to understand what qualities high scores could hide. Specifically, they aim to evaluate the robustness of these programs on sequences outside of the testing set but still valid under the training regime. On nine algorithmic tasks, the paper compares token accuracy with complete answer accuracy. They then use formal verifiers to find counterexamples. Across both, they find that token accuracy can mask incomplete answers that fail edge cases.

They use this diagnosis to guide interventions in the training data. On their histogram task, they found that adding in these examples can produce new models that are robust and fully correct. On other tasks, the benefits did not produce fully correct programs, but they still show how correct counts contribute to incorrect answers. They conclude with an analysis of the constraints on these programs and their training. They note that accuracy can decline as one moves toward discrete choices.

Critical strengths
  1. It is attempting to address an important question overall. Testing whether improvements actually produce reliable programs is a useful artifact to the field they are studying.
  2. The histogram result is strong and reproducible. All five retrained models achieved full correctness, whereas the strong baseline did not. Very clear and objective evidence in favor of their failure analysis and enrichment strategy.
  3. Their analysis does expose significant shortcomings between the reported scores and the actual programs being produced. This finding by itself is useful if it extends beyond the nine tasks they examine.
Critical weaknesses
  1. The paper is very poorly laid out and organized. Everything is all over the place, and it took me several careful readings to understand what was happening. The section organization feels more like a novel than scientific literature conveying an idea. There is no organization the hierarchy of contributions either. In many cases (like Table 4 on Page 8), essential information is separated from the results.
  2. The original transformer programs paper had already highlighted the use of program-based debugging while recognizing the accuracy shortcomings. This paper does the same, highlighting little to no new insight on this front. The Adaptive Transformer Programs comparison could have filled this gap but was not explored enough to justify.
  3. The overall frequency results mix two different scoring systems that do not seem to make a fair comparison. Further, enriching the training data changes the overall examples, so the comparisons are not fair. The authors do acknowledge this, but it is still a major weakness.
Questions for authors
  1. What is the baseline vs. enriched model's final score for soft, hard, and extracted Python scores on identical inputs? If you controlled the training budgets, would the results change?
  2. How consistently does repairing produce a correct artifact after execution?
  3. What are the mean/std across the seeds for each row in tab 4? and for how many tasks does the mean improve over the baseline mean?
7 comments

Thank you for recognizing the value of checking whether training improvements produce reliable programs, and for highlighting the histogram result. We appreciate your distinction between useful findings on the tasks studied and evidence of broader applicability. We will keep the bounded scope of the correctness results explicit.

Organization

  • Thank you for explaining where the presentation made the paper difficult to follow. We agree that readers should not have to reconstruct the experimental conditions from information scattered across sections.
  • We will reorganize the paper around its main questions and distinguish the contributions of evaluation, computational diagnosis and training interventions.
  • For Table 4, we will place the evaluated artifact, scoring convention and selection details alongside the results.

Prior Work

  • We agree that program based debugging is already demonstrated in the original Transformer Programs paper. Our contribution should not be framed as introducing that capability.
  • We will make the comparison more specific, explaining what our diagnoses and bounded repair checks add beyond the earlier examples.
  • Our ATP experiment is limited to the tested frequency configuration and implementation. It does not support a broad conclusion about ATP. We will qualify that comparison and distinguish its limited evidence from the main findings.
Reviewer_7g8EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Transformer Programs (TP) which constrain a Transformer model so that its learned computation can be extracted as a Python code. Across nine algorithmic tasks, it reveals that the commonly used token accuracy hides certain failures. Sequence accuracy is often much lower, and even perfect accuracy on sampled tests can miss counterexamples.

Two case studies trace failures through the extracted code. In histogram, a program counts correctly but cannot output the required count because that label never appeared in training data. Adding such examples yields models that pass exhaustive checks. In frequency ranking, correct counts lead to wrong answers through the classifier scores, both before and after data enrichment.

The paper also reports accuracy loss during annealing toward discrete choices, large variance across seeds, and the cost of hand-written reference constructions.

Critical strengths
  1. The paper is an unusually careful and transparent evaluation.

  2. It distinguishes soft models, hard models, and extracted programs, and confirms every counterexample by execution. It candidly reports seed variance, selection effects, and confounds.

  3. The histogram case, where a program computes the right quantity but lacks the output label, clearly shows why accuracy on sampled tests is not enough. This makes the paper a trustworthy and useful advice for anyone building on Transformer Programs.

Critical weaknesses
  1. The main thesis, that inspecting learned programs guides better training, is not demonstrated.

  2. The proposed insights are limited in novelty and scope for ICLR. The facts such as 'token accuracy overstates sequence accuracy', 'sampled tests miss rare failures', 'annealing toward discrete tokens can lose accuracy' are already known or easily expected.

  3. The interpretability analysis explain individual outputs through score accounting rather than recovering the learned algorithms.

Questions for authors
  1. Is there any case where program inspection led to a training change that the counterexamples alone would not have suggested? Such a case would directly support the paper's central claim.

  2. How were the reported best models selected? How do the main results look as seed averages under a single consistent metric, given that Table 4 mixes soft-model scores with hard-model and program scores?

  3. Do the extracted Python programs of the retrained models pass the same exhaustive checks as the hard neural models? The paper notes that the two cannot be assumed to agree.

7 comments

Thank you for recognizing the care and transparency of our evaluation, including the distinctions between soft models, hard models and extracted programs, and the execution checks on counterexamples. We appreciate your assessment of the histogram example. We will preserve its concrete diagnosis and the explicit reporting of seed variation, selection effects and confounds as we revise the paper.

Main Thesis

  • Thank you for pointing out the gap between our framing and the evidence. Our experiments do not isolate the additional contribution of program inspection to the training improvements.
  • For histogram, inspection shows that the highlighted program counts correctly but lacks the required output label. However, the counterexamples themselves also suggest adding examples with larger counts. For frequency ranking, program analysis happened after the training changes and did not guide them.
  • We will revise the abstract, contribution statement and conclusion to distinguish the explanations obtained through inspection from the improvements obtained through failure informed training changes.

Novelty And Scope

  • We agree that the general observations listed here are established and should not be presented as new discoveries.
  • Our intended contribution is the concrete analysis of learned TP artifacts, including a program that counts correctly but cannot express the required answer, and bounded checks of whether retraining removes failures beyond the motivating examples.
  • We will clarify what these findings add to the original TP paper and subsequent work. We will also make the restricted scope explicit, rather than suggesting that results on these small algorithmic tasks establish broader conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.