- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigate an interesting topic with transformer programming, with three claimed contributions: a. they found that accuracy is not reliable as even an accuracy (token & sentence) is high, it does not mean the program has been learned; b. they add data to guide this training; c. they take histogram as an example.
Critical strengths
- They have trained the model and evaluated it on 9 tasks, which builds the benchmark of the TP explainability.
- They provide useful insight from case study that it could be due to the classifier available labels and wrong logic when submitting the rank after correct frequency calculation.
Critical weaknesses
-
Motivation: The paper directly jumped into the TP error analysis, with very little related work discussion, the inspiration of the work, and contribution over previous work, which is the core issue of the paper writing and hinder the understanding of the paper. There are a lot to be compared such as: https://arxiv.org/abs/2301.05062v3; https://arxiv.org/abs/2301.05062
-
Novelty: I understand that it would be hard to argue against novelty but essentially this work is providing insights from analysis perspective. The first contribution is quite intuitive as a large number of explainable AI work stated; the second contribution is simply the data augmentation skills without neither a. large dataset creation as contribution nor b. out-of-domain test results. You are creating dataset specifically for an error you spotted, and it should work; the third contribution is not summarised as general findings but only comes out of limited experiments. The validation only N<8. It provides limited scalable insights for future work.
-
Experiments: a. all the tasks are math related or small tasks where the introduction and scope statements did not limit them, and the original TP paper has entity recognition etc deep learning tasks b. the non-attributed improvements: the paper did a lot improvements for example on frequency but we do not know where the gains come from c. it should discuss whether the failure modes found in one task would be generalised to another; d. it should prove at least your findings is a general case but not summarised from a single case.
-
Language: There are a lot of academic writing confused description such as line 23:
After adding training examples with larger counts'', what means larger counts (solely from abstract context here?), line 42:Having the code available allows us to understand the underlying computations'' - underlying computations?
Overall, I do not think it has significant findings in its current form and it is out of the scope of minor revision to be accepted in this round, so I recommend clear reject.
Questions for authors
See as above.
6 comments
Motivation
- Thank you for pointing out that the motivation and relationship to prior work need clearer development before the error analysis.
- Tracr is relevant to this framing. It compiles supplied RASP programs into Transformers with known computational structure, whereas TP learns from examples and exposes the resulting computation as a program. Our study examines whether those learned programs satisfy their task specifications and what inspection reveals when they fail.
- We will expand the discussion of RASP, Tracr, the original TP paper and subsequent work. The revised introduction will explain what our analysis adds, rather than presenting access to executable code or program based debugging as new contributions.
Novelty
- We agree that familiar limitations of accuracy are not, by themselves, novel findings. Our contribution needs to be stated through the specific evidence obtained from learned TP artifacts.
- We do not propose a new data augmentation technique or claim a dataset contribution. The repair experiments examine whether changes motivated by observed failures produce correctness beyond the motivating examples, within a defined bounded domain.
- The outcome is not uniformly positive. Histogram repair succeeds across five hard neural models in the reported bounded evaluation, while sorting enrichment can remove targeted failures and increase the overall failure count. We will give both outcomes appropriate prominence.
- We agree that these results do not establish generalization beyond the checked domains. Some checks extend through total length eight inclusive, but the broader limitation of small verification domains remains.
- We will sharpen the takeaways from the training and reference construction experiments without presenting them as universal constraints. In particular, the costs of our constructions are not proven architectural lower bounds.
Thank you for recognizing the evaluation across nine tasks and the diagnostic value of the histogram and frequency ranking examples. We appreciate your assessment of these case studies. We will distinguish their specific findings from claims about broader applicability or recovery of complete algorithms.