- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper considers the problem of extracting LTL formulas from hard attention transformers. Previous translations from transformers to LTL produce very large formulas, and previous work has not attempted the extraction. This works proposed an encoding of transformers into LTL that can be “unfolded” from top to bottom, focusing on the outputs which are of interest (eg those which capture a “sufficient reason” for acceptance of a string). Experiments are run on extracting formulas from small transformers trained on synthetic data.
Critical strengths
Novelty
Much theoretical work has discussed the possibility of extracting formulas from transformers, but they never actually did it. Doing this for the first time is quite interesting, and the successes/failures will be relevant for future work.
Rigor
The paper’s methodology is rigorous. The extraction of formulas is exact, unlike other kinds of interpretability methods. Also, the proofs/concepts are presented in a rigorous and readable style — a major strength of papers.
Critical weaknesses
Unclear empirical analysis
Several important details are unclear which renders the conclusions of the experiments slightly weaker. First, the test accuracy is never >90%. It’s unclear whether this was a design choice for the experiments or a limitation of training using hard attention. Second, analysis of the sufficient reason is lacking. In particular they are motivated for being more compact and efficient than extracting the whole network, but we need to know to what are the tradeoffs.
Questions for authors
Questions
- Was the suboptimal prediction accuracy form the transformers to be extracted from a design choice or an issue with optimization? For the latter, a typical trick to use is temperature annealing, which gradually lowers the temperature across training. If this was the problem then this may help with training.
- Can you clarify about the function of the sufficient reason? Is it meant to approximate the behavior of the whole network? If so, how well do the sufficient reasons capture the entire network, for the sufficient reasons extracted in the experiments?
- What distinguishes the proposed encoding from the one used by Jerad et al 2025? Is the transformer model the same?
3 comments
Response to the Rebuttal
Thank you for your response. In particular, the clarification on the exact model and the motivation behind the experiments was helpful.
I am left still wondering how to assess the efficacy of the sufficient reasons. What kind of analysis can they be used for explaining? Out of the words w' such that T(w')=T(w), how many are actually captured by the sufficient reason for w? Without this analysis or explicit use cases of sufficient reasons, it is unclear how to compare them to full extraction of the network.
For now I will maintain my rating. Thank you again to the authors.
We thank the reviewer for their suggestion. The advantage of sufficient reasons is that they can still be computed when the global formula cannot, as in our experiments with 1000 symbols. We agree that measuring how many inputs with the same label a sufficient reason covers would be a useful addition. For our six-symbol datasets this can be computed exactly, since by Proposition 2 it suffices to consider the words without repeated symbols, and we will include these numbers in the final version.
We thank the reviewer for their response and suggestions.
We did not tune for accuracy, since extraction is exact for any trained model. Annealing is a good suggestion, however, and it fits well our setup as the forward pass stays hard; we will try it.
A sufficient reason is specific to a single input w: it is a formula guaranteeing that any input satisfying it receives the same classification as w. Therefore, we can see it as describing the exact behaviour of the whole network on w, rather than approximating the network. Its purpose is to explain the network's classification for w.
No, the model is different. Jerad et al. (2025) also study transformers with left-most hard attention, but they consider future masking, whereas our HAT has no masking (and other minor differences). Their translation serves to prove an expressivity result and is not designed to be implemented. Unlike Jerad et al., we also extract sufficient reasons, which explain the classification of a single input.