- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper theoretically studies how transformer models employing causal attention solve multi-hop reasoning problems. In particular, it studies synthetic problems in which the model follows a chain of relationships, for example, given the separate facts , , , it must identify where a three-step chain starting at ends.
The main research question focuses on whether and how changing the order of the facts can change how many transformer layers are needed to answer a reasoning question. This is an interesting question: even though reordering these facts leaves the actual answer unchanged, with causal attention, an input position can use earlier facts but cannot look into future positions (facts), so their order can affect how the model builds intermediate answers.
The authors prove the number of reasoning steps (transformer layers) needed to reach the correct answer. The particularly interesting theoretical finding is that presenting facts in the order of the logical chain is not always the most efficient arrangement. They instead find that certain interleaved presentations supports more reasoning within a fixed number of layers, and construct a transformer that performs the proposed computation. Experiments with gradient descent trained transformers show a substantial accuracy advantage for four-step problems satisfying the predicted ordering condition, giving some empirical supports to the theory’s ordering prediction at the output level (though the hidden computation patterns the theory predicts were not verified).
Critical strengths
Counterintuitive dependence of layer efficiency on fact ordering. To me, the most interesting insight in this paper is that an interleaved presentation of the ( form) facts can be more efficient in the number of layers to combine intermediate results than the natural logical order -- the latter ordering which I think the (transformer) interpretability community had mostly believed so far. The paper explains how this happens and quantifies its consequences across layers. It also identifies an intermediate range of problem lengths / complexities where the fact ordering determines whether the proposed computation can finish (shorter chains can be handled regardless of order, while longer chains require more layers even under favorable ordering).
From my knowledge, prior works in mechanistic interpretability and theory of transformers had established that transformers can perform multi-step computations in parallel and that premise order affects reasoning accuracy. This paper adds a more specific account of how the two interact when the facts and correct answer remain unchanged. I find that account scientifically informative, even though it applies to the particular way of combining facts studied here.
I also like that the theoretical prediction on this fact-ordering layer-efficiency prediction got tested on transformers trained from scratch in a controlled setting, albeit the experiments can be expanded to give a more thorough mechanistic account (I will discuss this in the weakness section below).
Critical weaknesses
Encouraging, but unsystematic empirical support. The central experiment only establishes an accuracy difference at a fixed depth of three layers. It does not directly show whether adding a layer removes the disadvantage of unfavorable orders on the same problems. The aggregate comparison also does not separately report on the natural logical order.
I believe one useful extension would be to compare matched presentations of the same k-step chains using L-layer models, sweeping combinations of a few k and L, with relevant positions and distractors controlled or balanced and results reported across independent training runs. This should not only give us a better idea of the layer-efficiency and problem-length predictions made by the theory, but also just more statistically broader demonstration of the theory (perhaps a set of phase portraits would be illuminating). Another important aspect, though slightly more demanding in terms of experiment design, is to understand how the intermediate computations behave in the real transformers under the different fact-ordering arrangements – one possible way is simply to train linear probes to predict the intermediate clause-combinations or simply intermediate hops, to understand whether we indeed get the interesting merging behaviors predicted by the theory. A further (and stronger) examination beyond linear probing would be using causal interventions, where we have two problems with exactly the same variables and ordering, and only changing an intermediate value, and test whether the hidden representations indeed mediate the intermediate hops and clause-merging in the way predicted by the theory's construction or if interesting differences arise.
The presentation obscures the paper’s scientific insights. My impression when reading the paper was that, the formalism tends to hide the intuitive argument. For instance, I believe the fact-ordering layer-efficiency insight can be made clearer once one understands why placing a middle fact after its neighbors allows both sides of a chain to be joined within a layer (if I understand the argument right). However, this explanation is distributed across the formal rules and bounds. One possibility is perhaps to introduce a small, layer-by-layer toy example comparing logical and favorable interleaved orderings before the main formal development. This could help readers understand what the bounds teach us, rather than primarily encountering them as pure abstract expressions.
Questions for authors
First, for the accuracy comparison in Figure 6, are the two ordering groups constructed by rearranging the same underlying facts and query, or are examples sampled separately from each group? And jow many independent training runs support the reported result? If available, could the authors also report accuracy for the natural logical chain order separately from the other unfavorable orders?
Second, while I understand this is primarily a theory paper with implications on transformer interpretability, the biggest pragmatic applications of transformers is LLMs, and connecting the insights in the paper to LLMs is important. Can the authors elaborate on what their main theoretical insights implicate for LLMs in the wild, and how to design and / or better improve LLMs in multi-hop (latent) reasoning? Do the insights about fact ordering imply certain better principled ways we should present (propositional) logical clauses to LLMs in training and inference time? Do existing LLMs in the wild benefit from this interesting fact-ordering efficiency ?