- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies Transformers on symbolic inputs to understand whether findings on attention head behavior transfer to a specific natural language task the authors propose.
Critical strengths
The understanding of causal mechanisms of behaviours in large language models and in their components, transformers, are a very important problem. Studying them in synthetic experiments is a good proposition because, due to the controlled nature, many boundary conditions can be prescribed and transparently followed.
The paper proposes an experimental design which tries to remove certain biasing aspects, for example, with their label names being non-informative. The paper has many experimental ablations (although their exact meaning is hard to discern for myself).
Critical weaknesses
My biggest concern with the paper is that the writing is extremely hard to understand. They are using many terms, and do not always explain them. Even in cases where their meaning might be inferred (BF, FP, "write" as a noun), by putting at least a reference or a short definition, would significantly improve ease of understanding for the reader.
I find it also hard that many quantities are defined, but not explained: what does it mean that a contrast is "clean"? the utility of what does J represent?
Regarding the related work section, I see that the authors do not refer to the literature in studying formal grammars or symbolic tasks to understand model behaviour, including their extrapolation performance or whether they can compose different steps ahead. I would recommend the authors investigate this literature to contextualise the work, e.g., Meszaros et al. Also regarding the causal analysis, many of the formally grounded methods are missing, e.g., works by Geiger et al., many of which relate to causal abstarctions. Also, from checking the mentions of BUC in multiple sections, I was not entirely sure whether that is a method proposed by the authors or is coming from previous works (I could not find any statement that has the spirit of "we propose BUC" - although my understanding is that this is also a contribution of the paper).
The theoretical statements are hard to contextualize, as there is no explanation of what the results mean.
On the experimental design side and the metrics, the descriptions in the paper can't really be followed because many of the metric symbols are perhaps only named but not explained. The metric ranges are not given (at least not in the tables), and some metrics are not defined in the main text (e.g., slope reduction - I found the definition in the appendix, but I could only find it in teh appendix by searchign the pdf, as there were no clickable references, same for PairAcc/MinPairAcc). The same holds for task descriptions
For this reason, the paper is basically impossible to judge for me, thus I recommend rejection.
Questions for authors
In 5.6, some scope checks mention different models, what exactly does that mean? What were the experiments, and how were they run?
4 comments
Response to Reviewer EWCu (2/2)
W5. What the theoretical results mean
Section 4 now opens by stating the question each result answers, and each theorem is followed by a plain-language reading:
- T1. When can cheap attribution (projecting each head's output onto the answer direction) predict BUC's intervention score for a fixed head set? The bound says the prediction error is at most the measured intervention residual divided by the clean contrast.
- T2. With finitely many prompts, when does the estimated top- attribution list equal the true one? It does when the true scores have a clear gap at rank relative to the sampling error.
- T3. When is greedy selection close to the best set? It is when the raw utility behaves like a (weakly) submodular function, in which case greedy achieves at least a fraction of the optimum.
We also measure the quantities each result depends on (Tables 25, 26, 5, and 50) and state where a guarantee does not apply. For example, T3 concerns , not the normalized BUC score.
W6. Metric definitions, ranges, task descriptions, and links
- Definitions in the main text. Shrink and match are defined in Section 3.2 with a worked example. Slope reduction is defined where it is used, in Section 5.3: one minus the ratio of the ablated to the clean regression slope of the readout on count. Excess damage and the majority-minus-read-one interaction are defined in Section 5.4. PairAcc and MinPairAcc are no longer used in the main text. They appear only in an appendix table (Table 28), whose caption now links to their definition in Appendix B.3.
- Ranges. Shrink and match are at most 1, equal 1 for complete removal or recovery, equal 0 when the intervention changes nothing, and can be negative. This is stated in Section 3.2 and now also in the captions of Tables 1 and 2. Accuracies are fractions, and damage is in logits or percentage points as labeled.
- Task descriptions. The symbolic task is described with explicit example inputs in Section 1. The language tasks (majority, read-one, KEEP/DROP) are described in Section 5.4, with templates and scene design in Appendix B.3 (Tables 7–8).
- Links. All cross-references and citations are now colored, clickable links.
- Ablations. We consolidated the appendix so the many ablations are easier to navigate. Appendix C collects the supporting results for Section 5, Appendix D holds the extended experiments, and overlapping tables were merged.
Q. Section 5.6 (now Section 5.5): what are the scope checks with different models?
This check asks whether the symbolic localization result is specific to Pythia-410M. For each of six frozen models, we reran the same BUC selection from scratch on the same symbolic prompts and readout, separately for each ordering, and evaluated the selected heads against layer-matched random sets. The six models are Pythia-70M, -160M, -410M, and -1B, GPT-2, and Qwen3-4B-Base. Before running, we checked that the answer words "various" and "successful" are single tokens in every model's tokenizer (Table 43). Averaged over the four orderings, the selected sets give shrink 0.650–0.895 and match 0.700–0.925 across models, versus at most 0.048 and 0.059 for random sets (Table 24; per-ordering results in Tables 39–40, Appendix D.7). The heads are reselected in each model, so this shows that the procedure finds effective small sets in other models; it does not transfer one head set between architectures.
Two other model checks appear in the revision. The fixed-budget method comparisons in Section 5.2 use both Pythia-410M and GPT-2. The language tests in Section 5.4 repeat the majority versus read-one comparison in Pythia-1B and GPT-2, each with its own symbolically selected heads.
We hope the rewritten text, the clarified theory, and these explanations make the paper possible to evaluate. We would sincerely appreciate it if you could take another look, and we are happy to clarify anything that is still unclear.
I thank the authors for the detailed response. The clarifications help understanding to some extent, though as I checked the uploaded pdf again, parts such as sections 6&7 still do not read in a self-contained manner. I'd consider teh authors to rework the presentation of their results and resubmit.
Response to Reviewer EWCu (follow-up)
Thank you for reading the revised PDF again and for telling us exactly where it still falls short. You are right about Sections 6 and 7. Both were written as compressed summaries that assumed the reader had just finished Section 5, so they relied on terms such as the matched task pair, the screened pool, and within-seed intervals without saying what they meant. We have now revised the main text so that each part can be read on its own, and the updated PDF is uploaded:
- Sections 6 and 7. Section 6 (Discussion and limitations) now has four labeled paragraphs that follow the order of Section 5: symbolic localization, selection objective, language transfer, and theory and model scope. Each paragraph restates the terms it uses, says in plain terms what the experiment establishes, and then gives its limits. For example, it explains that ablation and mean patching show that the count effect (the shift in the model's answer preference when evidence counts change) reaches the answer through the selected heads' final-position outputs, but not that these heads compute the count alone. It also explains why larger damage on majority than on read-one points to aggregation, together with the confounds that remain, and says what the method comparison, each of the three theorems, the confidence intervals, and the cross-model tests do and do not establish. Section 7 (Conclusion) now tells the complete story with the key numbers.
- Section 5. It opens with five questions, one per subsection, and restates what shrink and match measure. Each paragraph now states its finding first and then gives the evidence in short sentences. For example, Section 5.4 states the exact majority and read-one questions and defines a scene where it first uses the term.
- Definitions throughout. Every term is now defined where it first appears, including direct attribution, the attribution shortlist, the screened pool, sensitivity ranking, donor inputs, and the three language tasks. When a symbol or term reappears far from its definition, the text restates what it is, for example "GREEDY-R, the recovery-only baseline of Section 3" or "the signed raw utility of Section 4", so readers do not need to search back.
If any other part still does not read on its own, we would be grateful if you could point us to it, and we will revise it right away. We hope the revised presentation makes the contribution easier to evaluate, and we would appreciate it if you could take it into account in your final assessment.
Response to Reviewer EWCu (1/2)
Thank you for the thoughtful review. We appreciate your recognition that understanding causal mechanisms in Transformers is important, that controlled synthetic settings let boundary conditions be set and followed transparently, and that our design tries to remove biasing factors such as informative label names. We also agree with your main concern. The submitted version made the paper much harder to judge than it should have been, including the meaning of the many ablations. Below we address each Weakness (W1–W6) and your question.
Abbreviations: BUC = balanced utility causal selection; BF16 = bfloat16 precision; FP32 = 32-bit floating-point precision; FR/BA/IN/RA = Front/Back/Interleave/Random orderings; pp = percentage points.
W1. Undefined terms (BF, FP, "write")
All three are now defined at first use. A head's write is its contribution to the residual stream, the model's running hidden representation (Section 1). BF16 and FP32 are spelled out in Section 3.3, where they first appear. We also checked the rest of the main text for the same problem, so that abbreviations and named quantities are defined before they are used.
W2. What "clean" means, and what the utility J represents
Clean means computed with no intervention. The clean contrast is the ordinary high-count minus low-count difference in the readout (Section 3.1), and every intervention is compared against it (Section 3.2).
J (Section 4, Equation 4.1) is the signed raw utility. It is a weighted average of how much ablating a head set reduces the count contrast and how much patching it moves the readout in the intended direction, both in logit units. The difference from BUC's score Qint is normalization. Qint divides by the clean contrast and rewards landing close to it, whereas J keeps growing when a patch overshoots. The greedy guarantee in T3 is stated for J, and Section 4 now says in one sentence how the two differ.
W3. Related work on formal languages and causal abstraction
Thank you for these pointers. Section 2 now discusses Mészáros et al. (2024) on rule extrapolation in formal languages, together with studies of compositional failures and task-trained symbolic models. We explain the difference: those works ask whether a completion follows a prescribed rule, while our symbolic prompts prescribe no correct answer and we measure a frozen model's preference. Section 2 also covers the causal-abstraction line (Geiger et al., 2021; 2024). Interchange interventions test an alignment between neural components and a high-level causal model. BUC tests removal and recovery of a controlled contrast without assigning the heads a high-level variable. Appendix F extends this comparison in a table.
W4. Is BUC our proposal?
Yes, and we now say so explicitly: "We propose balanced utility causal selection (BUC)…" (Section 1, third paragraph). Our contribution is the joint selection criterion, which scores removal and recovery in both directions for the same head set, together with its controlled evaluation.