- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies reasoning "cuircuits" via per task attention optimized masking on prefix at different sparsity with three types of suffix associated loss:
- answer correctness
- answer length
- probability conservation
In all three cases demonstrate ability to minimize the loss relative to random masking and baseline "local" method. The qualitative conclusion drawn by the analysis of the resultant masks is that no particular structure arises supporting a view that "reasoning models end up producing a thick soup of plans, facts, self-checking, ‘aha’ moments, and so on in their hidden reasoning."
Critical strengths
The understanding of reasoning attention circuits is a powerful, insightful and highly interesting topic. Showing a method applicable across different types of analysis is particularly interesting.
The results of the per-question prefix masks yield consequent attention token space interpretable analysis.
Critical weaknesses
-
The paper suggests a rather strong claim with respect to the absence of attention emerging structure. However non-uniqueness of outcome-optimized sparse masks is underpowered in distinguishing (a) the claimed unstructured reasoning from (b) a structured but redundant computation.
-
More fundamentally the identification of circuits itself under this methodology has inherent challenges driven by the methodology choice:
- The optimization ('learning') for mask finding is question specific --- as such it can potentially be used on that task, but can not distinguish task specific connections from attention "circuits" employed for reasoning more generally. A much stronger analysis would be required to disentangle generalizable circuits --- a non-task specific learned masking function which generalizes across unseen tasks.
- Exacerbating the above is the oracle / hindsight information used for the optimization of the mask per task/loss --- e.g. knowledge of correct answer, knowledge that unmasked model would have dominant mass on correct answer in the breakage point.
- The choice of masking prefix only (with non masked suffix) limits the interpretability of the attention circuit by construction
- The extension to problems which are not multi-answer but are verifiable for long reasoning horizons is preferable to show this work generalizes.
- the distinction between reasoning models and non reasoning models suggested in the discussion can and should be tested and shown. Similarly, models of different performances --- including progressively strong (with wide enough dynamic range of tasks) --- should be evaluated to distinguish noise in reasoning from structure. if Indeed non-structured reasoning emerges, then would this lack of structure be preserved irrespective of model perfromance?
Questions for authors
See above weaknesses section --- can you show generalization and disentangle the absence of structure from redundancy ?
4 comments
2.2) What we mean by circuits, and generalization to unseen tasks:
Our goal is to find the connections between sentences that influence a particular behavior on a particular reasoning prefix. The nodes of our circuits are the sentences of one prefix. A different question has different sentences, so a mask cannot be applied to it directly. We agree that such a mask does not show how reasoning works internally/across questions or datasets, and we do not claim this. (Also note that Attribution Graphs, Ameisen et al., also learn their circuits per prompt, while optimizing on a particular token's logit. It is still very useful!)
That said, we do also check which properties repeat across questions after we learn the question-specific masks:
- Across Model components (Appendix H). When we learn a separate mask for every attention head, the same heads matter across questions (rank correlation 0.74–0.88) and across objectives (0.82). Most of them lie in the middle layers.
- Across Sentences (Appendix F, paragraph "Comparing the masks of two tasks"): Two tasks keep different sentence pairs on the same question. For each question, we compare the answer-preservation mask with the correct-answer mask. Both masks choose from the same sentence pairs. They overlap about as much as two random masks, and far less than two training seeds of one task (see table below). A single mask shared across tasks would not capture what influences each task specifically.
| Model | Two masks for the same question | 30% sparsity | 50% sparsity | 70% sparsity |
|---|---|---|---|---|
| Both | Two random masks | 70% | 50% | 30% |
| Qwen3-8B | Answer preservation and correct answer (42 questions) | 71% | 51% | 33% |
| Qwen3-8B | Two training seeds of one task (2 questions) | 83–92% | 70–89% | 52–77% |
| Qwen3-32B | Answer preservation and correct answer (10 questions) | 71% | 52% | 32% |
| Qwen3-32B | Two training seeds of the correct-answer task (2 questions) | 83–90% | 70–81% | 54–68% |
(Shortening task cannot be included in the above table because it uses different prefixes)
A masking function that takes an unseen question and outputs a mask is an interesting next step. It needs a different parametrization (for example, a network that scores sentence pairs based on their representations), and we leave this for future work.
2.3) Masking only prefix:
Yes, this is a structural limitation. A sentence pair connection (i, j) cannot be trivially masked at an unrelated suffix sentence k, and sentences in the suffix are themselves subject to change, i.e., when we sample two distinct reasoning rollouts (suffixes), there is no sense in which the kth sentence in one rollout suffix is the same as the kth sentence in the other rollout suffix
However, prefix-only masks measurably influence the final answer, as we see in our results (Figure 3), and think that studying the downstream effect of removing i’s information at j under this setting is therefore meaningful. In contrast, if our results were negative, the lack of a suffix mask would be more concerning.
We isolate how much the unmasked suffix matters in Appendix I using a proxy task. Instead of masking sentence pairs, we mask entire sentences. During evaluation, we check the effect of “what if this sentence didn’t exist” by removing masked sentences from the context and rerunning the model. We can then compare the difference between training with prefix-only ablations and training with prefix-plus-suffix ablations on this evaluation. We find that the latter shows some improvement over prefix-only masking. However, it does not result in a much sparser solution.
- Prefix-only masks still improve over the local baseline. Under deletion, the prefix-only mask has a lower KL than the baseline on all 10 prompts (median area under the normalized-KL curve 0.13 against 0.23).
- Covering the suffix does not change the sparsity-performance curve qualitatively. The KL rises with sparsity in the same way for both trainings. The prefix-and-suffix mask is lower (median area 0.081), which we expect, because its training is closer to the deletion evaluation.
We present AUCs over 5–95% sparsity (similar to Figure 3, but for this task), with 95% bootstrap CIs below:
| fixed suffixes: normalized KL ↓ (×10⁻³) | generated suffixes: KL(q̄_clean ‖ q̄_M) ↓ (×10⁻³) | |
|---|---|---|
| prefix-and-suffix mask | 80.7 ± 51.1 | 17.4 ± 9.9 |
| prefix-only mask | 127.7 ± 65.0 | 31.0 ± 14.7 |
| local connection baseline | 234.8 ± 130.9 | 36.4 ± 17.8 |
| random mask | 281.2 ± 158.2 | 91.3 ± 69.6 |
| unmasked model | – | 3.1 ± 2.9 |
(Continued in the next comment)
Response to your Weakness 3: unstructured non-MCQ responses
We do include prompts from MATH, which do not have multiple choice / single token final answers, but unstructured math equations. We show how to grade them in Appendix E.1. Unfortunately, even longer-horizon runs with rewards as objectives are outside the scope and budget of our project. We are excited to consider doing this as future work.
Response to your Weakness 4: Extending to non-reasoning models and models of different scale
Thank you for this suggestion. We ran the test (Appendix F, paragraph "Non-reasoning models", Figures 19–21). It did not support our speculation that non-reasoning models would be more structured, and we have removed the speculation from the Discussion.
Non-reasoning model (Qwen2.5-7B-Instruct, prompted to think step by step): the masks are as redundant as for Qwen3-8B, and they keep the judged dependencies in the same order across tasks. We see two differences:
- Answer preservation objective does not perform as well: the learned masks and the local baseline are within noise of each other (Figure 19). For Qwen3-8B, the learned masks are clearly better (Figure 7).
- Raise P(correct answer) objective retains some semantic connections: the masks keep more judged dependencies than random masks (Figure 21). For Qwen3, they sit at the random level.
We want to point out to the reviewer that the step-by-step solutions of a non-reasoning model are much shorter than reasoning traces (with a median of 458 tokens against 7005 on GPQA Diamond). They follow a single approach and do not change direction, recheck steps, etc. In 50% of the questions checked (we filtered them out), removing every connection barely changes the answer, already pointing to highly performative reasoning.
Models of different scale:
Qwen3-8B and Qwen3-32B agree in most figures of Appendix F. We agree that 7B to 32B is not the wide range of capability that you ask for. Unfortunately, a larger sweep over progressively stronger models is beyond our compute capability, and we state this as a limitation.
We thank you again for your helpful comments and hope these additional results make you feel more confident about our work. If so, we kindly ask you to consider increasing your score. We are happy to answer any follow-up questions you may have.
Dear Reviewer 458W,
Thank you again for your thoughtful review and for the time you have devoted to evaluating our work. We do understand you may be busy at this time. If you have a chance before the discussion period ends, we would be grateful if you could take a look at our replies. We would appreciate hearing whether these responses address your concerns, and would be happy to discuss any questions that remain.
Best Regards,
Authors of Submission 53459
We thank reviewer 458W for their thorough and constructive review, it helped us deconfuse a lot of things in our draft. We address all raised weaknesses below:
Response to your Weakness 1: unstructured v/s structured but redundant
We have updated our draft’s main body and added details to Appendix F to address this.
Specifically, we separate three properties that the original draft mixed together, and all relate to notions of “structure” (or lack thereof):
To account for this ambiguity, our abstract and contributions (under Introduction) now state that the outcomes we study depend on sparse, partially redundant connections, and that semantic dependence between sentences does not account for the entire outcome. And claims about unstructured reasoning have been revised under Discussions (Section 5).
Response to your Weakness 2: Various points on our methodology
2.1) Optimizing on an objective:
Outcome supervision for guiding interpretability is common and defines the behavior we seek. E.g., for learning interventions (Geiger et al., 2024; Wu et al., 2023; Huang et al., 2024) or for circuit discovery along model components (Conmy et al., 2023; Syed et al., 2024; Hanna et al., 2024, De Cao et al., 2020; Cao et al., 2021). We do acknowledge that optimizing objectives for interpretability can have unintended effects (Makelov et al, Miller et al., Ramanujan et al.), and have updated our discussion section to reflect this. Despite the debate, we remain optimistic about such approaches and believe it is useful to study interventions optimized for a particular behavior.
(Point continued in the next comment)