acceptodds
Under review as a conference paper at ICLR 2027

Guiding Attention Circuit Discovery with Reasoning Outcomes

Abstract

We adapt attention circuit discovery to reasoning models by learning which attention connections between sentences in a reasoning prefix must be preserved to promote a given reasoning outcome. Because interventions can change the distribution of reasoning continuations, we train masks on continuations sampled from the unmasked model and test them on held-out unmasked continuations and on fresh continuations generated under the learned mask. We consider three objectives—preserving answer distributions, increasing correct-answer probability, and shortening reasoning—and find that our approach is more successful than a local connection baseline that scores connections by their effects on subsequent token predictions. We then investigate the learned circuits and find that retained connections show no consistent preferences for different types of reasoning steps. Moreover, different training seeds also produce distinct circuits with similar performance across sparsity levels. Together, these findings suggest that language model “reasoning” lacks the crisp, systematic structure evoked by the word.

Primary area:
interpretability and explainable AI
Keywords:
Interpreting Reasoning Traces, COT Faithfulness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_458WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies reasoning "cuircuits" via per task attention optimized masking on prefix at different sparsity with three types of suffix associated loss:

  • answer correctness
  • answer length
  • probability conservation

In all three cases demonstrate ability to minimize the loss relative to random masking and baseline "local" method. The qualitative conclusion drawn by the analysis of the resultant masks is that no particular structure arises supporting a view that "reasoning models end up producing a thick soup of plans, facts, self-checking, ‘aha’ moments, and so on in their hidden reasoning."

Critical strengths

The understanding of reasoning attention circuits is a powerful, insightful and highly interesting topic. Showing a method applicable across different types of analysis is particularly interesting.

The results of the per-question prefix masks yield consequent attention token space interpretable analysis.

Critical weaknesses
  1. The paper suggests a rather strong claim with respect to the absence of attention emerging structure. However non-uniqueness of outcome-optimized sparse masks is underpowered in distinguishing (a) the claimed unstructured reasoning from (b) a structured but redundant computation.

  2. More fundamentally the identification of circuits itself under this methodology has inherent challenges driven by the methodology choice:

  • The optimization ('learning') for mask finding is question specific --- as such it can potentially be used on that task, but can not distinguish task specific connections from attention "circuits" employed for reasoning more generally. A much stronger analysis would be required to disentangle generalizable circuits --- a non-task specific learned masking function which generalizes across unseen tasks.
  • Exacerbating the above is the oracle / hindsight information used for the optimization of the mask per task/loss --- e.g. knowledge of correct answer, knowledge that unmasked model would have dominant mass on correct answer in the breakage point.
  • The choice of masking prefix only (with non masked suffix) limits the interpretability of the attention circuit by construction
  1. The extension to problems which are not multi-answer but are verifiable for long reasoning horizons is preferable to show this work generalizes.
  2. the distinction between reasoning models and non reasoning models suggested in the discussion can and should be tested and shown. Similarly, models of different performances --- including progressively strong (with wide enough dynamic range of tasks) --- should be evaluated to distinguish noise in reasoning from structure. if Indeed non-structured reasoning emerges, then would this lack of structure be preserved irrespective of model perfromance?
Questions for authors

See above weaknesses section --- can you show generalization and disentangle the absence of structure from redundancy ?

4 comments

We thank reviewer 458W for their thorough and constructive review, it helped us deconfuse a lot of things in our draft. We address all raised weaknesses below:


Response to your Weakness 1: unstructured v/s structured but redundant

We have updated our draft’s main body and added details to Appendix F to address this.

Specifically, we separate three properties that the original draft mixed together, and all relate to notions of “structure” (or lack thereof):

  • Sparse / Reducible: a mask can remove most connections and still meet the objective.
    • Looking at the curves from Figure 7, across all 3 tasks, the performance remains high until 80% sparsity. Beyond that, they do not effectively influence the outcomes.
  • Redundant: more than one set of connections meets the objective at the same sparsity.
    • Our updated Figure 6 shows this more clearly: it plots the fraction of shared connections against sparsity of the masks. Up to 70% sparsity, performance differences across seeds are in the order of 10−210^{-2}, and hence are interchangeable.
    • Further, Appendix F shows how much the unique connections help performance. At up to 60% sparsity, the common nodes across seeds and the whole mask only marginally differ in performance (Figure 16). It also shows you can’t just replace them with totally random connections.
  • Semantically predictable: the kept connections follow a rule that can be stated in terms of the types of sentence connections that are kept, and this rule holds across training seeds.
    • We test this with a judge model that grades each sentence pair (i, j), j > i, on whether sentence j depends semantically on sentence i (Figure 17). Answer preservation keeps the most semantically meaningful connections. At 80% sparsity, its masks retain 70% of the sentence dependencies that a judge model marks, whereas a random mask retains only 20%. The correct-answer and shortening masks keep 23% and 17%, respectively (equal to random chance).
    • However, note that these are only a small fraction of connections at 80% sparsity and the connections that aren’t meaningful to a blackbox judge still hold a lot of influence on the outcome.

To account for this ambiguity, our abstract and contributions (under Introduction) now state that the outcomes we study depend on sparse, partially redundant connections, and that semantic dependence between sentences does not account for the entire outcome. And claims about unstructured reasoning have been revised under Discussions (Section 5).


Response to your Weakness 2: Various points on our methodology

2.1) Optimizing on an objective:
Outcome supervision for guiding interpretability is common and defines the behavior we seek. E.g., for learning interventions (Geiger et al., 2024; Wu et al., 2023; Huang et al., 2024) or for circuit discovery along model components (Conmy et al., 2023; Syed et al., 2024; Hanna et al., 2024, De Cao et al., 2020; Cao et al., 2021). We do acknowledge that optimizing objectives for interpretability can have unintended effects (Makelov et al, Miller et al., Ramanujan et al.), and have updated our discussion section to reflect this. Despite the debate, we remain optimistic about such approaches and believe it is useful to study interventions optimized for a particular behavior.

(Point continued in the next comment)

2.2) What we mean by circuits, and generalization to unseen tasks:

Our goal is to find the connections between sentences that influence a particular behavior on a particular reasoning prefix. The nodes of our circuits are the sentences of one prefix. A different question has different sentences, so a mask cannot be applied to it directly. We agree that such a mask does not show how reasoning works internally/across questions or datasets, and we do not claim this. (Also note that Attribution Graphs, Ameisen et al., also learn their circuits per prompt, while optimizing on a particular token's logit. It is still very useful!)

That said, we do also check which properties repeat across questions after we learn the question-specific masks:

  • Across Model components (Appendix H). When we learn a separate mask for every attention head, the same heads matter across questions (rank correlation 0.74–0.88) and across objectives (0.82). Most of them lie in the middle layers.
  • Across Sentences (Appendix F, paragraph "Comparing the masks of two tasks"): Two tasks keep different sentence pairs on the same question. For each question, we compare the answer-preservation mask with the correct-answer mask. Both masks choose from the same sentence pairs. They overlap about as much as two random masks, and far less than two training seeds of one task (see table below). A single mask shared across tasks would not capture what influences each task specifically.
ModelTwo masks for the same question30% sparsity50% sparsity70% sparsity
BothTwo random masks70%50%30%
Qwen3-8BAnswer preservation and correct answer (42 questions)71%51%33%
Qwen3-8BTwo training seeds of one task (2 questions)83–92%70–89%52–77%
Qwen3-32BAnswer preservation and correct answer (10 questions)71%52%32%
Qwen3-32BTwo training seeds of the correct-answer task (2 questions)83–90%70–81%54–68%

(Shortening task cannot be included in the above table because it uses different prefixes)

A masking function that takes an unseen question and outputs a mask is an interesting next step. It needs a different parametrization (for example, a network that scores sentence pairs based on their representations), and we leave this for future work.

2.3) Masking only prefix:
Yes, this is a structural limitation. A sentence pair connection (i, j) cannot be trivially masked at an unrelated suffix sentence k, and sentences in the suffix are themselves subject to change, i.e., when we sample two distinct reasoning rollouts (suffixes), there is no sense in which the kth sentence in one rollout suffix is the same as the kth sentence in the other rollout suffix

However, prefix-only masks measurably influence the final answer, as we see in our results (Figure 3), and think that studying the downstream effect of removing i’s information at j under this setting is therefore meaningful. In contrast, if our results were negative, the lack of a suffix mask would be more concerning.

We isolate how much the unmasked suffix matters in Appendix I using a proxy task. Instead of masking sentence pairs, we mask entire sentences. During evaluation, we check the effect of “what if this sentence didn’t exist” by removing masked sentences from the context and rerunning the model. We can then compare the difference between training with prefix-only ablations and training with prefix-plus-suffix ablations on this evaluation. We find that the latter shows some improvement over prefix-only masking. However, it does not result in a much sparser solution.

  • Prefix-only masks still improve over the local baseline. Under deletion, the prefix-only mask has a lower KL than the baseline on all 10 prompts (median area under the normalized-KL curve 0.13 against 0.23).
  • Covering the suffix does not change the sparsity-performance curve qualitatively. The KL rises with sparsity in the same way for both trainings. The prefix-and-suffix mask is lower (median area 0.081), which we expect, because its training is closer to the deletion evaluation.

We present AUCs over 5–95% sparsity (similar to Figure 3, but for this task), with 95% bootstrap CIs below:

fixed suffixes: normalized KL ↓ (×10⁻³)generated suffixes: KL(q̄_clean ‖ q̄_M) ↓ (×10⁻³)
prefix-and-suffix mask80.7 ± 51.117.4 ± 9.9
prefix-only mask127.7 ± 65.031.0 ± 14.7
local connection baseline234.8 ± 130.936.4 ± 17.8
random mask281.2 ± 158.291.3 ± 69.6
unmasked model–3.1 ± 2.9

(Continued in the next comment)

Response to your Weakness 3: unstructured non-MCQ responses
We do include prompts from MATH, which do not have multiple choice / single token final answers, but unstructured math equations. We show how to grade them in Appendix E.1. Unfortunately, even longer-horizon runs with rewards as objectives are outside the scope and budget of our project. We are excited to consider doing this as future work.


Response to your Weakness 4: Extending to non-reasoning models and models of different scale

Thank you for this suggestion. We ran the test (Appendix F, paragraph "Non-reasoning models", Figures 19–21). It did not support our speculation that non-reasoning models would be more structured, and we have removed the speculation from the Discussion.

Non-reasoning model (Qwen2.5-7B-Instruct, prompted to think step by step): the masks are as redundant as for Qwen3-8B, and they keep the judged dependencies in the same order across tasks. We see two differences:

  • Answer preservation objective does not perform as well: the learned masks and the local baseline are within noise of each other (Figure 19). For Qwen3-8B, the learned masks are clearly better (Figure 7).
  • Raise P(correct answer) objective retains some semantic connections: the masks keep more judged dependencies than random masks (Figure 21). For Qwen3, they sit at the random level.

We want to point out to the reviewer that the step-by-step solutions of a non-reasoning model are much shorter than reasoning traces (with a median of 458 tokens against 7005 on GPQA Diamond). They follow a single approach and do not change direction, recheck steps, etc. In 50% of the questions checked (we filtered them out), removing every connection barely changes the answer, already pointing to highly performative reasoning.

Models of different scale:

Qwen3-8B and Qwen3-32B agree in most figures of Appendix F. We agree that 7B to 32B is not the wide range of capability that you ask for. Unfortunately, a larger sweep over progressively stronger models is beyond our compute capability, and we state this as a limitation.


We thank you again for your helpful comments and hope these additional results make you feel more confident about our work. If so, we kindly ask you to consider increasing your score. We are happy to answer any follow-up questions you may have.

Dear Reviewer 458W,

Thank you again for your thoughtful review and for the time you have devoted to evaluating our work. We do understand you may be busy at this time. If you have a chance before the discussion period ends, we would be grateful if you could take a look at our replies. We would appreciate hearing whether these responses address your concerns, and would be happy to discuss any questions that remain.

Best Regards,

Authors of Submission 53459

Reviewer_khc7OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a discovery method for attention circuits to explain CoT reasoning.

Critical strengths

Strengths:

  • This paper is the most well-written one in my review batch (for humans as the reader).
  • The proposed technical approach is novel -- since we know transformer circuits (connecting transformer's internal components) very well, it is now a good time to study the connection between tokens.
  • The three tasks are well defined and well executed (preserving answer distribution, increasing correct answer likelihood, shortening reasoning).

This manuscript is the most interesting and (potentially) useful paper in my review batch in ICLR this year. So I would recommend a strong acceptance.

Critical weaknesses

Weaknesses:

The only significant weakness is this paper could better distinguish itself with the transformer circuits (which are sparse computation sub-graphs inside the transformer neural network model, which reverse engineer and reproduce the full model's performance, e.g., https://openreview.net/forum?id=Hf17y6u9BC).

  • I am sure the authors know these literatures well, can you add more in-depth discussions about the differences?
  • If my understanding is correct, this paper studies the circuit at the token level, while transformer circuits are at the components level. If it is correct, does it make sense to create a 2D version of the circuit discovery -- considering the causal impact of tokens (dimension 1) and components in the transformer (dimension 2)?
Questions for authors

Are there any actionable takeaways that LLMs practitioners can use your identified attention circuits?

AI assistance section: Did you use AI tools to polish your writing? I am also wondering did you use any specific AI tools to create the figures (e.g. Fig 1 and Fig 5)? They all look very nice, but I am not sure if they are disclosed in the section.

Can the authors show more in-depth analyses about the different variants' models' rollouts as case studies? It is interesting to see where are the important tokens in your circuits (esp. those segmentations, and "thinking" tokens).

1 comment

Thank you, Khc7, for your positive assessment of our work! We are glad you found it to stand out amongst your assigned reviews. We address each of your raised questions point-by-point below:


Causal effect along model components:

Added an analysis in Appendix H and expanded Related Work (Appendix A) based on your suggestion.

We can indeed learn masks along layers and individual heads. We find that the middle layers are very important, and head rankings are highly correlated across different objectives (Spearman rank correlation = 0.82, across 1152 heads). Crucially, we do not rank heads by ablating them. Instead, we learn masks over their attention patterns jointly and rank them after the optimization based on their individual sparsity. The following table shows how much the answer distribution changes (median KL in nats, over 25 prompts) when we zero the heads that are among the top heads of both tasks, compared with zeroing the same number of random heads

Top heads per taskHeads common to both tasksZero the common headsZero as many random headsZero as many random heads from the same layers
2070.120.020.06
50220.400.060.12
100591.050.240.27

Analyzing suffixes/rollouts and masks:
Suffixes for KL preservation and Answer Correctness are drawn only for 200 more tokens. These do not provide enough information for analysis and look very similar to regular rollouts. We provide an additional appendix in our updated draft (see Appendix G) analyzing the suffixes from the shortening task. We also add an end-to-end example of what the prefix mask does in Appendix J (one for each task objective).

In summary, here’s what our analysis of the rollouts say:

  • The mask works by increasing the probability that the model concludes the trace in the next few tokens, after an approach naturally ends with an answer (rather than pivoting to a new approach). Interestingly, we do not find a huge difference in how much this probability increased when the previous approach concludes with a wrong option wrt the correct option. The following table shows how many times more likely the model is to stop reasoning at a paragraph break with the mask than without it, on the same text:
P(conclusion | correct option)P(conclusion | approach ended on another option)
learned mask, 30% sparsity5.7×4.6×
learned mask, 70% sparsity11.8×8.5×
local connection baseline, 30% / 70%1.0× / 1.1×1.1× / 1.1×
random mask, 30% / 70%1.6× / 2.7×1.6× / 2.9×
  • All the patterns we see increase up to 60-70% sparsity, then start dropping along with output coherence, giving a U-shape (or inverted U-shape) to many metrics.
    • The largest effect we see is a net decrease in the fraction of sentences that start with: “Alternatively…”, “Wait …”, or “Let me try …”. (see Figure 22: left of our updated draft).
    • We also find that the mask removes keys containing such sentences (i.e., “Alternatively…”, “Wait …”, or “Let me try …”) more often than others: the sentences the mask hides most often are 1.5 times more likely to be from this set than from a random set.
    • Together, these results suggest that, at moderate sparsity, the masked model generates more efficient reasoning paths: it revisits the problem less often and stops sooner, while maintaining the same answer.

Practical Takeaways
Our shortening experiments suggest a potential route to reducing inference costs: attention interventions can encourage the model to stop after reaching an answer, reducing repeated attempts and reconsideration while maintaining or improving accuracy. However, our masks are learned separately for each reasoning prefix, so turning this finding into a practical method would require amortizing the cost of mask discovery or learning interventions that generalize across questions.


How we make figures and AI usage:
The figures are made using HTML. We used LLMs to convert matplotlib plots to HTML.
We used LLMs to draft the appendix and to check/edit the experimental details/math equations to better match the implementation. We have clarified the AI usage section to include this as suggested.


We thank you again for your helpful comments and are happy to answer any follow-up questions you may have.

Reviewer_LrAsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors tackle the problem of adopting circuit discovery methods to Large Reasoning models. Specifically, the authors extends the desiderate-based masking method by learning a sparse binary mask over coarse-grained sentence-level attention within reasoning prefix. To facilitate training, the authors produced samples of reasoning continuations from the unmasked (original) model. The training objective is to preserve the answer distribution, raise the probability of getting the correct answer, and reducing the token in the continuation. The trained masks are evaluated on held-out fixed suffixes and suffixes produced by masked models. The learned mask is evaluated on a range of datasets. Results show that the learn mask provide significantly better performances in terms of answer preservation, probability of correct answer, and token reduction in the reasoning continuation. Further analysis show that (1) the remaining attention connections show no preference for sentence type; and (2) training with different random seed result in different learned masks. These finding lead to the conclusion that "language model reasoning lacks the crisp, systematic structure evoked by the word."

Critical strengths
  1. The authors investigated approaches to conduct circuit discovery in Large Reasoning Models where assuming a fixed token sequence is not possible.
  2. The authors evaluated the learned masks on both their generalizability to held-out suffixes from the original unmasked model and the masked model.
  3. A lot of efforts have been putting into experiment design and ablation studies.
Critical weaknesses
  1. Most of the performance gains introduced by the revised Desiderata-based masking method could be attributed to the training objective indirectly optimizes for the evaluation metrics. While the metrics improved over the objective-agnostic baseline method, it require training which diminished the versatility and potentially generalizability that other circuit discovery methods possess.
  2. Figure 3a shows that the learned mask doubles the probability of getting an accurate answer on the generated suffixes while halves the number of thinking tokens. I wonder what have the mask changed in the continuation that make the reasoning to be seemingly "more efficient". An analysis based on the reasoning content would be very interesting.
  3. While the attention connection between (i,j)(i, j) in the prefix is masked, the continuation or suffix would still be able to attend to both sentences as their attentions are not masked. I wonder how much this diminishes the effect of the learned mask and how would the model performances change should we also regulate the attention connections in the continuation.
Questions for authors
  1. While the attention connection between (i,j)(i, j) in the prefix is masked, wouldn't the suffix still be able to attend to both sentences? How would this affect the performance of the model and the validity of the masks?
  2. How does the efficiency of the revised Desiderata-based masking method compared to the baseline method? It would be insightful to provide some metrics to quantify this, which would help people to better understand the tradeoff of learned attention masks.
  3. Are there any insights we can get from the phenomenon where learned mask improves the probability of getting a correct answer while reducing the length of the continuation. What has the content or "thought process" of the continuation changed compared to the continuation from the unmasked model.
3 comments

Dear Reviewer LrAs,
Thank you for your constructive and detailed review of our work. Your requested ablations provided valuable additions to the paper! We address each raised question below:


Response to your Weakness 1: Optimizing on an objective
While we do deliberately optimize for the behavior we want to investigate, this is not uncommon in mechanistic interpretability for guiding learning interventions (Geiger et al., 2024; Wu et al., 2023; Huang et al., 2024) or for circuit discovery along model components (Conmy et al., 2023; Syed et al., 2024; Hanna et al., 2024, De Cao et al., 2020; Cao et al., 2021). These works then evaluate their interventions/circuits on unseen prompts but with the same objective, which is an empirical question.

In our case, we want to ablate parts of the context to narrow down what matters for the objectives and properties we care about in a reasoning chain (as done unsupervised in Bogdan et al. and the local connection baseline). In both unsupervised and supervised setups, the scope of a mask only holds over a constant prefix mask and nowhere else. We ask the empirical question: do these properties hold even when we draw completely new generations from the masked prefix? Even when we train on fixed traces, this does hold. Generalization across unseen questions remains hard to formalize in this setup and is outside the present scope. (Note that Attribution Graphs, Ameisen et al., also learn their circuits per prompt, while optimizing on a particular token's logit. It is still very useful!)

We also acknowledge that optimizing objectives for interpretability can have unintended effects (Makelov et al., Miller et al., Ramanujan et al.), and have updated our discussion section to mention this.


Response to your Weakness 2 and Question 3: What is going on in the shortened rollouts?

Added Appendix G: Interpreting the suffixes generated under shortening masks.

In summary:

  • The mask works by increasing the probability that the model concludes the trace in the next few tokens, after an approach naturally ends with an answer (rather than pivoting to a new approach). Interestingly, we do not find a huge difference in how much this probability increased when the previous approach concludes with a wrong option wrt the correct option. The following table shows how many times more likely the model is to stop reasoning at a paragraph break with the mask than without it, on the same text:
P(conclusion | correct option)P(conclusion | approach ended on another option)
learned mask, 30% sparsity5.7×4.6×
learned mask, 70% sparsity11.8×8.5×
local connection baseline, 30% / 70%1.0× / 1.1×1.1× / 1.1×
random mask, 30% / 70%1.6× / 2.7×1.6× / 2.9×
  • All the patterns we see increase up to 60-70% sparsity, then start dropping along with output coherence, giving a U-shape (or inverted U-shape) to many metrics.
    • The largest effect we see is a net decrease in the fraction of sentences that start with: “Alternatively…”, “Wait …”, or “Let me try …”. (see Figure 22: left of our updated draft).
    • We also find that the mask removes keys containing such sentences more often than others: the sentences shortening masks hide most often are 1.7 times more likely to be from this set than from a random set.
    • Together, these results suggest that, at moderate sparsity, the masked model generates more efficient reasoning paths: it revisits the problem less often and stops sooner, while maintaining the same answer.

Apart from this, for prefixes, we also add an end-to-end example of what the prefix mask does in Appendix J (one for each task objective).


(continued in the next comment)

Response to your Weakness 3 and Question 1: Effect of masking suffix

Added an analysis in Appendix I

We would also like to emphasize that the success of our prefix-only masking demonstrates that we do not need to mask suffix sentences to identify attention circuits relevant to the reasoning rollout (Figure 3). Our masking changes how prefix representations are computed, which can affect subsequent generation even though those sentences remain accessible. Our positive results show that prefix interventions can significantly influence reasoning outcomes. By contrast, if our results were negative, the lack of a suffix mask would be more concerning.

Moreover, this is another question that is somewhat hard to formalize within our setup (from Figure 1), as

  1. A sentence pair connection (i, j) cannot be trivially masked at an unrelated suffix sentence k
  2. The sentences in the suffix are themselves subject to change, i.e., when we sample two distinct reasoning rollouts (suffixes), there is no sense in which the kth sentence in one rollout suffix is the same as the kth sentence in the other rollout suffix

So instead, we present a proxy comparison in Appendix I: Instead of masking sentence pairs, we mask entire sentences. During evaluation, we check the effect of “what if this sentence didn’t exist” by removing masked sentences from the context and rerunning the model. We can then compare the difference between training with prefix-only ablations and training with prefix-plus-suffix ablations on this evaluation. We find that the latter shows some improvement over prefix-only masking. However, it does not result in a much sparser solution.

fixed suffixes: normalized KL ↓ (×10⁻³)generated suffixes: KL(q̄_clean ‖ q̄_M) ↓ (×10⁻³)
prefix-and-suffix mask80.7 ± 51.117.4 ± 9.9
prefix-only mask127.7 ± 65.031.0 ± 14.7
local connection baseline234.8 ± 130.936.4 ± 17.8
random mask281.2 ± 158.291.3 ± 69.6
unmasked model–3.1 ± 2.9

AUC over 5–95% sparsity (similar to Figure 3, but for this task). Median over 10 Qwen3-8B prompts, ± half the width of the 95% bootstrap interval.


Response to your Question 2: Cost of learning masks

Added a compute cost table to Appendix D.7, along with a complexity analysis for the best methods we tested.

While doing this, we also benchmarked a promising concurrent work (Arora et al.). We found that it is as good as the subnetwork probing pipeline we used, while being much quicker! (More details in Appendix D.6). Thank you for raising this point.

Specifically, here are the median costs:

TaskLocal connection baseline (GPU-h)Subnetwork probing, one mask (passes / GPU-h)Matryoshka attribution, one run (passes / GPU-h)Memory (GiB)
Qwen3-8B
Answer preservation0.70 (one score matrix serves all tasks)16,000 / 0.888,000 / 0.7917–27
Correct answer1,000 / 0.092,000 / 0.1617–20
Shortening6,000 / 2.724,000 / 9.836–49
Qwen3-32B
Answer preservation2.7 (one score matrix serves all tasks)4,000 / 2.78,000 / 5.266–69
Correct answer1,000 / 0.462,000 / 0.9264–73
Shortening6,000 / 9.924,000 / 3680–127

We thank you again for your helpful comments and hope these additional results make you feel more confident about our work. If so, we kindly ask you to consider increasing your score. We are happy to answer any follow-up questions you may have.

Dear Reviewer LRAs,

Thank you again for your thoughtful review and for the time you have devoted to evaluating our work. We do understand you may be busy at this time. If you have a chance before the discussion period ends, we would be grateful if you could take a look at our replies. We would appreciate hearing whether these responses address your concerns, and would be happy to discuss any questions that remain.

Best Regards,

Authors of Submission 53459

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers again for their constructive feedback. We are happy to see that the reviewers found our experiments thorough (LrAs, khc7) and our research question interesting (all). We summarise how we addressed the main concerns below, with all new results in the revised PDF.

  1. Analysing rollouts (LrAs, khc7).

    • We added our analysis of how the rollouts changed in Appendix G. We observe that the fraction of sentences that start with Alternatively…”, “Wait …”, or “Let me try …” (which usually opens up a new point or a doubt in the reasoning trace), decrease significantly. Figure 22 shows a U-shape peaking at 60% sparsity (where the mask works best).
  2. Masking only the prefix (LrAs, 458W).

    • Appendix I adds a sentence-level variant in which the mask can also cover the suffix, evaluated by deleting the masked sentences. Covering the suffix helps, but does not give us qualitatively different sparsity curves. We also note that prefix-only masking significantly influences the final outcome. Had this not been the case, this would have been a larger concern.
  3. Outcome Supervision and prompt-specific masks (LRAs, 458W).

    • We add a new section to our main body (Section 5.1) to state our limitations and clarify that our scope is not to understand general latents responsible for reasoning as a whole, but instead is how a part of the trace helps the model get to X.
    • We also note that many interpretability methods optimize on an objective (eg: logit difference between the correct and incorrect answer, or KL divergence, both analogous to our objectives), and many works build prompt-specific circuits as well (Ameisen et al., Bogdan et al.). We go further by showing generalization across newly sampled rollouts as well. We therefore think these are very useful and practical, and that one should not stop doing interpretability on objectives due to possible downsides.
  4. Relation to component-level circuits (khc7).

    • We update our Literature Review to expand on this (Appendix A), and provide masks over heads as well (Appendix H). We observe that across different prompts as well as tasks, head rankings are highly correlated.
  5. Claims about interpretability of reasoning traces (458W).

    • We have clarified our claims about 'reasoning being unstructured' by splitting our observations into 3 parts: reducible (circuits are reducible up to 80% sparsity), redundancy (up to 60% sparsity), and semantically structured (answer preservation masks retain sentence pairs that are semantically dependent; the rest of the objectives do not, irrespective of the optimizer, initialization, or hyperparameters). Therefore, we find that the traces are reducible/has sparse important connections, with redundant information, and that semantic structure depends on the final objective.
    • We also formulate a new way to visualize our circuits by clustering similar sentences into ideas/approaches. The resulting circuits are shown in Appendix J, and we believe it gives us a very interpretable basis to understand what is going on in the trace and which connections are important.
  6. Compute cost (LrAs). Appendix D.7 adds a complexity analysis and measured GPU-hours (Table 1).

    • We also benchmark a new concurrent method, Matryoshka attribution (Appendix D.6), which learns a single ranking across all sparsities. It matches or improves on our pipeline while using 3–13× fewer GPU-hours for a full sparsity sweep.

Overall, we added new results in Appendixes F to J (and D6, D7), and revised our discussions and claims to be clearer and more specific. Unfortunately, reviewers did not engage with our rebuttals during the discussion period. Yet, we strongly believe that our new results and changes to the draft sufficiently address the concerns raised. We hope you consider them during your reviewer–AC discussion period.

Thanks again for devoting your time and taking a look at our work!

Best Regards,

Authors of Submission 53459

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.