acceptodds
Under review as a conference paper at ICLR 2027

A General Model Diffing Framework for Understanding Post-Training: Attributing Behavioral Differences to Parameter Updates

10 views

Abstract

Post-training dramatically changes the behavior of large language models, yet the mechanism behind these changes is hard to interpret because it originates from parameter updates of enormous scale and dimensionality. Prior model diffing works in the representation space and cannot trace behavioral differences back to the parameter updates that cause them. We propose a circuit-level model diffing framework that attributes the difference between two models within a single attribution graph. Running the base model and the post-trained model on the same input, we isolate the direct linear effect of the parameter difference at each block and convert it into an activation difference. We model these activation differences as nodes in an attribution graph built with a cross-layer transcoder, so that every path runs causally from a parameter-induced activation difference through interpretable features to the logits on which the two models disagree. Clustering these circuits across inputs yields a shared circuit that characterizes the behavioral difference. Applying the framework to a base model and the reasoning model obtained by reinforcement learning, we find that shallow-layer parameter updates on public tokens activate shared reasoning features in the middle-to-deep layers, which then exert a long-horizon influence across the generated response to drive specific reasoning behaviors. Extensive experiments show that activating only these few localized activation differences recovers most of the reasoning model's accuracy on in-distribution and out-of-distribution tasks, and steering the identified features elicits reasoning behavior.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Model Diffing, Post-Training, Mechanistic Interpretability, Cross-layer Transcoder

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pz8COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a mechanistic model diffing framework for analyzing differences between base and post-trained LLMs. The proposed method first translates parameter differences into localized activation differences, then traces their effects through CLTs, and finally clusters circuits across inputs to identify shared mechanisms. The framework is validated on Qwen3-8B and using both GRPO and SFT as post-training variants.

Critical strengths

The paper studies model diffing and uses CLTs on isolated activation differences to reflect parameter changes, which I consider is a novel combination. The proposed framework is well-motivated and technically sound.

Critical weaknesses
  1. The intro states that the framework can be applied to different base models and post-training variants. However, the experiments consider only Qwen3-8B and two post-training variants (GRPO and SFT). This makes it unclear whether the findings generalize across different model families, model sizes, and other post-training methods such as RLHF, DPO, or OPD.

  2. One of the paper’s main findings is that the most influential activation differences are concentrated on fixed prefix/template tokens. However, this finding may be an artifact of using the same prompt template across training samples. I wonder whether the authors could vary/paraphrase the prompt templates across samples and examine whether the same public-token mechanism still persists.

Questions for authors

Recent work on LLM mechanistic interpretability has also found that early layers are dominated by lexical processing and has focused on middle-to-late layers for identifying reasoning-related features, e.g.,

Ma, George, et al. "Do Sparse Autoencoders Identify Reasoning Features in Language Models?."

Given the similar layer-wise findings, I wonder whether this paper identifies a general property of LLM representations that has already been observed in prior work, or whether model diffing provides additional insights beyond the existing literature. It would be helpful to clarify this distinction in the related work.

4 comments

W1: Generalization across model families, scales, and post-training methods.

Thank you for raising these concerns. Our formulation and derivations, based on [1], primarily operate on differences in attention and MLP block outputs under fixed inputs. Since post-training changes only the parameters while preserving the architecture, the same formulation applies to other base models and their post-trained versions with attention and MLP blocks. They are not tied to a particular model family, scale, or post-training method, allowing the framework to transfer to other transformer-based LLMs, including those obtained through RLHF, DPO, or OPD.

We acknowledge that the framework’s applicability alone does not establish whether our empirical findings generalize to these additional settings. Parameter replacement and feature interventions in the original models validate the identified update locations and the relevant features’ behavioral effects, respectively. These experiments support our conclusions within the studied settings and provide a practical procedure for assessing results on new model pairs. We will clarify both the framework’s applicability and the empirical scope of our findings in the main paper.

[1] Transcoders Find Interpretable LLM Feature Circuits. NeurIPS 2024

W2: Does the public-token mechanism persist under template variation?

Thank you for this suggestion. We would like to clarify first that using a fixed task instruction template (e.g., "Solve the following problem...") during RL training is common practice, as exemplified by DeepSeek-R1-Zero [1]. Our analysis therefore examines a practically relevant training setup.

Moreover, our SFT experiment uses no additional system prompt or fixed task instruction template, retaining only the basic chat template, including markers such as <|im_start|>. These markers are part of the model’s native dialogue format, and we retain them following standard practice to preserve that format. Δ-activation influence still concentrates at these public-token positions, particularly in the chat template suffix. This shows that the localization phenomenon also occurs without a fixed natural-language task instruction.

To further examine robustness to template variation, we add two template ablation experiments.

First, we use an LLM to paraphrase the task instruction template. We then repeat circuit tracing with these modified test-time templates using the same method as in the paper. Figure 18 in Appendix F.3 of the paper PDF shows that Δ-activation influence still peaks at template positions, even when the templates change.

Second, we apply the same paraphrasing procedure and repeat the performance recovery experiment in Figure 6 with all other settings unchanged. Figure 19 in Appendix F.4 shows that progressively activating high-influence Δ-activations substantially improves performance toward that of the post-trained model. Activating fewer than 5% of all Δ-activation nodes achieves 84.8% accuracy, a 28.5 percentage point improvement over the base model.

[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.

Q1: Additional insights beyond previously observed layer patterns, and comparison with prior work.

Thank you for highlighting this related work. We will compare our work with it and then emphasize our substantive contributions beyond prior work.

Ma et al. [1] examine whether SAE features selected by contrasting reasoning and non-reasoning text capture reasoning or linguistic correlates. Their analysis focuses on middle and later layers and shows that many candidate features remain sensitive to surface cues. Our study instead examines how actual post-training parameter updates influence features and downstream outputs.

Crucially, we distinguish where parameter updates act from where the affected features activate. Early layers performing lexical processing does not by itself explain which updates in these layers affect later features or subsequent generation. Our framework connects these quantities within one attribution graph, providing an account of post-training changes beyond the distribution of representations across layers. We will clarify this distinction in the related work.

Compared with prior work, our substantive contributions are threefold.

(1) A general model-diffing framework. We extend circuit analysis from explaining model outputs to attributing the effects of parameter updates. We introduce Δ-activations under fixed block inputs as explicit sources in a unified graph connecting updates, intermediate features, and outputs. The formulation applies to transformer-based LLM checkpoints across model families, scales, and post-training methods. We demonstrate it on RL and SFT.

(2) Circuit tracing on real-world tasks. Beyond the short case studies in [2,3], adaptive tracing, source and path pruning, and memory optimizations extend our analysis beyond 500 tokens, supporting circuit analysis and intervention validation on realistic reasoning tasks such as AIME24 and MATH500.

(3) An explanation beyond template importance. Beyond the template importance established in [4,5], our circuits connect shallow parameter updates at template positions to sustained activation of shared features in the middle and later layers, which influence subsequent generation over long sequences. This explains how post-training acts through these positions and why they matter. Figure 2 summarizes this mechanism, supported by the localization, parameter replacement, and feature steering results in Figures 3–6.

We will release our framework’s code and trained CLT checkpoints to support further circuit research by the community.

[1] Do Sparse Autoencoders Identify Reasoning Features in Language Models? ICML 2026

[2] Circuit Tracing: Revealing Computational Graphs in Language Models. Anthropic's Blog

[3] Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders. EMNLP 2026

[4] Why Safeguarded Ships Run Aground? Aligned Large Language Models’ Safety Mechanisms Tend to Be Anchored in The Template Region. ACL 2025

[5] Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors. NeurIPS 2025 Workshop

I thank the authors for their detailed rebuttal. While I appreciate the conceptual clarifications, I agree with Reviewer 6EAS concern that the empirical evidence remains limited. I have decided to maintain my original score.

Reviewer_6EASOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a circuit-level model-diffing framework. By comparing a base and a post-trained model on identical inputs, it treats block-level differences as "Δ-activations" within a cross-layer transcoder attribution graph. It traces paths to diverging logits and intersects per-prompt subgraphs into a shared circuit. On Qwen3-8B-Base and a GRPO variant, shallow Δ-activations on template tokens trigger mid-to-deep reasoning features, consistent with a synthetic SFT task. Validating the circuit, injecting top-ranked Δ-activations into the base model recovers most post-trained accuracy, and steering shared features boosts reasoning-style outputs.

Critical strengths
  1. A circuit-level model-diffing method: block-output differences become source nodes in one attribution graph, linking parameter updates to logits.

  2. A causal intervention: swapping in post-trained blocks at a few hundred top-ranked positions recovers most of the post-trained accuracy, while random ∆-activations do not.

Critical weaknesses
  1. The findings may be artifacts of the method. The shared circuit keeps only nodes present in all prompts, which favors the template tokens, since they are the only positions shared by every prompt. No control is reported (e.g., varied templates).

  2. Limited contribution. The only clear cross-model finding is that template tokens matter. The authors themselves attribute this to the template being constant during training, and possibly to known special-token effects such as attention sinks. It is unclear what the framework reveals that simpler analyses would not.

  3. Weak validation and overclaiming. The evidence comes from one base model, a self-trained 100-step GRPO run, and one toy SFT task with 50 test examples, yet the paper calls the result "a universal principle". There is no comparison with existing model-diffing methods.

Questions for authors
  1. Template ablation. If the template changed at test time, does the ∆-activation influence also peak, and does the accuracy recovery still hold?

  2. Comparison with other methods. How does the framework compare with existing model-diffing methods? What does the diff graph identify that these methods do not?

  3. CLT formulation in the main text. Section 3 describes the CLT only in words and defers its defining equations to Appendices A.2 and A.4. Could the authors integrate them into Section 3 to make the methodology self-contained?

  4. Self-containedness of the main text. The main text relies heavily on the appendix for core empirical claims,specifically the key evidence for Section 5.1 (Figs. 11a, 12a, 13a) and all SFT results (Figs. 9b, 10b, 12b), making the paper difficult to follow on its own. How do the authors plan to restructure the main text within the page limit to incorporate these essential results, and which current material would be condensed or moved to the appendix to make room?

4 comments

W1, Q1: The findings may be artifacts of the method; template ablation.

Thank you for raising this concern. We recognize that our description of cross-example aggregation and the definition of shared nodes are insufficiently clear, leading to this ambiguity. In Line 267, "Clustering across prompts" refers to clustering across the complete sequences associated with different questions, including both prompts and responses, rather than clustering only within the prompt region. Thus, the candidate nodes for the shared circuit cover the template, question, and response regions, rather than only the prompt.

The shared circuit contains Δ-activation nodes and feature nodes. Δ-activation nodes are defined at all positions, with bucketing used to cluster sequences of different lengths and content. Feature nodes are identified solely by layer and feature index, so tokens at different positions, such as template and response positions, can activate the same feature. Shared circuits can therefore occur at any position, rather than only within the prompt. Figure 4 illustrates this with shared features in the response region.

We appreciate you bringing this ambiguity to our attention and will clarify the wording in the paper, for example, by replacing it with "Clustering across examples". Furthermore, to address your concern, we have added two template ablation experiments in which we replace the template, and the findings remain consistent.

To address "does the Δ-activation influence also peak", we use an LLM to paraphrase the task instruction template (e.g., "Solve the following problem..."). We then repeat circuit tracing with these modified test-time templates using the same method as in the paper. Figure 18 in Appendix F.3 of the paper PDF shows that Δ-activation influence still peaks at template positions, even when the templates change.

To address "does the accuracy recovery still hold", we apply the same paraphrasing procedure and repeat the performance recovery experiment in Figure 6 with all other settings unchanged. Figure 19 in Appendix F.4 shows that progressively activating high-influence Δ-activations substantially improves performance toward that of the post-trained model. Activating fewer than 5% of all Δ-activation nodes achieves 84.8% accuracy, a 28.5 percentage point improvement over the base model.

W2, W3.2, Q2: Comparison with related work and limited contribution.

Thank you for raising these concerns. We will compare our work with related model-diffing methods and then emphasize our substantive contributions beyond prior work.

Venhoff et al. [1] propose constructive model diffing, reconstructing differences between base and thinking models through learned category vectors and heuristics determining when to apply them. We trace the local effects of actual parameter updates through intermediate features to outputs within one attribution graph, without fitting additional steering vectors.

Crosscoder methods [2,3] identify shared and model-specific features, including features with causal effects on behavior. Our framework additionally represents the local effects of parameter updates under fixed block inputs as explicit graph sources, connecting these updates to intermediate features and outputs.

Activation patching across models [4] can localize improved submechanisms responsible for performance gains. Our framework provides a complementary analysis at the feature level, explicitly linking update sources and their downstream propagation within one attribution graph.

Beyond simpler analyses, position importance alone does not identify which shallow updates influence which intermediate features and subsequent outputs. Our diff graph identifies these connections, while parameter replacement and feature steering test the identified sources and features, respectively. Template consistency or attention sinks alone does not specify this relationship between updates, features, and outputs. We will clarify these distinctions in the revised paper.

Compared with prior work, our substantive contributions are threefold.

(1) A general model-diffing framework. We extend circuit analysis from explaining model outputs to attributing the effects of parameter updates. We introduce Δ-activations under fixed block inputs as explicit sources in a unified graph connecting updates, intermediate features, and outputs. The formulation applies to transformer-based LLM checkpoints across model families, scales, and post-training methods. We demonstrate it on RL and SFT.

(2) Circuit tracing on real-world tasks. Beyond the short case studies in [5,6], adaptive tracing, source and path pruning, and memory optimizations extend our analysis beyond 500 tokens, supporting circuit analysis and intervention validation on realistic reasoning tasks such as AIME24 and MATH500.

(3) An explanation beyond template importance. Beyond the template importance established in [7,8], our circuits connect shallow parameter updates at template positions to sustained activation of shared features in the middle and later layers, which influence subsequent generation over long sequences. This explains how post-training acts through these positions and why they matter. Figure 2 summarizes this mechanism, supported by the localization, parameter replacement, and feature steering results in Figures 3–6.

We will release our framework’s code and trained CLT checkpoints to support further circuit research by the community.

[1] Base Models Know How to Reason, Thinking Models Learn When. ICML 2026

[2] Sparse Crosscoders for Cross-Layer Features and Model Diffing. Anthropic's Blog

[3] Robustly Identifying Concepts Introduced during Chat Fine-Tuning Using Crosscoders. ICLR 2025 Workshop

[4] Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking. ICLR 2024

[5] Circuit Tracing: Revealing Computational Graphs in Language Models. Anthropic's Blog

[6] Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders. EMNLP 2026

[7] Why Safeguarded Ships Run Aground? Aligned Large Language Models’ Safety Mechanisms Tend to Be Anchored in The Template Region. ACL 2025

[8] Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors. NeurIPS 2025 Workshop

W3.1: Overstated conclusions.

Thank you for highlighting the need to clarify the scope of our conclusions. We agree that the current RL and SFT experiments are insufficient to establish public-token localization as a “universal principle” across all post-training settings. We will narrow this claim in the revision, describing it as a consistent mechanism observed in the two settings studied. Nevertheless, we believe this finding is interesting and meaningful. It reveals how parameter updates at specific positions influence downstream outputs through intermediate features, providing a distinct and deeper explanation compared with existing methods (see our response to W2, W3.2, Q2).


Q3, Q4: Organization of the paper.

Thank you for these specific suggestions. We agree that the CLT’s core definitions and the key empirical evidence should appear directly in the main text. We will incorporate the CLT's defining equations into Section 3 and bring key RL and SFT results into the main text, including evidence linking shallow updates to shared features in the middle and later layers.

We will combine related results into compact figures with shared legends and axes. To meet the page limit, we will condense the long response regions in Figures 3 and 4 while retaining the full distributions in the appendix. Detailed graph quality metric definitions and result tables from Section 5.3.2 will also move to the appendix. The main text will retain the meanings of key metrics, main results, and explicit references. These changes will make the methodology and core empirical evidence understandable from the main text alone while preserving the complete validation materials.

Thank you for the detailed response and the added experiments. The Template-random-k control (Fig. 17) is useful. However, my main concerns remain.

Contribution. The response restates the framework's capabilities but does not show a finding that the feature-level graph yields and simpler analyses (position-wise block patching, crosscoders) do not. The comparison with these methods is still only in words.

Scope. I appreciate that the authors will narrow the "universal principle" claim, but the evidence still rests on one base model, one 100-step GRPO run, and one small SFT task.

For these reasons I keep my score.

Reviewer_r2shOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a circuit-based methodology to distinguish internal computations between base model and post-trained models, aiming to attribute post-training behaviour to parameter updates. The design is developed to trace where distinctions become apparent. Authors apply it to Qwen3-8B-Base and a GRPO-trained reasoning variant, and find that shallow updates on template tokens lead to deeper "reasoning features" activation. When blocks of base model during forward run are replaced with post-trained ones at certain template positions, it recovers most of RL gain.

Critical strengths

S1. The reverse-engineering of post-training is still unsolved problem to my knowledge, and the method construction is simple and fits existing circuit-tracing tools. S2. The patching result is interesting. Most of RL gain is reproduced by a few hundred block replacements, all at template positions, which suggests post-training may be strongly template-conditioned. However, this finding is not particularly novel as is.

Critical weaknesses

W1. The methodology is hard to transfer. The graph explains the post-trained model logits, but nothing verifies that it accounts for the difference between the two models on the logit difference level. Some conclusions follow from the construction itself. Selecting paths by maximum product favours short paths, and intersecting circuits across prompts favours template positions. With a transcoder trained only on the base model, the finding that the features are “already present within M_base” could not have come out otherwise. Applying the method to another model pair requires training a transcoder, and the user still cannot tell which conclusions reflect the models. W2. The value of the framework for the main finding is not shown, and the finding’s relation to prior work is unclear. The recovery experiment replaces blocks at nodes the graph ranks highest, but compares them with random nodes drawn from the whole sequence rather than from template positions. It therefore does not separate the graph ranking from the choice of positions. Recovering RL gains by steering a base model at selected positions (Venhoff et al. cited in paper) and safety behaviour anchored in the template region ("Aligned Large Language Models’ Safety Mechanisms Tend to Be Anchored in The Template Region" by Leong et al.) are not compared with. There is also two nearby mechanistic papers: "Steering LLM Reasoning Through Bias-Only Adaptation" shows that RL gains might be recovered by training single steering vector per layer suggesting very simple alignment structure, and in subsequent paper "Small Vectors, Big Effects" they found close template-related behavior when training with the same methodology, that is prominent on first answer token right after the template. Perhaps some more discussion of related work and its mechanistic accounting would be helpful. Also, one RL run and one synthetic SFT task do not support the "universal principle" conclusion. I am willing to assess the paper positively but those points hold me back for now.

Questions for authors

Q1. Do the paths starting at the difference nodes account for the logit difference between the two models? At what share? How does recovery compare with random nodes restricted to template positions? Q2. What does your work establish beyond related literature mentioned above, have you compared in your preliminary studies? What would let others apply the method to new model pairs with confidence in its conclusions? Q3. There are known leakage problems in Qwen family (see "Spurious Rewards" and subsequent line of research on Qwen 2.5) that could make results be dependent on some underlying data or training property specific to the Qwen family, not the RL training itself. Have you considered this?

5 comments

W1.1, Q1.1: Can the paths explain the logit differences, and to what extent?

Thank you for this constructive comment. Our target logit nodes are selected at positions where the base and post-trained models make different predictions given the same token prefix. We decompose each target logit value into contributions from sources including embeddings and attention and MLP Δ-activations. In our attribution graph, Δ-activations can contribute to target logits both directly and through intermediate features. Table 4 shows that parameter updates outside transformer blocks, such as embeddings, have negligible effects on output behavior. Therefore, the vast majority of behavioral differences arise from attention and MLP Δ-activations, the starting points of all paths in our diff graph. Once these nodes are activated, the paths originating from them largely determine the logit differences.

We fully agree that direct experimental validation of logit differences is necessary. Beyond the guarantees provided by the attribution construction, we added a logit recovery experiment to quantify how much of the actual logit differences the identified updates account for.

Following exactly the Top-k design in Figure 6, we progressively activate the highest-influence Δ-activation nodes in the base model, measuring logit mean absolute error (MAE) instead of benchmark performance. On GSM8K, we first generate each response using the post-trained model, then feed it into the intervened model token by token. Wherever their predicted tokens differ (diff positions), we take the union of their top-3 predicted tokens and compute the absolute difference between their logits over this set. We compute the mean absolute error over all selected logits at all diff positions across all samples.

The results, added as Figure 16 in Appendix F.1 of the paper PDF, show substantial logit differences between the base model (k = 0) and the post-trained model. These differences decrease significantly as more nodes are activated, closely matching the performance recovery trend in Figure 6. Activating all template nodes (less than 10% of the total) reduces the average logit MAE by more than 50% relative to the base model.

Thus, activating the identified influential Δ-activation nodes (allowing their effects to propagate through downstream paths) significantly reduces logit differences. The identified nodes are effective both in recovering benchmark performance and at the logit level, providing direct empirical evidence addressing your question.


W2.1, Q1.2: The control group consists of nodes randomly sampled from the entire sequence. What if sampling is restricted to the template?

Thank you for suggesting this helpful control. Figure 6 evaluates the importance of the identified high-influence nodes through accuracy recovery. Our positional influence analysis covers both prompts and responses. Figure 3 also shows smaller influence peaks within the response, such as an influence fraction of approximately 0.006 near token index 293. Given the concentration of high-influence nodes at template positions, we progressively activate template nodes in descending order of influence and compare them with k randomly selected nodes from the entire sequence. Both interventions use exactly the same node budget, allowing a fair evaluation of our overall localization strategy relative to random selection across the full sequence. However, this does not separate the benefits of selecting template positions from ranking nodes within them.

Your suggestion provides a valuable baseline, and we therefore add a Template-random-k control (Figure 17 in Appendix F.2 of the paper PDF), randomly selecting k nodes from the same template candidate set as Top-k, keeping all other settings unchanged, and repeating across multiple random seeds. With the same budget of 800 nodes, Top-k achieves 88.1% accuracy on GSM8K, compared with a mean accuracy of 67.4% for Template-random-k (mean@32), an improvement of 20.7 percentage points. At k = 2196, both methods activate all template nodes and are therefore identical, but performance improves substantially more slowly with Template-random-k as k increases. This shows that the graph’s influence ranking identifies more effective update nodes even when the candidate set is restricted to the template region.

W1.2: Limitations of the method’s construction itself and the need to train a transcoder.

Thank you for raising these concerns. Our framework is built on standard practices in mechanistic interpretability:

Path selection. Multiplicative path scoring and selecting high-attribution paths have precedents [1,2]. Our pruning follows these principles, although we acknowledge that the resulting path-length distribution is sensitive to the selection criterion.

Shared circuits. Circuit intersections across settings are used to identify recurring mechanisms [3,4]. Our candidate nodes cover complete sequences, and shared features are matched by layer and feature index, not fixed token positions. Figure 4 also contains shared response-region features. This construction does not exclude non-template positions, although statistical selection bias remains possible.

Transcoder training. Trained feature dictionaries are standard and indispensable in SAE- and transcoder-based interpretability [1,2,5]. Our method does not require a separate transcoder for every model pair. Since the CLT decomposes the base model’s computations, different post-trained variants of the same base model can share it, amortizing the training cost.

We acknowledge that standard practices have limitations, including methodological biases and decomposition errors, and cannot guarantee the absolute correctness of conclusions. We therefore perform parameter replacement and feature interventions in the original models to test recovery at the identified update locations and the relevant features’ effects on output behavior, respectively. The results consistently support our confidence in these conclusions.

We will explicitly discuss these limitations in the revision. We will also narrow the scope of our claims about features being "already present within MbaseM_{\mathrm{base}}." Our results show that post-training strengthens the activations of several base-model features, but do not establish that post-training introduces no new features or mechanisms. However, our framework does not force all update effects to be explained through the base-model feature dictionary. Direct paths from Δ-activation nodes to logits (Figure 8) allow parameter updates to contribute to outputs without mediation by intermediate CLT features. This extends the analysis beyond changes in the identified base-model features.

[1] Circuit Tracing: Revealing Computational Graphs in Language Models. Anthropic's Blog

[2] Transcoders Find Interpretable LLM Feature Circuits. NeurIPS 2024

[3] Finding Transformer Circuits with Edge Pruning. NeurIPS 2024

[4] The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It. EMNLP 2025

[5] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. ICLR 2025

W2.2, Q2.1: Comparison with related work.

Thank you for highlighting these related works. We will compare our work with related works and then emphasize our substantive contributions beyond prior work.

Venhoff et al. [1] recover reasoning performance in base models through learned category-specific steering vectors and a mechanism determining when to apply them. We trace the original post-training updates through intermediate features to outputs, directly connecting behavioral recovery to actual parameter updates without fitting additional steering vectors.

Leong et al. [2] establish template importance for safety through activation interventions between harmful and harmless inputs within the same model. Our comparison of base and post-trained models on identical inputs isolates the local effects of parameter updates and traces their downstream influence, explaining how post-training changes behavior.

Bias-Only Adaptation [3] shows that training per-layer bias vectors while freezing base weights recovers substantial RL gains. We analyze actual attention and MLP updates in existing post-trained checkpoints, without requiring post-training to be restricted to additive vectors.

Small Vectors, Big Effects [4] analyzes the cross-layer propagation, first-token effects, and MLP-mediated mechanisms of learned steering vectors. We connect actual post-training updates, shared features, and subsequent outputs within one graph, providing feature-level explanations across layers and positions and identifying update locations for direct parameter replacement.

Overall, prior work has shown that a small set of parameter updates or activation interventions can improve reasoning benchmark performance and recover a substantial fraction of RL gains [1,3,4]. The template region and the first generation position immediately following it play important roles in post-training behavior [2,4]. On toy cases spanning a few dozen tokens, circuit analysis has identified parallel pathways for factual recall and addition [5], and shared multilingual representations with language-specific decoding [6]. We will discuss these works in the paper.

Compared with prior work, our substantive contributions are threefold.

(1) A general model-diffing framework. We extend circuit analysis from explaining model outputs to attributing the effects of parameter updates. We introduce Δ-activations under fixed block inputs as explicit sources in a unified graph connecting updates, intermediate features, and outputs. The formulation applies to transformer-based LLM checkpoints across model families, scales, and post-training methods. We demonstrate it on RL and SFT.

(2) Circuit tracing on real-world tasks. Beyond the short case studies in [5,6], adaptive tracing, source and path pruning, and memory optimizations extend our analysis beyond 500 tokens, supporting circuit analysis and intervention validation on realistic reasoning tasks such as AIME24 and MATH500.

(3) An explanation beyond template importance. Beyond the template importance established in [2,4], our circuits connect shallow parameter updates at template positions to sustained activation of shared features in the middle and later layers, which influence subsequent generation over long sequences. This explains how post-training acts through these positions and why they matter. Figure 2 summarizes this mechanism, supported by the localization, parameter replacement, and feature steering results in Figures 3–6.

We will release our framework’s code and trained CLT checkpoints to support further circuit research by the community.

[1] Base Models Know How to Reason, Thinking Models Learn When. ICML 2026

[2] Why Safeguarded Ships Run Aground? Aligned Large Language Models’ Safety Mechanisms Tend to Be Anchored in The Template Region. ACL 2025

[3] Steering LLM Reasoning Through Bias-Only Adaptation. EMNLP 2025.

[4] Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors. NeurIPS 2025 Workshop

[5] Circuit Tracing: Revealing Computational Graphs in Language Models. Anthropic's Blog

[6] Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders. EMNLP 2026

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Our Contributions in Relation to Prior Work.

We thank the reviewers for highlighting related work and encouraging us to clarify our contributions.

Prior work demonstrates reasoning recovery through steering [1,2,3], template importance [4], and feature or submechanism differences [5,6]. We explicitly connect actual parameter-update effects, intermediate features, and outputs within one attribution graph, without fitting additional steering vectors. Beyond layer-wise feature analyses [7] and short circuit case studies [8,9], we trace how updates affect downstream features and generation over sequences exceeding 500 tokens.

Compared with prior work, our substantive contributions are threefold.

(1) A general model-diffing framework. We extend circuit analysis from explaining model outputs to attributing the effects of parameter updates. We introduce Δ-activations under fixed block inputs as explicit sources in a unified graph connecting updates, intermediate features, and outputs. The formulation applies to transformer-based LLM checkpoints across model families, scales, and post-training methods. We demonstrate it on RL and SFT.

(2) Circuit tracing on real-world tasks. Beyond the short case studies in [8,9], adaptive tracing, source and path pruning, and memory optimizations extend our analysis beyond 500 tokens, supporting circuit analysis and intervention validation on realistic reasoning tasks such as AIME24 and MATH500.

(3) An explanation beyond template importance. Beyond the template importance established in [3,4], our circuits connect shallow parameter updates at template positions to sustained activation of shared features in the middle and later layers, which influence subsequent generation over long sequences. This explains how post-training acts through these positions and why they matter. Figure 2 summarizes this mechanism, supported by the localization, parameter replacement, and feature steering results in Figures 3–6.

We will clarify these comparisons in the revised paper. We will release our framework’s code and trained CLT checkpoints to support further circuit research by the community.

[1] Base Models Know How to Reason, Thinking Models Learn When. ICML 2026

[2] Steering LLM Reasoning Through Bias-Only Adaptation. EMNLP 2025.

[3] Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors. NeurIPS 2025 Workshop

[4] Why Safeguarded Ships Run Aground? Aligned Large Language Models’ Safety Mechanisms Tend to Be Anchored in The Template Region. ACL 2025

[5] Robustly Identifying Concepts Introduced during Chat Fine-Tuning Using Crosscoders. ICLR 2025 Workshop

[6] Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking. ICLR 2024

[7] Do Sparse Autoencoders Identify Reasoning Features in Language Models? ICML 2026

[8] Circuit Tracing: Revealing Computational Graphs in Language Models. Anthropic's Blog

[9] Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders. EMNLP 2026

AuthorsOpenReview ↗

Global Response to Reviewers’ Follow-up Comments

We thank all three reviewers for their follow-up comments. Reviewers r2sh and 6EAS question the depth and distinctiveness of our findings, while all three reviewers raise concerns about empirical support and generalizability. Reviewer r2sh also notes CLT training costs.

1. Mechanistic findings and comparison with prior methods.

Our circuits connect shallow updates at template positions, shared behavioral features, and generation hundreds of tokens later. For example, a first-person task framing feature activated at template positions influences outputs such as "I am stuck" much later. To our knowledge, this specific circuit-level connection has not been reported in prior model-diffing work. Steering the identified features only at public-token positions increases problem decomposition from 3% to 42% and self-doubt from 0% to 38% (Figure 5, Appendix C.2).

  • Position-wise block patching replaces activations at selected layers and positions to identify important components [1,2]. It does not itself resolve which features matter or how local parameter updates influence them.

  • Crosscoders jointly learn dictionaries that identify shared and model-specific features [3,4]. Dictionary comparison alone does not produce a directed graph connecting parameter-update sources, intermediate features, and logits.

Our result is this circuit-level model diff, with features and logits connected by attributed influences from Δ-activation sources. These approaches address different dimensions of analysis, so our conclusions cannot be directly compared with theirs.

Prior circuit studies have examined short toy cases, including factual recall and arithmetic [5] and multilingual completion [6]. Through adaptive tracing, source and path pruning, and memory optimizations, we extend circuit analysis to responses exceeding 500 tokens, enabling investigation of long-range effects in realistic reasoning tasks.

2. Empirical support and scope.

As reflected in our title, our primary contribution is a model-diffing framework, designed to understand how parameter changes during post-training affect model behavior. Our objective is not to train a stronger reasoning model. Its formulation compares attention and MLP blocks under fixed inputs and is independent of model family, scale, and post-training procedure for model pairs with unchanged architecture. RL and SFT demonstrate its explanatory value, rather than establish this applicability through empirical coverage.

Both settings produce substantial changes:

  • 100-step GRPO: GSM8K 52.46% → 93.97%, MATH-500 40.48% → 74.28%, and AIME24 4.01% → 30.36% (mean@64; Table 4).
  • SFT: French and uppercase response rates increase from 0% to 94% and 96%, respectively (Appendix B.3).

Approximation checks, parameter replacement, feature interventions, logit recovery, and the added controls support our interpretations within these settings.

3. CLT costs.

Our computational cost is substantial because we study an 8B model. We choose this scale to identify more generalizable features and observe long-range effects within reasoning circuits. Different post-trained variants of the same base model can reuse this CLT, amortizing the training cost.

Finally, regarding Reviewer r2sh’s request, our revisions have always been highlighted in blue since the revised PDF was uploaded. You can view these changes on pages 46–48.

[1] Towards Best Practices of Activation Patching in Language Models: Metrics and Methods.

[2] Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking.

[3] Sparse Crosscoders for Cross-Layer Features and Model Diffing.

[4] Robustly Identifying Concepts Introduced during Chat Fine-Tuning Using Crosscoders.

[5] Circuit Tracing: Revealing Computational Graphs in Language Models.

[6] Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.