- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a mechanistic model diffing framework for analyzing differences between base and post-trained LLMs. The proposed method first translates parameter differences into localized activation differences, then traces their effects through CLTs, and finally clusters circuits across inputs to identify shared mechanisms. The framework is validated on Qwen3-8B and using both GRPO and SFT as post-training variants.
Critical strengths
The paper studies model diffing and uses CLTs on isolated activation differences to reflect parameter changes, which I consider is a novel combination. The proposed framework is well-motivated and technically sound.
Critical weaknesses
-
The intro states that the framework can be applied to different base models and post-training variants. However, the experiments consider only Qwen3-8B and two post-training variants (GRPO and SFT). This makes it unclear whether the findings generalize across different model families, model sizes, and other post-training methods such as RLHF, DPO, or OPD.
-
One of the paper’s main findings is that the most influential activation differences are concentrated on fixed prefix/template tokens. However, this finding may be an artifact of using the same prompt template across training samples. I wonder whether the authors could vary/paraphrase the prompt templates across samples and examine whether the same public-token mechanism still persists.
Questions for authors
Recent work on LLM mechanistic interpretability has also found that early layers are dominated by lexical processing and has focused on middle-to-late layers for identifying reasoning-related features, e.g.,
Ma, George, et al. "Do Sparse Autoencoders Identify Reasoning Features in Language Models?."
Given the similar layer-wise findings, I wonder whether this paper identifies a general property of LLM representations that has already been observed in prior work, or whether model diffing provides additional insights beyond the existing literature. It would be helpful to clarify this distinction in the related work.
4 comments
W2: Does the public-token mechanism persist under template variation?
Thank you for this suggestion. We would like to clarify first that using a fixed task instruction template (e.g., "Solve the following problem...") during RL training is common practice, as exemplified by DeepSeek-R1-Zero [1]. Our analysis therefore examines a practically relevant training setup.
Moreover, our SFT experiment uses no additional system prompt or fixed task instruction template, retaining only the basic chat template, including markers such as <|im_start|>. These markers are part of the model’s native dialogue format, and we retain them following standard practice to preserve that format. Δ-activation influence still concentrates at these public-token positions, particularly in the chat template suffix. This shows that the localization phenomenon also occurs without a fixed natural-language task instruction.
To further examine robustness to template variation, we add two template ablation experiments.
First, we use an LLM to paraphrase the task instruction template. We then repeat circuit tracing with these modified test-time templates using the same method as in the paper. Figure 18 in Appendix F.3 of the paper PDF shows that Δ-activation influence still peaks at template positions, even when the templates change.
Second, we apply the same paraphrasing procedure and repeat the performance recovery experiment in Figure 6 with all other settings unchanged. Figure 19 in Appendix F.4 shows that progressively activating high-influence Δ-activations substantially improves performance toward that of the post-trained model. Activating fewer than 5% of all Δ-activation nodes achieves 84.8% accuracy, a 28.5 percentage point improvement over the base model.
[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
Q1: Additional insights beyond previously observed layer patterns, and comparison with prior work.
Thank you for highlighting this related work. We will compare our work with it and then emphasize our substantive contributions beyond prior work.
Ma et al. [1] examine whether SAE features selected by contrasting reasoning and non-reasoning text capture reasoning or linguistic correlates. Their analysis focuses on middle and later layers and shows that many candidate features remain sensitive to surface cues. Our study instead examines how actual post-training parameter updates influence features and downstream outputs.
Crucially, we distinguish where parameter updates act from where the affected features activate. Early layers performing lexical processing does not by itself explain which updates in these layers affect later features or subsequent generation. Our framework connects these quantities within one attribution graph, providing an account of post-training changes beyond the distribution of representations across layers. We will clarify this distinction in the related work.
Compared with prior work, our substantive contributions are threefold.
(1) A general model-diffing framework. We extend circuit analysis from explaining model outputs to attributing the effects of parameter updates. We introduce Δ-activations under fixed block inputs as explicit sources in a unified graph connecting updates, intermediate features, and outputs. The formulation applies to transformer-based LLM checkpoints across model families, scales, and post-training methods. We demonstrate it on RL and SFT.
(2) Circuit tracing on real-world tasks. Beyond the short case studies in [2,3], adaptive tracing, source and path pruning, and memory optimizations extend our analysis beyond 500 tokens, supporting circuit analysis and intervention validation on realistic reasoning tasks such as AIME24 and MATH500.
(3) An explanation beyond template importance. Beyond the template importance established in [4,5], our circuits connect shallow parameter updates at template positions to sustained activation of shared features in the middle and later layers, which influence subsequent generation over long sequences. This explains how post-training acts through these positions and why they matter. Figure 2 summarizes this mechanism, supported by the localization, parameter replacement, and feature steering results in Figures 3–6.
We will release our framework’s code and trained CLT checkpoints to support further circuit research by the community.
[1] Do Sparse Autoencoders Identify Reasoning Features in Language Models? ICML 2026
[2] Circuit Tracing: Revealing Computational Graphs in Language Models. Anthropic's Blog
[3] Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders. EMNLP 2026
[4] Why Safeguarded Ships Run Aground? Aligned Large Language Models’ Safety Mechanisms Tend to Be Anchored in The Template Region. ACL 2025
[5] Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors. NeurIPS 2025 Workshop
I thank the authors for their detailed rebuttal. While I appreciate the conceptual clarifications, I agree with Reviewer 6EAS concern that the empirical evidence remains limited. I have decided to maintain my original score.
W1: Generalization across model families, scales, and post-training methods.
Thank you for raising these concerns. Our formulation and derivations, based on [1], primarily operate on differences in attention and MLP block outputs under fixed inputs. Since post-training changes only the parameters while preserving the architecture, the same formulation applies to other base models and their post-trained versions with attention and MLP blocks. They are not tied to a particular model family, scale, or post-training method, allowing the framework to transfer to other transformer-based LLMs, including those obtained through RLHF, DPO, or OPD.
We acknowledge that the framework’s applicability alone does not establish whether our empirical findings generalize to these additional settings. Parameter replacement and feature interventions in the original models validate the identified update locations and the relevant features’ behavioral effects, respectively. These experiments support our conclusions within the studied settings and provide a practical procedure for assessing results on new model pairs. We will clarify both the framework’s applicability and the empirical scope of our findings in the main paper.
[1] Transcoders Find Interpretable LLM Feature Circuits. NeurIPS 2024