- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the question of when model differences observed after post-training can be interpreted as reproducible characteristics of a training recipe. We construct a co-activation graph with a fixed base-trained SAE encoder and evaluate the difference between checkpoints using the matched-base control and same-recipe rerun band. They report reproducible SFT-induced displacements in Qwen and Llama models, while adopting a more cautious interpretation for results involving DPO and GRPO. Furthermore, the study investigates whether structures appearing as features of post-training originate from the base model and whether these displaced features serve stage-specific functional roles. The key contribution is an empirical protocol for evaluating evidence supporting post-training interpretability claims, which limits the conclusion to structural displacement based on causal results.
Critical strengths
-
The scientific motivation for the research question is sound. It addresses the challenge of distinguishing between differences observed at specific checkpoints and the reproducible effects of a training recipe. An approach that explicitly considers the structure inherited from the base model and retraining variability provides a useful foundation for interpreting model differences.
-
There is empirical evidence to support the key observations. The SFT results encompass both independent reruns and cross-backbone replications. The fact that the observations were maintained even when the measurement basis was fixed and the graph budget was changed strengthens the claim that displacement appears repeatedly under the studied conditions.
-
The limitations of the interpretation are handled with care. The authors distinguish between descriptive distance and statistical significance, as well as between non-detection and equivalence. Furthermore, they conduct causal interventions and report findings that run counter to their proposed stage-specific interpretation. This commitment to aligning claims with the available evidence enhances both the credibility and the methodological value of the research.
Critical weaknesses
-
Additional contributions compared to prior research have not been sufficiently established. Ding et al. [1] have already evaluated representation distance based on seed variation, Du and Nguyen [2] investigated the representational instability of fine-tuning, and Minder et al. [3] showed that narrow fine-tuning leaves a readable activation trace. The distinguishing feature of this paper is that it applies a recipe-specific variability reference to the same model-diffing estimator. However, it remains unclear what significant scientific conclusions this procedure enables beyond what has already been established in the literature. Directly comparing the interpretations derived by each approach from the same subject will strengthen the claim of novelty.
-
The calibration provided by the experimental design is limited. The exchangeability proposition is based on assumptions that are not justified in an actual training setup, because the base checkpoint is the common starting point for all post-training runs. Although the authors acknowledge this limitation, the results do not establish a false-positive control for the base-to-post comparisons. While the rerun band provides a useful criterion for interpreting displacement relative to retraining variability, there is insufficient evidence to use it as a general statistical criterion for the training effect.
-
The evidence is concentrated on a narrow set of post-training conditions. The strongest controlled result is the result of evaluating narrow function-calling fine-tuning with a general-text co-activation graph. While cross-backbone replication is meaningful, significant recipe differences remain in SFT/DPO comparisons, and GRPO references include various variability factors. Furthermore, the comparisons across different training stages lack matched reruns. Consequently, it remains unclear which findings generalize to broader training distributions and which are contingent upon specific data, optimization settings, or readout methods.
-
The additional scientific understanding gained through displacement detection has not been sufficiently developed. The on-task readouts fail to pass positive control checks, and the causal sweeps do not support the hypothesized stage-specific interpretations. While these audit results are reported in a fair and appropriate manner, the practical significance of the reproducible SFT signature remains an open question. In addition, the manuscript acknowledges that the feature-level narrowing hypothesis under evaluation is the author's own extrapolation. The significance of this work would be clearer if the audit were to genuinely revise established interpretations or offer specific, testable predictions regarding learning processes or model behavior.
-
The paper has too many distracting references to Python scripts, Markdown files, and JSON files.
[1] Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding Representation Similarity Through Statistical Testing. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 1556–1568, 2021.
[2] Yupei Du and Dong Nguyen. Measuring the Instability of Fine-Tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6209–6230. Association for Computational Linguistics, 2023. DOI: 10.18653/v1/2023.acl-long.342.
[3] Julian Minder, Clément Dumas, Stewart Slocum, Helena Casademunt, Cameron Holmes, Robert West, and Neel Nanda. Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences. In International Conference on Learning Representations (ICLR), 2026.
Questions for authors
-
If this protocol is added to the existing seed-aware representation similarity or activation-difference approaches at the same checkpoint and probe, what scientifically significant conclusions change?
-
What statistical guarantees, if any, can be established when a base checkpoint and its descendants are not exchangeable? Can you define a realistic null hypothesis without such assumptions and evaluate the test properties of the procedure?
-
Which aspects of the differences between SFT and DPO persist even when key recipe components are controlled? Does the signature remain intact if the fine-tuning distribution is broadened or if the pretraining data is mixed?
-
Considering both on-task and causal outcomes, what verifiable claims about task behavior can be derived from the detected signature? Can these claims be supported by task-relevant readouts that pass a positive control?