acceptodds
Under review as a conference paper at ICLR 2027

SFT Leaves a Reproducible Signature in Activation Geometry: Calibrating Post-Training Interpretability Against Run-to-Run Noise

Abstract

A model diff credits post-training with whatever separates a base checkpoint from one post-trained descendant, presuming a rerun would land close by. On the sparse co-activation graph of Qwen3-1.7B it does not: two LoRA-SFT reruns sit over half as far apart as either sits from the base. We score each base-to-post contrast against its own recipe’s run-to-run band, and show that under run exchangeability k runs certify no better than p ≤ 1/(k + 1) however large the effect, so the band’s z is a descriptive scale. SFT leaves a large, reproducible signature: base→SFT displaces the graph by 19.9 pairwise-distance SDs beyond the mean distance between four independent LoRA-SFT runs, and the pre-registered estimator replicates on Llama-3.1-8B with Llama-Scope. At an identical 29 steps, four SFT and four DPO runs occupy disjoint displacement ranges, a gap 5.3× either arm’s spread, without any band. Three other readings stop short of a verdict: DPO’s band membership is dose-dependent (0/4, 2/4, 4/4 contrasts outside at 12, 29, 60 steps) while its displacement stays flat; light GRPO is a secondary non-detection, not an absence, and we run no equivalence test; and the graph concentration read as an RL signature is already in the zero-RL base, and a dictionary-width control removes an apparent scaling law in it. A pre-registered causal sweep (three edge budgets, 4,816 held-out prompts) found the displaced sources load-bearing (12× a matched random control in mean, 3.5× in median at 10,000 edges), but their stage-specificity interaction reversed at two budgets and the third failed its faithfulness gate, so we claim calibrated structural displacement, not a changed mechanism. Model diffs should be read beside their recipe’s rerun band, within-band diffs as non-detections. Scope: the full detection protocol covers one backbone (Qwen3-1.7B), one SFT recipe and general-text probe states; the matched-base control replicates across three families.

Primary area:
interpretability and explainable AI
Keywords:
post-training, supervised fine-tuning, DPO, activation geometry, seed-noise null, calibrated evaluation, sparse autoencoders

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dvCSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the question of when model differences observed after post-training can be interpreted as reproducible characteristics of a training recipe. We construct a co-activation graph with a fixed base-trained SAE encoder and evaluate the difference between checkpoints using the matched-base control and same-recipe rerun band. They report reproducible SFT-induced displacements in Qwen and Llama models, while adopting a more cautious interpretation for results involving DPO and GRPO. Furthermore, the study investigates whether structures appearing as features of post-training originate from the base model and whether these displaced features serve stage-specific functional roles. The key contribution is an empirical protocol for evaluating evidence supporting post-training interpretability claims, which limits the conclusion to structural displacement based on causal results.

Critical strengths
  1. The scientific motivation for the research question is sound. It addresses the challenge of distinguishing between differences observed at specific checkpoints and the reproducible effects of a training recipe. An approach that explicitly considers the structure inherited from the base model and retraining variability provides a useful foundation for interpreting model differences.

  2. There is empirical evidence to support the key observations. The SFT results encompass both independent reruns and cross-backbone replications. The fact that the observations were maintained even when the measurement basis was fixed and the graph budget was changed strengthens the claim that displacement appears repeatedly under the studied conditions.

  3. The limitations of the interpretation are handled with care. The authors distinguish between descriptive distance and statistical significance, as well as between non-detection and equivalence. Furthermore, they conduct causal interventions and report findings that run counter to their proposed stage-specific interpretation. This commitment to aligning claims with the available evidence enhances both the credibility and the methodological value of the research.

Critical weaknesses
  1. Additional contributions compared to prior research have not been sufficiently established. Ding et al. [1] have already evaluated representation distance based on seed variation, Du and Nguyen [2] investigated the representational instability of fine-tuning, and Minder et al. [3] showed that narrow fine-tuning leaves a readable activation trace. The distinguishing feature of this paper is that it applies a recipe-specific variability reference to the same model-diffing estimator. However, it remains unclear what significant scientific conclusions this procedure enables beyond what has already been established in the literature. Directly comparing the interpretations derived by each approach from the same subject will strengthen the claim of novelty.

  2. The calibration provided by the experimental design is limited. The exchangeability proposition is based on assumptions that are not justified in an actual training setup, because the base checkpoint is the common starting point for all post-training runs. Although the authors acknowledge this limitation, the results do not establish a false-positive control for the base-to-post comparisons. While the rerun band provides a useful criterion for interpreting displacement relative to retraining variability, there is insufficient evidence to use it as a general statistical criterion for the training effect.

  3. The evidence is concentrated on a narrow set of post-training conditions. The strongest controlled result is the result of evaluating narrow function-calling fine-tuning with a general-text co-activation graph. While cross-backbone replication is meaningful, significant recipe differences remain in SFT/DPO comparisons, and GRPO references include various variability factors. Furthermore, the comparisons across different training stages lack matched reruns. Consequently, it remains unclear which findings generalize to broader training distributions and which are contingent upon specific data, optimization settings, or readout methods.

  4. The additional scientific understanding gained through displacement detection has not been sufficiently developed. The on-task readouts fail to pass positive control checks, and the causal sweeps do not support the hypothesized stage-specific interpretations. While these audit results are reported in a fair and appropriate manner, the practical significance of the reproducible SFT signature remains an open question. In addition, the manuscript acknowledges that the feature-level narrowing hypothesis under evaluation is the author's own extrapolation. The significance of this work would be clearer if the audit were to genuinely revise established interpretations or offer specific, testable predictions regarding learning processes or model behavior.

  5. The paper has too many distracting references to Python scripts, Markdown files, and JSON files.

[1] Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding Representation Similarity Through Statistical Testing. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 1556–1568, 2021.

[2] Yupei Du and Dong Nguyen. Measuring the Instability of Fine-Tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6209–6230. Association for Computational Linguistics, 2023. DOI: 10.18653/v1/2023.acl-long.342.

[3] Julian Minder, Clément Dumas, Stewart Slocum, Helena Casademunt, Cameron Holmes, Robert West, and Neel Nanda. Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences. In International Conference on Learning Representations (ICLR), 2026.

Questions for authors
  1. If this protocol is added to the existing seed-aware representation similarity or activation-difference approaches at the same checkpoint and probe, what scientifically significant conclusions change?

  2. What statistical guarantees, if any, can be established when a base checkpoint and its descendants are not exchangeable? Can you define a realistic null hypothesis without such assumptions and evaluate the test properties of the procedure?

  3. Which aspects of the differences between SFT and DPO persist even when key recipe components are controlled? Does the signature remain intact if the fine-tuning distribution is broadened or if the pretraining data is mixed?

  4. Considering both on-task and causal outcomes, what verifiable claims about task behavior can be derived from the detected signature? Can these claims be supported by task-relevant readouts that pass a positive control?

Reviewer_eaJaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

unclear since this is AI slop

Critical strengths

None.

Critical weaknesses

clear AI slop from very first sentence:

A model diff credits post-training with whatever separates a base checkpoint from one post-trained descendant, presuming a rerun would land close by.

Questions for authors

None

Reviewer_BZUuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tests whether changes in a top-10k-edge SAE co-activation graph between a base and post-trained model exceed differences between same-recipe seed reruns. After 29 steps of LoRA SFT on BFCL data, base-to-SFT displacement exceeds the seed-to-seed spread on Qwen3-1.7B, with replication on Llama-3.1-8B. LoRA-DPO shows smaller displacement that stays roughly flat across the tested training durations, while light GRPO mostly stays within its rerun band. The causal ablations do not establish a stage-specific circuit.

Critical strengths
  1. The paper targets an important question: do changes in SAE co-activation graphs exceed differences between reruns of the same fine-tuning recipe? The SFT result replicates across models and SAE families, and survives changes in edge budget and a weighted-cosine measurement. These checks strengthen the descriptive finding that the observed displacement exceeds the variation between SFT reruns.

  2. The base-model controls catch concentration already present before fine-tuning, while the controls for SAE sparsity and dictionary width identify artifacts that could otherwise be mistaken for substantive results. I appreciate the use of pre-registration and the truthful reporting of negative and mixed results, including the limits of the causal interpretation.

Critical weaknesses
  1. The inferential status and added value of the measurement remain unclear. Proposition 1 assumes exchangeability, but the base checkpoint is the common starting point of the fine-tuning runs, not another draw from their distribution. The authors acknowledge this, leaving the rerun band as a descriptive reference without an established calibration guarantee for the experiment. A random encoder also detects the SFT change, and comparisons with simpler measurements such as CKA, covariance shifts, or activation mean shifts are missing. These gaps limit the methodological contribution beyond reporting run-to-run variability.

  2. The empirical findings support a narrow conclusion. The SFT/DPO comparison differs in learning rate by 10×, as well as dropout and training data, so it does not isolate the training objectives. The replicated SFT result concerns LoRA, and the causal tests do not establish a stage-specific mechanism. Although these limitations are disclosed, they leave a reproducible descriptive displacement whose broader significance remains unclear. This limits the paper’s impact and generalizability in my assessment.

Questions for authors
  1. Given that the base checkpoint is not an exchangeable draw from the fine-tuning recipe, which inferential guarantees, if any, does Proposition 1 provide for the reported experiments beyond a descriptive comparison with the rerun band?
  2. How does the graph-based measurement compare with CKA, covariance shifts, or activation mean shifts when each is evaluated against same-recipe reruns? Does the graph offer better detection, or is its main advantage localization?
  3. Does the SFT/DPO separation persist when the learning-rate difference is controlled? How do the recipes compare in weight-update magnitude (‖ΔW‖) and output KL on the same probe set?
  4. Is there evidence that the calibrated SFT result extends from LoRA to full-parameter fine-tuning, for example on a smaller model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.