acceptodds
Withdrawn from ICLR 2027

Representational Change Under Self-Consuming Training: A Sparse Autoencoder Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models are increasingly exposed to model-generated text, raising the possibility that future models will be trained recursively on the outputs of their predecessors. Prior work on such self-consuming training loops has largely focused on output quality and downstream performance, leaving their effects on internal representations poorly understood. We study how representations evolve across five generations of self-consumption and whether sparse autoencoders (SAEs) remain reliable under this shift. At each generation, we compare a lineage trained on synthetic data with a matched control trained on real data under the same training settings. We evaluate how well SAEs trained before self-consumption transfer to later generations, measuring reconstruction and sparsity across SAE widths, architectures, and model layers. We then fine-tune the original SAE on each generation and use sparse probing to test whether task-relevant information remains concentrated in a small number of latents or becomes more distributed across the representation. We find that zero-shot SAE performance degrades progressively under self-consumption, with growing reconstruction and sparsity gaps that vary substantially across layers. Fine-tuning largely restores reconstruction, but not feature-level usefulness: task-relevant information remains accessible in the full latent space while becoming less concentrated in individual features. These results show that self-consumption induces representational drift and that reconstruction fidelity alone is insufficient to establish that an adapted SAE remains equally useful for interpretation.

Primary area:
interpretability and explainable AI
Keywords:
Self-consuming models, Sparse Autoencoders, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C8p5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines how recursive training on model-generated text changes internal representations and affects the reliability of sparse autoencoders. The authors study five training generations in Pythia-160M and Gemma-2-2B, using real-data controls with matched training budgets and optimization settings to compare frozen and fine-tuned SAEs. Frozen SAEs generally deteriorate relative to these controls, and although fine-tuning largely closes the reconstruction gaps, probes using a small number of latents remain impaired in Pythia, while Gemma shows more mixed results across layers and metrics. Together with a simplified linear-autoencoder analysis, these findings suggest that recovering reconstruction quality does not necessarily restore feature-level usefulness, motivating evaluation of how task-relevant information is distributed and accessed across SAE latents.

Critical strengths
  1. The real-data controls use the same training budgets and optimization settings, helping separate the effects of synthetic data from those of continued training more generally. Comparing frozen and fine-tuned SAEs further distinguishes the mismatch of the original dictionary from performance gaps that persist after adaptation.
  2. In Pythia, the reconstruction gap largely closes after fine-tuning, yet probes using only a few latents remain impaired, providing concrete evidence that reconstruction quality alone is insufficient to assess SAE adaptation. The authors also clearly report the mixed results across Gemma layers and metrics, allowing readers to assess the scope of this finding.
Critical weaknesses
  1. Appendix A.2 selects the top-k latents based on differences in mean activation between positive and negative examples, but does not specify whether activations are standardized or test sensitivity to alternative selection methods. The decline in sparse-probing accuracy therefore does not establish that task-relevant information has become distributed across more latents, as changes in activation scale or feature rankings remain plausible alternative explanations.
  2. A near-zero damage score for full-SAE probing indicates only that average accuracy is close to that of the real-data control at the same generation, not that task-relevant information has been fully preserved relative to the initial model. Without absolute accuracies and per-task results for both lineages across generations, it is difficult to determine whether both have deteriorated.
  3. The paper does not adequately specify the SAE fine-tuning protocol, including the input corpus, number of activation samples, learning rate, training steps, sparsity targets, threshold updates, and checkpoint selection criteria. Table 1 primarily describes language-model training, while the SAE configuration in Figure 2 and some evaluation settings in Figure 3 remain unclear. These omissions hinder reproducibility and make it difficult to assess whether the residual performance gaps depend on the adaptation and evaluation settings.
  4. Persistent sparse-probing degradation after fine-tuning is observed mainly in Pythia and is not consistently reproduced in Gemma, leaving the generality of this finding across models uncertain. Each model-training lineage also uses only one random seed, so the reported error bars capture variation in SAE fine-tuning rather than uncertainty in the recursive training process. The current evidence therefore supports an observation under the evaluated settings, but does not yet establish its robustness across independent training runs.
Questions for authors
  1. The authors should clarify whether activations are standardized before top-k latent selection and test whether the findings hold under a scale-invariant selection method.
  2. The paper should report absolute probing accuracies and per-task results for both training lineages, starting from generation 0. The authors should then clarify whether information preservation means no additional degradation relative to the matched real-data control or maintained task predictability relative to the initial model.
  3. The authors should provide the complete SAE fine-tuning protocol, including data sources, activation sample counts, training budgets, optimization parameters, sparsity and threshold settings, and checkpoint selection criteria.
  4. The authors should validate the key findings across independent model-training seeds and clarify how the differing results in Pythia and Gemma affect the scope of their conclusions. If additional runs are not feasible, claims about persistent sparse-probing degradation should be limited to the evaluated models and training settings, with an explicit statement that the reported error bars reflect only variation in SAE fine-tuning.
Reviewer_qA7rOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how recursive training on model-generated data affects language-model representations using sparse autoencoders (SAEs). The authors compare models trained on synthetic and real data across multiple generations and evaluate SAE transferability and adaptation. The results show that SAEs become less effective as self-consuming training progresses, while fine-tuning largely restores reconstruction quality. However, the recovery of reconstruction does not necessarily imply the recovery of feature-level usefulness.

Critical strengths
  1. Well-controlled experimental design. The paper introduces generation-matched real-data controls to distinguish the effects of self-consuming training from those of continued training. It also evaluates different SAE architectures, widths, and model layers, providing a relatively comprehensive experimental comparison.

  2. Relevant research question. The paper explores how recursive training affects internal representations and the reliability of SAE-based interpretability. This provides a complementary perspective to previous studies that primarily focus on model outputs and downstream performance.

Critical weaknesses
  1. Insufficient motivation and unclear significance of the contribution. The paper mainly motivates its research by identifying a gap in previous studies, without sufficiently explaining why SAE transferability under self-consuming training is particularly important. Moreover, prior work has already investigated SAE transferability and the limitations of reconstruction-based evaluation. The paper does not clearly distinguish its contributions from these existing findings.

  2. The central conclusions lack sufficient direct mechanistic evidence. The claim that task-relevant information becomes more distributed across SAE latents is primarily supported by sparse probing results. However, the paper does not directly analyze how individual features evolve, split, or change across generations. More direct feature-level evidence would strengthen this conclusion.

  3. The key findings lack consistency across models. The persistent degradation in sparse localization is mainly observed in Pythia, while Gemma shows more mixed results across layers and metrics. The causal intervention results are also inconsistent, raising concerns about the generalizability of the main conclusions.

  4. The statistical reliability of the experiments remains limited. Each model-training lineage uses only a single random seed. Although multiple seeds are used for SAE fine-tuning, the paper does not sufficiently examine the variability introduced by the recursive training process itself.

Questions for authors

None

Reviewer_s7P9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper looks at how SAEs transfer over continued training of a model, specifically comparing trajectories of models that are trained on new/fresh human text to models trained on their own output. The findings are largely what one would expect: SAEs do not transfer zero shot across this lifetime, but the performance can be recovered with finetuning. Transfer is better between checkpoints trained on new/fresh text than between a checkpoint trained on new text and a later one that has been trained on its own generations.

Critical strengths

The themes that are present in the paper are interesting. It's now the case that we can assume models will be trained on their own output so asking about the implications of this for interpretability in particular is important and potentially impactful.

Critical weaknesses

novel or consequential insight that we learn from these experiments?) and the other concerning the presentation (there is a lot of math that as presented reads like filler).

Contribution: My understanding of the paper’s primary finding is that SAEs will expire when models are continually trained, especially when those models are trained on llm-generated data. The scientific insight here is that continued training leads to representational drift, especially when models are trained on llm generated text. These findings would be interesting if this was a new discover, but they seem quite obvious. In particular, I’d love if the authors could explain how these results aren’t entailed by several well known facts, namely: That AI generated text (especially recursively consumed/generated text) differs from human written text distributionally. That the internal state of models will reflect the text they are trained on That AEs/SAEs reconstruct this internal state Given this, of course we should not see transfer when the internal state changes, and of course we should expect the internal state to change. It's not that there is any flaw in study per se but the finding seems obvious and thus particularly consequential.

There is an implied engineering/practical implication – that SAEs need to be finetuned to adapt to changing models. But this seems like its probably already best practice, and there is no major additional insight to the findings unless I am missing something.

On the presentation note: I think equations starting around page 4 feel a bit like “rigormaxxing” i.e. an instance of a trend myself and many colleagues have noticed in which papers seem to be including significantly more equations and proofs appearing that was the case before LLMs were able to assist with these. The result is that many papers include math that is not novel to the paper and doesn’t make a real contribution, but takes up a lot of space and reader effort. Unless I was missing something, the equations are just explaining prerequisite things that largely can be taken for granted – i.e., giving the reconstruction error of an AE, showing that an SAE will not work if the underlying space changes (this seems true by definition), etc. These are not proving a new claim or offering a new theoretical contribution or insight, so why are they here? If they are offering novel insights that can’t be inferred from the definitions of the models involved, then the authors should write more clearly to explain what this insight is and why it matters for the field more generally.

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.