- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper examines how recursive training on model-generated text changes internal representations and affects the reliability of sparse autoencoders. The authors study five training generations in Pythia-160M and Gemma-2-2B, using real-data controls with matched training budgets and optimization settings to compare frozen and fine-tuned SAEs. Frozen SAEs generally deteriorate relative to these controls, and although fine-tuning largely closes the reconstruction gaps, probes using a small number of latents remain impaired in Pythia, while Gemma shows more mixed results across layers and metrics. Together with a simplified linear-autoencoder analysis, these findings suggest that recovering reconstruction quality does not necessarily restore feature-level usefulness, motivating evaluation of how task-relevant information is distributed and accessed across SAE latents.
Critical strengths
- The real-data controls use the same training budgets and optimization settings, helping separate the effects of synthetic data from those of continued training more generally. Comparing frozen and fine-tuned SAEs further distinguishes the mismatch of the original dictionary from performance gaps that persist after adaptation.
- In Pythia, the reconstruction gap largely closes after fine-tuning, yet probes using only a few latents remain impaired, providing concrete evidence that reconstruction quality alone is insufficient to assess SAE adaptation. The authors also clearly report the mixed results across Gemma layers and metrics, allowing readers to assess the scope of this finding.
Critical weaknesses
- Appendix A.2 selects the top-k latents based on differences in mean activation between positive and negative examples, but does not specify whether activations are standardized or test sensitivity to alternative selection methods. The decline in sparse-probing accuracy therefore does not establish that task-relevant information has become distributed across more latents, as changes in activation scale or feature rankings remain plausible alternative explanations.
- A near-zero damage score for full-SAE probing indicates only that average accuracy is close to that of the real-data control at the same generation, not that task-relevant information has been fully preserved relative to the initial model. Without absolute accuracies and per-task results for both lineages across generations, it is difficult to determine whether both have deteriorated.
- The paper does not adequately specify the SAE fine-tuning protocol, including the input corpus, number of activation samples, learning rate, training steps, sparsity targets, threshold updates, and checkpoint selection criteria. Table 1 primarily describes language-model training, while the SAE configuration in Figure 2 and some evaluation settings in Figure 3 remain unclear. These omissions hinder reproducibility and make it difficult to assess whether the residual performance gaps depend on the adaptation and evaluation settings.
- Persistent sparse-probing degradation after fine-tuning is observed mainly in Pythia and is not consistently reproduced in Gemma, leaving the generality of this finding across models uncertain. Each model-training lineage also uses only one random seed, so the reported error bars capture variation in SAE fine-tuning rather than uncertainty in the recursive training process. The current evidence therefore supports an observation under the evaluated settings, but does not yet establish its robustness across independent training runs.
Questions for authors
- The authors should clarify whether activations are standardized before top-k latent selection and test whether the findings hold under a scale-invariant selection method.
- The paper should report absolute probing accuracies and per-task results for both training lineages, starting from generation 0. The authors should then clarify whether information preservation means no additional degradation relative to the matched real-data control or maintained task predictability relative to the initial model.
- The authors should provide the complete SAE fine-tuning protocol, including data sources, activation sample counts, training budgets, optimization parameters, sparsity and threshold settings, and checkpoint selection criteria.
- The authors should validate the key findings across independent model-training seeds and clarify how the differing results in Pythia and Gemma affect the scope of their conclusions. If additional runs are not feasible, claims about persistent sparse-probing degradation should be limited to the evaluated models and training settings, with an explicit statement that the reported error bars reflect only variation in SAE fine-tuning.