- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To what extent can concepts like toxicity, harmful content, social bias, or others be measured and manifested in the internal representations of a language model during its pretraining process regarding safety? The authors compare probing techniques and make corrections for initial performance to determine surface level word differences from potential learning. They discover that the toxicity is measurable early in training, and the results for harm categories and bias are related to the measurement method. They also delete specific SAE features to assess their impact on the generated text, and see that the toxicity also dips but with little evidence of concept specificity. Overall, the paper suggests a framework to measure, but it's not clear whether that will be a metric that translates into meaningful safety improvements.
Critical strengths
[S1] The paper says that raw probing scores can be misleading since concepts can be separable even at the initial training checkpoint. Multiple readout methods and correcting for initial performance give a careful framework to study representations during pretraining.
[S2] The authors don't just probe, they also remove some of the feature directions from the SAE and see how the toxicity, benign-prompt behaviour and output diversity change. This gives empirical evidence of an association between internal representations and model behaviour, but the specificity of the observed effects is not known.
Critical weaknesses
[W1] The paper is mainly presenting the merging of the different techniques used to probe the concepts (baseline correction, activation interventions), and the adding of a limitted conceptual novelty and unclear significance: to estimate when the concepts related to safety are readable during the pretraining. It doesn't, however, provide enough evidence of a new fundamental insight or the importance of the emergence dates for informational or practical purposes of model safety. So, the practical or scientific value of the suggested framework is not yet clear.
[W2] Limited evidence that the proposed emergence dates are in fact evidence for conceptual learning: If the initial-checkpoint score is subtracted, it does not necessarily reveal semantic learning but it can be conceptualized as changes in lexical statistics or other representational properties during training. The dates of emergence reported as such are therefore dependent on the specific readout, baseline and threshold used, and thus make the paper's central claims about the dates of emergence more of a supposition than a fact.
[W3] Not anchoring the proposed SAE interventions to a specific concept: Directions associated with concepts unrelated to the interventions also do a very good job of reducing generated toxicity, comparable to toxicities selected by directions in some settings. This diminishes the argument that the chosen SAE features have a definite causal role in toxicity generation. The observed decreases could be due to a more general disruption of representation, not specifically to manipulation of toxicity, if there were less specificity of controls.
[W4] Limited evidence of safety improvements and poor cost assessment of interventions: experiments are mostly conducted on toxicity-related generation attributes in a few model configurations, and lack evidence of improvements in refusal behaviour and safety in aligned assistants. In addition, the interventions raise held-out perplexity, which may cause a loss of language quality. The overall applicability of the proposed approach is not established by the evaluation and the costs are not clearly shown to be acceptable when compared to the safety benefits.
Questions for authors
[Q1] How do the authors ensure that the emergence dates are based on learning of safety-relevant concepts and not on changes in the other representational properties (lexical statistics)? Specifically, what is the sensitivity of the conclusions to selection of the baseline correction and emergence threshold?
[Q2] Having seen that directions that are unrelated can also significantly decrease toxicity generated, what evidence shows that the selected SAE directions do play a specific causal role in the generation of toxicity instead of causing broader representational disruption?
[Q3] The authors do not connect the emergence timing with a better safety or refusal behavior, how is the practical utility of deciding on the emergence of safety relevant concepts during the pretraining justified then?
[Q4] There is an increase in held out perplexity with the interventions that suggests some degradation of the language. What is the relationship between the various sizes of the models and experimental conditions, and the consistency of the trade-off between toxicity and language quality, and how does this trade-off shape the implications for practical utility of the proposed method?
Response to Reviewer joiv
Thank you for the detailed review. We follow its numbering below.
W1. Novelty and significance. The paper does combine known tools. Its contribution is what the combination shows that no tool shows alone. Control tasks and random-encoder baselines already separate what a representation encodes from what a probe can extract (Section 2). Applied along a training trajectory and under more than one readout, the same idea shows how much a date owes to how it is measured. A raw score dates nine of the 32 concepts to the start of training. The same concept group receives different dates, or a date and none, under a probe and under an SAE readout. And at the threshold of 0.15, ten of the twelve group dates, four concept groups on three seeds, move between the original SAE dictionaries and those retrained on a constant token budget (Table 14). A study that follows the raw score of one readout cannot expose these dependences (Table 1).
W2 and Q1. Concept learning and sensitivity. The dates do depend on the readout, the baseline and the threshold, and this dependence is the result the paper reports. Section 1 says that a date "is well defined only for a stated readout", and Section 3.3 calls the first-checkpoint score "an operational baseline rather than an exact lexical/learned split". What a date states is a measured comparison: at that checkpoint and the next, a readout separates the two kinds of text better, by the stated margin, than the same readout did at the first checkpoint. It is not a claim of semantic learning, and we will reword the two sentences that can be read as such a claim.
On the first part of Q1, we ran a new analysis for this response with the TF-IDF classifiers of Appendix E, which see only the words or the characters of the same texts on the same folds. At the first checkpoint the linear probe is below both on 30 of the 32 concepts, and at the final checkpoint of layer 12 it is above both on 24. The probe passes the better classifier at step 1,000 for the toxicity group and at step 4,000 for the harm group, at layers 12 and 18 alike, and never for the bias group. Our response to Reviewer gd6K (Q1) has the table, the limits of this comparison and the two rewordings.
On the second part, Table 11 gives the share of bootstrap resamples in which each group is dated at 0.10, 0.15 and 0.20. Under the probe, harm is dated in every resample at 0.10, in about a quarter at 0.15 and in almost none at 0.20. Table 14 varies the threshold, the dictionary seed and the dictionary budget, which changes the first-checkpoint score, and the toxicity date stays between step 512 and step 2,000 in every cell. So the early toxicity date is stable and the harm date is not, as Section 5.3 reports.
W3 and Q2. Specificity. At 1.4B we have no evidence of a specific causal role. Section 5.5 says as much when it concludes that the criteria "measure the size and side effects of an edit, not whether the removed directions are specific to toxicity". We ran the control with unrelated concepts because random directions are a weak comparison: each of the five SAE subspaces we tested removes 2 to 8 percent of the activation variance, whichever concept it was selected for, against about 0.2 percent on average for five random directions (Appendix G.4). In that control, which uses a second prompt set (Table 18), the toxicity subspace cuts toxicity by 51 percent at 2.8B and the four control subspaces by 2 to 19 percent. This difference at the larger size is the only evidence of specificity we report.
W4, Q3 and Q4. Safety benefit and cost. The paper is a measurement study and does not propose removal as a safety method. Every model in it is a base model before alignment training, and the Limitations state: "Everything is pre-alignment; we claim nothing about refusal behaviour". The removal experiment, which asks whether the model relies on the directions a readout selects, covers toxicity, insult and threat at two Pythia sizes, and toxicity on OLMo-2.
On Q3, a date does not by itself make a model safer. It is useful earlier: it tells someone who monitors a training run from which checkpoint a probe has something to read, provided the date is right. The paper shows that a date obtained in the usual way can depend on how the dictionaries were trained. At the threshold of 0.15 the bias group was dated at step 2,000 on all three seeds, and it has no date once every dictionary is trained on the same number of tokens.
On Q4, Table 20 reports held-out perplexity for the methods of Table 3 at both sizes, and Section 6 names it as a cost that no criterion covers. On the test prompts of Table 3 the trade-off is more favourable at the larger size. At 1.4B the SAE subspace cuts toxicity by 38 percent and raises perplexity by 26 to 38 percent on the two held-out corpora. At 2.8B it cuts 41 percent at a cost of 15 to 20 percent. We will add the perplexity ratio to each row of Table 3.