acceptodds
Under review as a conference paper at ICLR 2027

When Do Safety-Relevant Concepts Become Readable in the Training Process of an LLM?

Abstract

Language models in everyday products are expected to avoid toxic, harmful, or stereotyped text. The concepts behind such text take shape during pretraining, so monitoring training requires knowing when each appears. Existing studies follow a concept across saved checkpoints with the raw score of a readout, a method that separates texts with and without the concept using the model's activations: a linear probe or a few sparse autoencoder (SAE) features. However, a raw score cannot tell when the concept became readable, because the two kinds of text differ in their words and the score can be high before any learning. We show that this point in training can be found, and that raw scores miss it. We measure every readout against its own score at the first checkpoint, when the model is still nearly random, and date a concept when this gain stays above a threshold. On 32 concepts across the checkpoints of our primary model, the raw score of a linear probe would date nine to the start of training. We instead find that toxicity emerges first, early in pretraining (by step 1,000). At our threshold, harm categories, which mark the kind of harm a text describes, emerge later and only under the SAE readout, and stereotype (bias) concepts under neither. The date thus depends on the readout, which a single-readout study cannot see. At the final checkpoint, removing a few SAE feature directions reduces generated toxicity. A set of five directions fixed in advance passes checks on benign prompts and output diversity, replicating in every pre-registered combination of three behaviours (toxicity, insult, threat) and two sizes of the primary model (1.4B and 2.8B).

Primary area:
interpretability and explainable AI
Keywords:
interpretability, sparse autoencoders, linear probes, pretraining dynamics, concept emergence, language model safety, toxicity, activation steering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_joivOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

To what extent can concepts like toxicity, harmful content, social bias, or others be measured and manifested in the internal representations of a language model during its pretraining process regarding safety? The authors compare probing techniques and make corrections for initial performance to determine surface level word differences from potential learning. They discover that the toxicity is measurable early in training, and the results for harm categories and bias are related to the measurement method. They also delete specific SAE features to assess their impact on the generated text, and see that the toxicity also dips but with little evidence of concept specificity. Overall, the paper suggests a framework to measure, but it's not clear whether that will be a metric that translates into meaningful safety improvements.

Critical strengths

[S1] The paper says that raw probing scores can be misleading since concepts can be separable even at the initial training checkpoint. Multiple readout methods and correcting for initial performance give a careful framework to study representations during pretraining.

[S2] The authors don't just probe, they also remove some of the feature directions from the SAE and see how the toxicity, benign-prompt behaviour and output diversity change. This gives empirical evidence of an association between internal representations and model behaviour, but the specificity of the observed effects is not known.

Critical weaknesses

[W1] The paper is mainly presenting the merging of the different techniques used to probe the concepts (baseline correction, activation interventions), and the adding of a limitted conceptual novelty and unclear significance: to estimate when the concepts related to safety are readable during the pretraining. It doesn't, however, provide enough evidence of a new fundamental insight or the importance of the emergence dates for informational or practical purposes of model safety. So, the practical or scientific value of the suggested framework is not yet clear.

[W2] Limited evidence that the proposed emergence dates are in fact evidence for conceptual learning: If the initial-checkpoint score is subtracted, it does not necessarily reveal semantic learning but it can be conceptualized as changes in lexical statistics or other representational properties during training. The dates of emergence reported as such are therefore dependent on the specific readout, baseline and threshold used, and thus make the paper's central claims about the dates of emergence more of a supposition than a fact.

[W3] Not anchoring the proposed SAE interventions to a specific concept: Directions associated with concepts unrelated to the interventions also do a very good job of reducing generated toxicity, comparable to toxicities selected by directions in some settings. This diminishes the argument that the chosen SAE features have a definite causal role in toxicity generation. The observed decreases could be due to a more general disruption of representation, not specifically to manipulation of toxicity, if there were less specificity of controls.

[W4] Limited evidence of safety improvements and poor cost assessment of interventions: experiments are mostly conducted on toxicity-related generation attributes in a few model configurations, and lack evidence of improvements in refusal behaviour and safety in aligned assistants. In addition, the interventions raise held-out perplexity, which may cause a loss of language quality. The overall applicability of the proposed approach is not established by the evaluation and the costs are not clearly shown to be acceptable when compared to the safety benefits.

Questions for authors

[Q1] How do the authors ensure that the emergence dates are based on learning of safety-relevant concepts and not on changes in the other representational properties (lexical statistics)? Specifically, what is the sensitivity of the conclusions to selection of the baseline correction and emergence threshold?

[Q2] Having seen that directions that are unrelated can also significantly decrease toxicity generated, what evidence shows that the selected SAE directions do play a specific causal role in the generation of toxicity instead of causing broader representational disruption?

[Q3] The authors do not connect the emergence timing with a better safety or refusal behavior, how is the practical utility of deciding on the emergence of safety relevant concepts during the pretraining justified then?

[Q4] There is an increase in held out perplexity with the interventions that suggests some degradation of the language. What is the relationship between the various sizes of the models and experimental conditions, and the consistency of the trade-off between toxicity and language quality, and how does this trade-off shape the implications for practical utility of the proposed method?

1 comment
AuthorsOpenReview ↗

Response to Reviewer joiv

Thank you for the detailed review. We follow its numbering below.

W1. Novelty and significance. The paper does combine known tools. Its contribution is what the combination shows that no tool shows alone. Control tasks and random-encoder baselines already separate what a representation encodes from what a probe can extract (Section 2). Applied along a training trajectory and under more than one readout, the same idea shows how much a date owes to how it is measured. A raw score dates nine of the 32 concepts to the start of training. The same concept group receives different dates, or a date and none, under a probe and under an SAE readout. And at the threshold of 0.15, ten of the twelve group dates, four concept groups on three seeds, move between the original SAE dictionaries and those retrained on a constant token budget (Table 14). A study that follows the raw score of one readout cannot expose these dependences (Table 1).

W2 and Q1. Concept learning and sensitivity. The dates do depend on the readout, the baseline and the threshold, and this dependence is the result the paper reports. Section 1 says that a date "is well defined only for a stated readout", and Section 3.3 calls the first-checkpoint score "an operational baseline rather than an exact lexical/learned split". What a date states is a measured comparison: at that checkpoint and the next, a readout separates the two kinds of text better, by the stated margin, than the same readout did at the first checkpoint. It is not a claim of semantic learning, and we will reword the two sentences that can be read as such a claim.

On the first part of Q1, we ran a new analysis for this response with the TF-IDF classifiers of Appendix E, which see only the words or the characters of the same texts on the same folds. At the first checkpoint the linear probe is below both on 30 of the 32 concepts, and at the final checkpoint of layer 12 it is above both on 24. The probe passes the better classifier at step 1,000 for the toxicity group and at step 4,000 for the harm group, at layers 12 and 18 alike, and never for the bias group. Our response to Reviewer gd6K (Q1) has the table, the limits of this comparison and the two rewordings.

On the second part, Table 11 gives the share of bootstrap resamples in which each group is dated at 0.10, 0.15 and 0.20. Under the probe, harm is dated in every resample at 0.10, in about a quarter at 0.15 and in almost none at 0.20. Table 14 varies the threshold, the dictionary seed and the dictionary budget, which changes the first-checkpoint score, and the toxicity date stays between step 512 and step 2,000 in every cell. So the early toxicity date is stable and the harm date is not, as Section 5.3 reports.

W3 and Q2. Specificity. At 1.4B we have no evidence of a specific causal role. Section 5.5 says as much when it concludes that the criteria "measure the size and side effects of an edit, not whether the removed directions are specific to toxicity". We ran the control with unrelated concepts because random directions are a weak comparison: each of the five SAE subspaces we tested removes 2 to 8 percent of the activation variance, whichever concept it was selected for, against about 0.2 percent on average for five random directions (Appendix G.4). In that control, which uses a second prompt set (Table 18), the toxicity subspace cuts toxicity by 51 percent at 2.8B and the four control subspaces by 2 to 19 percent. This difference at the larger size is the only evidence of specificity we report.

W4, Q3 and Q4. Safety benefit and cost. The paper is a measurement study and does not propose removal as a safety method. Every model in it is a base model before alignment training, and the Limitations state: "Everything is pre-alignment; we claim nothing about refusal behaviour". The removal experiment, which asks whether the model relies on the directions a readout selects, covers toxicity, insult and threat at two Pythia sizes, and toxicity on OLMo-2.

On Q3, a date does not by itself make a model safer. It is useful earlier: it tells someone who monitors a training run from which checkpoint a probe has something to read, provided the date is right. The paper shows that a date obtained in the usual way can depend on how the dictionaries were trained. At the threshold of 0.15 the bias group was dated at step 2,000 on all three seeds, and it has no date once every dictionary is trained on the same number of tokens.

On Q4, Table 20 reports held-out perplexity for the methods of Table 3 at both sizes, and Section 6 names it as a cost that no criterion covers. On the test prompts of Table 3 the trade-off is more favourable at the larger size. At 1.4B the SAE subspace cuts toxicity by 38 percent and raises perplexity by 26 to 38 percent on the two held-out corpora. At 2.8B it cuts 41 percent at a cost of 15 to 20 percent. We will add the perplexity ratio to each row of Table 3.

Reviewer_gd6KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates when safety-relevant concepts become readable during training using linear probes and sparse autoencoders (SAEs). The authors compare different readouts across training checkpoints and measure gains over each readout’s initial performance to account for separability already present before any substantial learning. They also investigate how removing selected directions affects the final model’s behavior.

The main contribution is an evaluation protocol showing that conclusions about concept emergence depend substantially on the readout, the data split, and the training of the SAE itself.

Critical strengths

The question addressed by the paper is relevant, and I think the experiments identify several real problems in how concept emergence is evaluated. In particular, keeping related texts together during evaluation and controlling the SAE training budget are useful additions, as both affect the conclusions drawn from the results.

The authors also include controls that challenge their own interpretation, especially the comparison with directions selected for unrelated concepts. Together with the additional checks on generated text, this makes the paper useful as a methodological study. I appreciate that the authors also report limitations and unsuccessful interventions.

Critical weaknesses

Interpretation of readability results. The initial-checkpoint baseline is a useful control, but I am not sure that gains over it establish readability beyond surface statistics. They could also reflect improved access to lexical differences already present in the texts. The authors acknowledge this in Sec. 3.3 and App. E, but the introduction appears to claim more.

The dates also depend on the threshold and SAE training setup. Layer choice also matters here. Safe-SAIL [1] traces safety-related features across layers, while [2] study how harmful-content features influence downstream refusal features. Although these studies concern already trained models, I think they motivate checking whether the corrected emergence dates hold across layers. The authors acknowledge this limitation, but the constant-budget correction was only carried out for the primary layer.

Interpretation of interventions and overall focus. I am also not convinced that the refitted probe’s chance performance establishes concept erasure. In Tab. 17, the fixed probe still detects information after the LEACE edit. App. A discusses why the fitting procedure can produce this discrepancy, but the erasure claim needs clearer qualification or a check using separate data for fitting the eraser and the new probe. At 1.4B, unrelated directions produce similar toxicity reductions, which also limits claims about specificity.

More generally, I found the paper difficult to follow. The different analyses and SAE training schedules make it hard to see which results support each conclusion and how the separate intervention study fits the overall contribution.

Overall, I think the methodological findings are useful, but the stronger interpretations of readability and erasure remain insufficiently supported. Clarifying these points and presenting the contribution more coherently could change my assessment.

[1] Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Framework, Weng et al., ACL Findings 2026

[2] Understanding Refusal in Language Models with Sparse Autoencoders, Yeo et al., ACL Findings 2025

Questions for authors

Clarifications

  1. Given the qualifications in Sec. 3.3 and App. E, what evidence distinguishes an increase over the initial-checkpoint baseline from improved access to lexical differences? It would be helpful to clarify how far the authors intend the results to support claims about concept emergence beyond those differences, and to make the introduction consistent with that scope.
  2. App. A explains that the eraser and refitted probe use the same training-fold activations. Could the authors report a check in which eraser fitting, probe refitting, and final evaluation use three separate sets of source groups? This would help assess which claims about concept erasure remain supported, especially given the fixed probe’s residual discrimination in Table 17.
  3. The layer comparison in App. D uses the original SAE training schedule, whereas the main timeline uses the corrected training budget. Do the authors have a corresponding comparison across layers with a constant budget? If not, it would be helpful to clarify which timing conclusions are only established for the primary layer.

Suggestions for Future Work

I was also wondering how the method would behave in a controlled experiment where safety-related content is introduced at a known point during training. Comparing otherwise matched runs with different introduction times, and one without the additional content, could show whether the estimated emergence dates shift with exposure. I think this would help clarify what the measure tracks, although it would not establish exactly when a concept is learned.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gd6K

Thank you for the careful review.

Q1. A gain over the baseline, or better access to lexical differences?

The gain alone does not distinguish them, and Section 3.3 calls the first-checkpoint score "an operational baseline rather than an exact lexical/learned split". For this response we compared the linear probe with the two text-only TF-IDF classifiers of Appendix E, fitted on the same texts and folds. At the first checkpoint the probe is below both on 30 of the 32 concepts. The table dates each concept group at the first checkpoint where the probe's median margin over the better classifier is positive and remains so at the next checkpoint.

Concept groupDate, the same at layers 12 and 18Median AUROC margin, final checkpoint, layer 12
Toxicitystep 1,0000.05 above
Harmstep 4,0000.03 above
Biasnever0.04 below

A rise up to the level of these classifiers can be explained by better access to which words are present, and we claim nothing more for it. Toxicity and harm go above that level, and bias does not. The classifiers are not an upper bound on lexical information, so we will call this readability beyond a text-only reference, not semantic learning. The paper's rule asks for a gain of 0.15 over the probe's own first-checkpoint score, which is already high for several harm categories, and gives harm no date under the probe (Table 14). Section 5.2 will report both rules.

In the introduction, "readable beyond surface statistics" will become "readable beyond what the first checkpoint already separates". In Section 2, "only the three together on one trajectory attribute a date to learning" will become "the three together on one trajectory date a change during training, under a stated readout and baseline".

Q2. Three separate sets of source groups.

We ran this check on the final checkpoint of Pythia-1.4B at layer 12, and the refitted probe is no longer at chance. With LEACE fitted on two folds, the probe refitted on a third and evaluated on the fourth, its AUROC averages 0.737 over the 32 concepts and the fold assignments, against 0.870 without the edit. The table gives this average for each edit. In both new columns the edit, the refit and the evaluation share no source group.

EditSame folds (Table 17)Edit on two folds, probe on a thirdEdit on one fold, probe on two
No edit0.8840.8700.881
SAE top-5 subspace0.8750.8560.871
LEACE0.5020.7370.828
Mean-difference direction0.5000.7220.824

So a refit at chance on the edit's own folds does not mean that no linear scorer separates the edited activations. The order of the edits is the same in all three columns, and the SAE subspace costs a refit probe almost nothing. We will not describe LEACE or the mean-difference removal as erasing the concept, and Table 17 will carry the two new columns.

In the Section 5.5 sentence on both edits, "LEACE, an affine eraser, holds the refit at chance" will become "LEACE costs a refit probe the most. The probe is at chance when refitted on the folds used to fit the edit, and reads the concept again, at a lower AUROC than without the edit, when refitted on separate source groups". The sentence after Table 3 that begins "Erasing the concept", and the others on these two edits, will be reworded in the same terms.

On specificity at 1.4B we agree, and Section 5.5 concludes that the criteria "measure the size and side effects of an edit, not whether the removed directions are specific to toxicity".

Q3. Is there a constant-budget comparison across layers?

No. Only at layer 12 were the early dictionaries retrained on the same number of tokens as the late ones, so the SAE dates of Section 5.3, and the finding that the budget moves them, are established for the primary layer only. Section 5.4 will say so. Timing results that use no dictionary do hold at layer 18: at a gain threshold of 0.15 the probe dates toxicity at step 1,000 at both layers and gives harm and bias no date at either (Table 7).

Which result supports which conclusion. Section 5 will open with a map from each conclusion below to its evidence, and will state which SAE schedule each part uses. First, a date must be stated with its readout, that readout's first-checkpoint score and, for an SAE readout, the dictionary's token budget (Table 2, Fig. 3, Table 14). Second, on Pythia-1.4B toxicity is dated before harm and bias, and whether those two receive a date depends on the readout, the threshold and the budget (Fig. 3, Tables 7, 11 and 14). Third, the removal study tests difficulty D4 of Section 1 and concludes that how well a direction reads does not tell what removing it does to generated text (Tables 3, 17 and 18).

We will cite [1] and [2] in Section 2 on the question of layers, and the Discussion will describe the experiment you suggest as validation still to be done.

Thank you for the response. Some of my concerns about the interpretation of the results remain.

The new experiment with separate source groups reinforces my concern about concept erasure. The much higher refitted AUROC for LEACE and mean difference removal shows that the original chance level performance does not establish erasure on independent data. I appreciate that the authors acknowledge this and intend to revise the corresponding claims.

Is an updated manuscript planned for closer inspection?

As of now, I lean towards maintaining my score, as the rebuttal has made it clear that the claims and contribution are much narrower than presented in the manuscript.

AuthorsOpenReview ↗

Response to Reviewer gd6K

Thank you for following up. We're preparing to revise the manuscript and have outlined the changes we'd make to Section 5.5 and Table 17. We don't have a date for sharing a revision.

Your question about independent data prompted us to test the erasure claim directly. We fitted each edit on two folds, trained a fresh probe on a third and tested it on the fourth, keeping the source groups separate. Across the 32 concepts, mean AUROC was 0.737 after LEACE and 0.722 after mean-difference removal, compared with 0.870 without an edit. Both edits made the signal harder to read, but a fresh probe could still pick it up on independent data.

Reviewer_EHKKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors use linear probes and SAEs to analyze the emergence of readable features corresponding to safety-related concepts during pre-training.

Critical strengths

The paper investigates the origins of representations of safety-related concepts during training. This has potential applications in early interventions to make models safer.

Critical weaknesses

The paper's headline question---when do safety-related concepts become readable during pretraining---is not really answered in a meaningful way, and I'm not sure its a very interesting question to begin with. They show that it seems to emerge around 10^3---10^4 steps, but their definition of "becomes readable" depends on the choice of threshold and classifier. As the paper points out, lexical features are already sufficient to read concepts to some degree at the start of training, so what they really measure is when the strengthening features cross an arbitrary threshold. The fact that early on, as the model learns to represent deeper features of natural language, the readability of safety concepts increases, is not very surprising to me. Not all papers need to have surprising results, but the alternative hypothesis in this case, that readability only becomes available at the end of training, seems highly unlikely. I'm not sure what we gain from confirming this.

The comparison between SAEs and linear probes does not seem to add much to the existing literature comparing the two methods, as far as I can tell.

Overall, I do not think that the paper rises to the standard of the conference in terms of scientific contribution.

Questions for authors

Non-critical feedback (does not factor into my assessment, putting here because there is no other field for this): There is no gap between paragraphs or indent on the first line of paragraphs.

4 comments
AuthorsOpenReview ↗

Response to Reviewer EHKK

Thank you for the review and the clear statement of your concerns. We answer them in order.

1. The date "depends on the choice of threshold and classifier". We agree, and measuring this dependence is the paper's main claim. Section 1 states that a date "is well defined only for a stated readout", together with that readout's first-checkpoint score and, for an SAE readout, the token budget of its dictionary. The experiments measure how far a date moves with each choice.

  • Raw score. After one gradient step from random weights the linear probe already separates nine of the 32 concepts with AUROC above 0.8 (Fig. 3a), so a raw score dates all nine to the start of training.
  • Readout. A concept group is dated when its median gain over the first-checkpoint score exceeds 0.15 at two successive checkpoints. By this rule the harm group has no date under the linear probe. Under the SAE readout its date lies between step 2,000 and step 32,000, depending on the dictionary seed (Table 14).
  • Dictionary budget. When the dictionaries of early checkpoints were trained on fewer tokens than those of late ones, the bias group was dated at step 2,000 on all three seeds. With the same budget for every dictionary it has no date on any seed at this threshold.

The value 0.15 is varied as well: Section 5.3 repeats the dating at 0.10 and 0.20 with a bootstrap over concepts and folds (Fig. 3c, Table 11).

2. The question "is not really answered in a meaningful way". For toxicity it is. On Pythia-1.4B the toxicity group is dated between step 512 and step 2,000 of 143,000, and the date stays in this range under both readouts, three dictionary seeds and all three thresholds (Table 14). For harm and bias, whether a date exists and where it falls depend on these choices. With equal budgets, across the two readouts and the three thresholds, harm is dated between step 1,000 and step 32,000 or not at all. One thing is stable for harm too: at these three thresholds, wherever it has a date, that date is later than the toxicity date. A study with a single readout could have reported any one of these outcomes as the answer.

3. "the alternative hypothesis in this case, that readability only becomes available at the end of training, seems highly unlikely". We also find that alternative unlikely, and the paper does not set out to reject it. What Table 5 shows is a rise that is far from uniform across the safety concepts. Under the linear probe all seven toxicity axes are dated by step 1,000. Of the 13 harm categories, six are dated between step 2,000 and step 32,000, and the other seven never sustain a gain of 0.15. None of the six CrowS-Pairs bias axes does, and the gender axis ends at AUROC 0.640 after 143,000 steps, from 0.633 at the first checkpoint.

4. "what they really measure is when the strengthening features cross an arbitrary threshold". The threshold is a choice, which is why point 1 varies it. For this response we also ran a dating rule that does not use the value 0.15. Appendix E describes TF-IDF classifiers that see only the words or the characters of the same texts, on the same folds, and we asked from which checkpoint the probe beats the better of the two. The toxicity group passes this reference at step 1,000 and the harm group at step 4,000, and the bias group never does. This rule again places toxicity first and harm later, and gives bias no date. The table is in our response to Reviewer gd6K (Q1).

5. "I'm not sure its a very interesting question to begin with" and "I'm not sure what we gain from confirming this." Studies that follow features across checkpoints already report when they form, and a formation time is read as a statement about what the model has learned by that step. The paper shows that a date obtained in the usual way can differ by more than a factor of ten between dictionary seeds, as it does for harm, or can disappear when the dictionaries are trained on equal budgets, as it does for bias. Anyone who wants to know from which checkpoint a probe-based monitor has something to read needs dates without these problems. The removal experiment adds that reading a concept well does not tell which directions can be removed without side effects (Table 3).

6. "The comparison between SAEs and linear probes does not seem to add much to the existing literature". For the final checkpoint this is right, and Section 2 says that "we find the same" as the existing comparisons. The comparison is in the paper for other reasons. The two readouts date the same concept group differently, as the harm example in point 1 shows, which a final-checkpoint study cannot see. And the way folds are built can decide such a comparison: when the two sentences of a minimal pair fall in different folds, a linear probe can score below chance, 0.36 on gender and 0.39 on age in Table 4.

7. Paragraph spacing. We will restore the template's paragraph spacing in the revision.

This response, like the paper, has a strong AI-generated smell. I have low confidence that engaging with the response is worth the effort, given that I see no signs of effort on the part of the authors. Here is my ai agent's response.

Thank you for the detailed response and the new analyses.

The TF-IDF comparison is a real improvement. Dating a concept by when the probe beats a text-only classifier is more meaningful than a fixed 0.15 gain over the first checkpoint. If the paper is revised, I'd suggest making this the main dating rule.

But the response confirms my main concern. Its central claim is now that a date "is well defined only for a stated readout," and that dates move with the readout, threshold, dictionary seed and dictionary budget. That warns readers about measurement, but it doesn't answer the question in the title. What's left that holds across these choices is: • toxicity becomes readable within roughly the first 1% of training; • harm categories become readable later, if at all; • bias never clearly becomes readable.

I don't find this surprising, and the response doesn't show why it matters. The monitoring use case is still hypothetical. The response doesn't describe any training-time decision that would change depending on whether toxicity is readable at step 1,000 or step 4,000.

The specific ways the dates break also seem fairly expected: • SAE features depend on the seed and the training budget. • A minimal pair split across folds can push a probe below chance.

Both are worth noting, but I don't think they add up to a main contribution.

I agree with Reviewer gd6K that the rebuttal makes the claims narrower than the manuscript presents them. The promised changes are substantial: a new dating rule, rewritten erasure claims, and a new map from conclusions to evidence. The paper would need to be reviewed again with those changes in place.

I'm keeping my score.

Upon reflection, my response was perhaps overly harsh. I am overwhelmed with the high volume low-quality ai research outputs. If the authors would like to engage in good faith discussion on how to improve the paper without obvious ai-sounding text I am willing to engage.

AuthorsOpenReview ↗

Response to Reviewer EHKK

Thank you for coming back to the discussion. I understand how frustrating it is to put time into a review and feel that the authors haven't taken the same care with their reply. I'm sorry our wording gave that impression.

I sometimes use AI and translation tools to help me express myself in English. We ran the experiments and take responsibility for what we wrote about them. A specialist in our field also reviewed the English manuscript. I hope you'll judge the paper on whether the experiments support our conclusions.

On your question about steps 1,000 and 4,000, we haven't tested whether intervening at one step works better than intervening at the other. For bias, the choice we faced was whether to report a date at all. With our original dictionary budgets, the bias group crossed our threshold at step 2,000 on all three seeds. When we trained each dictionary on the same number of tokens, we saw no crossing on any seed. The model checkpoints were the same. Looking only at the first result, we would have said that bias became readable early in training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.