acceptodds
Under review as a conference paper at ICLR 2027

Fighting the Drift: Probe Transfer Across Post-Training Stages

Abstract

Linear activation probes are used for monitoring large language models. Updating a probe for a new checkpoint requires re-extracting activations and may require new labels. First, we study probe transfer across SFT, DPO, and RL in OLMo-3 7B and across SFT and GRPO in Qwen3-4B. On the most disruptive SFT transfers, frozen probes achieve only about half the true positive rate of a probe refit on the target checkpoint, at 1% FPR. DPO and RL updates leave much less headroom for refitting than SFT updates. A refit on the target checkpoint can outperform the probe on its source checkpoint, so we measure transfer performance against the target refit. Label-free representation alignment can close much of this headroom but can plateau below the refit or perform worse than the frozen probe. Second, we develop an orthogonal drift correction that adjusts the old probe, with or without representation alignment, using a small number of labels. With a strong starting probe, correction achieves the best performance at most tested labeling budgets, can match the full refit, and can outperform a from-scratch refit trained on the same labeled data. Cheap probe transfer is a step toward overseeing systems that update faster than their monitors can keep up.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
linear probes, activation alignment, post-training, safeguards, LLM monitoring, AI safety, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xDeyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper considers the problem of how linear probes get affected by various training setups such as SFT, DOP and other RL methods. They find that SFT causes the most significant change in probe performance compared to DPO and RL. In the second half of the paper, the authors then propose methods to mitigate this drop in probing performance without having to obtain new activations and labels. Specifically, a label-free orthogonal drift correction mechanism is proposed in the paper.

Critical strengths

Given that probing is an established technique for checking for various useful properties of a model (although there are criticisms as to the validity of linear probing as a method for this such as https://arxiv.org/abs/2004.03061) the paper might be of interest to practitioners who would like to track a particular property of a model across multiple checkpoints without having to retrain the probe by extracting labelled activations every time.

Critical weaknesses

The paper is rather poorly written and the authors do not explain why it is important to study this issue and why one would expect a linear probe trained on one checkpoint to even generalise to another. Linear probes are trained for various purposes (as briefly mentioned in the first paragraph of the intro) for detecting various properties of a model such as its social biases. However, the idea is to study a specific model (checkpoint) and given that training (using any method) would substantially modify the weights of a model, there is no expectation for a linear probe to generalise (applicable from) one checkpoint to another. What benefits should we see if this happens? is something that the authors should have motivated well in the introduction. Although I can "guess" this because of my expertise of the domain, I think that should not be an excuse not to make this explicit in the introduction. I am not sure whether the paper has been written using LLMs, but it has not helped at all to make it more readable in this case.

Given that linear probing has been criticised and more complex non-linear probes have been recommended in prior work (https://arxiv.org/abs/2004.03061) I wonder what the value of the findings of this paper be. I have formulated this also as a question below.

The label-free methodology is effectively orthogonal procrustes, which has been widely applied in NLP/ML for cross-lingual or cross-domain feature alignment (see https://aclanthology.org/N15-1104/ for example). Can the authors explain the novelty of the method proposed in your work against this backdrop?

Questions for authors

Given that linear probing has been criticised and more complex non-linear probes have been recommended in prior work (https://arxiv.org/abs/2004.03061) I wonder what the value of the findings of this paper be.

The label-free methodology is effectively orthogonal procrustes, which has been widely applied in NLP/ML for cross-lingual or cross-domain feature alignment (see https://aclanthology.org/N15-1104/ for example). Can the authors explain the novelty of the method proposed in your work against this backdrop?

1 comment

Thank you for taking the time to review our paper and recognizing the practical value of probe transfer. We address each of your concerns below:

  1. “the authors do not explain why it is important to study this issue”

The motivation is already explicitly provided in the opening paragraphs of our introduction, which explain that probes are used in LLM monitoring, model updates can negatively affect probe performance, and retraining probes “is limited by the cost of generating new responses, labeling them and, in some cases, extracting terabytes of per-token activations.”

  1. “Linear probing has been criticised and more complex non-linear probes have been recommended in prior work (https://arxiv.org/abs/2004.03061) I wonder what the value of the findings of this paper be.”

Linear probes have a concrete application in LLM safeguards. Anthropic’s August 2026 Risk Report describes streaming linear probes being used as a component of deployed safeguards, linking this design to Cunningham et al.’s Constitutional Classifiers++, which our introduction already cites. We have edited our introduction to make the connection to safeguards clearer.

The paper you cited formulates probing as estimating mutual information between representations and linguistic features and recommends the best performing probe for that purpose. Our objective is reliable, lightweight detection and we do not use metrics for linear probes to establish causal relationships or to measure all information in representations.

The limitations of linear probes as tools for interpreting representations do not negate their practical use as lightweight detectors or the importance of studying their transfer across model updates (especially since safeguards should be kept effective as the model updates).

  1. “there is no expectation for a linear probe to generalise (applicable from) one checkpoint to another. What benefits should we see if this happens? is something that the authors should have motivated well in the introduction. Although I can "guess" this because of my expertise of the domain, I think that should not be an excuse not to make this explicit in the introduction.”

We do not assume that probes generalize from one checkpoint to another. Studying whether and when they generalize is one of the main empirical questions we aim to answer with this study. Empirically, we find that in some cases, linear probes generalize very well without intervention (such as between DPO and RL checkpoints as shown in our paper).

The practical benefit of probe generalization follows directly from the motivation provided in the introduction (that probe retraining is costly). To make this connection clearer, we have also explicitly added that successfully transferring probes can help avoid these retraining costs.

  1. “The label-free methodology is effectively orthogonal procrustes.”

Our paper differentiates between label-free representation alignment in Section 5 and our supervised correction method that we propose in Section 6.

The orthogonal drift correction, which we propose in Section 6, is supervised and does not perform Procrustes or label-free alignment (though it can be used alongside Procrustes or any other label-free representation alignment method.)

We agree that label-free alignment has been well-studied and also have explicitly acknowledged that it has been used in prior probe transfer papers (Duan (2026) studies probe transfer using label-free alignment when the model undergoes quantization, LoRA, and QLoRA.) Our main contribution in Section 5 is to evaluate how well these methods perform in transferring probes across post-training checkpoints. Note that Duan (2026) also does not analyze probe repair performance as a function of the data budget. We also find that in some probe transfer cases label-free alignment can degrade performance.

Our results in Section 5 also motivate why our proposed orthogonal drift correction method in section 6 is useful. Our proposed orthogonal drift correction method in Section 6 can improve transfer beyond using label-free alignment alone.

We hope these clarifications address your concerns. If they do, we would appreciate reconsideration of your assessment of our paper.

Reviewer_LjrwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how linear activation probes transfer when language models undergo post-training. It evaluates harm and refusal probes across SFT, DPO, and RL checkpoints in OLMo-3 7B, with additional experiments on Qwen3-4B and domain-specific SFT. Transfer performance is measured against a probe refitted on the target checkpoint. The experiments find that SFT creates most of the opportunity for refitting in the tested pipelines, while subsequent DPO and RL generally produce smaller gaps. The authors evaluate representation alignment and propose an orthogonal drift correction that combines an existing probe with a direction learned from labeled target-checkpoint activations. With a suitable starting probe, correction often outperforms refitting from scratch at the same training-prompt budget and can approach the full-data refit.

Critical strengths
  1. The paper identifies a practically relevant failure mode using appropriate evaluation metrics. Figure 2 shows that, on the most disruptive SFT transfers, frozen refusal probes achieve approximately half the target refit’s true-positive rate at 1% false-positive rate, despite remaining relatively close in AUROC. This demonstrates why aggregate ranking performance alone can give an incomplete picture of monitor transfer. Comparing against a target-checkpoint refit also accounts for changes in how well the concept can be detected after post-training.

  2. The empirical investigation covers multiple aspects of probe maintenance. The experiments include two model families, different post-training stages, multiple layers, and additional domain-specific SFT. They compare frozen probes, representation alignment, correction, and refitting, including a proximal-refit baseline in the harm experiments. The results provide evidence that retaining an existing probe can be useful when target training data are limited. The paper also reports cases where representation mapping hurts performance and includes total-label-budget plots in Appendix C.1, which makes the practical trade-offs easier to assess.

Critical weaknesses
  1. The added value of the main methodological contribution is not sufficiently established.

The results show that retaining information from an existing probe can improve adaptation compared with training a new probe from limited data. However, they do not establish how much of this improvement requires the proposed orthogonal drift correction. Equation 4 expresses the final detector as a combination of the existing probe and a newly fitted direction. The orthogonalization in Equation 5 keeps the detector within the span of these same directions. A direct combination of the two probe scores, tuned on the same estimation set, is therefore a particularly relevant comparison. The from-scratch and proximal-refit baselines do not isolate this distinction. Without this comparison, I find the evidence for the specific correction procedure less convincing than the evidence for probe reuse more generally. This limits the strength of the paper’s methodological contribution.

  1. The evaluation is insufficiently specified to assess the reliability and scope of the main empirical finding.

A central result is that frozen probes can lose substantial detection performance after SFT when evaluated at a 1% false-positive rate. However, the provided manuscript does not specify the source corpora and construction of the probe benchmarks, the test-set sizes and class counts, or the exact activation-pooling procedure. These details directly affect the interpretation of this result. In particular, the number and composition of negative test examples determine how reliably performance can be assessed at such a low false-positive rate. The reported variation across seeds does not, by itself, clarify uncertainty from the finite test sample. Since the empirical characterization of probe transfer is a major contribution, these omissions limit my ability to judge how robust and reproducible the headline finding is.

Questions for authors
  1. Can you compare orthogonal drift correction with directly combining the same existing and newly fitted probe scores, using the same training data, estimation set, and selection objective? Please also separate the effect of within-class centering from the effect of combining the probes. A demonstrated advantage over this simpler alternative would strengthen my assessment of the methodological contribution.

  2. Can you provide the benchmark sources and construction procedure, test-set class counts, and exact activation-pooling rule? For the headline TPR-at-1%-FPR result, how many negative test examples support the estimate, and do you have uncertainty estimates that account for the finite test sample? This would help establish whether the large SFT transfer losses are reliably measured.

2 comments

Thank you for taking the time to write your thoughtful review. We address your concerns below:

  1. “The added value of the main methodological contribution is not sufficiently established.”

The main methodological contribution is not the fact that orthogonalization is used, but rather that we have introduced a supervised method that achieves pareto-optimal transfer performance, especially where pre-existing label-free probe transfer methods fail (addressed in Section 5). We would argue that a direct combination of probe scores, tuned on the same estimation set would be an equally novel methodological contribution to the problem of probe transfer. Why orthogonalization is specifically used is addressed below.

  1. “Can you compare orthogonal drift correction with directly combining the same existing and newly fitted probe scores, using the same training data, estimation set, and selection objective? Please also separate the effect of within-class centering from the effect of combining the probes. A demonstrated advantage over this simpler alternative would strengthen my assessment of the methodological contribution.”

We compared orthogonal drift correction with a signed and weighted combination of the existing and newly-fitted probe scores, using identical fitted probe directions, training data, estimation set, AUROC selection objectives and test examples. Both methods use bounded Brent line search with identical optimiser settings. We also compare within-class and global centering under the same Brent search configurations.

The table below reports the largest absolute mean difference between the orthogonal drift correction and direct probe combination in TPR percentage points, across the different fitting and estimation budgets that were used.

\hline \text{Transfer} & \text{Max. absolute mean }\Delta\text{TPR (pp)} \\\\ \hline \text{Base}\rightarrow\text{Think-SFT (refusal)} & 0.05 \\\\ \text{Base}\rightarrow\text{Instr-SFT (refusal)} & 0.04 \\\\ \text{Think-SFT}\rightarrow\text{Instr-SFT (refusal)} & 0.06 \\\\ \text{Think-SFT}\rightarrow\text{Think-DPO (refusal)} & 0.04 \\\\ \text{Base}\rightarrow\text{Think-DPO (harm)} & 0.03 \\\\ \text{Base}\rightarrow\text{Think-SFT (harm)} & 0.02 \\\\ \hline \end{array}$$ To isolate within-class centering, we compare it with global centering while retaining the same Brent search configurations. $$\begin{array}{lr} \hline \text{Comparison} & \text{Absolute mean }\Delta\text{TPR (pp)} \\\\ \hline \text{Within-class vs global centring} & \leq 0.07 \\\\ \text{Globally centred correction vs direct blend} & \leq 0.07 \\\\ \hline \end{array}$$ While the results do not establish a significant detection performance benefit, orthogonalisation provides a way to decompose the update that is more interpretable than the direct combination. It retains the starting score and adds a residual correction after removing its pooled within-class linear overlap with that score on the estimation set. The weight controls the magnitude of this residual update. We will distinguish this structural interpretation from an additional performance benefit, which these comparisons do not establish. 3. “Can you provide the benchmark sources and construction procedure, test-set class counts, and exact activation-pooling rule? For the headline TPR-at-1%-FPR result, how many negative test examples support the estimate, and do you have uncertainty estimates that account for the finite test sample? This would help establish whether the large SFT transfer losses are reliably measured.” Our harm and refusal benchmarks are custom datasets derived from subsets of multiple existing corpora and model-generated texts. Each dataset undergoes extensive cleaning, filtering, and decontamination, and includes model-generated traces and labels that are not from the source corpora. We do not release the final datasets to reduce the risk of contamination. Our primary comparisons concern relative performance (rather than absolute performance) across probe transfer methods evaluated on identical held-out examples so we hope that providing the construction procedure, filtering criteria, class counts, evaluation protocol, and other details below will be enough to support your assessment of the results.

Construction procedure. For the harm dataset, we use datapoints each consisting of a single prompt and response (this is called a prompt-response exchange). A data-point is labelled positive when either the prompt or response is harmful. For refusal, we generate several responses per prompt at each checkpoint. At each prompt, each response in the set of responses is judged for refusal. If a majority of the responses are judged to exhibit refusal then the prompt is labelled as producing refusal at that model checkpoint. Prompts with tied or missing judgements for any checkpoint, are excluded across all other checkpoints.

The filtering procedure includes normalising the prompts into a common format and then checking for matches against post-training data. Prompts are assigned to the train/estimation/test splits based on prompt groups. Estimation examples are excluded from testing.

Test-set class counts and negative support. The approximate per-seed counts are below. Harm counts refer to prompt-response exchanges; refusal counts refer to prompts with checkpoint-specific behavioural labels.

\hline \text{Evaluation} & \text{Positives (approx.)} & \text{Negatives (approx.)} \\\\ \hline \text{Harm, Figure 2} & 36{,}500 & 37{,}100 \\\\ \text{Harm, main budget experiments} & 28{,}800 & 29{,}300 \\\\ \text{Refusal: Base}\rightarrow\text{Think-SFT, Figure 2} & 8{,}100 & 19{,}600 \\\\ \text{Refusal: Think-SFT}\rightarrow\text{Instr-SFT, Figure 2} & 10{,}200 & 17{,}600 \\\\ \hline \end{array}$$ Approximate counts are given because the counts vary slightly across seeds. The variation is due to the fact that the prompt groups vary in size slightly, and each seed assigns different prompt groups to the estimation set. The main budget experiments reserve a different number of prompt groups than the experiments whose results are presented in Figure 2. All methods within each comparison share the same test examples. Refusal class counts differ between checkpoints because some prompts might be refused by one checkpoint but not another. Negatives are non-refusals, not necessarily harmless prompts. Exact activation-pooling rule. We extract residual-stream activations after block 16 for OLMo and block 18 for Qwen, using zero-based indexing, without applying the final block’s normalisation to the activations. For harm, we average activations over every token in the exchange. For refusal, we use the final token of the checkpoint-specific prompt. Uncertainty from the finite test sample. For the two headline results concerning the TPR advantage of a full refit at 1% FPR, we estimated their uncertainty to a 95% confidence interval through bootstrapping, performing 10,000 paired prompt-group bootstrap replicates. For each replicate we recomputed the probe classification threshold. The full-refit minus frozen-probe TPR differences at 1% FPR are presented in the table below: $$\begin{array}{lrr} \hline \text{Transfer} & \text{Difference (pp)} & \text{95\\% CI (pp)} \\\\ \hline \text{Base}\rightarrow\text{Think-SFT} & 45.1 & [42.3,47.8] \\\\ \text{Think-SFT}\rightarrow\text{Instr-SFT} & 42.7 & [40.9,44.4] \\\\ \hline \end{array}$$ These intervals account for finite-test-sample uncertainty. Both intervals support the claim that the advantage of the full refit over the frozen probe remains large after accounting for finite-test-sample uncertainty. All of these data details have been added to the appendix of the revised paper. We hope these clarifications address your concerns. If they do, we would appreciate the reconsideration of your original assessment of our paper.
Reviewer_27EqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how post-training methods, such as SFT, DPO, and RL, affect the transferability of linear probes across model checkpoints. It then proposes a method that reuses existing probes instead of refitting new ones on post-trained model checkpoints. By learning correction directions, the proposed method aims to recover probe performance while reducing the cost of obtaining new labels.

Critical strengths

1.Maintaining probe performance across model updates is an interesting and practically relevant research direction

2.Method is reasoable and makes sense. I think it is logically sound. The concept of splitting the gap into frozen loss and ceiling drift is intuitive and itneresting

3.The paper attempts to reduce the cost of obtaining new labels by adapting existing probes

4.I appreciate that the authors honestly and openly report cases where their methods underperform, such as when label-free activation mapping performs worse than the frozen probe.

Critical weaknesses

1.I think the paper's advantage/contribution limits to cases where obtaining new labels is expensive. but for tasks where existing labels can be reused, i think it is not really that advantageous than just simply refitting. Thus, I think further discussion is needed on whether the proposed method provides meaningful cost savings in realistic settings and how to determine when correction may fail. In other words, I believe the authors should further discuss the practical significance of their method despite this limitation. If the authors disagree with this interpretation, I would appreciate further clarification or justification of the method's practical benefits.

2.It remains unclear whether the method generalizes to other models, domains, and probing objectives.

3.I think more analysis is needed. Specifically, I have included several questions below and would be happy to reconsider my score based on the authors' responses.

Questions for authors

1.The paper finds that SFT creates most of the refit headroom, while subsequent DPO and RL stages contribute relatively little. Could the authors provide some explanation why this pattern can happen?

2.Since SFT, DPO, and RL are evaluated as sequential post-training stages, could the observed differences in refit headroom be influenced by the training order? Do the authors expect the same pattern to hold if these methods were applied separately (not sequentially) to the same base model?

3.The correction method uses a separate labeled estimation set in addition to the training subset. I think the paper's most important contribution is reducing labeling costs, so could the authors clarify whether and how the estimation set is accounted for in the reported labeling budgets? If not, wouldn't the additional labels required for this set introduce a substantial cost burden, potentially reducing the practical advantage of the proposed method?

4.Could the results/effectiveness of the method be influenced by training configurations, such as learning rate, number of training steps, or training data size, rather than the post-training objectives themselves? Have the authors investigated how sensitive probe transfer and the proposed correction method are to these factors?

5.The experiments focus on harm and refusal probes. Do the authors expect the proposed correction method to generalize to other probing tasks or domains beyond safety?

2 comments

Thank you very much for your review. We appreciate your time in coming up with a comprehensive set of comments and questions. We address your concerns below:

  1. “I think the paper's advantage/contribution limits to cases where obtaining new labels is expensive. but for tasks where existing labels can be reused, i think it is not really that advantageous than just simply refitting. Thus, I think further discussion is needed on whether the proposed method provides meaningful cost savings in realistic settings and how to determine when correction may fail. In other words, I believe the authors should further discuss the practical significance of their method despite this limitation. If the authors disagree with this interpretation, I would appreciate further clarification or justification of the method's practical benefits.”

As mentioned in the introduction, retraining probes can be limited by the extraction of very large scale activation sets. This problem is particularly bad for probes that require per-token activations to train. As a hypothetical example, caching 16-bit activations for 50 million tokens (which corresponds to an exchange dataset with 50,000 rows and 1,000 tokens per exchange) across 64 layers of width 4,096 would take about 26.2 TB. Extracting this many activations for multiple different safeguard probes for each time the model is updated can be especially inconvenient. Being able to achieve pareto-optimal probe transfer performance using a smaller data budget is thus particularly helpful. Cunningham et al. report that for Constitutional Classifers++ safeguards (which has a all-layer probe component and required per-token activation extraction), moving activations to storage was slow enough that recomputing them during training was faster.

  1. “It remains unclear whether the method generalizes to other models, domains, and probing objectives.”

The paper already empirically tested 2 different models with two different post-training recipes, and three domain-specific SFT (science, coding, clinical) on separate copies of the model after the normal post-training recipes. For many of these scenarios we try both harm detection and refusal prediction as probing objectives. A lot of these results are in the paper appendix. Our method empirically tunes how much we trust the old probe (or a different probe used as the prior such as a ridge-map correct probe) versus a noisy correction, and thus should not be worse than the prior probe in most cases by construction.

  1. “The paper finds that SFT creates most of the refit headroom, while subsequent DPO and RL stages contribute relatively little. Could the authors provide some explanation why this pattern can happen?”

We ran additional preliminary experiments to answer this. The results suggest that SFT produces larger accumulated changes to the model, possibly contributing to the larger refit headroom:

\hline \text{Transfer} & \text{Weight change (\\%)} & \text{Harm headroom (pp)} \\\\ \hline \text{Base} \to \text{Think-SFT} & 10.8 & +4.11 \\\\ \text{Think-SFT} \to \text{Instruct-SFT} & 5.1 & +3.26 \\\\ \text{Think-SFT} \to \text{Think-DPO} & 0.031 & +0.22 \\\\ \text{Think-DPO} \to \text{Think-RLVR} & 0.061 & -0.03 \\\\ \hline \end{array}$$ Weight change refers to the change in the weight norms between the embedding and the blocks feeding the layer-16 readout. Headroom is full-refit minus frozen TPR at 1% FPR on fixed harm examples and labels. these diagnostic values use one probe seed. SFT changes these weights approximately 164 times as much as DPO assuming the same Think SFT parent. Additionally, accumulated SFT updates span more directions, and the headroom associated with harm grows with training progress before reaching a plateau. 4. “Since SFT, DPO, and RL are evaluated as sequential post-training stages, could the observed differences in refit headroom be influenced by the training order? Do the authors expect the same pattern to hold if these methods were applied separately (not sequentially) to the same base model?” While we do not do explicit post-training stage order ablations, there is some indirect evidence that training order might not be a prominent reason for the results observed. Note that the reason we don’t try these ablations explicitly is because we think that the paper could provide a more valuable contribution if it focused on more realistic post-training recipes and training orders. For example, DPO and RL are not often performed when the base model has not undergone instruction fine-tuning in the initial SFT phase. Appendix C.8 also shows that post-training on Qwen without a DPO phase still shows a similar trend in headroom where the RL phase results in a significantly smaller headroom than the SFT phase.
  1. “The correction method uses a separate labeled estimation set in addition to the training subset. I think the paper's most important contribution is reducing labeling costs, so could the authors clarify whether and how the estimation set is accounted for in the reported labeling budgets? If not, wouldn't the additional labels required for this set introduce a substantial cost burden, potentially reducing the practical advantage of the proposed method?”

The main budget plots report the fitting budget m, while Appendix C includes the estimation set. Appendix C reports m+|E|. Since |E|=4,000, the 100-fitting-group setting uses 4,100 labelled prompt groups. Both the orthogonal drift correction and the partial refit baseline use the same estimation set, so their original comparisons already match the total labelled budgets. We agree that this overhead matters. We therefore tested smaller estimation sets at fixed fitting budgets of 500 and 2,000 groups, across three checkpoint transfer settings and eight seeds:

\hline \text{Estimation groups} & \text{AUROC change (pp)} & \text{TPR change (pp)} \\\\ \hline 1{,}000 & -0.017\text{ to }0.000 & -0.19\text{ to }+0.03 \\\\ 500 & -0.029\text{ to }-0.012 & -0.36\text{ to }+0.02 \\\\ \hline \end{array}$$ The values in the table above are ranges of the mean changes relative to 4,000 estimation groups across the six transfer–budget settings; TPR is measured at 1% FPR. The transfers are Base to Think-SFT, Think-SFT to Instruct-SFT refusal, and Base to DPO harm. These results show that the estimation overhead can be substantially reduced in these settings, with only small observed changes in performance. For fixed harm exchanges, estimation labels can also be reused across checkpoints, although activations must be re-extracted, and this reusability applies to both probing methods. Refusal labels require generating and judging each updated checkpoint’s responses, so they are not reusable. The results above have been added to a new appendix. 6. “Could the results/effectiveness of the method be influenced by training configurations, such as learning rate, number of training steps, or training data size, rather than the post-training objectives themselves? Have the authors investigated how sensitive probe transfer and the proposed correction method are to these factors?” Appendix B already has some transfer results on intermediate SFT checkpoints after different amounts of training steps when using the map method. We have added additional detail to appendix B that compares frozen probes, correction and refitting at these checkpoints and the final checkpoint. The detail is what you see below. The table below reports mean harm AUROC. The orthogonal drift correction and partial refit budget for training used to obtain the results in the table was 3,200 prompt groups. $$\begin{array}{lrrr} \hline \text{Method} & \text{15,000 steps} & \text{30,000 steps} & \text{Final checkpoint} \\\\ \hline \text{Frozen probe} & 0.9636 & 0.9622 & 0.9623 \\\\ \text{Correction} & 0.9661 & 0.9657 & 0.9659 \\\\ \text{partial refit} & 0.9602 & 0.9606 & 0.9608 \\\\ \text{Full refit} & 0.9688 & 0.9685 & 0.9687 \\\\ \text{Refit headroom} & 0.0052 & 0.0064 & 0.0063 \\\\ \hline \end{array}$$ The orthogonal drift correction achieves higher mean AUROC than the partial refit at all three checkpoints, including across all 6 of the fitting budgets that we tested in the extended analysis. These results cover the effect of the number of training steps within one SFT run. We agree that learning rate or training-data size could also influence transfer. However, we focus on optimal training recipes to provide the results for most realistic transfer scenarios and so ablating these variables is not a priority for the study. We would expect similar trends where the orthogonal drift correction should not be worse than the prior probe in most cases for reasons explained earlier. 7. “The experiments focus on harm and refusal probes. Do the authors expect the proposed correction method to generalize to other probing tasks or domains beyond safety?” As mentioned earlier, our proposed correction method empirically tunes how much we trust the old probe (or a different probe used as the prior such as a ridge-map correct probe) versus a noisy correction, and thus should not be worse than the prior probe in most cases by construction. We deliberately focused our experiments on harm and refusal so that our empirical results would be differentially useful to safety practitioners deciding when to reuse a probe and how many labels to allocate. We hope these clarifications address your concerns. If they do, we would appreciate reconsideration of your assessment of our paper.

Comments on the paper

1 comment

Dear Authors,

As your rebuttal was submitted right at the cutoff of the author-reviewer discussion window, there was insufficient time for the reviewers to respond prior to the deadline. We will review the points raised in your response during the reviewer-AC discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.