- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed Supervised Transcoder Replacement (STR), which aim to mitigate the side effects of model steering. The core idea is training a new sparse transcoder to replace the original MLP, which be capable of reconstructing the functionality of the original MLP, while also ensuring that, when subjected to steering interventions, it enhances the target behavior with minimal disruption to other behaviors. This method shifts the optimization target, with the motivation of training a representation space that is inherently easier to control safely.
Critical strengths
- This paper proposes an interesting direction for activation steering. Instead of optimizing the steering method itself, it explores making the model's internal representations more suitable for steering interventions.
- The experiments cover multiple steering methods and model families, demonstrating that STR can be integrated with existing steering techniques and effectively reduce safety-related side effects.
Critical weaknesses
- STR trains the transcoder using Corrigibility and Safety data, and then learns a steering vector for the same target behavior. This raises concerns about whether the learned representation is generalizable or simply tailored to this specific task pair. It would be important to evaluate whether a transcoder trained on Corrigibility can also reduce side effects when steering other behaviors, such as Truthfulness or Sycophancy, without retraining.
- It remains unclear whether replacing the original MLP with a transcoder affects the model's general capabilities. Additional evaluations on standard benchmarks would help clarify this.
- One of the key advantages of activation steering is its lightweight nature, but STR introduces additional training and inference overhead. The actual computational costs and whether the safety improvements justify them remain unclear.
Questions for authors
- Can the authors provide cross-target transfer experiments? For example, can a transcoder trained on Corrigibility and Safety data reduce steering side effects for other target behaviors without retraining? This would help clarify whether STR learns a generalizable representation space or one tailored to a specific task pair.
- Have the authors evaluated whether replacing the original MLP affects the model's general capabilities, such as reasoning and knowledge performance, even without steering?
- Could the authors report the additional training and inference costs introduced by STR, and discuss whether these costs are justified by the improvements in steering safety?
Response to Reviewer w78H
We thank the reviewer for these constructive suggestions. We agree that cross-target transfer, general capability preservation, and computational cost are important for assessing STR’s generality and practical value. We therefore conducted three supplementary evaluations addressing these concerns.
Regarding cross-target transfer, we evaluated Truthfulness and Sycophancy using an implementation based on the publicly available CAA codebase of Rimsky et al. and its released behavioral datasets. We reused the existing transcoders trained using Corrigibility and Safety data, keeping both the backbone and transcoder frozen. Only the steering vectors for the new targets were fitted separately in the original-model and STR representation spaces.
We evaluated CAA, SSV, and OPIUM on Gemma-3-4B-it, Llama-3.1-8B-Instruct, and Gemma-3-12B-it. Each target used 1,000 paired training examples from the corresponding CAA training split. Truthfulness was evaluated on a predetermined subset of 100 TruthfulQA questions covering 30 categories, with equal weighting across categories; Sycophancy used all 50 questions in the corresponding test set. The intended directions were increased truthfulness and increased sycophancy, respectively.
Target utility U is the conditional probability assigned to the target A/B option, expressed as a percentage, rather than free-generation accuracy or the full TruthfulQA MC1/MC2 score. ASR follows the paper’s safety evaluation protocol. All entries below are original model / STR, in percentages.
Truthfulness steering
Sycophancy steering
STR lowers ASR in all 18 steered configurations, by 0.94%–10.67%. For Gemma-3-12B with Sycophancy–SSV, the target-utility increase relative to each branch’s unsteered baseline is 3.30% under STR versus 0.15% on the original model, while the corresponding ASR increase is only 0.27% versus 10.11%. These findings support transfer of STR’s safety benefits to both additional behaviors without transcoder retraining.
Regarding general capabilities without steering, we evaluated MMLU, GSM8K, and IFEval, covering knowledge, mathematical reasoning, and instruction following. We used 570 MMLU questions with 5-shot prompting and subject weighting based on the full test distribution, 200 GSM8K questions with 0-shot prompting and strict answer extraction, and 200 IFEval prompts with official strict prompt-level accuracy.
All entries below are original model / STR, in percentages, with no steering.
Across the nine model–benchmark pairs, score changes range from −2.50% to +2.67%. These evaluations suggest approximate preservation of general capabilities without steering.
Regarding computational cost, we measured training and inference time on a single datacenter GPU with 4.8 TB/s memory bandwidth
Training time. Averaged across three seeds, complete STR training jobs took 29.1, 27.3, and 40.8 minutes for Gemma-3-4B, Llama-3.1-8B, and Gemma-3-12B, respectively.
Inference time. The table reports mean inference compute time for the same eight requests under CAA steering, using 128 input tokens and 64 generated tokens per request, averaged over three runs after warm-up. Times are original model / STR; batch size 1 processes the eight requests sequentially. Batch size 8 processes them simultaneously.
In conclusion, the training cost can be amortized across steering targets, while the observed safety improvements incur at most 2.79% additional inference compute time under the tested conditions.
We hope these supplementary experiments will address your concerns.