- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors show that 19 small (<=12B param) open-weight models linearly represent the reversibility of actions, but rarely warn of risk of irreversible actions given a neutral (or even lightly priming) prompt. This “reversibility gap” is reduced when explicitly asking the models to flag irreversible actions. Activation steering in the reversible-irreversible mean-diff direction also often causes the models to warn of risks-- suggesting that the model’s representation of reversibility is causally related to warning behavior (if only weakly) as one might expect. The authors show that activation probes can outperform model behavior itself in detecting irreversible actions; this suggests a probe-based mitigation, surfacing a warning of irreversible actions to the user.
Critical strengths
They show a clear, strong effect that exists reliably across a large number of models. I believe that this result will generalize at least to other models in this capability range. They perform a good control showing the degree to which elicitation affects the result. I suspect this result is not that sensitive to exact prompting conditions. They neatly synthesize the relationship of steering and activation decoding under different amounts of regularization (fig. 2, which argues that stability of the steering direction estimate explains the gap between decoding and steering performance). They show an advantage of their proposed activation-based classifier over a reasonable blackbox baseline (prompted classifier).
Critical weaknesses
The scope of the paper is somewhat limited, showing linear decodability of a single safety-relevant property in synthetic short-context settings. The comparison to blackbox-only baselines is incomplete. I would be curious to compare the performance of the explicit-risk prompted model (whose FPR is not explicitly stated) with the probe thresholded at similar FPR. This seems important given that including explicit-risk language in the system prompt could be a reasonable baseline for the proposed probe-based control mitigation. Additionally, do probes stack with the explicit-risk prompt? It is not clear how the results transfer to stronger models; for example, do stronger closed-source models accessed via API warn of risks given neutral or lightly priming prompts? Although closed-source models don’t permit activation probing, they do permit comparison of warning behavior across prompting settings-- i.e., do SOTA models fail to flag irreversible actions in the neutral to explicit-risk settings? Showing that this issue persists in SOTA models (particularly in more naturalistic settings) would be persuasive with respect to the importance of these results. The taxonomy seems to be subtle; LLM-generated labels used in the majority of the paper seem to have poor agreement with human labels (which in turn agree poorly with each other). As the authors admit, it may be that probe performance is partly explained by spurious correlations between LLM judgments, rather than human-understood notions of irreversibility. If a probe using mean diff activations from positive and negative human labels of irreversibility achieves good performance in predicting “ground truth” LLM-generated labels of irreversibility (and vice-versa, if the paper’s probe performs well on human-generated labels), that would resolve this concern.
Questions for authors
Does the explicit-risk prompt result in nonzero FPR? What is the TPR of probes at matched FPR? It would be valuable to show results for prompted settings overlaid on a probe ROC curve for ease of comparison.