acceptodds
Under review as a conference paper at ICLR 2027

Agents Represent Action Reversibility, But Rarely Say So

Abstract

Models can tell an irreversible tool call from a recoverable one. Under a neutral system prompt, a mean of 91.9% of the irreversible ones draw no caution at all on the external set. Reversibility—whether an action is idempotent, reversible, compensable, or irreversible—is linearly decodable from a model's internal activations in all 19 open-weight models we test, above an action-name baseline on our internal set and in 13 of 18 models on externally authored schemas. The silence is elicitation rather than ignorance: telling the model to flag risky steps cuts it, on our internal set, from 84.9% to 27.9% without closing it, and a generic instruction to take care closes almost none of it. Asking outright does not fix it either. Models assert that an irreversible action can be undone in 26% of the judgments they make. On our internal set a linear probe's scores rank an irreversible action above a recoverable one 86.9% of the time, and tell a right spoken judgment from a wrong one 53.9% of the time, where 50% is chance. The same dissociation appears in causal control. Steering works in seven of ten models when the direction is the gap between the two classes' average activations, and in one when it is our probe's own. What decides this is how the direction is fitted, not which model it is—our probe ranks actions well, but two fits on different halves point different ways. The direction that decodes best is close to the worst one to steer with. The readout is useful in its own right. The probe raises detection F1 over the model's own caution language by 1.70× across eleven instruct models, and beats a prompted classifier. It transfers across models and to schemas written outside this work, though not to pooled live decoding, and under a threshold fixed in advance it clears about 30% of our internal action set (23% on a current scikit-learn release) while letting through only a few percent of the irreversible ones, at negligible latency. We release the benchmark, the probes, and our evaluation.

Primary area:
interpretability and explainable AI
Keywords:
LLM agents, tool use, linear probing, activation steering, AI safety, mechanistic interpretability, action reversibility, reversibility gap, identifiability, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DGzmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors show that 19 small (<=12B param) open-weight models linearly represent the reversibility of actions, but rarely warn of risk of irreversible actions given a neutral (or even lightly priming) prompt. This “reversibility gap” is reduced when explicitly asking the models to flag irreversible actions. Activation steering in the reversible-irreversible mean-diff direction also often causes the models to warn of risks-- suggesting that the model’s representation of reversibility is causally related to warning behavior (if only weakly) as one might expect. The authors show that activation probes can outperform model behavior itself in detecting irreversible actions; this suggests a probe-based mitigation, surfacing a warning of irreversible actions to the user.

Critical strengths

They show a clear, strong effect that exists reliably across a large number of models. I believe that this result will generalize at least to other models in this capability range. They perform a good control showing the degree to which elicitation affects the result. I suspect this result is not that sensitive to exact prompting conditions. They neatly synthesize the relationship of steering and activation decoding under different amounts of regularization (fig. 2, which argues that stability of the steering direction estimate explains the gap between decoding and steering performance). They show an advantage of their proposed activation-based classifier over a reasonable blackbox baseline (prompted classifier).

Critical weaknesses

The scope of the paper is somewhat limited, showing linear decodability of a single safety-relevant property in synthetic short-context settings. The comparison to blackbox-only baselines is incomplete. I would be curious to compare the performance of the explicit-risk prompted model (whose FPR is not explicitly stated) with the probe thresholded at similar FPR. This seems important given that including explicit-risk language in the system prompt could be a reasonable baseline for the proposed probe-based control mitigation. Additionally, do probes stack with the explicit-risk prompt? It is not clear how the results transfer to stronger models; for example, do stronger closed-source models accessed via API warn of risks given neutral or lightly priming prompts? Although closed-source models don’t permit activation probing, they do permit comparison of warning behavior across prompting settings-- i.e., do SOTA models fail to flag irreversible actions in the neutral to explicit-risk settings? Showing that this issue persists in SOTA models (particularly in more naturalistic settings) would be persuasive with respect to the importance of these results. The taxonomy seems to be subtle; LLM-generated labels used in the majority of the paper seem to have poor agreement with human labels (which in turn agree poorly with each other). As the authors admit, it may be that probe performance is partly explained by spurious correlations between LLM judgments, rather than human-understood notions of irreversibility. If a probe using mean diff activations from positive and negative human labels of irreversibility achieves good performance in predicting “ground truth” LLM-generated labels of irreversibility (and vice-versa, if the paper’s probe performs well on human-generated labels), that would resolve this concern.

Questions for authors

Does the explicit-risk prompt result in nonzero FPR? What is the TPR of probes at matched FPR? It would be valuable to show results for prompted settings overlaid on a probe ROC curve for ease of comparison.

Reviewer_QVvWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study a range of small models which they use as agents, largely non-reasoning models and models from before agent training was a significant focus, and they study whether we can cheaply identify irreversible tool calls. They find that the agent is not very reliable when asked, but training a simple probe can be highly reliable. They also do some interpretability analysis on how steerable these probes are, and show that difference-in-mean probes are significantly more steerable, and they show that the probes can be transferred between models with Procrustes alignment.

Critical strengths
  • There is some practical utility to having a cheap way of gating irreversible decisions even on a model too weak to reliably tell you
  • I appreciated the authors avoiding anthropomorphizing or implying the models "knew" the actions were reversible and were being deceptive about it
  • I appreciated the analysis of why logistic regression probes were worse for steering
  • I find it interesting that Procrustes alignment works here
Critical weaknesses
  • While the split-half analysis is nice, the core takeaway that class means steer better than probe weights is already known see eg Marks and Tegmark, 2024
  • The protocol matched comparison of the two directions is done on three models chosen because of the performance of class-mean steering
  • A regex scorer is used to measure whether they say action reversibility, rather than an LLM judge, and may miss things
  • The representation for probing is read from an additional forced tool call, reducing the practical utility
  • The 12-of-36 and 1-of-36 Fisher tests assume independence of corelated rows
  • The probes do not beat a TF-IDF baseline trained on the full vendor schema. The authors include this, to their credit, but only in Appendix I
  • The papers main results are in a single turn forced setting, when practical utility is in a multi-turn setting. The authors show it doesn't transfer, though can be retrained (on only 4 of 8 models), but their analysis is not studying the setting that matters
  • Poor human agreement on the LLM supplied labels
Questions for authors
  • Is there a practical reason to transfer a probe with Procrustes rather than just training it again?
  • Why do tables 2 and 32 disagree on the AUC for some of the same probes?
Reviewer_rerhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study whether LLM agents encode whether an action can be undone or not. If so, they also explore if that encoding can shape what the agents can do. They motivate these questions primarily with respect to irreversibility, where an action has no recovery path, where it is important for the agent to be cautious and hence, the authors are concerned with measuring caution. They explore the first question via linear probes and explore the second RQ with two action sets spanning 41 domains (214 actions from real tool schemas and 342 from unedited operations from public API and Berkeley Function-Calling Leaderboard) and measures them on 19 model checkpoints and steer on a 12 model subset. They find that reversibility is linearly decodable in every model, yet under neutral prompt, irreversible actions draw no warning. The steering also succeeds in most models when the difference lies between class means but not on the probe’s own weight vector.

Critical strengths

S1. The problem setup is well-motivated! It is important for agents to be cautious and to be able to flag that risk before acting on it. This gap between what the model represents and says can be useful for agent safety.

S2. The experimental setups are mostly well-controlled. The probes that were trained are compared with name-only baseline, permuted labels, and more. For steering, the authors had two fitted control directions and the outputs were screened for degenerate generation.

Critical weaknesses

W1. In live multi-turn runs, the scores of the trained probe drift across turns and the accuracy also drops a lot. It does not seem deployable as a gate as claimed.

W2. The name-only baseline is relatively weak. Since it matches character patterns, it may not recognize synonyms of the actions. See Q1.

W3. The ground truth labels of reversibility are uncertain. The human annotators seem to weakly agree with the LLM labeled actions. So, there is a possibility that the probe is learning the model’s notion of reversibility rather than true reversibility.

W4. The behavior numbers are not really convincing. The warnings were measured by keyword matching and the precision was 0.71.

W5. Certain parts of the paper were hard to follow. The abstract itself seemed very dense and it seems the number of models changes across experiments. A table listing which models are used in each experiment would be helpful.

Questions for authors

Q1. Can you test the probe against a stronger control? Maybe fit it on the model’s own activations for the action names (without the agent context)?

The rest of my questions/clarifications/suggestions are listed as Ws in the above section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.