acceptodds
Under review as a conference paper at ICLR 2027

TrustMI: Causally controlling how assistants trust their users

Abstract

Large Language Model (LLM) assistants routinely decide whether they can trust users and third parties whose competence, intentions, and integrity they cannot verify. This uncertainty matters for safety, as trusting the wrong party can lead an agent to comply with harmful requests or act on malicious instructions encountered during tool use. To study this problem, we define trust as an assistant's willingness to accept vulnerability to the actions of another party and ask whether such behavior can be causally controlled through model activations. We build 2,000 contrastive conversations spanning ability, benevolence, and integrity, where paired responses complete the same request but differ in whether the assistant trusts the user. From these pairs, we learn steering matrices while keeping the model parameters frozen and test them across six instruction-tuned models from three families, finding that steering changes trust decisions monotonically in both directions. We then ask whether this effect extends to several safety-related agent settings involving harmful requests, prompt injections, and insider threats, while using benign-task and reasoning as controls. Our findings provide evidence that trust in the user can be causally controlled along linear directions in model activations and provide a way to study how trust shapes safety-relevant behavior in language models.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, representation engineering, activation steering, large language models, agents, trust, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FxZQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether assistants’ reliance on unverifiable user claims can be controlled through activation interventions. It constructs 2,000 contrastive conversations that hold the request fixed and keep both responses helpful while varying reliance on the user’s claim. Building on BiPO, the method learns multilayer steering matrices with frozen model parameters, separately intervening on user inputs or assistant continuations. Experiments cover six models from three families, including trust evaluations, three safety settings, and capability controls.

Critical strengths
  1. The problem is practically meaningful, and the dataset includes useful controls. Reliance on unverifiable information is directly relevant to tool use and prompt-injection risks. Holding the request fixed while keeping both contrastive responses helpful helps reduce the conflation of distrust with refusal.

  2. Some results demonstrate a convincing improvement in the safety–utility trade-off. The evaluation covers multiple model families and includes benign-task and capability controls. For Qwen3.5-27B at α=−1\alpha=-1, AgentDojo attack success decreases from 33.6% to 20.8%, utility under attack increases from 56.8% to 66.8%, and benign utility decreases only from 76.4% to 75.0%. This supports the view that some safety gains are not merely consequences of general capability degradation.

Critical weaknesses
  1. Some reported scores are inconsistent with the stated protocol, and the small-sample confidence intervals are problematic. Appendix D.4 states that every benchmark except AgenticMisalignment is run three times, while Table 11 lists 50 τ²-bench airline tasks. With binary outcomes and equal weighting across all tasks and runs, accuracy should be 100k/150100k/150. However, values such as 43.6%, 49.2%, and 51.2% cannot result from this calculation after rounding; only 5 of the 30 reported values are compatible with a denominator of 150. Furthermore, Tables 8–10 report zero observed events among only 25 samples per condition as a 95% confidence interval of 0.0±0.00.0\pm0.0, understating uncertainty. Evaluation logs and statistical calculation details are needed to resolve these concerns.

  2. The algorithmic advances are incremental, while the central empirical contribution lacks sufficient conceptual separation and direct controls. The core optimization follows BiPO, with multilayer interventions, an auxiliary NLL term, and sparsity regularization providing incremental extensions. The contribution therefore depends substantially on establishing new insights into assistant trust. However, keeping both responses helpful does not adequately rule out correlated changes in caution, compliance, sycophancy, or refusal. Matched comparisons with these neighboring behavioral directions, as well as direct comparisons with single-layer BiPO and CAA using the same data, are missing. The results support a causal intervention on reliance behavior, but do not sufficiently establish its trust specificity or the necessity of the added design choices.

Questions for authors
  1. Evaluation-protocol clarification Please provide the actual number of τ²-bench tasks, samples per task, valid outcomes, scoring rule, and aggregation formula used in Table 11. How do values such as 43.6%, 49.2%, and 51.2% arise under the stated protocol of 50 tasks and three repetitions? Please provide per-task results sufficient to reproduce the reported scores.

  2. Statistical-reporting clarification How were the 95% confidence intervals in Tables 8–10 computed? Why do zero events in 25 samples yield 0.0±0.00.0\pm0.0? Do the main conclusions about different steering strengths hold when using intervals appropriate for small binomial samples?

  3. Controls that would affect the assessment of the contribution With training data, intervention locations, and safety–utility trade-offs matched as closely as possible, what distinctive effects does TrustMI exhibit relative to single-layer BiPO, CAA, and refusal or sycophancy directions? Which results distinguish trust control from general changes in caution or compliance?

2 comments

Dear reviewer FxZQ, thank you for the valuable feedback and questions. We hope these additional results and clarifications address your concerns and provide grounds for revising your rating upward.

On result precision clarification

Reviewer FxZQ raised concerns about rounded number that do not match the three runs announced in the appendix D.4.

W1. Some reported scores are inconsistent with the stated protocol, and the small-sample confidence intervals are problematic [...]

Thank you for checking these numbers. The τ²-bench experiments reported in Table 11 used five runs per task. Appendix D.4 incorrectly stated three runs, and we will correct this description in the revised manuscript. The experiments and reported results remain unchanged, only the stated number of runs is being corrected.

Q1. [...] Please provide the actual number of τ²-bench tasks, samples per task, valid outcomes, scoring rule, and aggregation formula used in Table 11 [...]

We evaluate on the 50 tasks of τ²-bench’s airline domain, using the Inspect implementation (inspect_evals 0.17.0). Each task is run 5 times per model and steering strength, giving 250 samples per cell. The steered model acts as the agent (temperature 0.8, up to 4,096 tokens per turn) and gpt-oss-120b simulates the user. All 250 conversations in each of the 30 cells are scored, and runs that exceed the 30-minute task timeout count as failures. We use τ²-bench’s standard binary reward. The conversation gets 1 if the final airline database matches the gold database and the agent communicated all required information, and 0 otherwise. The reported score is the fraction of successful runs out of 250, so every value in Table 11 is a multiple of 0.4% (e.g., 43.6% = 109/250, 49.2% = 123/250, 51.2% = 128/250). We report clipped Wald 95%confidence intervals with tasks as the unit averaging the 5 runs within each task. We provide the code as an anonymous github repo.

On statistical significance

Reviewer FxZQ raised concerns about the use of clipped Wald confidence intervals.

Q2. Statistical-reporting clarification How were the 95% confidence intervals in Tables 8–10 computed?

Thank you for highlighting this issue. Tables 8 through 10 used Wald 95% CIs, p^±1.96p^(1−p^)/n\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}, clipped to [0,1]. With zero observed events, the estimated standard error is zero, yielding 0.0 ± 0.0. We agree this understates uncertainty. Each cell contains 25 samples and 300 is the total across the 12 variants aggregated in Figure 3c. In the revised appendix, we will replace cell-level intervals with Wilson 95% CIs and report explicit lower and upper bounds: 0/25 yields [0.0%, 13.3%], and 25/25 yields [86.7%, 100.0%]. Wilson intervals preserve evidence for large reductions under negative steering. With an explicit goal and replacement threat, Qwen3.5-27B’s blackmail rate decreases from 76.0% [56.6%, 88.5%] at α = 0 to 8.0% [2.2%, 25.0%] at α = −2, Llama3.1-70B’s leaking rate decreases from 80.0% [60.9%, 91.1%] at α = 0 to 4.0% [0.7%, 19.5%] at α = −1. The average scores remain unchanged. The revised intervals more accurately reflect uncertainty while preserving the large reductions illustrated above. For the aggregated rates in Figure 3c, Wilson and Wald bounds differ by at most 0.6%, leaving our conclusions unaffected. We will update Tables 8–10 in the final revision.

On trust representation

Reviewer FxZQ raised concerns about similarities of the studied trust representation and previously studied direction such as refusal and sycophancy.

W2. [...] However, keeping both responses helpful does not adequately rule out correlated changes in caution, compliance, sycophancy, or refusal. Matched comparisons with these neighboring behavioral directions [...] are missing. [...]

Q3. [...] what distinctive effects does TrustMI exhibit relative to [...] refusal or sycophancy directions? Which results distinguish trust control from general changes in caution or compliance?

As this concern was shared by several reviewers, we address it in our general comment and summarize the results here. We computed the refusal and sycophancy directions following the exact methodology of Arditi et al. (2024) [1] and Rimsky et al. (2024) [2], respectively, and observe that they are near orthogonal to all our trust steering matrices (see the general comment for cosine similarities).

We then steered the models with both directions and evaluated them on the same safety benchmarks (new Appendix D.8). Refusal steering degrades general capabilities and lowers both harmful and benign scores regardless of sign, while sycophancy steering has little effect on either. Only trust steering substantially shifts safety behaviour while mostly preserving utility. Together, these results show that the effects of our trust matrices are neither explained by refusal nor reproduced by sycophancy.

On single layer steering baseline

Reviewer FxZQ asked for a steering baseline with only one steered layer against our steering matrices.

W2. [...] Matched comparisons with these neighboring [...] are missing. [...]

Q3. [...] what distinctive effects does TrustMI exhibit relative to single-layer BiPO, CAA [...]?

Our choice to use steering matrices instead of single layer steering vectors is justified in the original paper (line 190-203). A single layer steering baseline is not needed to support the claims of the paper.

Nonetheless, we trained a single-layer steering vector using the same methodology as in the original paper and reported its effect on the safety benchmarks in the new Appendix D.9. The vector is trained with BiPO on the same data and the same user span as our matrices, at layer 13 (about 40% of the depth). We chose this layer because intermediate layers hold the most abstract representations of a language model and are where steering usually works best [3, 4]. We observe that it moves the benchmarks in the same direction as the full matrices, but with a much smaller amplitude. At α = −2, it lowers the misaligned actions of Qwen3.5-9B from 21.0% to 14.7%, where the full matrix reaches 1.7%, and the AgentHarm harmful score of Olmo-3-7B from 8.4% to 5.2%, against 1.7%. A single layer therefore captures part of the trust direction, and steering all layers is what produces the full effect.

[1] Arditi et al., “Refusal in Language Models Is Mediated by a Single Direction,” NeurIPS 2024
[2] Rimsky et al., “Steering Llama 2 via Contrastive Activation Addition,” ACL 2024
[3] Zou et al.,” Representation Engineering: A Top-Down Approach to AI Transparency”, 2023
[4] Skean et al., “Layer by Layer: Uncovering Hidden Representations in Language Models”, ICML 2025

Reviewer_quYdOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TrustMI, a framework that uses bidirectional preference optimization (BiPO) to learn activation-steering matrices for controlling LLM assistants' trust in users. The authors construct 2,000 synthetic contrastive conversations and evaluate the steering effects across six models, including on harmful requests, prompt injections, and agentic misalignment benchmarks.

Critical strengths

The paper studies a relevant problem at the intersection of mechanistic interpretability and agent safety. The experiments cover multiple model families and include both behavioral and capability evaluations.

Critical weaknesses
  1. Limited innovation. This method primarily applies existing activation guidance technology and BiPO to trust-related behaviors.
  2. The mechanism is unclear. The direction of learning may capture general tendencies of obedience, skepticism, or rejection, rather than unique trust representations.
  3. Insufficient validation. The claimed transformation to tool output was only evaluated on two Qwen models, and relying on judgments based on Large Language Models (LLMs) further limits the strength of the conclusions.
Questions for authors
  1. What evidence demonstrates that the learned steering direction represents trust rather than general compliance, skepticism, or refusal behavior?
  2. How much improvement does TrustMI provide over simpler activation-steering baselines and existing refusal or prompt-injection defenses?
2 comments

Dear reviewer quYd, thank you for the valuable feedback and questions. We hope these additional results and clarifications address your concerns and provide grounds for revising your rating upward.

On paper contributions

Reviewer quYd raises concern about the contributions of the paper reusing existing training and steering methods.

W1. Limited innovation. This method primarily applies existing activation guidance technology [...]

As stated in our primary contributions, the main point of our study is not a new steering technique but trust representation in LLMs. We build upon existing interpretability work to train steering matrices [1]. However, we extend this work on multiple axes (multi layer, steering span and sparsity regularization) and which allows us to study trust representations at a granularity that single-layer, single-vector steering does not permit.

On distinguishing trust from related behaviors

Reviewer quYd questions whether the learned steering directions capture trust specifically or general tendencies toward compliance, skepticism, or refusal.

W2. The mechanism is unclear. The direction of learning may capture general tendencies of obedience, skepticism [...]
Q1. What evidence demonstrates that the learned steering direction represents trust [...] ?

In our study, we use the trust definition of Mayer et al. (1995), under which trust is a broader concept than simple behaviors such as systematic refusal or skepticism. We built a dataset following the core components of trust (ability, benevolence, integrity) [2], while controlling for refusal and helpfulness, and optimized our steering vector accordingly. LLM responses from a refusal-steered model and a distrust-steered model can be similar in specific scenarios,

As this concern was shared by several reviewers, we address it in our general comment and summarize the results here. Additional experiments show that our trust vectors are near-orthogonal to the refusal [3] and sycophancy [4] directions, and that steering with those directions produces significantly different behavior in safety benchmarks.

On the scope and reliability of the validation

Reviewer quYd questions the validation of tool-output steering, which was evaluated on only two Qwen models, and the reliance on LLM judges.

W3. Insufficient validation. The claimed transformation to tool output was only evaluated on two Qwen models [...]

The submitted paper already evaluates steering user+tool outputs across all six models (Table 7). Appendix D.7 presents an additional ablation on two Qwen models to isolate the contribution of steering tool outputs. We have now extended this ablation to all six models across three families (see revised Figure 12). At α=−2\alpha = -2, steering both sources yields lower attack success rates than steering user turns alone for every model, while benign utility remains broadly comparable. These results strengthen the evidence that steering matrices learned on user turns transfer to tool outputs across Qwen, Llama, and OLMo, without additional training.

W3. [...] relying on judgments based on Large Language Models (LLMs) further limits the strength of the conclusions.

Using LLMs as judges is now a common and accepted practice in ML research. It is a valid methodology [1,2, to name but a few] if the judge model has strong evaluation capabilities. In our submitted paper, we validated the judge capabilities and found high agreement with our own human annotations (App. D.3).

Moreover, the safety benchmarks corroborate the direction of LLM-judges: negative steering reduces harmful behavior across the six models. AgentDojo provides independent behavioral corroboration, as its attack-success and task-utility metrics are scored programmatically. Our evidence therefore combines human validation, LLM-judged trust evaluations, and independently scored behavioral outcomes.

Additionally, we report here examples of papers with valuable contributions to the scientific community accepted to ICLR that use LLM as a judge in their experiments:

  • Cunningham et al., Sparse Autoencoders Find Highly Interpretable Features in Language Models, ICLR 2024
  • Lin et al., WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild, ICLR 2025

On comparison to baselines and existing defenses

Reviewer quYd asks how much TrustMI improves over simpler activation-steering baselines and existing refusal or prompt-injection defenses.

Q2. How much improvement does TrustMI provide over simpler activation-steering baselines and existing refusal or prompt-injection defenses?

We report the comparison of our work to other activation baselines in our revision (Appendix D.9) showing that single layer steering impact less the model behavior than the trust steering matrices. As for comparison to other defenses, we would like to clarify the scope of our contributions. Our paper is purely analytical and studies how LLM models user/assistant interactions and is not a defense methodology. We find the generalisation of trust steering vectors, trained from benign conversation, to behavior shift in safety relevant scenarios, across 6 LLMs, very interesting and consider it one of the main findings of our work.

[1] Cao et al., “Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization”, NeurIPS 2024
[2] Mayer et al. “An integrative model of organizational trust”, Academy of management review 1995
[3] Arditi et al., “Refusal in Language Models Is Mediated by a Single Direction,” NeurIPS 2024
[4] Rimsky et al., “Steering Llama 2 via Contrastive Activation Addition,” ACL 2024

Reviewer_Nk8SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TrustMI steers an LLM's trust in its user, which is defined as its willingness to act on unverifiable user claims, by adding learned vectors to its hidden activations at each layer. The training uses 2,000 paired conversations that vary the user's ability, benevolence, and integrity, following Mayer's trust model. An LLM judge finds that these interventions raise or lower trust in six models on held-out conversations.

Critical strengths

When the learned vectors lower trust, this paper reports that every tested model behaves less harmfully on agent-safety benchmarks. So a direction learned only from everyday conversations about user claims might help make agents safer.

This paper provides some interesting insights. For example, this paper finds that steering tool outputs protects against injected instructions while steering only the user turns does not. This suggests that agent defenses could set reliance separately for user and tool information.

Critical weaknesses
  1. Some interpretation may be questionable. For example, the paper interprets the learned direction as user trust but omits comparisons with vectors inducing refusal, agreement with users, or random activation changes. So the safety effects' relation to general compliance or caution remains unclear. In more details, most distrust replies in Figure 6 add caution, such as a safer git flag. At α = −2, benign AgentHarm refusals for Llama-3.1-70B rise from 0.8% to 13.3% (see Table 6), although "distrust, in our formulation, is not refusal". The authors may consider comparing the safety results with Arditi et al.'s (2024) refusal vector, Rimsky et al.'s (2024) sycophancy vector, and a random vector with equal magnitude would test these alternatives.
  2. There may be some claim-evidence misalignment. For example, especially some key arguments are not well-supported.
    • The paper states that steering changes agent behavior "without merely degrading the models' abilities", but the largest AgentHarm reductions appear at α = −2, where benign scores fall. So the role of capability loss remains unclear. Specifically, at α = −1 the reductions are small, with overlapping 95% intervals for five of six models (See Table 6). At α = −2, harmful-task refusals rise little or fall for most models, and OLMo-3-7B refuses less while its benign score drops from 30.3% to 7.2%. Steering tool outputs may impair tool use. It is suggested that the authors report both harmful and benign refusal rates under user-only steering, which would separate these effects.
    • Section 3.1 interprets steering as evidence of an internal trust representation. But optimizing activation changes only shows that outputs can be altered. So the argument is not supported. Optimized activation changes can alter outputs through directions the model does not normally use [1]. Layer vectors total about 655,000 parameters for Llama-3.1-70B. Vectors applied to user text and assistant replies both change trust, yet point in nearly perpendicular directions (See Table 5). One suggestion is that a classifier predicting trust from unchanged activations like detectors of injected instructions [2] would help test this interpretation.
  3. The code and the trained steering matrices are promised only upon acceptance, so the training, trust judging, and benchmark results cannot currently be reproduced or verified. Considering the exponentially growing volume of machine learning submissions, it is strongly encouraged that the authors may submit the code along with the submission. Otherwise, it is impossible for reviewers to verify the correctness of the methodology, whether the code align with the method, whether the code can reproduce the reported results, and whether the results truly support the method.

[1] Aleksandar Makelov et al. Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024.

[2] Sahar Abdelnabi et al. Get My Drift? Catching LLM Task Drift with Activation Deltas. IEEE SaTML 2025.

Questions for authors
  1. What is the cosine similarity between the trust matrices and the refusal direction or a sycophancy vector. Do those directions reproduce the safety effects in Figure 3?
  2. At α = −2, how do the AgentHarm harmful and benign refusal rates change when only user turns are steered?
  3. Do unsteered activations projected onto the learned direction separate conversations with credible user claims from conversations with doubtful ones?
2 comments

Dear reviewer Nk8S, thank you for the valuable feedback and questions. We hope these additional results and clarifications address your concerns and provide grounds for revising your rating upward.

On trust vs refusal and sycophancy representation

Reviewer Nk8S raised concerns about similarities of the studied trust representation and previously studied direction such as refusal and sycophancy.

W1. [...] the paper interprets the learned direction as user trust but omits comparisons with vectors inducing refusal, agreement with users, or random activation changes [...]

Q1. What is the cosine similarity between the trust matrices and the refusal direction or a sycophancy vector [...]

To compare our trust representation to refusal and sycophancy, we now compute their linear direction following the exact methodology of Arditi et al. (2024) and Rimsky et al. (2024), respectively. The cosine similarities we obtained indicate near orthogonal representation between our trust direction and the refusal/sycophancy directions.

ModelTrust vectorRefusal (Arditi et al., 2024)Sycophancy (Rimsky et al., 2024)
Qwen3.5-9BUser span+0.007−0.012
Assistant span+0.017+0.006
OLMo-3-7BUser span−0.018+0.008
Assistant span+0.001+0.013
Llama-3.1-8BUser span−0.010+0.014
Assistant span+0.007+0.000

Q1 [...] Do those directions reproduce the safety effects in Figure 3?

Our additional experiments with refusal and sycophancy steering show that both directions produce different behaviours from our trust steering matrices (all new results are reported in Appendix D.8). At α = ±2, refusal steering breaks the capabilities of all three models, which lowers both harmful and benign scores at both signs (e.g. the benign AgentHarm score of Qwen3.5-9B falls from 68.6% to 27.4% and 4.8%). Sycophancy steering changes most harmful and benign scores by less than 5 points. Trust steering instead has significant effects while preserving most of its utility. Together, these results show that the safety effects of our trust matrices are neither reproduced by sycophancy steering nor explained by refusal.

On trust internal representation

Reviewer Nk8S raised concerns that the trained steering matrices were not corresponding to internal representation of trust in LLMs.

W2. The paper states that steering changes agent behavior "without merely degrading the models' abilities" [...] Steering tool outputs may impair tool use. It is suggested that the authors report both harmful and benign refusal rates under user-only steering, which would separate these effects.

Q2. At α = −2, how do the AgentHarm harmful and benign refusal rates change when only user turns are steered?

To test whether “steering tool outputs impairs tool use”, we extended our “user steered” vs “user & tool output steered” ablation on AgentDojo (App. D.7) to all models and report now both benign and harmful scores in the new revised figure 12 (cf. pdf). Benign scores remain stable under both setups, which indicates that steering tool outputs does not substantially degrade tool-use ability. We acknowledge that on AgentHarm benign tasks, steering significantly affects benign scores for some models, which diverges from the other capability benchmarks we evaluated (AgentDojo benign, GPQA Diamond, BBEH mini, BFCL core and τ²-bench airline). We do not yet have a full explanation for this result. One likely explanation is that, because AgentHarm benign tasks are constructed as mirrors of the harmful ones and share their sensitive sounding topics [1], a shift toward distrust may also increase caution on these benign requests, unlike on the other capability benchmarks. In addition, after a qualitative inspection of OLMo 7B and 32B generation, we note that the models refuse more often, with justifications ranging from safety concerns to claimed capability limitations (e.g. “I am unable to execute actual system commands directly in this environment [...]”, “I’m sorry but I can’t assist with that request [...]”). We will add this discussion and nuance our claim about capability preservation accordingly in the final revision.

On trust internal representation

Reviewer Nk8S raised concerns that the trained steering matrices were not corresponding to internal representation of trust in LLMs.

W2. Section 3.1 interprets steering as evidence of an internal trust representation. But optimizing activation changes only shows that outputs can be altered [...] Vectors applied to user text and assistant replies both change trust, yet point in nearly perpendicular directions (See Table 5). One suggestion is that a classifier predicting trust from unchanged activations like detectors of injected instructions [2] would help test this interpretation.

Q3. Do unsteered activations projected onto the learned direction separate conversations with credible user claims from conversations with doubtful ones?

To confirm that our steering matrices correspond to internal trust representations, we try to classify the raw model activations of the assistant response of our test dataset representing trust or distrustful replies with our assistant and user steering vector. Our results show that our trust matrices have specific layers that are able to differentiate trustful and distrustful replies as we report high paired accuracy for all models.

ModelUser layerUser paired acc.Assistant layerAssistant paired acc.
Qwen3.5-9B17 (53%)0.9411 (34%)0.87
Qwen3.5-27B41 (64%)0.9332 (50%)0.91
Llama-3.1-8B14 (44%)0.9214 (44%)0.81
Llama-3.1-70B30 (38%)0.8918 (22%)0.81
Olmo-3-7B13 (41%)0.8517 (53%)0.83
Olmo-3.1-32B21 (33%)0.9259 (92%)0.85

Previous works [2] show that, depending on the steering span (e.i. user vs assistant), representation for abstract concepts can be different which explain our near orthogonal user assistant matrices.

On reproducibility

Reviewer Nk8S raised concerns about the reproducibility of our results.

W3. The code and the trained steering matrices are promised only upon acceptance [...]

We released the codebase used to obtain the paper and rebuttal results under an anonymous github repository.

We warmly want to thank the reviewer again as we believe that their review really helped us to improve our work.

[1] Andriushchenko et al., “AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents,” ICLR 2025
[2] Sofroniew et al., “Emotion Concepts and their Function in a Large Language Model”, Transformer Circuits, 2026
[3] Arditi et al., “Refusal in Language Models Is Mediated by a Single Direction,” NeurIPS 2024
[4] Rimsky et al., “Steering Llama 2 via Contrastive Activation Addition,” ACL 2024

Comments on the paper

2 comments

Dear reviewers, thank you for the valuable feedback and questions.

As you raised concerns about similarities between our trust representation and previously studied representation (e.i. refusal [1] and sycophancy [2]), we provide here evidence that our representation is different via two lines of experiments.

First, we computed the refusal and sycophancy directions following the exact methodology of Arditi et al.(2024) and Rimsky et al.(2024) and compared them via cosine similarity. We observe near orthogonal directions for all three trust steering matrices, showing that our studied trust representations are different.

ModelTrust vectorRefusal (Arditi et al., 2024)Sycophancy (Rimsky et al., 2024)
Qwen3.5-9BUser span+0.007−0.012
Assistant span+0.017+0.006
OLMo-3-7BUser span−0.018+0.008
Assistant span+0.001+0.013
Llama-3.1-8BUser span−0.010+0.014
Assistant span+0.007+0.000

Second, steering along these directions produces behaviour that differs clearly from trust steering (new Appendix D.8). Refusal steering degrades general capabilities, lowering both harmful and benign scores regardless of sign. Sycophancy steering has little effect on either. Only trust steering substantially shifts safety behaviour while mostly preserving utility. This shows that its effects are neither explained by refusal nor reproduced by sycophancy.

We hope these additional results address your concerns and that you will consider revising your score accordingly.

[1] Arditi et al., “Refusal in Language Models Is Mediated by a Single Direction,” NeurIPS 2024
[2] Rimsky et al., “Steering Llama 2 via Contrastive Activation Addition,” ACL 2024

Dear Reviewers,

Thank you again for your insightful comments and suggestions. We would be grateful to hear whether our responses have clarified the points you raised, and we are be happy to address any remaining questions or concerns. Thank you for your time and consideration.

Warm regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.