- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether assistants’ reliance on unverifiable user claims can be controlled through activation interventions. It constructs 2,000 contrastive conversations that hold the request fixed and keep both responses helpful while varying reliance on the user’s claim. Building on BiPO, the method learns multilayer steering matrices with frozen model parameters, separately intervening on user inputs or assistant continuations. Experiments cover six models from three families, including trust evaluations, three safety settings, and capability controls.
Critical strengths
-
The problem is practically meaningful, and the dataset includes useful controls. Reliance on unverifiable information is directly relevant to tool use and prompt-injection risks. Holding the request fixed while keeping both contrastive responses helpful helps reduce the conflation of distrust with refusal.
-
Some results demonstrate a convincing improvement in the safety–utility trade-off. The evaluation covers multiple model families and includes benign-task and capability controls. For Qwen3.5-27B at , AgentDojo attack success decreases from 33.6% to 20.8%, utility under attack increases from 56.8% to 66.8%, and benign utility decreases only from 76.4% to 75.0%. This supports the view that some safety gains are not merely consequences of general capability degradation.
Critical weaknesses
-
Some reported scores are inconsistent with the stated protocol, and the small-sample confidence intervals are problematic. Appendix D.4 states that every benchmark except AgenticMisalignment is run three times, while Table 11 lists 50 τ²-bench airline tasks. With binary outcomes and equal weighting across all tasks and runs, accuracy should be . However, values such as 43.6%, 49.2%, and 51.2% cannot result from this calculation after rounding; only 5 of the 30 reported values are compatible with a denominator of 150. Furthermore, Tables 8–10 report zero observed events among only 25 samples per condition as a 95% confidence interval of , understating uncertainty. Evaluation logs and statistical calculation details are needed to resolve these concerns.
-
The algorithmic advances are incremental, while the central empirical contribution lacks sufficient conceptual separation and direct controls. The core optimization follows BiPO, with multilayer interventions, an auxiliary NLL term, and sparsity regularization providing incremental extensions. The contribution therefore depends substantially on establishing new insights into assistant trust. However, keeping both responses helpful does not adequately rule out correlated changes in caution, compliance, sycophancy, or refusal. Matched comparisons with these neighboring behavioral directions, as well as direct comparisons with single-layer BiPO and CAA using the same data, are missing. The results support a causal intervention on reliance behavior, but do not sufficiently establish its trust specificity or the necessity of the added design choices.
Questions for authors
-
Evaluation-protocol clarification Please provide the actual number of τ²-bench tasks, samples per task, valid outcomes, scoring rule, and aggregation formula used in Table 11. How do values such as 43.6%, 49.2%, and 51.2% arise under the stated protocol of 50 tasks and three repetitions? Please provide per-task results sufficient to reproduce the reported scores.
-
Statistical-reporting clarification How were the 95% confidence intervals in Tables 8–10 computed? Why do zero events in 25 samples yield ? Do the main conclusions about different steering strengths hold when using intervals appropriate for small binomial samples?
-
Controls that would affect the assessment of the contribution With training data, intervention locations, and safety–utility trade-offs matched as closely as possible, what distinctive effects does TrustMI exhibit relative to single-layer BiPO, CAA, and refusal or sycophancy directions? Which results distinguish trust control from general changes in caution or compliance?
2 comments
On single layer steering baseline
Reviewer FxZQ asked for a steering baseline with only one steered layer against our steering matrices.
W2. [...] Matched comparisons with these neighboring [...] are missing. [...]
Q3. [...] what distinctive effects does TrustMI exhibit relative to single-layer BiPO, CAA [...]?
Our choice to use steering matrices instead of single layer steering vectors is justified in the original paper (line 190-203). A single layer steering baseline is not needed to support the claims of the paper.
Nonetheless, we trained a single-layer steering vector using the same methodology as in the original paper and reported its effect on the safety benchmarks in the new Appendix D.9. The vector is trained with BiPO on the same data and the same user span as our matrices, at layer 13 (about 40% of the depth). We chose this layer because intermediate layers hold the most abstract representations of a language model and are where steering usually works best [3, 4]. We observe that it moves the benchmarks in the same direction as the full matrices, but with a much smaller amplitude. At α = −2, it lowers the misaligned actions of Qwen3.5-9B from 21.0% to 14.7%, where the full matrix reaches 1.7%, and the AgentHarm harmful score of Olmo-3-7B from 8.4% to 5.2%, against 1.7%. A single layer therefore captures part of the trust direction, and steering all layers is what produces the full effect.
[1] Arditi et al., “Refusal in Language Models Is Mediated by a Single Direction,” NeurIPS 2024
[2] Rimsky et al., “Steering Llama 2 via Contrastive Activation Addition,” ACL 2024
[3] Zou et al.,” Representation Engineering: A Top-Down Approach to AI Transparency”, 2023
[4] Skean et al., “Layer by Layer: Uncovering Hidden Representations in Language Models”, ICML 2025
Dear reviewer FxZQ, thank you for the valuable feedback and questions. We hope these additional results and clarifications address your concerns and provide grounds for revising your rating upward.
On result precision clarification
Reviewer FxZQ raised concerns about rounded number that do not match the three runs announced in the appendix D.4.
Thank you for checking these numbers. The τ²-bench experiments reported in Table 11 used five runs per task. Appendix D.4 incorrectly stated three runs, and we will correct this description in the revised manuscript. The experiments and reported results remain unchanged, only the stated number of runs is being corrected.
We evaluate on the 50 tasks of τ²-bench’s airline domain, using the Inspect implementation (inspect_evals 0.17.0). Each task is run 5 times per model and steering strength, giving 250 samples per cell. The steered model acts as the agent (temperature 0.8, up to 4,096 tokens per turn) and gpt-oss-120b simulates the user. All 250 conversations in each of the 30 cells are scored, and runs that exceed the 30-minute task timeout count as failures. We use τ²-bench’s standard binary reward. The conversation gets 1 if the final airline database matches the gold database and the agent communicated all required information, and 0 otherwise. The reported score is the fraction of successful runs out of 250, so every value in Table 11 is a multiple of 0.4% (e.g., 43.6% = 109/250, 49.2% = 123/250, 51.2% = 128/250). We report clipped Wald 95%confidence intervals with tasks as the unit averaging the 5 runs within each task. We provide the code as an anonymous github repo.
On statistical significance
Reviewer FxZQ raised concerns about the use of clipped Wald confidence intervals.
Thank you for highlighting this issue. Tables 8 through 10 used Wald 95% CIs, p^±1.96p^(1−p^)/n, clipped to [0,1]. With zero observed events, the estimated standard error is zero, yielding 0.0 ± 0.0. We agree this understates uncertainty. Each cell contains 25 samples and 300 is the total across the 12 variants aggregated in Figure 3c. In the revised appendix, we will replace cell-level intervals with Wilson 95% CIs and report explicit lower and upper bounds: 0/25 yields [0.0%, 13.3%], and 25/25 yields [86.7%, 100.0%]. Wilson intervals preserve evidence for large reductions under negative steering. With an explicit goal and replacement threat, Qwen3.5-27B’s blackmail rate decreases from 76.0% [56.6%, 88.5%] at α = 0 to 8.0% [2.2%, 25.0%] at α = −2, Llama3.1-70B’s leaking rate decreases from 80.0% [60.9%, 91.1%] at α = 0 to 4.0% [0.7%, 19.5%] at α = −1. The average scores remain unchanged. The revised intervals more accurately reflect uncertainty while preserving the large reductions illustrated above. For the aggregated rates in Figure 3c, Wilson and Wald bounds differ by at most 0.6%, leaving our conclusions unaffected. We will update Tables 8–10 in the final revision.
On trust representation
Reviewer FxZQ raised concerns about similarities of the studied trust representation and previously studied direction such as refusal and sycophancy.
As this concern was shared by several reviewers, we address it in our general comment and summarize the results here. We computed the refusal and sycophancy directions following the exact methodology of Arditi et al. (2024) [1] and Rimsky et al. (2024) [2], respectively, and observe that they are near orthogonal to all our trust steering matrices (see the general comment for cosine similarities).
We then steered the models with both directions and evaluated them on the same safety benchmarks (new Appendix D.8). Refusal steering degrades general capabilities and lowers both harmful and benign scores regardless of sign, while sycophancy steering has little effect on either. Only trust steering substantially shifts safety behaviour while mostly preserving utility. Together, these results show that the effects of our trust matrices are neither explained by refusal nor reproduced by sycophancy.