acceptodds
Under review as a conference paper at ICLR 2027

Knowing Whom to Trust: Reliability States and Information Use in Large Language Models

Abstract

Agents based on large language models (LLMs) increasingly make decisions using information supplied by people and other agents. Their choice of whom to trust shapes both the information they accept and the resources they entrust. Alongside studying whether agents themselves are trustworthy, an equally important question is how they learn whom to trust and which internal processes connect that experience to later decisions. We investigate this question across eight general-purpose LLMs spanning multiple model families and parameter scales. Models receive reports and verifiable feedback during interactions with four unfamiliar simulated participants whose reporting reliability is experimentally controlled, then allocate resources in a trust game and choose between conflicting recommendations. Participants with higher reporting reliability receive larger transfers and are selected more often when recommendations conflict. In Qwen3.5-27B, verbalizer readouts and linear probes distinguish participants with high versus low reporting reliability at the output and activation levels. Activation patching and steering shift recommendation-choice probabilities and expected transfers to the corresponding participants; the same steering direction affects both decision tasks. These findings link interaction experience to subsequent decisions through internal evaluations of reporting reliability, providing behavioral and causal evidence for how LLMs evaluate others and decide whom to trust.

Primary area:
interpretability and explainable AI
Keywords:
large language models, mechanistic interpretability, trust and reputation, activation patching, activation steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nVyyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLMs use interaction histories to evaluate unfamiliar participants and make subsequent trust-related decisions. The authors use a take-advice game to establish different reporting-reliability histories, followed by resource-allocation and conflicting-recommendation tasks. They then use descriptive readouts, linear probes, activation steering, and patching to examine how reliability-related representations influence these decisions.

Critical strengths
  1. Clear presentation: The paper is well organized and clearly separates behavioral observations, representational readouts, and causal interventions. This structure makes the evidence supporting each claim relatively easy to assess.

  2. Broad behavioral evaluation: The behavioral experiments cover eight LLMs and two downstream tasks. This breadth helps establish that the observed reliance on reporting history extends across model families and different forms of decision-making.

Critical weaknesses
  1. Limited mechanistic specificity: The interventions demonstrate that reliability-associated activations influence decisions, but do not sufficiently distinguish reliability from general favorability, positive evaluation, or other correlated attributes. The analyses also leave unclear how these evaluations form and update during interaction. Consequently, the proposed mechanism remains a relatively broad description of a behaviorally relevant representation.
  2. Limited methodological and empirical novelty: The study combines an established TAG–TG paradigm with standard probing and activation-intervention methods. Prior work has also examined source-dependent choices and internal trust representations. The main increment is therefore the connection between interaction-derived evaluations and two downstream decisions, but the paper offers limited broader insight into when this connection should hold, when it should fail, or what it implies beyond the tested setting.
Questions for authors
  1. Would directions representing general favorability or other positive evaluations produce similar effects? Does steering one participant also change decisions about other participants?
  2. How consistent are the intervention effects across models? The eight-model behavioral results are broad, but the causal evidence comes from Qwen3.5-27B alone.
1 comment

Thank you for your careful review and constructive suggestions. Three new experiments address specificity, novelty, and generality.

1. Reliability vs. general favorability, and how evaluations form (W1, Q1)

We did not manipulate reliability orthogonally to favorability; instead, TAG provides behavioral trajectories varying verifiable accuracy across neutral IDs (equalized reward chances, no persona/warmth cues), so social impressions emerge in a data-driven way: without a preset reliability word list, verbalizer selection naturally surfaces reliability-centered terms (reliable, consistent vs. unreliable, inconsistent), and our steering vector contrasts high- vs. low-accuracy description activations. We agree this manipulation is not orthogonal to favorability, as repeated accurate reporting fosters broader positive impressions. Prior trust work likewise shows positive evaluations can shape cooperative choices (Mayer et al., 1995; Bellucci et al., 2019), so a favorability vector could plausibly shift decisions too. Our focus is thus not abstract favorability, but how a data-driven positive social evaluation formed from verifiable trajectories causally shapes advice acceptance and resource allocation.

On formation dynamics, our experiments focus on post-interaction states after completed trajectories and have not yet modeled turn-by-turn updating; because recent observations and cumulative accuracy vary together in completed histories (App. D), separating turn-by-turn accumulation from recency effects is a next step noted in Limitations.

2. Advance over prior work, and when this link holds vs. weakens (W2)

Prior work studies two separate halves of trust: Khan et al. vary static source cues to measure choice without internal states or interaction histories, while Debnath et al. analyze static trust-concept geometry from contrastive stories without interaction or action. We connect verifiable trajectories -> participant-indexed evaluations -> cross-objective decisions without task-specific direction fitting.

To test when this link holds vs. weakens, a new prompt-variation test (App. B.2) on 10 fixed Qwen3.5-27B histories (360 separately branched decisions, 120 per formulation) compared original wording with two paraphrases (question alone, or also opening and TG link). Conflict choice stayed stable at 76.25% and 78.75% (vs. 75.00% original), while the high-low TG gap stayed positive, shifting from 2.75 pts [1.50, 4.15] to 1.35 [0.20, 2.85] (paired change -1.40 [-2.60, -0.40]) and 1.90 [-0.35, 4.25]. Thus, the link holds robustly when decisions directly query the experienced dimension (Conflict), but weakens (with narrower gaps or zero-spanning CIs under paraphrased framing) when extrapolated across domains to resource allocation without repayment feedback (TG).

3. Does steering one participant alter decisions about others? (Q1)

In a new TG participant-selectivity test (Sec. 5.2; App. E.1) on Qwen3.5-27B (10 histories, 40 balanced pairs; A first, B next, separate 10-pt budgets), we applied the frozen layer-43 direction over alpha in {-4,-2,-1,0,1,2,4} only at A's query ID (B unedited), inserting A's unsteered reply verbatim before querying B while reapplying A's earlier edit:

TG measure (pts vs. alpha=0)Edited AUnedited B
Expected TG shift, alpha=-4/+4-0.463 / +0.527-0.008 / +0.014
Bidirectional effect [95% CI]0.495 [0.378, 0.607]0.011 [0.003, 0.019]

The mean bidirectional effect on edited A is ~45x that on unedited B, showing steering mainly shifts allocations to the targeted participant with much smaller effects on the other participant.

4. How consistent are the intervention effects across models? (Q2)

In a new replication (Sec. 5.4; App. F), we extended these analyses to Gemma-4-31B-it (20 dev, 10 eval histories), selecting verbalizer word sets, fitting probes, choosing layer 30 from dev descriptions, and reusing that frozen direction over 7 doses:

Mean shift at alpha=-4/+4Qwen3.5-27BGemma-4-31B-it
Verbalizer readout score-0.607 / +0.738-1.375 / +1.380
Expected TG transfer (pts)-0.461 / +0.518-0.519 / +0.478
Conflict choice (log odds)-0.338 / +0.322-0.854 / +1.196

Both show increasing 7-dose curves; Gemma's 95% CIs exclude zero at alpha=-4/+4 across readouts, patching shifts all readouts (+1.138/-2.407 TG pts), and on descriptions and TG Gemma's direction exceeds all five random controls. Qwen3.5-27B and Gemma-4-31B-it represent two mainstream ~30B dense open-weights families with independent architectures, tokenizers, and post-training stacks, showing consistent cross-task reuse across model families.

Reviewer_YCQsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks changes in LLM behavior in two iterated scenarios where a model has to make a decision based on partial information provided by another user but that information is selectively incorrect at certain rates. Using two setups, the paper shows that models are sensitive to repeated untrustworthy information and are more likely to behave in a way that favors the more trustworthy partner. In two downstream experiments, the paper shows that when asked to describe partners, the language used mirrors what a human might reasonably use to indicate a level of trust. In the second, the paper shows that model activations are predictive of the behavior (based on trust) and can be used to steer the output behavior.

Critical strengths
  • Clean experimental setup where LLM outcome is not tied to the trust level. The use of two setups (TAG and TG) both reinforce the results

  • Experiments with multiple models of a variety of capability levels

  • Goes beyond verbalization to study activation patterns and demonstrate through activation patching that these have a causal effect

Critical weaknesses
  • While I appreciate both games, it's not clear how much the games have ecological validity for more real-world LLM interactions or behaviors. The games themselves are highly stylized—which is helpful, admittedly—but it would have been nice to show that the trust behavior seen in the games is predictive of some downstream behavior, e.g., a model is more/less likely to provide answers.

  • The games use very structured prompts. Some minimal prompt variation would help show that this is general behavior (this is a minor, not critical point though).

Questions for authors
  • I would think that the model reasoning would be interesting to look at, but all of the runs have thinking as off (Table 3). Do you have a sense from any pilot work whether reasoning causes a model to be more or less sensitive to trust behaviors?
3 comments

Thank you for your careful review and constructive suggestions. We address real-world relevance, prompt sensitivity, and native-thinking behavior and cost.

1. Ecological validity and how the stylized setup connects to real-world LLM behaviors

In real-world LLM interactions, such as deciding whether to answer a user, adopt external claims, or delegate subtasks, reliance on others is entangled with pre-existing reputations, explicit authority labels, and persona cues, obscuring whether behavior reflects experienced reliability or prompt heuristics. Stylized TAG is designed not as an isolated end, but to strip away those confounds so participant evaluations arise from verifiable reporting history across unfamiliar participants.

Within this framework, TAG is only the experience-formation stage, while Conflict and TG branch from that history as two distinct downstream decision archetypes mirroring everyday agentic choices: epistemic acceptance (whose claim to trust under disagreement) and risky entrustment (how much resource or autonomy to commit without repayment feedback). Crucially, the bridge from TAG to these behaviors is mechanistic rather than merely correlational: across both Qwen3.5-27B and our new Gemma-4-31B-it replication (Sec. 5.4; App. F), one activation direction estimated solely from post-interaction participant descriptions causally shifts both downstream decisions without task-specific direction fitting (complemented by opposite-history patching).

Why does this matter beyond the games? It shows that LLMs do not treat interaction history as game-bound text, but compress verifiable participant behavior into a reusable internal evaluation that carries over across task boundaries. This provides a mechanistic basis for how prior counterpart reliability can shape real-world LLM interactions and behaviors (e.g., selective answer provision, advice adoption, or task delegation), along with a direct way to test how internal participant evaluations shape subsequent choices across tasks.

2. Does reliability-sensitive behavior persist under prompt variation?

In a new prompt-sensitivity experiment (App. B.2, Table 5) on 10 fixed Qwen3.5-27B evaluation histories (64 TAG trials, 4 participants each), we compared the original post-test wording against two pre-specified paraphrases across 360 separately branched post-test decisions (40 TG + 80 Conflict per formulation; rules, IDs, JSON schemas, and seeds fixed):

Post-test formulation (10 fixed histories)High-minus-low TG transfer, pts [95% CI]High-reliability Conflict choice [95% CI]
Original wording2.75 [1.50, 4.15]75.00% [60.00%, 88.75%]
Decision question paraphrased1.35 [0.20, 2.85]76.25% [62.50%, 90.00%]
Task opening, question, and TG link paraphrased1.90 [-0.35, 4.25]78.75% [66.25%, 90.00%]

95% bootstrap CIs resample the 10 histories. Conflict choice stays stable across paraphrases, while mean TG transfers stay higher for reliable participants with framing-dependent magnitude (question paraphrase shifts the TG gap by -1.40 pts, paired 95% CI [-2.60, -0.40], and the broader paraphrase's CI spans zero). Beyond showing that reliability sensitivity is not an artifact of one rigid template, this reveals an informative boundary: carryover from verifiable accuracy is highly invariant in direct epistemic acceptance (Conflict), yet sensitive to contextual framing when extrapolated across domains to unverified cooperative risk (TG).

3. How does native thinking affect trust sensitivity and computational cost?

In preliminary work, we explored native thinking mode and observed repeated deliberation over output formatting, multiple reviews of the same interaction evidence, and reconsideration of stated strategies. To answer your question directly, we report a matched single-run pilot comparison from our Qwen3.5-27B preliminary testing in App. B.1 (Table 4; 64 TAG trials, 4 TG decisions, 8 Conflict choices):

Single-run metric (Qwen3.5-27B, 76 items)Non-thinkingNative thinking
High-minus-low TG transfer (pts)5.07.5
High-reliability Conflict choice8/88/8
Total output tokens, incl. retries21,818246,296

Successful thinking replies averaged 2,346 thinking tokens plus an 84-token answer; including retries, thinking used 11.29x as many output tokens (~12.53 A100 GPU-hours for one run). Both modes thus show a consistent directional preference for more reliable participants (both 8/8 in Conflict and favoring reliable participants in TG), while native thinking yields stronger TG differentiation (7.5 vs. 5.0 pts) alongside an 11.29x output-token cost. Non-thinking mode therefore captures the same qualitative trust sensitivity while keeping multi-model, multi-run behavioral evaluation practical at scale.

I thank the authors for their thoughtful reply. For the ecological validity, I was more referring to general tasks and "everyday" model behavior, rather than what is shown in App. F, though I get the argument you're trying to make here. The response clarifies several points and I maintain my positive score.

Thank you for the clarification and for your positive assessment. We appreciate your suggestion to connect these findings more directly to everyday model behavior.

Reviewer_EwpBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies trust behaviors exhibited by a variety of LLMs when playing well-studied trust games, then performs mechanistic interpretability analysis to better understand how certain internal representations contribute to the trust behaviors of one model.

Critical strengths
  • The paper covers a holistic range of experimental methods: model behaviors in the actual games (sec 3), logit-weighted model descriptions (sec 4.1), probing analysis (sec 4.2), mean-difference steering (sec 5.1), and activation patching (sec 5.2). The experiments are clearly described and narrowly well-designed.
Critical weaknesses
  • The motivation is not especially clear. That is: as the authors point out (in abstract and sec 1), trust behaviors are clearly important when agents take nontrivial actions on behalf of users; but the experimental settings examined in this work do not reflect this setting at all, and consider only minimal "toy" experimental settings.
    • Of course, more naturalistic and potentially impactful settings are more difficult to study, so selecting a simplified proxy to first gain a foothold in understanding the problem is reasonable -- but the authors do not make any argument for how their findings could be plausibly connected to (e.g., predictive of, useful for steering, etc.) any such naturalistic settings.
    • Even if one were to assume that these toy settings were in fact naturalistic, high-impact scenarios, the import of the findings is still not clear.
  • Complaints about "toy" setting aside, the import of these findings are not clear even in the context of these trust games -- it seems to me that common findings across many interpretability studies are simply repeated here, without any clear sense for what these results mean in this context or how they connect to any larger questions regarding trustworthiness, reliability, interpretability, etc.
    • That is: it is observed (in sec 3) that models generally lean towards trusting more trustworthy simulated participants (it is hard to imagine how it could be otherwise, for any but the very simplest LLMs); in sec 4.1, that models are able to provide simple descriptions about participant trustworthiness that agree with observed behaviors in-context (again, it would be shocking if this were not true of any modern model); in sec 4.2, that participant trustworthiness can be predicted from internal representations via probes (which is not surprising, given that probes are well known for being able to predict many features that are not necessarily used by models -- per, e.g., the probing works cited in sec 4.2 and 6); and in sec 5, that models can be steered and patched to change their trust behaviors (indicating that these features are in fact used, but not how they are inferred, how/when/why their usage of such features can be considered reliable, etc).
    • What does any of this this tell us about how LLMs internally process trust-relevant information? Could these findings offer some help toward predicting or improving model trust behaviors in new environments? Might they help us to select or train more reliable or discerning models to be used in sensitive settings? Or indicate features that models rely on to infer trustworthiness that diverge from features used by humans? There are many such questions that could be pursued in even this simplified toy setting; but no such questions are addressed in this work -- or even speculatively discussed.
  • In all experiments beyond section 3's simple behavioral analysis (i.e., all of those throughout sec 4 and 5), only one model is examined. This severely limits the generality of all associated findings, and precludes comparative analyses between methods across models (which methods are most broadly revealing? how do they differ by model?), between models for a given method (how do models differ in their representation and use of trust-related features?), etc.
    • While the authors do provide limited explanation for why this specific model is chosen (sec 4), they do not clarify why only one model is considered -- if it is compute limitations, then at least the experiments in, e.g., sec 4.1 could easily be performed across many models, as (unless I am missing something), it would actually be much cheaper than the behavioral studies in sec 3. Most of the other experiments also require only a minimal compute budget, and could be performed on at least a small subset of the available open-weights models to show whether the observed results generalize across models, etc.
Questions for authors
  • Imagine an idealized empirical scenario, with robust results similar to those in the paper but observed across a dozen leading open-weights models in all of the settings you considered here, or even more complex agentic trust settings like those gestured at in the intro. What might the impact of such findings be? What would the purpose of the study have been?
    • I ask because, in the current draft, it comes across that a variety of experiments have been run, but I cannot string together a clear story for why they have been run, or what the broader goal of this paper is. The specific findings are clear, but what is unclear is the way they are intended to fit together, and what that broader story implies. There are a variety of impactful questions such work might be intended to address (see, e.g., the list of potential research questions enumerated in the relevant weaknesses bullet above) -- which of these, if any, are related to what you had in mind? Can you provide an alternative question in the same vein that these experiments cumulatively work towards answering?
  • Why are the specific layers and positions selected for probing and steering? Why not other layers or positions? Ablations aside, what is the assumed significance of these sites?
1 comment

Thank you for your careful review and constructive suggestions.

1. What is the overarching purpose, and how does this setting connect to naturalistic agents?

Viewed separately, each stage admits a simpler account: behavioral preference (Sec. 3) could reflect surface pattern matching; descriptions (Sec. 4.1) could be post-hoc rationalizations; probe decodability (Sec. 4.2) could capture unused features; and task-fitted steering shows only output steerability. Our experiments jointly address these concerns through a connected causal test: when an LLM accumulates verifiable experience with unfamiliar participants, does the evaluation surfaced when describing a participant act as a shared causal substrate across decisions with distinct objectives? Sec. 5 thus never fits a "TG vector" or "Conflict vector," reusing one description-query direction without task-specific direction fitting across epistemic acceptance (Conflict) and resource allocation without repayment feedback (TG), plus opposite-history patching.

  • Linking expressed evaluations to distinct choices: Rather than leaving verbalized assessments as post-hoc text, showing one description-derived direction causally shifts advice acceptance and resource allocation indicates LLMs can compress interaction records into participant-associated internal states guiding distinct forms of reliance.
  • Connecting controlled histories to realistic interactions: In realistic LLM and agent interactions (adopting claims, answering or deferring to users, delegating subtasks), reliance is confounded with pre-existing reputations, explicit authority labels, and persona cues. Stripping away those confounds first shows interaction-formed participant evaluations carry over from epistemic judgment to risky resource commitment, giving a basis to study and predict how prior history shapes multi-turn reliance.

Regarding scope, our experiments examine how post-interaction participant states guide controlled choices and do not yet test open-ended deployment or training more discerning models; Sec. 7 discusses open-ended interactions as a next step.

2. Does the internal mechanism generalize across model families, and what is the compute profile?

Our newly added Gemma-4-31B-it replication (Sec. 5.4; App. F) required open-ended adjective discovery without a preset reliability word list, complete-answer scoring across candidate trees, layer scans, and 7-dose/control interventions (133.68 recorded A100 GPU-hours, reusing behavioral runs). On 20 dev and 10 eval histories, we selected verbalizer word sets, fitted probes, chose layer 30 from dev descriptions, and reused that frozen direction over alpha in {-4,-2,-1,0,1,2,4}:

Measure (10 held-out histories)Qwen3.5-27BGemma-4-31B-it
Verbalizer readout AUC0.9780.900
Query-ID / Answer-prep probe AUC0.840 / 0.9880.893 / 0.968
Expected TG shift, alpha=-4/+4 (pts)-0.461 / +0.518-0.519 / +0.478
Conflict shift, alpha=-4/+4 (log odds)-0.338 / +0.322-0.854 / +1.196

Both models show monotonic mean increases in verbalizer score, expected TG transfer, and conflict target-choice log odds across seven doses. Gemma's 95% CIs exclude zero at alpha=-4/+4 in TG ([-0.887,-0.184], [0.186,0.791]) and Conflict ([-1.143,-0.538], [0.794,1.638]), patching shifts all readouts (+1.138/-2.407 TG pts), alternate wording keeps 0.985 answer-prep AUC (vs. 0.870 Qwen / 0.676 Gemma at query ID), and on descriptions and TG Gemma's direction exceeds all five random controls. These mainstream ~30B dense open-weight models differ in architecture, tokenizer, and post-training. These findings support cross-family generalization of the causal role of participant evaluations in both decisions.

3. Why these sites and layers, and what is their mechanistic significance?

Probing samples three stages from evidence to action (last TAG feedback, query ID, and answer preparation), while steering targets the query ID, which names each task's participant before generating descriptions or decisions (outperforming dev within-TAG sites). In both models, query-ID decoding starts near chance at layer 0 and rises to 0.840 at layer 62 of 64 (Qwen) and 0.893 at layer 59 of 60 (Gemma), consistent with dynamic participant-history retrieval across the forward pass (App. D, Fig. 8); in a new TG participant-selectivity test (Sec. 5.2; App. E.1; 40 Qwen pairs, A before B), steering only A's query ID shifts transfers to A ~45x more than B (0.495 vs. 0.011 pts), confirming a participant-selective locus. On 20 dev histories, probes select late layers (and 53, 58 at answer prep), while causal scans select mid-to-late layers (43; 30). A plausible interpretation is that mid-to-late layers route reliability-related information that becomes more linearly decodable later, showing why probe peaks need not match optimal intervention layers.

Comments on the paper

5 comments
Reviewer_EwpBOpenReview ↗

No rebuttal visible

Hi AC, I'm not able to see any such rebuttal -- no comment on my own review, that of any other reviewers, or even a general response. Possibly they set the "Readers" field incorrectly, so reviewers cannot see it?

Adding that I can't see any rebuttals either, unfortunately.

You are all correct. The reviewers did not post any rebuttal.

Apologies!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.