- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how LLMs use their impressions of others when making decisions over time. The authors introduce a two-stage experiment where models first observe a partner's honesty and then play 20 rounds of a trust game with that partner. They find that larger models initially invest less in dishonest partners, but increase their investments after receiving positive feedback. Explicitly asking models to evaluate their partners makes these initial impressions more influential. For smaller models, the authors show that information about honesty is present in their internal representations but does not always affect their decisions. Additional prompts can help these models use this information.
Critical strengths
- The two-stage experimental design separates impression formation from later decision-making, which helps study how past social information affects future behavior.
- The paper studies trust as a changing process rather than relying on a single decision, which I found to be interesting.
- The comparison between explicit evaluation, no evaluation, and a color control is well designed and helpful.
- The combination of behavioral experiments, internal representation analysis, and prompt interventions is very helpful.
Critical weaknesses
- The model comparison is heavily focused on Qwen models. Six of the seven models belong to the Qwen family, making it difficult to know whether the findings hold across different model families. Testing additional model families would strengthen the general claims.
- The honesty penalty is not necessarily a measure of better trust decisions. Since partners return the same favorable amount regardless of their honesty, investing less in dishonest partners may actually reduce the model's payoff. The paper should analyze the trade-off between using past impressions and maximizing rewards.
- The paper does not clearly separate learning from reaching the maximum investment. Some models quickly reach the 10-token limit, which can hide differences in how they update their decisions. An additional analysis that accounts for this ceiling would make the results clearer.
- The probing results may partly reflect information about honesty levels rather than a learned social impression. The probes predict a score that is itself closely related to the observed honesty history. Comparing against a simple honesty-count baseline would help establish what additional information the probes capture.
- The prompt interventions may directly encourage cautious investment rather than simply help models use existing impressions. A control prompt that mentions caution without referring to the partner's previous behavior would help separate these effects.
- The paper reports average investment patterns but gives less attention to differences across individual trials. Analyzing how often models actually change their investment strategy, rather than only comparing average trajectories, would help establish how consistent the findings are.
Questions for authors
- Does the order of honest and dishonest reports in Stage 1 affect the initial investment? For example, do models give more importance to recent behavior?
- Have the authors examined whether the explicit partner evaluations are consistent across trials? Could differences in the words models choose help explain the variation in investment behavior?