acceptodds
Withdrawn from ICLR 2027

A Thousand Faces: A Persuasion Benchmark with Experience-Grounded Persuadees Informed by Psychology and Pragmatics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

People can express the same beliefs and desires yet need different reasons to change their minds. What makes an argument relevant depends on how those views arose: experiences shape what a person finds credible, what matters to them, and what could prompt reconsideration. These personal grounds may also be expressed indirectly, through contextual clues in conversation. Benchmarks that specify beliefs and desires without these connections can reward arguments that match a description while overlooking whether they address the person's reasons. We introduce A Thousand Faces (ATF-Persu), with 1,000 personas across 200 topics and nine domains. Drawing on personality and attitude theories and conversational implicature, we construct five personas per factual context, linking experiences to judgments, expression, and grounds for revision. Evaluations show lower persuasion success than three comparison benchmarks under their respective protocols. Complete backgrounds yield broader use of supplied personal grounds and higher persona faithfulness than simplified specifications; blinded reviewers also rate ATF-Persu above CToMPersu on identifiable personal reasons, their influence on responses, and contextual expression.

Primary area:
datasets and benchmarks
Keywords:
Persuasion, Large Language Models, User Simulation, Persona Modeling, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_u67SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ATF-Persu, which is a benchmark for evaluating persuasion against synthetic personas, whose beliefs / preferences are grounded in personal experiences. The key argument is that people can reach the same conclusion, but require different types of persuasion to influence their decisions. The authors validate through cross-benchmark performance, automated evaluation of persona realization and blinded human assessment.

Critical strengths
  1. Benchmark addresses a limitation in existing persona-based persuasion evaluation. This could help evaluate if a persuasion method responds to a person's underlying reasons, rather than simply targeting their stated positions. The five-personas per topic set-up allows controlled comparisons.
Critical weaknesses
  1. Differences in evaluation protocol. Persuasion success is much lower on ATF-Persu, but the cross-benchmark experiments do not isolate the benchmark design from differences in evaluation protocol. There should have been a common success criterion to make evaluation overall more objective.

  2. Insufficient support to show that the personas simulate real human behavior. The personas are llm-generated, the dialogue is llm-generated, and much of the evaluation is also automatic and llm-based. The human evaluation component is nice, but it primarily checks if readers can identify personal reasons and recognize their influence on simulated responses. This is concerning, because a synthetic persona could consistently reject arguments that do not address a very specific concern - so this reflects as being very faithful to the persona, but may not be psychologically realistic.

Questions for authors
  1. As the benchmarks retain their respective success evaluation protocols, could you provide common-judge / common-criterion comparison or an analysis showing that the results are robust to evaluation criteria?

  2. Is there any evidence that can support that these simulated personas are approximations of realistic human behavior? They could really just be internally consistent fictional personas. Do you have a better way to judge realism, beyond just the human evaluation checking to identify personal reasons or influence on responses?

  3. For the complete vs simplified, there's a lot of proposal about causal links between experiences, beliefs and grounds for revision. Why can't you use an equally detailed but unstructured background?

Reviewer_AuiDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces A Thousand Faces persuasion benchmark (ATF-Persu), which provides simulated persuasion in which the persuadee personas are grounded in factual context (e.g. past experiences, then connected to beliefs and views). These experience-grounded persuadees are constructed through psychology theories (cognitive-affective personality system, causal attitude network model). They show that this benchmark is more difficulty (lower persuasion success rate) and has higher persona faithfulness than other persuasion benchmarks.

Critical strengths

The grounding of this construction in psychological theory and empiricism is well-motivated. The work points out an intuitive gap in existing persona benchmarks, and the blind human eval suggests that this benchmark succeeds in making a benchmark that intuitively grounds persuasion reasoning better. Hence, aspects are intuitively well-motivated, positively influencing my score.

The paper's experiments establish a considerable gap in performance on several models with previous persuasion benchmarks. The others are approaching saturation; this is far from it on these models -- this establishes some significance to the utility of the benchmark.

Critical weaknesses

My main concern is in a lack of validation suggesting that this actually benchmarks what (presumably) it is intended to benchmark: whether a model would be good at persuading a person. The evidence for persona faithfulness and human rating on identifiable personal reasons is useful for suggesting this, but as it stands, I think there is a big open question as to the motivation for hillclimbing on this benchmark: is it a hard benchmark for persuading a simulated agent, or, does it really measure something about persuading people?

I also think it would be helpful to run the benchmark on better models. These models are doing poorly, but they are also cheaper variants of models. Costs might be a concern here, but the error bars are so far from other benchmark results, so I think you could run more expensive models on a subset of the benchmark and still give readers a clear sense of how the frontier does here.

Questions for authors

all asked above

Reviewer_eDX4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

LLMs have been used as persuaders in different scenarios, calling for more reliable benchmarks. However, even when people with similar beliefs and desires are persuaded, such change may be due to different personal grounds, like specific experience. And simply specifying beliefs and desires in a benchmark may not cover those personal grounds. Inspired by such, this paper is mainly to provide a new benchmark, consisting of synthetic personas with more comprehensive backgrounds. The paper begins with introducing the method of constructing such benchmark. The benchmark can cover a wide range of topics. For each topic, there is shared context and a group of personas. With such setting, the papers introduces the procedure for constructing evidence-supported factual contexts, persona construction, and finally the persona-grounded interaction stream. Review and refinement are also performed to finalize the benchmark. With such constructed benchmark, in the experiments three persuasion methods are considered, and three different LLMs are used to play the role of persuadees. Among concerned benchmarks, the proposed one, ATF-Persu, consistently achieves the lowest success rate. In addition, it is observed that conversations due to ATF-Persu have the highest effective individualization breadth and persona faithfulness. They also report some further results on consistency in different aspects.

Critical strengths

Important question

  • The paper tries to provide a new benchmark for LLMs persuasion, by especially creating persona that incorporate personal experience and backgrounds, which provided more detailed settings for analyzing persuasion behaviours and addressing individual concerns.
Critical weaknesses

Insufficient validation of realistic responses to persuasion

  • Although there are some empirical results on the consistency between personal grounds from the benchmark and the resulting conversations, such checks still may not be sufficient to show that the synthetic personas are close enough to real people when responding to persuaders.
  • In table 3, observing that ATF-Persu having scores closer to human references might have rather limited implications. Since different conversations are used in different groups, such closeness may not necessarily justify that in the same situations humans and the synthetic personas would lead to similar responses.
  • Further validation or discussion would be helpful to clarify whether the benchmark captures realistic responses to persuasion.

Insufficient explanation of the construction procedure and theoretical connections

  • Throughout the method section, a lot of notations are introduced, though many of they are just for describing some data generating step, which can actually be easier to understand if using natural language.
  • Some operational steps are still not clear enough. For example, how 'Select' chooses topics; how the seed materials are created.
  • Some use of notations may be confusing. For example, the seed materials and the topic-wise record both use X\mathcal{X}.
  • The paper refers to several psychological theories, but their connections to the actual prompts and implementation could be explained more explicitly.
Questions for authors

For clarification

  • Could the authors clarify some construction steps, like how topics are selected and how the seed materials are generated?
  • Could the authors explain more explicitly how the psychological theories inform the actual prompts and implementation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.