- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ATF-Persu, which is a benchmark for evaluating persuasion against synthetic personas, whose beliefs / preferences are grounded in personal experiences. The key argument is that people can reach the same conclusion, but require different types of persuasion to influence their decisions. The authors validate through cross-benchmark performance, automated evaluation of persona realization and blinded human assessment.
Critical strengths
- Benchmark addresses a limitation in existing persona-based persuasion evaluation. This could help evaluate if a persuasion method responds to a person's underlying reasons, rather than simply targeting their stated positions. The five-personas per topic set-up allows controlled comparisons.
Critical weaknesses
-
Differences in evaluation protocol. Persuasion success is much lower on ATF-Persu, but the cross-benchmark experiments do not isolate the benchmark design from differences in evaluation protocol. There should have been a common success criterion to make evaluation overall more objective.
-
Insufficient support to show that the personas simulate real human behavior. The personas are llm-generated, the dialogue is llm-generated, and much of the evaluation is also automatic and llm-based. The human evaluation component is nice, but it primarily checks if readers can identify personal reasons and recognize their influence on simulated responses. This is concerning, because a synthetic persona could consistently reject arguments that do not address a very specific concern - so this reflects as being very faithful to the persona, but may not be psychologically realistic.
Questions for authors
-
As the benchmarks retain their respective success evaluation protocols, could you provide common-judge / common-criterion comparison or an analysis showing that the results are robust to evaluation criteria?
-
Is there any evidence that can support that these simulated personas are approximations of realistic human behavior? They could really just be internally consistent fictional personas. Do you have a better way to judge realism, beyond just the human evaluation checking to identify personal reasons or influence on responses?
-
For the complete vs simplified, there's a lot of proposal about causal links between experiences, beliefs and grounds for revision. Why can't you use an equally detailed but unstructured background?