- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes PERSUASIONTRACE, a framework for studying belief dynamics in multi-turn human–LLM persuasion. It collects turn-level belief reports, analyzes human belief trajectories, and develops a Bayesian-network-based simulated target. The paper further evaluates the simulator through human-likeness judgments, forced replay, and several diagnostic tests, showing that the choice of simulated target can substantially affect the measured quality of a persuader.
Critical strengths
The paper goes beyond standard pre/post persuasion evaluation and directly measures belief changes across turns. This is a useful direction for studying both human persuasion and simulated users. The observation that different simulated targets can lead to different rankings of persuaders is also important.
Critical weaknesses
The paper is not always easy to follow. The distinction between endpoint-level, process-level, and turn-level evaluation could be explained more clearly. Some figures are also insufficiently integrated into the text; for example, Figures 1 and 3 are not explicitly referenced in the main text. Several figures could also be more polished and easier to interpret.
Several human-study cohorts differ in topic source, number of turns, participant allocation, and collection conditions, which makes some cross-cohort comparisons difficult to interpret.
The proposed Bayesian simulator is not cleanly separated from its LLM-based components. A stronger ablation would be needed to show that the Bayesian state representation and update mechanism, rather than the additional LLM processing pipeline, are responsible for the improvement.
The process-level findings are currently rather limited. The reported trajectory clusters are mainly separated by how much participants move overall, so it is not yet clear that turn-level tracing reveals substantially richer belief dynamics beyond high-shift versus low-shift behavior.
Repeated belief elicitation may itself affect the interaction. The pre/post-only comparison addresses aggregate persuasion effects, but does not establish that repeated elicitation leaves the shape of the measured belief trajectories unchanged.
Questions for authors
These issues are intertwined with the paper’s framing and experimental design, and in my view cannot be resolved by minor clarification alone.