acceptodds
Under review as a conference paper at ICLR 2027

A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing

Abstract

Large language models can shift human beliefs across high-stakes domains, but most persuasion studies rely on pre/post belief change. These endpoint measures identify whether persuasion occurred, yet miss where and how beliefs moved within a dialogue. We present PERSUASIONTRACE, a framework for studying persuasion in human-LLM interaction. Built on a web-based experimental platform, PERSUASIONTRACE contributes a tool for multi-turn persuasion studies and a process-level evaluation protocol: it records multi-turn belief reports from human or simulated targets of persuasion and supports comparisons of how their beliefs move during dialogue. Using this framework, we find that human targets exhibit heterogeneous patterns of multi-turn belief updates, including low movement and early movement followed by partial regression, and that LLMs are persuasive across generic and personalized topics and multi-turn interactions. Prior work has chiefly used vanilla-prompted LLMs to simulate human targets, but we show that simulator choice materially affects apparent persuader quality. We introduce a Bayesian-network simulated target that maintains an explicit latent belief state over time so each persuader message yields a structured belief update. In blinded pairwise human-likeness evaluation with two annotators (76.1% agreement), real human targets score higher than our Bayesian target in 96.7% of matched comparisons, while the Bayesian target scores higher than the two baseline LLM targets in 77.5% of those head-to-head comparisons on average. PERSUASIONTRACE reframes persuasion evaluation from endpoint movement alone to turn-level measurement, providing a basis for studying when beliefs move and auditing simulator-based evaluations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
persuasion, user simulator, bayesian network, human experiment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LVnXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes PERSUASIONTRACE, a framework for studying belief dynamics in multi-turn human–LLM persuasion. It collects turn-level belief reports, analyzes human belief trajectories, and develops a Bayesian-network-based simulated target. The paper further evaluates the simulator through human-likeness judgments, forced replay, and several diagnostic tests, showing that the choice of simulated target can substantially affect the measured quality of a persuader.

Critical strengths

The paper goes beyond standard pre/post persuasion evaluation and directly measures belief changes across turns. This is a useful direction for studying both human persuasion and simulated users. The observation that different simulated targets can lead to different rankings of persuaders is also important.

Critical weaknesses

The paper is not always easy to follow. The distinction between endpoint-level, process-level, and turn-level evaluation could be explained more clearly. Some figures are also insufficiently integrated into the text; for example, Figures 1 and 3 are not explicitly referenced in the main text. Several figures could also be more polished and easier to interpret.

Several human-study cohorts differ in topic source, number of turns, participant allocation, and collection conditions, which makes some cross-cohort comparisons difficult to interpret.

The proposed Bayesian simulator is not cleanly separated from its LLM-based components. A stronger ablation would be needed to show that the Bayesian state representation and update mechanism, rather than the additional LLM processing pipeline, are responsible for the improvement.

The process-level findings are currently rather limited. The reported trajectory clusters are mainly separated by how much participants move overall, so it is not yet clear that turn-level tracing reveals substantially richer belief dynamics beyond high-shift versus low-shift behavior.

Repeated belief elicitation may itself affect the interaction. The pre/post-only comparison addresses aggregate persuasion effects, but does not establish that repeated elicitation leaves the shape of the measured belief trajectories unchanged.

Questions for authors

These issues are intertwined with the paper’s framing and experimental design, and in my view cannot be resolved by minor clarification alone.

Reviewer_EeFLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

While most persuasion studies focus on pre/post belief states, this paper presents PersuasionTrace: a framework for collecting multi-turn belief trajectories that introduces a process-level evaluation tool for persuasion. This tool helps understand specifically when belief changes throughout a dialog.

Using PersuasionTrace, the authors analyze persuasion dynamics in humans and present specific insights, such as heterogeneous belief-updates. Then, they present a Bayes Net belief-state simulator of persuadees and show that they have much stronger alignment with real humans compared to baseline LLM simulators.

Critical strengths

The paper conducts a thorough human study of persuasion, then simulates the persuadee with multiple baselines and demonstrates the effectiveness of their approach. The study is well-designed and analysis addresses many important questions.

Critical weaknesses

While the paper demonstrates that a Bayesian Network simulator helps across several metrics related to belief updates, the evaluation neglects several other dimensions of simulation such as fidelity, naturalness, coherence, or realism along social and emotional dimensions. It would be helpful to see how the simulator compares along such other dimensions. Additionally, further validation of their existing metrics would also be helpful (i.e. validating human-likeness LLM judge).

The paper claims to study process-level belief changes, but their analysis of these dynamics is limited. The multi-turn data serves primarily as a baseline to validate their simulator against human data, and there doesn’t seem to be much insight into the actual cognitive process.

Questions for authors

n/a

Reviewer_zRSoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Bayesian network model for simulating human persuadees in multi-turn dialogues. The approach assumes that a person’s stance is supported by a set of interconnected beliefs, such that changing individual beliefs can influence their overall stance. For each target proposition, the authors construct a Bayesian network whose nodes represent the proposition and related beliefs, with dependencies encoded by edges and conditional probabilities. The simulator updates its probabilistic belief state in response to persuasive messages and generates replies conditioned on the updated state. Experimental results indicate that the proposed simulator is judged more human-like than the evaluated LLM baselines and is less susceptible to unsupported stance assertions.

Critical strengths
  1. The paper models a person’s stance through a network of interconnected beliefs, providing an intuitive account of how changes in individual beliefs can influence their overall position. This structured representation offers a promising approach to capturing the dynamics of human belief revision during persuasive dialogues.

  2. The authors conduct experiments with human participants and collect turn-by-turn belief trajectories, using real human responses and belief changes to evaluate the simulators. This substantial data collection effort is commendable and provides a valuable empirical foundation for assessing the realism of simulated persuadees.

Critical weaknesses
  1. Insufficient ablation of the proposed components: The full method simultaneously introduces argument atomization, evidence-strength scoring, externally maintained numerical belief states, predefined probabilistic update rules, dependencies among beliefs, and a dedicated prompt for generating responses. In contrast, the two baselines primarily prompt the LLM to respond and report its current beliefs. Although all methods use the same underlying model, several components change together. The results therefore demonstrate the effectiveness of the overall pipeline but do not establish whether explicitly modeling dependencies among beliefs is a key source of improvement. Component-level ablations are needed to isolate the contribution of the Bayesian network.

  2. The paper compares human and simulated distributions of four-dimensional belief-update vectors using energy distance, on which the BN simulator outperforms the baselines. However, this metric does not preserve the correspondence between simulated trajectories and their associated human conditions. For example, humans might be easily persuaded on proposition A but resistant on proposition B, while the simulator exhibits the opposite pattern. If the resulting trajectory sets are identical, the aggregate distance would still indicate a perfect distributional match despite these mismatches. Consequently, this result alone does not establish improved fidelity for the corresponding propositions and initial conditions. Complementing it with paired or condition-specific trajectory comparisons would provide stronger evidence.

  3. The paper does not sufficiently examine whether its improvements in matching human trajectories and its reported rankings of persuaders remain stable when belief graphs are regenerated, update strengths are moderately varied, or modest changes are made to the graph structure. Without such sensitivity analyses, it remains unclear how strongly the findings depend on a particular choice of graph and update configuration, limiting confidence in the robustness of the reported results.

Questions for authors
  1. Section 4.2 reports an overall human-likeness win rate of 52.8% for the BN simulator and an average of 25.0% for the two baselines, whereas the Discussion reports 66% and 19%, respectively. Could you clarify this discrepancy? Do these figures reflect different evaluation settings or aggregation procedures, or have I misinterpreted them?

  2. Do any of the concerns raised in the Weaknesses section stem from a misunderstanding of the methodology or evaluation? If so, please clarify the relevant details and explain how they address these concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.