- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces the Contrastive Stance Probe Estimator (CSPE), a method that uses large language models (LLMs) to estimate how persuasive a message is. CSPE tracks how a message changes the model’s likelihood of agreeing with statements that support or oppose a claim. It compares these changes with a neutral reference to produce a persuasiveness score, without asking the model to rate the message or predict a person’s reaction.
The authors test CSPE on 720 persuasive messages covering 10 political topics, using three LLMs. They compare it with Likert-scale and rubric-based methods and use results from human experiments as a reference. They also examine how well the method holds up across different prompts, models, and evaluation settings.
Critical strengths
-
A simple and interesting approach to measuring persuasion. The paper introduces a different way to evaluate persuasive messages without directly asking LLMs to rate them. Instead, it uses the likelihood of supporting and opposing statements to estimate persuasiveness. I find this approach interesting because it reduces reliance on explicit LLM judgments, which can be sensitive to how evaluation questions are phrased.
-
Evaluation using real human experiments. One strength of the paper is that the authors compare CSPE with results from randomized human experiments rather than relying only on LLM-generated judgments. They also use statistical methods, such as uncertainty-weighted WPC and paired bootstrapping, to account for uncertainty in the comparisons. The additional soft-label experiments help strengthen their findings on stability. I also appreciate that they report paired confidence intervals transparently. This makes the results trustworthy and easy to access.
Critical weaknesses
- It is unclear whether CSPE measures persuasion or mainly stance agreement: A message can strongly support a claim without changing the reader’s mind. Persuasion also depends on the recipient’s prior beliefs and trust in the speaker, but CSPE’s probes and neutral reference are generated from the claim without directly accounting for these factors. The persona experiments are useful, but they do not establish whether differences in scores reflect how real audiences respond. The current experiments do not fully separate these explanations. An LLM may assign higher likelihood to a statement simply because it agrees with the preceding message. The NLI baseline’s correlation with human outcomes in Table 9 suggests that stance information already explains some performance. RQ3 attempts to address this concern, but the high- and low-persuasion texts lack independent human validation. Persuasion is also manipulated by changing the main text, while stance is changed through a short prefix, making the comparison difficult to interpret. Can the authors show that CSPE distinguishes messages that support the same stance but differ in their ability to change a recipient’s views? An evaluation using observed human outcomes and recipient information, potentially from the ChangeMyView subreddit or PersuasionForGood dataset, would help establish its value beyond stance agreement.
- Hard to follow: There is considerable room to improve the paper’s wording. In some sections, unnatural phrasing made the explanation difficult to follow.
Questions for authors
Check weaknesses.
6 comments
2. Whether Stance Agreement Alone Explains CSPE's Predictive Ability
Thank you for raising this central issue. To further test whether CSPE merely scores semantic compatibility or the direction of a message's advocated stance, we have added an analysis that controls for stance direction and examines whether CSPE can distinguish the relative persuasion effects of messages advocating the same stance.
The Spearman correlation analysis in the original RQ2 is conducted within policy issues, each containing 72 messages supporting a target claim. Building on this setup, we additionally control for the advocated stance direction and compare how well CSPE, Likert, Rubric, and the semantic-entailment-based NLI method correspond to observed human persuasion effects within the same issue and stance. The results are equally averaged across nine configurations formed by three evaluator models and three persona conditions.
Table: Experiment B — 677 Messages (Grouped by Issue × Stance)
| Method | Spearman ρ ↑ | WPC ↑ | Quartile discrimination rate ↑ |
|---|---|---|---|
| CSPE | 0.2095 | 0.5901 | 63.80% |
| Likert | 0.1093 | 0.5431 | 57.06% |
| Rubric | 0.1106 | 0.5456 | 57.10% |
| NLI | 0.0608 | 0.5235 | 54.62% |
| CSPE improvement over the best baseline | +0.0989 | +0.0445 | +6.70 pp |
The results show that even when messages share the same issue and advocated stance direction, the methods differ substantially in their ability to distinguish relative persuasion effects. After controlling for stance direction, the semantic-entailment-based NLI measure has a relatively weak correlation with observed human persuasion effects, whereas CSPE achieves the strongest results on all three metrics. Compared with the strongest baseline, Rubric, CSPE improves Spearman correlation by 0.0989, WPC by 0.0445, and the quartile discrimination rate by 6.70 percentage points. This indicates that CSPE's predictive ability is not attributable solely to identifying a message's advocated stance direction; it also reflects differences in persuasion effects among messages advocating the same stance.
To further examine CSPE's ability to distinguish different magnitudes of human persuasion effects, we pair messages within the same issue and stance, obtaining 19,249 message pairs in total. Within each issue, we divide message pairs into four groups according to the difference between their estimated persuasion effects in the human experiments. We then calculate, for each group, the accuracy with which CSPE identifies which message has the greater persuasion effect.
| Difference in human persuasion effects | Llama-2 | Llama-3.1 | GPT-OSS-120B |
|---|---|---|---|
| Smallest 25% | 51.69% | 50.82% | 51.88% |
| 25%–50% | 54.53% | 55.86% | 54.66% |
| 50%–75% | 57.80% | 59.40% | 57.82% |
| Largest 25% | 62.69% | 64.45% | 62.13% |
For all three models, ranking accuracy increases progressively as the difference in observed human persuasion effects increases. Compared with the smallest-difference group, accuracy in the largest-difference group is higher by 10.99, 13.63, and 10.25 percentage points, respectively. The 95% confidence intervals for these differences, calculated using an issue-clustered bootstrap, are all entirely above zero.
This further shows that for two messages on the same issue advocating the same stance, a larger difference in observed human persuasion effects is generally associated with a higher probability that CSPE correctly identifies the more persuasive message.
Taken together, these two additional analyses provide further validation of CSPE as a message-level persuasiveness measure: CSPE not only identifies a message's advocated stance direction but also distinguishes relative human persuasion effects among messages with the same stance direction. Direct comparisons with NLI, Likert, and Rubric indicate that CSPE's directional probability-based measurement captures information about persuasiveness beyond semantic compatibility and performs better in predicting observed human persuasion effects.
Combined with the scoring-stability experiments in the original RQ1 and the large-scale randomized human experimental validation in RQ2, these findings show that CSPE provides stable message-level scores with meaningful correspondence to observed human persuasion effects.
3. Lack of Independent Human Validation for the Matched-Quartet Experiment
Thank you for suggesting independent validation.
The original RQ3 uses 50 matched quartets (200 messages) in a 2 × 2 design that independently manipulates stance strength and persuasion level.
With high and low stance strength held fixed, CSPE's contrasts between the high- and low-persuasion conditions are 2.3886 and 1.5755, respectively.
We additionally recruited 12 human raters to evaluate the 50 matched quartets, allowing us to examine whether the intended manipulations are reflected in human judgments.
Here, HH, HL, LH, and LL correspond to high stance/high persuasion, high stance/low persuasion, low stance/high persuasion, and low stance/low persuasion, respectively.
| Manipulated dimension | Text contrast | Difference in human ratings [95% CI] |
|---|---|---|
| Persuasion level (high stance) | HH − HL | +1.32 [0.77, 1.87] |
| Persuasion level (low stance) | LH − LL | +1.47 [0.65, 2.29] |
| Stance strength (high persuasion) | HH − LH | +0.35 [−0.12, 0.83] |
| Stance strength (low persuasion) | HL − LL | +0.74 [0.12, 1.36] |
The results show that when stance strength is held fixed, human raters consistently assign higher persuasiveness ratings to texts in the high-persuasion condition.
The human-rating differences are +1.32 and +1.47 at high and low stance strength, respectively, and neither 95% CI includes zero.
The human-rating contrasts for stance strength are also positive, with a difference of +0.74 under low persuasion and a 95% CI of [0.12, 1.36].
These human-rating results further support that the manipulations of persuasion level and stance strength in the matched-quartet design operate as intended. They are also consistent with the RQ3 finding that CSPE distinguishes high- from low-persuasion texts.
We will add the human annotation results to the revised manuscript and discuss CSPE's measurement validity together with the observed human attitude-change data used in RQ2.
4. The ChangeMyView and PersuasionForGood Datasets
Thank you for recommending these two datasets.
ChangeMyView (CMV) uses delta awards from natural discussions to assess whether a reply changes a particular original poster's view. Its evaluation target therefore shifts from the message itself to the match between the message and a specific recipient, and it cannot directly quantify the persuasive effect of a single message independently of that recipient. PersuasionForGood (PFG), meanwhile, focuses on charitable donations and personalized persuasion of particular recipients through multi-turn interactions.
By contrast, CSPE focuses on the effect of a message on average attitudes at the population level, as in political and policy communication, commercial advertising and brand communication, public health communication, and public-interest campaigns.
Given this research objective, we selected the PNAS dataset because it provides persuasion-effect estimates for 720 distinct persuasive messages from randomized human experiments involving 25,982 respondents. This allows us to directly evaluate correspondence between automated scores and observed attitude changes in real audiences. The dataset also includes multiple messages for each policy issue, enabling within-issue comparisons and our additional analysis distinguishing messages with higher and lower human persuasion effects under the same stance direction.
We will further explain this dataset selection in the revised manuscript. We also recognize that multi-turn conversational persuasion targeting specific recipients is an important but complex research problem. Its effects depend not only on the persuasive message itself but also on persuasion strategies, recipients' profiles and initial attitudes, persuader characteristics, interpersonal relationships and trust, contextual factors, multi-turn interaction dynamics, and psychological mechanisms. Systematically modeling these factors and their interactions is an important direction for future research.
5. Language and Readability
Thank you for this suggestion. We will carefully edit the manuscript to improve its language, simplify complex sentences and repetitive passages, and clarify key concepts, experimental settings, and figure and table descriptions to improve readability.
Thank you again for your comments. To address the central concern about stance agreement, we have added same-stance comparisons against observed human persuasion effects and a human-rating analysis of the matched quartets. Together with the original validation against randomized human experiments, these results provide stronger empirical support for CSPE as a measure of message-level persuasiveness.
We thank the reviewer for the assessment of CSPE's methodological design and its validation against human experiments. Your central concern is whether CSPE measures message persuasiveness beyond stance agreement. We address this question using the existing experiments and additional comparisons with observed human persuasion effects.
We would also like to emphasize that CSPE's contribution is not limited to distinguishing the stance direction of messages or their relative persuasive effects. It provides a stable message-level measure whose scores exhibit meaningful correspondence with observed human persuasion effects. Below, we address your concerns in relation to our research objective, existing findings, and additional validation.
1. Persuasion Also Depends on Recipients' Prior Beliefs and Trust in the Speaker
Thank you for highlighting the importance of recipients' prior beliefs, trust in the persuader, and communication context. We agree that these factors play important roles in persuasion. We also recognize that our original presentation could have more clearly distinguished message-level persuasiveness from recipient-level persuasion effects, and we appreciate the opportunity to clarify this distinction.
As illustrated in Figure 1, our paper distinguishes message-level persuasiveness from recipient-level persuasion effects. The former concerns the relative persuasiveness of different messages at the population level, whereas the latter concerns the attitude change experienced by a particular recipient after exposure to a message, which jointly depends on prior beliefs, individual characteristics, trust relationships, and communication context.
CSPE aims to construct a message-level persuasiveness measure, not to directly predict the attitude change of a specific individual. To evaluate its validity, RQ2 uses randomized human experimental results covering ten policy issues, 720 persuasive messages, and 25,982 respondents as an external criterion. We examine correspondence between CSPE scores and the average persuasion effects observed among real human audiences. This provides empirical support for the validity of message-level scoring, but does not imply that the scores can accurately predict each recipient's individual response.
A further important advantage of CSPE is measurement stability. RQ1 of the original submission systematically evaluates ranking consistency across variations in instruction wording, persona conditions, and evaluator models. With the evaluator model and persona fixed, changing only semantically equivalent scoring instructions yields CSPE message-ranking agreement of 0.835–0.986, substantially higher than that of Likert and Rubric. On the additional ConvinceQA dataset of 5,052 messages, CSPE achieves ranking agreement of 0.910–0.913 across persona and prompt configurations.
These findings show that CSPE not only exhibits correspondence with observed human persuasion effects but also preserves relatively stable message rankings under non-substantive variations in evaluation settings, reducing measurement variation caused by changes in scoring instructions. Stability and human-outcome correspondence are complementary evaluation dimensions that jointly support CSPE's utility as a message-level persuasiveness measure.
We are also interested in recipient-level persuasion effects, as you suggest. Our persona experiments primarily assess scoring behavior under different simulated audience conditions and serve as an initial exploration of audience-conditioned evaluation. We agree that persona experiments alone do not establish whether the scores reflect attitude changes in real individuals. Further work will require modeling and validation using real recipients' initial beliefs, individual characteristics, trust relationships, and multi-turn interactions.
We will clarify the distinction between these two levels of persuasion, the research objective of CSPE, and the conclusions and limitations of the persona experiments in the revised manuscript.