acceptodds
Under review as a conference paper at ICLR 2027

Reaction-Free Persuasion Measurement via Contrastive Likelihood Probing

Abstract

Scalable evaluation of persuasive messages is increasingly important as large language models (LLMs) generate persuasive content. Yet human experiments are costly, while common LLM-based evaluators rely on simulated audience reactions or holistic judgments that can be sensitive to seemingly innocuous changes in evaluation instructions. This sensitivity makes it difficult to distinguish message-level persuasiveness from measurement reactivity. We study the estimation of message-level persuasiveness using aggregate human treatment effects as an external criterion and introduce the Contrastive Stance Probe Estimator (CSPE), a reaction-free readout that measures how a message shifts the conditional likelihood of balanced third-party pro and con stance probes relative to a claim-matched neutral reference. We evaluate CSPE against Likert and Rubric-score baselines on 720 messages spanning ten policy issues and three evaluator models, and test stability transfer on an additional dataset of 5,052 messages. Across evaluator models and persona conditions, CSPE is substantially more stable to meaning-preserving changes in scoring instructions and consistently achieves the highest observed correspondence with human persuasion effects. A separate matched-quartet validation that independently manipulates stance strength and persuasion level further shows that CSPE remains sensitive to persuasion at both stance levels, whereas an entailment-based reference does not. Together, these results establish CSPE as a reproducible message-level estimator that combines high configuration stability with consistently stronger observed correspondence to aggregate human persuasion effects than the evaluated elicited-response baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Persuasion Measurement, Human-Grounded Evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_iw7ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces the Contrastive Stance Probe Estimator (CSPE), a method that uses large language models (LLMs) to estimate how persuasive a message is. CSPE tracks how a message changes the model’s likelihood of agreeing with statements that support or oppose a claim. It compares these changes with a neutral reference to produce a persuasiveness score, without asking the model to rate the message or predict a person’s reaction.

The authors test CSPE on 720 persuasive messages covering 10 political topics, using three LLMs. They compare it with Likert-scale and rubric-based methods and use results from human experiments as a reference. They also examine how well the method holds up across different prompts, models, and evaluation settings.

Critical strengths
  1. A simple and interesting approach to measuring persuasion. The paper introduces a different way to evaluate persuasive messages without directly asking LLMs to rate them. Instead, it uses the likelihood of supporting and opposing statements to estimate persuasiveness. I find this approach interesting because it reduces reliance on explicit LLM judgments, which can be sensitive to how evaluation questions are phrased.

  2. Evaluation using real human experiments. One strength of the paper is that the authors compare CSPE with results from randomized human experiments rather than relying only on LLM-generated judgments. They also use statistical methods, such as uncertainty-weighted WPC and paired bootstrapping, to account for uncertainty in the comparisons. The additional soft-label experiments help strengthen their findings on stability. I also appreciate that they report paired confidence intervals transparently. This makes the results trustworthy and easy to access.

Critical weaknesses
  • It is unclear whether CSPE measures persuasion or mainly stance agreement: A message can strongly support a claim without changing the reader’s mind. Persuasion also depends on the recipient’s prior beliefs and trust in the speaker, but CSPE’s probes and neutral reference are generated from the claim without directly accounting for these factors. The persona experiments are useful, but they do not establish whether differences in scores reflect how real audiences respond. The current experiments do not fully separate these explanations. An LLM may assign higher likelihood to a statement simply because it agrees with the preceding message. The NLI baseline’s correlation with human outcomes in Table 9 suggests that stance information already explains some performance. RQ3 attempts to address this concern, but the high- and low-persuasion texts lack independent human validation. Persuasion is also manipulated by changing the main text, while stance is changed through a short prefix, making the comparison difficult to interpret. Can the authors show that CSPE distinguishes messages that support the same stance but differ in their ability to change a recipient’s views? An evaluation using observed human outcomes and recipient information, potentially from the ChangeMyView subreddit or PersuasionForGood dataset, would help establish its value beyond stance agreement.
  • Hard to follow: There is considerable room to improve the paper’s wording. In some sections, unnatural phrasing made the explanation difficult to follow.
Questions for authors

Check weaknesses.

6 comments

We thank the reviewer for the assessment of CSPE's methodological design and its validation against human experiments. Your central concern is whether CSPE measures message persuasiveness beyond stance agreement. We address this question using the existing experiments and additional comparisons with observed human persuasion effects.

We would also like to emphasize that CSPE's contribution is not limited to distinguishing the stance direction of messages or their relative persuasive effects. It provides a stable message-level measure whose scores exhibit meaningful correspondence with observed human persuasion effects. Below, we address your concerns in relation to our research objective, existing findings, and additional validation.

1. Persuasion Also Depends on Recipients' Prior Beliefs and Trust in the Speaker

Thank you for highlighting the importance of recipients' prior beliefs, trust in the persuader, and communication context. We agree that these factors play important roles in persuasion. We also recognize that our original presentation could have more clearly distinguished message-level persuasiveness from recipient-level persuasion effects, and we appreciate the opportunity to clarify this distinction.

As illustrated in Figure 1, our paper distinguishes message-level persuasiveness from recipient-level persuasion effects. The former concerns the relative persuasiveness of different messages at the population level, whereas the latter concerns the attitude change experienced by a particular recipient after exposure to a message, which jointly depends on prior beliefs, individual characteristics, trust relationships, and communication context.

CSPE aims to construct a message-level persuasiveness measure, not to directly predict the attitude change of a specific individual. To evaluate its validity, RQ2 uses randomized human experimental results covering ten policy issues, 720 persuasive messages, and 25,982 respondents as an external criterion. We examine correspondence between CSPE scores and the average persuasion effects observed among real human audiences. This provides empirical support for the validity of message-level scoring, but does not imply that the scores can accurately predict each recipient's individual response.

A further important advantage of CSPE is measurement stability. RQ1 of the original submission systematically evaluates ranking consistency across variations in instruction wording, persona conditions, and evaluator models. With the evaluator model and persona fixed, changing only semantically equivalent scoring instructions yields CSPE message-ranking agreement of 0.835–0.986, substantially higher than that of Likert and Rubric. On the additional ConvinceQA dataset of 5,052 messages, CSPE achieves ranking agreement of 0.910–0.913 across persona and prompt configurations.

These findings show that CSPE not only exhibits correspondence with observed human persuasion effects but also preserves relatively stable message rankings under non-substantive variations in evaluation settings, reducing measurement variation caused by changes in scoring instructions. Stability and human-outcome correspondence are complementary evaluation dimensions that jointly support CSPE's utility as a message-level persuasiveness measure.

We are also interested in recipient-level persuasion effects, as you suggest. Our persona experiments primarily assess scoring behavior under different simulated audience conditions and serve as an initial exploration of audience-conditioned evaluation. We agree that persona experiments alone do not establish whether the scores reflect attitude changes in real individuals. Further work will require modeling and validation using real recipients' initial beliefs, individual characteristics, trust relationships, and multi-turn interactions.

We will clarify the distinction between these two levels of persuasion, the research objective of CSPE, and the conclusions and limitations of the persona experiments in the revised manuscript.

2. Whether Stance Agreement Alone Explains CSPE's Predictive Ability

Thank you for raising this central issue. To further test whether CSPE merely scores semantic compatibility or the direction of a message's advocated stance, we have added an analysis that controls for stance direction and examines whether CSPE can distinguish the relative persuasion effects of messages advocating the same stance.

The Spearman correlation analysis in the original RQ2 is conducted within policy issues, each containing 72 messages supporting a target claim. Building on this setup, we additionally control for the advocated stance direction and compare how well CSPE, Likert, Rubric, and the semantic-entailment-based NLI method correspond to observed human persuasion effects within the same issue and stance. The results are equally averaged across nine configurations formed by three evaluator models and three persona conditions.

Table: Experiment B — 677 Messages (Grouped by Issue × Stance)

MethodSpearman ρ ↑WPC ↑Quartile discrimination rate ↑
CSPE0.20950.590163.80%
Likert0.10930.543157.06%
Rubric0.11060.545657.10%
NLI0.06080.523554.62%
CSPE improvement over the best baseline+0.0989+0.0445+6.70 pp

The results show that even when messages share the same issue and advocated stance direction, the methods differ substantially in their ability to distinguish relative persuasion effects. After controlling for stance direction, the semantic-entailment-based NLI measure has a relatively weak correlation with observed human persuasion effects, whereas CSPE achieves the strongest results on all three metrics. Compared with the strongest baseline, Rubric, CSPE improves Spearman correlation by 0.0989, WPC by 0.0445, and the quartile discrimination rate by 6.70 percentage points. This indicates that CSPE's predictive ability is not attributable solely to identifying a message's advocated stance direction; it also reflects differences in persuasion effects among messages advocating the same stance.

To further examine CSPE's ability to distinguish different magnitudes of human persuasion effects, we pair messages within the same issue and stance, obtaining 19,249 message pairs in total. Within each issue, we divide message pairs into four groups according to the difference between their estimated persuasion effects in the human experiments. We then calculate, for each group, the accuracy with which CSPE identifies which message has the greater persuasion effect.

Difference in human persuasion effectsLlama-2Llama-3.1GPT-OSS-120B
Smallest 25%51.69%50.82%51.88%
25%–50%54.53%55.86%54.66%
50%–75%57.80%59.40%57.82%
Largest 25%62.69%64.45%62.13%

For all three models, ranking accuracy increases progressively as the difference in observed human persuasion effects increases. Compared with the smallest-difference group, accuracy in the largest-difference group is higher by 10.99, 13.63, and 10.25 percentage points, respectively. The 95% confidence intervals for these differences, calculated using an issue-clustered bootstrap, are all entirely above zero.

This further shows that for two messages on the same issue advocating the same stance, a larger difference in observed human persuasion effects is generally associated with a higher probability that CSPE correctly identifies the more persuasive message.

Taken together, these two additional analyses provide further validation of CSPE as a message-level persuasiveness measure: CSPE not only identifies a message's advocated stance direction but also distinguishes relative human persuasion effects among messages with the same stance direction. Direct comparisons with NLI, Likert, and Rubric indicate that CSPE's directional probability-based measurement captures information about persuasiveness beyond semantic compatibility and performs better in predicting observed human persuasion effects.

Combined with the scoring-stability experiments in the original RQ1 and the large-scale randomized human experimental validation in RQ2, these findings show that CSPE provides stable message-level scores with meaningful correspondence to observed human persuasion effects.

3. Lack of Independent Human Validation for the Matched-Quartet Experiment

Thank you for suggesting independent validation.

The original RQ3 uses 50 matched quartets (200 messages) in a 2 × 2 design that independently manipulates stance strength and persuasion level.

With high and low stance strength held fixed, CSPE's contrasts between the high- and low-persuasion conditions are 2.3886 and 1.5755, respectively.

We additionally recruited 12 human raters to evaluate the 50 matched quartets, allowing us to examine whether the intended manipulations are reflected in human judgments.

Here, HH, HL, LH, and LL correspond to high stance/high persuasion, high stance/low persuasion, low stance/high persuasion, and low stance/low persuasion, respectively.

Manipulated dimensionText contrastDifference in human ratings [95% CI]
Persuasion level (high stance)HH − HL+1.32 [0.77, 1.87]
Persuasion level (low stance)LH − LL+1.47 [0.65, 2.29]
Stance strength (high persuasion)HH − LH+0.35 [−0.12, 0.83]
Stance strength (low persuasion)HL − LL+0.74 [0.12, 1.36]

The results show that when stance strength is held fixed, human raters consistently assign higher persuasiveness ratings to texts in the high-persuasion condition.

The human-rating differences are +1.32 and +1.47 at high and low stance strength, respectively, and neither 95% CI includes zero.

The human-rating contrasts for stance strength are also positive, with a difference of +0.74 under low persuasion and a 95% CI of [0.12, 1.36].

These human-rating results further support that the manipulations of persuasion level and stance strength in the matched-quartet design operate as intended. They are also consistent with the RQ3 finding that CSPE distinguishes high- from low-persuasion texts.

We will add the human annotation results to the revised manuscript and discuss CSPE's measurement validity together with the observed human attitude-change data used in RQ2.

4. The ChangeMyView and PersuasionForGood Datasets

Thank you for recommending these two datasets.

ChangeMyView (CMV) uses delta awards from natural discussions to assess whether a reply changes a particular original poster's view. Its evaluation target therefore shifts from the message itself to the match between the message and a specific recipient, and it cannot directly quantify the persuasive effect of a single message independently of that recipient. PersuasionForGood (PFG), meanwhile, focuses on charitable donations and personalized persuasion of particular recipients through multi-turn interactions.

By contrast, CSPE focuses on the effect of a message on average attitudes at the population level, as in political and policy communication, commercial advertising and brand communication, public health communication, and public-interest campaigns.

Given this research objective, we selected the PNAS dataset because it provides persuasion-effect estimates for 720 distinct persuasive messages from randomized human experiments involving 25,982 respondents. This allows us to directly evaluate correspondence between automated scores and observed attitude changes in real audiences. The dataset also includes multiple messages for each policy issue, enabling within-issue comparisons and our additional analysis distinguishing messages with higher and lower human persuasion effects under the same stance direction.

We will further explain this dataset selection in the revised manuscript. We also recognize that multi-turn conversational persuasion targeting specific recipients is an important but complex research problem. Its effects depend not only on the persuasive message itself but also on persuasion strategies, recipients' profiles and initial attitudes, persuader characteristics, interpersonal relationships and trust, contextual factors, multi-turn interaction dynamics, and psychological mechanisms. Systematically modeling these factors and their interactions is an important direction for future research.

5. Language and Readability

Thank you for this suggestion. We will carefully edit the manuscript to improve its language, simplify complex sentences and repetitive passages, and clarify key concepts, experimental settings, and figure and table descriptions to improve readability.

Thank you again for your comments. To address the central concern about stance agreement, we have added same-stance comparisons against observed human persuasion effects and a human-rating analysis of the matched quartets. Together with the original validation against randomized human experiments, these results provide stronger empirical support for CSPE as a measure of message-level persuasiveness.

Reviewer_yFQxOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CSPE, a training-free metric that aims to reduce the prompt sensitivity of conventional Likert- and rubric-based persuasion measures while achieving closer correspondence with persuasion effects observed in human experiments. For each target claim, CSPE uses balanced sets of supporting and opposing stance statements, together with a claim-matched neutral reference text. For each statement, it computes the change in conditional log-likelihood when the neutral reference is replaced by the persuasive message. The final score is the mean likelihood shift for supporting statements minus the mean shift for opposing statements. This contrast is intended to distinguish a directional shift toward the advocated claim from a general increase in topic salience. The paper evaluates the proposed metric along three dimensions: stability, human-outcome correspondence, and construct validation.

Critical strengths
  1. CSPE is grounded in a clear intuition: changes in the conditional likelihoods of supporting and opposing statements can provide a meaningful signal of how a message steers the model’s distribution. Using a fixed likelihood-based contrast reduces reliance on explicitly elicited ratings and provides a more transparent basis for measurement than rubric-based scoring.

  2. The paper separately evaluates whether the metric produces consistent rankings across evaluator configurations and whether those rankings correspond to persuasion effects observed in human experiments. This distinction is important, as stability alone does not establish the validity of a persuasion measure. The results show substantial improvements in stability and consistently higher point estimates for the primary human-correspondence metrics relative to the evaluated Likert and Rubric baselines.

  3. The evaluation considers a meaningful alternative explanation for CSPE’s stability advantage: that continuous scores may inherently be more stable than discrete ratings. The inclusion of soft-label baselines directly addresses this concern, while component ablations examine the contributions of the pro–con contrast and neutral reference. Together, these analyses make the evaluation more thorough and strengthen the empirical support for the proposed design.

Critical weaknesses
  1. Insufficient comparison with closely related persuasion metrics. There is already substantial work on automatically measuring persuasiveness, yet the paper does not adequately position CSPE against closely related approaches, particularly Nguyen et al. (EMNLP 2025), How Persuasive Is Your Context?, and Du et al. (ACL 2024), Context versus Prior Knowledge in Language Models. Both examine how contextual information shifts a language model’s output distribution, making them relevant to CSPE’s underlying measurement principle. Although their evaluation settings differ from the human persuasion setting studied here, a clearer methodological comparison, together with appropriately adapted empirical baselines where feasible, would help establish CSPE’s incremental contribution. Without these comparisons, its novelty and advantages over existing probability-based measures remain difficult to assess.

  2. Insufficient separation of persuasive influence from stance strength and continuation compatibility. Statements can affect the likelihood of pro and con continuations simply by expressing different stances or degrees of certainty, without providing substantive supporting arguments. For example, “I am not particularly inclined to support this policy” and “This policy is clearly right, and none of the objections hold up” could produce different probe likelihoods because a language model favors continuations consistent with the preceding stance and tone. Such a likelihood difference does not, by itself, establish a difference in persuasive effectiveness. Although the matched-quartet experiment attempts to address this concern, it lacks an independent manipulation check confirming that the intended variation in persuasiveness is successfully separated from changes in overall stance strength, certainty, or semantic compatibility. Consequently, the current evidence does not sufficiently rule out these factors as alternative explanations for part of the CSPE signal.

  3. Limited evidence that measurement validity generalizes to new claims and probe constructions. CSPE requires a separate set of pro probes, con probes, and a neutral reference for every target claim. Although these materials can be generated automatically and reused across messages targeting the same claim, the resulting scores depend on the claim-specific measurement instrument. It remains unclear whether the reported correspondence with human persuasion effects reliably carries over to new claims and independently constructed probe panels. The ConvinceQA transfer experiment primarily establishes configuration stability, while the alternative probe-generation experiment provides limited evidence about the preservation of human-outcome correspondence across panels. A stronger evaluation would fix the material-generation and quality-control procedures in advance, then assess human-outcome correspondence on new claims and across multiple independently generated panels.

Questions for authors
  1. Could the authors address these concerns and clarify whether any of them stem from a misunderstanding of the method or experimental setup? Additional explanations or supporting evidence would help assess these points more fairly.

  2. Under the GPT-OSS-120B, no-persona setting, Table 1 reports mean wording stability of 0.491 for Likert and 0.578 for Rubric, whereas Table 2 reports 0.255 and 0.264, respectively, for the corresponding model and persona. Could the authors explain why these values differ and clarify whether the two tables use different data subsets, evaluation settings, or aggregation procedures?

6 comments

We thank the reviewer for the careful assessment of CSPE's methodological design and experimental analyses. In response to your concerns about comparisons with related work, construct separation, and measurement generalization, we have conducted the following additional analyses and experiments.

1. Comparison with Existing Probability-Based Persuasion Measures

Thank you for pointing out the relevant work by Du et al. (ACL 2024) and Nguyen et al. (EMNLP 2025).

Like CSPE, these approaches use language models' conditional probability distributions to measure contextual influence, but their measurement targets differ. Du et al. use mutual information to characterize how context affects a model's use of knowledge, while Nguyen et al.'s TPS uses Wasserstein distance to measure movement of an answer distribution toward a target distribution.

By contrast, CSPE constructs a message-level score from changes in the conditional likelihoods of pro and con stance probes relative to a claim-matched neutral reference, and evaluates its predictive ability against observed persuasion effects from randomized human experiments.

Following your suggestion, we have added adapted evaluations of measureLM KL, BasicTPS, and Ordinal TPS. Results for the two evaluator models are shown below:

Evaluator modelMethodPearson rSpearman ρ
Llama-2-7B-ChatmeasureLM KL−0.084−0.003
Llama-2-7B-ChatBasicTPS−0.035−0.002
Llama-2-7B-ChatOrdinal TPS0.0760.048
Llama-3.1-8BmeasureLM KL−0.050-0.003
Llama-3.1-8BBasicTPS0.2070.160
Llama-3.1-8BOrdinal TPS−0.079−0.232

For reference, under the original 4+4, no-persona, direct, w1 configuration, CSPE achieves Human Spearman correlations of 0.321 and 0.333 on the two models, respectively.

Among the newly adapted baselines, BasicTPS achieves a Spearman correlation of 0.160 on Llama-3.1-8B, while the other adapted baselines show lower correlations. CSPE achieves higher point estimates of human-outcome correspondence on both evaluator models.

These results further support the CSPE design: a directional measure tailored to persuasion, constructed from fixed, bidirectional stance probes and a neutral reference, can more effectively predict the relative persuasion effects of messages observed in real human experiments.

We will add a more detailed discussion of the related work and these experimental results to the revised manuscript.

2. Distinguishing Persuasiveness from Stance Strength and Semantic Compatibility

This is an important question concerning the construct validity of our measure.

CSPE measures the directional likelihood changes induced by a message through two contrasts: the contrast between pro and con stance probes and the contrast between the persuasive message and the claim-matched neutral reference.

To further separate persuasiveness from stance strength, RQ3 constructs 50 matched quartets (200 texts in total) in a 2 × 2 design that independently manipulates persuasion level and stance strength.

The original results show CSPE persuasion contrasts of 2.3886 and 1.5755 when high and low stance strength, respectively, are held fixed.

In response to your suggestion for independent validation, we additionally recruited 12 human raters to evaluate the 50 matched quartets. We compared their ratings to examine whether the intended manipulations of persuasion level and stance strength are reflected in human evaluations.

Here, HH, HL, LH, and LL denote high stance/high persuasion, high stance/low persuasion, low stance/high persuasion, and low stance/low persuasion, respectively.

Manipulated dimensionText contrastDifference in human ratings [95% CI]
Persuasion level (high stance)HH − HL+1.32 [0.77, 1.87]
Persuasion level (low stance)LH − LL+1.47 [0.65, 2.29]
Stance strength (high persuasion)HH − LH+0.35 [−0.12, 0.83]
Stance strength (low persuasion)HL − LL+0.74 [0.12, 1.36]

At both levels of stance strength, texts in the high-persuasion condition receive higher human persuasiveness ratings. The rating difference is +1.32 under high stance strength and +1.47 under low stance strength; neither 95% CI includes zero.

The stance-strength contrasts also show positive differences: +0.35 under high persuasion and +0.74 under low persuasion, with the 95% CI for the latter excluding zero.

These findings indicate that the manipulations of persuasion level and stance strength in the matched-quartet design operate in the intended directions.

The human evaluations complement the CSPE results in RQ3: with stance strength held fixed, human raters recognize the intended differences in persuasiveness, and those differences correspond to changes in CSPE scores.

We will include the human-rating results in the revised manuscript to strengthen the construct validation in RQ3.

3. Measurement Validity on New Claims and Independently Constructed Probe Panels

Thank you for raising the questions of probe construction and measurement generalization.

Regarding the probe-construction protocol, CSPE generates four pro probes, four con probes, and one claim-matched neutral reference for each target claim. The original experiments used GPT-5.6 to generate these measurement materials. All messages targeting the same claim share the same probe sets and neutral reference, and probe generation uses only the target claim as input.

In the revised manuscript, we will provide the full generation prompts and probe quality-control procedures.

Regarding robustness across probe panels, Appendix D.4 of the original submission already reports results on Llama-3.1-8B using probes regenerated with DeepSeek-V3.2.

Following your suggestion, we have additionally constructed three independent probe panels (v2, v3, and v4) and compared their measurement results on Llama-2 and Llama-3.1.

Evaluator modelPanelHuman Spearman ↑WPC ↑Wording stability ↑
Llama-2v20.3180.6380.984
Llama-2v30.3090.6350.984
Llama-2v40.3110.6390.987
Llama-3.1v20.3260.6410.976
Llama-3.1v30.3110.6340.977
Llama-3.1v40.3240.6400.980

The results show that all three additional panels maintain high wording stability, with relatively limited variation in human-outcome correlations and WPC. This provides further evidence of CSPE's measurement robustness across different probe panels.

We will provide the full generation prompts for both probes and fillers in a separately uploaded supplementary file.

Regarding generalization to new claims, RQ2 currently evaluates CSPE using a dataset of 720 messages across ten policy issues with observed human persuasion effects, whereas ConvinceQA is used to assess stability across evaluator configurations. We will make the distinct validation purposes of these experiments explicit and identify human-outcome validation on independent new issues as a direction for future work.

4. Differences Between the Values in Table 1 and Table 2

Thank you for pointing out the apparent discrepancy between Tables 1 and 2. We realize that our original presentation did not clearly distinguish the data sources and statistical scopes of these two tables, which may have caused confusion.

To clarify, the left-hand portion of Table 1 presents a motivating experiment illustrating prompt sensitivity, with its underlying data described in Appendix B. Table 2, in contrast, uses all 720 messages from the dataset of Hackenburg et al. (2025), involving 25,982 respondents.

We will explicitly identify the data sources and experimental settings of both tables in the revised manuscript to avoid this ambiguity.

Thank you again for your suggestions. We have added experiments involving related baselines, human construct validation, and independent probe panels, and will improve the corresponding methodological descriptions and results analyses in the revised manuscript.

Reviewer_id2iOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces the Contrastive Stance Probe Estimator (CSPE), a method for evaluating persuasive messages. The work notes that existing LLM-based methods for evaluating persuasive messages both fail to be stable under semantically-equivalent prompts and fail to correlate with human rankings. CSPE measures how a given message (intended for persuasion) shifts the likelihood of sets of fixed pro- and con- statements relative to a neutral reference. They demonstrate superiority of this method on stability/reproducibility and human-outcome correspondence and demonstrate construct validity as well as necessity of each term in CSPE through ablations.

Critical strengths

This work is well-motivated: persuasion is of substantial interest, and considerable work has been devoted to modeling human persuasion by LLMs as well as how persuasive LLM-based methods are. Given the expense of human subject experiments, a method that can measure persuasiveness better than others is quite useful.

The empirical support for their method is substantive -- results appear significant and do a good job of both establishing the utility of the method as well as the necessity of its components.

Both parts of these are critical to my overall assessment: the work is well-motivated, and they have good evidence that their method works.

Critical weaknesses

While analyses of significance appear to be sound, from a clarity perspective I do not know why the authors suppressed almost all CI statistics to the appendix -- this is immediately important and doing this hurts readability. This is of course very fixable, and hence, my overall assessment leans towards acceptance.

Questions for authors

How big do the probe set sizes need to be/how much does it help to have more probes? I see a 1-probe holdout in Appendix D.4 but it would be helpful to learn how much more probes help/fewer hurt, since changing the number could be costly or uncostly depending on the application of the method.

4 comments

We thank the reviewer for the positive assessment of our research motivation, methodological design, and empirical results. We respond to your two points below.

1. Presentation of Confidence Intervals (CIs)

In the original submission, we placed the complete 95% confidence intervals (CIs) in the appendix because of the nine-page limit for the main paper and the multiple evaluator models, methods, and persona conditions covered in Table 2.

In the revision, we will adjust the table layout to present CIs for the primary results and key method comparisons in the main text, while retaining the complete statistical results in the appendix.

2. Probe Set Size and Its Effect on Performance

Thank you for raising this question. The original CSPE uses four pro probes and four con probes. By aggregating multiple semantically similar statements with different wordings, this design reduces the influence of any individual probe's wording on the measurement.

Appendix D.4 of the original submission already reports a leave-one-probe-out experiment. Following your suggestion, we have additionally conducted a probe-count sensitivity experiment to systematically examine how reducing or increasing the number of probes affects human-outcome correspondence.

We use Llama-2-7B-Chat and Llama-3.1-8B, with the no-persona, direct, and w1 configuration held fixed. Taking the original 4+4 probe configuration as the reference, we compare performance using one to eight probes per stance direction. For each probe count, we report the mean metrics across balanced probe subsets rather than selecting the best-performing subset.

Number of probesLlama-2 SpearmanLlama-2 WPCLlama-3.1 SpearmanLlama-3.1 WPC
1+10.1770.5790.1780.577
2+20.2370.6050.2440.606
3+30.2780.6240.2940.628
4+4 (original)0.3210.6420.3330.644
5+50.3030.6340.3180.639
6+60.3030.6340.3170.638
7+70.3090.6360.3210.639
8+80.3150.6390.3240.640

The results show a clear pattern.

Using fewer probes reduces correspondence with human persuasion effects, whereas adding more probes does not yield further improvements in average performance.

Overall, the original 4+4 configuration achieves the highest average human-outcome correspondence in this experiment while avoiding the additional computational cost of using more probes. These results provide direct empirical support for the current choice of probe set size in CSPE.

We will include this sensitivity analysis and a comparison of probe count in the revised manuscript.

We sincerely thank you again for your positive assessment, particularly your recognition of the motivation and empirical support for our work. We hope our response, including the additional probe-count sensitivity analysis and clarification regarding confidence interval presentation, has adequately addressed your concerns. We would be happy to provide any further clarification and greatly appreciate your time and consideration.

Thanks for your update! I appreciate the additional experimentation. It's helpful to see...performance looks fairly stable but doesn't get better for higher count. I've adjusted my score appropriately.

Thank you very much for your thoughtful feedback and for taking the time to consider our additional experiments. We sincerely appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.