Reaction-Free Persuasion Measurement via Contrastive Likelihood Probing

Scalable evaluation of persuasive messages is increasingly important as large language models (LLMs) generate persuasive content. Yet human experiments are costly, while common LLM-based evaluators rely on simulated audience reactions or holistic judgments that can be sensitive to seemingly innocuous changes in evaluation instructions. This sensitivity makes it difficult to distinguish message-level persuasiveness from measurement reactivity. We study the estimation of message-level persuasiveness using aggregate human treatment effects as an external criterion and introduce the Contrastive Stance Probe Estimator (CSPE), a reaction-free readout that measures how a message shifts the conditional likelihood of balanced third-party pro and con stance probes relative to a claim-matched neutral reference. We evaluate CSPE against Likert and Rubric-score baselines on 720 messages spanning ten policy issues and three evaluator models, and test stability transfer on an additional dataset of 5,052 messages. Across evaluator models and persona conditions, CSPE is substantially more stable to meaning-preserving changes in scoring instructions and consistently achieves the highest observed correspondence with human persuasion effects. A separate matched-quartet validation that independently manipulates stance strength and persuasion level further shows that CSPE remains sensitive to persuasion at both stance levels, whereas an entailment-based reference does not. Together, these results establish CSPE as a reproducible message-level estimator that combines high configuration stability with consistently stronger observed correspondence to aggregate human persuasion effects than the evaluated elicited-response baselines.