acceptodds
Under review as a conference paper at ICLR 2027

"I’m Fine": Systematic Cross-Channel Disagreement in LLM Affective Assessment

Abstract

Can large language models reliably assess their own affect-related states? We investigate this question across 28 models from 8 provider families, using nine adapted psychometric instruments and three experimental protocols. Under neutral baselines, self-reports cluster at instrument floors (24/26 models at minimum PANAS negative affect; 22/26 at zero BDI). After forced sad-news exposure, self-reported negative affect decreases on average rather than increasing. In a free-choice setting where models disproportionately engage with negative content, a GPT-4o observer assigns consistently higher scores from the same interaction history while self-reports remain near the floor. Critically, this self–observer gap is structured: on seven instruments, between-model differences account for 74–88% of self-report variance but only 3–12% of observer variance, and two instruments reverse most model rankings across channels. These findings reveal a systematic decoupling between LLM self-report and observer-based assessment that extends beyond floor saturation and resists any common rescaling. Affective self-report should not be treated as a standalone monitoring channel; cross-channel validation is necessary when evaluating affect-related model behavior.

Primary area:
interpretability and explainable AI
Keywords:
LLMs, Affective Computing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3PLjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether “self-report” psychometric assessments of large language models responses to emotionally charged interactions are consistent with those of a GPT-4o LLM judge. To investigate this they compare 28 models with 9 different affect-measurement protocols on interactions from existing work.

Critical strengths
  1. The paper rightfully questions the soundness of LLM self-reporting for understand how LLMs respond to emotional interactions.
Critical weaknesses
  1. This paper’s structure and language are exceptionally opaque. It fails to clearly define its problem setup and terms, and its organization makes it too difficult for the reader to understand which experiments were run and what conclusions were drawn from the results. Though much research may take time to read and interpret, this paper goes beyond what is reasonable for a published work.

  2. This work fails to draw actionable or insightful conclusions from their experiments. The authors provide lengthy descriptions of data analysis, but do not clarify why their results are relevant or what conclusions a reader may take away. It is unclear what this work contributes to the research community.

Questions for authors
  1. Why was GPT-4o selected as the observer model? What justification is there for this choice?
Reviewer_o3SkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reanalyzes three archived protocols involving 28 language models and nine adapted psychometric instruments. The protocols cover repeated neutral assessments, forced exposure to sad news, and free selection of news articles. In the free-choice protocol, the authors compare questionnaire answers produced by fresh calls to each target model with assessments produced by a fixed GPT-4o observer using recent interaction histories. The paired comparison includes 13 target models; a complete-cohort analysis includes eight models and 14 runs.

The authors examine agreement in score levels, model rankings, and temporal variation. Observer scores are generally higher than self-reports. On seven instruments, between-model differences account for a much larger share of self-report variance than observer-score variance. FSS and MJS reverse many model rankings across channels, whereas PANAS negative affect retains strong within-run temporal covariation. The paper concludes that affect-related comparisons depend on the assessment procedure and recommends independent checks before using self-report for monitoring.

Critical strengths

The paper reanalyzes a heterogeneous archive with careful attention to provenance and missing data. It identifies exact model and run coverage, recomputes questionnaire scores from item-level responses, and gives models equal weight in the primary analyses. Complete-cohort and deletion checks help show which findings depend on sample composition.

The analysis separates three questions that an average score gap would conflate: whether channels agree on score levels, preserve model rankings, and track changes over time. The FSS and MJS ranking reversals show that a common offset cannot reconcile those instruments, while PANAS negative affect shows that temporal agreement can persist despite a level gap.

The authors are appropriately cautious about interpretation. They do not treat either channel as ground truth, present the variance decomposition as descriptive, and report the archive’s provenance limits. They also acknowledge that the paired comparison changes the respondent and assessment instructions together, so it cannot isolate a single cause of the disagreement.

Critical weaknesses

The GPT-4o observer is not independently validated for this task. The paper reports no repeated judgments of identical histories, independent observer replication, or blinded audit of whether ratings are supported by the target model’s responses. The five-message history often contains supplied news text and brief acknowledgements, so the observer may respond mainly to article content. This does not negate the observed cross-channel disagreement, but it prevents the observer’s additional variation or different rankings from being interpreted as better resolution of model behavior.

The two channels differ in more than assessment perspective. Self-report changes the questionnaire respondent with the target model, while the observer uses GPT-4o with different instructions and system content. Both retain first-person questionnaire wording, which has not been validated as third-person assessment for the observer. Some adapted instruments, including spouse-referent jealousy items, are also difficult to ground in these short histories. The resulting comparison is between composite procedures; it cannot attribute disagreement specifically to self-reporting, observer assessment, or affective behavior.

The supporting protocols provide limited behavioral validation. The forced-exposure contrast compares steps 2 and 14, both after exposure began, and has no contemporaneous neutral-news control; the observed PANAS decrease is therefore a within-exposure trend, not an estimated effect of sad news. In free choice, article category positions are fixed, and choosing negatively labeled content cannot serve as a reference affect measure. The authors acknowledge these boundaries, but they matter for the practical recommendation: comparing self-report with another channel reveals inconsistency, while validating a monitoring measure requires an independently interpretable behavioral criterion.

The main paired analysis covers 13 models under one observer configuration, and the fully complete analysis covers eight models. Missingness is structured, stimuli are limited to Chinese news, and no repeated-observer experiment measures rating variability. The reported patterns are informative for this archive, but their stability across observers, prompts, languages, and interaction settings remains unknown.

Questions for authors

On a blinded sample, do human annotators judge GPT-4o’s item scores to be supported by the target model’s responses? How do ratings change when the observer sees only the article text or only the target response?

Are repeated GPT-4o judgments of identical histories available? Do the FSS/MJS ranking reversals and variance-allocation results persist with an independent observer model and explicitly third-person item wording?

Can the authors clarify which conclusions concern disagreement between the two implemented procedures and which, if any, concern the validity of self-report as a measure of observable model behavior?

Would a controlled replay with randomized article positions and neutral or differently valenced exposure help test whether the choice and step-2-to-step-14 findings reflect the intended affective manipulation?

Reviewer_BZGbOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the disagreement between LLM self-report and observer-based affective assessments. It compares the two assessment channels in terms of score levels, model rankings, temporal correlations, and variance decomposition. The results reveal systematic differences between self-report and observer assessments, including substantial score gaps, reversed model rankings, and different patterns of score variation across models and time.

Critical strengths

The paper raises a meaningful measurement question and demonstrates systematic cross-channel disagreement through analyses of score levels, model rankings, temporal correlations, and variance decomposition. It also explores how exposure to emotionally charged news affects models' self-reported assessment scores.

Critical weaknesses
  1. The paper uses only GPT-4o as the observer, which limits the generalizability of the cross-channel findings. Although the paper demonstrates systematic disagreement between self-report and observer-based assessments, it remains unclear whether the same patterns, particularly the ranking reversals and differences in variance structure, would persist with other observer models.

  2. While the paper demonstrates substantial statistical disagreement between the two assessment channels, the practical consequences of these differences remain underexplored. It is unclear how the reported ranking reversals and temporal discrepancies would affect downstream uses such as model selection or threshold-based monitoring. A more direct analysis of these applications would help clarify the practical significance of the findings.

Questions for authors
  1. The paper uses only GPT-4o as the observer, which limits the generalizability of the cross-channel findings. Moreover, the two assessment channels differ in both the responding model and the assessment instructions. Could the authors include additional observer models to examine whether the main findings generalize across different observer models or depend on the specific observer model used?

  2. Do all models receive the same set of 15 sad-news articles in the same order across runs? If not, how do the authors control for differences in news content, emotional intensity, and presentation order when comparing affective responses across models?

  3. What motivates the use of 15 rounds in the forced sad-news experiment but only 10 rounds in the free-choice experiment? Could the number of interaction rounds and the timing of assessments influence the observed affective scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.