acceptodds
Under review as a conference paper at ICLR 2027

Beyond Sentiment Classification: A Generative Framework for Emotion Intensity Evaluation in Text

Abstract

We introduce a novel approach to emotion modeling that shifts the focus from identification to evaluation, addressing the limitations of discrete classification in applied domains such as finance. By constructing a dataset of emotional intensity scores and fine-tuning open-weight generative language models to output continuous values from 0-100, we demonstrate a more expressive, generalizable framework for sentiment and emotion analysis. Our findings not only outperform classification baselines but also reveal surprising generalization capabilities and transfer effects to related constructs such as sentiment and arousal. This work contributes to the interdisciplinary recontextualization of NLP by introducing emotion intensity evaluation as an alternative to classification, arguing that this shift better aligns with the needs of domains—such as finance—where the degree of emotional content is central to interpretation and decision-making.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
emotion intensity, affective computing, generative language models, LoRA fine-tuning, sentiment analysis, valence and arousal, dataset construction, natural language processing

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_drWCOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that affective NLP should move beyond discrete emotion classification toward fine-grained, continuous emotion intensity modeling. It introduces a small human-annotated dataset of 1,177 short conversational phrases, each scored on ten dimensions: eight discrete emotions (anger, anxiety, fear, sadness, disgust, optimism, excitement, surprise), valence (−100 to 100), and arousal (0–100). The proposed method fine-tunes generative LLMs (Mistral-7B and Mistral-24B) with LoRA to output a structured JSON containing intensity scores for all emotions plus valence and arousal in a single inference pass. The paper compares these models against zero-shot GPT-4/GPT-4o, a GoEmotions-based RoBERTa classifier, and per-emotion RoBERTa regression baselines. It reports CCC and a so-called “zero-match F1,” and further conducts leave-one-emotion-out (LOO) generalization tests and an emotion-only (EO) transfer experiment to valence/arousal.

Critical strengths
  1. Relevant problem framing. The paper targets a genuine gap: most affective benchmarks remain classification-based, while many applications need graded intensity profiles rather than binary presence/absence labels.

  2. Structured multi-output formulation. Asking a single generative model to produce a full emotion-intensity vector plus valence/arousal in one JSON output is a practically convenient design, and the paper evaluates it across multiple affective dimensions rather than a single emotion.

  3. Useful exploratory analyses. The leave-one-emotion-out experiment and the emotion-only transfer to valence/arousal are interesting probes of whether intensity supervision induces latent affective structure. The 24B results, if robust, would be a nontrivial finding.

  4. Clear and reproducible reporting of training setup. The paper documents model choices, LoRA hyperparameters, data splits, training hardware, and the prompt template, which supports reproducibility better than many short papers.

  5. Honest reporting of a negative result. The paper explicitly notes that Mistral-7B sometimes fails to produce well-formed outputs in LOO settings and imputes missing predictions with 0 (marked with asterisks), rather than hiding the failure. This is a good practice, even though the analysis of this failure remains thin.

Critical weaknesses

Dataset

At present, Chapter 3 cannot be trusted.

  1. Small dataset: can 7B LoRA training work? 706 training samples, 176 validation set checkpoints, 10 epochs, r=16. Under this combination, the reported 24B outperforms GPT-4, LOO is close to fully supervised, and EO transfer Valence reaches 95.2%. Statistically, this is untenable. You should at least provide the mean ± variance of the seeds, the difference in CCC between the validation set and the test set, as well as rank ablation (r=4/8/16). Otherwise, readers cannot determine whether the advantage of 24B is real generalization or just a chance selection of checkpoints on the 176 validation set.

  2. What is the significance of the Valence and Arousal modification intervals? Why can't the previous [-1, 1], [0, 1], [1, 9] be used? You only stated using [-100, 100]/[0, 100], but did not provide an argument. CCC is invariant to linear transformation, so it is fundamentally impossible to see any benefits from a 100-point scale from the indicators. Either provide scale ablation (retrain CCC and format error rate after mapping back to [0, 1] with linear mapping) or explain that this is for generating stability in decoding and report the missing/formatted error rate as evidence. Now this modification is just "for the sake of modification".

  3. The main text says 'detailed rubric' (see Appendix A), but Appendix A is not detailed at all. There are no examples of score segment anchors; no basis for distinguishing adjacent scores (anger 10, 11, 12, 13, 14, what exactly is the difference); no boundary cases; no rules for handling disagreements; no instructions for handling mixed emotions (such as both sadness and anger); no text examples. It is all based on subjective judgment. Also, the annotators should ideally be more than 3; taking the average of the two annotators to cover the differences can't estimate the label noise.

Methodology

You are actually just filling the data into the prompt and then LoRA fine-tuning to fit the manually annotated data. The formalization of the problem in 4.1 (f: x → JSON(...) ) does not need to be written in a separate section; it does not provide any information beyond "input text, output score".

More seriously, there is a structural issue: 4.2, 4.3, 4.4 (model architecture, training details, evaluation metrics) should have been placed in the Experiments chapter, but are now placed in the Methodology chapter to pad the length, resulting in a thin experimental analysis (Chapter 5) instead. This is a misalignment in chapter organization.

Experiments

  1. Baselines are too weak for the claims made. The encoder-only comparison includes only two RoBERTa variants (a GoEmotions classifier and per-emotion MSE regression heads), with no DeBERTa-v3, multi-task regression, frozen-encoder-plus-regression baseline, or tuning details. The conclusion that “encoder-only models fail to capture emotion intensity” is therefore overgeneralized. GPT-4 is evaluated only zero-shot; few-shot or instruction-tuned variants are not tested, weakening the “24B outperforms GPT-4” claim.

  2. No ablations are provided. LoRA rank, label scale, prompt design (few-shot examples, placement of valence/arousal in the main instruction vs. “also include,” explicit zero-score instruction), validation-split sensitivity (single 176-sample split vs. cross-validation), and joint multi-output vs. per-dimension prompting are all left unexamined. Given the small dataset described in Section 3, these choices may materially affect results but are treated as fixed configurations.

  3. No uncertainty quantification. All tables report single-run point estimates; no multiple seeds, confidence intervals, or paired significance tests are given. In Table 3, zero-score samples number only 15 for valence and 34 for arousal, making two-decimal F1 scores statistically unreliable. CCC confidence intervals on the 295-sample test set are also omitted. Whether the 24B–7B gap reflects a real advantage or noise cannot be assessed.

  4. 7B format failures are under-analyzed. Section 5.3 and Table 4 acknowledge missing predictions for Mistral-7B under LOO, imputed as 0 and marked with asterisks. However, the paper does not report the missing rate, the gold-score distribution of affected samples, or sensitivity analyses comparing imputation with exclusion. Given the small training set and the absence of IAA reported in Section 3, this instability should be discussed as a reliability issue, not dismissed as a need for “additional control mechanisms.”

  5. The EO setup conflicts with the prompt. Section 5.4 states that EO models are trained only on the eight emotion scores, excluding valence and arousal, yet the Appendix B prompt always requests valence and arousal. The exclusion mechanism is unspecified: Are JSON fields removed while the instruction remains? Is a different prompt used? Is the loss masked? This ambiguity undermines the EO experiment. Similarly, the construction of %emotions_list under LOO (target emotion removed or all eight retained) and its consistency between training and test are not described, although the LOO claim depends on this detail.

  6. The “calibrated” claim is unsupported. Only CCC and F1 are reported. There are no reliability plots, mean-bias analyses, regression slopes, or prediction-distribution analyses. High CCC does not imply calibration; a model with systematic linear overestimation can still achieve high CCC.

  7. No direct comparison with EmoLLMs (Liu et al., 2024). The paper positions EmoLLMs as the closest prior work and criticizes its one-variable-at-a-time design, but provides no experimental evidence for the claimed advantage of single-pass multi-output generation—no CCC comparison, no inference cost/latency comparison, and no internal-consistency analysis across outputs. The core contribution therefore remains at the level of method description.

  8. Conclusions exceed the evidence. Claims such as “24B outperforms GPT-4 across all emotions,” “LOO enables evaluation even without labeled data,” and “EO transfer reveals latent affective structure” go beyond what can be supported by a 295-sample test set, single-run results, and the dataset limitations noted in Section 3. The Discussion’s extrapolation to finance, economics, and political science is not backed by any downstream experiment.

Questions for authors

Same to weaknesses.

Reviewer_BBRjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors generate a synthetic dataset containing ~1200 "short conversational phrases" containing 8 emotions with ChatGPT, then have two annotators score them both in terms of valence and arousal (using the average score). They then evaluate (metric: Concordance Correlation Coefficient) zero-shot performance for four LLMs (GPT-4, GPT-4o, Mistral-7B-Instruct, Mistral-Small-24B), plus two encoder baselines. Further they fine-tune the two Mistral models, leaving one emotion out and evaluate the regression task on that emotion. They show that fine-tuning improves CCC score and their own proposed zero-match F1-score.

Critical strengths
  • The authors motivate a relevant problem in that there is a lack of fine-grained, high-quality valence and arousal data.
  • The training setup and chosen metrics seem methodologically sound
Critical weaknesses
  • A large part of the contribution is the collected dataset- it is motivated through the identified gap and subsequently used in all evaluations. However, but the collection procedure does not meet the bar. The number of annotators is low (2), average scores are used with no information about agreement, and only a minimal rubric is presented. The lack of a more detailed rubric would be defendable if the authors had a larger pool of annotators, since then we would get a reasonable sample of perceived valence and arousal, but in it's current form, it cannot form a representative sample.
  • It is not clear if improvements on the fine-tuning, held-out-emotion task mainly stem from basic calibration to the expected scale, and/or, for example, better alignment with the required output format. In the current form, the paper cannot show actual transfer capabilities to other emotions. It thus disagree with the certainty of the conclusion that the authors draw in that respect ("The Mistral-24B model demonstrates strong transfer from emotion scoring to affective dimensions: it achieves 95.2% on Valence and 67.5% on Arousal without ever seeing supervision for those targets, nearly matching the fully supervised model on Valence and recovering most of the fine-tuning gain on Arousal")
  • In addition, these models, as the authors report, already have valence/arousal detection capabilities in the zero-shot task, which further makes it hard to disentangle the confound of pre-existing capabilities before fine-tuning.
  • No confidence intervals or significance is reported on the results. Some differences in score the authors draw conclusions from might as well be insignificant. For example, the authors state "Mistral-24B-FT achieves the highest CCC in 8 out of 10 categories" - the numbers seem mostly inconclusive to me without multiple seeds (e.g., Anxiety: 86.4 vs 87.1, Optimism: 90.8 vs 91.2)
  • The choice of the 8 specific emotions is not motivated, and there seems to be larger number of negative emotions than positive ones. Also, "Anxiety" and "Fear" seem to be very similar.
  • The scored utterances in the dataset were completely generated by one (relatively weak) model, no human examples, no stronger models
Questions for authors
  • Why did the authors only resort to two annotators?
  • Why are metrics for annotator disagreements not reported?
  • Will the dataset, with per-rater scores, be released? Why has it not been released as a supplementary, given that it carries a large part of the paper's story?
  • The phrases were generated by ChatGPT and rated from a "imagine yourself saying this" perspective. How do the authors imagine these results translating to naturally occurring text (that is often not in this very specific form), and transfer to domains like finance, like the authors motivate initially?
Reviewer_RDqiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes reframing text emotion analysis from discrete multi-label classification to continuous multi-dimensional intensity prediction. The authors construct a dataset of 1,177 short conversational phrases generated by ChatGPT and annotated by two Upwork raters on a 0-to-100 scale across eight discrete emotions (anger, anxiety, fear, sadness, disgust, optimism, excitement, surprise), valence, and arousal. They fine-tune two open-weight instruction-tuned models (Mistral-7B and Mistral-24B) using 4-bit QLoRA and standard causal language modeling cross-entropy loss to generate JSON strings containing all ten numerical scores simultaneously. Evaluated on 295 test sentences using the Concordance Correlation Coefficient (CCC) and a Zero-Match F1 metric, the fine-tuned Mistral models outperform zero-shot LLMs (GPT-4, GPT-4o) and fine-tuned RoBERTa regression baselines while retaining performance in leave-one-emotion-out (LOO) and emotion-only-to-valence/arousal (EO) transfer settings.

Critical strengths
  1. A well-designed joint multidimensional formulation: Predicting a full vector of discrete emotion intensities alongside dimensional affect (valence and arousal) in a single pass is practically convenient compared to maintaining ten separate single-target regression models.

  2. Appropriate choice of primary evaluation metric: Using CCC instead of scale-invariant Pearson Correlation properly penalizes location and scale shifts in predicted intensity values.

Critical weaknesses
  1. A Fundamental Experimental Flaw: The RoBERTa regression baseline (RoBERTa-FT) is clearly broken/misconfigured, invalidating a primary claim of the paper. Their evidence in Table 1 is RoBERTa-FT, which achieves a CCC of 0.8% on Valence, 0.0% on Arousal, 0.1% on Anxiety, and 1.7% on Surprise. This is a clear training failure.

  2. Standard next-token cross-entropy loss is clearly not well-suited for continuous regression and lacks uncertainty calibration. Methodologically, the paper simply applies off-the-shelf 4-bit QLoRA with standard token-level cross-entropy loss to emit JSON strings. Under token-level cross-entropy, predicting "79" when the ground truth is "80" incurs the same loss penalty as predicting "10", ignoring the metric structure.

  3. Misclaimed motivation: The Abstract, Introduction, and Discussion repeatedly motivate the work via finance and market behavior, however, the paper never evaluates a single financial document, earnings call, or existing real-world benchmark

Questions for authors

What is the raw agreement (Pearson rr, CCC, and MAE) between your two Upwork annotators across each of the 10 dimensions before averaging?

Why did RoBERTa-FT achieve 0.8% CCC on Valence, 0.0% on Arousal, and 0.1% on Anxiety? Did you normalize the target scores to [0,1][0, 1] or [−1,1][-1, 1] before computing MSE loss??

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.