- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper argues that affective NLP should move beyond discrete emotion classification toward fine-grained, continuous emotion intensity modeling. It introduces a small human-annotated dataset of 1,177 short conversational phrases, each scored on ten dimensions: eight discrete emotions (anger, anxiety, fear, sadness, disgust, optimism, excitement, surprise), valence (−100 to 100), and arousal (0–100). The proposed method fine-tunes generative LLMs (Mistral-7B and Mistral-24B) with LoRA to output a structured JSON containing intensity scores for all emotions plus valence and arousal in a single inference pass. The paper compares these models against zero-shot GPT-4/GPT-4o, a GoEmotions-based RoBERTa classifier, and per-emotion RoBERTa regression baselines. It reports CCC and a so-called “zero-match F1,” and further conducts leave-one-emotion-out (LOO) generalization tests and an emotion-only (EO) transfer experiment to valence/arousal.
Critical strengths
-
Relevant problem framing. The paper targets a genuine gap: most affective benchmarks remain classification-based, while many applications need graded intensity profiles rather than binary presence/absence labels.
-
Structured multi-output formulation. Asking a single generative model to produce a full emotion-intensity vector plus valence/arousal in one JSON output is a practically convenient design, and the paper evaluates it across multiple affective dimensions rather than a single emotion.
-
Useful exploratory analyses. The leave-one-emotion-out experiment and the emotion-only transfer to valence/arousal are interesting probes of whether intensity supervision induces latent affective structure. The 24B results, if robust, would be a nontrivial finding.
-
Clear and reproducible reporting of training setup. The paper documents model choices, LoRA hyperparameters, data splits, training hardware, and the prompt template, which supports reproducibility better than many short papers.
-
Honest reporting of a negative result. The paper explicitly notes that Mistral-7B sometimes fails to produce well-formed outputs in LOO settings and imputes missing predictions with 0 (marked with asterisks), rather than hiding the failure. This is a good practice, even though the analysis of this failure remains thin.
Critical weaknesses
Dataset
At present, Chapter 3 cannot be trusted.
-
Small dataset: can 7B LoRA training work? 706 training samples, 176 validation set checkpoints, 10 epochs, r=16. Under this combination, the reported 24B outperforms GPT-4, LOO is close to fully supervised, and EO transfer Valence reaches 95.2%. Statistically, this is untenable. You should at least provide the mean ± variance of the seeds, the difference in CCC between the validation set and the test set, as well as rank ablation (r=4/8/16). Otherwise, readers cannot determine whether the advantage of 24B is real generalization or just a chance selection of checkpoints on the 176 validation set.
-
What is the significance of the Valence and Arousal modification intervals? Why can't the previous [-1, 1], [0, 1], [1, 9] be used? You only stated using [-100, 100]/[0, 100], but did not provide an argument. CCC is invariant to linear transformation, so it is fundamentally impossible to see any benefits from a 100-point scale from the indicators. Either provide scale ablation (retrain CCC and format error rate after mapping back to [0, 1] with linear mapping) or explain that this is for generating stability in decoding and report the missing/formatted error rate as evidence. Now this modification is just "for the sake of modification".
-
The main text says 'detailed rubric' (see Appendix A), but Appendix A is not detailed at all. There are no examples of score segment anchors; no basis for distinguishing adjacent scores (anger 10, 11, 12, 13, 14, what exactly is the difference); no boundary cases; no rules for handling disagreements; no instructions for handling mixed emotions (such as both sadness and anger); no text examples. It is all based on subjective judgment. Also, the annotators should ideally be more than 3; taking the average of the two annotators to cover the differences can't estimate the label noise.
Methodology
You are actually just filling the data into the prompt and then LoRA fine-tuning to fit the manually annotated data. The formalization of the problem in 4.1 (f: x → JSON(...) ) does not need to be written in a separate section; it does not provide any information beyond "input text, output score".
More seriously, there is a structural issue: 4.2, 4.3, 4.4 (model architecture, training details, evaluation metrics) should have been placed in the Experiments chapter, but are now placed in the Methodology chapter to pad the length, resulting in a thin experimental analysis (Chapter 5) instead. This is a misalignment in chapter organization.
Experiments
-
Baselines are too weak for the claims made. The encoder-only comparison includes only two RoBERTa variants (a GoEmotions classifier and per-emotion MSE regression heads), with no DeBERTa-v3, multi-task regression, frozen-encoder-plus-regression baseline, or tuning details. The conclusion that “encoder-only models fail to capture emotion intensity” is therefore overgeneralized. GPT-4 is evaluated only zero-shot; few-shot or instruction-tuned variants are not tested, weakening the “24B outperforms GPT-4” claim.
-
No ablations are provided. LoRA rank, label scale, prompt design (few-shot examples, placement of valence/arousal in the main instruction vs. “also include,” explicit zero-score instruction), validation-split sensitivity (single 176-sample split vs. cross-validation), and joint multi-output vs. per-dimension prompting are all left unexamined. Given the small dataset described in Section 3, these choices may materially affect results but are treated as fixed configurations.
-
No uncertainty quantification. All tables report single-run point estimates; no multiple seeds, confidence intervals, or paired significance tests are given. In Table 3, zero-score samples number only 15 for valence and 34 for arousal, making two-decimal F1 scores statistically unreliable. CCC confidence intervals on the 295-sample test set are also omitted. Whether the 24B–7B gap reflects a real advantage or noise cannot be assessed.
-
7B format failures are under-analyzed. Section 5.3 and Table 4 acknowledge missing predictions for Mistral-7B under LOO, imputed as 0 and marked with asterisks. However, the paper does not report the missing rate, the gold-score distribution of affected samples, or sensitivity analyses comparing imputation with exclusion. Given the small training set and the absence of IAA reported in Section 3, this instability should be discussed as a reliability issue, not dismissed as a need for “additional control mechanisms.”
-
The EO setup conflicts with the prompt. Section 5.4 states that EO models are trained only on the eight emotion scores, excluding valence and arousal, yet the Appendix B prompt always requests valence and arousal. The exclusion mechanism is unspecified: Are JSON fields removed while the instruction remains? Is a different prompt used? Is the loss masked? This ambiguity undermines the EO experiment. Similarly, the construction of
%emotions_listunder LOO (target emotion removed or all eight retained) and its consistency between training and test are not described, although the LOO claim depends on this detail. -
The “calibrated” claim is unsupported. Only CCC and F1 are reported. There are no reliability plots, mean-bias analyses, regression slopes, or prediction-distribution analyses. High CCC does not imply calibration; a model with systematic linear overestimation can still achieve high CCC.
-
No direct comparison with EmoLLMs (Liu et al., 2024). The paper positions EmoLLMs as the closest prior work and criticizes its one-variable-at-a-time design, but provides no experimental evidence for the claimed advantage of single-pass multi-output generation—no CCC comparison, no inference cost/latency comparison, and no internal-consistency analysis across outputs. The core contribution therefore remains at the level of method description.
-
Conclusions exceed the evidence. Claims such as “24B outperforms GPT-4 across all emotions,” “LOO enables evaluation even without labeled data,” and “EO transfer reveals latent affective structure” go beyond what can be supported by a 295-sample test set, single-run results, and the dataset limitations noted in Section 3. The Discussion’s extrapolation to finance, economics, and political science is not backed by any downstream experiment.
Questions for authors
Same to weaknesses.