acceptodds
Withdrawn from ICLR 2027

RLE-Reasoner: Building Trustworthy Emotion Reasoning from Complementary EEG/EOG Evidence through Counterfactual Diagnosis and Diagnosis-Guided Repair

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) can generate natural-language explanations for emotion judgments, yet fluent reasoning may describe physiological patterns absent from the recorded signals. We propose RLE-Reasoner, a build–diagnose–repair framework that connects emotion reasoning to verifiable EEG/EOG response evidence. In Stage 1, RLE-MLLM combines separate EEG/EOG encoders, Complementarity-Aware Response Fusion (CARF), and a physiological prefix to predict seven-class emotions and generate structured physiological evidence with a natural-language explanation. Seven attributes independently extracted from raw signals supervise the structured EEG/EOG evidence fields without entering the model input; their verbalization supplies the explanation target. In Stage 2, RLE-Eval uses controlled replacement of real EEG/EOG recordings to test whether the generated evidence reflects the current signals. In Stage 3, RLE-PostTrain turns these diagnoses into GRPO rewards that encourage correct physiological descriptions and responses to changed evidence. On a 5,040-trial, twelve-subject dataset with 1,260 held-out test trials, RLE-MLLM 14B achieves 66.52% seven-class cross-subject emotion accuracy and 62.08% matched physiological-state accuracy. Post-training improves overall physiological-grounding accuracy from 63.84% to 74.15% and state Macro-F1 from 46.21% to 58.10%, with matched emotion accuracy changing from 66.52% to 66.66%. Blinded human assessment of free-form text against supplied physiological reference states finds that consistency increases from 58.33% to 69.06%. These results show that physiological evidence can guide the diagnosis and repair of generated emotion reasoning.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Large Language Models, EEG, EOG, Emotion Recognition, Explainable AI, Reinforcement Learning, GRPO, Trustworthy AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_E3NrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents RLE-Reasoner, a three-stage framework for emotion recognition from EEG and EOG with a large language model.

In the first stage, separate EEG and EOG encoders are combined by a gated fusion module (CARF), and the result is passed to a LoRA-tuned Qwen model (4B, 8B, 14B) as a short sequence of input tokens. The model outputs an emotion label, five EEG states, two EOG states, and a short text that the authors call an explanation. The seven states are produced by a fixed rule-based extractor applied to the raw signals and discretized into three levels using statistics from the training subjects. Qwen-Max turns the label and the states into the target texts.

In the second stage, the EEG and/or EOG of a test trial is replaced with a recording from another subject. The generated states are then compared with those computed on the modified input.

In the third stage, the language model is further trained with GRPO, using a reward based on the accuracy of the generated states. Responses whose emotion differs from the first-stage prediction are penalized.

On a private dataset of twelve subjects and seven classes, the 14B model reaches 66.52% emotion accuracy. Post-training raises state accuracy from 63.84% to 74.15% without changing emotion accuracy, and three blinded raters judge the texts of the post-trained model more consistent with the reference states. A further experiment on DEAP evaluates classification only.

Critical strengths

A concrete test for whether the generated evidence reflects the actual signals. A common failure of emotion-reasoning models is producing fluent, physiological-sounding text ("elevated frontal theta, reduced alpha...") that follows from the predicted label rather than from the recorded signals. RLE-Eval addresses this directly: it replaces the EEG, the EOG, or both with a recording from a different subject viewing a different stimulus, chosen so that at least one reference attribute changes, and then checks whether the generated states match the attributes of the signals actually presented. Because the reference is recomputed from the input rather than assumed from the label, agreement becomes meaningful evidence that the model is reading the signal instead of collapsing .

Critical weaknesses

-The main results rely on a single split into eight training subjects, one validation subject, and three test subjects. LOSO evaluation is the standard in cross-subject EEG emotion recognition, and several of the cited baselines use it. The six additional runs in Table 3 cover only the first stage, and the paper does not describe how training and validation subjects were assigned in them.

-In post-training, Qwen-Max assigns a score of 0, 0.5 or 1 for how well the text organizes the evidence, avoids causal overstatement, and handles conflicting evidence. The prompt, the criteria, and how causal overstatement is detected are not given, and the judge is not compared with human ratings. Because the targets are written by Qwen-Max, the judge is Qwen-Max, and the trained model is also a Qwen model, the score may reward similarity of style rather than correctness, since language models tend to favor text resembling their own (Panickssery et al., 2024).

-The thresholds at +-0.75 robust standard deviations imply that, for a roughly Gaussian attribute, about 55% of trials fall in the "typical" state. A model that always predicts "typical" could therefore match or exceed the 4B model (44.83%) and the 14B accuracy on posterior alpha (46.52%). No such baseline is reported.

-The authors should train a simple classifier on the frozen fused features to predict the seven states. Without it, one cannot tell whether the language model is limited by the information it receives or by its own decoding. The encoders are trained only to classify emotions, they receive inputs normalized per trial, and they are frozen afterwards. Meanwhile, amplitude based attributes, such as the EOG and the evoked responses, are computed before normalization. Part of the information to be predicted may simply be absent from the model's input.

-Replacement recordings always come from another person, and several attributes, such as EOG amplitude or posterior alpha, vary strongly between individuals. A model could "follow" the replacement by recognizing the donor's typical level rather than by reading the single trial. A baseline that predicts each state from the subject's average would clarify this. It also remains unexplained why accuracy on modified inputs is higher than on original ones for all model sizes, and how often replacing only the EEG also changes the generated EOG states.

-The paper calls the generated text an explanation of the emotion judgment. However, the text verbalizes the extractor's states and the label, following targets written by Qwen-Max, and nothing connects it to how the model reaches its decision. An explanation, in the usual sense, should reflect the reasons for the prediction. What is evaluated here is closer to a description of the input. Two design choices reinforce this gap. The emotion is generated before the states, so the states are produced already knowing the label. Second, the reward ties the emotion to the first-stage prediction. This makes the preservation of emotion accuracy largely a consequence of the design rather than an empirical result, and it prevents post-training from making the decision more consistent with the evidence. No experiment tests whether the reported evidence affects the prediction, or whether the seven single-trial attributes are related to the emotion categories at all.

-Finally, the reward measures agreement with states the authors already know. The first stage never saw modified inputs, so the gain could come from simply training on this new kind of input. A comparison with standard supervised training on the same 13440 modified instances is needed. Without it, the specific contribution of the third stage cannot be assessed.


Minor

Two citations are missing: one in the Introduction, after "(Ma et al., 2026; Jiang et al., 2024; ?)", and one in Related Work, after "emotion recognition (?)".

The language model is described only as "Qwen 4B/8B/14B", without version or citation. The available sizes and Appendix H suggest Qwen3, but this is never stated for the main experiments. The paper also does not say which layers LoRA adapts or where the input tokens are placed in the prompt.

Qwen-Max, a proprietary model available only through an online service, is used both to write the training targets and to score texts during post-training. Neither its version, the date of use, nor the prompts are reported.

The main text and the appendix disagree on the number of post-training instances (13440 versus 400 in Table 4) and on how the reward is computed when no attribute is left unchanged.

Questions for authors

1.Can the second and third stages be evaluated with LOSO? For the six runs in Table 3, how were training and validation subjects assigned?

  1. What prompt and criteria does Qwen-Max use for the soft reward, and how is causal overstatement detected? Was the judge compared with human ratings, or with a judge from a different model family?

  2. What accuracy and Macro-F1 does a model that always predicts "typical" achieve, overall and for each attribute? What does a simple classifier trained on the frozen fused features achieve on the seven states?

  3. What accuracy is obtained by predicting each state from the subject's average? Why is accuracy higher on modified inputs than on original ones, and how often do the generated EOG states change when only the EEG is replaced?

  4. Does the predicted emotion change when the generated evidence changes? Are the seven single-trial attributes predictive of the emotion categories?

The autors could generate the evidence before the emotion, and letting the emotion vary during post-training. This would allow a direct test of whether the decision depends on the evidence.

Reviewer_dMTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue that natural language explanations generated by multimodal large language models (MLLMs) for emotion recognition may lack support from actual physiological signals, and proposes RLE-Reasoner. The framework consists of three stages: (1) RLE-MLLM jointly generates emotion classifications, structured physiological evidence, and natural language explanations using separate EEG/EOG encoders, a Complementarity-Aware Response Fusion (CARF) module, and physiological feature prefixes; (2) RLE-Eval constructs modality-specific intervention inputs by replacing real EEG/EOG recordings to examine whether the generated physiological state descriptions are consistent with the current input signals; and (3) RLE-PostTrain converts the resulting diagnostic signals into GRPO reward signals for post-training the model, with the aim of improving the physiological grounding of generated content. Experiments are conducted on a private EEG/EOG dataset comprising 5,040 trials from 12 participants. In cross-subject testing, the 14B model achieves a seven-class classification accuracy of 66.52% and a physiological-state accuracy of 62.08% on matched inputs.

Critical strengths

The main strength of this paper lies in extending the focus of emotion recognition research beyond traditional classification performance to the reliability of explanations generated by multimodal large language models. Unlike studies that focus solely on the accuracy of emotion label prediction, this work examines whether the generated physiological evidence is supported by the input EEG/EOG signals. By integrating structured physiological state supervision, controlled signal interventions, and diagnosis-guided post-training, the authors establish a complete framework encompassing generation, diagnosis, and repair. This approach offers a valuable research direction for evaluating and improving the physiological grounding of generated explanations in emotion reasoning.

Critical weaknesses

The validation of the reliability of generated explanations in this paper is limited. The paper primarily diagnoses generated content by measuring the consistency between structured physiological states and the reference states corresponding to the current input. However, such consistency does not sufficiently establish the overall reliability of natural language explanations. Specifically, the paper acknowledges that its method cannot verify every statement in free-form explanations, and the additional quality reward for natural language explanations is activated only when all seven structured physiological states are correct. Although the authors report human evaluation results in Table 6, 28.57% of explanations after post-training were still judged to contain unsupported physiological content. Therefore, the current evidence supports improvements in structured physiological evidence consistency, but remains insufficient to establish the overall reliability of the generated explanations.

Questions for authors
  1. The paper uses seven structured physiological states derived from a fixed feature extractor as supervision and evaluation targets. However, agreement with these predefined reference states does not independently establish the physiological validity of the generated explanations. The supplementary material acknowledges that these states cannot establish actual emotional experiences or the causal features used internally by CARF. Moreover, the external validation on DEAP evaluates only classification and physiological-prefix conditioning, without validating the seven-attribute intervention protocol or post-training repair. Could the authors provide further evidence that the proposed diagnosis-and-repair mechanism generalizes beyond the private dataset
  2. The paper reports that, after post-training, the consistency between natural language explanations and physiological references increases from 58.33% to 69.06%, while the proportion of explanations containing unsupported physiological content decreases from 38.69% to 28.57%. However, the explanation quality reward is activated only when all seven structured physiological states are correct, while the existing ablation experiments primarily investigate the effects of counterfactual reward terms on structured physiological state performance. The contribution of the natural language explanation reward has not been independently evaluated. The authors are encouraged to conduct an ablation study specifically targeting the explanation reward, together with corresponding human evaluations of free-form explanations, to clarify its independent contribution to explanation quality and distinguish improvements in structured physiological state prediction from improvements in natural language explanation quality.
  3. The Related Work section lacks a sufficiently explicit methodological comparison with the most closely related approaches, making it difficult to assess the novelty of the proposed framework. The authors should conduct experimental comparisons between the proposed method and existing methods to demonstrate its specific contributions. Additionally, two unresolved citation placeholders, "(?)", should be corrected to ensure the relevant claims are properly supported.
Reviewer_cZMSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RLE-Reasoner, a three-stage framework for improving the physiological grounding of emotion reasoning generated by multimodal large language models. The framework consists of RLE-MLLM for EEG/EOG-based emotion recognition and evidence generation, RLE-Eval for diagnosing unsupported physiological claims through signal replacement, and RLE-PostTrain for improving evidence consistency using physiology-constrained GRPO.

The proposed method is evaluated on a private EEG/EOG dataset containing 5,040 trials from twelve subjects. The results demonstrate improvements in structured physiological grounding, intervention responsiveness, and human-evaluated explanation consistency, while largely preserving emotion classification performance.

The problem is interesting and relevant to trustworthy multimodal reasoning. However, the evaluation raises several important questions regarding the validity of the physiological references, the reliability of cross-subject generalization, and whether the reported gains establish improvements in genuine reasoning rather than agreement with predefined physiological rules.

Critical strengths

1 The paper addresses an important limitation of multimodal LLMs: a plausible natural-language explanation does not necessarily correspond to the evidence contained in the model's actual input. Evaluating explanation grounding independently of classification accuracy is a valuable research direction. 2 The build–diagnose–repair framework provides a clear methodological organization. In particular, separating evidence generation, intervention-based evaluation, and reward-guided post-training makes the proposed approach relatively easy to understand and analyze. 3 The paper evaluates several model scales, compares classification against relevant baselines, and includes an ablation of counterfactual reward terms. The improvement in physiological grounding with relatively stable classification accuracy is promising. 4 The authors appropriately distinguish deterministic physiological-state accuracy from human evaluation of natural-language explanations. This avoids treating every generated sentence as verified solely because the structured attributes are correct.

Critical weaknesses

1 The seven physiological attributes are generated using predefined extraction rules and statistical thresholds. These same attributes are subsequently used for supervised training, intervention evaluation, and GRPO reward computation. Although this provides a consistent evaluation protocol, agreement with these automatically generated targets does not necessarily establish physiologically faithful reasoning. The authors should further validate the extracted attributes against independently assessed physiological evidence and investigate the sensitivity of the results to the chosen extraction rules and thresholds. 2 The paper states that physiological targets are extracted before encoder normalization, while the EEG/EOG recordings are normalized channel-wise per trial. However, several target attributes, including event-related potential amplitudes and EOG RMS activity, depend on signal magnitude. If normalization removes absolute amplitude information, some target attributes may not be fully recoverable from the model input. The authors should clarify the exact normalization procedure and explain how the relevant physiological information is preserved. 3 The evaluation relies on a private dataset containing only twelve subjects. The primary experiment uses eight training subjects, one validation subject, and three testing subjects. Although additional subject-disjoint evaluations are provided for Stage 1, the robustness of the complete diagnosis-guided post-training framework across different subject splits remains unclear. The authors should evaluate the full framework across additional subject-independent splits, report statistical uncertainty, and ideally validate it on an independent dataset. 4 The post-training experiments compare the proposed method against supervised fine-tuning and GRPO without counterfactual reward terms. However, the evaluation metrics are closely aligned with the physiological rules used to construct the rewards. It remains unclear whether the improvements arise from the proposed diagnosis-guided mechanism or from directly optimizing agreement with predefined attribute labels. Stronger comparisons with intervention-augmented supervised fine-tuning and simpler physiology-constrained generation methods would help establish the necessity of the proposed framework. 5 The emotion categories correspond to presented facial-expression stimuli rather than independently measured emotional experiences. Moreover, the human evaluation covers only 25 base trials under four intervention conditions, and reviewers assess generated explanations using the same seven physiological reference states. These evaluations support consistency with predefined signal attributes but do not fully establish the faithfulness of emotion reasoning. The authors should better distinguish stimulus classification from emotion-state inference and provide stronger independent evaluation of free-form explanations.

Questions for authors

1 How does the per-trial signal normalization preserve the amplitude information required to predict N170, LPP, and EOG RMS states? 2 How sensitive are physiological-grounding accuracy and State Macro-F1 to alternative extraction thresholds and independent expert annotations? 3 Do the improvements from RLE-PostTrain remain consistent across different held-out subject splits, rather than only the primary split?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.