acceptodds
Under review as a conference paper at ICLR 2027

A-MBER: Affective Memory Benchmark for Multimodal Emotion Recognition

Abstract

Long-horizon interactive systems need to remember not only facts, but also the affective and relational history that gives a user's current utterance its meaning. Existing emotion-recognition benchmarks mainly evaluate local utterance labels or short-context affective cues, while long-term memory benchmarks mainly emphasize factual recall, temporal reasoning, or knowledge update. We introduce A-MBER, an affective memory benchmark for evaluating whether language models can interpret a current anchor turn using historically relevant evidence from prior sessions. A-MBER contains 2,047 benchmark items derived from 34 multi-session interaction trajectories and covers three task families: judgment, evidence retrieval, and explanation. Each item is annotated with content type, memory dependency level, reasoning structure, and robustness condition. We evaluate ten large language models under local-context, full-history, and retrieved-history settings. Results show that historical access substantially improves performance, with the largest gains on long-term implicit affect, high-memory items, multi-hop reasoning, and trajectory-based reasoning. These findings suggest that A-MBER captures a distinct affective-memory capability beyond local emotion recognition and generic long-context question answering.

Primary area:
datasets and benchmarks
Keywords:
affective memory, emotion recognition in conversation, long-horizon interaction, long-term memory benchmark, multi-session dialogue, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_q2wYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces A-MBER, a framework that evaluates a model's ability to leverage cross-session history to explain current emotional or relational states through judgment, evidence retrieval, and explanation tasks. The dataset comprises 34 synthetic interaction trajectories and 2,047 items; experiments compare the performance of ten models under conditions involving local context, full history, and retrieved history.

Critical strengths

Distinguishing between sentiment judgments, the identification of historical evidence, and explanations enables an analysis that is richer and more accurate than one based on simple sentiment labels.

The authors conducted trajectory-level manual inspections, spot checks, and calibration of explanation scoring to provide initial quality assurance for the synthetic benchmark.

Critical weaknesses

Section 4.3 stipulates that the overall score is the average of the scores across all items. Table 6 further indicates that "content type" and "memory level" each constitute a complete partition of the total 2,047 items. Therefore, calculating the weighted average score for the groups within either complete partition—based on the number of items—should yield the overall score found in the main table. However, for the "Retrieved" condition, Claude-Opus yields a result of approximately 0.590 when weighted by the number of items and scores across memory levels (Tables 6 and 9), and approximately 0.601 when weighted by content type; yet, Table 2 reports a score of 0.613. These two complete partitions differ not only from the main table but also from each other.

There is already direct precedent for memory evaluation in long-term emotional support. ES-MemEval (WWW 2026) has investigated implicit and scattered information as well as evolving user states within this context, covering aspects such as information extraction, temporal reasoning, conflict detection, refusal to answer, and user modeling, while evaluating memory utilization through question answering, summarization, and dialogue generation. Its EvoEmo dataset combines existing emotional support conversations with synthetic cross-session trajectories. While the overlap between this work and A-MBER is evident, the current manuscript fails to cite or discuss it. Consequently, characterizing the intersection of long-term memory and emotional understanding as lacking evaluation is insufficient to substantiate the paper's claims of novelty. A-MBER must clearly demonstrate that its approaches to current utterance interpretation, relationship dynamics, and evidence chain design introduce challenges not addressed by ES-MemEval, rather than merely rebranding the task.

The use of historical implicit states to explain subsequent behavior has also been evaluated. LoCoMo-Plus assesses whether models can apply implicit user states, goals, and value constraints by leveraging the semantic disconnect between historical cues and subsequent requests. Its appendix explicitly includes examples where "prior emotional states influence subsequent behavior," and it employs a constraint-consistency evaluation that allows for multiple valid responses; the experiments cover standard models, RAG, and specialized memory systems. Although A-MBER cites this work, it fails to demonstrate—through task-aligned experiments—what additional insights its emotion-related items capture. Performance gains derived from accessing history do not, in themselves, prove the existence of a novel emotional memory capability.

Clearer methods now exist to distinguish between retrieval failures and evidence integration failures. Beyond reporting retrieval performance, LongMemEval employs an "oracle" condition, providing only the correct evidence—to examine the reading and reasoning errors a model makes even when the necessary evidence is already available. While A-MBER utilizes the top-10 results from BGE-M3, it lacks a corresponding control using correct evidence and fails to analyze upstream retrieval coverage. Residual errors under retrieval conditions could stem from missed retrievals, missing segments, or interpretation failures; current experiments do not adequately differentiate among these causes. Since existing work has already demonstrated this diagnostic approach, the present paper should either adopt it or explain why it is inapplicable.

The “multimodal” claim, however, shows a significant discrepancy with the actual input: tone, pauses, and expressive style are described via text, yet the paper provides no experiments using the corresponding raw audio or video inputs. Comprehending a textual description of tone is a different task from recognizing emotion based on acoustic or visual signals. While manual inspection addresses concerns regarding synthetic data quality, the kappa value of 0.82 measures consistency in decisions to retain, modify, or rewrite content; it does not directly demonstrate consistency in the ground-truth emotional interpretations. Furthermore, the 2,047 items originate from only 34 trajectories, and item-level bootstrapping fails to adequately account for the statistical dependencies arising from these shared trajectories.

Questions for authors

see weakness. If the authors can address the issues raised in the "Weakness" section, I will raise my rating.

Reviewer_3JPnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an affective memory benchmark (A-MBER) for evaluating whether LLM can interpret a current anchor turn using historically relevant evidence. A-MBER contains 2047 items from 34 multi-session interaction trajectories and covers three task families: judgment, evidence retrieval, and explanation. Results show that historical access improves performance.

Critical strengths
  1. The topic is clear and practical.
  2. The human quality verification improve the credibility of benchmark.
Critical weaknesses
  1. The title is "multimodal emotion recognition", but the benchmark only contain text, and not real voice record and video.
  2. Only 34 multi-session interaction trajectories, the diversity is limited.
  3. The existing controls are insufficient to demonstrate that an independent "emotional memory capacity" has been measured. The construction process itself was intentionally designed to select items that rely on historical context.
Questions for authors

See the weakness.

Reviewer_grEROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A-MBER evaluates historically grounded affective interpretation in multi-session interaction. It contains 2,047 items from 34 generated trajectories, with judgment, evidence-retrieval and explanation tasks. The documented construction pairs a student user with a teacher- or counselor-like AI and retains textual descriptions of delivery cues. Ten language models are evaluated under local, full-history and BGE-M3 retrieved-history conditions. The authors report larger historical-access gains on implicit-affect and high-memory subsets, supported by a 410-item quality audit and a 180-output explanation-judge validation. The contribution is a benchmark and evaluation protocol; the principal question is how convincingly it measures warranted affective interpretation beyond ordinary history-dependent question answering.

Critical strengths
  1. The target has practical value, particularly for educational support: previous disappointment, failed reassurance or changes in trust can alter the meaning of a present utterance. Separating judgment, evidence localization and explanation is consequential because a plausible emotion label alone does not establish that a system used the relevant personal history.

  2. The within-item context comparisons keep the question, anchor and scoring target fixed, and the retrieved condition uses pre-anchor embedding retrieval rather than oracle evidence. The factual/local controls, reasoning metadata, human item audit and judge-calibration study provide a useful foundation for diagnostic evaluation. These are meaningful safeguards, even though the broader validity questions below remain open.

Critical weaknesses
  1. The evidence for a distinct, broadly applicable affective-memory capability is incomplete. Section 3.2 and Figure 2 describe student–teacher/counselor-like roles, while the introduction motivates companion and general personalized assistants. A role/topic inventory is needed to establish the actual coverage. Diversity among 34 trajectories within one relationship type would not establish transfer across relationships: authority, disclosure, and conversational expectations can change which interpretation is warranted. This preserves the benchmark's educational value while limiting demonstrated generalization. Likewise, the larger gains on implicit-affect and high-memory items establish historical dependence, but do not isolate affect-specific difficulty from evidence distance, context length or reasoning complexity.

  2. The gold answers may not capture all reasonable interpretations. In Table 10, the student’s APA-formatting question is interpreted as avoiding a personal topic. However, the student might simply want help with formatting. How do the authors distinguish these possibilities? Please clarify whether independent readers reached the same interpretation before seeing the gold answer, and how other reasonable or appropriately cautious answers are scored.

Questions for authors
  1. Do all 34 trajectories involve students and teacher- or counselor-like AI? Please list the roles and topics covered and explain how this coverage supports claims about affective memory.

  2. How much error comes from missing relevant history, and how much comes from misunderstanding it? Please report how much annotated evidence BGE-M3 retrieves in its top 10 results and model performance when that evidence is available. Compare retrieval with recent history under the same input budget, and test models with the annotated evidence provided directly. Also compare emotion and factual questions with similar memory demands, evidence amounts and evidence distances.

  3. Did independent annotators reach similar interpretations before seeing the reference answers or planned emotional storyline? How are reasonable alternative interpretations and cautious answers scored? Does performance change when questions omit clues about the intended interpretation? Please specify the model, version and settings used to score explanations.

  4. The overall and subgroup scores do not match. Weighting Claude-Opus’s local-context scores in Table 7 by the item counts in Table 6 gives 0.3271, while Table 2 reports 0.371. Please reconcile these values. The three task-family counts also total 1,882 rather than 2,047; which task families contain the remaining 165 items?

Reviewer_fP1zOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a dataset for measuring how LLMs can use long contexts to understand the implicit emotion in an utterance (which often cannot be fully determined just from the local context). The dataset is generated with LLMs and the evaluation of the LLMs is performed by LLMs. The results show that LLMs do better when they have access to the context, and the gap is larger for harder, less local determination tasks.

Critical strengths

The problem of how an LLM can understand a human's emotions from long interactions seems very important to me. The authors assert there are no datasets on this. I am not an expert in this area so I cannot easily verify, but if this is true then contributing a dataset of this form seems potentially valuable.

Critical weaknesses
  1. Biggest issue: the dataset is not provided. There is no supplementary material and no anonymized link in the paper. Thus, there is no way to review the dataset itself.
  2. The experiments do not seem particularly informative. A good experimental section in a datasets paper should identify or hypothesize what the structural failure modes are. The observation that long history is necessary to solve a long-horizon task is a zeroth-order sanity check but not otherwise informative.
  3. The interactions in the dataset are fully LLM generated and fully LLM evaluated. I understand that there may be ethical issues with getting real human data, but for a benchmark about ability to interact with humans, this seems unideal. Maybe there are middle grounds such as using conversations from fiction.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.