acceptodds
Under review as a conference paper at ICLR 2027

AffectSim: A Controllable Interactive 3D Simulation Benchmark for Embodied Affective Perception

Abstract

Existing affective benchmarks largely rely on fixed recordings, where observation conditions are predetermined before inference. Consequently, they mainly evaluate passive affect recognition while overlooking a key question for embodied agents: how to actively acquire informative affective evidence. We introduce AffectSim, a controllable interactive 3D simulation benchmark for embodied affective perception. AffectSim represents affective behaviors as replayable 3D episodes and separates the underlying behavior from how it is observed, enabling systematic control over distance, orientation, occlusion, scene geometry, and agent viewpoint. It contains 27,647 episodes across five emotion categories and 57 scenes. Across 24 frozen perception-model configurations, we show that observation quality substantially affects recognition. To examine whether active sensing can recover this gap, we establish a training-free heuristic baseline, which improves recognition in 21 of 24 settings. Building on this, we propose Evidence-Aware Observation Gate (EAOG) to adaptively assess whether additional observation is still beneficial. These results show that affective perception depends not only on the observed behavior, but also on how evidence is acquired. By making observation acquisition experimentally controllable, AffectSim provides a foundation for studying embodied affective perception in interactive 3D environments. All benchmark assets, code, and simulation environments will be released.

Primary area:
datasets and benchmarks
Keywords:
affective computing, emotion recognition, active affective perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eagzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AffectSim is a simulation benchmark for active body-based emotion recognition. It separates the underlying motion from observation conditions, allowing matched comparisons across viewpoints. The paper also introduces EAOG, a learned gate that decides when to stop observing.

Critical strengths

I like the controlled-replay design. Observing the same behavior under different conditions gives the simulator a clear purpose beyond producing more data. Keeping variants of the same performance within one split is also a sensible choice.

The paper is clear and includes useful controlled comparisons. With the same adapted recognizer, learned active observation improves Macro-F1 from 32.74% to 45.47% over the initial view. Fixing the navigator in the stopping experiments also helps separate the benefit of movement from the benefit of observation timing.

Critical weaknesses

The emotion-specific insight is not yet clear. The navigation teacher mainly encourages visibility, framing, and a suitable distance. Emotion-specific supervision enters through the observation gate. This leaves me wondering how much the benchmark requires beyond obtaining a clearer body view. With the navigator fixed, EAOG improves Macro-F1 by only 1.20 points over random stopping. A simple visibility-based gate at matched observation cost would help establish what the affect-specific component adds.

The scope also limits the broader motivation. The benchmark uses five categories and looping, predefined body motions. People do not respond to the robot, and facial and speech cues are left for future work. These choices support controlled perception experiments, but do not establish benefits for natural interaction or collaboration. The authors acknowledge these limitations. Given the focus on body motion, a pose-estimation-plus-emotion-classification baseline using the same RGB clips would be particularly useful. It would test whether the observation gains carry over to a motion-based representation.

Label reliability and dataset coverage need a clearer account. Human majority votes match the source labels on 78% of the 500 checked motions. This is useful validation, but the remaining disagreement deserves analysis. Is it caused by ambiguous performances, source annotations, or retargeting? It is also unclear whether uncertain samples are retained, relabeled, or flagged.

Table B reports actor counts for the action-grounded subset, but not for the other two components. A source-by-source breakdown of distinct performers, original performances, and split overlap would make the dataset’s diversity easier to judge. The paper also describes manual episode screening without clearly quantifying its coverage. The large episode count should not be taken as equivalent to independent human performances.

Questions for authors
  • How are disagreements between human judgments and source labels handled? Can you report reviewer agreement and compare judgments before and after retargeting?
  • How many distinct performers and original performances come from each source? Are performers disjoint across splits in every component, and how many episodes received manual review?
  • How does EAOG compare with a visibility-based gate and a validation-tuned fixed stopping time at similar observation cost?
  • Are pose-based recognition results available using poses estimated from the same RGB clips, with matched fixed-view and active-view budgets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer eagz (Part 1/2)

Thank you for your constructive review. We appreciate your recognition of the controlled-replay design, and useful controlled comparisons that separate movement from observation timing. Following your suggestions, we conducted additional stopping comparisons, pose-based evaluations, and detailed label and dataset analyses.

 

R3-W1 & Q3. Emotion-Specific Contribution of EAOG

(1) Visibility-Based and Budget-Matched Comparisons.

We agree that the navigation teacher mainly improves person framing, while EAOG introduces affect-conditioned evidence assessment through stopping. To isolate this contribution, all methods use the same frozen navigator, trajectories, candidate windows, and recognizer; only stopping differs.

The Visibility-and-Scale gate uses YOLO11x-Pose keypoint visibility and target bounding-box area, averaged over 32 frames, with thresholds tuned on validation data. Fixed 34s matches EAOG's validation observation budget; Fixed 45s is the best validation-selected fixed time. We also sample stopping times from EAOG's validation-set stopping-time distribution (five seeds), matching its marginal time distribution.

EAOG outperforms Visibility-and-Scale by 3.22 pp (95% CI [0.16, 5.91]) and Fixed 34s by 1.48 pp ([0.14, 3.49]). Against distribution-matched random stopping, the gain is 1.23 pp ([0.26, 2.39]). Compared with Fixed 45s, EAOG uses 25.1% less time (Table 1).

Table 1. Stopping strategies with identical navigation and recognizer. F1 is evaluated on 3,941 episodes; fixed times/thresholds are validation-selected. Random results are mean ± SD over five seeds. CIs use paired source-group bootstrap comparisons.

StoppingStrategyMacro-F1(%)↑Avg.Time(s)↓
Visibility-and-Scale42.2536.03
Fixed 34s (budget-matched)43.9934.00
Fixed 45s (validation-best)44.6645.00
Uniform Random (5 seeds)44.17 ± 0.4230.04 ± 0.17
Validation-Distribution Random (5 seeds)44.24 ± 0.5633.50 ± 0.24
EAOG45.4733.72

 

(2) Affect-Specific Supervision and Scope.

Removing EAOG's auxiliary emotion loss reduces Macro-F1 from 45.47% to 42.65% (−2.82 pp). Together, these comparisons show a benefit beyond the tested geometric-visibility heuristic.

 

R3-W2 & Q4. Pose-Based Recognition and Benchmark Scope

Thank you for suggesting a pose-based baseline. We agree that predefined body motions and the absence of facial, vocal, or reactive behaviors limit claims about natural human–robot interaction.

(1) Matched RGB-Based Protocol.

We used YOLO11x-Pose to estimate skeletons from the exact RGB clips used in the original evaluation, followed by CTR-GCN for emotion recognition. Within each P-Init or A-Obs condition, the pose-based and original recognizers receive identical clips, trajectories, temporal windows, and stopping times. This matches inputs and budgets across recognizers within each condition, not between P-Init and A-Obs. CTR-GCN is trained on P-Ref and evaluated unchanged; pose-estimation failures remain in the evaluation.

 

(2) Results and Limitations.

Active observation improves pose-based Macro-F1 on Emilya (+5.12 pp) and KDAE (+5.81 pp), with +4.15 pp overall (Table 2). Valid pose estimation rises from 55.77% (P-Init) to 79.24% (A-Obs). The Dyadic Pilot declines by 5.62 pp; our current estimator cannot reliably associate detected poses with the designated target actor. Thus, the result supports active observation for single-person RGB-derived motion recognition, not consistent gains in dyadic interaction.

Table 2. Pose-based Macro-F1 (%) from the same RGB observations. CTR-GCN is trained on P-Ref and held fixed; failed pose estimates remain in the evaluation.

SourceP-InitA-ObsΔ(pp)
Emilya19.3124.43+5.12
KDAE38.8844.69+5.81
Dyadic Pilot25.1919.57−5.62
Overall25.8329.98+4.15
AuthorsOpenReview ↗

Response to Reviewer eagz (Part 2/2)

R3-W3 & Q1. Emotion-Label Reliability and Disagreement Handling

Thank you for raising label reliability after retargeting and disagreement handling. We conducted blind evaluations across representations and separately reassessed disputed cases.

(1) Human Agreement Across Representations.

We sampled 500 source motions (320 KDAE, 162 Emilya, 18 dyadic) and presented each as an original skeleton (S), retargeted skeleton (T), and final RGB animation (A). Different groups of three independent blind raters evaluated S, T, and A (Table 3).

For pairs with valid majority-vote labels in both representations, consistency is 88.53% (386/436) for S-T. 83.93% (376/448) for T-A, and 85.29% (377/442) for S-A. This indicates that perceived emotion is largely preserved across stages.

Table 3. Blind recognition across representations. Macro-F1 measures majority-vote agreement with source labels; Fleiss' κ measures within-representation inter-rater agreement.

RepresentationMacro-F1(%)Fleiss'κ
Original Skeleton (S)76.390.665
Retargeted Skeleton (T)74.240.679
Final RGB Animation (A)79.860.734

 

(2) Disagreement Tracing and Label Reassessment.

Of the 500 final RGB animations, 111 do not match their source labels by majority vote: 85 receive a different emotion and 26 have no clear majority. Of these, 81/111 already disagree or remain ambiguous at S, including 32 assigned the same non-source emotion across S, T, and A. The other 30 first differ at T (6) or A (24). This suggests that many disagreements originate in source-motion ambiguity, though different rater groups prevent definitive causal attribution.

In a label-informed reassessment of all 111 cases, three raters independently judged source-label consistency with the final RGB animation.** 102/111 (91.89%) received majority acceptance, including 77/85 previously assigned another emotion and 25/26 previously lacking a majority; 72/111 were unanimously accepted. Individual acceptance was 278/333 (83.48%), supporting the perceptual plausibility of source labels for most disputed cases.

 

(3) Label Handling and Auditability.

We retain original source labels, rather than automatically relabeling or removing disagreements, to preserve a consistent benchmark target. To enable uncertainty analysis, we will release human judgments and audit flags for all 500 evaluated motions, identifying agreement, disagreement, and ambiguity across S/T/A.

 

R3-W4 & Q2. Dataset Coverage and Manual Screening

Thank you for asking about independent source performances and screening coverage. We distinguish source performances from episodes and report source-level actor/split statistics and human review coverage.

(1) Dataset Diversity and Split Integrity.

AffectSim contains 1,307 source motions or synchronized dyadic sequences, generating 27,647 episodes. Table 4 reports source counts, distinct actors, and their overlap across training (Tr), validation (Val), and test (Te).

Emilya is actor-disjoint, with 278/76/68 source motions in Tr/Val/Te; KDAE (669/84/84) and Dyadic Pilot (34/7/7) are source-motion-disjoint but not actor-disjoint. Across all sources, each source motion and all its episodes stay in one split, with zero source-motion overlap.

Table 4. Source-level dataset diversity and cross-split actor overlap. Overlap counts actors shared by each split pair; episode counts do not represent independent performances.

SourceActorsSource MotionsEpisodesActor Overlap(Tr–Val/Tr–Te/Val–Te)
Emilya1242217,9010 / 0 / 0
KDAE228378,30621 / 21 / 20
Dyadic Pilot2481,4402 / 2 / 2
Total—1,30727,647—

 

(2) Quality Control and Manual Screening Coverage.

We distinguish episode-level human screening from source-motion-level checks. For Emilya, scene-dependent actions require valid body–scene geometry and object contacts. After configuration refinement and emotion filtering, 52,573 candidate episodes remained; 28,765 underwent individual manual review, assessing interaction plausibility, object contacts, and rendered motions. All 17,901 retained Emilya episodes passed individual human screening (Table 5).

Table 5. Emilya episode-level manual screening by action type. All retained episodes were individually reviewed.

Action TypeManually ReviewedRetained
BS (Being Seated)11,9007,684
KD (Knocking)4,7001,791
SD (Sitting Down)4,1651,076
SW (Walking)8,0007,350
Total28,76517,901

For KDAE, all 8,306 episodes were manually checked. For Dyadic Pilot, 48 of 60 captured interaction sequences were retained and instantiated into 1,440 episodes.

Appendix B will clarify review levels and actor/split statistics, distinguishing episodes from independent performances.

Reviewer_yYFHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AffectSim, an interactive 3D simulation benchmark built in Habitat to evaluate embodied affective perception under controllable observation conditions. By decoupling underlying human affective motions (retargeted from existing datasets and a dyadic capture) from scene placement and camera trajectories, AffectSim generates 27,647 replayable episodes across 57 scenes and five emotion categories. Using matched passive and active evaluation protocols across 24 frozen vision-language and affect recognition models, the authors demonstrate a clear performance gap between initial egocentric views and privileged reference views. They further propose a training-free heuristic active baseline and a learned closed-loop system featuring an Evidence-Aware Observation Gate (EAOG) to balance recognition quality against observation time.

Critical strengths
  1. Well-Motivated Paradigm Shift: Moving affective computing from passive, pre-framed recordings to active embodied viewpoint acquisition addresses a practical bottleneck in human-robot interaction. Separating the instantiated behavior from how it is observed allows systematic, repeatable stress-testing across distance, orientation, and occlusion.

  2. Rigorous Matched Evaluation Protocol: Evaluating 24 open- and closed-source models across paired observation settings (P-Init, P-Mon, P-Ref, and A-Obs) cleanly isolates the impact of observation quality from model capacity. The proposed metrics: Observation Gap Recovery (OGR), Recovery Rate, and Emotion-SPL (E-SPL), are well-suited for active perception.

  3. Effective Active Sensing Reference Solutions: Both the training-free heuristic controller and the learned EAOG termination module demonstrate consistent empirical gains over passive initial views, showing that adaptive stopping improves both Macro-F1 and observation efficiency.

Critical weaknesses
  1. Simulation-Only Scope and Modality Constraints: The benchmark and evaluations are conducted exclusively in simulation using retargeted skeletal avatars without facial expressions, vocal/audio cues, or reactive human behaviors. Because human affect in real-world robotics is heavily multimodal and interactive, it remains unclear how well observation policies learned on rendered body-only avatars transfer to physical environments.

  2. Unvalidated Label Inheritance After Scene Composition: In Eq. (1) (Line 147), an episode is composed as e=C(m,S,ph,x0)e = \mathcal{C}(m, S, p_h, x_0) and its emotion label is directly inherited from the source motion mm. While Section 3.1 (Line 158) and Appendix B.3 report a 77.81% Macro-F1 human validation score on 500 isolated retargeted motions, and Section 3.3 notes that reviewers screen composed episodes for physical plausibility, there is no human validation verifying that the perceived emotion remains unchanged after composing the motion into diverse 3D scenes and environmental contexts.

  3. Missing Dataset Proportions and Annotation Protocol for the Dyadic Pilot: Section 3.1 (Line 152) states that the 1,307 motion assets combine KDAE, Emilya, and a newly captured dyadic pilot dataset. However, neither the main text nor Appendix Table B specifies the exact proportion of motions and episodes originating from KDAE versus Emilya (Table B groups single-person data by "Action-grounded" and "Observation-controlled" rather than source dataset, and omits the motion count for the Action-grounded split). Furthermore, while Table B indicates 48 motions (1,440 episodes) come from the newly captured dyadic pilot, the paper provides no details on the capture hardware, actor instructions/scripts, or how ground-truth emotion labels were annotated and verified for this new dataset.

  4. Conceptual Ambiguity in Sharing Labels Across Degraded Observation Configurations: A core premise of the paper is that poor viewpoints (e.g., far distance or severe occlusion in P-Init or early search steps) render affective cues ambiguous or unobservable, often appearing "neutral" or "unclear" (as illustrated in Figure 1 and Figure C). Yet, all observation configurations inherit the exact same ground-truth label from mm, and EAOG's auxiliary emotion loss (Lemotion\mathcal{L}_{\text{emotion}} in Eq. 23) directly supervises intermediate windows t∈Tt \in \mathcal{T} using the episode's target emotion label ycy_c regardless of whether affective evidence is actually visible in window Wt\mathcal{W}_t. Evaluating passive views or supervising intermediate active views with the actor's underlying label, rather than viewpoint-specific perceived labels or an explicit "unobservable/uncertain" category, conflates the absence of visual evidence with model recognition error.

Questions for authors
  1. Dataset Breakdown and Dyadic Capture Protocol: What are the exact proportions (in terms of source motions and generated episodes) of KDAE, Emilya, and the dyadic pilot dataset across the benchmark? Additionally, how was the new dyadic pilot dataset captured, and what annotation protocol (e.g., self-report, script label, or external raters) was used to assign target emotion labels to each interacting actor? For the newly captured dyadic pilot dataset, could the authors also confirm whether the human motion-capture collection received IRB (or institutional ethics committee) approval and provide brief details on the participant consent and annotation protocol in Appendix B?

  2. Label Validity of Composed Episodes: Did the authors conduct any human perceptual study on the final composed episodes e=C(m,S,ph,x0)e = \mathcal{C}(m, S, p_h, x_0) (e.g., under P-Ref) to verify that scene context, lighting, and object interactions do not alter the perceived emotion inherited from mm?

  3. Perceptual Labels Under Degraded Views: Since observation conditions significantly alter what affective cues are visible, should degraded observation configurations (such as P-Init under heavy occlusion) be annotated with viewpoint-specific human perceptual labels (or an "uncertain/unobservable" state) rather than inheriting the source motion label? Similarly, in Eq. (23), does supervising Lemotion\mathcal{L}_{\text{emotion}} with the ground-truth emotion label ycy_c on early, occluded candidate windows introduce noisy supervision into EAOG's representation learning?

  4. Sim-to-Real Generalization: How sensitive are the behavior-cloned Uni-NaVid policy and EAOG module to visual domain shifts or real-world human appearance when deployed outside the HSSD/Habitat rendering pipeline?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yYFH (Part 1/2)

Thank you for your constructive review. We appreciate your recognition of the well-motivated paradigm shift, rigorous matched evaluation protocol, and effective active sensing reference solutions. Following your suggestions, we conducted additional human validation, visual-domain robustness tests, and supervision analyses to strengthen the study.

 

R2-W1 & Q4. Simulation Scope and Visual-Domain Generalization

(1) Why Controlled Simulation?

AffectSim focuses on body-motion-based embodied affective perception: how viewpoint, distance, and occlusion affect emotion recognition, and whether an agent can actively obtain better affective evidence. Simulation lets us replay the same behavior under different viewpoints, distances, and occlusion conditions while holding the source motion fixed.

The same underlying behavior can be replayed under different observation conditions—difficult to achieve in natural human interactions. Across 24 frozen recognizer configurations, we observe substantial performance differences while holding the source behavior and recognizer fixed.

 

(2) Visual-Domain Robustness.

Following your suggestion, we evaluated 400 stratified test episodes from 125 source-motion groups, spanning all three data sources and five emotions. We tested an unseen virtual avatar, Gaussian blur (7×7, σ=1.5), Gaussian noise (σ=8/255), and exposure changes (±1 EV). All models were frozen. We replayed identical navigation trajectories to isolate visual-domain effects, regenerated or perturbed RGB observations, and recomputed EAOG stopping decisions and emotion predictions. Relative to Clean (44.94% Macro-F1), unseen-avatar substitution and noise cause small decreases (−0.66 and −0.36 pp), whereas reduced exposure and blur cause larger drops (−3.02 and −5.62 pp). This indicates robustness to some tested changes, but sensitivity to degraded visual evidence (Table 1).

Table 1. Emotion recognition under visual domain shifts. The same 400 test episodes and fixed navigation trajectories are used throughout; stopping decisions and predictions are recomputed under each condition.

Visual ConditionMacro-F1(%)↑Δvs.Clean(pp)
Clean44.94—
Unseen virtual avatar44.28−0.66
Gaussian noise44.58−0.36
Exposure +1 EV43.50−1.44
Exposure −1 EV41.92−3.02
Gaussian blur39.32−5.62

 

(3) Scope of the Evidence.

This is a fixed-trajectory evaluation of the EAOG–recognizer pipeline, not a closed-loop robustness test of Uni-NaVid. It also does not isolate EAOG from recognizer sensitivity. The benchmark currently excludes facial, vocal, and reactive human behaviors; virtual-avatar generalization is not real-world transfer. We will clarify these boundaries and highlight multimodal interaction and physical deployment as future work.

 

R2-W2 & Q2. Perceptual Validity and Cross-Composition Stability

Following your suggestion, we evaluated final composed episodes, rather than relying only on validation of isolated retargeted motions and physical-plausibility screening.

(1) Human Validation of Final Episodes.

We sampled 500 P-Ref episodes from the training set: 100 distinct source motions × five scene or character-placement variants, spanning 31 scenes, Emilya/KDAE/NOKOV, and five emotions (100 episodes each). Three independent raters blindly labeled each episode. Majority votes matched the inherited labels in 403/500 episodes (80.60%), yielding 81.66% Macro-F1; 10 episodes had no clear majority.

 

(2) Cross-Composition Stability.

Across five variants of the same motion, 76/100 source groups retained the same perceived emotion, including 71 groups consistently matching the source label. Of the remaining groups, 15 exhibited a change in majority emotion and nine included at least one ambiguous variant. Thus, perceived emotion is stable under most tested compositions (Table 2).

Table 2. Cross-composition emotion consistency. Each source-motion group contains five composed variants, each assessed by three blind raters.

Judgments Across Five VariantsSource Groups
Consistent and matching source label71
Consistent but different from source label5
Different majority emotions across variants15
At least one variant without a majority9
Total100

 

(3) Label-Informed Check.

Different raters also assessed whether a provided source label was plausible for each composed episode. 436/500 (87.20%) received majority acceptance, including 368/500 (73.60%) with unanimous acceptance. This complements the blind study but may be affected by label prompting. We will report these results and limitations in the revision.

AuthorsOpenReview ↗

Response to Reviewer yYFH (Part 2/2)

R2-W3 & Q1. Dataset Composition, Dyadic Capture, Annotation, and Ethics

(1) Dataset Breakdown and Splits.

AffectSim contains 1,307 source motions/dyadic sequences and 27,647 episodes. Action-grounded corresponds to Emilya (422 motions); Observation-controlled to KDAE (837). Episodes are scene/view instantiations, not independent performances. All variants of a motion share one split: Emilya is actor-disjoint; KDAE and Dyadic Pilot are source-motion-disjoint (actor overlap possible).

Table 3. Dataset composition. Percentages refer to the full benchmark; each dyadic sequence counts as one source unit.

SourceProtocolSource Motions(%)Episodes(%)
KDAEObservation-controlled837 (64.04%)8,306 (30.04%)
EmilyaAction-grounded422 (32.29%)17,901 (64.75%)
Dyadic PilotDyadic interaction48 (3.67%)1,440 (5.21%)
Total—1,307 (100%)27,647 (100%)

 

(2) Dyadic Collection and Labels.

Two adults were synchronously recorded using NOKOV optical motion capture (60 Hz), performing scripts for Angry, Fearful, Happy, Sad, and Neutral. Scripts specify roles and affective posture, movement intensity, rhythm, and interpersonal positioning, without speech or props. We retained 48/60 recordings, each instantiated across 10 scenes and three observation conditions (1,440 episodes).

Labels are script-assigned intended emotions, not self-reports. Each episode identifies one target actor; the partner does not automatically share that label. Of 48 sequences, 24 target the male actor and 24 the female actor. Target identities and labels remain fixed across instantiations.

For validation, three independent blind raters evaluated 50 final episodes from 10 source sequences, balanced across five emotions. Majority votes matched script labels in 46/50 (92.0%), achieving 91.67% Macro-F1 and Fleiss' κ = 0.829.

 

(3) Ethics and Consent.

Our university's Artificial Intelligence Ethics Committee approved the study. Both adults provided written informed consent before collection. Released virtual-avatar episodes contain no identifiable faces or speech. We will document these details in Appendix B.

 

R2-W4 & Q3. Emotion Labels and Supervision under Limited Observability

Thank you for distinguishing underlying emotion from viewpoint-dependent perceptual evidence. We clarify the task definition and test supervision under occlusion.

(1) Fixed Labels vs. Viewpoint Observability.

AffectSim evaluates recovery of the underlying source-motion emotion across observation conditions. Fixed labels enable controlled passive–active comparisons. As you noted, recognition failure under occlusion may reflect insufficient evidence, not misinterpretation. Our metrics thus measure recognition under available observations, not recognizer capability independently of visibility. Viewpoint-specific labels or an uncertain/unobservable category could complement this evaluation.

 

(2) Emotion-Loss Masking Ablation.

To examine potentially noisy supervision, we mask auxiliary emotion cross-entropy when the person is absent across all 32 frames. Visibility is measured using paired RGB renders with/without the person under identical camera/scene settings, with an absolute channel-difference threshold of 12.

Of 543,345 training windows, 13,367 (2.46%) meet this criterion. We retrained EAOG with the same seed, data, architecture, and training budget, masking only these emotion-loss gradients. Other losses remain active; masking is training-only.

We evaluated the masked model at a validation-tuned threshold. As shown in Table 4, masked EAOG retains most of the original recognition performance, with a 1.89-percentage-point decrease in Macro-F1 and a 2.25s (6.67%) reduction in mean stopping time (Table 4). The masked model thus retains most of EAOG’s recognition performance while stopping earlier.

Table 4. Auxiliary emotion-loss masking on fully unobservable windows. Other losses remain unchanged; threshold tuning uses validation data. All results use the full test set.

EAOG VariantMasked WindowsMacro-F1(%)↑Avg.Time(s)↓
Original EAOG045.4733.72
Masked EAOG13,367 (2.46%)43.5831.47
Reviewer_yYFHOpenReview ↗

Concern regarding emotion label

Thank the authors for the clarification; most of my concerns have been addressed.

However, regarding the emotion label, my concern would remain the same as Reviewer 5TSB (W1) and Reviewer eagz (W3), since the most valuable contribution is the dataset AffectSim, while the emotion label is the most important property for the dataset, and for other downstream tasks built upon this dataset. Current validation can hardly provide strong evidence for the reliabiility of the emotion label.

One interesting question is: If human evaluators disagree with the inherited labels, why not replace them with human-annotated labels?

AuthorsOpenReview ↗

Emotion-Label Reliability: Intended Emotion vs. Perceived Emotion

Thank you for emphasizing this important issue. We fully agree that emotion-label reliability is fundamental to AffectSim and its downstream applications. We had carefully considered this issue during dataset construction. Regarding whether human annotations should replace the original labels, we distinguish source-defined intended emotions from observer-perceived emotions, as they represent different annotation targets.

 

(1) Source Labels Are Grounded in Expression Protocols and Prior Validation.

The source labels are based on established emotion-expression protocols and validation procedures.

  • Emilya: Eleven actors received seven professional training sessions (three hours each). A perceptual study involving 1,008 participants evaluated 664 motion segments, with 24 ratings per segment. Inter-rater Cronbach's α exceeded 0.7 for 90.36% of the stimuli [A, B].
  • KDAE: Seventy participants evaluated 70 emotion-eliciting scenarios, from which the five most recognizable scenarios per emotion were selected. Subsequently, 22 semi-professional actors performed scenario-based and free expressions based on their own interpretations. Motion-signal quality and performance naturalness were also checked individually [C].
  • Dyadic Pilot: Emotion labels were assigned through predefined interaction scripts to designated target actors. A blind evaluation of 50 final episodes achieved 92.0% majority agreement with script labels, 91.67% Macro-F1, and Fleiss' κ = 0.829.

These procedures support the provenance and plausibility of source labels, although they do not guarantee unambiguous perception of every motion.

 

(2) Human Perceptual Labels Are Complementary, Not Substitutes.

Source labels define what the performer intends to express; human annotations reveal what observers actually perceive. Both are valuable, but they serve different research purposes.

Source labels define the intended emotion to be recovered, whereas human annotations reflect what observers perceive under particular visual conditions, which may vary with viewpoint, occlusion, and scene composition.

To examine label validity after scene composition, we evaluated 500 P-Ref episodes covering 100 source motions with five composed variants each. Three independent raters blindly annotated each episode. Majority votes matched the inherited labels in 403/500 episodes (80.60%), yielding 81.66% Macro-F1; 10 episodes had no clear majority.

Across the five variants of each source motion, 76/100 groups retained consistent perceived emotions, including 71 consistently matching the source labels and five consistently receiving a different emotion. The remaining 24 showed varying judgments (15) or at least one ambiguous variant (9).

Replacing source labels with episode-level majority votes could assign different targets to different observations of the same motion. Aggregating votes across variants would instead redefine the task as predicting observer-perceived emotion rather than recovering the intended emotion.

 

(3) We Preserve Source Labels While Providing Complementary Human Annotations.

We therefore retain the original source labels rather than automatically replacing them with human majority votes or excluding samples based on perceptual disagreement. We will provide human perceptual judgments and audit flags for the evaluated samples, distinguishing agreement, disagreement, and cases without a clear majority.

This allows downstream researchers to study both intended-emotion recognition and emotion perception under specific observation conditions.

We believe preserving both complementary annotation layers provides greater transparency and value for downstream research than replacing one with the other.

 

References

[A] Emilya: Emotional Body Expression in Daily Actions Database, Proceedings of LREC, 2014.

[B] Perception of Emotions and Body Movement in the Emilya Database, IEEE Transactions on Affective Computing, 2018.

[C] Kinematic Dataset of Actors Expressing Emotions, Scientific Data, 2020.

Reviewer_5TSBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

AffectSim separates affective body-motion assets from the conditions under which an embodied observer sees them. It instantiates 27,647 episodes from 1,307 motions in 57 scenes, with five emotion categories and single-person and dyadic behaviors. Matched initial, reference, monitor, and active observations are used to study observation-dependent recognition. The paper evaluates a training-free navigation/controller baseline across 24 frozen recognizer configurations and introduces a learned observation system combining behavior-cloned navigation with the Evidence-Aware Observation Gate (EAOG). EAOG is trained to choose a recognition window using a recognition-risk-plus-delay objective, auxiliary emotion supervision, and quality prediction.

Critical strengths

Strengths

  1. A meaningful experimental distinction. The separation of motion identity from observation conditions is clear and useful (§§3–4, pp. 3–5). Treating simulation as a controlled re-observation instrument is more compelling than merely generating additional rendered clips.
  2. Breadth of paired recognizer evaluation. Table D supports the reported improvements for 21 of 24 heuristic configurations and the best passive performance of P-Ref in 23 of 24 configurations (pp. 7, 23). Within-configuration comparisons are more informative here than a simple model leaderboard.
  3. Several important controls already exist. Table 4 fixes the learned navigator and recognizer when comparing stopping strategies; Table 5 ablates training objectives. The paper also provides human semantic validation, class-level results, episode-family diagnostics, and navigation-cost measurements. These should not be overlooked when assessing experimental completeness.
Critical weaknesses

Major Concerns

  1. Label preservation after retargeting needs stronger validation The benchmark inherits emotion labels after source filtering, label harmonization, retargeting, and avatar substitution. Appendix B.3 and Table C (p. 17) provide useful validation: three reviewers independently label 500 retargeted motion assets, and majority-vote predictions achieve 78% agreement with source labels and 77.81% Macro-F1. However, this does not distinguish ambiguity in the source performances from semantic changes introduced by retargeting or rendering. It also leaves unclear how well the sampled assets represent the different motion sources. The disagreement should not be interpreted as a 22% labeling-error rate or a human-performance ceiling. Nevertheless, its origin matters because benchmark errors may otherwise conflate recognition failure, observation difficulty, and label ambiguity. Please provide the source-specific label mappings and filtering criteria, the validation sample composition, and inter-rater agreement in addition to the existing class-wise results. A focused analysis of disputed examples would be particularly useful. If feasible, a small paired source-versus-retargeted assessment would more directly test semantic preservation without requiring exhaustive reannotation.
  2. Affect-specific observation gains are not sufficiently separated from generic person framing The navigator follows a fixed instruction to find the person and maintain a clear, well-framed body view. Its expert supervision emphasizes navigability, distance, visibility, and stable observation (Section 5.1; Appendices D.1–D.2, pp. 18–19). As described, affect-conditioned stopping does not feed back into the choice of the next movement action. Consequently, much of the movement behavior could also benefit generic human-activity recognition, leaving the affect-specific requirements of active observation insufficiently characterized. Importantly, EAOG is not affect-agnostic: it uses emotion-conditioned queries, auxiliary emotion supervision, and recognizer-derived risk targets. Table 5 also reports a decrease from 45.47% to 42.65% Macro-F1 when the emotion loss is removed. These are meaningful results, but they do not fully establish what is gained beyond generic visual-quality-based observation selection. Please distinguish the contribution of person framing from that of affect-conditioned stopping and explain the task-specific motivation for the Uni-NaVid backbone. One targeted comparison with a visibility/framing-based stopping rule using the same admissible observations and navigation trajectories would help test whether affect-conditioned evidence provides additional value. A simpler navigation controller would be useful if the authors wish to make stronger claims about the backbone itself, but a new architecture is not a prerequisite for a benchmark contribution.
  3. The incremental benefit and quality–cost advantage of EAOG remain insufficiently established Table 4 (p. 8) already provides an appropriate gate-specific comparison by fixing the learned navigator and recognizer. Under this control, EAOG achieves 45.47% Macro-F1, versus 44.27% for uniform random stopping and 43.94% for the best displayed fixed-time rule. The incremental gains are therefore 1.20 and 1.53 percentage points, respectively. These gains are promising, but the larger improvement over P-Init in Table 3 reflects the combined navigation-and-stopping system and should not be read as EAOG’s isolated contribution. The observation budgets also differ. EAOG stops at 33.72 seconds on average (Table E, p. 24), whereas the fixed 30-second rule uses less time. Uniform sampling over the 37 integer candidates from 12 to 48 seconds has an expected mean of 30 seconds, although its realized evaluation mean is not reported. Stopping earlier than the 48-second endpoint demonstrates savings relative to that endpoint, but does not establish superiority at matched average observation cost. E-SPL provides complementary path-efficiency information, not a substitute for a time-matched comparison. Please report the realized random-stopping time and a matched-mean-time comparison or a recognition-versus-observation-time curve with operating points selected on validation data. The small differences also warrant paired uncertainty estimates, repeated random-stopping draws, and disclosure of available training-run variability. Since multiple episodes derive from the same source performance (Table B, p. 17), uncertainty estimation should account for that dependence, for example through grouped paired resampling. Much of this analysis may be possible using existing trajectories and predictions.

Additional Clarifications

  • Explain the low zero-shot scores. Table D (p. 23) reports very low Macro-F1 for several recognizer configurations. Prediction distributions, confusion matrices, and the frequency of invalid or unmapped outputs would help distinguish domain mismatch and label bias from observation-quality limitations.
  • Clarify the fearful-class trade-off. Figure 5 (p. 9) reports a 20.6-percentage-point decrease in fearful recall for EAOG relative to P-Init. P-Ref also decreases fearful recall by 12.6 points, so the effect cannot be attributed entirely to the gate. Please provide absolute class-wise metrics and compare EAOG with fixed stopping under the same learned navigator to better separate viewpoint effects from stopping-induced changes.
Questions for authors

see the above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5TSB (Part 1/3)

Thank you for your constructive review. We appreciate your recognition of the meaningful experimental distinction, the breadth of paired recognizer evaluation, and the important controls already exist. Following your suggestions, we conducted additional human validation, stopping comparisons, and error analyses to further strengthen the study.

 

R1-W1. Emotion-Label Preservation After Retargeting

We clarify the label mappings and filtering criteria and conduct paired human evaluations across original motions, retargeted motions, and final RGB animations to distinguish source-motion ambiguity from processing-induced changes.

(1) Label Mapping and Motion Filtering.

We harmonize labels according to their original definitions:

  • Emilya: Joy → Happy, Anger → Angry, Panic Fear → Fearful, Sadness → Sad; Neutral is retained. Anxiety, Shame, and Pride are excluded.
  • KDAE: Happiness → Happy, Anger → Angry, Fear → Fearful, Sadness → Sad; Neutral is retained. Disgust and Surprise are excluded.
  • Dyadic Pilot: Interaction scripts directly target Angry, Fearful, Happy, Sad, and Neutral, without label remapping.

Emilya retains walking, knocking, and sitting motions compatible with scene geometry, excluding incompatible object-dependent actions. KDAE retains scenario-elicited affective motions passing quality checks.

 

(2) Human Validation Across Representations.

We sampled 500 source motions approximately proportional to the motion library: 320 KDAE, 162 Emilya, and 18 dyadic motion pairs, with 105/105/105/95/90 samples for Angry/Fearful/Happy/Sad/Neutral, respectively. Three independent groups of three evaluators blindly annotated the same motions as original skeletons (S), retargeted skeletons (T), and final RGB animations (A), respectively. Three additional evaluators assessed paired posture and motion-dynamics consistency.

As shown in Tables 1–2, Macro-F1 ranges from 74.24% to 79.86%, with Fleiss' κ of 0.665–0.734. Importantly, 85.29% of valid S–A pairs retain the same perceived emotion. Paired posture and motion-dynamics ratings also exceed 4.96/5 for S–T and S–A, supporting substantial preservation of affective motion characteristics.

Table 1. Human emotion recognition and inter-rater agreement across representations. Macro-F1 compares majority-vote judgments with source labels; Fleiss' κ measures agreement among three independent raters.

MetricOriginal(S)Retargeted(T)Final RGB(A)
Macro-F1 (%)76.3974.2479.86
Fleiss' κ0.6650.6790.734

Table 2. Paired perceptual emotion consistency across processing stages. Valid pairs require clear majority votes in both representations; agreement denotes identical perceived emotions.

ComparisonConsistent/Valid PairsAgreement
S–T386/43688.53%
T–A376/44883.93%
S–A377/44285.29%

 

(3) Disagreement Analysis and Label Reassessment.

Among 500 final RGB animations, 111 lack a majority-vote match with their source labels (85 different labels; 26 without a majority). Of these, 81 already show disagreement or ambiguity at S, including 32 consistently assigned the same non-source emotion across S, T, and A. The remaining 30 first show disagreement at T (6) or A (24). Thus, most discrepancies are observable before retargeting, although later processing may also affect perception. Different rater groups prevent definitive causal attribution.

We further conducted a label-informed reassessment of all 111 disputed cases. Three raters independently judged whether the provided source label matched each final animation. 102/111 (91.89%) received majority acceptance, including 77/85 previously assigned another emotion and 25/26 previously lacking a majority; 72 received unanimous acceptance. Individual acceptance was 83.48% (278/333). This supports source-label plausibility rather than definitive correctness, given potential label-prompting effects.

 

(4) Label Handling.

We retain original source labels rather than automatically relabeling or excluding disputed motions. We will release audit flags and human judgments for the 500 evaluated motions, distinguishing agreement, disagreement, and ambiguous cases across S, T, and A. Other motions will be marked as unaudited.

AuthorsOpenReview ↗

Response to Reviewer 5TSB (Part 2/3)

R1-W2. Affect-Specific Observation Beyond Generic Person Framing

Following your suggestion, we introduce a Visibility-and-Scale stopping baseline to determine whether EAOG adds value beyond generic person-view quality.

(1) Visibility-Based Comparison.

Using YOLO11x-Pose, the baseline measures (i) the proportion of 12 reliably detected body keypoints and (ii) the target's relative bounding-box area. Both are averaged over EAOG's 32-frame observation window. At candidate times from 12 to 48s, it stops when both measures exceed validation-tuned thresholds, or at 48s otherwise. Thresholds maximize validation Macro-F1 under approximately matched observation budgets.

The navigator, candidate observations, and recognizer remain fixed; only the stopping rule changes. As shown in Table 3, EAOG improves Macro-F1 by 3.22 points (95% CI [0.16, 5.91]) while observing 2.31s less, demonstrating an advantage beyond the tested generic visibility-and-scale heuristic.

Table 3. EAOG versus visibility-based stopping. Both use identical trajectories, observation windows, and a frozen recognizer on test episodes. The gain's 95% CI uses 20,000 paired bootstrap resamples clustered by 159 source-motion groups.

Stopping StrategyMacro-F1(%)↑Avg.Time(s)↓
Visibility-and-Scale42.2536.03
EAOG45.4733.72

 

(2) Affect-Specific Contribution and Module Roles.

Removing EAOG's auxiliary emotion loss reduces Macro-F1 from 45.47% to 42.65% (−2.82 points). Together, these results support the benefit of affect-conditioned evidence evaluation beyond generic visual quality.

Uni-NaVid handles person framing and viewpoint acquisition, using video-history-based closed-loop navigation; EAOG evaluates affective evidence and decides when to stop. The current navigation actions are not directly emotion-conditioned. We will clarify this division of responsibilities in the revision.

 

R1-W3. EAOG's Incremental Benefit and Observation Cost

As you pointed out, the improvement from P-Init (32.74%) to the full system (45.47%) combines navigation and adaptive stopping, rather than EAOG alone. We therefore isolate stopping effects using identical navigation trajectories, candidate windows, and a frozen recognizer.

(1) Cost-Matched Comparisons.

Using validation data only, we select Fixed 34s, nearest to EAOG's mean validation stopping time (33.58s), and Fixed 45s, which achieves the best validation Macro-F1 among fixed-time rules.

We also introduce Validation-Distribution Random, sampling each test episode's stopping time independently from EAOG's validation-set stopping-time distribution. Unlike Uniform Random, this approximately matches EAOG's marginal stopping-time distribution without using episode-specific evidence. Both random baselines are repeated with five seeds across all 3,941 test episodes.

As shown in Table 4, EAOG achieves two significant improvements at similar observation costs:

  • Fixed 34s: +1.48 Macro-F1 points (95% CI [0.14, 3.49]); 33.72s versus 34.00s.
  • Validation-Distribution Random: +1.23 points (95% CI [0.26, 2.39]); 33.72s versus 33.50s.

Both confidence intervals exclude zero. The distribution-matched comparison further supports episode-adaptive stopping beyond simply reproducing a favorable stopping-time distribution. Against validation-best Fixed 45s, EAOG achieves comparable Macro-F1 with 25.1% less observation time.

Table 4. Recognition performance and observation cost under fixed navigation. Fixed stopping points are validation-selected; random results report mean ± standard deviation over five seeds. Gain intervals use 5,000 paired bootstrap resamples, stratified by data source and clustered by 159 source-motion groups. Random-baseline comparisons average the five independently evaluated Macro-F1 scores without prediction ensembling.

Stopping StrategyMacro-F1(%)↑Avg.Time(s)↓EAOG Gain95%CI
Fixed 30s43.9430.00+1.53 [−0.24, 2.36]
Fixed 34s (time-matched)43.9934.00+1.48 [0.14, 3.49]
Fixed 45s (validation-best)44.6645.00+0.81 [−0.33, 1.43]
Uniform Random (5 seeds)44.17 ± 0.4230.04 ± 0.17+1.30 [0.19, 1.78]
Validation-Distribution Random (5 seeds)44.24 ± 0.5633.50 ± 0.24+1.23 [0.26, 2.39]
EAOG45.4733.72—

 

(2) Training Variability.

Three independently trained EAOG models achieve 45.47%, 45.09%, and 45.28% Macro-F1 (45.28 ± 0.19%). Mean stopping times range from 25.36 to 33.72s, indicating relatively stable recognition performance but greater observation-cost variability.

AuthorsOpenReview ↗

Response to Reviewer 5TSB (Part 3/3)

R1-Q1. Understanding Low Zero-Shot Scores

Following your suggestion, we analyzed prediction distributions, confusion matrices, and invalid outputs across all 24 recognizer configurations.

(1) Prediction Bias.

The main finding is a strong bias toward Neutral, which represents only 16.75% of test labels. As shown in Table 5, open-source models predict Neutral in 89.55%/83.21% of P-Init/P-Ref cases on average, versus 75.19%/60.15% for closed-source models.

Qwen2.5-VL-32B-INT8 predicts Neutral in over 99.7% of cases, explaining its very low Macro-F1. Qwen2.5-VL-7B shows similar behavior (99.77% under both views). Better viewpoints alone cannot resolve such near-single-class predictions.

Table 5. Zero-shot Macro-F1 and Neutral prediction frequency. Group values average 19 open-source or five closed-source configurations.

Model/GroupP-InitF1(%)P-RefF1(%)Neutral Predictions(P-Init/P-Ref)
Open-source (19)9.8915.5689.55% / 83.21%
Closed-source (5)22.6130.7875.19% / 60.15%
Qwen2.5-VL-32B-INT86.126.2199.77% / 99.70%
VideoLLaMA2.1-7B-AV18.6727.6570.74% / 61.41%

 

(2) Confusion Analysis.

VideoLLaMA2.1-7B-AV predicts more diverse emotions. Under P-Ref, its Macro-F1 increases from 18.67% to 27.65%, while Neutral predictions decrease from 70.74% to 61.41%, particularly improving Happy and Sad recognition (Table 6). Better viewpoints therefore alleviate prediction bias in some models.

Table 6. VideoLLaMA2.1-7B-AV confusion matrix. Entries are P-Init / P-Ref counts on 3,941 episodes; rows are true labels and columns are predictions.

True\Pred.AngryFearfulHappyNeutralSad
Angry100 / 12120 / 927 / 50557 / 49633 / 61
Fearful83 / 10122 / 2339 / 64484 / 45345 / 32
Happy95 / 16724 / 1180 / 131598 / 52463 / 27
Neutral68 / 3217 / 732 / 59494 / 53949 / 23
Sad96 / 12928 / 1342 / 13655 / 408190 / 448

 

(3) Invalid Outputs.

Across four protocols, open-source models produce 97/299,516 invalid outputs (0.0324%), versus 10/78,820 (0.0127%) for closed-source models. These are unmappable emotion labels, not parsing failures, and all are counted as incorrect. Thus, low zero-shot scores mainly reflect class prediction bias rather than invalid outputs.

 

R1-Q2. More Details about Fearful-Class Results

Thank you for highlighting the unexpected Fearful recall decline. We identified a scoring inconsistency in Figure 5(b).

(1) Evaluation Correction.

Figure 5(b) used mean token log-probabilities instead of the summed token log-probabilities including EOS specified in Eq. (18) and used in Tables 3–5 and EAOG training. We recomputed the class-wise metrics accordingly. No retraining is needed, and the main results remain unchanged.

 

(2) Corrected Results.

As shown in Table 7, P-Ref increases Fearful recall from 39.52% to 46.21% (+6.69 pp), rather than decreasing it by 12.6 points. EAOG achieves 36.85% recall, only 2.67 points below P-Init, instead of the previously reported 20.6-point decline.

Table 7. Corrected class-wise results on 3,941 episodes. Entries report Precision / Recall / F1 (%), using the summed token-log-probability protocol. Fixed 34s and EAOG share the navigator and recognizer.

EmotionP-InitP-RefFixed 34sEAOG
Angry32.46 / 10.04 / 15.3446.00 / 32.02 / 37.7639.25 / 21.30 / 27.6233.11 / 27.00 / 29.75
Fearful23.52 / 39.52 / 29.4929.45 / 46.21 / 35.9728.55 / 30.76 / 29.6125.51 / 36.85 / 30.15
Happy38.65 / 30.70 / 34.2262.48 / 48.02 / 54.3147.05 / 50.93 / 48.9145.97 / 45.12 / 45.54
Neutral25.87 / 56.36 / 35.4648.88 / 62.73 / 54.9438.10 / 62.58 / 47.3648.45 / 56.67 / 52.23
Sad78.52 / 35.81 / 49.1890.63 / 77.45 / 83.5275.16 / 59.55 / 66.4581.65 / 60.73 / 69.65
Macro-F132.7453.3043.9945.47

 

(3) Viewpoint vs. Stopping Effects.

Following your suggestion, we compare EAOG with Fixed 34s under identical navigation trajectories and nearly matched observation times (33.72 vs. 34.00s).

EAOG improves Fearful recall from 30.76% to 36.85% (+6.09 pp), while precision decreases from 28.55% to 25.51%. Fearful F1 increases from 29.61% to 30.15% (+0.54 pp). Thus, adaptive stopping partially mitigates the recall decline under fixed-time stopping, with a remaining precision–recall trade-off. We will correct Figure 5(b) and its analysis in the revision.

AuthorsOpenReview ↗

Follow-up on Our Response

Thank you again for your initial assessment. We sincerely hope you could take another look at the additional experiments and analyses provided in our rebuttal.

We have made substantial efforts to strengthen the paper and clarify its contributions. We believe AffectSim offers a valuable research setting and benchmark for the community, extending affective computing beyond passive recognition toward active affective perception. We would greatly appreciate it if you could reconsider the overall contribution of the work in light of the additional evidence.

Thank you again for your time and consideration.

Comments on the paper

2 comments
Area_Chair_28P7OpenReview ↗

Reminder

Dear Reviewers,

Thank you for reviewing this paper. If you haven't done so, please respond to the rebuttal as discussion deadline is approaching.

Best, AC

AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

Thank you for your thoughtful and constructive feedback. Your suggestions have helped us further strengthen AffectSim through additional experiments, more thorough validation, and clearer discussions of its scope and limitations. We sincerely appreciate this opportunity to improve our work.

We would like to highlight three contributions that we believe are particularly valuable to the research community.

(1) AffectSim establishes a new research setting for active affective perception.

AffectSim moves beyond passive emotion recognition to investigate how embodied agents actively acquire affective evidence. Its controlled-replay design separates underlying behaviors from observation conditions, enabling systematic, repeatable studies of observation-dependent emotion recognition.

(2) AffectSim provides a substantial, reusable benchmark and systematic experimental evidence.

With 27,647 episodes across 57 scenes and evaluations involving 24 frozen recognizer configurations, AffectSim reveals the importance of observation conditions for emotion recognition. Our additional stopping comparisons and RGB-derived pose-based experiments further strengthen the evidence for active affective perception.

(3) Emotion-label reliability was carefully considered and extensively evaluated.

Label reliability was considered from the outset through established emotion-expression protocols, motion-quality screening, and human validation. Source labels represent performer-intended emotions, while human perceptual labels reflect what observers perceive; both serve complementary research purposes. Our rebuttal adds cross-representation, composed-episode, and dyadic validation, alongside transparent handling of perceptual disagreements.

We believe AffectSim provides a valuable experimental foundation for advancing embodied affective intelligence beyond conventional passive recognition.

We sincerely appreciate your time and thoughtful consideration. We welcome any further questions or concerns during the remaining discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.