- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AffectSim is a simulation benchmark for active body-based emotion recognition. It separates the underlying motion from observation conditions, allowing matched comparisons across viewpoints. The paper also introduces EAOG, a learned gate that decides when to stop observing.
Critical strengths
I like the controlled-replay design. Observing the same behavior under different conditions gives the simulator a clear purpose beyond producing more data. Keeping variants of the same performance within one split is also a sensible choice.
The paper is clear and includes useful controlled comparisons. With the same adapted recognizer, learned active observation improves Macro-F1 from 32.74% to 45.47% over the initial view. Fixing the navigator in the stopping experiments also helps separate the benefit of movement from the benefit of observation timing.
Critical weaknesses
The emotion-specific insight is not yet clear. The navigation teacher mainly encourages visibility, framing, and a suitable distance. Emotion-specific supervision enters through the observation gate. This leaves me wondering how much the benchmark requires beyond obtaining a clearer body view. With the navigator fixed, EAOG improves Macro-F1 by only 1.20 points over random stopping. A simple visibility-based gate at matched observation cost would help establish what the affect-specific component adds.
The scope also limits the broader motivation. The benchmark uses five categories and looping, predefined body motions. People do not respond to the robot, and facial and speech cues are left for future work. These choices support controlled perception experiments, but do not establish benefits for natural interaction or collaboration. The authors acknowledge these limitations. Given the focus on body motion, a pose-estimation-plus-emotion-classification baseline using the same RGB clips would be particularly useful. It would test whether the observation gains carry over to a motion-based representation.
Label reliability and dataset coverage need a clearer account. Human majority votes match the source labels on 78% of the 500 checked motions. This is useful validation, but the remaining disagreement deserves analysis. Is it caused by ambiguous performances, source annotations, or retargeting? It is also unclear whether uncertain samples are retained, relabeled, or flagged.
Table B reports actor counts for the action-grounded subset, but not for the other two components. A source-by-source breakdown of distinct performers, original performances, and split overlap would make the dataset’s diversity easier to judge. The paper also describes manual episode screening without clearly quantifying its coverage. The large episode count should not be taken as equivalent to independent human performances.
Questions for authors
- How are disagreements between human judgments and source labels handled? Can you report reviewer agreement and compare judgments before and after retargeting?
- How many distinct performers and original performances come from each source? Are performers disjoint across splits in every component, and how many episodes received manual review?
- How does EAOG compare with a visibility-based gate and a validation-tuned fixed stopping time at similar observation cost?
- Are pose-based recognition results available using poses estimated from the same RGB clips, with matched fixed-view and active-view budgets?
2 comments
Response to Reviewer eagz (Part 2/2)
R3-W3 & Q1. Emotion-Label Reliability and Disagreement Handling
Thank you for raising label reliability after retargeting and disagreement handling. We conducted blind evaluations across representations and separately reassessed disputed cases.
(1) Human Agreement Across Representations.
We sampled 500 source motions (320 KDAE, 162 Emilya, 18 dyadic) and presented each as an original skeleton (S), retargeted skeleton (T), and final RGB animation (A). Different groups of three independent blind raters evaluated S, T, and A (Table 3).
For pairs with valid majority-vote labels in both representations, consistency is 88.53% (386/436) for S-T. 83.93% (376/448) for T-A, and 85.29% (377/442) for S-A. This indicates that perceived emotion is largely preserved across stages.
Table 3. Blind recognition across representations. Macro-F1 measures majority-vote agreement with source labels; Fleiss' κ measures within-representation inter-rater agreement.
| Representation | Macro-F1(%) | Fleiss'κ |
| Original Skeleton (S) | 76.39 | 0.665 |
| Retargeted Skeleton (T) | 74.24 | 0.679 |
| Final RGB Animation (A) | 79.86 | 0.734 |
(2) Disagreement Tracing and Label Reassessment.
Of the 500 final RGB animations, 111 do not match their source labels by majority vote: 85 receive a different emotion and 26 have no clear majority. Of these, 81/111 already disagree or remain ambiguous at S, including 32 assigned the same non-source emotion across S, T, and A. The other 30 first differ at T (6) or A (24). This suggests that many disagreements originate in source-motion ambiguity, though different rater groups prevent definitive causal attribution.
In a label-informed reassessment of all 111 cases, three raters independently judged source-label consistency with the final RGB animation.** 102/111 (91.89%) received majority acceptance, including 77/85 previously assigned another emotion and 25/26 previously lacking a majority; 72/111 were unanimously accepted. Individual acceptance was 278/333 (83.48%), supporting the perceptual plausibility of source labels for most disputed cases.
(3) Label Handling and Auditability.
We retain original source labels, rather than automatically relabeling or removing disagreements, to preserve a consistent benchmark target. To enable uncertainty analysis, we will release human judgments and audit flags for all 500 evaluated motions, identifying agreement, disagreement, and ambiguity across S/T/A.
R3-W4 & Q2. Dataset Coverage and Manual Screening
Thank you for asking about independent source performances and screening coverage. We distinguish source performances from episodes and report source-level actor/split statistics and human review coverage.
(1) Dataset Diversity and Split Integrity.
AffectSim contains 1,307 source motions or synchronized dyadic sequences, generating 27,647 episodes. Table 4 reports source counts, distinct actors, and their overlap across training (Tr), validation (Val), and test (Te).
Emilya is actor-disjoint, with 278/76/68 source motions in Tr/Val/Te; KDAE (669/84/84) and Dyadic Pilot (34/7/7) are source-motion-disjoint but not actor-disjoint. Across all sources, each source motion and all its episodes stay in one split, with zero source-motion overlap.
Table 4. Source-level dataset diversity and cross-split actor overlap. Overlap counts actors shared by each split pair; episode counts do not represent independent performances.
| Source | Actors | Source Motions | Episodes | Actor Overlap(Tr–Val/Tr–Te/Val–Te) |
| Emilya | 12 | 422 | 17,901 | 0 / 0 / 0 |
| KDAE | 22 | 837 | 8,306 | 21 / 21 / 20 |
| Dyadic Pilot | 2 | 48 | 1,440 | 2 / 2 / 2 |
| Total | — | 1,307 | 27,647 | — |
(2) Quality Control and Manual Screening Coverage.
We distinguish episode-level human screening from source-motion-level checks. For Emilya, scene-dependent actions require valid body–scene geometry and object contacts. After configuration refinement and emotion filtering, 52,573 candidate episodes remained; 28,765 underwent individual manual review, assessing interaction plausibility, object contacts, and rendered motions. All 17,901 retained Emilya episodes passed individual human screening (Table 5).
Table 5. Emilya episode-level manual screening by action type. All retained episodes were individually reviewed.
| Action Type | Manually Reviewed | Retained |
| BS (Being Seated) | 11,900 | 7,684 |
| KD (Knocking) | 4,700 | 1,791 |
| SD (Sitting Down) | 4,165 | 1,076 |
| SW (Walking) | 8,000 | 7,350 |
| Total | 28,765 | 17,901 |
For KDAE, all 8,306 episodes were manually checked. For Dyadic Pilot, 48 of 60 captured interaction sequences were retained and instantiated into 1,440 episodes.
Appendix B will clarify review levels and actor/split statistics, distinguishing episodes from independent performances.
Response to Reviewer eagz (Part 1/2)
Thank you for your constructive review. We appreciate your recognition of the controlled-replay design, and useful controlled comparisons that separate movement from observation timing. Following your suggestions, we conducted additional stopping comparisons, pose-based evaluations, and detailed label and dataset analyses.
(1) Visibility-Based and Budget-Matched Comparisons.
We agree that the navigation teacher mainly improves person framing, while EAOG introduces affect-conditioned evidence assessment through stopping. To isolate this contribution, all methods use the same frozen navigator, trajectories, candidate windows, and recognizer; only stopping differs.
The Visibility-and-Scale gate uses YOLO11x-Pose keypoint visibility and target bounding-box area, averaged over 32 frames, with thresholds tuned on validation data. Fixed 34s matches EAOG's validation observation budget; Fixed 45s is the best validation-selected fixed time. We also sample stopping times from EAOG's validation-set stopping-time distribution (five seeds), matching its marginal time distribution.
EAOG outperforms Visibility-and-Scale by 3.22 pp (95% CI [0.16, 5.91]) and Fixed 34s by 1.48 pp ([0.14, 3.49]). Against distribution-matched random stopping, the gain is 1.23 pp ([0.26, 2.39]). Compared with Fixed 45s, EAOG uses 25.1% less time (Table 1).
Table 1. Stopping strategies with identical navigation and recognizer. F1 is evaluated on 3,941 episodes; fixed times/thresholds are validation-selected. Random results are mean ± SD over five seeds. CIs use paired source-group bootstrap comparisons.
(2) Affect-Specific Supervision and Scope.
Removing EAOG's auxiliary emotion loss reduces Macro-F1 from 45.47% to 42.65% (−2.82 pp). Together, these comparisons show a benefit beyond the tested geometric-visibility heuristic.
Thank you for suggesting a pose-based baseline. We agree that predefined body motions and the absence of facial, vocal, or reactive behaviors limit claims about natural human–robot interaction.
(1) Matched RGB-Based Protocol.
We used YOLO11x-Pose to estimate skeletons from the exact RGB clips used in the original evaluation, followed by CTR-GCN for emotion recognition. Within each P-Init or A-Obs condition, the pose-based and original recognizers receive identical clips, trajectories, temporal windows, and stopping times. This matches inputs and budgets across recognizers within each condition, not between P-Init and A-Obs. CTR-GCN is trained on P-Ref and evaluated unchanged; pose-estimation failures remain in the evaluation.
(2) Results and Limitations.
Active observation improves pose-based Macro-F1 on Emilya (+5.12 pp) and KDAE (+5.81 pp), with +4.15 pp overall (Table 2). Valid pose estimation rises from 55.77% (P-Init) to 79.24% (A-Obs). The Dyadic Pilot declines by 5.62 pp; our current estimator cannot reliably associate detected poses with the designated target actor. Thus, the result supports active observation for single-person RGB-derived motion recognition, not consistent gains in dyadic interaction.
Table 2. Pose-based Macro-F1 (%) from the same RGB observations. CTR-GCN is trained on P-Ref and held fixed; failed pose estimates remain in the evaluation.