acceptodds
Withdrawn from ICLR 2027

AIRIS: Benchmarking Social Embodied Intelligence in Egocentric Streaming Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied agents operating around people must not only recognize social cues, but also decide when the available evidence is sufficient, whether to intervene, and how to respond without violating social norms. Existing benchmarks typically evaluate these abilities in isolation or with access to completed videos, obscuring where an online social decision fails. We introduce AIRIS, a causal streaming benchmark that formulates social embodied intelligence as a time-anchored chain of Attention, Intention, Reaction, and Interaction (A-I-R-I). AIRIS contains 8,000 expert-curated questions grounded in 357 egocentric clips across 11 context categories and 36 capability dimensions. At each query, models receive only the observation prefix available at that time; decision-aware metrics reward calibrated waiting and complete reactions while penalizing premature commitment. Evaluating a pool of 45 proprietary and open-source general-purpose, embodied, streaming-video, and omni-modal models reveals fragmented and temporally miscalibrated competence. General-purpose VLMs perform strongly in cue grounding and intent recognition, whereas embodied specialists are more competitive on aggregate Reaction; nevertheless, the best Intent Anticipation Score is only 0.237, category-level False Commitment Rates range from 0.26 to 0.51, and no model exceeds 0.147 Reaction Joint-EM. These findings characterize the perception–execution gap as a failure of temporal calibration and reaction composition. Ultimately, reliable social embodied intelligence requires models to actively calibrate anticipation to observable evidence, composing perception, inference, and action into seamless responses that align with human social norms.

Primary area:
datasets and benchmarks
Keywords:
Social Interaction, Ego-centric, VLM Evaluation, Human-robot Interaction, Social Embodied Intelligence, Perception-Execution Gap

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dmKeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AIRIS, a benchmark for evaluating whether VLMs can behave as socially competent agents from a first-person video stream. Instead of testing social perception, intent inference, action selection, and norm compliance as separate tasks, the authors chain them within the same episode (the A-I-R-I stages), so that a wrong final response can be traced back to the stage where it went wrong. The benchmark is built from egocentric web videos annotated through a VLM-assisted, human-adjudicated pipeline, and is used to evaluate a large pool of general-purpose, embodied, streaming, and omnimodal models. The clips come from everyday settings such as shops, markets, service counters, and casual encounters. The questions range from noticing where a person is looking, to deciding what the camera wearer should do next, to judging afterward whether the exchange ended well and politely. The most notable aspect is the evaluation protocol. Models are questioned at specific moments in each video and can only see what has happened up to that moment. If there is not yet enough evidence to tell what the person wants, the correct answer is to wait or not act. This moves the question from whether a model recognizes a social cue to whether it knows when it has enough evidence to act. The proposed metrics follow the same idea, penalizing premature commitment and collapsed policies rather than rewarding raw accuracy. The paper's central claim follows from this framing: models that perceive social cues well are still poorly calibrated over time and rarely produce a complete and correct reaction. The authors also report that letting models reason step by step does not help, and often hurts on questions where the right choice is to hold back.

Critical strengths
  1. A time-based protocol that rewards knowing when to act.
    • Questions are asked at set moments in the video, and the model only sees what has happened up to that moment. Before the person's intent is clear, the correct answer is "not enough evidence". In the Reaction stage, waiting or not stepping in can also be the correct answer.
    • Most video and social benchmarks do not test whether a model knows when it has enough evidence to act, and AIRIS does. For an agent that shares space with people, acting too early is as much a failure as misreading the scene.
    • The task is far from solved. The best model achieves an Intent Anticipation Score of only 0.237, and it still guesses on 42% of questions before the intent is clear.
    • I see this as the paper's main contribution and the main reason the benchmark could be useful to the community.
  2. Linking all stages in one episode makes it possible to find where a model fails.
    • All four stages are asked about the same exchange, so a wrong reaction can be traced back to its cause, whether that is a missed cue, a wrong reading of intent, or a wrong choice of action.
    • The 36 capabilities split skills that are usually scored together. For example, naming what a person is looking at is scored separately from finding it in the frame, and many models do much better at naming than at finding.
    • Reaction is scored at four levels, namely whether to act, which type of action, the specific action, and what to say. Scoring these separately lets the benchmark point to the exact decision a model gets wrong, instead of only marking the final response as right or wrong.
  3. Metrics that penalize simple shortcuts, extensive experiments, and clear documentation.
    • The metrics are designed to catch models that take the easy way out. The Intent Anticipation Score gives no credit to a model that always says "not enough evidence" and the engagement score gives a low score to a model that almost always chooses "refrain".
    • The paper provides extensive experiments on 45 models from five model families, along with the full capability list and the main annotation prompts. This makes both the benchmark and its results easier to check and build on.
Critical weaknesses
  1. The labels are not shown to be reliable, which is the main evidence a benchmark paper needs.

    • The headline results rest on the benchmark's most subjective labels. These are the moments when intent becomes clear (T1), the Reaction labels that describe what Ego should do rather than what Ego actually did, and the norm-compliance judgments.
    • The paper does not report agreement between annotators for any of these labels, how often the expert panel changed the proposed labels, or how well humans perform on the benchmark. Without this, we cannot tell whether low scores on intent anticipation, engagement or Joint-EM come from the limits of the models or from noisy labels.
    • This should be fixable. Two annotators and a VLM proposed labels independently for the Intention, Reaction, and Interaction stages, so the agreement data should be presented and discussed.
  2. Key metrics have no reference baselines, and one headline result is mostly explained by a trivial answer pattern.

    • The appendix shows that a model which always answers "refrain" gets a Joint-EM of 88/611, which is about 0.144. Eleven models score exactly this, and the best model reaches only 0.147.
    • So the main-text claim that no model exceeds 0.147 Joint-EM says more about this trivial baseline than about the ability of models to put together a full reaction. Models that actually try to respond score below it, such as Gemini-3.7-Flash at 0.083. This explanation appears only in the appendix and should be in the main text.
    • The paper reports no random-chance, majority-answer, or blind baselines, such as answering from the question alone or from the transcript alone. Without them, it is hard to judge what a score means, especially for the compliance questions and when comparing stages that use different question formats. The paper itself notes that most episodes contain no violation, and many models score about 0.90 on Compliance.
  3. Section 3.2 says the advantage of general-purpose models disappears in Reaction and Interaction and that embodied models take the lead. This holds for the single best Reaction model, HY-Embodied-VLM-1.0, but not more broadly. In Table 2, closed models score higher on average than embodied models in both stages (Reaction and Interaction), and the two Gemini models share the best Interaction score.

  4. Omni-modal models receive raw audio while the other models receive speech transcripts, so the comparison between model types is not like-for-like.

  5. No confidence intervals are reported, even though the benchmark has only 357 clips and many questions come from the same clip. Some conclusions rest on very small differences. For example, Qwen3-VL-32B is said to lead Compliance with 0.923 against 0.918 for RynnBrain1.1-9B.

  6. There are some inconsistencies. Gemini-3.7-Flash's Joint-EM is 0.083 in Tables 2 and 9 but 0.084 in the appendix text, and the key-moment definition in A.3.2 describes T2 as the partner's response rather than Ego's.

Suggestions:

  • Add confidence intervals computed by resampling clips rather than individual questions.
  • Move the explanation of the 0.144 Joint-EM baseline into the main text.
Questions for authors
  1. Section 2.2 says a VLM and two annotators separately proposed the Intention, Reaction and Interaction labels before an expert panel decided the final labels. What was the agreement between the two annotators, and between the annotators and the VLM, for T1, intent, Reaction and compliance labels? How often did the panel change the proposed labels, and how many questions were revised or removed in the final question review?
  2. Which proprietary VLM was used for screening, segmentation and pre-annotation? Is it from the same family as any evaluated model, such as Gemini, GPT or Doubao? Did the expert panel see the VLM's proposals when deciding the final labels?
  3. Can you report how humans perform on a subset of the benchmark, at least for the Intention and Reaction questions?
  4. The ethics statement says clips that cannot be redistributed will be shared only as annotations and source metadata. What fraction of the 357 clips can be released as video, and how can others run the benchmark on the rest?
  5. No ablation presented on random-chance, majority-answer, and blind baselines (question and video only, question and transcript only) for each stage, and Joint-EM computed only on the 523 active events? Does the Joint-EM claim in the main text still hold with these?
  6. In Intent Anticipation Score.
    • How is "stable correctness" defined?
    • How is the score computed when T2 is null, which Prompt 3 allows, or when no query falls between T1 and T2?
    • How many queries are asked per event around T1?
  7. Each question gives the model about 11.5 frames on average, while clips are about 110 seconds long on average. The annotators set T1 by watching the full video, and the visual cue that made the intent clear may last only a moment, such as a glance or a gesture. If that moment is not among the sampled frames, the model never sees it. A model that then answers "not enough evidence" after T1 is being reasonable, but it is scored as wrong. How do you pick which frames the model sees from the video up to the question time? For questions asked right after T1, how often do these frames include the moment that made the intent clear?
  8. Evaluation setup.
    • Were the streaming models run in their own streaming mode, or given the same sampled frames as the other models?
    • Which reasoning setting (thinking or no thinking) was used for each model in Table 2, including the proprietary models?
    • In clips with several speakers, such as Figures 42 and 44, do the transcripts given to models mark which speaker is Ego?
  9. Section 2.4 says clip-macro metrics are reported. Where can they be found?
Reviewer_mDGNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AIRIS, a benchmark for evaluating social embodied intelligence in egocentric streaming videos. The authors organize social interaction understanding and decision-making into four stages: Attention, Intention, Reaction, and Interaction (A-I-R-I). The benchmark uses four temporal anchors and a causal-prefix evaluation protocol to assess social cue recognition, intention inference, response selection, and interaction outcomes under temporally restricted observations.

AIRIS contains 8,000 questions derived from 357 egocentric video clips, covering 11 social contexts and 36 capability dimensions. The dataset is constructed through model-assisted annotation, human verification, and expert review. The authors evaluate 45 multimodal models across general-purpose, embodied, streaming-video, and omni-modal categories.

The experiments suggest that current models perform relatively well on social cue perception but struggle with timely intention prediction and complete response selection. The authors identify temporal calibration and response composition as important limitations of current multimodal models.

Critical strengths

S1. Relevant research problem and comprehensive evaluation coverage.

The paper addresses a meaningful problem: evaluating whether multimodal models can understand evolving social situations and select appropriate responses under partial observations. The A-I-R-I framework provides a clear organization of social perception, intention inference, response decisions, and interaction assessment. Including justified waiting and non-intervention makes the evaluation relevant to practical human-agent interaction.

S2. Substantial benchmark construction and experimental effort.

The authors have invested considerable effort in dataset collection, annotation, and evaluation. AIRIS includes 8,000 questions across 357 egocentric clips and 36 capability dimensions, supported by a multi-stage annotation and human-review process. Evaluating 45 models across different architectural families provides broad empirical coverage. The detailed quantitative results and qualitative examples also make the benchmark useful for examining model capabilities and failure patterns.

Critical weaknesses

W1. Limited demonstrated novelty over existing benchmarks and insufficient comparison with the most closely related work.

My main concern is that the proposed benchmark largely integrates evaluation dimensions that have already been explored in previous studies. Online video understanding, intention anticipation, proactive response timing, and socially appropriate embodied decisions have all been investigated in related benchmarks.

The paper discusses several relevant works, including OVO-Bench, ESTP-Bench, EgoNormia, and RobotEQ. However, the comparison in Table 1 primarily emphasizes differences in task coverage and does not sufficiently establish the substantive novelty of the proposed evaluation framework. More importantly, EgoSocial (Wang et al., 2025), which specifically evaluates proactive intervention in egocentric social interactions, is missing from the related-work discussion despite its close relevance.

Although combining these capabilities into a unified benchmark has practical value, the paper does not convincingly demonstrate that AIRIS introduces substantially new research questions or evaluation capabilities beyond existing benchmarks. The contribution therefore appears closer to a comprehensive integration and extension of established evaluation tasks.

The authors should provide a more focused comparison with the most closely related benchmarks, particularly EgoSocial and ESTP-Bench, and clarify which capabilities, evaluation settings, or failure modes AIRIS uniquely captures.

W2. Insufficient evidence supporting the proposed causal framework and the interpretation of its key results.

The paper presents A-I-R-I as an interdependent causal chain connecting social perception, intention inference, reaction decisions, and interaction outcomes. However, the experiments primarily evaluate these stages through separately scored question-answering tasks. The results therefore demonstrate differences in model performance across task categories without directly establishing how upstream errors affect downstream decisions.

Consequently, the claimed perception-to-execution gap is not fully validated as a causal phenomenon. Moreover, the Reaction stage evaluates the selection of predefined actions rather than their execution in an interactive environment, which further limits the interpretation of this claim.

The reported Reaction Joint-EM results also raise questions about the informativeness of the metric. As discussed in Appendix A.4.2, a constant non-intervention policy can achieve approximately 0.144 Joint-EM, whereas the best-performing model reaches only 0.147. Although the authors acknowledge this issue and report engagement Macro-F1, additional analysis is needed to establish whether the proposed metric meaningfully distinguishes successful active-response decisions.

Finally, the temporal decidability anchor T1 and the normative Reaction labels involve potentially subjective human judgments. The paper describes expert review but does not provide sufficient quantitative evidence regarding annotation agreement or sensitivity to temporal ambiguity.

These limitations suggest that the empirical results do not yet fully substantiate the central methodological claims. The authors could strengthen the paper with episode-level cross-stage error analysis, active-response-only evaluation, and quantitative annotation reliability measurements.

Questions for authors

Q1. Novelty relative to existing benchmarks. What evaluation capabilities does AIRIS provide that cannot be adequately assessed by existing benchmarks such as EgoSocial and ESTP-Bench? In particular, how does the proposed A-I-R-I framework differ from integrating previously established social understanding and proactive response tasks?

Q2. Validation of the A-I-R-I framework. Can the authors provide episode-level evidence that errors in Attention or Intention are associated with downstream Reaction failures? Such an analysis would help establish whether the proposed framework captures meaningful dependencies between stages.

Q3. Interpretation of Reaction Joint-EM. Given that a constant non-intervention baseline achieves approximately 0.144 Joint-EM, close to the best reported score of 0.147, can the authors report results on active-response events separately? How do the model rankings change under this evaluation?

Q4. Annotation reliability. How consistent are human annotators when determining the intention-decidability timestamp T1 and selecting socially appropriate reactions? Can the authors provide agreement statistics and examine the sensitivity of IAS to temporal annotation uncertainty?

Reviewer_kKAGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a streaming benchmark, consisting of 8,000 multiple-choice questions and 357 web-sourced egocentric clips, that scores a model at timestamps along an interaction using only the video prefix. It establishes social embodied intelligence as a causal chain: Attention (locate the interacting person and their cues), Intention (infer their goal and whether it is yet inferable), Reaction (decide what the camera-wearer (“Ego”) should do), Interaction (judge the exchange afterwards). The paper evaluates 45 models, finding that models perceive cues well but commit to intents too early and cannot produce a complete, correct reaction.

Critical strengths
  1. The paper fills a gap earlier benchmarks lack, specifically that they score “insufficient evidence” as the correct answer before intent is decidable, and penalize premature commitment (FCR).
  2. The paper's evaluation is broad, with 45 models evaluated and per-stage breakdowns in the appendix.
Critical weaknesses
  1. The benchmark does not measure what the paper claims. The ground truth labels cannot be checked, since Reaction (“what Ego should do”) and Interaction (norm violations, “mutual understanding”) are normative judgments. The manuscript does not indicate how they are operationalized, and no inter-annotator agreement is reported for any stage. Moreover, the headline Reaction result is a metric artifact, as always answering “refrain” scores Joint-EM 0.144 (§A.4.2) whereas the best model scores 0.147 and every model that attempts actions scores lower. The claim that “No model closes the full reaction decision chain” is therefore not established. Finally, some simple baselines are missing, that would better delineate the performance of the evaluated models. Indeed, no video-blind, chance, or majority baseline is reported, although ASR transcripts are supplied and the query time is visible in the prompt, so it is unknown how much of Intention and Reaction is solvable without vision. The protocol does not match the claims, given that questions are independent truncated clips, so streaming models are never evaluated as streams, and the architecture conclusion in §3.2 does not follow. Reaction queries at τ ≥ T2 include Ego’s observed response in the prefix. Ego is a human and no action is ever executed, so “perception-execution gap” is a misnomer.

  2. The paper is hard to verify and hard to follow. Core terms are used before they are defined (“Ego” as the camera wearer treated as the robot, “grounding” as localizing a cue’s target, “interaction partner,” and the norm/compliance labels), Attention is defined as "interaction", i.e., the partner it is supposed to identify, the introduction does not state the research gap concretely or introduce it well, and figures are crowded and difficult to follow. In terms of reproducibility, clips are scraped web video, release is conditional on an unspecified rights/privacy review, and the appendix transcripts contain personal names (Figs. 40, 43).

Questions for authors
  1. Define, in one sentence each, what makes a Reaction or compliance answer correct, and give agreement statistics between annotators.
  2. Which VLM produced the pre-annotations, and is it in the evaluated pool?
  3. Perform Joint-EM on the 523 active events only. Which model leads?
  4. What fraction of clips will be redistributed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.