- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AIRIS, a benchmark for evaluating whether VLMs can behave as socially competent agents from a first-person video stream. Instead of testing social perception, intent inference, action selection, and norm compliance as separate tasks, the authors chain them within the same episode (the A-I-R-I stages), so that a wrong final response can be traced back to the stage where it went wrong. The benchmark is built from egocentric web videos annotated through a VLM-assisted, human-adjudicated pipeline, and is used to evaluate a large pool of general-purpose, embodied, streaming, and omnimodal models. The clips come from everyday settings such as shops, markets, service counters, and casual encounters. The questions range from noticing where a person is looking, to deciding what the camera wearer should do next, to judging afterward whether the exchange ended well and politely. The most notable aspect is the evaluation protocol. Models are questioned at specific moments in each video and can only see what has happened up to that moment. If there is not yet enough evidence to tell what the person wants, the correct answer is to wait or not act. This moves the question from whether a model recognizes a social cue to whether it knows when it has enough evidence to act. The proposed metrics follow the same idea, penalizing premature commitment and collapsed policies rather than rewarding raw accuracy. The paper's central claim follows from this framing: models that perceive social cues well are still poorly calibrated over time and rarely produce a complete and correct reaction. The authors also report that letting models reason step by step does not help, and often hurts on questions where the right choice is to hold back.
Critical strengths
- A time-based protocol that rewards knowing when to act.
- Questions are asked at set moments in the video, and the model only sees what has happened up to that moment. Before the person's intent is clear, the correct answer is "not enough evidence". In the Reaction stage, waiting or not stepping in can also be the correct answer.
- Most video and social benchmarks do not test whether a model knows when it has enough evidence to act, and AIRIS does. For an agent that shares space with people, acting too early is as much a failure as misreading the scene.
- The task is far from solved. The best model achieves an Intent Anticipation Score of only 0.237, and it still guesses on 42% of questions before the intent is clear.
- I see this as the paper's main contribution and the main reason the benchmark could be useful to the community.
- Linking all stages in one episode makes it possible to find where a model fails.
- All four stages are asked about the same exchange, so a wrong reaction can be traced back to its cause, whether that is a missed cue, a wrong reading of intent, or a wrong choice of action.
- The 36 capabilities split skills that are usually scored together. For example, naming what a person is looking at is scored separately from finding it in the frame, and many models do much better at naming than at finding.
- Reaction is scored at four levels, namely whether to act, which type of action, the specific action, and what to say. Scoring these separately lets the benchmark point to the exact decision a model gets wrong, instead of only marking the final response as right or wrong.
- Metrics that penalize simple shortcuts, extensive experiments, and clear documentation.
- The metrics are designed to catch models that take the easy way out. The Intent Anticipation Score gives no credit to a model that always says "not enough evidence" and the engagement score gives a low score to a model that almost always chooses "refrain".
- The paper provides extensive experiments on 45 models from five model families, along with the full capability list and the main annotation prompts. This makes both the benchmark and its results easier to check and build on.
Critical weaknesses
-
The labels are not shown to be reliable, which is the main evidence a benchmark paper needs.
- The headline results rest on the benchmark's most subjective labels. These are the moments when intent becomes clear (T1), the Reaction labels that describe what Ego should do rather than what Ego actually did, and the norm-compliance judgments.
- The paper does not report agreement between annotators for any of these labels, how often the expert panel changed the proposed labels, or how well humans perform on the benchmark. Without this, we cannot tell whether low scores on intent anticipation, engagement or Joint-EM come from the limits of the models or from noisy labels.
- This should be fixable. Two annotators and a VLM proposed labels independently for the Intention, Reaction, and Interaction stages, so the agreement data should be presented and discussed.
-
Key metrics have no reference baselines, and one headline result is mostly explained by a trivial answer pattern.
- The appendix shows that a model which always answers "refrain" gets a Joint-EM of 88/611, which is about 0.144. Eleven models score exactly this, and the best model reaches only 0.147.
- So the main-text claim that no model exceeds 0.147 Joint-EM says more about this trivial baseline than about the ability of models to put together a full reaction. Models that actually try to respond score below it, such as Gemini-3.7-Flash at 0.083. This explanation appears only in the appendix and should be in the main text.
- The paper reports no random-chance, majority-answer, or blind baselines, such as answering from the question alone or from the transcript alone. Without them, it is hard to judge what a score means, especially for the compliance questions and when comparing stages that use different question formats. The paper itself notes that most episodes contain no violation, and many models score about 0.90 on Compliance.
-
Section 3.2 says the advantage of general-purpose models disappears in Reaction and Interaction and that embodied models take the lead. This holds for the single best Reaction model, HY-Embodied-VLM-1.0, but not more broadly. In Table 2, closed models score higher on average than embodied models in both stages (Reaction and Interaction), and the two Gemini models share the best Interaction score.
-
Omni-modal models receive raw audio while the other models receive speech transcripts, so the comparison between model types is not like-for-like.
-
No confidence intervals are reported, even though the benchmark has only 357 clips and many questions come from the same clip. Some conclusions rest on very small differences. For example, Qwen3-VL-32B is said to lead Compliance with 0.923 against 0.918 for RynnBrain1.1-9B.
-
There are some inconsistencies. Gemini-3.7-Flash's Joint-EM is 0.083 in Tables 2 and 9 but 0.084 in the appendix text, and the key-moment definition in A.3.2 describes T2 as the partner's response rather than Ego's.
Suggestions:
- Add confidence intervals computed by resampling clips rather than individual questions.
- Move the explanation of the 0.144 Joint-EM baseline into the main text.
Questions for authors
- Section 2.2 says a VLM and two annotators separately proposed the Intention, Reaction and Interaction labels before an expert panel decided the final labels. What was the agreement between the two annotators, and between the annotators and the VLM, for T1, intent, Reaction and compliance labels? How often did the panel change the proposed labels, and how many questions were revised or removed in the final question review?
- Which proprietary VLM was used for screening, segmentation and pre-annotation? Is it from the same family as any evaluated model, such as Gemini, GPT or Doubao? Did the expert panel see the VLM's proposals when deciding the final labels?
- Can you report how humans perform on a subset of the benchmark, at least for the Intention and Reaction questions?
- The ethics statement says clips that cannot be redistributed will be shared only as annotations and source metadata. What fraction of the 357 clips can be released as video, and how can others run the benchmark on the rest?
- No ablation presented on random-chance, majority-answer, and blind baselines (question and video only, question and transcript only) for each stage, and Joint-EM computed only on the 523 active events? Does the Joint-EM claim in the main text still hold with these?
- In Intent Anticipation Score.
- How is "stable correctness" defined?
- How is the score computed when T2 is null, which Prompt 3 allows, or when no query falls between T1 and T2?
- How many queries are asked per event around T1?
- Each question gives the model about 11.5 frames on average, while clips are about 110 seconds long on average. The annotators set T1 by watching the full video, and the visual cue that made the intent clear may last only a moment, such as a glance or a gesture. If that moment is not among the sampled frames, the model never sees it. A model that then answers "not enough evidence" after T1 is being reasonable, but it is scored as wrong. How do you pick which frames the model sees from the video up to the question time? For questions asked right after T1, how often do these frames include the moment that made the intent clear?
- Evaluation setup.
- Were the streaming models run in their own streaming mode, or given the same sampled frames as the other models?
- Which reasoning setting (thinking or no thinking) was used for each model in Table 2, including the proprietary models?
- In clips with several speakers, such as Figures 42 and 44, do the transcripts given to models mark which speaker is Ego?
- Section 2.4 says clip-macro metrics are reported. Where can they be found?