- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Egocentric Action Selection (EAS): given a single image of a multi-person scene in which the camera wearer’s body is not visible, the model must choose the action that the viewer standing at the camera position could take right now. To make the task diagnostic, distractors are built from the actions of other people in the scene whose roles or objects can only be held by one person (the “singleton” constraint), plus an “off-vantage” action that cannot be performed from the viewer’s position. EgoGapBench contains 1,000 questions on 440 COCO images, with a 107-question Wikimedia set and a 168-question training split. All evaluated MLLMs fall well below humans (66.1% for the best configuration vs. 94.5%), most errors go to other people’s actions, and fine-tuning on EgoIT-99K lowers EAS accuracy while leaving EgoThink almost unchanged.
Critical strengths
-
The singleton construction is the main contribution and a good one. It turns “copying what someone else in the scene is doing” into an error that can be scored unambiguously, which existing action-selection benchmarks such as EgoNormia and EgoPlan-Bench do not test. The pilot in Appendix B makes this concrete: replacing one EgoNormia option with another person’s action from the same scene drops Gemini-3-Flash from 66.9% to 32.4%, while a random action from another video does not. This is the main reason I think the problem is worth studying.
-
The controls are stronger than in most benchmark papers. Text-only accuracy on direct questions is at chance (28.5% vs. 30.1%), the model ranking holds under a different prompt, and the pattern repeats on the Wikimedia set. So the gap is unlikely to be a text shortcut or a prompt artifact.
Critical weaknesses
-
The central claim is a diagnosis, third-to-first-person confusion, but the only evidence for it is where the errors land, and two other explanations are not ruled out. (a) The evaluation prompt (Appendix C) never says the model is not one of the people in the image, yet the prompt used to generate training rationales (Appendix G) says exactly this. (b) The model may not recognize that the role is already taken. The exclusion instruction adds 28.1 points for Qwen3.5 but only 8.6 for InternVL2.5, which stays below chance (Table 4), so the failure is probably not the same across models. This matters because the broader point, that MLLMs struggle to reason from a specified viewpoint, is already documented in the perspective-taking literature (e.g., COMFORT, ICLR 2025). Work on telling the user apart from others in egocentric video (Ego-Grounding for Personalized Question-Answering in Egocentric Videos, CVPR 2026) is not discussed either. What is new here is the role-level confusion in action selection, so that mechanism needs to be shown directly. A prompt that only clarifies the viewer’s identity, plus a systematic categorization of the rationales the models already produce (Figure 4 shows two), would do this.
-
The fine-tuning result highlighted in the abstract (−33.9% on EgoGapBench vs. −0.5% on EgoThink) compares different formats and metrics: multiple-choice accuracy against GPT-4o-graded open-ended answers. After LoRA on mostly video dialogues, the drop could come from output-format drift or general forgetting rather than from egocentric data itself. A same-format control is needed, such as an egocentric multiple-choice benchmark (e.g., EgoPlan-Bench) or the same LoRA on a matched amount of non-egocentric instruction data. Table 2 also omits the singleton selection rate after tuning, which is the number that would support the authors’ reading.
Questions for authors
-
What happens if the default prompt adds one sentence stating that the model is not any of the people in the image and is standing where the photo was taken, without the exclusion rule? If accuracy approaches the exclusion-prompt level, the finding is mainly about prompt interpretation; if not, it supports the confusion claim.
-
After EgoIT-99K tuning, what are the singleton and off-vantage selection rates on n4 questions, and how does the same LoRA affect an egocentric multiple-choice benchmark?
-
What instructions did the three human annotators receive? Were they told explicitly that they are a bystander at the camera position?