acceptodds
Withdrawn from ICLR 2027

EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) deployed in devices such as smart glasses need to identify appropriate next actions from the user's perspective. However, some questions in existing evaluations based on first-person visual inputs can be answered based solely on the camera wearer's body cues. When other people's actions are absent from the answer choices, a correct response provides limited evidence that the model distinguishes the user's situation from those of others, so it is difficult to assess action selection from the user's perspective separately from input understanding. To address this limitation, we use images of multiple people that contain no body cues from the camera wearer and ask models to respond as if they were directly viewing the scene. We define Egocentric Action Selection (EAS) as the task of selecting an appropriate action from the user's perspective in a given situation and introduce EgoGapBench, a diagnostic benchmark for EAS with 1,000 multiple-choice questions on 440 COCO images. EgoGapBench includes other people's actions as distractors when duplicating those actions would be inappropriate in the given situation. All evaluated open- and closed-source MLLMs achieve substantially lower accuracy than the human baseline, and for most models errors concentrate on the actions of other people in the scene. Moreover, after instruction tuning on first-person visual input data, all five open-source models score nearly the same on an existing benchmark of first-person visual inputs ( relative) but drop on EgoGapBench ( relative).

Primary area:
datasets and benchmarks
Keywords:
egocentric vision, egocentric perspective understanding, multimodal large language models, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KhkKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Egocentric Action Selection (EAS): given a single image of a multi-person scene in which the camera wearer’s body is not visible, the model must choose the action that the viewer standing at the camera position could take right now. To make the task diagnostic, distractors are built from the actions of other people in the scene whose roles or objects can only be held by one person (the “singleton” constraint), plus an “off-vantage” action that cannot be performed from the viewer’s position. EgoGapBench contains 1,000 questions on 440 COCO images, with a 107-question Wikimedia set and a 168-question training split. All evaluated MLLMs fall well below humans (66.1% for the best configuration vs. 94.5%), most errors go to other people’s actions, and fine-tuning on EgoIT-99K lowers EAS accuracy while leaving EgoThink almost unchanged.

Critical strengths
  1. The singleton construction is the main contribution and a good one. It turns “copying what someone else in the scene is doing” into an error that can be scored unambiguously, which existing action-selection benchmarks such as EgoNormia and EgoPlan-Bench do not test. The pilot in Appendix B makes this concrete: replacing one EgoNormia option with another person’s action from the same scene drops Gemini-3-Flash from 66.9% to 32.4%, while a random action from another video does not. This is the main reason I think the problem is worth studying.

  2. The controls are stronger than in most benchmark papers. Text-only accuracy on direct questions is at chance (28.5% vs. 30.1%), the model ranking holds under a different prompt, and the pattern repeats on the Wikimedia set. So the gap is unlikely to be a text shortcut or a prompt artifact.

Critical weaknesses
  1. The central claim is a diagnosis, third-to-first-person confusion, but the only evidence for it is where the errors land, and two other explanations are not ruled out. (a) The evaluation prompt (Appendix C) never says the model is not one of the people in the image, yet the prompt used to generate training rationales (Appendix G) says exactly this. (b) The model may not recognize that the role is already taken. The exclusion instruction adds 28.1 points for Qwen3.5 but only 8.6 for InternVL2.5, which stays below chance (Table 4), so the failure is probably not the same across models. This matters because the broader point, that MLLMs struggle to reason from a specified viewpoint, is already documented in the perspective-taking literature (e.g., COMFORT, ICLR 2025). Work on telling the user apart from others in egocentric video (Ego-Grounding for Personalized Question-Answering in Egocentric Videos, CVPR 2026) is not discussed either. What is new here is the role-level confusion in action selection, so that mechanism needs to be shown directly. A prompt that only clarifies the viewer’s identity, plus a systematic categorization of the rationales the models already produce (Figure 4 shows two), would do this.

  2. The fine-tuning result highlighted in the abstract (−33.9% on EgoGapBench vs. −0.5% on EgoThink) compares different formats and metrics: multiple-choice accuracy against GPT-4o-graded open-ended answers. After LoRA on mostly video dialogues, the drop could come from output-format drift or general forgetting rather than from egocentric data itself. A same-format control is needed, such as an egocentric multiple-choice benchmark (e.g., EgoPlan-Bench) or the same LoRA on a matched amount of non-egocentric instruction data. Table 2 also omits the singleton selection rate after tuning, which is the number that would support the authors’ reading.

Questions for authors
  1. What happens if the default prompt adds one sentence stating that the model is not any of the people in the image and is standing where the photo was taken, without the exclusion rule? If accuracy approaches the exclusion-prompt level, the finding is mainly about prompt interpretation; if not, it supports the confusion claim.

  2. After EgoIT-99K tuning, what are the singleton and off-vantage selection rates on n4 questions, and how does the same LoRA affect an egocentric multiple-choice benchmark?

  3. What instructions did the three human annotators receive? Were they told explicitly that they are a bystander at the camera position?

Reviewer_p79dOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces EgoGapBench, a diagnostic benchmark for Egocentric Action Selection (EAS), defined as selecting an action that is appropriate for the viewer rather than simply reproducing an action performed by another person in the scene. The central concern is "third-to-first-person confusion": an MLLM may recognize what another person is doing yet incorrectly recommend that action to the user.

The benchmark contains 1,000 multiple-choice questions constructed from 440 COCO images containing multiple people while excluding visible body cues from the putative camera wearer. A key design device is the singleton constraint: distractors correspond to actions tied to roles or resources already occupied by another person, such that performing the same action would be inappropriate for the viewer. Some questions additionally contain an off-vantage distractor that cannot be performed from the camera position.

The experiments show a substantial gap between humans and evaluated MLLMs. The strongest reported model configuration reaches 66.1% accuracy compared with a 94.5% human baseline, with errors frequently corresponding to actions performed by other people in the image. The authors further show that instruction tuning on first-person visual data can leave performance on an existing egocentric benchmark approximately unchanged while substantially reducing EgoGapBench accuracy. Fine-tuning directly on EAS examples improves performance but does not eliminate the human-model gap. Prompt, text-only, explicit-exclusion, and out-of-domain analyses provide useful controls.

Critical strengths
  1. The paper isolates a meaningful failure mode that existing egocentric benchmarks do not directly test. The distinction between identifying an action in a scene and determining whether that action belongs to the viewer is important for interactive multimodal systems. The singleton-distractor construction provides a simple and reasonably controlled way to expose cases in which models copy another actor's behavior rather than selecting an action appropriate for the viewer. There is also an interesting conceptual connection to prior work on tracking, where successful behavior requires maintaining the correct correspondence or target independently of potentially misleading appearance information. For example, PathTracker showed substantial failures of neural networks when target identity must be maintained from motion rather than appearance (Linsley et al., 2021; Malik et al., 2021), while FeatureTracker subsequently demonstrated failures when an object's appearance changes substantially over time (Muzellec et al., 2025). EgoGapBench exposes an analogous, although importantly different, binding issue at the level of agent and action rather than object identity: recognizing an observed action does not imply correctly assigning that action to the agent for whom it is appropriate.

  2. The paper contains a thoughtful collection of controls rather than relying only on benchmark accuracy. The human comparison, direct versus indirect analysis, distractor-type analysis, alternate prompt, text-only condition, explicit exclusion instruction, out-of-domain Wikimedia evaluation, and two fine-tuning experiments collectively make the empirical story considerably stronger. In particular, the finding that first-person instruction tuning can preserve performance on EgoThink while degrading substantially on EgoGapBench is a useful demonstration that conventional first-person understanding and egocentric action selection are not equivalent capabilities.

Critical weaknesses
  1. The diagnostic setup may partially conflate perspective binding with learning a benchmark-specific exclusion rule. By construction, visible people's singleton actions are inappropriate for the viewer. This makes the diagnostic clean, but real multi-agent settings also contain actions that another person is performing and that the viewer could appropriately perform as well. The explicit-exclusion experiment is revealing here: directly stating the dataset rule causes large gains for several models. A stronger test would include matched cases in which copying another person's action is appropriate, requiring the model to infer agent/action compatibility rather than learning a general "do not choose observed actions" heuristic. Without such a control, I think the paper should be careful about interpreting the results as a general measure of third-to-first-person perspective confusion.

  2. Parts of the benchmark provide answer-type cues independent of the image. The text-only experiment reports substantially above-chance performance on the indirect subset. Since indirect correct answers involve responses such as watching while distractors describe singleton-role actions, linguistic structure can provide information about the correct choice. The authors appropriately acknowledge this and identify the direct subset as the more meaningful control, but the issue complicates interpretation of overall EgoGapBench accuracy. Reporting direct performance as a primary diagnostic metric, or constructing linguistically matched indirect distractors, would strengthen the benchmark.

Questions for authors
  1. Clarification that could affect my assessment: Have the authors evaluated a matched control containing multi-person scenes where an action performed by another person is nevertheless also appropriate for the viewer? This would help distinguish genuine agent-action binding from a strategy of suppressing actions associated with visible people. The strong gains from the explicit exclusion instruction make this distinction particularly important.

  2. Clarification that could affect my assessment: How sensitive are the principal conclusions to considering only the direct subset? Since the text-only results suggest substantial linguistic leakage for indirect questions, it would be useful to know whether the main model comparisons, fine-tuning effects, and conclusions remain qualitatively unchanged when direct questions are treated as the primary evaluation.

  3. Suggestion rather than a requirement for acceptance: The authors may want to discuss the benchmark in the broader context of visual binding/correspondence problems. Prior work on appearance-free and appearance-changing tracking demonstrates that neural systems can recognize relevant visual information while failing to maintain the appropriate target correspondence over time (Linsley et al., 2021; Malik et al., 2021; Muzellec et al., 2025). EgoGapBench appears to expose a related binding problem across agents rather than across object trajectories.

References

Linsley, D., Malik, G., Kim, J., Govindarajan, L. N., Mingolla, E., & Serre, T. (2021). Tracking without re-recognition in humans and machines. Advances in Neural Information Processing Systems, 34, 19473–19486.

Malik, G., Linsley, D., Serre, T., & Mingolla, E. (2021). The challenge of appearance-free object tracking with feedforward neural networks. arXiv preprint arXiv:2110.02772.

Muzellec, S., Linsley, D., Ashok, A. K., Mingolla, E., Malik, G., VanRullen, R., & Serre, T. (2025). Tracking objects that change in appearance with phase synchrony. International Conference on Learning Representations.

Reviewer_7hvYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark, EgoGapBench, aiming to evaluate whether MLLMs can select actions appropriate for the current user from the perspective of the current scene. The authors also define a task called Egocentric Action Selection (EAS). EgoGapBench consists of 440 images and 1,000 multiple-choice questions to evaluate the EAS task. Through evaluations of open-source and closed-source models, the authors find that most errors are concentrated on selecting actions performed by other people in the scene.

Critical strengths
  1. To define the Egocentric Action Selection task, the authors specifically select images in which other people are present but the camera wearer's body is not visible. This clearly distinguishes the proposed benchmark from existing egocentric benchmarks. The authors also design specific distractors to facilitate analysis and evaluation.

  2. The authors introduce the singleton share metric to analyze model failures in Egocentric Action Selection. This helps reveal the differences between current models and human performance.

Critical weaknesses
  1. In the evaluation experiments, fine-tuning on EgoIT leads to a substantial performance drop when tested on EgoGapBench, revealing considerable differences between the benchmarks. Could fine-tuning on EgoGapBench also help models achieve improvements on first-person perspective benchmarks such as EgoThink? In addition, the prompt variation experiments lack evaluations using Chain-of-Thought (CoT) or similar deep reasoning approaches, which could better reflect the models' ability to understand and switch between different perspectives.

  2. The current benchmark contains an insufficient amount of data for evaluation. The authors should further expand the evaluation scale and analyze model performance across different scenes and activities.

Questions for authors

Why do the authors not construct the EAS task using more commonly used egocentric datasets? Such recordings should better reflect the daily usage scenarios of smart glasses users. As for the camera wearer's body, it could also be filtered out through post-processing, since not all egocentric recordings contain visible body parts of the wearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.