acceptodds
Under review as a conference paper at ICLR 2027

EEMAG : Egocentric-Exocentric Multimodal Affordance Grounding

Abstract

With the rise of intelligent systems, interactions between humans and robots need to be more precise and understandable. To this end, affordance detection, linking each affordance to precise objects' regions, has become more important than ever. This bridge between affordances and regions is learned by humans through a mix of experimentations and instructions. In this paper, a novel framework is presented to based on this multimodality of affordance learning, between exocentric experiments and semantic instructions : Egocentric-Exocentric Multimodal Affordance Grounding (EEMAG). It extends the multimodal information extraction from a semantic description to the multimodal description of ongoing action. This knowledge is then transferred to an Egocentric Branch through a process of distillation and fusion. To achieve this goal, we propose a new branch to replace textual branches : the Multimodal Branch. It uses a Visual Language Model and Perceivers to extract visual and semantic information about the exocentric image. This branch effectively separates affordances and guide the Egocentric Branch towards useful visual features. Our framework is trained using the weakly supervised paradigm, using only image-level labels. Different experiments demonstrate the performance of our framework compared to state of the art affordance grounding models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual affordance detection, contextual affordance, text embedding, computer vision, deep learning model

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_UFXXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses affordance grounding: localizing where a human would naturally interact with an object upon seeing it. In particular, it tackles this problem in a weakly supervised setting, learning from only two inputs, an exocentric image showing a human interacting with the object and an egocentric image containing only the object, rather than from dense and costly labels. The authors argue that existing methods either rely solely on visual information or use exocentric images only during training, and therefore cannot exploit that context at inference time. WSMA adds a text branch, but its text is fixed and independent of the current scene. To address these limitations, the authors propose EEMAG. Built on top of WSMA, EEMAG uses a VLM so that exocentric images can also provide information at inference time. It outperforms WSMA on two evaluation datasets, AGD20K and HICO-IFF.

Critical strengths
  • Affordance grounding is an important problem for both interpretability and robotics, especially as the era of Physical AI approaches. The weakly supervised setting is a data-efficient and scalable choice, since it does not require fine-grained labels.
  • The paper shows that adding a multi-modal branch can benefit the WSMA architecture.
  • The authors are candid about the method’s weaker performance, and they transparently disclose the limitation that training is unstable.
Critical weaknesses
  • Above all, EEMAG appears to underperform Selective-CL. The gap is not large, but the trend holds on both datasets (AGD20K and HICO-IFF) regardless of the split, which is disappointing given that a large vision-language model is used. Moreover, LoopTrans [1], a published work, outperforms Selective-CL. Taking this into account, there is no metric/dataset pair on which EEMAG achieves the best performance.
  • The paper clearly lays out how it is similar to and different from WSMA, and this reviewer appreciates that. Even so, proposing a multi-modal branch falls short on novelty for the acceptance bar of a top-tier venue such as ICLR.
  • A paper should be self-contained. This includes experimental hyperparameters, the computational cost of the experiments, how the metrics are computed (it is risky to assume that the reviewer already knows them), and dataset details. These are usually expected in the Appendix, but this submission covers them only thinly, which hurts both reproducibility and the review process.
  • Finally, I cannot be confident that the paper compares fairly against the other baselines. Unlike the baselines, EEMAG uses an exocentric image, which (1) gives it additional information at inference time, (2) undermines the zero-shot inference that a text encoder would otherwise offer if an exocentric image is strictly required, and (3) is not explained in the paper: how the exocentric image is selected is never disclosed. Is it chosen from validation data, or at random? If from validation data, which data? If at random, what is the standard deviation? If neither, how robust is the method?

[1] Jiajin Tang, Zhengxuan Wei, Ge Zheng, and Sibei Yang. Closed-loop transfer for weakly-supervised affordance grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9530–9539, 2025.

Questions for authors
  • As far as I know, the ICLR guidelines place table captions above the table. Please revise accordingly.
  • In Table 2, the SIM metric for EEMAG on the AGD20K-Seen split is listed as 0.419, but shouldn’t it be 0.420, as in the Table 3 ablation?
  • It appears that DINO-ViT-S and Qwen2.5 are used as the image feature encoder and the VLM, respectively. DINO has progressed to DINOv3 and comes in a range of model sizes, so I would like to see experiments on these variants. Do newer models yield better performance? If not, the method may not generalize well. A similar experiment on the VLM backbone should support the same kind of conclusion.
  • INTRA demonstrates zero-shot inference results by using a text encoder. I am curious whether EEMAG can do the same. If not, there would be no advantage to using a separate text encoder.
  • When using quotation marks in LaTeX, please use `` for opening quotes rather than the straight or curly double quote “, as this is the correct LaTeX syntax.
Reviewer_caUCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces EEMAG framework to address the affordance learning problem.

Critical strengths

None.

Critical weaknesses
  1. The figures in the image is ugly. This manuscript doesn't match the level of ICLR.
  2. The performance gains are limited and do not represent a comprehensive lead. As shown in Table 2, EEMAG underperforms Selective-CL on the AGD20K-Unseen dataset in terms of KLD and SIM, as well as on the AGD20K-Seen dataset. Although the paper claims to "outperform most of these approaches," the lack of a consistent advantage over the strongest baseline weakens the persuasiveness of the method's contributions.
Questions for authors
  1. Compare the EEMAG with Selective-CL, what's the advantage of EEMAG? The performance is worse than the Selective-CL in Table 2.
Reviewer_qJZjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission proposes to use a vision language model (VLM) to serve as a multimodal branch to replace the text encoder being used in existing affordance grounding methods. In particular, the proposed approach follows WSMA (Xu et al., AAAI 2024), where a VLM (Qwen2.5-3B-instruct) is used to replace WSMA's text encoder. This raises a novelty concern considering the proposed approach is similar to WSMA with a straightforward modification and incorporating a VLM in affordance grounding has been explored in existing methods.

The writing of the paper could be improved (see minor issues for details). As a submission to ICLR, it is expected that the presentation of the paper is clear and consistent, however, the writing and the presentation of figures and tables show inconsistencies across the paper. 

The quantitative results show the proposed approach is comparable to existing methods (or even worse), which does not support the effectiveness of the proposed approach and its core modules. In addition, several related methods are not included in the comparison, which makes the comparison incomplete.

Critical strengths

None.

Critical weaknesses

The major weakness of this paper is the novelty. The proposed approach is similar to WSMA (Xu et al., AAAI 2024) with a VLM to replace the text encoder. Simply changing an encoder is not enough to claim this as a major technical contribution or novelty, where the experimental results also do not support the significance (Table 2 WSMA vs. the proposed approach). Incorporating or utilizing VLM or MLLM for affordance grounding has been explored in existing methods [Review 1,2], which are not discussed or compared in this submission.

The writing and presentation of this submission need to be improved (see minor issues for some typos or inconsistencies). A proofreading is needed to fix the issues. The figures are not presented professionally, the positions and the scale of images appear to be random. The tables also show some issues, where Tables 2 and 3 use alternating shading for odd and even rows, so the last row in Table 3 shading should be applied.

Several affordance grounding methods are not included in the discussion or quantitative comparisons [Review 1-4]. In particular, WSAG-PLSP [Review 3] is supposed to be included in the quantitative comparison, Affordance-r1 is another consideration. ICLR does not require authors to compare the papers published in the last two months, so the authors can decide if they'd like to include the ECCV papers.

[Review 1] Affordance-r1: Reinforcement learning for generalizable affordance reasoning in multimodal large language models, AAAI 2026.

[Review 2] Token-Based Affordance Grounding with Large Vision-Language Models, ECCV 2026

[Review 3] Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors, ICLR 2025

[Review 4] A4-Agent: An Agentic Framework for Zero-Shot Affordance Reasoning, ECCV 2026

Questions for authors

Questions:

  • Clarify the novelty of the proposed approach, discuss the relationship with related methods that utilize VLMs or MLLMs.

  • Consider including the related methods in the comparison, which helps to assess the performance of the proposed approach.

Suggestions (minor issues):

  • Line 26: "... state of the art affordance grounding models." - state-of-the-art affordance grounding models. Similar issues in the caption of Fig. 1 (Line 50-51), line 67.

  • Line 57, 59: Inconsistent capitalization. (1) affordances ... (2) Several affordances.

  • Line 408: Tables 3. Why is it "tables"?

Future work:

  • Do a careful proofreading of the whole paper is helpful. Also, please check and update the figures and tables based on the comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.