acceptodds
Under review as a conference paper at ICLR 2027

Preserving How Across Objects: Execution-Conditioned Bimanual Contact Grounding

Abstract

Most cross-object interaction models learn where an action can be performed on a new object. Describing an interaction by its action makes this knowledge transferable, but can obscure how two hands carry it out: different executions of the same action assign different roles to the hands and may require different contacts. We represent an execution by its action and the functional roles of both hands, independently of the target object. We learn this representation from the visual features of regions contacted under the same execution across training objects. Given a new target image, we use the learned representation directly as the grounding query to predict where the queried hand should make contact. We evaluate on unseen instances of GigaHands training categories, in external transfer to HOI4D, and through execution contrasts that change hand roles while fixing the object, action, and queried hand. Our method achieves the highest mean localization scores and better distinguishes executions that require different contacts, with a score of 26.4% versus 18.8% for a non-learned prototype baseline and 12.0% for the strongest learned baseline. It also reaches 21.8% versus 10.1% for that baseline on role combinations never seen in training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-object interaction, affordance grounding, interaction representation, cross-object generalization, bimanual interaction, compositional representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t1eVOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Existing affordance grounding (weakly supervised) methods typically focus on generalizing across different objects (e.g., shape and color), and bimanual models predict both hands' affordance. In this submission, the authors aim to associate the functional role to a specific hand, which is not explicitly modeled in existing methods.

It is discussed in the paper that localization alone cannot demonstrate if a model follows a specified execution, this seems to be a plausible statement. However, the rest of the paper is organized in a way that it is challenging for a reader to find evidence to support the proposed approach effectively addressing the problem (e.g., Table 1 contact localization). Fig. 2 is supposed to show that but it again just focuses on localization (affordance grounding).

As the authors address the affordance problem from a new angle, it is important to make sure the benchmark datasets used are suitable to validate the main hypothesis. In the paper, it is explained that GigaHands is filtered and repurposed to be used in the evaluation of this submission, and HOI4D (subset) is used as well to test capture conditions shift. Additional information is also provided in the App. A.1, but it'd be helpful to provide qualitative examples (images and labels) to show the data being used in the evaluation of the paper. And the subset seems to be relatively small compared to the original dataset. For example, HOI4D includes 1,506 recordings, only 154 recordings are used as stated in the App. A.1.

In terms of baseline comparisons, this is a major shortcoming of the submission, only two baselines are included in the comparison and several newer methods are not included, which are potentially applicable for the problem this submission aims to solve. That said, this part is another area that can be improved.

Critical strengths

The authors tackle affordance grounding from a new angle beyond localization alone. In my opinion, this is a meaningful step in the field. In daily life, we often complete a task using both hands, and each hand performs a different role. So, functional association with each hand might provide a fine-grained affordance analysis.

Critical weaknesses

The writing and readability could be improved. 1) Motivation is not clear enough, where the authors discussed existing methods largely ignored the functional association part, but did not provide the benefit of doing this. Adding this part would strengthen the motivation. 2) It is not easy to follow the paper to understand what the key differences between the proposed approach compared to the related methods are. 3) As briefly mentioned in the summary, the provided comparisons (quantitative and qualitative) do not focus on supporting the advantage of the proposed approach. The main table (Table 1) and qualitative results (Fig. 2) both focus on localization.

The experimental setup could be improved. 1) Benchmark datasets are important, especially when a paper tackles the problem from a new angle, in this case, no existing benchmark can be used or evaluation protocol could be directly used. In this case, it's important to establish that the benchmark datasets being used or the subsets selected are suitable to evaluate the proposed approach (the subset not too small). 2) Two baselines are included in the comparison, a number of recent affiance grounding methods are not included [r1 - r4]. Including these methods in the comparison would allow a reader to understand the performance of these recent methods (some are zero-shot) on the problem this submission focuses on. Note: Based on the ICLR's guidelines, papers officially published within the last two months are considered contemporaneous and the authors are not required to compare to those papers. The ECCV 2026 papers are published within two months of the ICLR paper submission, so it's not required to compare unless the authors would like to.

[r1] Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors, ICLR 2026

[r2] Affordance-r1: Reinforcement learning for generalizable affordance reasoning in multimodal large language models, AAAI 2026.

[r3] Token-Based Affordance Grounding with Large Vision-Language Models, ECCV 2026

[r4] A4-Agent: An Agentic Framework for Zero-Shot Affordance Reasoning, ECCV 2026

Questions for authors

Questions:

  • Clarify the motivation in terms of the benefit of providing functional association in affordance grounding.

  • Add the information regarding the benchmark datasets and the subset being used, as a reader, not enough information is provided in the initial submission.

  • Consider including more baselines to strengthen the comparison, which helps readers to understand whether the recent methods can address the problem this submission focuses on.

Suggestions for future work:

  • It's worth considering a dedicated benchmark to focus on the proposed problem. In this paper, as the focus is the proposed approach, not enough information is provided regarding the datasets being used, or whether they are suitable or providing statistical significance.

  • Improving the writing and readability can help readers understand how the proposed approach differentiates from the related methods.

  • Organize the results to support the key claim of the proposed approach, otherwise, it is not clear whether the proposed approach is effective in terms of the problem the paper focuses on addressing.

Reviewer_4LhGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that action-conditioned affordance grounding could not distinguish different bimanual executions of the same action, thus proposed to construct a query-relative execution representation and supervises it using contact prototypes obtained by pooling frozen DINOv2 features over annotated contact regions. This representation serves as the query for a cross-attention decoder. The paper also introduces execution contrasts to evaluate whether the predicted contacts could follow the changes in hand-role assignments. Extensive results show improvements on GigaHands objects and unseen role compositions, as well as cross-dataset contact localization on HOI4D.

Critical strengths

The distinction between affordance and execution-conditioned contact grounding is meaningful and reasonable, which could bring new understandings to the community.

The execution contrast benchmark is a valuable, also further emphasizing the necessity of the distinction between affordance and execution-conditioned contact grounding, it highlights that conventional localization metrics can remain strong even when predictions fail to distinguish different hand-role assignments.

Comprehensive ablation studies are provided to validate the importance of the proposed training design.

Evaluating on HOI4D without fine-tuning is useful. The authors also explicitly acknowledge important limitations, including the reliance on annotated roles, within-category object generalization, and proximity-based contact labels.

Critical weaknesses
  • Demonstrating what score should be expected given prediction pairs that differ without encoding the correct execution semantics could be helpful as reference.

  • The SelectiveCL baseline differs from the proposed method in visual backbone and grounding head, while 2HandedAfforder was not originally designed for structured hand-role conditioning. These are reasonable reference methods but not sufficiently controlled alternatives. A stronger comparison would retain the same DINOv2 backbone, grounding decoder, spatial losses, and role-conditioning inputs, while replacing the proposed contact-semantic interface with a conventional role-conditioned embedding or MLP.

  • The held-out composition evaluation contains 36 compositions and 72 contrasts across 24 test objects. Although removing the corresponding role assignments and their mirrored counterparts from training is appropriate, per-composition results would clarify whether the improvement reflects systematic role composition or interpolation among closely related observed executions. Contrasts that modify only the partner role would be particularly informative.

  • The localization improvement on held-out GigaHands objects is small (.323 v.s. .319). Additionally, the HOI4D experiment supports cross-dataset localization but does not directly establish cross-dataset preservation of bimanual role assignments.

Questions for authors
  • What is the Group score of independent random maps and role-permuted predictions on the exact evaluation contrasts? These could be valuable references.

  • Could the authors compare against a model using the same visual backbone and decoder with direct role conditioning, but without contact-semantic supervision?

  • How are the held-out compositions distributed across actions and constituent roles?

Reviewer_JVGvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates two-hand contact localization under execution conditions: given an image of a target object, an action, the functional roles of the left and right hands, and the query hand, it predicts the region of the object that the query hand should contact. The core motivation is that different divisions of labor between the two hands for the same action may correspond to different contact regions, and using only action labels is insufficient to distinguish these execution modes. This method encodes the action, the role of the queried hand, and the role of the other hand into an execution representation relative to the query. It uses frozen DINOv2 features trained on object contact regions to supervise this representation and directly employs it as the query for the target image localization decoder. The paper further constructs execution contrasts by fixing the object, action, and queried hand while varying only the role assignment, to distinguish between general contact localization ability and the ability to maintain a specified execution mode. Besides that, experiments cover unseen object instances and unseen role combinations on GigaHands, as well as HOI4D external transfer without fine-tuning. Key results show that the method achieves the highest mean scores on standard localization metrics and demonstrates more significant improvements in execution contrast evaluations.

Critical strengths
  1. The paper points out a significant distinction: predicting a reasonable contact region does not imply that the model understands the specified division of labor between the two hands. The performance comparison minimizes interference from other factors by fixing the object, action, and queried hand. Furthermore, the bidirectional matching metric can rule out models that consistently predict the same region across different executions. For example, in Table 2, the gap between this paper’s model and the adapted SelectiveCL in locating the AP is relatively small; however, the combined Group scores are 26.4% and 12.0%, respectively, while the uncombined scores are 21.8% and 10.1%. This result supports the authors’ conclusion that there is a distinction between localization capability and execution retention capability.

  2. The core design of the direct supervised localization query adopted in this paper is supported by relatively clear controlled ablation results. I note that the progressive experiments in Table 3 show that Group’s performance increased from 14.0% with direct conditioning to 16.0% with the normalization bottleneck, 18.0% with independent auxiliary semantic heads, and finally reached 26.4% with the direct supervised localization interface. This set of comparisons distinguishes the roles of normalization, the addition of auxiliary supervision, and the supervision of position.

Critical weaknesses
  1. The independent contribution of the relative encoder has not yet been distinguished from the effect of retaining complete role information. The role ablation experiments in Table 3 were primarily conducted by sharing the role type vector, masking the role of the other hand, or masking the hand identity. The first two methods render some action pairs indistinguishable in the input representation; therefore, the performance decline reflects both the impact of information loss and the impact of structural changes. The current results do not yet demonstrate that SetEnc–Readout outperforms sequential concatenation or a simple MLP encoder that receives the same complete input. This is also acknowledged in Section 3.5 of this paper.

  2. There is no evidence of mechanism verification for the combination, and the coverage of core ability assessments remains limited. For example, the combination experiment did not include an execution control across 24 objects, so the scope of coverage remains relatively limited. HOI4D provides valuable external localization results, but since its tasks primarily involve one-handed operations and lack an execution control evaluation, it supports the transfer of contact-based localization but cannot yet directly support the external transfer of two-handed execution and grasping abilities. Finally, the authors are encouraged to provide more qualitative visualization results to support their work.

Questions for authors
  1. After maintaining the input of complete actions, own role, partner role, and hand identity, and fixing the visual branch, decoder, and training loss, how does a simple ordered concatenation/MLP encoder perform?

  2. On the unseen combinations in Table 2b, does the advantage of the full model over models without contact semantic supervision and those with contact semantic supervision applied only to the auxiliary head still hold?

  3. The main manuscript states that SelectiveCL shares the encoder, loss, and training protocol, while Appendix A.2 explicitly specifies only the use of the same spatial loss. Does this baseline use LcsL_{cs}? If so, on which representation is the supervision applied?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.