acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Cross-View Alignment: Interaction-Grounded Part Semantic Guidance for Weakly Supervised Affordance Grounding

Abstract

Affordance grounding aims to localize object regions that support specific actions. Since dense affordance annotation is expensive, weakly supervised affordance grounding (WSAG) has attracted increasing attention. Recent WSAG methods exploit exocentric human–object interactions as weak supervision for egocentric localization. They mainly extract visual cues from exocentric interactions and transfer them through cross-view feature or region alignment. However, large differences in appearance, viewpoint, and occlusion can make such visual transfer less reliable. Motivated by this limitation, we propose Interaction-Grounded Part Semantic Guidance (IGPSG), which explores interaction-grounded functional-part semantics as a complementary cross-view transfer signal. IGPSG uses foundation models to identify the functional part relevant to the Exo interaction and represents it with a semantic prototype. This prototype indicates which part should be transferred, while Ego visual features determine where it is localized. We further introduce Part–Verb Consensus Grounding (PVCG), which refines the transferred part query with multi-level Ego features and part–verb spatial consensus. Extensive experiments on AGD20K and HICO-IIF demonstrate consistent improvements over existing methods in both Seen and Unseen settings, supporting the effectiveness of functional-part semantics as a complementary cue for cross-view affordance grounding.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Weakly Supervised Affordance Grounding, Cross-View Learning, Affordance Localization, Part Semantics, Semantic Transfer, Human–Object Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bHL3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Interaction-Grounded Part Semantic Guidance (IGPSG) for weakly supervised affordance grounding. Instead of relying solely on cross-view visual correspondence, the method transfers interaction-relevant functional-part semantics from exocentric to egocentric images. Specifically, Functional-Part Semantic Transfer (FPST) constructs semantic prototypes using pretrained foundation models and provides additional supervision for egocentric localization. Part–Verb Consensus Grounding (PVCG) further refines affordance predictions using multi-level visual features and spatial consensus between part and verb representations. Experiments on AGD20K and HICO-IIF demonstrate improvements over existing methods in both Seen and Unseen settings.

Critical strengths
  • The paper addresses the limitations of direct visual correspondence between exocentric and egocentric images by exploiting functional-part semantics, which can provide complementary guidance under substantial appearance and viewpoint variations.

  • The proposed method achieves improvements over existing approaches on AGD20K and HICO-IIF, including both Seen and Unseen settings. The component-wise ablation study also supports the contributions of FPST and PVCG.

Critical weaknesses
  1. Limited novelty. The novelty over WSAG-PLSP is not sufficiently established, as PLSP already incorporates part-level semantic priors and pretrained foundation models such as VLPart and SAM. The paper should more clearly explain what fundamentally distinguishes the proposed approach from PLSP and why the additional mechanisms lead to better performance.

  2. Unclear complexity-performance trade-off. The additional computational and architectural complexity is not adequately analyzed, making it difficult to assess whether the performance gains justify the increased complexity.

  3. Numerous hyperparameters and insufficient analysis. The proposed method introduces many hyperparameters and heuristic design choices, including multiple temperature parameters, filtering thresholds, reliability weights, and so on. However, the paper does not sufficiently explain how these values were selected or provide sensitivity analyses. It is also unclear whether these hyperparameters were tuned separately for each dataset or whether the same configuration generalizes across datasets. Given the number of design choices, these analyses are important for evaluating the robustness and practicality of the proposed method.

Questions for authors
  1. What exactly does the "Base model" in Table 3 refer to? How does it differ from the full WSAG-PLSP model?

  2. What percentage of the training samples have valid semantic candidates and receive FPST supervision?

  3. Are there failure cases caused by inaccurate part detection or segmentation from VLPart and SAM? How sensitive is the method to such errors?

  4. How is the egocentric object mask in Eq. (6) obtained?

6 comments
AuthorsOpenReview ↗

Response to Reviewer bHL3(PART I)

We greatly appreciate the reviewer's careful assessment and valuable suggestions. We have taken each concern seriously and made corresponding revisions, including methodological clarifications and supplementary experiments. We respond to each point in detail below.

W1. Novelty Beyond Visual Feature Alignment

Thank you for raising this important concern. We acknowledge that WSAG-PLSP already incorporates part-level semantic priors and pretrained foundation models such as VLPart and SAM. Our novelty lies not in using these models, but in exploring a complementary approach to cross-view affordance learning beyond conventional visual feature alignment.

Specifically, while PLSP employs part-level semantic priors to refine weak supervision, our IGPSG transfers Exo-conditioned functional-part semantics to Ego observations through sample-adaptive semantic prototypes. FPST uses Exo-instance evidence to select and reliability-weight candidate part semantics, which then provide auxiliary spatial guidance for Ego affordance localization. This reduces dependence on direct visual correspondence under cross-view variations in object shape, scale, and texture. Unlike conventional cross-view feature alignment, FPST retrieves semantic support in the Ego image and supervises the predicted probability mass over this support, without enforcing direct Exo–Ego feature correspondence. PVCG further refines the Ego query through multi-level visual evidence and part–verb spatial consensus.

The key distinction is therefore how functional-part semantics are constructed from Exo evidence and transferred to Ego observations, rather than the introduction of part-level priors themselves. The component ablations in Table 2, together with the controlled Text-only and Direct alignment comparisons in Table 3, support the distinct contributions of semantic transfer and Ego-side grounding.

W2. Complexity–Performance Trade-off

Thank you for highlighting this concern. We have conducted a complexity and efficiency analysis comparing IGPSG with WSAG-PLSP under identical inference settings (RTX 3090, FP32, 224 × 224 inputs).

MetricWSAG-PLSPOurs
Total parameters112.52 M114.20 M
Inference parameters112.38 M113.01 M
Estimated computation18.91 G19.28 G
Inference latency (batch size 20)59.98 ms62.22 ms

1. Performance gains with limited overhead. Our method increases total parameters by only 1.5%, inference parameters by 0.6%, and estimated computation by 2.0%, with an additional 2.24 ms per batch in inference latency. Meanwhile, compared with WSAG-PLSP, IGPSG achieves 4.8%/7.3% reductions in KLD, 2.2%/6.0% improvements in SIM, and 3.9%/7.7% improvements in NSS on Seen/Unseen, respectively.

2. Source of additional complexity. FPST operates only during training and introduces no additional inference computation. The modest inference overhead primarily arises from PVCG's multi-level part-conditioned attention and part–verb spatial consensus. Neither Exo inputs nor VLPart/SAM detection and segmentation are required at inference.

These results demonstrate that IGPSG achieves improved affordance localization with limited additional inference cost.

AuthorsOpenReview ↗

Response to Reviewer bHL3(PART II)

W3. Numerous Hyperparameters and Insufficient Analysis

Thank you for raising this concern. We have conducted additional sensitivity experiments to assess the key hyperparameter choices and evaluated a shared configuration on AGD20K Seen and HICO-IIF. The experimental setup and findings are summarized below.

1. Parameter selection and sensitivity analysis.

We examine five key hyperparameter groups: the reliability cutoff rmin⁡r_{\min}, reliability weights α\boldsymbol{\alpha}, prototype temperature τproto\tau_{\mathrm{proto}}, transfer weight λtransfer\lambda_{\mathrm{transfer}}, and PVCG temperature τPVCG\tau_{\mathrm{PVCG}}. We select these groups because they directly control candidate filtering, evidence scoring, semantic aggregation, transfer supervision, and spatial attention, respectively, covering the main mechanisms introduced by FPST and PVCG.

We vary one group at a time while keeping the remaining settings fixed. All configurations follow the same training and checkpoint-selection protocol as the main experiments.

Among the tested configurations, the shared default—rmin⁡=0.48r_{\min}=0.48, α=(0.42,0.28,0.20,0.10)\boldsymbol{\alpha}=(0.42,0.28,0.20,0.10), τproto=0.15\tau_{\mathrm{proto}}=0.15, λtransfer=0.01\lambda_{\mathrm{transfer}}=0.01, and τPVCG=0.10\tau_{\mathrm{PVCG}}=0.10—achieves the best score in five of the six dataset–metric combinations. The only exception is a marginal improvement in AGD20K Seen SIM with τPVCG=0.05\tau_{\mathrm{PVCG}}=0.05, accompanied by worse KLD and NSS. These results support the effectiveness and stability of the shared default within the tested parameter ranges.

FactorSettingAGD20K Seen KLD ↓SIM ↑NSS ↑HICO-IIF KLD ↓SIM ↑NSS ↑
DefaultShared0.84720.52141.60810.80590.57971.9503
Prototype temp. (τproto\tau_{\mathrm{proto}})0.100.86250.51821.59320.81740.57631.9365
Prototype temp. (τproto\tau_{\mathrm{proto}})0.200.85980.51941.60020.81360.57781.9421
Stricter cutoff (rmin⁡r_{\min})0.750.87860.51481.58240.82930.57261.9198
Stricter cutoff (rmin⁡r_{\min})0.850.90610.50971.55030.85420.56081.8841
Reliability weights (α\boldsymbol{\alpha})Equal (0.25 each)0.88410.51361.57750.83350.57121.9164
Transfer weight (λtransfer\lambda_{\mathrm{transfer}})0.0050.87020.51731.58860.82180.57491.9306
Transfer weight (λtransfer\lambda_{\mathrm{transfer}})0.0200.87580.51681.58020.82460.57381.9257
PVCG temp. (τPVCG\tau_{\mathrm{PVCG}})0.050.85630.52211.60140.81270.57821.9432
PVCG temp. (τPVCG\tau_{\mathrm{PVCG}})0.200.87150.51421.58710.82050.57401.9284

For the “Equal” setting, the four reliability components—detection confidence, object containment, area plausibility, and interaction-role prior—each receive a coefficient of 0.25.

2. Cross-dataset consistency and practicality.

The same default configuration is applied to AGD20K Seen and HICO-IIF without dataset-specific tuning, including the temperatures, filtering thresholds, reliability coefficients, and transfer weight. The shared default achieves consistently strong results on both datasets, supporting its practical applicability without dataset-specific adjustment.

AuthorsOpenReview ↗

Response to Reviewer bHL3(PART III)

Q1. Clarification of the Base Model in Table 3

Thank you for this question. The “Base model” in Table 3 of the original submission (now Table 2 in the revised manuscript) refers to our reproduction of the full WSAG-PLSP model, retaining its original architecture and training objectives, without FPST or PVCG. It is not a simplified version of WSAG-PLSP.

The WSAG-PLSP results in Table 1 are taken from the original publication, whereas Table 2 reports our own experimental runs under a consistent ablation setup. Using this reproduced baseline, Table 2 isolates the contributions of FPST and PVCG by progressively introducing Exo-conditioned semantic transfer and Ego-side localization refinement under matched experimental conditions.

Q2. FPST Supervision Coverage

Thank you for this question. We quantified FPST supervision coverage on both AGD20K training splits. Valid candidate coverage is the percentage of all training instances with at least one valid Exo semantic candidate, while effective supervision coverage is the percentage of all training instances contributing to the FPST support-mass loss after activation.

Training splitTraining instancesValid candidatesEffective FPST supervision
Seen6,92986.23%72.20%
Unseen4,81585.81%72.32%

These results show that valid semantic candidates are available for over 85% of training instances, while approximately 72% of all training instances receive effective FPST supervision after activation.

Q3. Failure Cases and Sensitivity to VLPart/SAM Errors

Thank you for raising this important concern. We acknowledge that inaccurate detections from VLPart or segmentation errors from SAM can affect the quality of candidate part evidence and, consequently, the transferred semantic prototypes. For example, missing a relevant functional part may prevent FPST supervision, while an incorrectly detected part may introduce noisy semantic guidance.

1. How does FPST handle detection and segmentation errors? FPST employs role-based filtering and reliability-weighted aggregation to mitigate the influence of unreliable candidates. Specifically, candidate reliability considers detection confidence, object containment, area plausibility, and interaction-role compatibility. These mechanisms reduce dependence on individual detections but cannot entirely eliminate errors.

2. How sensitive is the method to such errors? Our existing ablations provide indirect evidence of robustness. The “w/o reliability” ablation in Table 3 demonstrates the benefit of reliability weighting, while Appendix E.1 shows that weighted aggregation outperforms Top-1 selection across all six evaluation metrics. These results suggest that integrating multiple reliability-weighted candidates is preferable to relying on a single detection.

Q4. Construction of the Egocentric Object Mask

Thank you for this question. The Ego object mask in Eq. (6) is constructed from precomputed VLPart object detections. Specifically, we rasterize the selected object bounding box onto the Ego patch grid, marking overlapping patches as valid. This coarse mask restricts semantic support retrieval to the detected object region without requiring ground-truth segmentation. We have clarified this procedure in Section 3.3, “Ego-space Semantic Support Retrieval.”

Manuscript updates

The revised manuscript includes the following additions and clarifications.

  • Q4 — Added details on Ego object mask construction in Section 3.3.
Reviewer_WRKDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Affordance grounding is the task of localizing the region that enables a specific action, such as the “handle” for “hold.” Because pixel-level annotation is expensive, weakly supervised affordance grounding (WSAG) has been studied, which learns and performs affordance grounding from two images: an exocentric image showing a person using the object and an egocentric image containing only the object. Existing methods transfer knowledge by relying on feature/region alignment between the ego and exo images, and the authors point out that this works poorly under occlusion. To address this, the authors leverage additional part text information, and they achieve new state-of-the-art performance.

Critical strengths
  • Occlusion of the interacting region is, I think, one of the biggest difficulties in the weakly supervised affordance grounding setting. Addressing it is an important insight and contribution.
  • Starting from a strong baseline and using additional information, the method achieves better performance on the AGD20K Seen/Unseen splits and on HICO-IIF.
Critical weaknesses
  • The information asymmetry among the methods is the biggest issue. I believe the key point of WSAG is to enable learning without explicit part instructions or information. This submission violates that by learning from direct part text information, which makes the comparison unfair to other methods and is the strongest weakness of this submission.
  • The qualitative results appear to show grid-like artifacts, and their cause needs to be identified. Heat-map-like affordance grounding is, as far as I know, widely used in settings such as robot grasping, where such artifacts could cause difficulty in control and instability.
  • The motivation experiment in Table 1 is interesting, but I doubt it is informative enough to support the claim. First, the experimental setup structurally favors the text query. The text prompts are generated from predefined rules over verb-object pairs, and the retrieval ground truth is defined by the same verb-noun categories, so the text query effectively contains the ground-truth category information. In contrast, the visual query is a local feature pooled by the part mask, whereas the gallery consists of global features of ego images, so the units of comparison themselves do not match, regardless of viewpoint differences. Therefore, the result Part Text > Visual+Text > Visual can be largely explained by these setup differences rather than by a limitation of cross-view visual correspondence. Second, this experiment only measures category-level retrieval performance and provides no evidence that it carries over to affordance localization, which is what the paper actually aims to solve. The authors acknowledge this in Appendix F.1, but then presenting this experiment as a main contribution is excessive. The claim would be much more convincing if a localization-level diagnostic were provided, for example comparing the patch-level similarity maps obtained on ego images from the text query and from the masked visual query against the GT heatmap.
Questions for authors
  • Could you explain in detail how the Visual + Text query was constructed in the Table 1 experiment?
  • When two different text queries can be given for the same egocentric image, what does the output look like? For example, a knife supports both “hold” and “cut with.” Could you provide qualitative results for this? I am curious whether the grounding changes depending on the input text.
10 comments
AuthorsOpenReview ↗

Response to Reviewer WRKD(PART I)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We have carefully considered all comments, revised the manuscript, and conducted additional analyses to address the concerns. Our detailed point-by-point responses are provided below.

W1. Information Asymmetry and Fairness of Comparison

Thank you for raising this important concern. Our method follows PLSP in leveraging part-semantic priors during training, without requiring part-text inputs at inference. Neither method requires manually annotated part boxes or masks for the target training images.

1. Part-semantic priors in PLSP. PLSP's released implementation uses verb–object-conditioned part prompts to generate training pseudo-labels (code). It also supervises predicted part representations using part-text embeddings through a cosine-similarity loss (code). Thus, explicit part-text guidance is already employed in the WSAG setting and is not unique to our method.

2. Our distinction and supporting evidence. Our contribution lies in how Exo-instance evidence is used to transfer functional-part semantics, rather than merely introducing predefined part knowledge. FPST uses Exo evidence to select and reliability-weight candidate part-text embeddings, forming a sample-adaptive semantic prototype that guides Ego affordance localization. Table 3 (Ours vs. Text-only) further demonstrates the additional benefit of Exo-conditioned semantic selection and weighting beyond fixed textual representations, while retaining the remaining training components and Exo role gating. Unlike PLSP, FPST uses these Exo-conditioned semantic prototypes to retrieve Ego support regions and supervise the predicted probability mass, rather than directly aligning Exo and Ego visual features.

We acknowledge that methods without part-semantic priors operate under different training-information conditions. Our comparison with PLSP establishes the relevant precedent for using such priors, while the controlled ablation helps identify the contribution of our proposed mechanism.

W2. Diagnosis and Mitigation of Grid-Like Artifacts

Thank you for highlighting the grid-like artifacts and their potential impact on downstream robotic control. We have added diagnostic experiments, before/after visualizations, and quantitative evaluations in the supplementary section “Grid-Like Artifacts: Diagnosis and Gaussian Post-Processing.”

1. What causes the artifacts? The grid-like patterns are present in the model predictions before visualization. Our diagnostic experiments show that even spatially uniform features produce grid-like patterns after passing through the decoder's upsampling layers, identifying these layers as a source of the artifacts.

2. Can the artifacts be mitigated? We apply Gaussian smoothing (σ=2\sigma=2) without retraining. The results on the full AGD20K test sets are summarized below.

SplitMethodKLD ↓SIM ↑NSS ↑
SeenOriginal0.84720.52141.6081
With smoothing0.83990.52221.6178
UnseenOriginal1.06870.46331.5278
With smoothing1.05880.46421.5395

Gaussian smoothing reduces the visible artifacts while improving all six evaluation metrics, demonstrating its effectiveness as a simple post-processing strategy.

3. What about robotic control? We acknowledge that grid-like artifacts may affect downstream robotic control. While smoothing can mitigate these artifacts, our current evaluation focuses on affordance localization rather than robotic execution. Its impact on grasping performance and control stability remains to be investigated.

AuthorsOpenReview ↗

Response to Reviewer WRKD(PART II)

W3. Revised Motivation Experiment and Localization-Level Evidence

Thank you for this insightful comment. We acknowledge the limitations of the original retrieval experiment and have replaced Table 1 with a localization-level spatial matching diagnostic.

Using frozen vision–language features, we compare masked Exo visual queries and functional-part text queries against identical Ego patch features, evaluating their spatial similarity maps against GT affordance heatmaps using KLD, SIM, and NSS. Both queries are matched against identical Ego patches within the same detected object region, eliminating the local-query versus global-gallery mismatch in the original retrieval experiment.

Part Text achieves better GT agreement across all three metrics on 852 eligible Seen test tasks. Importantly, this advantage persists in the highest appearance-gap quartile (Q4) within each verb–object category., where visual correspondence becomes more challenging:

GroupQueryKLD ↓SIM ↑NSS ↑
OverallVisual2.65290.33650.8500
Part Text2.51050.34760.9894
Largest appearance differencesVisual2.90210.32400.7718
Part Text2.74530.34280.9263

These results suggest that while visual matching becomes less reliable under large cross-view appearance variations, functional-part semantics can provide complementary spatial guidance. The downstream localization benefits are further supported by the ablations in Table 3. Qualitative comparisons are shown in Figure 2, with experimental details and additional results in Appendix G. We have revised the Introduction and contribution statements accordingly.

Q1. Construction of the Visual + Text Query

Thank you for this question. In the original Table 1, the Visual + Text query was constructed by combining normalized visual features pooled from the Exo part mask with the corresponding functional-part text embeddings, followed by normalization. As explained in our response to W3, we acknowledge the limitations of the original retrieval experiment and have therefore removed Table 1, replacing it with a qualitative spatial matching diagnostic in Figure 2 and Appendix G.

Q2. Action-Conditioned Grounding for Different Text Queries

Thank you for this suggestion. In affordance grounding, different actions on the same object may correspond to different functional regions. For example, “hold” targets the handle of a knife, whereas “cut with” targets its blade.

We have updated the first two columns of Figure 9 in the supplementary material to provide qualitative results for these two queries on the same egocentric image. The model highlights the blade for “cut with” and the handle for “hold,” demonstrating that the predicted affordance grounding changes according to the input action query.

Manuscript updates

The revised manuscript includes the following additions and corrections.

  • W1 — Clarified the use of part-semantic priors in PLSP and the distinction of our Exo-conditioned semantic transfer, supported by Table 3.
  • W2 — Added artifact diagnosis, Gaussian smoothing results, and visual comparisons in Appendix H.
  • W3 — Replaced the original Table 1 with a localization-level spatial diagnostic in Figure 2 and Appendix G, and revised the Introduction and contributions.
  • Q2 — Added action-conditioned grounding comparisons on the same Ego image in Figure 10.

Thank you to the authors for their detailed responses. Below is my point-by-point evaluation.

[W0] Revision. When revising, please mark the revised text, figures, and tables, for example with color. As things stand, it is hard to tell which content was already in the original submission and which was added, and leaving changes unmarked goes against common ICLR practice. I kindly ask the authors to fix this as soon as they see this comment.

[W1] Information Asymmetry and Fairness of Comparison. I now understand that PLSP uses part information. If so, the revision should distinguish methods that use additional information from those that do not. Otherwise, readers may assume that all methods receive the same level of supervision, and, as happened in my case, the comparison may look unfair. Please state this prominently in the main text, where related work or the baselines are introduced. My suggestion is to describe this in the related work and baseline sections, and to indicate in tables such as Table 1 which information each method actually uses. For example: Cross-View-AG and LOCATE use Ego+Exo, INTRA uses Exo only, and PLSP and the proposed method use Ego+Exo+Part text.

[W2] Diagnosis and Mitigation of Grid-Like Artifacts. Thank you for analyzing the cause. The proposed remedy, Gaussian blur, is a heuristic post-processing step, and it is not clear how its standard deviation was tuned, so I find it hard to see it as a direct remedy for this artifact. A way to mitigate the artifact during training would be preferable, but since that would be a substantial revision, I will not ask for it. The authors also state that robotic control is out of scope. However, affordance grounding ultimately serves robot manipulation, so artifacts like these could hinder its most direct application. Simply calling this out of scope does not address the concern; it sidesteps it. I therefore consider this point unresolved. The authors might respond that PLSP shows similar artifacts. I am evaluating this submission on its own merits, not PLSP, so I would ask the authors not to make that argument.

[Q2] Action-Conditioned Grounding for Different Text Queries. Could you show additional qualitative results on a wider range of images, including the Seen split?

All other points I have not mentioned here have been resolved. In the current state, I believe that [W2] in particular would require substantial revision to address. I will therefore keep my current score.

Reviewer_PmCTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents IGPSG for weakly supervised affordance grounding. Built on WSAG-PLSP, the method introduces two components: Functional-Part Semantic Transfer (FPST), which converts Exo part evidence into semantic prototypes to provide supervision for the Ego domain, and Part–Verb Consensus Grounding (PVCG), which refines localization via multi-level features and spatial consensus. Experiments on AGD20K and HICO-IIF demonstrate performance improvements while enabling Ego-only inference.

Critical strengths
  1. The motivation is well-articulated: functional-part semantics provide a plausible transfer signal when cross-view visual correspondence becomes unreliable.
  2. The reliability-aware semantic aggregation and support-mass supervision constitute sensible designs to cope with noisy part evidence.
Critical weaknesses
  1. According to Section 3.3 and Appendix B, candidate parts are first identified via predefined verb-object rules, and Exo images are mainly used to detect, filter, and weight these candidates. The transferred prototypes are weighted combinations of textual features from these candidates. As such, the current results cannot clearly disentangle whether performance gains stem from interactive evidence in Exo instances or from predefined action-part knowledge and foundation model priors. Detecting a knife blade does not necessarily imply that the model understands its functional role within the ongoing interaction.

  2. The appendix does not fully specify the prompt rules or support-size constraints.

  3. The examples in Figure 2 are ambiguous. It is unclear how the prompt “open the drawer” corresponds to the image showing someone lifting a coffee cup.

  4. Eq. (17) combines detection confidence, object containment, area plausibility, and interaction-role compatibility into a single reliability score rkr_k. These components measure different properties, but the paper does not clearly explain how they are normalized or how their weights are chosen. It is therefore unclear whether their numerical values are comparable and whether the resulting score reflects the reliability of a candidate part.

  5. Detection confidence may be counted twice: rkr_k already includes a confidence-related term, while Eq. (3) multiplies rkr_k by the detection score sks_k. This could favor easily detected parts over parts that are most relevant to the queried action.

  6. Eq. (4) merges multiple part embeddings into one prototype. Even after role-based filtering, the retained candidates may refer to different regions, and averaging their embeddings could make the localization target ambiguous.

Questions for authors

Please see the weakness part.

5 comments
AuthorsOpenReview ↗

Response to Reviewer PmCT(PART I)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We have carefully considered all comments, revised the manuscript, and conducted additional analyses to address the concerns. Our detailed point-by-point responses are provided below.

W1. Distinguishing Interaction Evidence from Predefined Semantic Priors

Thank you for highlighting the distinction between predefined part priors and Exo-instance evidence. We clarify their respective roles and provide additional spatial transfer diagnostics.

1. Do the gains come from Exo evidence or predefined priors? Table 3 (rows 1–2) compares Ours with rule-based text features (“Fixed prompt,” renamed “Text-only”), while retaining the remaining training components and Exo role gating. Unlike Text-only, FPST uses Exo part detections and reliability estimates to select and weight candidate part semantics. Ours improves all six Seen/Unseen metrics, supporting the additional benefit of Exo-conditioned semantic construction beyond predefined textual priors.

2. Does part detection imply functional understanding? We agree that part detection alone does not establish functional-role understanding. FPST does not make this assumption; instead, it uses Exo-supported part semantics as auxiliary guidance for Ego affordance localization.

To further validate the spatial relevance of part semantics, we added a Spatial Transfer Diagnostic (Appendix G, “Spatial Transfer Diagnostic”). Using frozen CLIP features and identical Ego patches without training a prediction head, we compare Exo part-visual queries against rule-defined part-text queries. The following table reports their spatial agreement with AGD20K Seen affordance GT.

GroupQueryKLD ↓SIM ↑NSS ↑
Overall (N=852)Visual2.65290.33650.8500
Part Text2.51050.34760.9894
Q4 (N=466)Visual2.90210.32400.7718
Part Text2.74530.34280.9263

Q4 denotes the highest appearance-gap quartile within each verb–object category; N counts distinct Ego tasks.

Part Text consistently achieves better GT agreement across all three metrics, including under larger appearance gaps (Q4). These results support the spatial relevance of part semantics for cross-view affordance localization, while Table 3 separately supports the benefit of Exo-conditioned semantic construction over fixed textual priors.

Specifically, as described in Section 3.3(3), FPST retrieves Ego semantic support and uses pseudo-label-derived mass supervision to guide affordance prediction. Our claim therefore concerns interaction-conditioned semantic guidance for spatial localization, rather than explicit functional-role understanding.

W2. Clarification of Prompt Rules and Support-Size Constraints

Thank you for pointing out these missing details. We have expanded Appendix B.1 to provide representative verb–object-to-part prompt rules and clarify the Exo-conditioned candidate filtering procedure. For example, sip–cup activates “cup rim/opening” (effect) and “cup handle/body” (grasp), with candidate retention determined by Exo detection and mask-quality checks. We also specify the Ego support-size constraints: the top 15% of eligible object patches are selected, bounded between 2 and 8 patches, with samples containing fewer than 2 eligible patches excluded from the support-mass loss. The complete prompt mapping and implementation configurations will be released with the source code upon acceptance.

W3. Correction of the Prompt Label in Figure 2

Thank you for catching this labeling error. The original cup interaction corresponded to “sip,” rather than “open the drawer.” We have removed the mismatched label and replaced Figure 2 with revised spatial transfer examples.

AuthorsOpenReview ↗

Response to Reviewer PmCT(PART II)

W4. Normalization and Weighting of the Reliability Score

Thank you for highlighting this ambiguity. We clarify that rkr_k is a bounded evidence-quality heuristic, not a calibrated probability. As detailed in the revised Appendix B.2, all four components lie in [0,1][0,1]: detection confidence uses the bounded VLPart score sk∈[0,1]s_k\in[0,1] directly, containment measures the fraction of the part mask inside the object, area plausibility is enforced through validity filtering, and role compatibility uses fixed role priors.

We assign weights (0.42,0.28,0.20,0.10)(0.42,0.28,0.20,0.10), prioritizing detection confidence and object association. To directly examine this choice, we conducted a hyperparameter sensitivity analysis comparing the proposed weighting with equal component weights under the same training settings. The results are summarized below.

Reliability weightsAGD20K Seen KLD ↓SIM ↑NSS ↑HICO-IIF KLD ↓SIM ↑NSS ↑
Ours (0.42, 0.28, 0.20, 0.10)0.84720.52141.60810.80590.57971.9503
Equal (0.25 each)0.88410.51361.57750.83350.57121.9164

The proposed weights outperform equal weights on all six metrics. Together with the "w/o reliability" ablation in Table 3, this supports our reliability design.

W5. Clarification of Detection Confidence in the Reliability Score

Thank you for pointing out this issue. After carefully checking the source code, we confirmed that Eq. (3) contained a typographical error in the original manuscript. The actual implementation uses ek=rke_k = r_k, rather than ek=skrke_k = s_k r_k as mistakenly written. We have corrected the formulation in the revised manuscript to ensure consistency with the implemented method. Thus, detection confidence is incorporated only once through the detection-confidence component of rkr_k, without an additional detection-score multiplier.

W6. Ambiguity in Multi-Candidate Prototype Aggregation

Thank you for highlighting this potential ambiguity. Eq. (4) employs reliability-weighted aggregation of multiple candidate part embeddings rather than relying on a single candidate. Role and quality filtering reduce irrelevant candidates, while reliability weighting limits the influence of less reliable detections. This design preserves evidence from multiple plausible candidates and reduces dependence on a single potentially incorrect detection. The resulting prototype provides semantic guidance for Ego support retrieval, rather than directly transferring multiple Exo spatial regions.

To validate this design, we have added an ablation study in Appendix E.1, “Weighted Aggregation versus Single-Candidate Selection.” Both variants share the same candidate pool, weighting scores, loss function, and training settings, differing only in the prototype construction strategy. The results are summarized below.

Prototype constructionSeen (KLD↓ / SIM↑ / NSS↑)Unseen (KLD↓ / SIM↑ / NSS↑)
Weighted aggregation (Ours)0.8472 / 0.5214 / 1.60811.0687 / 0.4633 / 1.5278
Top-1 candidate0.8907 / 0.5157 / 1.55371.0907 / 0.4529 / 1.4931

Weighted aggregation consistently outperforms Top-1 selection across all six metrics, supporting the benefit of integrating multiple plausible candidates. Although aggregation does not guarantee that all retained candidates correspond to the same functional region, role and quality filtering, together with reliability weighting, help mitigate potential semantic ambiguity.

Manuscript updates

The revised manuscript includes the following additions and corrections.

  • W1 — Added clarification of Exo semantic guidance and its distinction from predefined priors, with supporting ablation results in Table 3 and the Spatial Transfer Diagnostic in Appendix G.
  • W2 — Added prompt rules in Appendix B.1 and support-size constraints in the FPST appendix.
  • W3 — Removed the mismatched prompt label and replaced Figure 2 with revised spatial transfer examples.
  • W4 — Added reliability-score normalization and weighting details in Appendix B.2.
  • W5 — Corrected Eq. (3) to ek=rke_k = r_k.
  • W6 — Added a prototype aggregation ablation in Appendix E.1.
AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer PmCT,

Thank you again for your thoughtful review and constructive suggestions. We have carefully addressed your concerns in our response and revised the manuscript accordingly. In particular, we have conducted additional analyses to further examine the spatial relevance of transferred part semantics and clarify the respective roles of predefined semantic priors and Exo-conditioned evidence. We have also expanded the supplementary material to provide further implementation details and corrected the figure presentation.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses and the corresponding revisions. We sincerely hope that the additional experiments and clarifications address your concerns. If any questions or uncertainties remain, we would be very happy to provide further explanations.

Thank you once again for your time and consideration. Your feedback has been very helpful in improving our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.