- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Interaction-Grounded Part Semantic Guidance (IGPSG) for weakly supervised affordance grounding. Instead of relying solely on cross-view visual correspondence, the method transfers interaction-relevant functional-part semantics from exocentric to egocentric images. Specifically, Functional-Part Semantic Transfer (FPST) constructs semantic prototypes using pretrained foundation models and provides additional supervision for egocentric localization. Part–Verb Consensus Grounding (PVCG) further refines affordance predictions using multi-level visual features and spatial consensus between part and verb representations. Experiments on AGD20K and HICO-IIF demonstrate improvements over existing methods in both Seen and Unseen settings.
Critical strengths
-
The paper addresses the limitations of direct visual correspondence between exocentric and egocentric images by exploiting functional-part semantics, which can provide complementary guidance under substantial appearance and viewpoint variations.
-
The proposed method achieves improvements over existing approaches on AGD20K and HICO-IIF, including both Seen and Unseen settings. The component-wise ablation study also supports the contributions of FPST and PVCG.
Critical weaknesses
-
Limited novelty. The novelty over WSAG-PLSP is not sufficiently established, as PLSP already incorporates part-level semantic priors and pretrained foundation models such as VLPart and SAM. The paper should more clearly explain what fundamentally distinguishes the proposed approach from PLSP and why the additional mechanisms lead to better performance.
-
Unclear complexity-performance trade-off. The additional computational and architectural complexity is not adequately analyzed, making it difficult to assess whether the performance gains justify the increased complexity.
-
Numerous hyperparameters and insufficient analysis. The proposed method introduces many hyperparameters and heuristic design choices, including multiple temperature parameters, filtering thresholds, reliability weights, and so on. However, the paper does not sufficiently explain how these values were selected or provide sensitivity analyses. It is also unclear whether these hyperparameters were tuned separately for each dataset or whether the same configuration generalizes across datasets. Given the number of design choices, these analyses are important for evaluating the robustness and practicality of the proposed method.
Questions for authors
-
What exactly does the "Base model" in Table 3 refer to? How does it differ from the full WSAG-PLSP model?
-
What percentage of the training samples have valid semantic candidates and receive FPST supervision?
-
Are there failure cases caused by inaccurate part detection or segmentation from VLPart and SAM? How sensitive is the method to such errors?
-
How is the egocentric object mask in Eq. (6) obtained?
6 comments
Response to Reviewer bHL3(PART II)
W3. Numerous Hyperparameters and Insufficient Analysis
Thank you for raising this concern. We have conducted additional sensitivity experiments to assess the key hyperparameter choices and evaluated a shared configuration on AGD20K Seen and HICO-IIF. The experimental setup and findings are summarized below.
1. Parameter selection and sensitivity analysis.
We examine five key hyperparameter groups: the reliability cutoff , reliability weights , prototype temperature , transfer weight , and PVCG temperature . We select these groups because they directly control candidate filtering, evidence scoring, semantic aggregation, transfer supervision, and spatial attention, respectively, covering the main mechanisms introduced by FPST and PVCG.
We vary one group at a time while keeping the remaining settings fixed. All configurations follow the same training and checkpoint-selection protocol as the main experiments.
Among the tested configurations, the shared default—, , , , and —achieves the best score in five of the six dataset–metric combinations. The only exception is a marginal improvement in AGD20K Seen SIM with , accompanied by worse KLD and NSS. These results support the effectiveness and stability of the shared default within the tested parameter ranges.
| Factor | Setting | AGD20K Seen KLD ↓ | SIM ↑ | NSS ↑ | HICO-IIF KLD ↓ | SIM ↑ | NSS ↑ |
|---|---|---|---|---|---|---|---|
| Default | Shared | 0.8472 | 0.5214 | 1.6081 | 0.8059 | 0.5797 | 1.9503 |
| Prototype temp. () | 0.10 | 0.8625 | 0.5182 | 1.5932 | 0.8174 | 0.5763 | 1.9365 |
| Prototype temp. () | 0.20 | 0.8598 | 0.5194 | 1.6002 | 0.8136 | 0.5778 | 1.9421 |
| Stricter cutoff () | 0.75 | 0.8786 | 0.5148 | 1.5824 | 0.8293 | 0.5726 | 1.9198 |
| Stricter cutoff () | 0.85 | 0.9061 | 0.5097 | 1.5503 | 0.8542 | 0.5608 | 1.8841 |
| Reliability weights () | Equal (0.25 each) | 0.8841 | 0.5136 | 1.5775 | 0.8335 | 0.5712 | 1.9164 |
| Transfer weight () | 0.005 | 0.8702 | 0.5173 | 1.5886 | 0.8218 | 0.5749 | 1.9306 |
| Transfer weight () | 0.020 | 0.8758 | 0.5168 | 1.5802 | 0.8246 | 0.5738 | 1.9257 |
| PVCG temp. () | 0.05 | 0.8563 | 0.5221 | 1.6014 | 0.8127 | 0.5782 | 1.9432 |
| PVCG temp. () | 0.20 | 0.8715 | 0.5142 | 1.5871 | 0.8205 | 0.5740 | 1.9284 |
For the “Equal” setting, the four reliability components—detection confidence, object containment, area plausibility, and interaction-role prior—each receive a coefficient of 0.25.
2. Cross-dataset consistency and practicality.
The same default configuration is applied to AGD20K Seen and HICO-IIF without dataset-specific tuning, including the temperatures, filtering thresholds, reliability coefficients, and transfer weight. The shared default achieves consistently strong results on both datasets, supporting its practical applicability without dataset-specific adjustment.
Response to Reviewer bHL3(PART III)
Q1. Clarification of the Base Model in Table 3
Thank you for this question. The “Base model” in Table 3 of the original submission (now Table 2 in the revised manuscript) refers to our reproduction of the full WSAG-PLSP model, retaining its original architecture and training objectives, without FPST or PVCG. It is not a simplified version of WSAG-PLSP.
The WSAG-PLSP results in Table 1 are taken from the original publication, whereas Table 2 reports our own experimental runs under a consistent ablation setup. Using this reproduced baseline, Table 2 isolates the contributions of FPST and PVCG by progressively introducing Exo-conditioned semantic transfer and Ego-side localization refinement under matched experimental conditions.
Q2. FPST Supervision Coverage
Thank you for this question. We quantified FPST supervision coverage on both AGD20K training splits. Valid candidate coverage is the percentage of all training instances with at least one valid Exo semantic candidate, while effective supervision coverage is the percentage of all training instances contributing to the FPST support-mass loss after activation.
| Training split | Training instances | Valid candidates | Effective FPST supervision |
|---|---|---|---|
| Seen | 6,929 | 86.23% | 72.20% |
| Unseen | 4,815 | 85.81% | 72.32% |
These results show that valid semantic candidates are available for over 85% of training instances, while approximately 72% of all training instances receive effective FPST supervision after activation.
Q3. Failure Cases and Sensitivity to VLPart/SAM Errors
Thank you for raising this important concern. We acknowledge that inaccurate detections from VLPart or segmentation errors from SAM can affect the quality of candidate part evidence and, consequently, the transferred semantic prototypes. For example, missing a relevant functional part may prevent FPST supervision, while an incorrectly detected part may introduce noisy semantic guidance.
1. How does FPST handle detection and segmentation errors? FPST employs role-based filtering and reliability-weighted aggregation to mitigate the influence of unreliable candidates. Specifically, candidate reliability considers detection confidence, object containment, area plausibility, and interaction-role compatibility. These mechanisms reduce dependence on individual detections but cannot entirely eliminate errors.
2. How sensitive is the method to such errors? Our existing ablations provide indirect evidence of robustness. The “w/o reliability” ablation in Table 3 demonstrates the benefit of reliability weighting, while Appendix E.1 shows that weighted aggregation outperforms Top-1 selection across all six evaluation metrics. These results suggest that integrating multiple reliability-weighted candidates is preferable to relying on a single detection.
Q4. Construction of the Egocentric Object Mask
Thank you for this question. The Ego object mask in Eq. (6) is constructed from precomputed VLPart object detections. Specifically, we rasterize the selected object bounding box onto the Ego patch grid, marking overlapping patches as valid. This coarse mask restricts semantic support retrieval to the detected object region without requiring ground-truth segmentation. We have clarified this procedure in Section 3.3, “Ego-space Semantic Support Retrieval.”
Manuscript updates
The revised manuscript includes the following additions and clarifications.
- Q4 — Added details on Ego object mask construction in Section 3.3.
Response to Reviewer bHL3(PART I)
We greatly appreciate the reviewer's careful assessment and valuable suggestions. We have taken each concern seriously and made corresponding revisions, including methodological clarifications and supplementary experiments. We respond to each point in detail below.
W1. Novelty Beyond Visual Feature Alignment
Thank you for raising this important concern. We acknowledge that WSAG-PLSP already incorporates part-level semantic priors and pretrained foundation models such as VLPart and SAM. Our novelty lies not in using these models, but in exploring a complementary approach to cross-view affordance learning beyond conventional visual feature alignment.
Specifically, while PLSP employs part-level semantic priors to refine weak supervision, our IGPSG transfers Exo-conditioned functional-part semantics to Ego observations through sample-adaptive semantic prototypes. FPST uses Exo-instance evidence to select and reliability-weight candidate part semantics, which then provide auxiliary spatial guidance for Ego affordance localization. This reduces dependence on direct visual correspondence under cross-view variations in object shape, scale, and texture. Unlike conventional cross-view feature alignment, FPST retrieves semantic support in the Ego image and supervises the predicted probability mass over this support, without enforcing direct Exo–Ego feature correspondence. PVCG further refines the Ego query through multi-level visual evidence and part–verb spatial consensus.
The key distinction is therefore how functional-part semantics are constructed from Exo evidence and transferred to Ego observations, rather than the introduction of part-level priors themselves. The component ablations in Table 2, together with the controlled Text-only and Direct alignment comparisons in Table 3, support the distinct contributions of semantic transfer and Ego-side grounding.
W2. Complexity–Performance Trade-off
Thank you for highlighting this concern. We have conducted a complexity and efficiency analysis comparing IGPSG with WSAG-PLSP under identical inference settings (RTX 3090, FP32, 224 × 224 inputs).
1. Performance gains with limited overhead. Our method increases total parameters by only 1.5%, inference parameters by 0.6%, and estimated computation by 2.0%, with an additional 2.24 ms per batch in inference latency. Meanwhile, compared with WSAG-PLSP, IGPSG achieves 4.8%/7.3% reductions in KLD, 2.2%/6.0% improvements in SIM, and 3.9%/7.7% improvements in NSS on Seen/Unseen, respectively.
2. Source of additional complexity. FPST operates only during training and introduces no additional inference computation. The modest inference overhead primarily arises from PVCG's multi-level part-conditioned attention and part–verb spatial consensus. Neither Exo inputs nor VLPart/SAM detection and segmentation are required at inference.
These results demonstrate that IGPSG achieves improved affordance localization with limited additional inference cost.