acceptodds
Withdrawn from ICLR 2027

Action-Aware Spatial Supervision for Weakly Supervised Affordance Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Weakly supervised affordance grounding (WSAG) aims to localize action-relevant object regions using only image-level supervision. Existing methods exploit exocentric human–object interaction images to support affordance localization in egocentric images, with recent approaches further leveraging vision–language foundation models to generate dense pseudo labels for the egocentric view. However, exocentric cues are still aggregated over object regions without explicit guidance from the queried action, while egocentric pseudo labels and predictions may remain spatially imprecise. We therefore propose a unified framework that improves spatial reliability across exocentric transfer, pseudo-label learning, and prediction optimization. First, we introduce an Action-Aware Exocentric Pooling (AAEP) module that combines queried-action-guided feature aggregation with fine-grained exocentric interaction pseudo-mask supervision, emphasizing action-relevant interaction regions. Second, we develop a Reliability-Calibrated Pseudo-Label Refinement strategy that performs action-aware refinement of egocentric pseudo labels and adaptively fuses the refined labels with the originals. Finally, we introduce a Peak-Anchored Support Expansion (PASE) Loss that encourages compact, appearance-consistent support around reliable peaks, reducing overly concentrated responses. Experiments on AGD20K demonstrate that our model achieves state-of-the-art performance, with a 4.2% reduction in KLD and gains of 1.6% in SIM and 4.1% in NSS on the Unseen split.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Weakly Supervised Affordance Grounding, Cross-View Learning, Affordance Localization, Pseudo-Label Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_d28zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper extends WSAG-PLSP for weakly supervised affordance grounding with three components: Action-Aware Exocentric Pooling (AAEP), reliability-calibrated pseudo-label refinement, and Peak-Anchored Support Expansion (PASE). The method aims to improve spatial precision in exocentric transfer, pseudo-label supervision, and final prediction, and reports improved results on the Seen and Unseen splits of AGD20K.

Critical strengths

The paper addresses a reasonable limitation of existing weakly supervised affordance grounding methods: object-level exocentric cues may be too coarse when different actions correspond to different functional regions of the same object. The proposed three-stage spatial refinement framework is conceptually coherent. The method achieves strong quantitative results on AGD20K, improving over the reported WSAG-PLSP results on both Seen and Unseen splits.

Critical weaknesses
  1. The current ablations are not sufficient to tell where the improvement actually comes from. Since the method is built directly on WSAG-PLSP, I would expect a controlled baseline using the authors' own code and training setup with all three new components removed. Instead, Table 2 only removes one component at a time, while the WSAG-PLSP numbers in Table 1 are copied from the original paper. AAEP is also not cleanly ablated: it adds both action-conditioned pooling and VLPart/SAM-based spatial supervision, so removing AAEP removes both at once. I think the paper needs at least a same-setting WSAG-PLSP baseline, an action-aware pooling variant without the extra pseudo-mask supervision, and a variant with the pseudo-mask supervision but without action-aware pooling. Otherwise, it is hard to know which part is actually responsible for the reported gain.

  2. The pseudo-label refinement contains many handcrafted choices, but these choices are not really validated. The method manually assigns different actions to spatial templates, such as upper regions for support actions, object edges for grasping, and ends/edges for tool-use actions. Its reliability score is then mainly determined by the area of the refined mask. These rules may work on AGD20K, but the current experiments do not show whether they are necessary or robust. I would like to see at least a refinement variant without the handcrafted action prior, a comparison between fixed fusion and the proposed reliability-based fusion, and sensitivity results for the main coverage/reliability thresholds. Without these experiments, it is difficult to tell whether the improvement comes from the proposed refinement idea or from dataset-specific tuning.

Questions for authors

Can the authors provide a controlled WSAG-PLSP baseline under exactly the same implementation and training setup, with AAEP, label refinement, and PASE all disabled?

Can the authors disentangle action-conditioned pooling from the additional VLPart/SAM pseudo-mask supervision in AAEP, e.g., by evaluating each component independently?

How were the handcrafted spatial templates and thresholds selected, and how sensitive are the results to these choices?

Reviewer_tUpzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the task of weakly supervised affordance grounding, which aims to predict affordance regions corresponding to the action query in an object image. During training, pixel-level groundtruth annotations are unavailable, and only exocentric images depicting human-object interactions are provided as guidance. Building upon existing pipelines based on pseudo-labeling and cross-view feature alignment, the authors introduce three improvements. First, AAEP leverages action features to guide the extraction of exocentric image features, and employs object part masks generated by SAM to supervise the attention maps. Second, the pseudo labels for egocentric images undergo multiple refinement steps based on reliability calibration. Third, a novel PASE loss is proposed to account for uncertainty in pseudo labels and suppress isolated peaks in the predictions. Experiments on the AGD20K dataset demonstrate that the proposed method achieves state-of-the-art performance, with ablation studies further validating the effectiveness of each individual component.

Critical strengths

The proposed method achieves strong empirical performance, and the effectiveness of its components is demonstrated through ablation studies.

Critical weaknesses

My primary concern is the limited novelty of the proposed approach. The paper largely follows the existing pseudo-labeling framework, introducing only incremental modifications to several components, many of which appear relatively trivial or engineering-driven. For example, the authors propose a multi-stage pseudo-label refinement strategy involving numerous heuristically chosen hyper-parameters and hand-crafted rules. This procedure lacks a clear underlying insight and may primarily represent a set of dataset-specific heuristics with limited generalizability and broader significance.

Furthermore, the validity of certain design choices is questionable. For instance, Eq. (7) requires segmenting the relevant object part in exocentric images. However, the target part is often occluded by the human, which may harm the accuracy of the interaction mask and the effectiveness of the corresponding attention loss. The "Reliability-calibrated Candidate Refinement" module employs a fixed threshold on the part/object area ratio, despite the fact that the relative sizes of affordance regions can vary substantially across different objects and actions.

Questions for authors

How does each stage of the refinement process affect the pseudo-labels? It might be helpful to provide some visualizations.

How sensitive is the proposed method to its key hyper-parameters?

Reviewer_Ss5AOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper builds on WSAG-PLSP (Xu & Mu, 2025) for weakly supervised affordance grounding and adds three training-time components:

  1. AAEP: action-conditioned exocentric attention pooling, supervised by VLPart/SAM interaction pseudo-masks.
  2. Pseudo-label refinement: coverage-based refinement of egocentric pseudo labels using verb-specific spatial templates.
  3. PASE: a loss that pushes predictions to cover appearance-consistent regions around pseudo-label peaks.

It reports the best results among the listed baselines on the AGD20K benchmark.

Critical strengths

The motivation is reasonable: different actions engage different parts of the same object, so conditioning exocentric aggregation on the queried action is a sensible idea. Beyond this, I find the strengths limited.

Critical weaknesses

1. The manuscript is not self-contained, and key details are missing

It is difficult to understand how the full method is trained and run without consulting Xu & Mu (2025).

  • Table 1 lists "WSAG-PLSP baseline" and "WSAG-PLSP full" without defining the difference or stating which one this method builds on. So it is unclear how much of the gain comes from the proposed modules.
  • Role of f_act unclear. The paper never explains how f_act enters cross-view transfer or which loss consumes it.
  • Objective underspecified. L_base is described only as the "original" objectives.

The base architecture, the full training objective, and the role of f_act should be described within the paper itself.

2. So many hand-set hyperparameters, with no sensitivity analysis and no stated selection protocol

These include the coverage thresholds (0.62/0.35/0.45), λ, γ, the reliability step function (0.45/0.80/0.25), the PASE thresholds and caps (0.12, 0.035, 8%, 1.8%, 20%), τ_a, τ, the loss weights, and the activation and warm-up epochs. η, m_min, and m_max (Eq. S18) are listed but never given values. The verb-specific templates (Table S2) are also hand-designed. None of these values is ablated, and the paper does not say how they were chosen. Since AGD20K has no standard validation split, tuning on the test set cannot be ruled out. This is especially concerning given the small margins in point 1. Please describe the selection protocol and report sensitivity for the main thresholds.

3. Neither the main results nor the ablations are shown to be statistically meaningful.

The state-of-the-art claim rests on small margins over WSAG-PLSP full, and the ablation in Table 2 likewise rests on small differences between variants. The paper reports no standard deviations, multi-seed results, or significance tests for either. It is therefore unclear whether the improvement over the baseline, or the contribution attributed to each module, exceeds run-to-run variance. The large number of manually tuned hyperparameters makes this concern more serious.

4. The evaluation is limited to a single dataset.

All results are on AGD20K, and the Unseen split is a held-out split of the same dataset rather than a true distribution shift. It is therefore unclear whether they transfer to other objects, actions, or image distributions.

Questions for authors

How were the more than 20 hand-set hyperparameters selected? Can you provide sensitivity plots for at least τ_obj, γ, and the PASE support caps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.