acceptodds
Under review as a conference paper at ICLR 2027

Efficient 3DGS-Based Affordance Reasoning via Semantically Structured Shape Tokens

Abstract

3D affordance reasoning requires accurately mapping high-level interaction intent to fine-grained local functional regions. However, existing 3DGS-based affordance methods typically perform cross-modal interaction directly between spatially organized low-level Gaussian features and textual intent, without effectively modeling geometric relationships through hierarchical semantic structures. This can lead to incomplete localization and local fragmentation in object structures, while also introducing substantial computational redundancy. Meanwhile, reliance on low-level spatial patterns limits the abstraction of transferable functional structures and cross-category generalization. To address this issue, we propose ST-Afford, an efficient framework for 3DGS-based affordance reasoning that organizes dense Gaussian geometry into compact semantically structured representations and further dynamically selects task-relevant geometric information based on interaction intent. Specifically, we introduce a Semantically Structured Geometry Extractor, which compresses and encodes high-dimensional 3DGS features into compact shape-prior tokens. Based on these tokens, an Intent-Guided Dynamic Fusion module uses interaction text as a condition to dynamically select task-relevant geometric semantic information and fuse it with local Gaussian features, enabling accurate localization of fine-grained functional regions from high-level interaction intent. Experimental results show that ST-Afford substantially reduces inference overhead while generating more complete and accurate affordance reasoning, and achieves stronger zero-shot cross-category generalization on unseen categories.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Affordance Reasoning, 3D Gaussian Splatting, Structured Shape Representation, Intent-Guided Fusion, Zero-Shot Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qoJYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ST-Afford, a 3DGS-based framework for 3D affordance reasoning. It compresses dense Gaussian features into semantically structured shape tokens and dynamically selects and fuses relevant geometric information according to textual intent, aiming to improve affordance grounding accuracy, inference efficiency, and cross-category generalization.

Critical strengths
  1. ST-Afford organizes dense Gaussian features into structured representation and recovers Gaussian-level predictions through text-conditioned fusion, providing a reasonable approach to improving the efficiency of 3DGS-based affordance reasoning.

  2. Extensive experiments demonstrate that ST-Afford achieves superior performance on both the seen-category and unseen-category datasets.

Critical weaknesses
  1. The experimental baselines are insufficient. IAGNet and LASO are from around 2024, while recent methods specifically targeting affordance generalization, such as QueryMe[1] or HAMMER[2], are not compared. Without these comparisons, the claimed improvements in grounding accuracy and cross-category generalization are not validated against the current state of the art.
  2. The evidence for cross-category generalization is limited. The unseen-category evaluation is compared only with 3DAffordSplat, without further analysis of unseen categories. This is insufficient to support the paper’s claim of broad zero-shot cross-category generalization.

[1] Zhao W, Li R, Liu J, et al. QueryMe: Query-Driven Open-Vocabulary 3D Object Affordances Grounding from Multimodal Evidence[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 2603-2612.

[2] Yao L, Chen Y, Su Y, et al. HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Questions for authors
  1. Can you provide fair comparisons with recent affordance-generalization methods such as QueryMe or HAMMER, under a consistent evaluation protocol, to validate the claimed gains in grounding accuracy and cross-category generalization?

  2. Beyond comparing only with 3DAffordSplat on the unseen-category set, can you provide further analysis of unseen categories to better support the claim of broad zero-shot cross-category generalization?

3 comments

We sincerely thank Reviewer qoJY for their careful reading and positive recognition of our framework's efficiency, token design, and superior performance across both seen and unseen benchmarks. We appreciate the opportunity to address your concerns regarding additional state-of-the-art baselines and cross-category generalization analysis. All corresponding revisions in our updated manuscript are highlighted in blue.


1. Expanding Baseline Comparisons: HAMMER and QueryMe (CW1 & Q1).

... The experimental baselines are insufficient... recent methods specifically targeting affordance generalization, such as QueryMe or HAMMER, are not compared ...

A: We fully agree that benchmarking against recent advanced multimodal architectures is crucial. Here is how we addressed both suggested baselines:

  • Regarding QueryMe: To date, QueryMe has not publicly released its official source code or pre-trained model weights, precluding an accurate and fair reproduction on our 3DGS dataset. We have explicitly noted this limitation in Sec. 4.3 and Appendix D.
  • Rigorous Adaptation of HAMMER: To provide a rigorous comparison, we adapted HAMMER to our 3DGS evaluation protocol. To ensure a strictly fair comparison without altering HAMMER's core learning capacity, we aligned its input pipeline by extending its geometric encoder from 3D coordinates to accept the identical 10D Gaussian physical attributes (μ,s,q\mu, s, q) while keeping all other components and training schedules unmodified.

We evaluated the adapted HAMMER alongside 3DAffordSplat and ST-Afford on our zero-shot Unseen test set (175 instances across held-out object categories):

MethodModality / ArchitectureUnseen MAE ↓\downarrowUnseen SIM ↑\uparrowUnseen AUC ↑\uparrowUnseen IoU ↑\uparrow
3DAffordSplatNative 3DGS0.21550.45020.79470.2424
HAMMER (Adapted 10D)3DGS 10D + MLLM0.24650.34730.76230.1740
ST-Afford (Ours)3DGS + SSGE + IGDF0.19180.51540.82370.3793

As shown above and visualized in Fig. 4, while adapted HAMMER achieves competitive scores on seen objects, its zero-shot generalization drops drastically on unseen geometries (0.1740 IoU). Why does HAMMER degrade on 3DGS compared to standard point clouds? Unlike uniformly surface-sampled point clouds, 3DGS consists of non-uniform, continuous volumetric primitives optimized for radiance rendering—containing dense clusters in high-frequency texture regions and sparse primitives on flat surfaces. Point-cloud backbones rely heavily on isotropic neighborhood grouping (e.g., FPS and ball queries), which suffers from a severe modality gap when handling non-uniform 3DGS distributions on unseen geometries. In contrast, ST-Afford chooses 3DGS for its continuous surface fidelity and rich appearance/opacity cues, and uses SSGE to project these irregular primitives into structured, transferable shape tokens—achieving 0.3793 Unseen IoU (more than double HAMMER's performance).


2. Deepening Analysis on Zero-Shot Cross-Category Generalization (CW2 & Q2).

... The evidence for cross-category generalization is limited... without further analysis of unseen categories ...

A: We appreciate this suggestion. Beyond the quantitative improvements in Table 2, our zero-shot unseen benchmark is specifically designed to test whether models can recognize functional structures (such as handles, graspable rims, and trigger regions) across objects with entirely different global shapes (e.g., transferring grasping from seen daily objects to unseen instances like pistols, remotes, microphones, and baskets).

  • Why ST-Afford Generalizes Better: Without structural tokenization, baseline methods rely heavily on global shape context, causing severe localization shifts when global geometry changes (as seen in the qualitative heatmaps of Fig. 4). ST-Afford overcomes this by using SSGE to construct ordered, semantically consistent shape priors and IGDF to dynamically query task-relevant geometry, enabling robust part-level transfer across diverse object categories.

Specifically, we updated the following sections, highlighted in blue in the revised paper:

  • Sec. 2.1 & Sec. 4.3: Expanded related work and detailed the analysis of zero-shot cross-category generalization.
  • Table 2 & Fig. 4: Integrated the adapted HAMMER quantitative metrics and qualitative visual comparisons on the Unseen dataset.
  • Appendix D: Added a dedicated protocol description for the 10D geometry adaptation of HAMMER.

Dear Reviewer qoJY,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

Dear Reviewer,

Thank you again for your valuable comments. Since the discussion period will end on November 18, we really hope you can let us know whether the revised version addresses your main concerns. We also welcome further discussion with you.

Best regards, The Authors

Reviewer_TdUsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ST-Afford, an efficient framework for 3DGS-based affordance reasoning. Instead of performing cross-modal interaction directly between dense low-level Gaussian features and textual intent, the method first organizes the dense Gaussian geometry into a compact, semantically structured representation, and then dynamically selects task-relevant geometric information conditioned on the interaction intent. A Semantically Structured Geometry Extractor compresses high-dimensional 3DGS features into compact ordered shape-prior tokens, and an Intent-Guided Dynamic Fusion module uses interaction text as a condition to select task-relevant geometric-semantic information and fuse it with local Gaussian features, enabling accurate grounding of high-level interaction intent to fine-grained functional regions via a lightweight point-wise head. Experiments show that ST-Afford substantially reduces inference overhead while producing more complete and accurate affordance predictions.

Critical strengths
  1. The efficiency gain is obtained without sacrificing accuracy: all four metrics improve simultaneously. The underlying mechanism is transparent, as cross-modal interaction is conducted against a fixed set of 512 tokens rather than against every primitive, and the heavy spatial decoder employed by prior 3DGS pipelines is removed.
  2. The token design constitutes a principled adaptation of existing techniques. Prefix-based semantic ordering originates from generative shape tokenization (LoST); retaining only the encoder while discarding the generator is appropriate for a perception task, and this combination has not previously been applied to 3DGS affordance reasoning.
  3. The experimental evaluation is rigorous and transparent. Each component is isolated in the ablation study, and the per-category results reported in the appendix candidly reveal the failure cases, such as large planar functional regions (beds, chairs, and support surfaces).
Critical weaknesses
  1. The training procedure is cumbersome. It requires three separate stages and depends on pre-computed DiffGS tri-plane features; the framework is therefore not end-to-end, and the aggregate training cost is not reported.
  2. The zero-shot evaluation is limited in scope. It comprises 175 instances, covers only the grasp and wrap-grasp affordances, and compares against a single baseline, which is insufficient to substantiate a broad claim of cross-category generalization.
  3. The paper provides no comparison with MLLM-based methods. SeqAfford and 3D-AffordanceLLM are cited but not evaluated, and open-vocabulary capability is not assessed.
Questions for authors
  1. The method is not validated on physical robots, which weakens its practical significance for affordance learning. Furthermore, no FLOPs, parameter counts, or per-module latency figures are provided beyond a single FPS measurement.
3 comments

We sincerely thank Reviewer TdUs for their encouraging recognition of our efficiency gains, principled token design, and rigorous component-wise ablation. We appreciate the opportunity to address your questions regarding the training cost, evaluation scope, MLLM comparisons, and hardware computational profiling. All corresponding revisions in our updated manuscript are highlighted in blue.


1. Total Training Cost and Pipeline Breakdown (CW1).

... The training procedure is cumbersome... and the aggregate training cost is not reported ...

A: We agree that reporting the full training budget is essential. In our framework, semantic pre-training is a one-time, task-agnostic investment over 23,142 objects that builds reusable geometric shape-prior tokens for downstream tasks. On NVIDIA A100 GPUs, the exact stage-wise GPU-hour breakdown is:

  • Stage 1a — RIDA Semantic Encoder Pre-training: 100 epochs on 1×A1001 \times \text{A100} GPU (≈\approx 18.5 GPU-hours).
  • Stage 1b — SSGE Ordered Tokenizer Pre-training: 200 epochs on 2×A1002 \times \text{A100} GPUs (≈\approx 38.4 GPU-hours). (Combined Semantic Pre-training: 56.9 GPU-hours).
  • Stage 2 — Downstream Affordance Finetuning: 200 epochs on 2×A1002 \times \text{A100} GPUs with LoRA (r=32r=32) applied to IGDF projections (≈\approx 29.15 GPU-hours).

Thus, the aggregate training cost for the entire pipeline is 86.1 GPU-hours. Furthermore, because the pre-trained SSGE weights are frozen/adapted via LoRA, adapting ST-Afford to new affordance datasets requires only the lightweight downstream stage (29.15 GPU-hours). We have added this complete breakdown to Appendix B.4 (Table 5) in the revised paper.


2. Expanding Evaluation Scope and Baseline Comparisons (CW2 & CW3).

... The zero-shot evaluation is limited in scope... and lacks comparison with MLLM-based methods (e.g., SeqAfford, 3D-AffordanceLLM) ...

A: We appreciate this suggestion and have expanded our evaluation from two aspects:

  • Expanded Unseen Benchmark (HAMMER Comparison): To strengthen cross-category generalization, we newly incorporated HAMMER [1] (a recent multimodal affordance baseline) into our 3DGS protocol using identical 10D geometric inputs. As shown in Table 2 and Fig. 4, ST-Afford achieves an Unseen IoU of 0.3793, significantly outperforming both 3DAffordSplat (0.2424) and adapted HAMMER (0.1740).
  • Discussion with MLLM-based Methods: While methods like SeqAfford and 3D-AffordanceLLM leverage powerful MLLM backbones for open-vocabulary reasoning, they typically operate over text/2D prompts or coarse bounding boxes, lacking native dense 3DGS geometric alignment. We have expanded our Related Work (Sec. 2) to explicitly discuss these MLLM paradigms and contrast them with our structural tokenization approach.

3. Comprehensive Hardware Profiling: FLOPs, Parameters, Latency, and Memory (Q1).

... No FLOPs, parameter counts, or per-module latency figures are provided beyond a single FPS measurement ...

A: To address the lack of granular profiling, we have conducted rigorous benchmarking on an NVIDIA A100 GPU (N=32,768N=32,768 Gaussian primitives). The complete hardware profile is summarized below and included in Appendix B.4 (Table 5):

Hardware MetricValue / ScaleMeasurement Protocol
Total Parameters223.8 MEntire model (SSGE + IGDF + PAGH)
Computational Complexity214.5 GFLOPsAt batch size 1 (N=32,768N=32,768)
Peak GPU Memory1,996 MiBBatch size 1 inference footprint
Per-Sample Latency (Batch 1)50.87 msPure-forward execution (warmup=20\text{warmup}=20, 100 iterations)
Throughput (Batch 1 / 16)19.66 / 27.21 FPSSynchronized CUDA events (excluding I/O & metrics)
  • Protocol Clarification: We also clarified the distinction between our system-level evaluation throughput ($\sim$11.74 FPS, which includes dataset I/O and dense online metric calculations like IoU/AUC) and our pure-forward hardware limit (19.66 FPS at batch 1 / 27.21 FPS at batch 16).
  • Why ST-Afford is Efficient: By compressing 32,768 primitives into K=512K=512 shape tokens via SSGE, we replace heavy O(N2)\mathcal{O}(N^2) dense cross-modal decoders with lightweight point-to-token cross-attention (O(N×K)\mathcal{O}(N \times K)), achieving real-time performance.

Specifically, we updated the following sections, highlighted in blue in the revised paper:

  • Sec. 4.2 & Appendix B.4: Integrated the comprehensive compute breakdown table, training GPU-hours, parameter counts, FLOPs, peak memory, and dual-protocol FPS explanations.
  • Sec. 2: Added literature discussions addressing MLLM-based affordance techniques.

[1] Yao L, Chen Y, et al. HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Dear Reviewer TdUs,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

Dear Reviewer,

Thank you again for your valuable comments. Since the discussion period will end on November 18, we really hope you can let us know whether the revised version addresses your main concerns. We also welcome further discussion with you.

Best regards, The Authors

Reviewer_garWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces ST-Afford, a framework that groups dense 3D Gaussian features into compact semantic tokens and uses text instructions to select relevant geometry. This improves the accuracy and completeness of functional region localization, reduces inference cost, and supports better generalization to unseen object categories.

Critical strengths
  • The paper addresses an important problem in affordance learning.
  • The main idea is clear and generally easy to follow
Critical weaknesses
  • The motivation for combining Gaussian features with point-cloud features is not fully convincing, as both represent the same object and may encode redundant information. In practice, requiring corresponding 3D Gaussian primitives also adds complexity compared with using point clouds alone.
  • The comparison in Table 1 is not entirely fair, since IAGNet and LASO do not use 3D Gaussian inputs. It is therefore unclear whether the gains come from the proposed method or the additional representation.
  • Beyond combining Gaussian and point-cloud features, the paper offers limited methodological novelty for the ML community.
Questions for authors

I mainly have concerns about the motivation of using Gaussian as another input, the the experiment is not entire fare. The proposed method also does not have any new theory.

3 comments

We sincerely thank Reviewer garW for their constructive feedback. We appreciate the opportunity to clarify our single-modality 3DGS design, the deep theoretical advantages of 3D Gaussians over point clouds as formulated in recent literature (e.g., 3DAffordSplat), our ablation interpretation, and our zero-shot generalization performance. All corresponding manuscript revisions are highlighted in blue.


1. Comprehensive Clarification on Input Representation and the Inherent Advantages of 3DGS over Point Clouds.

... The motivation for combining Gaussian features with point-cloud features is not fully convincing, as both represent the same object ...

A: We explicitly clarify that ST-Afford operates strictly on a single 3DGS modality; it does not combine point clouds and Gaussians as dual inputs.

Our pipeline takes exclusively 3DGS assets as input through two complementary branches without dual-modality mixing:

  • Structural Tri-Plane Branch (SSGE): Projects the full 59-dimensional Gaussian primitives onto orthogonal tri-planes (X∈R256×768X \in \mathbb{R}^{256 \times 768}) to capture high-level object semantics and ordered shape-prior tokens.
  • Local Geometry Branch (IGDF): Slices physical geometry attributes directly from the exact same primitives to preserve fine-grained spatial correspondence.

Why 3DGS is Superior and More Faithful to Real Objects than Point Clouds ( grounded in 3DAffordSplat ):

  • Overcoming Point Cloud Sparsity & Coordinate Sensitivity: As rigorously demonstrated in recent 3DGS affordance research, traditional 3D point clouds are inherently discrete, sparse, and sensitive to coordinate variations. Their discrete sampling nature fundamentally constrains their ability to represent continuous surfaces and intricate structural details essential for precise affordance reasoning. Increasing point density often incurs prohibitive computational costs without resolving fundamental discreteness.
  • Continuous Surfaces & Rich Appearance/Opacity Cues: In contrast, 3D Gaussian Splatting (3DGS) represents 3D scenes as dense, continuous distributions that are far closer to real-world objects. By naturally encoding continuous geometric surfaces, smooth curves, rich structural details, and high-fidelity appearance/opacity attributes, 3DGS provides superior surface preservation and fine-grained visual cues without heuristic neighborhood sampling noise. This continuous nature is vital for predicting smooth functional regions over complex surfaces.
  • In-Depth Interpretation of Ablation A1 (No DiffGS + SSGE, Table 3): Variant A1 is an architectural ablation removing SSGE rather than a point-cloud input. Its severe performance drop (IoU collapsing from 0.4511 down to 0.0918) proves that without an explicit structural aggregation mechanism (such as our SSGE), raw primitive-level attributes alone cannot bridge high-level interaction intent and local functional regions.

2. Expanding Baseline Comparisons and Rigorous Analysis on Unseen Generalization.

... The comparison is not entirely fair, since IAGNet and LASO do not use 3D Gaussian inputs. It is therefore unclear whether the gains come from the proposed method or the additional representation ...

A: To validate generalization and substantiate our methodological novelty, we adapted HAMMER using the identical physical attributes extracted from 3DGS under our unified evaluation protocol:

MethodModality / ArchitectureUnseen MAE ↓\downarrowUnseen SIM ↑\uparrowUnseen AUC ↑\uparrowUnseen IoU ↑\uparrow
3DAffordSplatNative 3DGS0.21550.45020.79470.2424
HAMMER (Adapted)3DGS + MLLM0.24650.34730.76230.1740
ST-Afford (Ours)3DGS + SSGE + IGDF0.19180.51540.82370.3793

As shown above and visualized in Fig. 4, while adapted HAMMER achieves competitive scores on seen training objects, its zero-shot generalization drops drastically on unseen geometries (0.1740 IoU due to instance-specific overfitting to training categories). In contrast, ST-Afford achieves an Unseen IoU of 0.3793—more than double HAMMER’s performance, validating that SSGE successfully distills transferable shape tokens for robust cross-category generalization.

Specifically, we updated the following sections (highlighted in blue in the revised paper):

  • Sec. 2.1 & Sec. 4.3: Included related work discussion, clarified the 10D vs. point-cloud distinction, and updated the zero-shot Unseen analysis.
  • Table 2 & Fig. 4: Added adapted HAMMER quantitative metrics and qualitative visual comparisons on the Unseen dataset.
  • Appendix D: Added a dedicated protocol description for the 10D geometry adaptation of HAMMER.

Dear Reviewer garW,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

Dear Reviewer,

Thank you again for your valuable comments. Since the discussion period will end on November 18, we really hope you can let us know whether the revised version addresses your main concerns. We also welcome further discussion with you.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.