acceptodds
Under review as a conference paper at ICLR 2027

MAGPIE: Memory-augmented Affordance Grounding via Point-cloud Instance Selection

Abstract

Language-guided 3D affordance grounding requires a model to segment, directly in a 3D scene, the functional element that an instruction refers to. The leading methods answer it with scale: an 8B–72B vision–language model operates inside the grounding loop, every query incurs its forward-pass cost, and the decision rests on captured RGB-D frames in which the element may be occluded by its own carrier or absent from every view. Such models are designed to turn language into an executable goal, whereas the choice among instances of the same type depends on where each candidate lies relative to the others, a 3D signal that prior work largely leaves unused. MAGPIE therefore restricts the large models to a coarse region and settles the question in 3D. Specifically, a frozen vision–language model parses the instruction and selects a rendered view, a frozen grounder delimits the candidate set to one region of that view, and no large model is consulted again. MAGPIE stacks three lightweight heads on frozen point-cloud features: a type head proposes candidate instances, a set-selector resolves which same-category instance the instruction refers to, and a refine head tightens the selection into the final mask. The heads consume no image, captured or rendered, and disambiguate visually near-identical instances from 3D structure alone. On SceneFun3D, MAGPIE leads the strongest point-cloud-only prior work on every reported metric, surpasses peer-reviewed RGB pipelines up to 72B on AP₅₀ and AP₂₅, and exceeds concurrent 30B systems on AP₅₀ with three orders of magnitude fewer trainable parameters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D affordance grounding, affordance segmentation, 3D scene understanding, SceneFun3D

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5xb5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MAGPIE addresses language-guided, scene-level 3D affordance grounding. A frozen VLM parses the instruction and selects a rendered view, and a frozen grounding model identifies a coarse search region. Three lightweight modules operating on frozen point-cloud features then generate affordance candidates, select the referred instance through language-conditioned set reasoning, and refine its mask. The central proposal is to move fine-grained instance disambiguation from a large VLM to a lightweight 3D selector. The method reports competitive results on SceneFun3D and evaluates an adaptation of its refinement module on object-level benchmarks.

Critical strengths
  1. It decomposes the task with a small trainable footprint. Separating affordance candidate generation, language-conditioned instance selection, and mask refinement is a motivated design for small, repetitive functional elements. Training only 5.05M parameters on frozen point-cloud features offers a practical and parameter-efficient adaptation strategy. Importantly, this suggests that improving fine-grained grounding may not require fine-tuning a large multimodal model.
  2. It shows competitive scene-level performance. MAGPIE improves substantially over the reproduced AffordBot baseline and achieves AP50 and AP25 results competitive with RGB-based pipelines.
Critical weaknesses
  1. The central contribution is not sufficiently distinguished from prior 2D-guided 3D methods or isolated experimentally. The paper contrasts prior view-based pipelines with its own 3D decision mechanism. However, TASA already combines 2D guidance with learned 3D geometric refinement, so the distinction cannot simply be whether a method uses 3D geometry after an image-based localization stage. MAGPIE’s more specific contribution appears to be explicit, language-conditioned selection among a set of 3D instances.

  2. More importantly, the ablations do not directly test whether candidate-to-candidate reasoning is necessary. Table 2 shows refinement, clustering thresholds, and oracle selection, but does not compare the full selector against a language-conditioned independent candidate scorer or a selector without candidate self-attention.

  3. The view-dependent localization interface is underspecified, and its effects are not adequately isolated in the evaluation.

    (1) Section 3.2 states that eight center-out views are rendered and that the VLM selects the view in which the root object is most visible. The supplement provides several rendering constants, but the definition of the scene center and the exact camera construction remain unclear. In particular, the camera positions, look-at targets, orientation convention, and view-selection prompt are insufficient to reproduce this stage unambiguously.

    (2) Beyond implementation details, there is a another issue: the selected camera determines the frame for interpreting "left" and "right," but choosing a view for object visibility does not guarantee that this is the reference frame intended by the instruction. The paper does not explain how this alignment is ensured to alternative camera frames.

    (3) This stage strongly affects the final result. Table 5 attributes 19.8% of queries to ROI exclusion, versus only 7.0% to incorrect selection after the correct candidate survives upstream filtering. Thus, although the final selector operates in 3D and can retain occluded candidates, the system remains highly dependent on view-based localization.

    (4)The evaluation also does not fully separate parsing from view selection. In Table 3, replacing the query-understanding model changes both, while "Parse" measures only affordance-type accuracy despite the structured query containing five fields. Similar type accuracy therefore does not establish that the other parsed fields are equally reliable, or that performance differences primarily stem from view selection. These ambiguities weaken both reproducibility and the interpretation of the upstream ablation.

  4. The public Fun3DU/AffordBot evaluators label thresholded pointwise precision as AP, unlike SceneFun3D’s official IoU-based instance AP. If this implementation was used, the reported AP may reward incomplete predictions and is not directly comparable to official benchmark results. Which evaluator was used? If it follows Fun3DU/AffordBot, could you reevaluate your method and key baselines using the official SceneFun3D protocol?

Questions for authors

Please see the weakness.

11 comments

Comment 1. What MAGPIE contributes beyond 2D-guided pipelines with learned 3D refinement such as TASA, and whether that contribution is isolated experimentally.

Response. We thank the reviewer for the comment, and respectfully clarify that MAGPIE contributes a grounding pipeline that completes the whole task on the point cloud alone and resolves the choice among near-identical sibling elements in 3D, the step at which the difficulty of scene-level affordance grounding concentrates. The frozen upstream models only parse the instruction, select a rendered view of the point cloud, and delimit a coarse search region on that view, a region that restricts the candidate pool and identifies no element. Within that region, the type head and feature-cut propose candidates from frozen point features, the Set-Selector resolves the instruction against the relative arrangement of the candidate set, and the refine head produces the final mask, as Section 3 describes. Every element-level decision therefore takes place on the point cloud, and no posed RGB-D imagery enters the pipeline. Prior view-based pipelines take the opposite route and identify the element in 2D images before lifting that choice into 3D. TASA follows this route [1]: a vision-language model locates the manipulable points on captured RGB frames, Molmo and SAM convert those points into 2D masks, the masks are projected into the point cloud through the camera parameters, and the Point Transformer refines the boundary within the neighborhood of that projection. The learned 3D stage of TASA thus inherits the element that its 2D stage selected, whereas the learned 3D stage of MAGPIE performs the selection.

The advantage of a 3D decision over the 2D route of TASA lies in the two conditions under which scene-level grounding is hardest: near-identical siblings and hidden targets. Under the first condition, the element an instruction designates is visually interchangeable with its siblings, and only its position within the local arrangement sets the target apart, as Fig. 1 illustrates. The instruction names that position with words such as leftmost or second, and whether a given handle is the leftmost one depends on where the other handles stand rather than on the handle itself. The first condition therefore requires every sibling to be in view when the decision is taken. Under the second condition, an element of this size may sit behind furniture or lie outside every captured RGB frame, which requires the decision to retain candidates that no image shows. No single captured frame satisfies either requirement. TASA selects the element within individual captured frames, each taken from one viewpoint in which some siblings may be hidden or foreshortened, and the 3D refinement of TASA adjusts only the boundary of that selected element. MAGPIE instead selects the element from the complete candidate set in 3D and satisfies both requirements. The Set-Selector reads the rank of every candidate from the arrangement of the whole set, and candidates hidden in the selected view remain in that set since the search region is a view frustum without an occlusion test. We measured the second requirement against the 2D route on the same descriptions. TASA is quoted from its paper and offers no per-description predictions, hence the two baselines we reproduced, Fun3DU-9B and AffordBot-72B, stand in for that route. Table R2 groups the 445 validation descriptions by the number of captured frames that observe the target, under the depth-based occlusion test of the Fun3DU lifting, in which a frame observes the target when at least 10% of its points are visible within a 25% relative depth tolerance. On the 69 descriptions whose target is observed in at most 100 frames, MAGPIE reaches 26.6 mIoU, against 4.8 for Fun3DU-9B and 10.9 for AffordBot-72B, whereas over all 445 descriptions the three reach 22.3, 10.8, and 14.2. In every group of Table R2 with 100 frames or fewer, the two reproduced baselines fall below their overall mIoU, and MAGPIE does not. Within the view that MAGPIE selects, the 55 descriptions whose target lies in the frame and is occluded reach 20.3 mIoU, as Appendix E of the original submission reports. Moreover, the two routes place very different demands on the model that selects the element. In TASA the element is selected by a 72B vision-language model, since identifying a handle or a knob directly in a captured image is a perception problem at the scale of the whole scene. In MAGPIE the frozen point features separate the candidates before the selection begins, and the Set-Selector compares only the few candidates inside the search region by their positions and the instruction. The learned capacity of MAGPIE is therefore placed at the selection itself, where the difficulty of the task concentrates, and three heads with 5.05M trained parameters carry the decision that TASA assigns to a 72B vision-language model.

(to be continued...)

Comment 1. What MAGPIE contributes beyond 2D-guided pipelines with learned 3D refinement such as TASA, and whether that contribution is isolated experimentally.

(continued...)

Table R1 isolates this selection stage experimentally. The parser, the selected view, the search region, and the candidate sets are held fixed at the values of Table 1, and only the rule that selects one candidate changes. When the same candidates are handed to Qwen2.5-VL-72B or Qwen3.5-9B, the selection accuracy is 0.200 and 0.238 and AP25 is 22.3 and 26.3, whereas the Set-Selector reaches 0.679 and 47.2 AP25. With every upstream input shared, the margin lies in the selection stage itself. The end-to-end results in Table 1 agree with this reading. Without captured RGB frames, MAGPIE surpasses the peer-reviewed RGB pipelines on AP50 and AP25, TASA-72B among them by 8.6 and 18.6 points, and reaches 22.3 mIoU against 19.7 for TASA-72B. The margin also does not come from the vision-language model. With the Qwen2.5-VL-72B model that TASA uses placed in the upstream of MAGPIE, the pipeline reaches 34.2 AP50, 44.5 AP25, and 21.8 mIoU, as Table 3 reports. The revised Section 2 states this distinction from view-based pipelines, TASA included.

Table R1. Selection rules on identical candidate sets (SceneFun3D validation, predicted type). Sel. Acc. follows the definition in Appendix E.

Selection ruleSel. Acc.AP50AP25mIoU
Random0.12310.614.46.5
Sort by ordinal + qualifier0.12910.814.46.9
Sort by ordinal only0.12912.816.97.9
Proposal nearest to the set centroid0.14014.818.28.6
Largest cluster0.28515.722.311.2
Qwen2.5-VL-72B0.20017.322.310.8
Qwen3.5-9B0.23819.626.312.5
Set attention, no language0.31018.225.813.0
Selector w/o candidate self-attention0.44927.236.918.1
Selector, SA replaced by parameter-matched FFN0.44127.036.017.6
Set-Selector (ours)0.67935.547.222.3

Table R2. mIoU by the number of captured frames that observe the target under the depth test of the Fun3DU lifting (SceneFun3D validation, 445 descriptions). Fun3DU-9B values come from our per-description rerun and differ from Table 1 by 0.1 mIoU overall. The 0–60 row includes the one description observed in no frame. The revised appendix reports the breakdown in steps of 20 frames.

Frames observing the targetnMAGPIEFun3DU-9B*AffordBot-72B*
All44522.310.814.2
> 10037621.511.914.8
≤ 1006926.64.810.9
61–1004726.34.413.2
0–602227.25.76.2

Comment 2. Whether the gain of the Set-Selector arises from interaction among candidates, tested against an input-matched independent scorer and a selector without candidate self-attention.

Response. We thank the reviewer for the question, and respectfully confirm that both requested controls are in Table R1 of our response to Comment 1, with inputs identical to those of the full Set-Selector. Removing the interaction among candidates lowers the selection accuracy from 0.679 to 0.449 and AP25 from 47.2 to 36.9. With the same number of parameters placed in a feed-forward network that processes each candidate alone instead of in self-attention, the accuracy is 0.441 and AP25 is 36.0. In the architecture of MAGPIE the two requested controls coincide. Each candidate token carries the mean frozen point feature of the candidate, its size, and its offset from the mean of the set, expressed as the displacement from the set centroid in the world frame and in the selected view and as the height relative to the set mean. Each token therefore carries the position of the set as a whole and not the position of any other candidate. Inside the network, the self-attention among candidate tokens is the only operation that compares one candidate with another, since layer normalization, cross-attention to the instruction, and the feed-forward layers act on each token separately. Removing that self-attention leaves a language-conditioned scorer that reads each candidate on its own, which is the row (Selector w/o candidate self-attention). A second row rules out capacity as the explanation. The self-attention block is replaced by a feed-forward network that processes each candidate alone and still reads the same offsets from the set mean, with 262,912 parameters against 263,168 in the block it replaces. This is the row (Selector, SA replaced by parameter-matched FFN).

Both rows follow the protocol of Table 1. The affordance type is predicted, the parser, the selected view, the search region, and the candidate sets are fixed, and only the selection rule changes. Selection accuracy follows Appendix E. Its denominator is the 365 descriptions whose candidate pool under the predicted type contains a cluster that covers the ground truth, and the numerator counts the descriptions whose selected cluster has the largest overlap with the ground truth. The denominator keeps the descriptions for which the search region excluded the correct candidate, which holds every row to the same conservative measure.

We kindly refer the reviewer to the two rows of Table R1 that we ran for this question, (Selector w/o candidate self-attention) and (Selector, SA replaced by parameter-matched FFN), both evaluated end to end with the full pipeline. The loss comes from the missing interaction rather than from missing capacity. The parameter-matched row keeps the parameter count of the full Set-Selector, 3.014M against 3.015M, and spends them on a feed-forward network that processes each candidate alone. We would like to bring to the attention of the reviewer that this row scores no higher than the row that only removes self-attention, 0.441 against 0.449 in selection accuracy and 36.0 against 36.9 in AP25, although it holds as many parameters as the full model. This dependence on interaction follows from the instructions themselves. A qualifier such as leftmost or second describes where a candidate stands among the others, and a scorer that reads one candidate at a time has access to that standing only through its set-relative offsets. The row (Set attention, no language) supplies the complementary reading, since interaction without the instruction reaches only 0.310 and the arrangement alone does not identify which position the instruction names. On the full pipeline, with the parser, the selected view, the search region, and the candidate sets unchanged, the full Set-Selector reaches 35.5 AP50, 47.2 AP25, and 22.3 mIoU, and the row without candidate self-attention reaches 27.2 AP50, 36.9 AP25, and 18.1 mIoU, a difference of 8.3 AP50, 10.3 AP25, and 4.2 mIoU that the interaction alone accounts for. Handing the same candidates to a vision-language model widens the gap further. Qwen2.5-VL-72B selects the correct candidate in 0.200 of the 365 descriptions and reaches 22.3 AP25, and Qwen3.5-9B selects it in 0.238 and reaches 26.3 AP25, against 0.679 and 47.2 for the Set-Selector. The revised Section 4.4 carries Table R1, as Table 6, with both controls.

Reviewer_cLkaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MAGPIE addresses language-guided affordance grounding in 3D scenes. Frozen vision-language models parse instructions and identify a coarse region in rendered views. Three lightweight heads then generate functional-part candidates, select an instance using language and candidate relationships, and refine its mask. On SceneFun3D, MAGPIE achieves 35.5 AP50 and 22.3 mIoU, outperforming the reproduced AffordBot-72B. Only 5.05M parameters are trained, with additional object-level evaluations on LASO and AGPIL.

Critical strengths
  1. Well-motivated decomposition. Separating localization, candidate generation, instance selection, and refinement provides an intuitive approach to grounding small, repeated functional parts.

  2. Substantial empirical improvements. MAGPIE clearly improves over the reproduced AffordBot-72B and achieves competitive SceneFun3D results against RGB-based pipelines.

  3. Low adaptation cost and useful diagnostics. Training takes approximately five minutes with cached features. Stage-wise failure analysis and refinement ablations help explain the system’s behavior, although inference still requires large frozen models.

Critical weaknesses
  1. This is the main concern. The central ablation does not isolate relational modeling. In Table A9, linear baselines receive candidate features, optionally with geometry, whereas the full selector additionally receives language and uses a more expressive attention network. The improvement from 50.9% to 88.6% therefore confounds language access, model capacity, and candidate interaction. Input-matched independent scoring and a selector without candidate self-attention are essential missing controls.

  2. “Geometry alone” is insufficiently supported, and the classification of 2D information is inconsistent. The encoder consumes coordinates, color, and normals, while Concerto uses joint 2D–3D pretraining. Independence from captured images does not establish geometry-only recognition. Moreover, CMAT is classified as using 2D information because of its pretraining features, whereas MAGPIE is not. Inference inputs and pretraining information should be distinguished consistently.

  3. View-selection scalability remains unclear. The fixed eight-view arrangement and single-view ROI grounding may inadequately cover large or multi-room scenes. No evaluation by scene extent or layout is provided.

  4. Candidate quality and type-head capability are incompletely characterized. Under the ground-truth type, candidate generation and ROI filtering account for 13.0% and 19.8% of query failures. Coverage requires only one overlapping voxel. Meanwhile, the 84.6% probe accuracy measures classification on annotated parts, rather than scene-wide foreground detection.

Questions for authors
  1. Is it possible to add an independent candidate scorer with identical language, feature, and geometry inputs, and an otherwise matched selector without candidate self-attention?

  2. How does removing color affect performance? Could you consistently distinguish methods’ inference inputs from their use of 2D information during pretraining?

  3. How does view selection perform in large or multi-room scenes? Would multiple camera centers or selected views improve coverage?

  4. Could you report candidate recall at multiple overlap thresholds, before and after ROI filtering, alongside scene-wide type-head precision/recall and per-type results?

  5. Why use CLIP for selection? Could you compare Qwen text features or encoded structured fields, especially on spatial and ordinal instructions?

  6. How strong is a nearest-to-set-centroid baseline? Does perturbing training centers or training with predicted ROIs change performance?

9 comments

Comment 1. Whether the gain of the Set-Selector over the linear probes of Table A9 comes from the interaction among candidates rather than from access to the instruction or from model capacity, tested with an input-matched independent scorer and a selector without candidate self-attention.

Response. We thank the reviewer for the careful reading of Table A9 and for the question, and respectfully confirm that we ran both requested controls with inputs identical to those of the full Set-Selector, and that Table R1 reports them. Under identical inputs, the interaction among candidates accounts for 23.0 points of selection accuracy, from 0.449 to 0.679, and for 8.3 AP50, 10.3 AP25, and 4.2 mIoU end to end, and the same number of parameters placed in a per-candidate feed-forward network recovers none of this difference, at 0.441 and 36.0 AP25. Access to the instruction accounts for the larger share of the gap, since the selector that keeps the interaction and omits the instruction reaches 0.310.

The two rows correspond to the two controls the reviewer names, and in the architecture of MAGPIE they differ in one component. Each candidate token carries the mean frozen point feature of the candidate, its size, and its offset from the mean of the set, expressed as the displacement from the set centroid in the world frame and in the selected view and as the height relative to the set mean. Each token therefore carries the position of the set as a whole and not the position of any other candidate. Inside the network, the self-attention among candidate tokens is the only operation that compares one candidate with another, since layer normalization, cross-attention to the instruction, and the feed-forward layers act on each token separately. Removing that self-attention leaves a language-conditioned scorer that reads each candidate on its own, which is the row (Selector w/o candidate self-attention). A second row rules out capacity as the explanation. The self-attention block is replaced by a feed-forward network that processes each candidate alone and still reads the same offsets from the set mean, with 262,912 parameters against 263,168 in the block it replaces, and the whole model keeps 3.014M parameters against 3.015M. This is the row (Selector, SA replaced by parameter-matched FFN). Both rows follow the protocol of Table 1. The affordance type is predicted, the parser, the selected view, the search region, and the candidate sets are fixed at their values in Table 1, and only the selection rule changes. Selection accuracy follows Appendix E of the original submission. Its denominator is the 365 validation descriptions whose candidate pool under the predicted type contains a cluster that covers the ground truth, and the numerator counts the descriptions whose selected cluster has the largest overlap with the ground truth. The denominator keeps the descriptions for which the search region excluded the correct candidate, which holds every row to the same conservative measure.

The dependence on the interaction follows from the instructions themselves. A qualifier such as leftmost or second describes where a candidate stands among the others, and a scorer that reads one candidate at a time has access to that standing only through its set-relative offsets. The row (Set attention, no language) supplies the complementary reading, since interaction without the instruction reaches only 0.310 and the arrangement alone does not identify which position the instruction names. The three controls therefore separate the two contributions behind the 0.679 of the full Set-Selector, 0.449 without the interaction, 0.310 without the instruction, and 0.441 with the interaction replaced by parameters, which is the separation the reviewer requests. We would like to clarify how this separation relates to Table A9 of the original submission. Table A9 answers a different question. It measures how far linear probes, with neither the instruction nor interaction among candidates, separate the correct candidate from the others in the set. The probes read the frozen point features alone or together with the set-relative geometry of each candidate, as Appendix G of the original submission states. Table R1 addresses the question the reviewer raises on the candidate pools that the pipeline forms at inference. The revised Fig. 2 and Section 4.4 report these controls next to the linear probes of Table A9 and attribute the gap to the instruction and to the interaction separately. We hope that these controls answer the question the reviewer raised, and we would be glad to report any further control that the reviewer finds useful.

(to be continued...)

Comment 1. Whether the gain of the Set-Selector over the linear probes of Table A9 comes from the interaction among candidates rather than from access to the instruction or from model capacity, tested with an input-matched independent scorer and a selector without candidate self-attention.

(continued...)

Table R1. Selection rules on identical candidate sets (SceneFun3D validation, predicted type). Sel. Acc. follows the definition in Appendix E.

Selection ruleSel. Acc.AP50AP25mIoU
Random0.12310.614.46.5
Sort by ordinal + qualifier0.12910.814.46.9
Sort by ordinal only0.12912.816.97.9
Proposal nearest to the set centroid0.14014.818.28.6
Largest cluster0.28515.722.311.2
Qwen2.5-VL-72B0.20017.322.310.8
Qwen3.5-9B0.23819.626.312.5
Set attention, no language0.31018.225.813.0
Selector w/o candidate self-attention0.44927.236.918.1
Selector, SA replaced by parameter-matched FFN0.44127.036.017.6
Set-Selector (ours)0.67935.547.222.3

Comment 2. The scope of the claim that MAGPIE resolves near-identical instances from 3D structure alone, as the frozen encoder reads coordinates, colors, and normals and Concerto is pretrained jointly on 2D and 3D data. The effect of removing color, and one rule for the 2D column of Table 7 that marks 2D data of the task at inference or in training and reports the pretraining of frozen backbones separately.

Response. We thank the reviewer for the careful reading, and would like to kindly bring to the attention of the reviewer that the question touches three different things. The first is 2D input at inference: whether the pipeline reads any image of the task, captured or rendered, when it grounds an instruction. The second is 2D data of the task in training: whether the method is trained with images of the task, with views rendered from the task point clouds, or with 2D features computed from them. The third is the pretraining of a frozen, general-purpose backbone, which is not specific to the task and which every method built on a pretrained encoder carries. MAGPIE consumes no image at inference, the three trained heads are trained with no 2D data of the task, and the frozen point encoder Concerto is a general-purpose backbone pretrained by its authors through self-supervision jointly on 2D and 3D data, with no supervision from the task, as Appendix B of the original submission states. The claim of 3D structure alone in the abstract and Section 3 refers to the first two: the heads read frozen point features and the instruction, and the choice among near-identical siblings rests on the relative arrangement of the candidates. We would also like to kindly remind the reviewer that two results demonstrate the remaining two points, color and pretraining: removing color from the encoder input changes the end-to-end results by less than two points on every measure (Table R3), and an encoder pretrained on 3D data only keeps MAGPIE ahead of the strongest point-cloud-only baseline on every metric (Table 4 of the original submission). For the color question, we set the point colors fed to the encoder to zero, kept the coordinates and normals, retrained the type head, the Set-Selector, and the refine head on the color-free features with the recipe of Table 1, and evaluated the pipeline end to end on the 445 validation descriptions. MAGPIE reaches 36.4 AP50, 49.0 AP25, and 22.1 mIoU without color against 35.5, 47.2, and 22.3 with color, as Table R3 reports, hence the results do not rest on color. The four remaining measures of Table 1 change by at most 1.4 points, with 24.0 AP, 23.5 AR, 36.6 AR50, and 45.6 AR25 without color against 22.7, 24.8, 38.0, and 46.7 with color.

For the pretraining of the backbone, Table 4 of the original submission replaces Concerto-large with Sonata, a Point Transformer V3 encoder pretrained by 3D-only self-distillation, and retrains the three heads with an unchanged recipe. MAGPIE with Sonata reaches 27.9 AP50, 40.9 AP25, and 16.1 mIoU against 20.0, 24.5, and 14.2 for the reproduced AffordBot-72B, the strongest point-cloud-only baseline (Table R3), and Concerto raises these scores further, in step with the nine-way probe accuracy of the frozen features, from 0.767 to 0.846, as Section 4.4 of the original submission reports. The advantage of the pipeline therefore does not require an encoder that has seen 2D data, and the encoder is a replaceable component of MAGPIE rather than a contribution of this work.

The same distinction settles the 2D column of Table 6 and Table 7, Tables 7 and 8 of the revised manuscript, with one rule for every method. The revised tables mark 2D data of the task, at inference or in training the method, and report the pretraining of each frozen backbone in a footnote of the table rather than in the column. Under this rule, GEAL is marked, since it renders 2D views of the task point clouds, CMAT is unmarked and footnoted, since 2D foundation-model features enter only the pretraining of its point encoder on Objaverse and Behavior-1K, outside the data of the task, IAGNet on LASO is unmarked, since its image backbone is replaced by a language model, and MAGPIE is unmarked, with the footnote stating that Concerto is pretrained through self-supervision jointly on 2D and 3D data, with no supervision from the task. The revised Section 4.5 and Appendix E compare the methods under this rule, and the revised abstract states the claim in these terms: the heads consume no image, and the selection rests on the relative arrangement of the candidates.

(to be continued...)

Reviewer_xyTWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a promising framework for language-guided 3D affordance grounding. By introducing several lightweight heads over frozen point-cloud features, MAGPIE aims to reduce the involvement of large models in the grounding process. The framework is interesting and potentially practical for embodied AI applications. However, my main concern is whether the reported performance improvements can be sufficiently attributed to the proposed Set-Selector rather than other components, particularly the upstream ROI module. Additionally, the paper may overstate its robustness to occluded objects, given the substantial performance gap between visible and hidden settings.

Critical strengths
  1. The proposed framework is promising and insightful. Using several lightweight heads to address instance and mask granularity ambiguity is an elegant design. Compared with previous pipelines that heavily rely on large foundation models throughout the grounding process, this work reduces their involvement in downstream grounding decisions.

  2. The experiments cover multiple aspects of the proposed framework, and the paper provides several detailed failure analyses, which are helpful for understanding the strengths and limitations of the method.

  3. The approach is well aligned with the needs of embodied manipulation, where instance disambiguation among similar operable parts is often a key bottleneck. Additionally, it has the potential to be applied to future embodied agents, provided that its end-to-end runtime efficiency can be demonstrated.

Critical weaknesses
  1. Limited evidence of the Set-Selector's contribution to end-to-end performance. Although the Set-Selector shows clear improvements in the isolated selection experiments, Table 5 indicates that most failures occur in the upstream candidate generation and ROI stages (146 vs. 31). I would like to see a more direct end-to-end ablation to better understand how much the Set-Selector contributes to the overall improvement.

  2. Limited robustness to occlusion. The paper emphasizes its ability to handle occluded targets, but the performance gap between visible and hidden targets remains substantial (25.8 vs. 3.3 mIoU). This suggests that the method still depends heavily on successful ROI localization, which limits the strength of its occlusion robustness claims.

Questions for authors
  1. Could the authors provide an end-to-end comparison between the proposed Set-Selector and a simpler independent instance-selection baseline?

  2. Given the substantial performance gap between visible and hidden targets, could the authors clarify to what extent MAGPIE improves occlusion robustness over existing methods?

3 comments

Comment 1. How much of the end-to-end result is attributable to the Set-Selector rather than to the upstream stages, where most failures in Table 5 occur, and how the Set-Selector compares with a rule that scores each candidate on its own.

Response. We thank the reviewer for the question, and report the end-to-end ablation it requires in Table R1. Table R1 follows the protocol of Table 1: the affordance type is predicted, the parser, the selected view, the search region, and the candidate sets are fixed, and only the rule that selects one candidate changes. Every row is evaluated end to end with the full pipeline. The independent baseline that the reviewer requests is the row (Selector w/o candidate self-attention), a language-conditioned scorer with the inputs of the Set-Selector that reads each candidate on its own. It reaches 27.2 AP50, 36.9 AP25, and 18.1 mIoU, whereas the Set-Selector reaches 35.5 AP50, 47.2 AP25, and 22.3 mIoU. When the same candidates are handed to Qwen3.5-9B, the pipeline reaches 19.6 AP50, 26.3 AP25, and 12.5 mIoU. With every upstream input shared, these differences come from the selection stage alone.

We would also like to kindly bring to the attention of the reviewer that Table 5 counts the stage at which each description fails, and not the contribution of each stage. Under the ground-truth type, 299 of the 445 descriptions reach the Set-Selector with a covering candidate in the set, and the Set-Selector selects the correct one for 268 of them. The 31 failures of the selection stage are therefore the remainder of a stage that resolves 89.6% of the descriptions it receives. The revised Section 4.4 reports Table R1 as Table 6, next to Table 5. We hope that this comparison answers the question the reviewer raised.

Table R1. Selection rules on identical candidate sets (SceneFun3D validation, predicted type). Sel. Acc. follows the definition in Appendix E.

Selection ruleSel. Acc.AP50AP25mIoU
Random0.12310.614.46.5
Sort by ordinal + qualifier0.12910.814.46.9
Sort by ordinal only0.12912.816.97.9
Proposal nearest to the set centroid0.14014.818.28.6
Largest cluster0.28515.722.311.2
Qwen2.5-VL-72B0.20017.322.310.8
Qwen3.5-9B0.23819.626.312.5
Set attention, no language0.31018.225.813.0
Selector w/o candidate self-attention0.44927.236.918.1
Selector, SA replaced by parameter-matched FFN0.44127.036.017.6
Set-Selector (ours)0.67935.547.222.3

Comment 2. How MAGPIE compares with existing methods on targets that are hard to observe, and how much its result on hidden targets depends on the search region, in view of the visible and hidden groups in Fig. 6.

Response. We thank the reviewer for this question. We would first like to clarify the definition behind the 25.8 vs. 3.3 mIoU gap. Fig. 6 and Table A6 group the descriptions by whether the target falls inside the image frame of the selected rendered view, without any occlusion test (Appendix E). The 70 descriptions at 3.3 mIoU are those whose target does not fall inside that frame. This reflects view selection rather than occlusion.

The targets that are truly occluded, i.e., those that lie inside the frame and fail the depth test, number 55 and reach 20.3 mIoU (Appendix E). The search region admits every 3D cluster whose centroid projects inside it, occluded or not, and a candidate therefore need not be visible in any image (Section 3.2).

To directly address the extent to which MAGPIE improves over existing methods, we conducted the following additional analysis. We project each ground-truth target into all captured high-resolution frames and apply the same depth-based occlusion test as the lifting of Fun3DU: a frame is counted as observing the target when at least 10% of its points are visible. Following the Fun3DU criterion, we then group the descriptions by the number of frames that observe the target.

Table R2. mIoU by the number of captured frames that observe the target under the depth test of the Fun3DU lifting (SceneFun3D validation, 445 descriptions). Fun3DU-9B values come from our per-description rerun and differ from Table 1 by 0.1 mIoU overall. The 0–60 row includes the one description observed in no frame. The revised appendix reports the breakdown in steps of 20 frames.

Frames observing the targetnMAGPIEFun3DU-9B*AffordBot-72B*
All44522.310.814.2
> 10037621.511.914.8
≤ 1006926.64.810.9
61–1004726.34.413.2
0–602227.25.76.2

On the descriptions whose target is observed in at most 100 frames, Fun3DU drops from 11.9 to 4.8 and AffordBot from 14.8 to 10.9, whereas MAGPIE does not degrade (21.5 → 26.6). In every group of Table R2 with 100 frames or fewer, the two reproduced baselines fall below their overall mIoU, and MAGPIE does not. For clarity, the revised caption of Fig. 6 states that its two groups are divided by whether the target falls inside the frame of the selected view, with no occlusion test. We hope that these measurements answer the question the reviewer raised.

Thank you for the detailed responses and additional experiments. The rebuttal addresses most of my concerns, and I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.