- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MAGPIE addresses language-guided, scene-level 3D affordance grounding. A frozen VLM parses the instruction and selects a rendered view, and a frozen grounding model identifies a coarse search region. Three lightweight modules operating on frozen point-cloud features then generate affordance candidates, select the referred instance through language-conditioned set reasoning, and refine its mask. The central proposal is to move fine-grained instance disambiguation from a large VLM to a lightweight 3D selector. The method reports competitive results on SceneFun3D and evaluates an adaptation of its refinement module on object-level benchmarks.
Critical strengths
- It decomposes the task with a small trainable footprint. Separating affordance candidate generation, language-conditioned instance selection, and mask refinement is a motivated design for small, repetitive functional elements. Training only 5.05M parameters on frozen point-cloud features offers a practical and parameter-efficient adaptation strategy. Importantly, this suggests that improving fine-grained grounding may not require fine-tuning a large multimodal model.
- It shows competitive scene-level performance. MAGPIE improves substantially over the reproduced AffordBot baseline and achieves AP50 and AP25 results competitive with RGB-based pipelines.
Critical weaknesses
-
The central contribution is not sufficiently distinguished from prior 2D-guided 3D methods or isolated experimentally. The paper contrasts prior view-based pipelines with its own 3D decision mechanism. However, TASA already combines 2D guidance with learned 3D geometric refinement, so the distinction cannot simply be whether a method uses 3D geometry after an image-based localization stage. MAGPIE’s more specific contribution appears to be explicit, language-conditioned selection among a set of 3D instances.
-
More importantly, the ablations do not directly test whether candidate-to-candidate reasoning is necessary. Table 2 shows refinement, clustering thresholds, and oracle selection, but does not compare the full selector against a language-conditioned independent candidate scorer or a selector without candidate self-attention.
-
The view-dependent localization interface is underspecified, and its effects are not adequately isolated in the evaluation.
(1) Section 3.2 states that eight center-out views are rendered and that the VLM selects the view in which the root object is most visible. The supplement provides several rendering constants, but the definition of the scene center and the exact camera construction remain unclear. In particular, the camera positions, look-at targets, orientation convention, and view-selection prompt are insufficient to reproduce this stage unambiguously.
(2) Beyond implementation details, there is a another issue: the selected camera determines the frame for interpreting "left" and "right," but choosing a view for object visibility does not guarantee that this is the reference frame intended by the instruction. The paper does not explain how this alignment is ensured to alternative camera frames.
(3) This stage strongly affects the final result. Table 5 attributes 19.8% of queries to ROI exclusion, versus only 7.0% to incorrect selection after the correct candidate survives upstream filtering. Thus, although the final selector operates in 3D and can retain occluded candidates, the system remains highly dependent on view-based localization.
(4)The evaluation also does not fully separate parsing from view selection. In Table 3, replacing the query-understanding model changes both, while "Parse" measures only affordance-type accuracy despite the structured query containing five fields. Similar type accuracy therefore does not establish that the other parsed fields are equally reliable, or that performance differences primarily stem from view selection. These ambiguities weaken both reproducibility and the interpretation of the upstream ablation.
-
The public Fun3DU/AffordBot evaluators label thresholded pointwise precision as AP, unlike SceneFun3D’s official IoU-based instance AP. If this implementation was used, the reported AP may reward incomplete predictions and is not directly comparable to official benchmark results. Which evaluator was used? If it follows Fun3DU/AffordBot, could you reevaluate your method and key baselines using the official SceneFun3D protocol?
Questions for authors
Please see the weakness.
11 comments
Comment 1. What MAGPIE contributes beyond 2D-guided pipelines with learned 3D refinement such as TASA, and whether that contribution is isolated experimentally.
(continued...)
Table R1 isolates this selection stage experimentally. The parser, the selected view, the search region, and the candidate sets are held fixed at the values of Table 1, and only the rule that selects one candidate changes. When the same candidates are handed to Qwen2.5-VL-72B or Qwen3.5-9B, the selection accuracy is 0.200 and 0.238 and AP25 is 22.3 and 26.3, whereas the Set-Selector reaches 0.679 and 47.2 AP25. With every upstream input shared, the margin lies in the selection stage itself. The end-to-end results in Table 1 agree with this reading. Without captured RGB frames, MAGPIE surpasses the peer-reviewed RGB pipelines on AP50 and AP25, TASA-72B among them by 8.6 and 18.6 points, and reaches 22.3 mIoU against 19.7 for TASA-72B. The margin also does not come from the vision-language model. With the Qwen2.5-VL-72B model that TASA uses placed in the upstream of MAGPIE, the pipeline reaches 34.2 AP50, 44.5 AP25, and 21.8 mIoU, as Table 3 reports. The revised Section 2 states this distinction from view-based pipelines, TASA included.
Table R1. Selection rules on identical candidate sets (SceneFun3D validation, predicted type). Sel. Acc. follows the definition in Appendix E.
| Selection rule | Sel. Acc. | AP50 | AP25 | mIoU |
|---|---|---|---|---|
| Random | 0.123 | 10.6 | 14.4 | 6.5 |
| Sort by ordinal + qualifier | 0.129 | 10.8 | 14.4 | 6.9 |
| Sort by ordinal only | 0.129 | 12.8 | 16.9 | 7.9 |
| Proposal nearest to the set centroid | 0.140 | 14.8 | 18.2 | 8.6 |
| Largest cluster | 0.285 | 15.7 | 22.3 | 11.2 |
| Qwen2.5-VL-72B | 0.200 | 17.3 | 22.3 | 10.8 |
| Qwen3.5-9B | 0.238 | 19.6 | 26.3 | 12.5 |
| Set attention, no language | 0.310 | 18.2 | 25.8 | 13.0 |
| Selector w/o candidate self-attention | 0.449 | 27.2 | 36.9 | 18.1 |
| Selector, SA replaced by parameter-matched FFN | 0.441 | 27.0 | 36.0 | 17.6 |
| Set-Selector (ours) | 0.679 | 35.5 | 47.2 | 22.3 |
Table R2. mIoU by the number of captured frames that observe the target under the depth test of the Fun3DU lifting (SceneFun3D validation, 445 descriptions). Fun3DU-9B values come from our per-description rerun and differ from Table 1 by 0.1 mIoU overall. The 0–60 row includes the one description observed in no frame. The revised appendix reports the breakdown in steps of 20 frames.
| Frames observing the target | n | MAGPIE | Fun3DU-9B* | AffordBot-72B* |
|---|---|---|---|---|
| All | 445 | 22.3 | 10.8 | 14.2 |
| > 100 | 376 | 21.5 | 11.9 | 14.8 |
| ≤ 100 | 69 | 26.6 | 4.8 | 10.9 |
| 61–100 | 47 | 26.3 | 4.4 | 13.2 |
| 0–60 | 22 | 27.2 | 5.7 | 6.2 |
Comment 2. Whether the gain of the Set-Selector arises from interaction among candidates, tested against an input-matched independent scorer and a selector without candidate self-attention.
Response. We thank the reviewer for the question, and respectfully confirm that both requested controls are in Table R1 of our response to Comment 1, with inputs identical to those of the full Set-Selector. Removing the interaction among candidates lowers the selection accuracy from 0.679 to 0.449 and AP25 from 47.2 to 36.9. With the same number of parameters placed in a feed-forward network that processes each candidate alone instead of in self-attention, the accuracy is 0.441 and AP25 is 36.0. In the architecture of MAGPIE the two requested controls coincide. Each candidate token carries the mean frozen point feature of the candidate, its size, and its offset from the mean of the set, expressed as the displacement from the set centroid in the world frame and in the selected view and as the height relative to the set mean. Each token therefore carries the position of the set as a whole and not the position of any other candidate. Inside the network, the self-attention among candidate tokens is the only operation that compares one candidate with another, since layer normalization, cross-attention to the instruction, and the feed-forward layers act on each token separately. Removing that self-attention leaves a language-conditioned scorer that reads each candidate on its own, which is the row (Selector w/o candidate self-attention). A second row rules out capacity as the explanation. The self-attention block is replaced by a feed-forward network that processes each candidate alone and still reads the same offsets from the set mean, with 262,912 parameters against 263,168 in the block it replaces. This is the row (Selector, SA replaced by parameter-matched FFN).
Both rows follow the protocol of Table 1. The affordance type is predicted, the parser, the selected view, the search region, and the candidate sets are fixed, and only the selection rule changes. Selection accuracy follows Appendix E. Its denominator is the 365 descriptions whose candidate pool under the predicted type contains a cluster that covers the ground truth, and the numerator counts the descriptions whose selected cluster has the largest overlap with the ground truth. The denominator keeps the descriptions for which the search region excluded the correct candidate, which holds every row to the same conservative measure.
We kindly refer the reviewer to the two rows of Table R1 that we ran for this question, (Selector w/o candidate self-attention) and (Selector, SA replaced by parameter-matched FFN), both evaluated end to end with the full pipeline. The loss comes from the missing interaction rather than from missing capacity. The parameter-matched row keeps the parameter count of the full Set-Selector, 3.014M against 3.015M, and spends them on a feed-forward network that processes each candidate alone. We would like to bring to the attention of the reviewer that this row scores no higher than the row that only removes self-attention, 0.441 against 0.449 in selection accuracy and 36.0 against 36.9 in AP25, although it holds as many parameters as the full model. This dependence on interaction follows from the instructions themselves. A qualifier such as leftmost or second describes where a candidate stands among the others, and a scorer that reads one candidate at a time has access to that standing only through its set-relative offsets. The row (Set attention, no language) supplies the complementary reading, since interaction without the instruction reaches only 0.310 and the arrangement alone does not identify which position the instruction names. On the full pipeline, with the parser, the selected view, the search region, and the candidate sets unchanged, the full Set-Selector reaches 35.5 AP50, 47.2 AP25, and 22.3 mIoU, and the row without candidate self-attention reaches 27.2 AP50, 36.9 AP25, and 18.1 mIoU, a difference of 8.3 AP50, 10.3 AP25, and 4.2 mIoU that the interaction alone accounts for. Handing the same candidates to a vision-language model widens the gap further. Qwen2.5-VL-72B selects the correct candidate in 0.200 of the 365 descriptions and reaches 22.3 AP25, and Qwen3.5-9B selects it in 0.238 and reaches 26.3 AP25, against 0.679 and 47.2 for the Set-Selector. The revised Section 4.4 carries Table R1, as Table 6, with both controls.
Response. We thank the reviewer for the comment, and respectfully clarify that MAGPIE contributes a grounding pipeline that completes the whole task on the point cloud alone and resolves the choice among near-identical sibling elements in 3D, the step at which the difficulty of scene-level affordance grounding concentrates. The frozen upstream models only parse the instruction, select a rendered view of the point cloud, and delimit a coarse search region on that view, a region that restricts the candidate pool and identifies no element. Within that region, the type head and feature-cut propose candidates from frozen point features, the Set-Selector resolves the instruction against the relative arrangement of the candidate set, and the refine head produces the final mask, as Section 3 describes. Every element-level decision therefore takes place on the point cloud, and no posed RGB-D imagery enters the pipeline. Prior view-based pipelines take the opposite route and identify the element in 2D images before lifting that choice into 3D. TASA follows this route [1]: a vision-language model locates the manipulable points on captured RGB frames, Molmo and SAM convert those points into 2D masks, the masks are projected into the point cloud through the camera parameters, and the Point Transformer refines the boundary within the neighborhood of that projection. The learned 3D stage of TASA thus inherits the element that its 2D stage selected, whereas the learned 3D stage of MAGPIE performs the selection.
The advantage of a 3D decision over the 2D route of TASA lies in the two conditions under which scene-level grounding is hardest: near-identical siblings and hidden targets. Under the first condition, the element an instruction designates is visually interchangeable with its siblings, and only its position within the local arrangement sets the target apart, as Fig. 1 illustrates. The instruction names that position with words such as leftmost or second, and whether a given handle is the leftmost one depends on where the other handles stand rather than on the handle itself. The first condition therefore requires every sibling to be in view when the decision is taken. Under the second condition, an element of this size may sit behind furniture or lie outside every captured RGB frame, which requires the decision to retain candidates that no image shows. No single captured frame satisfies either requirement. TASA selects the element within individual captured frames, each taken from one viewpoint in which some siblings may be hidden or foreshortened, and the 3D refinement of TASA adjusts only the boundary of that selected element. MAGPIE instead selects the element from the complete candidate set in 3D and satisfies both requirements. The Set-Selector reads the rank of every candidate from the arrangement of the whole set, and candidates hidden in the selected view remain in that set since the search region is a view frustum without an occlusion test. We measured the second requirement against the 2D route on the same descriptions. TASA is quoted from its paper and offers no per-description predictions, hence the two baselines we reproduced, Fun3DU-9B and AffordBot-72B, stand in for that route. Table R2 groups the 445 validation descriptions by the number of captured frames that observe the target, under the depth-based occlusion test of the Fun3DU lifting, in which a frame observes the target when at least 10% of its points are visible within a 25% relative depth tolerance. On the 69 descriptions whose target is observed in at most 100 frames, MAGPIE reaches 26.6 mIoU, against 4.8 for Fun3DU-9B and 10.9 for AffordBot-72B, whereas over all 445 descriptions the three reach 22.3, 10.8, and 14.2. In every group of Table R2 with 100 frames or fewer, the two reproduced baselines fall below their overall mIoU, and MAGPIE does not. Within the view that MAGPIE selects, the 55 descriptions whose target lies in the frame and is occluded reach 20.3 mIoU, as Appendix E of the original submission reports. Moreover, the two routes place very different demands on the model that selects the element. In TASA the element is selected by a 72B vision-language model, since identifying a handle or a knob directly in a captured image is a perception problem at the scale of the whole scene. In MAGPIE the frozen point features separate the candidates before the selection begins, and the Set-Selector compares only the few candidates inside the search region by their positions and the instruction. The learned capacity of MAGPIE is therefore placed at the selection itself, where the difficulty of the task concentrates, and three heads with 5.05M trained parameters carry the decision that TASA assigns to a 72B vision-language model.
(to be continued...)