- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ActFun3D, which identifies the task-relevant functional region (like a handle, button, or knob), given a natural language task, a global 3D point cloud, and posed RGB-D images. ActFun3D first extracts the structured semantics from the task description, and then combines them with language features in 3D to narrow the scene to a smaller region. Then a trained point-based network predicts the functional masks within this region. In the end, selected RGB-D views are checked by a VLM and segmented using SAM2. This method achieves 30.8 AP50 on SceneFun3D, outperforming the reproduced baselines.
Critical strengths
- ActFun3D addresses a challenging problem of localizing small functional regions in large 3D scenes. The proposed approach outperforms the reproduced baselines on the SceneFun3D dataset.
- The experimental analysis is relatively comprehensive, including ablations on different components. The paper also provides useful failure analyses and discusses the limitations of the approach.
Critical weaknesses
-
Limited novelty and unclear core contribution. The proposed method combines several existing components, including LLM parsing, CLIP features, a point-based network, VLM verification, SAM2, and multi-view fusion. This results in a rather complicated pipeline, with many manually tuned parameters and a runtime of 126.6 seconds per task. More importantly, the idea of using task semantics and contextual objects to identify functional parts has already been explored in previous studies, such as Fun3DU and FunGraph. It is therefore not very clear to me what the main methodological contribution is beyond combining these existing techniques.
-
The need for such a complicated pipeline is not well justified. Given that point clouds and posed RGB-D images are already available, I wonder whether the task could be solved with a much simpler pipeline using recent foundation models. For example, a VLM could directly identify the functional part, followed by SAM 3 for segmentation and projection into 3D. Another possibility would be to use a 3D part segmentation model, such as PartSAM, to generate candidate parts and then use a VLM to select the relevant one. The paper does not provide comparisons with such simpler alternatives or more recent models, making it difficult to judge whether the proposed complexity is really necessary.
Questions for authors
- Necessity of the trained network. Is the trained LocalFunctionalGrounder really necessary? Could existing foundation models, such as SAM 3 or PartSAM, achieve comparable results without training a dedicated network?
- Structured semantic parsing. Given the capabilities of recent VLMs, is explicit structured semantic parsing still necessary? Could a VLM directly understand the task and identify the relevant functional part without this intermediate step?
- Direct grounding with stronger VLMs. The paper uses Qwen3-8B for task parsing and qwen3.8-flash for visual verification. Could larger and stronger VLMs directly identify functional targets from the task and RGB-D images, followed by segmentation and projection into 3D? How would such a simpler pipeline compare with ActFun3D?
- Related work and novelty. Several relevant recent works are missing from the discussion, including FunGraph (IROS 2025), T-FunS3D (ICRA 2026), and FunFact (CVPR 2026). Could the authors discuss these works and clarify the novelty of ActFun3D compared with existing approaches?
- Explicit 3D mapping. The authors could also discuss whether explicit semantic maps or scene graphs, such as ConceptGraphs or OpenFunGraph, could provide a simpler and more reusable alternative.
5 comments
Response to Reviewer CWMd (2/3)
Q1. Can SAM3 or PartSAM replace the trained LocalFunctionalGrounder?
Under the evaluated settings, both foundation-model alternatives leave a substantial performance gap. W2 explains the practical challenges of these routes: finding views that show both the part and its context, and ensuring that the 3D proposal set contains the part specified by the task. We tested both alternatives on all 445 SceneFun3D split0 tasks. The 2D route uses a VLM to localize the part and SAM3 to segment it for projection into 3D. The 3D route uses PartSAM to generate candidates from scene object proposals and a VLM to select the task-relevant part. We generated the proposals without ground-truth object crops or ActFun3D’s learned predictions (Table 10, Appendix B.7).
| Method | AP50 | AP25 | mIoU |
|---|---|---|---|
| GPT-6 Sol + SAM3, Structured input | 16.5 | 29.7 | 14.3 |
| PartSAM + Qwen3.8-Max | 18.2 | 30.4 | 15.1 |
| PartSAM + GPT-6 Sol | 19.0 | 31.3 | 15.7 |
| ActFun3D | 30.8 | 41.5 | 23.0 |
We then isolated the grounder’s contribution within ActFun3D. We kept the Structured crop, retrieval rule, 12-view cap, VLM/SAM2 backend, and signed refinement update fixed, and replaced the learned point scores with coarse narrowing scores. AP50 dropped from 30.8 to 18.4. Visibility voting raised coarse support to 19.2 AP50. With SAM3 and stronger VLMs, learned support still improved AP50 by 7.5 points with Qwen3.8-Max and 7.1 points with GPT-6 Sol (Tables 3(b) and 12(b)).
This control gives both variants the same candidate region. The difference is how precisely they identify the interaction points inside it. Those point scores guide view retrieval and give visual refinement a more precise 3D prediction to verify (Appendix D).
Q2. Does structured parsing still help a strong VLM that can interpret the raw task directly?
We gave both VLMs the raw instruction and then repeated the experiment with the structured description while keeping the selected views and segmenter fixed (Table 10(a)).
| Direct pipeline | Raw AP50 | Structured AP50 | Gain |
|---|---|---|---|
| Qwen3.8-Max + SAM3 | 12.4 | 15.1 | +2.7 |
| Qwen3.8-Max + SAM3.1 | 12.8 | 15.4 | +2.6 |
| GPT-6 Sol + SAM3 | 14.2 | 16.5 | +2.3 |
| GPT-6 Sol + SAM3.1 | 14.6 | 16.3 | +1.7 |
The raw instruction gives the VLM enough information to attempt localization. The structured description makes the interaction target explicit. For example, in “turn on the TV using the remote,” it identifies the remote’s power button as the part to manipulate and retains the TV as context. The gains across all four combinations show that this distinction helps even when the same strong VLM sees the same images.
Q3. How does direct grounding with stronger VLMs perform after projection into 3D?
The best raw-task configuration reaches 14.6 AP50, and the best structured-input configuration reaches 16.5, compared with 30.8 for ActFun3D. The direct baselines use depth and camera poses for projection, with visibility and cross-view consistency checks (Table 10(a), Appendix B.7). These checks can filter inconsistent evidence, but a mask on the wrong cabinet can still project consistently into 3D. A part hidden in the selected views also contributes no mask evidence.
We then held the views, task text, SAM3 backend, and projection procedure fixed and changed only the allowed 3D region (Table 12(a)).
| VLM + SAM3 | Global AP50 | Object-matched AP50 | Structured AP50 |
|---|---|---|---|
| Qwen3.8-Max | 15.1 | 19.7 | 22.8 |
| GPT-6 Sol | 16.5 | 20.5 | 23.6 |
Structured improves on Object-matched by 3.1 AP50 with either VLM. Since the views and visual backend are fixed, the gain supports task-conditioned selection of the 3D region even when the functional masks are produced in 2D.
Figure 3 in the main paper shows the projected 3D masks for drawer, cabinet-door, and toilet-flush tasks, including the two SAM3.1 baselines. Figure 9 adds direct SAM3 and PartSAM comparisons for cabinet, closet, remote-control, and washing-machine tasks. Qualitative results are also available under Foundation Models at https://anonymous.4open.science/w/ActFun3D-Web/#controlled-comparisons
Response to Reviewer CWMd (3/3)
Q4. What problem remains open in fine-grained functional grounding, and how does ActFun3D address it?
Fine-grained functional grounding requires connecting the meaning of an action to the exact 3D surface that supports it. Prior work has made progress on task interpretation, candidate retrieval, and functional scene representation. Fun3DU (CVPR 2025) identifies contextual and functional objects from the instruction and lifts functional masks from selected views. FunGraph (IROS 2025) detects and stores affordance-relevant parts in a 3D scene graph. T-FunS3D (ICRA 2026) uses task decomposition and graph retrieval to focus image-based part segmentation on relevant instances. FunFact (CVPR 2026) jointly reasons about functional relations in an object-and-part map.
For point-level grounding, identifying the relevant object or relation can still leave several plausible places to act. A contextual cue may identify the correct cabinet while leaving several handles inside it. The search stage therefore determines whether the geometry needed to make the finer distinction is available to the local predictor. ActFun3D uses the current task to construct a high-recall search region directly over the 3D points under a fixed budget, then learns to resolve the interaction points inside that region. Selected images refine that point-level prediction.
We test these two decisions separately. Table 2(b) holds candidate point counts and the downstream pipeline fixed to assess which geometry the search retains. Table 3(b) holds the selected region fixed to assess what learned point-level grounding adds. The results establish the contribution of task-conditioned 3D region selection and explain how it works with local prediction to produce the final mask.
Q5. Could ConceptGraphs or OpenFunGraph provide a simpler and more reusable alternative?
Thank you for this suggestion. A reusable scene graph is an interesting direction when multiple tasks are issued in the same scene. ConceptGraphs (ICRA 2024) provides object instances and their relations, and OpenFunGraph (CVPR 2025) adds interactive parts and functional links. Such a representation could share object and context information across instructions and supply candidate regions to the local grounder, especially when several similar objects are present.
The present paper focuses on retaining the relevant 3D geometry for a given task and resolving the exact interaction points. During the rebuttal period, we prioritized the controlled grounding experiments and foundation-model comparisons requested in the reviews. We would like to explore graph-based candidate retrieval in future work to measure how much scene search can be reused across tasks and whether explicit graph relations improve contextual disambiguation before point-level grounding.
Response to Reviewer CWMd (1/3)
Dear Reviewer CWMd,
Thank you for asking us to clarify the contribution of ActFun3D and to evaluate simpler alternatives.
W1. What is new beyond combining existing components, and what justifies the complexity and runtime?
A task such as “open the left door of the cabinet next to the doorway” requires two decisions at different scales. The method must find the correct cabinet among similar instances across the room, then isolate the few points on its handle that support the action. Searching the whole scene under a fixed point budget can lose evidence of the small part during sampling. Starting from the wrong cabinet leaves the later stages with a plausible handle in the wrong location.
Our contribution is task-conditioned narrowing of the 3D point search space before fine-grained functional grounding. The design has three stages. Task parsing and CLIP-based scoring select a high-recall region in the global point cloud. The local grounder resolves the interaction points within that region. View retrieval, VLM verification, and SAM2 then supply visual evidence to refine the existing 3D prediction.
Structured narrowing retains 94.2% of the target within 15.9% of the scene on average. To test whether the semantics select better points under the same budget, we retrained a shared grounder with balanced sampling across region types and matched candidate point counts per task. With that grounder and the same sampling and refinement settings, Structured improves AP50 from 21.6 to 24.5 before refinement and from 27.2 to 30.6 afterward relative to Object-matched (Table 2(b)). Figure 5 shows how changing the instruction from the cabinet by the doorway to the cabinet under the shelf shifts the selected 3D region. The task changes which geometry the local grounder sees, and this improves grounding even at the same candidate size (Appendix D).
The inference settings are shared across tasks. Spatial expansion preserves nearby parts that may score weakly on their own, and the 12-view cap bounds the verification budget. The initial 3D prediction reaches 24.1 AP50 in 16.3 s/task, geometry-based verification reaches 26.2 AP50 in 49.7 s/task, and full refinement reaches 30.8 AP50 in 126.6 s/task, including API latency (Table 3(a)). These measurements show the accuracy gained at each level of visual processing and its cost.
W2. Could direct VLM + SAM or PartSAM + VLM solve the task with fewer stages?
A direct 2D pipeline still needs to decide which images to inspect. In our local copy of the 30 SceneFun3D split0 scenes, we counted 3,218–17,746 captured RGB frames per scene, with a median of 6,078. Evaluating every frame for each task would mean thousands of image evaluations. Generic filtering can remove blur and redundancy, but a sharp frame may show a handle without the doorway or shelf needed to identify it. Task-aware filtering can help and adds a retrieval step. Rendering new views requires selecting a camera trajectory, viewing distances, and a view count. Wide views preserve context but may make the interaction part too small. Close-ups reveal the part but may omit the context that identifies the correct instance.
We evaluated the direct pipeline with up to 12 views retrieved from the captured RGB-D pool. For the stress test, we randomly removed 50% of paired frames per scene before retrieval, keeping the scans, tasks, and view budget fixed. Both segmenters use Qwen3.8-Max and the raw instruction. The full-pool results are from Table 10(a), and the reduced-pool results are from Table 10(c). Appendix B.7 describes the evaluation protocol.
AP50 drops by 1.0 and 1.3 points, respectively, under this random frame-loss setting. PartSAM takes a different route by proposing parts directly in 3D. The selector still needs to distinguish the task-relevant part from similar proposals, and the proposal set must contain that part at the required granularity. We report both PartSAM configurations in Q1.