acceptodds
Under review as a conference paper at ICLR 2027

ActFun3D: Task-Conditioned Scene Narrowing for Fine-Grained 3D Functional Grounding

Abstract

Fine-grained 3D functional grounding aims to identify the physical region that supports a natural-language task, such as the handle to pull, the button to press, or the control to adjust. This is particularly challenging in large 3D scenes, where task-relevant regions occupy only a small fraction of the scene and are easily confused with visually or geometrically similar candidates. We introduce ActFun3D, a task-conditioned coarse-to-fine framework that uses task semantics to reduce the effective search space before fine-grained grounding. Given a task, ActFun3D first extracts structured functional semantics describing the target, its context, affordance, and spatial constraints, and uses them to localize a high-recall candidate region over language-aligned 3D features. A trainable local 3D grounder then integrates local geometry, appearance, multi-view visual features, and task semantics to predict the task-relevant functional region at the point level. To further resolve ambiguous predictions, ActFun3D retrieves a compact set of informative posed RGB-D views and projects verified visual evidence back into 3D for multi-view refinement. Experiments on SceneFun3D show that ActFun3D achieves 30.8 in fine-grained 3D functional grounding. Notably, task-conditioned narrowing retains 94.2% of the target while reducing the effective search space to only 15.9% of the scene, demonstrating the effectiveness of narrowing large 3D scenes before fine-grained grounding. Project page: https://anonymous.4open.science/w/ActFun3D-Web

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene understanding, Vision-language grounding, Affordance understanding, 3D functional grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CWMdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ActFun3D, which identifies the task-relevant functional region (like a handle, button, or knob), given a natural language task, a global 3D point cloud, and posed RGB-D images. ActFun3D first extracts the structured semantics from the task description, and then combines them with language features in 3D to narrow the scene to a smaller region. Then a trained point-based network predicts the functional masks within this region. In the end, selected RGB-D views are checked by a VLM and segmented using SAM2. This method achieves 30.8 AP50 on SceneFun3D, outperforming the reproduced baselines.

Critical strengths
  1. ActFun3D addresses a challenging problem of localizing small functional regions in large 3D scenes. The proposed approach outperforms the reproduced baselines on the SceneFun3D dataset.
  2. The experimental analysis is relatively comprehensive, including ablations on different components. The paper also provides useful failure analyses and discusses the limitations of the approach.
Critical weaknesses
  1. Limited novelty and unclear core contribution. The proposed method combines several existing components, including LLM parsing, CLIP features, a point-based network, VLM verification, SAM2, and multi-view fusion. This results in a rather complicated pipeline, with many manually tuned parameters and a runtime of 126.6 seconds per task. More importantly, the idea of using task semantics and contextual objects to identify functional parts has already been explored in previous studies, such as Fun3DU and FunGraph. It is therefore not very clear to me what the main methodological contribution is beyond combining these existing techniques.

  2. The need for such a complicated pipeline is not well justified. Given that point clouds and posed RGB-D images are already available, I wonder whether the task could be solved with a much simpler pipeline using recent foundation models. For example, a VLM could directly identify the functional part, followed by SAM 3 for segmentation and projection into 3D. Another possibility would be to use a 3D part segmentation model, such as PartSAM, to generate candidate parts and then use a VLM to select the relevant one. The paper does not provide comparisons with such simpler alternatives or more recent models, making it difficult to judge whether the proposed complexity is really necessary.

Questions for authors
  1. Necessity of the trained network. Is the trained LocalFunctionalGrounder really necessary? Could existing foundation models, such as SAM 3 or PartSAM, achieve comparable results without training a dedicated network?
  2. Structured semantic parsing. Given the capabilities of recent VLMs, is explicit structured semantic parsing still necessary? Could a VLM directly understand the task and identify the relevant functional part without this intermediate step?
  3. Direct grounding with stronger VLMs. The paper uses Qwen3-8B for task parsing and qwen3.8-flash for visual verification. Could larger and stronger VLMs directly identify functional targets from the task and RGB-D images, followed by segmentation and projection into 3D? How would such a simpler pipeline compare with ActFun3D?
  4. Related work and novelty. Several relevant recent works are missing from the discussion, including FunGraph (IROS 2025), T-FunS3D (ICRA 2026), and FunFact (CVPR 2026). Could the authors discuss these works and clarify the novelty of ActFun3D compared with existing approaches?
  5. Explicit 3D mapping. The authors could also discuss whether explicit semantic maps or scene graphs, such as ConceptGraphs or OpenFunGraph, could provide a simpler and more reusable alternative.
5 comments
AuthorsOpenReview ↗

Response to Reviewer CWMd (1/3)

Dear Reviewer CWMd,

Thank you for asking us to clarify the contribution of ActFun3D and to evaluate simpler alternatives.

W1. What is new beyond combining existing components, and what justifies the complexity and runtime?

A task such as “open the left door of the cabinet next to the doorway” requires two decisions at different scales. The method must find the correct cabinet among similar instances across the room, then isolate the few points on its handle that support the action. Searching the whole scene under a fixed point budget can lose evidence of the small part during sampling. Starting from the wrong cabinet leaves the later stages with a plausible handle in the wrong location.

Our contribution is task-conditioned narrowing of the 3D point search space before fine-grained functional grounding. The design has three stages. Task parsing and CLIP-based scoring select a high-recall region in the global point cloud. The local grounder resolves the interaction points within that region. View retrieval, VLM verification, and SAM2 then supply visual evidence to refine the existing 3D prediction.

Structured narrowing retains 94.2% of the target within 15.9% of the scene on average. To test whether the semantics select better points under the same budget, we retrained a shared grounder with balanced sampling across region types and matched candidate point counts per task. With that grounder and the same sampling and refinement settings, Structured improves AP50 from 21.6 to 24.5 before refinement and from 27.2 to 30.6 afterward relative to Object-matched (Table 2(b)). Figure 5 shows how changing the instruction from the cabinet by the doorway to the cabinet under the shelf shifts the selected 3D region. The task changes which geometry the local grounder sees, and this improves grounding even at the same candidate size (Appendix D).

The inference settings are shared across tasks. Spatial expansion preserves nearby parts that may score weakly on their own, and the 12-view cap bounds the verification budget. The initial 3D prediction reaches 24.1 AP50 in 16.3 s/task, geometry-based verification reaches 26.2 AP50 in 49.7 s/task, and full refinement reaches 30.8 AP50 in 126.6 s/task, including API latency (Table 3(a)). These measurements show the accuracy gained at each level of visual processing and its cost.

W2. Could direct VLM + SAM or PartSAM + VLM solve the task with fewer stages?

A direct 2D pipeline still needs to decide which images to inspect. In our local copy of the 30 SceneFun3D split0 scenes, we counted 3,218–17,746 captured RGB frames per scene, with a median of 6,078. Evaluating every frame for each task would mean thousands of image evaluations. Generic filtering can remove blur and redundancy, but a sharp frame may show a handle without the doorway or shelf needed to identify it. Task-aware filtering can help and adds a retrieval step. Rendering new views requires selecting a camera trajectory, viewing distances, and a view count. Wide views preserve context but may make the interaction part too small. Close-ups reveal the part but may omit the context that identifies the correct instance.

We evaluated the direct pipeline with up to 12 views retrieved from the captured RGB-D pool. For the stress test, we randomly removed 50% of paired frames per scene before retrieval, keeping the scans, tasks, and view budget fixed. Both segmenters use Qwen3.8-Max and the raw instruction. The full-pool results are from Table 10(a), and the reduced-pool results are from Table 10(c). Appendix B.7 describes the evaluation protocol.

SegmenterAvailable frame poolAP50AP25mIoU
SAM3Full pool12.424.911.4
SAM350% randomly removed11.423.310.3
SAM3.1Full pool12.825.011.5
SAM3.150% randomly removed11.523.310.5

AP50 drops by 1.0 and 1.3 points, respectively, under this random frame-loss setting. PartSAM takes a different route by proposing parts directly in 3D. The selector still needs to distinguish the task-relevant part from similar proposals, and the proposal set must contain that part at the required granularity. We report both PartSAM configurations in Q1.

AuthorsOpenReview ↗

Response to Reviewer CWMd (2/3)

Q1. Can SAM3 or PartSAM replace the trained LocalFunctionalGrounder?

Under the evaluated settings, both foundation-model alternatives leave a substantial performance gap. W2 explains the practical challenges of these routes: finding views that show both the part and its context, and ensuring that the 3D proposal set contains the part specified by the task. We tested both alternatives on all 445 SceneFun3D split0 tasks. The 2D route uses a VLM to localize the part and SAM3 to segment it for projection into 3D. The 3D route uses PartSAM to generate candidates from scene object proposals and a VLM to select the task-relevant part. We generated the proposals without ground-truth object crops or ActFun3D’s learned predictions (Table 10, Appendix B.7).

MethodAP50AP25mIoU
GPT-6 Sol + SAM3, Structured input16.529.714.3
PartSAM + Qwen3.8-Max18.230.415.1
PartSAM + GPT-6 Sol19.031.315.7
ActFun3D30.841.523.0

We then isolated the grounder’s contribution within ActFun3D. We kept the Structured crop, retrieval rule, 12-view cap, VLM/SAM2 backend, and signed refinement update fixed, and replaced the learned point scores with coarse narrowing scores. AP50 dropped from 30.8 to 18.4. Visibility voting raised coarse support to 19.2 AP50. With SAM3 and stronger VLMs, learned support still improved AP50 by 7.5 points with Qwen3.8-Max and 7.1 points with GPT-6 Sol (Tables 3(b) and 12(b)).

This control gives both variants the same candidate region. The difference is how precisely they identify the interaction points inside it. Those point scores guide view retrieval and give visual refinement a more precise 3D prediction to verify (Appendix D).

Q2. Does structured parsing still help a strong VLM that can interpret the raw task directly?

We gave both VLMs the raw instruction and then repeated the experiment with the structured description while keeping the selected views and segmenter fixed (Table 10(a)).

Direct pipelineRaw AP50Structured AP50Gain
Qwen3.8-Max + SAM312.415.1+2.7
Qwen3.8-Max + SAM3.112.815.4+2.6
GPT-6 Sol + SAM314.216.5+2.3
GPT-6 Sol + SAM3.114.616.3+1.7

The raw instruction gives the VLM enough information to attempt localization. The structured description makes the interaction target explicit. For example, in “turn on the TV using the remote,” it identifies the remote’s power button as the part to manipulate and retains the TV as context. The gains across all four combinations show that this distinction helps even when the same strong VLM sees the same images.

Q3. How does direct grounding with stronger VLMs perform after projection into 3D?

The best raw-task configuration reaches 14.6 AP50, and the best structured-input configuration reaches 16.5, compared with 30.8 for ActFun3D. The direct baselines use depth and camera poses for projection, with visibility and cross-view consistency checks (Table 10(a), Appendix B.7). These checks can filter inconsistent evidence, but a mask on the wrong cabinet can still project consistently into 3D. A part hidden in the selected views also contributes no mask evidence.

We then held the views, task text, SAM3 backend, and projection procedure fixed and changed only the allowed 3D region (Table 12(a)).

VLM + SAM3Global AP50Object-matched AP50Structured AP50
Qwen3.8-Max15.119.722.8
GPT-6 Sol16.520.523.6

Structured improves on Object-matched by 3.1 AP50 with either VLM. Since the views and visual backend are fixed, the gain supports task-conditioned selection of the 3D region even when the functional masks are produced in 2D.

Figure 3 in the main paper shows the projected 3D masks for drawer, cabinet-door, and toilet-flush tasks, including the two SAM3.1 baselines. Figure 9 adds direct SAM3 and PartSAM comparisons for cabinet, closet, remote-control, and washing-machine tasks. Qualitative results are also available under Foundation Models at https://anonymous.4open.science/w/ActFun3D-Web/#controlled-comparisons

AuthorsOpenReview ↗

Response to Reviewer CWMd (3/3)

Q4. What problem remains open in fine-grained functional grounding, and how does ActFun3D address it?

Fine-grained functional grounding requires connecting the meaning of an action to the exact 3D surface that supports it. Prior work has made progress on task interpretation, candidate retrieval, and functional scene representation. Fun3DU (CVPR 2025) identifies contextual and functional objects from the instruction and lifts functional masks from selected views. FunGraph (IROS 2025) detects and stores affordance-relevant parts in a 3D scene graph. T-FunS3D (ICRA 2026) uses task decomposition and graph retrieval to focus image-based part segmentation on relevant instances. FunFact (CVPR 2026) jointly reasons about functional relations in an object-and-part map.

For point-level grounding, identifying the relevant object or relation can still leave several plausible places to act. A contextual cue may identify the correct cabinet while leaving several handles inside it. The search stage therefore determines whether the geometry needed to make the finer distinction is available to the local predictor. ActFun3D uses the current task to construct a high-recall search region directly over the 3D points under a fixed budget, then learns to resolve the interaction points inside that region. Selected images refine that point-level prediction.

We test these two decisions separately. Table 2(b) holds candidate point counts and the downstream pipeline fixed to assess which geometry the search retains. Table 3(b) holds the selected region fixed to assess what learned point-level grounding adds. The results establish the contribution of task-conditioned 3D region selection and explain how it works with local prediction to produce the final mask.

Q5. Could ConceptGraphs or OpenFunGraph provide a simpler and more reusable alternative?

Thank you for this suggestion. A reusable scene graph is an interesting direction when multiple tasks are issued in the same scene. ConceptGraphs (ICRA 2024) provides object instances and their relations, and OpenFunGraph (CVPR 2025) adds interactive parts and functional links. Such a representation could share object and context information across instructions and supply candidate regions to the local grounder, especially when several similar objects are present.

The present paper focuses on retaining the relevant 3D geometry for a given task and resolving the exact interaction points. During the rebuttal period, we prioritized the controlled grounding experiments and foundation-model comparisons requested in the reviews. We would like to explore graph-based candidate retrieval in future work to measure how much scene search can be reused across tasks and whether explicit graph relations improve contextual disambiguation before point-level grounding.

Reviewer_bFL6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ActFun3D proposes a coarse-to-fine framework for fine-grained 3D functional grounding. The pipeline consists of task parsing, task-conditioned scene narrowing, local 3D functional segmentation, view selection, VLM verification and SAM2 segmentation, and projection back into 3D for refinement. The central idea is to use task semantics to concentrate a limited point-cloud processing budget on a relevant region before resolving small functional targets with a local grounding network.

Critical strengths
  1. The scale mismatch between large scenes and small functional targets is well motivated. Reporting both search-space ratio and target retention makes the narrowing stage directly assessable. Retaining 94.2% of target points within 15.9% of the scene demonstrates useful candidate-region selection.

  2. The complete method improves AP50, AP25, and mIoU over the reproduced baselines. The refinement ablations provide evidence for the benefits of task-aware verification and visibility-aware evidence aggregation.

Critical weaknesses
  1. The narrowing experiments don’t really separate semantic selection from point-budget allocation. Table 2 gives the same 16,384-point budget to whole-scene and narrowed inputs. So narrowing mainly increases sampling density around potential targets and makes it less likely to lose small functional regions during subsampling. That’s a real practical benefit, but the comparison still doesn’t tell us how much comes from structured semantics and how much from spatial concentration.
    Also, the similarity-and-clustering formulation never clearly says how relations between target and context objects are resolved.
  2. The necessity of the Local 3D Grounding stage is not established. Scene narrowing already produces a task-conditioned candidate region. It is therefore unclear whether a separate trainable local grounder is needed before VLM verification and SAM2 segmentation.
Questions for authors
  1. Can the authors better separate structured semantic selection from point-budget concentration, for example through matched-size candidate regions? How reliably does narrowing distinguish semantically similar candidates that differ primarily in their spatial relationship to context objects?

  2. How does the complete pipeline perform when the local grounder is replaced with a simpler support derived from the narrowed region, under comparable view-selection and visual-verification budgets?

  3. Although the paper states that it follows SceneFun3D Task-2, could the authors confirm whether the reported AP25/AP50 were computed using the official IoU-based instance evaluator? Some public Fun3DU/AffordBot implementations label thresholded pointwise precision as “AP,” which can favor incomplete predictions and is not directly comparable with official benchmark results. If this alternative evaluation was used, could the authors reevaluate their method and key baselines using the official SceneFun3D protocol?

3 comments
AuthorsOpenReview ↗

Response to Reviewer bFL6 (1/2)

Dear Reviewer bFL6,

Thank you for your careful reading and for asking us to clarify the roles of semantic selection, spatial context, and learned local grounding.

W1. The whole-scene comparison mixes semantic region selection with the effect of concentrating a fixed point budget.

The original comparison tests whether narrowing makes better use of the same 16,384-point budget. The new sampling diagnostics show how it does so. The grounder directly receives 75.0% of the target points after sampling from the Structured region, compared with 32.1% under whole-scene sampling (Table 11).

To test whether structured semantics help decide where to spend that budget, we retrained a shared grounder with balanced sampling across region types. We kept the architecture, optimizer, loss, number of training steps, and effective batch size the same as for the original grounder. At inference, the variants use the same checkpoint, grounding description, sampling procedure, and input budget, with candidate point counts matched per task. Structured improves initial AP50 from 21.6 to 24.5 over Object-matched, so the benefit is already measurable before visual refinement (Table 2(b)).

We observe the same effect in a direct mask-projection pipeline. With the views and projection procedure fixed, Structured improves AP50 over Object-matched by 3.1 points with each of two VLM backends (Table 12(a)). Together, these comparisons support the role of structured semantics in selecting which part of the scene receives the point budget.

W2. How do target–context relations affect the similarity scores and clustering?

The parser retains the context object and spatial relation wording in the textual cues used for narrowing. These cues affect the scores assigned to 3D points in Eq. (3). Clustering then groups high-scoring points into spatially coherent regions. In this way, the relation influences which region has stronger semantic support before clustering.

Figure 5 shows two instructions in the same room. Both ask to open the left door of a wooden cabinet. Changing the reference from “to the right of the doorway” to “under the shelf with the picture frames” directs the Structured query to a different cabinet. Appendix A.1 explains how these cues enter the scoring.

W3. What does the trainable local grounder add when narrowing already provides a task-conditioned region?

A crop can retain nearly all of a handle and still contain the entire door and nearby handles. This is the distinction behind the 94.2% target retention: it measures how much of the target is available for grounding. The local grounder uses geometry, visual features, and task cues to identify the interaction points within that region. Its prediction guides view retrieval and gives the verifier more precise 3D support to check. Table 3(b) tests this role by fixing the crop and replacing the learned scores with coarse narrowing scores. Appendix D discusses why candidate coverage and point-level grounding serve different purposes.

AuthorsOpenReview ↗

Response to Reviewer bFL6 (2/2)

Q1. Can matched-size regions separate semantic selection from budget concentration? How well does narrowing distinguish similar candidates using their spatial relation to context objects?

At matched candidate point counts, Structured retains 94.2% of the target and reaches 30.6 refined AP50, compared with 88.6% and 27.2 for Object-matched. The non-semantic crop, which also matches point density, retains 18.6% and reaches 4.8 refined AP50. With the grounder and refinement procedure held fixed, these results show that regions with the same number of points can differ substantially in how much useful geometry they preserve (Table 2(b)).

We tested the role of spatial context on 277 tasks with explicit target–context relations, selected from the descriptions independently of model predictions. We changed only the query used for narrowing. The shared grounder, its full task description, and the sampling procedure were kept fixed.

Narrowing queryTarget retained before sampling (%)Initial AP50Initial mIoU
Complete query92.819.215.9
Without the context anchor85.414.612.9
Without the relation phrases87.713.812.3

Removing the relation phrases reduces target retention from 92.8% to 87.7% and initial AP50 by 5.4 points, even though the grounder still receives the full instruction. This shows why the relation matters at the narrowing stage: telling the grounder where the target should be cannot bring back geometry already excluded from its input region. Figure 5 illustrates this for similar cabinets in the same scene (Table 12(c), Appendix C.6).

Q2. How does the full pipeline perform when coarse support replaces the learned grounder under comparable view-selection and verification budgets?

We replaced the learned point scores with normalized narrowing scores inside the same Structured crop. Each variant retrieves views from its own initial support, using the same retrieval rule, 12-view cap, VLM verifier, and SAM2 segmenter. The selected views can differ because the initial support guides retrieval. This lets us evaluate the downstream effect of the support under a common budget.

Initial support and refinementAP50AP25mIoU
Coarse support + signed update18.427.614.7
Coarse support + visibility voting19.228.515.2
Learned support + signed update30.841.523.0

With the signed update fixed, learned support improves AP50 by 12.4 points. We also tested visibility voting so coarse support could aggregate visual evidence through a different rule. It reaches 19.2 AP50. With SAM3 and stronger verifiers, learned support still improves AP50 from 23.4 to 30.9 with Qwen3.8-Max and from 24.2 to 31.3 with GPT-6 Sol. Across these settings, a more precise initial 3D prediction gives the visual stage better support to verify (Tables 3(b) and 12(b)).

Qualitative comparisons are available under Local Grounder at https://anonymous.4open.science/w/ActFun3D-Web/#controlled-comparisons

Q3. Were AP25 and AP50 computed with the official SceneFun3D Task-2 IoU-based instance evaluator?

Yes. We used the unmodified official SceneFun3D Task-2 evaluator on split0 for ActFun3D and the reproduced baselines, including Fun3DU and AffordBot. For ActFun3D, the 0.5 point-probability threshold is used to form the foreground mask. We group its spatial components into scored instances and export them in the original Faro point order. AP25 and AP50 then come from the official evaluator’s instance matching at mask IoU thresholds of 0.25 and 0.50, respectively. We converted the reproduced baseline predictions to the same Task-2 format and evaluated them with the same script. Pointwise precision is reported separately as a diagnostic (Appendices B.4 and B.7).

AuthorsOpenReview ↗

Follow-up on Our Response and Revision

Dear Reviewer bFL6,

Apologies for reaching out again during the discussion period, and thank you for your valuable review and feedback. Regarding your concerns about the necessity of the LocalFunctionalGrounder, we have provided additional controlled experiments and updated the manuscript accordingly. We would appreciate hearing whether our responses and revisions have addressed your main concerns, or if any questions remain.

Thank you for your time and consideration.

Reviewer_NkU7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ActFun3D targets fine-grained 3D functional grounding on SceneFun3D. The pipeline has four stages. An LLM parses the task into a structured query. The query is matched against lifted CLIP features to narrow the scene to a candidate region. A trainable local grounder predicts the functional region inside it. A VLM and SAM2 then refine the prediction using a few retrieved views. On average the narrowing stage keeps 15.9% of the scene and retains 94.2% of the target. On split0, which has 30 scenes and 445 tasks, ActFun3D reaches 30.8 AP50, 41.5 AP25 and 23.0 mIoU. This beats the best reproduced baselines by 3.9, 8.2 and 3.3 points.

Critical strengths
  1. Clear diagnostics. SSR and TR make the trade-off between search-space reduction and target retention explicit. The paper also reports the failure tail: 37 of 445 tasks keep less than 80% of the target. This shows where errors enter the cascade.

  2. Credible gains. All baselines are evaluated with the same official evaluator. The component ablations, the runtime numbers and the separate UniFunc3D comparison give useful context. The results are easy to interpret.

Critical weaknesses
  1. The contribution of task-conditioned narrowing is not isolated. Table 2 is meant to show this, but it has three confounds. First, the Whole Scene row is capped at 16,384 points, so its low score mixes semantic filtering with point density. Second, all rows share one grounder that was trained partly on structured-narrowing regions. The structured row is therefore in-distribution and the other rows are not. Third, the variants differ in both region size and target retention, so size is never controlled.

  2. The end-to-end evidence is also missing. Without refinement, ActFun3D reaches 24.1 AP50 and trails TASA at 26.9. The margin over baselines appears only after VLM and SAM2 refinement, which raises runtime from 16.3 to 126.6 s per task. A size-matched control with the same refinement is needed to show what narrowing adds.

Questions for authors
  1. What are the point counts of full scenes and of narrowed regions? How often is the 16,384-point cap reached, and how much of the target survives subsampling?
  2. Can the authors add a non-semantic crop matched in region size and point density, plus an oracle crop centered on the target? Please run the same refinement on these controls so the end-to-end benefit can be judged.
  3. Which data were used to choose the parser prompt, the narrowing parameters and K? Can the authors report scene-level bootstrap intervals for the main differences?
3 comments
AuthorsOpenReview ↗

Response to Reviewer NkU7 (1/2)

Dear Reviewer NkU7,

Thank you for pointing out the confounds in the original comparison and suggesting matched controls.

W1. The original Table 2 mixes the effects of semantic selection and sampling density. The grounder was trained partly on Structured regions, and the candidate regions differ in size and target retention.

We trained a new shared grounder with balanced sampling across region types, keeping the original architecture, optimizer, and number of training steps. We then matched candidate point counts per task and used the same grounding text, sampling procedure, and 16,384-point budget. Under these controls, Structured retains 94.2% of the target, compared with 88.6% for Object-matched, and improves initial AP50 from 21.6 to 24.5. At the same candidate size and with the same grounder, the structured query selects a region that contains more of the target. This advantage is already measurable before visual refinement (Section 4.3, Table 2(b)).

W2. Initial ActFun3D trails TASA, and the advantage over the reproduced baselines appears after costly visual refinement. Does narrowing contribute when refinement is held fixed?

Initial ActFun3D reaches 24.1 AP50 in 16.3 s/task, while TASA reaches 26.9 AP50. Full ActFun3D reaches 30.8 AP50 in 126.6 s/task, including API latency for view verification (Tables 1 and 3(a)). To test the contribution of narrowing within the full pipeline, we applied the same view retrieval rule, view budget, and refinement procedure to both matched regions. Structured reaches 30.6 AP50, versus 27.2 for Object-matched (Table 2(b)). The 3.4-point gap shows that choosing a better 3D candidate region improves the final result under the same refinement procedure.

AuthorsOpenReview ↗

Response to Reviewer NkU7 (2/2)

Q1. How many points are in the full scenes and narrowed regions, how often is the 16,384-point budget exceeded, and how much of the target survives sampling?

Across 445 tasks, we counted candidate points in the 2 cm voxelized point cloud. Input retention measures the fraction of target points actually passed to the grounder, before nearest-neighbor propagation.

RegionCandidate points, median [P10, P90]Tasks over budgetTarget retained before samplingTarget retained in input
Whole Scene398,089 [148,382, 1,164,024]445/445 (100.0%)100.0%32.1%
Raw Task94,765 [30,857, 264,320]434/445 (97.5%)80.8%44.0%
Functional Object66,918 [22,612, 197,164]426/445 (95.7%)89.4%62.2%
Structured58,866 [20,485, 174,504]415/445 (93.3%)94.2%75.0%

Structured narrowing retains an average of 15.9% of the scene. Most crops still exceed the input budget, yet the grounder sees 75.0% of the target on average, compared with 32.1% under whole-scene sampling (Tables 2(a) and 11, Appendix C.5).

Q2. How do a size- and density-matched non-semantic crop and a ground-truth-centered oracle perform under the same refinement?

We matched candidate point counts per task and additionally matched point density for the non-semantic crop. The oracle uses the ground-truth target location to center its crop. All three use the same grounder, sampling procedure, view budget, and refinement settings.

RegionInitial AP50Refined AP50Refined AP25Refined mIoU
Non-semantic matched3.44.88.54.2
Structured24.530.641.222.9
GT-centered oracle28.034.045.826.0

The non-semantic crop retains 18.6% of the target before sampling, versus 94.2% for Structured. Matching size and density still leaves a large difference in target coverage. Visual refinement suppresses unsupported predictions within the initial 3D support, so it cannot recover target points excluded by the crop (Appendix D). The target-centered oracle reaches 34.0 AP50, compared with 30.6 for Structured under the same refinement (Table 2(b)). Qualitative comparisons are available at https://anonymous.4open.science/w/ActFun3D-Web/#controlled-comparisons

Q3. Which data informed the parser prompt, narrowing parameters, and view budget KK? Can you report scene-level bootstrap intervals?

We designed the parser prompt around the structure of SceneFun3D task descriptions. It separates the part to manipulate from the object whose state changes. For example, “turn on the TV using the remote” should direct the search toward the remote's power button. For narrowing, we tried different settings with the goal of retaining the target in a compact crop. The target cue receives the highest weight to locate plausible parts, context helps distinguish similar candidates, and spatial expansion keeps nearby functional parts whose semantic scores may be weak. We use one configuration across scenes. The available records do not identify a specific split for the preliminary prompt and narrowing trials (Appendices B.1 and B.3).

We chose K=12K=12 to give the verifier several complementary views under occlusion while limiting per-view verification and segmentation. In the split0 sweep, increasing KK from 6 to 12 improves AP50 from 29.4 to 30.8. Doubling it to 24 yields 30.4 AP50 and only 0.1 more AP25, supporting the twelve-view cap (Table 9(b)).

The refined Structured versus Object-matched difference (+3.4 AP50) and the full ActFun3D versus TASA difference (+3.9 AP50) are point estimates. Scene-level bootstrap intervals for these differences remain unavailable.

AuthorsOpenReview ↗

Follow-up on Our Response and Revision

Dear Reviewer NkU7,

Apologies for reaching out again during the discussion period, and thank you for your valuable review and feedback. Regarding your concerns about the fairness of scene-narrowing comparisons, we have provided additional controlled experiments and updated the manuscript accordingly. We would appreciate hearing whether our responses and revisions have addressed your main concerns, or if any questions remain.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.