acceptodds
Withdrawn from ICLR 2027

AcFunSG: Active Evidence Seeking for Open-Vocabulary Functional 3D Scene Graph Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Functional 3D scene understanding goes beyond scene geometry and object-level semantics to model part-level affordances for embodied interaction. However, incomplete or ambiguous observations can introduce errors in cross-view association and functional reasoning that propagate into the scene graph. Existing methods typically construct graphs from a fixed observation set, with limited support for subsequent evidence acquisition and graph revision. To address this limitation, we propose AcFunSG, a training-free, closed-loop framework for open-vocabulary functional 3D scene graph generation that progressively completes and refines an initial graph through uncertainty-guided evidence acquisition. Given posed RGB-D observations, it builds an initial graph through cross-view association and relation-specific global reasoning while retaining entities and relations unresolved by the current evidence. These uncertainties guide further evidence acquisition. Under a limited budget, Ac-Agents selects informative views and queries, integrates the resulting evidence, and revises affected predictions. A deterministic verifier enforces the same consistency constraints used during construction and accepts only updates that pass structural and evidential checks. The graph is progressively completed and corrected, while indeterminate elements remain unresolved. Experiments on SceneFun3D, FunGraph3D, and FunThor show state-of-the-art overall instance recall, with a functional part mapping Recall@3 of 85.1% and relation F1 of 42.0% on FunThor.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Graph Generation;Open-Vocabulary Functional 3D Scene Graph Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_gTmmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a method to iteratively complete and correct predicted functional scene graphs from fixed RGB-D observations through active agentic evidence seeking. The authors argue that the current two-stage, one-shot feedforward pipeline for 3D scene understanding, which identifies object and part nodes first and then infers relations over the fixed detected nodes, is especially fragile for functional parts because they are difficult to detect, and a missed detection prevents all corresponding functional relations from being generated in the second stage. To this end, the authors propose using one agent to identify potentially missing parts and seek evidence and another agent to review the proposals, alternating their roles to progressively complete the graph. The method is benchmarked against baselines on three indoor scene-level datasets, and the results show improvements over the baselines, especially on metrics related to functional parts.

Critical strengths

The paper identifies a meaningful limitation of current functional scene-graph pipelines: missing a functional part during initial perception prevents its associated relations from being recovered later. Using the current graph to guide iterative evidence acquisition is a reasonable solution to this problem.

Critical weaknesses
  1. The source of the improvement is unclear. AcFunSG uses GPT-4.1 for the initial FunFact pipeline but GPT-5.6 Luna during active repair, while also performing substantially more inference. It is therefore difficult to determine whether the gains come from the proposed agentic procedure, the stronger model, or simply additional access to the images. I suggest using GPT-5.6 Luna for the initial scene analysis in FunFact and reporting the results to clarify whether the gains come from feeding the image data to a stronger model during the active evidence acquisition stage.
  2. The computational cost is too high relative to the improvement. Active repair requires an additional 194.5–403.5 logical model calls per scene. This makes it unclear whether the active-agent component provides a favorable performance–cost trade-off.
Questions for authors
  1. The abstract mentions “uncertainty-guided evidence acquisition” and the selection of “informative views and queries.” However, Section 3.4 mainly describes an agent choosing an object, query, view, and update. It is not clear to me how uncertainty and informativeness are represented. How are parent objects selected? How are available views ranked?
  2. In the section “Hierarchical consistency (HC) for multi-view identity association,” the authors state that partial visibility and segmentation errors can cause different geometric extents for the same object, resulting in incorrect merging and motivating hierarchical consistency. The motivation is not clear to me, as detections with different geometric extents would tend to be harder to merge.
  3. Does the system fix existing graph errors, or does it mainly add missing parts? The manuscript repeatedly mentions graph correction. However, Section 3.4 mainly formalizes the addition of missing parts through hypothesis proposals. What graph-edit operations are supported?
Reviewer_zUnVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AcFunSG, a training-free framework for open-vocabulary functional 3D scene graph construction. Given posed multi-view RGB-D observations, the method builds on FunFact’s perception and factor-graph inference pipeline. It introduces Hierarchical Consistency (HC), a bidirectional geometric consistency constraint that improves cross-view instance association, and Relation Reasoning Calibration (RRC), a relative-ranking potential that calibrates functional correspondences expected to be one-to-one. Subsequently, two multimodal model sessions alternate between the roles of proposer and reviewer, revisiting existing observations and invoking perception tools to search for missing functional parts. Candidate updates must receive semantic approval and pass geometric and structural checks; in particular, each newly added part must be supported by at least two distinct views.

The authors evaluate AcFunSG on FunThor, FunGraph3D, and SceneFun3D. Compared with FunFact, functional-part mapping recall on FunThor improves from 69.5% to 85.1%, while relation F1 increases from 38.7% to 42.0%. Overall triplet Recall@10 improves from 63.9% to 67.4% on FunGraph3D and from 57.9% to 65.1% on SceneFun3D. The main contribution is to use hypotheses about missing elements in the current graph to guide subsequent visual search, while constraining graph updates through evidence review.

I appreciate the problem formulation and the value of closed-loop completion, as well as the demonstrated recall gains. However, I currently lean toward a weak rejection. My main concerns are that the additional benefit of the two-session collaboration remains insufficiently isolated from model choice and computation, and that the precision–recall trade-off introduced by active repair is not yet fully characterized, particularly for newly added predictions that do not match the available annotations.

Critical strengths

Quantitative evidence supports functional-part recovery. In Table 3, the full method achieves a functional-part Recall@3 of 85.1%, compared with 78.0% for HC+RRC and 80.1% without role swapping. Under the exclusive-matching evaluation in Appendix B, the improvement over FunFact persists (81.6% vs. 64.5%). The experimental coverage is relatively comprehensive, spanning three benchmarks, component ablations, calibration metrics, a budget curve, and qualitative examples. The combination of semantic approval and deterministic checks also provides a sensible safeguard against committing unconstrained model outputs to the graph.

Critical weaknesses
  1. The benefit of two-session collaboration is not fully isolated. The Role Swap ablation shows gains over the no-swap variant, but the full method is not compared with single-session iterative repair using the same model, tools, and actual computation. Because initialization and repair also use different models, the current results cannot cleanly separate the effect of collaboration from model choice and additional computation. A related gap is that component ablations are conducted only on FunThor. On the real-world datasets, the gains over FunFact are mixed, and SceneFun3D triplet Recall@10 remains below OpenFunGraph (65.1% vs. 70.3%). Comparisons between HC+RRC and the full method on FunGraph3D and SceneFun3D are therefore needed to establish where active repair helps.
  2. Active repair improves coverage but reduces relation precision and relation F1 relative to HC+RRC. In Table 3, recall increases, while relation precision drops from 35.5% to 31.3% and F1 from 44.0% to 42.0%. The authors should clarify when this trade-off is preferable and report how it varies with the repair budget or acceptance threshold. The evaluation also does not separately quantify the correctness of repair-added predictions, especially those unmatched to ground truth. Qualitative examples and multi-view consistency cannot distinguish valid unannotated completions from semantic errors. A human evaluation of a random sample would make the benefit of active repair substantially clearer.
Questions for authors
  1. Under matched models, tools, and actual computation, how does the full method compare with single-session iterative repair?
  2. Can you report relation precision–recall trade-offs under different repair budgets or acceptance thresholds?
  3. Could you compare HC+RRC with the full method on FunGraph3D and SceneFun3D, and report variability across repeated runs?
  4. Could you independently assess a random sample of repair-added predictions, particularly those unmatched to ground truth? Please also clarify the minor numerical discrepancies between Tables 1 and 3 on FunThor.
  5. The proposed method generates scene graph considering cross-view associations, similar idea appeared in “DSM: Building A Diverse Semantic Map for 3D Visual Grounding”, please identify your major difference.
Reviewer_TsPfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a real failure mode of open-vocabulary functional 3D scene graphs. Entities are frozen at feed-forward time of graph generation, so a functional part that is never instantiated can nerver support a functional relation. The author propose a training-free framework AcFunSG, which keeps a FunFact-style pipeline, including open-vocabulary part proposals, hierarchical consistency matching, relative-rank potential-based relation calibration and an active repair loop. The experimental results are claimed to be competitive.

Critical strengths
  1. Treating the current graph as a source of missing-part hypotheses is well motivated. The mechanism that separates the candidate generation from graph commitment is more substantive than unconstrained self-review.
  2. Solid engineering improvements support the usefulness complete pipeline.
  3. The ablation study further prove the effectiveness of the proposed pipoline.
Critical weaknesses
  1. The theoretical improvements are intuitive, lacking analysis or research on the essence of the flaw of FunFact or existing frameworks.
  2. HC and RRC are relatively lightweight extensions to an explicitly inherited pipeline, yet the paper does not provide enough theoretical evidence or reason for these incremental designs. For example, why bidirectional geometric-mean aggregation or the particular rank mapping is preferable to similarly simple alternatives.
  3. AcFunSG uses substantial extra repair computation. The budget curve shows that the complete method benefits from more processing, but does not isolate the benefit of its adaptive selection policy. A matched-budget non-adaptive repair baseline would help distinguish evidence selection from repeated querying and review.

Some minor issues:

  • In Eq. 3, target positions receive discrete ranks, but a source position is mapped into rank space by linear normalization over the extreme coordinates. This is not generally the same as an ordinal correspondence when target spacing is nonuniform.
  • The values of λ in Eq. 2 and the bounding-box expansion δ in Eq. 1 are not clearly provided.
  • ECE-All and ECE-Amb are named and attributed to FunFact, but the confidence quantity, ambiguous subset, and binning used here should be made clear. Table 3 does not show a uniform calibration improvement: ECE-Amb is 0.12 for Full versus 0.07 for FunFact.
Questions for authors
  1. Under the same models, frame pool, verification rules, and inference budget, how does adaptive acquisition compare with random or fixed scheduling? Can you report a consistent accounting between model calls and tool actions, plus one practical cost comparison?
  2. How are relation groups, ranking axes, and reliable ordering determined without ground truth? How is Eq. 3 handled for nonuniform target spacing and degenerate coordinate ranges? How does it compare with a matched-strength distance potential?
  3. How much of the F1 drop from 44.0 to 42.0 is explained by valid predictions absent from the annotations? Can you quantify this without replacing the primary benchmark evaluation?
  4. Please provide \lambda, \delta, the mapping from the named implementation parameters to the equations, executable verification criteria, BP settings, and calibration definitions. Can repeated-run or per-scene uncertainty be reported for the principal comparisons?
Reviewer_RdchOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a method to build functional 3D scene graphs from posed RGB-D frames. In this graph, vertices correspond to scene objects (cabinets, lights) or functional parts (switches, handles), and edges describe a functional reaction between them using free-form natural language (open-vocabulary). To construct and refine the graph, two agents talk to each other suggesting modifications and finding supporting evidence for the modifications - these talking agents seem to be the main contribution. The method is evaluated on three benchmarks (FunGraph3D, SceneFun3D, and FunThor).

Critical strengths

The approach starts from an initial scene graph (based on FunFact) that is then subsequently refined: based on this initial graph, the agents can reason which parts of the graph are missing, the agents then revisit the scene to look again for potentially missing parts. This higher-level reasoning for identifying missing parts, based on the existing evidence is a reasonable usage of agents.

Critical weaknesses

While the paper uses the SceneFun3D dataset, it does not evaluate on the SceneFun3D hidden test set - this would allow to better appreciate the results. It seems possible to leverage the predicted scene graph for the two benchmark tasks “Functionality Segmentation” and “Task-Driven Affordance Grounding”.

On a higher level, while the method evaluates on public benchmarks it does not demonstrate practical down-stream applications which makes me question the relevance of the reported improvements. In a way scene-graph prediction is a made up mid-task that in itself has little relevance, so showing some actual applications would make the submission stronger.

Questions for authors

What is the performance on the SceneFun3D benchmark tasks, in particular on the hidden test split - it seems possible to adapt the predicted scene graph to these tasks?

A similar dataset is Affordance3D, would it make sense to compare the approach on this dataset as well?

Why are two agents needed, could the same be achieved with a single one?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.