acceptodds
Under review as a conference paper at ICLR 2027

Let the Scene Decide: Retrieval-Augmented 3D Reasoning Segmentation

Abstract

3D reasoning segmentation predicts a mask in a scanned scene from a query that describes an object's purpose without naming it. Existing systems learn this mapping end to end from annotated scans with narrow indoor vocabularies, and struggle when the target lies outside them. The bottleneck is often grounding rather than language: an LLM can infer the intended category, but the model has not learned its 3D appearance. We propose Retrieval-Augmented 3D Reasoning Segmentation (RA3D), which supplies this missing visual knowledge at inference time. An LLM proposes candidate categories, retrieves images for each, and lifts them into point clouds. Each exemplar is placed into the query scene before encoding, reducing the gap between isolated objects and scene geometry. RA3D matches exemplars through background-contrastive similarity maps, ranks candidates with parameter-free distribution-calibrated scores, and independently verifies shortlisted masks against the retrieved images, which also resolves category selection. RA3D requires no training and can be integrated with existing models. We further introduce NO-Reason3D, a benchmark whose targets lie outside the indoor vocabularies used for training. RA3D improves trained models on Reason3D without retraining and outperforms all evaluated methods on NO-Reason3D by a wide margin. The code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Reasoning Segmentation, Retrieval-Augmented Segmentation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LEvCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work presents RA3D: an LLM generates candidate categories from implicit queries; web-retrieved images are reconstructed into 3D exemplars and embedded into the scene for contextual encoding. Target masks are obtained via background contrastive matching, distribution calibration and independent visual verification. The authors further propose NO-Reason3D for out-of-vocabulary targets and demonstrate RA3D’s gains on Reason3D, Instruct3D and a new benchmark.

Critical strengths
  1. The technical design of introducing external visual knowledge at inference time constitutes a substantial contribution. Instead of task-specific training, RA3D reconstructs web exemplars into 3D and sequentially addresses the isolated-object/context gap, cross-exemplar score incomparability, and candidate self-verification. Ablations in Tables 3, 6, 8 and 9 collectively support that these performance gains are not incidental benefits from a single module. Prior works such as OpenScene and PLA have established open-vocabulary 3D perception, yet their core mechanisms differ from this inference-time visual exemplar retrieval, so the proposed direction is clearly distinct.

  2. The merit of the method does not fully rely on the new benchmark. On the real-world Reason3D dataset, Reason3D+RA3D improves mIoU over the vanilla model from 31.20 to 34.12 on ScanNet and from 19.54 to 23.22 on Matterport3D. It also lifts performance from 11.70 to 13.14 on Instruct3D. These results demonstrate that even with a more conservative view on the generalization claims of NO-Reason3D, the retrieval branch still delivers standalone practical value.

Critical weaknesses
  1. NO-Reason3D experiments fail to separate gains from retrieval versus open-vocabulary 3D perception. Table 1 only compares conventional reasoning segmentation baselines (3D-STMN, Reason3D). A fairer control uses identical Qwen category predictions but replaces exemplar retrieval with off-the-shelf open-vocabulary models like OpenScene/PLA. Existing results only beat closed-vocabulary baselines and cannot confirm retrieval drives novel-object improvements. The authors should add these baselines with [email protected], [email protected] and mIoU, including oracle-category tests to decouple language and visual grounding errors under fixed scenes and candidate classes.
  2. NO-Reason3D’s full CAD mesh insertion may bias results toward RA3D’s complete exemplar matching. While insertion artifacts are mitigated, NO-Reason3D uses full meshes unlike partial surfaces from real scans. RA3D also reconstructs complete object geometry for verification, so the benchmark may favor shape matching. The authors should report geometric statistics (visible surface ratio, point density) between CAD inserts and real scans. Preferably, add scan-like occlusion to inserted objects and re-evaluate, or include a geometry-only baseline to rule out spurious gains from intact CAD models.
Questions for authors

1.On the NO-Reason3D validation set, what are Qwen’s top-1 and top-3 recall for target categories? Among RA3D’s failure cases, what fraction has the correct category missing from the Qwen top-3 predictions? This statistic directly tests the paper’s motivation that the main bottleneck lies in visual grounding rather than language reasoning.

2.Have the authors tested OpenScene, PLA or other open-vocabulary 3D models using exactly the same Qwen-predicted categories as RA3D? If yes, please provide NO-Reason3D mIoU and Acc results. If not, specify concrete barriers in input modality or evaluation protocol. Since OpenScene and PLA natively support language queries for unseen classes, this comparison may substantially change my assessment of the main empirical contributions.

3.Do the authors have completeness or visibility statistics comparing the full meshes inserted in NO-Reason3D against real objects from ScanNet/Matterport3D, or existing results on partially occluded CAD objects? Appendix G shows geometric completeness changes how verification depends on view count, so this information shapes my interpretation of benchmark realism and the large gains reported in Table 1.

Reviewer_VxoaOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a 3D scene segmentation pipeline called RA3D. The pipeline is training-free and can be plugged into existing methods. Its main goal is to improve the segmentation of out-of-distribution objects and objects from long-tail categories in 3D reasoning segmentation. The approach is as follows: first, an LLM predicts object categories, and corresponding images are retrieved. A 3D generative model then generates 3D exemplars of these objects, which are placed in the scene. Next, contextual encoding, background-contrastive matching, calibrated scoring, and visual verification produce the final segmentation. The paper also introduces a test set called NO-Reason3D to evaluate models’ ability to segment object categories unseen during training. RA3D improves performance on Reason3D when combined with existing models and outperforms the baseline methods on NO-Reason3D.

Critical strengths

I believe that using retrieval at inference time to supplement visual knowledge of new categories is a meaningful research direction. Existing 3D reasoning segmentation models are often limited by the coverage of categories in their training data. The pipeline proposed in this paper supplements frozen segmentation systems with external visual knowledge, providing a way to expand the set of target categories without updating model parameters. This idea offers some useful insights, especially for scenarios where continually collecting 3D annotations for new categories is difficult. Moreover, the pipeline appears to work as a plug-in that can be combined with existing models to improve performance, suggesting that the information provided by retrieval is somewhat complementary to the existing models. The paper provides extensive experimental analysis, numerous comparisons with alternatives, and experiments examining design trade-offs, making the pipeline’s design convincing.

Critical weaknesses
  1. The paper’s 2D-bank pipeline relies on multiple sequential components but lacks a systematic analysis of failure cases and quantitative errors. Specifically, the candidate categories, category scales, and hints about placement surface types come from an LLM, while the 3D reference exemplars are generated by TRELLIS from images. Although the authors introduce candidate verification and discuss the relevant limitations, for a paper centered on a pipeline, the ability of these mechanisms to correct upstream errors has not been sufficiently demonstrated. My core concern is that errors may propagate through successive stages and affect the final segmentation.

  2. As a training-free plug-in, the method’s cost–benefit trade-off does not seem ideal. The paper states that each query that misses the cache requires 30 image-to-3D shape generation operations, in addition to LLM calls, repeated 3D encoding, and rendering-based verification, all of which are time-consuming components. Table 10 reports that a single query still takes approximately 45 seconds even with 15 processes running in parallel. This means that, in complex real-world tasks, one must either accept long waiting times or maintain a relatively large 3D database. Even with such a complex pipeline, the method’s gains on existing benchmarks are relatively limited, with the larger improvements mainly appearing on the authors’ own benchmark.

Questions for authors
  1. Line 419 suggests that improvements in single-image reconstruction quality could lead to downstream gains, but this statement contradicts the results in Table 4: the stronger TRELLIS2 actually performs worse than TRELLIS. Have the authors considered the following issue? The generated 3D shapes are watertight meshes, whereas scene scans are non-watertight (meaning that object scans may have missing parts and that there may be no clear boundaries between the point clouds of different objects). In my experience, there are some differences between the features of watertight and non-watertight point clouds. The resulting issue is that both the NO-Reason3D test set and the RA3D method use clean, complete geometry, which naturally makes matching easier. I would like to see results on real scans for categories outside the existing training vocabulary.

  2. More baselines. Although the paper focuses on reasoning segmentation, the “reasoning” relies entirely on LLM outputs. I think this problem is approximately equivalent to segmentation given an object category. If my understanding is correct, could first using the same LLM category predictions as RA3D and then passing them to an off-the-shelf open-vocabulary 3D segmentation model achieve similar results? However, the paper lacks comparisons with such methods. Please report the metrics for these baselines; alternatively, would adding RA3D to these models still yield improvements?

  3. Please provide some failure cases and corresponding analyses to show how the model behaves when the LLM or TRELLIS makes mistakes, as well as errors in the model’s own predictions. This would help me assess the robustness of the pipeline.

Reviewer_sog4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

RA3D segments a 3D scene from a query that describes what an object is for, without naming the category. An LLM proposes categories; the method retrieves images, lifts them to point clouds, places each exemplar into the query scene, and then matches and verifies it. The authors also introduce NO-Reason3D. They insert 12 ShapeNet categories into ScanNet and Matterport3D scans and pair each object with a human-checked query that does not name it.

Critical strengths

0 Reason3D and similar benchmarks test categories the model has already been trained on. NO-Reason3D uses 12 ShapeNet categories outside that vocabulary. Every inserted object sits on a supporting surface, does not intersect the scan, and is resampled to the scan’s point density.

  • With a 2D retrieval bank, NO-Reason3D mIoU is 14.50 / 20.08, against 4.19 / 5.21 for Reason3D and under 2 for 3D-STMN.
  • Routing complements trained models: Reason3D mIoU on ScanNet rises from 31.20 to 34.12. Ablations of placement, in-scene encoding, and verification move in the same direction, and the paper reports run-to-run variance.
Critical weaknesses
  • On categories the model has already seen, retrieval alone is weaker. On Reason3D ScanNet, mIoU is 31.20 for Reason3D and 20.93 for RA3D with the 2D bank. The gain appears on categories outside the training vocabulary.
  • The new objects are ShapeNet meshes placed into real scans. Resampling matches the local point density, but the geometry and occlusion still come from a CAD model rather than from the scanner.
  • Absolute accuracy stays low: [email protected] on these new categories is 12.56–23.31. There is little sensitivity analysis for thresholds, the LLM size prior, or whether the bank is built with the validation categories known in advance. Runtime for the LLM, retrieval, and per-exemplar encoding is not reported.
Questions for authors
  • Are validation category names known when the bank is built? What happens if retrieval is allowed only for the current query?
  • How does mIoU change if the LLM size prior is removed, or if objects are placed on holes and unusual supports?
  • What confidence threshold is used when routing to the base model, and how sensitive are the results to it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.