- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents RA3D: an LLM generates candidate categories from implicit queries; web-retrieved images are reconstructed into 3D exemplars and embedded into the scene for contextual encoding. Target masks are obtained via background contrastive matching, distribution calibration and independent visual verification. The authors further propose NO-Reason3D for out-of-vocabulary targets and demonstrate RA3D’s gains on Reason3D, Instruct3D and a new benchmark.
Critical strengths
-
The technical design of introducing external visual knowledge at inference time constitutes a substantial contribution. Instead of task-specific training, RA3D reconstructs web exemplars into 3D and sequentially addresses the isolated-object/context gap, cross-exemplar score incomparability, and candidate self-verification. Ablations in Tables 3, 6, 8 and 9 collectively support that these performance gains are not incidental benefits from a single module. Prior works such as OpenScene and PLA have established open-vocabulary 3D perception, yet their core mechanisms differ from this inference-time visual exemplar retrieval, so the proposed direction is clearly distinct.
-
The merit of the method does not fully rely on the new benchmark. On the real-world Reason3D dataset, Reason3D+RA3D improves mIoU over the vanilla model from 31.20 to 34.12 on ScanNet and from 19.54 to 23.22 on Matterport3D. It also lifts performance from 11.70 to 13.14 on Instruct3D. These results demonstrate that even with a more conservative view on the generalization claims of NO-Reason3D, the retrieval branch still delivers standalone practical value.
Critical weaknesses
- NO-Reason3D experiments fail to separate gains from retrieval versus open-vocabulary 3D perception. Table 1 only compares conventional reasoning segmentation baselines (3D-STMN, Reason3D). A fairer control uses identical Qwen category predictions but replaces exemplar retrieval with off-the-shelf open-vocabulary models like OpenScene/PLA. Existing results only beat closed-vocabulary baselines and cannot confirm retrieval drives novel-object improvements. The authors should add these baselines with [email protected], [email protected] and mIoU, including oracle-category tests to decouple language and visual grounding errors under fixed scenes and candidate classes.
- NO-Reason3D’s full CAD mesh insertion may bias results toward RA3D’s complete exemplar matching. While insertion artifacts are mitigated, NO-Reason3D uses full meshes unlike partial surfaces from real scans. RA3D also reconstructs complete object geometry for verification, so the benchmark may favor shape matching. The authors should report geometric statistics (visible surface ratio, point density) between CAD inserts and real scans. Preferably, add scan-like occlusion to inserted objects and re-evaluate, or include a geometry-only baseline to rule out spurious gains from intact CAD models.
Questions for authors
1.On the NO-Reason3D validation set, what are Qwen’s top-1 and top-3 recall for target categories? Among RA3D’s failure cases, what fraction has the correct category missing from the Qwen top-3 predictions? This statistic directly tests the paper’s motivation that the main bottleneck lies in visual grounding rather than language reasoning.
2.Have the authors tested OpenScene, PLA or other open-vocabulary 3D models using exactly the same Qwen-predicted categories as RA3D? If yes, please provide NO-Reason3D mIoU and Acc results. If not, specify concrete barriers in input modality or evaluation protocol. Since OpenScene and PLA natively support language queries for unseen classes, this comparison may substantially change my assessment of the main empirical contributions.
3.Do the authors have completeness or visibility statistics comparing the full meshes inserted in NO-Reason3D against real objects from ScanNet/Matterport3D, or existing results on partially occluded CAD objects? Appendix G shows geometric completeness changes how verification depends on view count, so this information shapes my interpretation of benchmark realism and the large gains reported in Table 1.