- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on open-vocabulary 3D scene understanding based on 3D Gaussian Splatting. It points out that existing methods typically rely on one-shot static inference. As a result, they easily make irreversible errors under complex occlusion, viewpoint bias, or semantic ambiguity, while lacking a mechanism for self-correction. To address this issue, the paper proposes RSI-LGS, a framework that recasts 3D open-vocabulary querying as a recursive self-improvement loop at inference time. To reduce the cost of querying large models, the authors introduce a hierarchical asymmetric architecture called HALCO. In this design, the local tier handles high-frequency 2D bounding-box prediction, SAM segmentation, and 3D geometric voting. Meanwhile, the cloud tier inspects novel-view relevance rendering maps and uses a set of tools to adaptively adjust pipeline configurations. On both the LERF-Mask and 3D-OVS benchmarks, this method achieves performance superior to existing baselines.
Critical strengths
- The paper clearly discusses the practical limitations of one-shot static inference methods. It points out that in complex scenarios involving occlusions and viewpoint variations, localization and semantic errors inevitably persist without a self-correction mechanism, thereby providing a compelling motivation for introducing recursive self-improvement.
- Reducing complex 3D semantic understanding to processing with mature 2D foundation models, followed by back-projecting into 3D space for geometric consistency filtering, is an elegant and sound design.
- The HALCO architecture establishes a clear division of responsibilities between the local tier and the cloud. By keeping high-frequency, computation-intensive localization, segmentation, and voting on the local device, and sending only compact novel-view renderings to the cloud for low-frequency review, it efficiently allocates computational resources and prevents the exhaustion of the large model's context window.
- The paper fully discloses the state machine template, prompts, and the parameter and functional definitions of the six specialized tool operators in the appendix, demonstrating a high degree of academic transparency and reproducibility.
- The proposed method significantly outperforms existing SOTA baselines on both the LERF-Mask and 3D-OVS benchmarks, achieving substantial mIoU improvements of 18.6 and 11.6 mIoU points, respectively, which demonstrates encouraging experimental results.
Critical weaknesses
- The technical novelty appears to lie primarily in inference-time pipeline orchestration rather than in a new 3D representation or learning algorithm. While this system-level contribution can be practically valuable, the paper does not sufficiently isolate the novelty of its recursive review policy from existing tool-using or iterative-refinement strategies. In particular, it remains unclear whether the gains arise from the review policy itself or simply from additional views, parameter adjustments, and inference-time computation.
- The request_more_views tool is invoked most frequently, yet it merely executes additional random view sampling for the subsequent round without providing directional camera poses or targeted observations toward occluded areas. Consequently, the large multimodal model fails to demonstrate genuine 3D spatial causal reasoning, effectively degrading the tool into a blind trial-and-error operation. Furthermore, the number of initial views is strictly hard-coded to 10, resulting in request_more_views accounting for 42.2% of all tool calls. The authors need to justify why they did not simply increase the initial number of views to reduce redundant tool invocations.
- Although the paper reports the invocation frequency of each tool, tool call counts alone do not prove that the model's decisions are correct or effective. Reporting step-by-step performance changes before and after each tool invocation would much better substantiate the actual value and efficacy of the tool design.
- The evaluation is conducted on LERF-Mask and 3D-OVS, which are very small in scale (totaling only 53 queries across 8 scenes). Although the paper emphasizes that the method improves accuracy in complex environments through iterative refinement, the limited evaluation scenarios are insufficient to convincingly demonstrate the system's generalization capability across more diverse scenes.
- The proposed method takes up to 105.3 seconds for a single query, exhibiting a latency gap compared to the baseline methods. The paper does not verify whether existing baselines could achieve comparable accuracy under a similar computational budget (e.g., aggregating information across multiple views). Consequently, it remains unclear whether the reported accuracy gains justify the additional computation and latency.
Questions for authors
- The request_more_views tool accounts for 42.2% of all tool calls, yet it merely executes random supplementary sampling for the next round without specifying camera poses or targeted viewpoints aimed at occluded areas. Have the authors tested the effect of increasing the initial number of views? Could appropriately increasing the initial views significantly reduce the invocation frequency of this tool and shorten the overall runtime? If so, given that the model does not perform targeted spatial geometric planning, what is the necessity of incurring a lengthy cloud round-trip merely to trigger random view sampling?
- A high invocation frequency for a tool does not necessarily equate to correct decision-making. Could the authors report the performance changes before and after invoking each individual tool, or provide fine-grained tool ablation experiments?
- Currently, the evaluation is based on only 8 scenes with a total of 53 text queries, which represents a very small sample size. Given that the paper emphasizes the framework's strong error-correction capability in complex scenarios, the authors should evaluate the method on more complex or more open-ended 3D scenes to convincingly demonstrate its advantages over existing methods.
Q1 —
request_more_views(42.2%) and initial views. The high frequency follows our deliberate policy of actively requesting views to improve accuracy. Since real scenes vary, processing all views or uniformly enlarging the initial set wastes HALCO-Local computation and burdens HALCO-Cloud with redundant evidence; on-demand acquisition adds views only when review finds evidence insufficient.Such exploration prevents premature acceptance of a plausible but suboptimal result merely because no better alternative has been observed.
undo_and_finishallows failed adjustments to be rolled back, making exploration safer.Thus, the tool balances accuracy and cost across diverse scenes.
Q2 — Tool frequency ≠ decision quality; per-tool ablation. We agree that call frequency does not establish decision quality. Using round-by-round traces, Table R3-B links each decisive intervention to its mechanism and before/after prediction, directly evaluating individual tools rather than aggregate counts.
Table R3-B. Per-tool attribution on the four decisive chains.
The traces show coverage completion (
request_more_views), wrong-cluster correction (select_cluster), and rollback of a failed adjustment (set_eps+undo_and_finish). Gains depend on corrected supervision, not call count: “porcelain hand” gains +12.3 with three calls versus +0.9 for “paper napkin” with seven.Q3 — Complex referring descriptions on Ref-LERF. To test error correction with complex language, we re-evaluated RSI-LGS on Ref-LERF
figurinesusings0descriptions. Unlike category words, these prompts specify attributes, shape, function, and spatial relations (Table R3-C).Table R3-C. Ref-LERF
s0referring descriptions used onfigurines.s0referring descriptionUsing the same
s0prompts, RSI-LGS achieves the best mean 0.7426 IoU / 0.7197 BIoU (Table R3-D), exceeding Gaussian Grouping (0.4997 / 0.4875) by 24.3 / 23.2 points. It avoids complete failure on all seven queries (minimum IoU 0.5339), while baselines produce several near-zero results.Table R3-D. Ref-LERF
s0results onfigurines. Each cell is IoU / BIoU.Thus, RSI-LGS remains robust to complex attribute- and relation-based descriptions, supporting the value of reviewing and correcting intermediate predictions over one-shot text-to-mask matching.