acceptodds
Under review as a conference paper at ICLR 2027

RSI-LGS: Recursive Self-Improvement for Language Gaussian Splatting

Abstract

Open-vocabulary 3D querying aims to localize and identify objects in 3D scenes according to arbitrary natural language prompts. Existing methods typically adopt a one-stage pipeline that aligns 3D scene representations with vision-language semantics in a static manner. Consequently, these models lack effective mechanisms for diagnosing and rectifying erroneous predictions. To address this limitation, we propose RSI-LGS, an open-vocabulary 3D querying framework equipped with a pipeline-level self-improvement loop. RSI-LGS reviews intermediate query results to diagnose localization and semantic matching errors, and adaptively adjusts the querying pipeline for subsequent iterations. This enables recursive refinement of the inference strategy and improves 3D open-vocabulary query results.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-vocabulary 3D querying, 3D Gaussian Splatting, Multimodal large language models, Recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LXhxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on open-vocabulary 3D scene understanding based on 3D Gaussian Splatting. It points out that existing methods typically rely on one-shot static inference. As a result, they easily make irreversible errors under complex occlusion, viewpoint bias, or semantic ambiguity, while lacking a mechanism for self-correction. To address this issue, the paper proposes RSI-LGS, a framework that recasts 3D open-vocabulary querying as a recursive self-improvement loop at inference time. To reduce the cost of querying large models, the authors introduce a hierarchical asymmetric architecture called HALCO. In this design, the local tier handles high-frequency 2D bounding-box prediction, SAM segmentation, and 3D geometric voting. Meanwhile, the cloud tier inspects novel-view relevance rendering maps and uses a set of tools to adaptively adjust pipeline configurations. On both the LERF-Mask and 3D-OVS benchmarks, this method achieves performance superior to existing baselines.

Critical strengths
  1. The paper clearly discusses the practical limitations of one-shot static inference methods. It points out that in complex scenarios involving occlusions and viewpoint variations, localization and semantic errors inevitably persist without a self-correction mechanism, thereby providing a compelling motivation for introducing recursive self-improvement.
  2. Reducing complex 3D semantic understanding to processing with mature 2D foundation models, followed by back-projecting into 3D space for geometric consistency filtering, is an elegant and sound design.
  3. The HALCO architecture establishes a clear division of responsibilities between the local tier and the cloud. By keeping high-frequency, computation-intensive localization, segmentation, and voting on the local device, and sending only compact novel-view renderings to the cloud for low-frequency review, it efficiently allocates computational resources and prevents the exhaustion of the large model's context window.
  4. The paper fully discloses the state machine template, prompts, and the parameter and functional definitions of the six specialized tool operators in the appendix, demonstrating a high degree of academic transparency and reproducibility.
  5. The proposed method significantly outperforms existing SOTA baselines on both the LERF-Mask and 3D-OVS benchmarks, achieving substantial mIoU improvements of 18.6 and 11.6 mIoU points, respectively, which demonstrates encouraging experimental results.
Critical weaknesses
  1. The technical novelty appears to lie primarily in inference-time pipeline orchestration rather than in a new 3D representation or learning algorithm. While this system-level contribution can be practically valuable, the paper does not sufficiently isolate the novelty of its recursive review policy from existing tool-using or iterative-refinement strategies. In particular, it remains unclear whether the gains arise from the review policy itself or simply from additional views, parameter adjustments, and inference-time computation.
  2. The request_more_views tool is invoked most frequently, yet it merely executes additional random view sampling for the subsequent round without providing directional camera poses or targeted observations toward occluded areas. Consequently, the large multimodal model fails to demonstrate genuine 3D spatial causal reasoning, effectively degrading the tool into a blind trial-and-error operation. Furthermore, the number of initial views is strictly hard-coded to 10, resulting in request_more_views accounting for 42.2% of all tool calls. The authors need to justify why they did not simply increase the initial number of views to reduce redundant tool invocations.
  3. Although the paper reports the invocation frequency of each tool, tool call counts alone do not prove that the model's decisions are correct or effective. Reporting step-by-step performance changes before and after each tool invocation would much better substantiate the actual value and efficacy of the tool design.
  4. The evaluation is conducted on LERF-Mask and 3D-OVS, which are very small in scale (totaling only 53 queries across 8 scenes). Although the paper emphasizes that the method improves accuracy in complex environments through iterative refinement, the limited evaluation scenarios are insufficient to convincingly demonstrate the system's generalization capability across more diverse scenes.
  5. The proposed method takes up to 105.3 seconds for a single query, exhibiting a latency gap compared to the baseline methods. The paper does not verify whether existing baselines could achieve comparable accuracy under a similar computational budget (e.g., aggregating information across multiple views). Consequently, it remains unclear whether the reported accuracy gains justify the additional computation and latency.
Questions for authors
  1. The request_more_views tool accounts for 42.2% of all tool calls, yet it merely executes random supplementary sampling for the next round without specifying camera poses or targeted viewpoints aimed at occluded areas. Have the authors tested the effect of increasing the initial number of views? Could appropriately increasing the initial views significantly reduce the invocation frequency of this tool and shorten the overall runtime? If so, given that the model does not perform targeted spatial geometric planning, what is the necessity of incurring a lengthy cloud round-trip merely to trigger random view sampling?
  2. A high invocation frequency for a tool does not necessarily equate to correct decision-making. Could the authors report the performance changes before and after invoking each individual tool, or provide fine-grained tool ablation experiments?
  3. Currently, the evaluation is based on only 8 scenes with a total of 53 text queries, which represents a very small sample size. Given that the paper emphasizes the framework's strong error-correction capability in complex scenarios, the authors should evaluate the method on more complex or more open-ended 3D scenes to convincingly demonstrate its advantages over existing methods.
1 comment

Q1 — request_more_views (42.2%) and initial views. The high frequency follows our deliberate policy of actively requesting views to improve accuracy. Since real scenes vary, processing all views or uniformly enlarging the initial set wastes HALCO-Local computation and burdens HALCO-Cloud with redundant evidence; on-demand acquisition adds views only when review finds evidence insufficient.

Such exploration prevents premature acceptance of a plausible but suboptimal result merely because no better alternative has been observed. undo_and_finish allows failed adjustments to be rolled back, making exploration safer.

Thus, the tool balances accuracy and cost across diverse scenes.

Q2 — Tool frequency ≠ decision quality; per-tool ablation. We agree that call frequency does not establish decision quality. Using round-by-round traces, Table R3-B links each decisive intervention to its mechanism and before/after prediction, directly evaluating individual tools rather than aggregate counts.

Table R3-B. Per-tool attribution on the four decisive chains.

QueryDeciding toolMechanismBefore → after
rubber duck with red hatrequest_more_views (+6 views)voting cluster expanded 7→11 accepted frames, completing 3D coverage0.6443 → 0.9094 (+26.5)
porcelain handselect_clusterdefault voting cluster was the wrong object (blue rabbit); supervision rewritten to target0.6360 → 0.7586 (+12.3)
cookies on a plateset_eps + undo_and_finishtightened ε, detected degradation in next render, rolled back0.5405 → 0.6592 (+11.9)
chopsticksrequest_more_views (+6 views)scattered clusters merged into the target's main structure0.4607 → 0.5645 (+10.4)

The traces show coverage completion (request_more_views), wrong-cluster correction (select_cluster), and rollback of a failed adjustment (set_eps + undo_and_finish). Gains depend on corrected supervision, not call count: “porcelain hand” gains +12.3 with three calls versus +0.9 for “paper napkin” with seven.

Q3 — Complex referring descriptions on Ref-LERF. To test error correction with complex language, we re-evaluated RSI-LGS on Ref-LERF figurines using s0 descriptions. Unlike category words, these prompts specify attributes, shape, function, and spatial relations (Table R3-C).

Table R3-C. Ref-LERF s0 referring descriptions used on figurines.

Querys0 referring description
green appleA green object placed on the edge of a table, shaped like a fruit.
green toy chairA green object placed between a pumpkin and a red chair.
old cameraA black device with a lens and buttons for taking photos or videos.
porcelain handA small white ornament, shaped like a hand, made of smooth and delicate material, standing on a table.
red appleA round red fruit with a smooth surface sits quietly on the edge of the table.
red toy chairA chair toy next to the box in bright colors.
rubber duck with red hatA yellow duckling toy sits quietly on the surface of a colorful Rubik's Cube.

Using the same s0 prompts, RSI-LGS achieves the best mean 0.7426 IoU / 0.7197 BIoU (Table R3-D), exceeding Gaussian Grouping (0.4997 / 0.4875) by 24.3 / 23.2 points. It avoids complete failure on all seven queries (minimum IoU 0.5339), while baselines produce several near-zero results.

Table R3-D. Ref-LERF s0 results on figurines. Each cell is IoU / BIoU.

QueryRSI-LGS (ours)Gaussian GroupingLangSplatReferSplat
green apple0.5339 / 0.50210.0000 / 0.00000.1528 / 0.13400.7975 / 0.7566
green toy chair0.9362 / 0.92040.8761 / 0.85770.2595 / 0.24560.8820 / 0.8636
old camera0.5790 / 0.54260.8822 / 0.83540.3870 / 0.33150.4798 / 0.4436
porcelain hand0.7586 / 0.75770.8478 / 0.84650.0029 / 0.00310.1023 / 0.1019
red apple0.8731 / 0.81230.0000 / 0.00000.1116 / 0.07330.5910 / 0.5305
red toy chair0.9323 / 0.91920.8916 / 0.87260.2245 / 0.22000.8339 / 0.8238
rubber duck with red hat0.5849 / 0.58380.0001 / 0.00010.0127 / 0.01270.0000 / 0.0000
Mean0.7426 / 0.71970.4997 / 0.48750.1644 / 0.14570.5266 / 0.5028

Thus, RSI-LGS remains robust to complex attribute- and relation-based descriptions, supporting the value of reviewing and correcting intermediate predictions over one-shot text-to-mask matching.

Reviewer_G6w9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RSI-LGS for text-conditioned object segmentation in reconstructed 3D Gaussian scenes. A local vision-language model predicts bounding boxes in sampled camera views, SAM produces segmentation masks, and a geometric filtering step rejects unreliable views. Per-Gaussian relevance scores are then fitted to the accepted masks. A cloud-hosted model reviews renderings of the result and can request additional views, adjust clustering, select a cluster, or rewrite the query.

The method is evaluated against LangSplat, Gaussian Grouping, OpenGaussian, and ReferSplat on LERF-Mask and 3D-OVS, with higher overall segmentation scores reported on both benchmarks.

Critical strengths

The paper includes comparisons against relevant Gaussian-based segmentation and grounding methods, and reports substantial improvements on both evaluated benchmarks. These results deserve consideration even though the source of the gains needs further investigation. The loop-removal ablation also provides some evidence that review is useful

Critical weaknesses
  1. The central claim overstates the methodological contribution. The reviewer selects from a fixed set of predefined operations and reruns the pipeline. Neither the reviewer nor its improvement procedure is modified. Although the paper specifies that its recursion operates at the pipeline level, the demonstrated mechanism is adaptive iterative refinement for an individual query. I do not find it convincing to call this “recursive self-improvement.”

The underlying task is established open-vocabulary 3D segmentation. Combining 2D localization and segmentation with geometric filtering and a fitted 3D relevance field is a reasonable approach, but the paper needs to establish what the review controller contributes beyond this combination.

  1. computational cost is too high: The reported runtime is 105.3 seconds for a single query on the figurines scene, including 58.4 seconds for cloud review and the pipeline work it triggers. This is substantial latency for selecting an object in an already reconstructed scene.
Questions for authors
  1. since the title includes RSI: What specifically distinguishes the proposed loop from iterative refinement with a fixed tool-using controller? Does any part of the improvement procedure itself change?
  2. see the weakness above
1 comment
AuthorsOpenReview ↗

RSI vs. fixed controller; review contribution

Q1 — Loop vs. iterative refinement with a fixed controller. Our loop is not iterative refinement under a fixed tool-using controller: the improvement procedure itself changes, because it is history-conditioned and adapts to evidence rather than executing a fixed schedule, so it can correct errors a schedule cannot even observe. A fixed controller must be triggered by a scalar criterion, but the causes of error here require perceptual judgement of rendered 3D semantics, which no scalar expresses; accordingly the intervention space is mostly non-schedulable. Moreover, we pair every HALCO-Cloud tool with a reversible undo (undo_and_finish), which lets the loop aggressively pursue the best performance while always being able to roll back to a previous state — a behavior a fixed controller flow cannot offer.

W1 — Whether query-scoped adaptive refinement constitutes “recursive self-improvement.” We agree that our loop operates through adaptive iterative refinement within a single query; our claim is not model-level self-improvement or cross-task learning, but history-conditioned improvement of the querying process. Unlike a fixed controller that follows a predetermined intervention schedule, our reviewer retains its context and the history of previous tool calls throughout the loop. Each decision is therefore conditioned not only on the current rendered prediction and pipeline state, but also on earlier interventions and their observed consequences. This allows the reviewer to update its strategy from round to round—for example, by acquiring additional views, adjusting clustering, selecting another cluster, or terminating when the evidence is sufficient—rather than repeatedly executing a fixed procedure.

We further provide the reversible undo_and_finish operator, which reverts the most recent adjustment and restores the previous state. If an intervention degrades the result, the reviewer can withdraw it instead of committing to the erroneous operation. Together, the preserved history and rollback mechanism give the reviewer room to explore different interventions, learn from their outcomes within the current query, and pursue a better final prediction without being locked into a failed trajectory.

W2 — Contribution beyond localization + geometric filtering + relevance fitting. Our contribution is an effective self-correction framework. RSI-LGS uses rendered predictions as visual feedback: HALCO-Cloud reviews the intermediate result, diagnoses remaining errors, and modifies the configuration of the entire inference pipeline. This closed-loop review allows the system to correct its own prediction instead of accepting the first output as final.

Our ablations demonstrate that this review strategy is necessary. On figurines, the full model achieves 0.8690 mIoU / 0.8334 mBIoU. Removing the review loop while retaining geometric voting reduces performance to 0.8210 / 0.7987, drops of 4.80 mIoU and 3.47 mBIoU points under otherwise identical settings. Thus, even with localization, voting, and relevance fitting intact, the pipeline loses a substantial part of its accuracy without review.

Removing both geometric voting and review further reduces performance to 0.7706 mIoU / 0.7462 mBIoU, corresponding to substantial drops of 9.84 and 8.72 points. This combined loss is nearly additive with the individual losses from removing voting (4.75 / 3.24 points) and review (4.80 / 3.47 points), with an mIoU interaction term of only −0.0029. The two modules therefore address complementary failure modes: voting rejects geometrically inconsistent 2D evidence, whereas review corrects predictions that are geometrically coherent but semantically wrong.

In summary, the ablations show that HALCO-Cloud review is not redundant with the underlying segmentation pipeline: visual-feedback-driven self-correction and pipeline-level revision are essential to the final performance.

Reviewer_3aY2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents RSI-LGS, a framework for open-vocabulary 3D segmentation in 3D Gaussian Splatting. Prior methods use one-shot predictions as the final output. RSI-LGS instead builds the 3D segmentation for a text query through an iterative self-refinement loop with two stages. In HALCO-Local, a local MLLM predicts per-view bounding boxes, and SAM converts them into masks. The masked pixels are back-projected and clustered with DBSCAN. Only the 2D masks with high IoU against the dominant cluster are used to fit the 3D relevance field. In HALCO-Cloud, a cloud-hosted MLLM reviews novel-view renderings of the prediction and refines the 3D segmentation by invoking pipeline-level tools, such as adjusting the DBSCAN radius, requesting more views, selecting a cluster, or rewriting the query.

Critical strengths
  • The motivation is clear. Existing one-shot methods have no mechanism to correct an erroneous 3D segmentation once it is produced.
  • The method outperforms existing one-shot approaches on both LERF-Mask and 3D-OVS. This makes it useful when very high-quality 3D segmentation is required.
  • The two proposed modules (HALCO-Local and HALCO-Cloud) refine the 3D segmentation in complementary ways.
  • The paper introduces a set of tools that an MLLM can call to refine the 3D segmentation at the pipeline level.
Critical weaknesses
  • Slow inference speed Prior one-shot methods complete SAM-based preprocessing offline, which allows interactive querying in milliseconds at inference time. In contrast, RSI-LGS must call the models for every query and takes 105.3 s per query (Table 5). This substantially limits its applicability to latency-sensitive domains such as robotics and AR/VR.
  • Limited technical novelty Each component is an existing technique: MLLM grounding, SAM, DBSCAN, and mask lifting. The contribution therefore remains largely at the level of system integration.
  • Reliance on MLLMs Beyond inference speed, the cloud MLLM introduces dependencies on internet connectivity and API cost. Comparing an MLLM-based pipeline against CLIP-based baselines is also not a fair comparison.
  • Unfair Model size The MLLMs used, LLaVA-OneVision-2-8B for localization and DeepSeek V4.1 Flash for review, are much larger than the models used in prior work. This further weakens the fairness of the comparison.
Questions for authors
  • Is there an ablation on the size of the MLLM? Prior work relies on CLIP and SAM (roughly 1B parameters), so I would like to see the performance with a ~1B-scale model.
  • Could the authors provide comparisons with recent methods such as THGS [1], Dr. Splat [2], and LUDVIG [3]?

[1] Training-Free Hierarchical Scene Understanding for Gaussian Splatting with Superpoint Graphs

[2] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration

[3] LUDVIG: Learning-free Uplifting of 2D Visual features to Gaussian Splatting scenes

3 comments

Q1 — Ablation on MLLM size. We ablate the size of the local MLLM and show that our self-improvement mechanism is not merely a benefit of large models — it is an effective, general framework for boosting performance that holds even at the ~1B scale used by prior work. To separate the backbone scale from the architecture, we replace HALCO-Local with Grounded SAM and keep all other settings unchanged. As shown in Table R1-A, RSI-LGS-light still attains a macro 0.7743 mIoU / 0.7361 mBIoU; by contrast, Gaussian Grouping — which uses the same Grounded-SAM detector — scores only 0.6808 / 0.6349, so adopting our framework alone yields +9.4 / +10.1, directly evidencing the effectiveness of the self-improvement loop rather than the local model.

Table R1-A. Combined LERF-Mask results, including RSI-LGS-light with the localizer replaced by Grounded SAM (≈0.4B). Each cell is mIoU / mBIoU; the last row is the equally weighted average over the three scenes.

SceneRSI-LGSRSI-LGS-lightGaussian GroupingReferSplatDr. SplatLangSplatOpenGaussian
figurines0.8690/0.83340.8860/0.86770.7311/0.71420.5445/0.52110.6580/0.63750.2167/0.19350.1739/0.1422
ramen0.8660/0.75950.6884/0.63920.6106/0.53500.7749/0.72670.6676/0.55140.6049/0.53640.4143/0.3271
teatime0.8653/0.79910.7477/0.70230.7006/0.65560.6149/0.56410.6467/0.59850.2822/0.24250.2757/0.1968
Overall0.8668/0.79730.7743/0.73610.6808/0.63490.6448/0.60400.6574/0.59580.3680/0.32410.2879/0.2220

Q2 — Comparison with Dr. Splat. We have supplemented our evaluation with Dr. Splat under the identical protocol used for our method and the other baselines (same queries, test views, ground truth, and metric implementation). The results are reported in Tables R1-A and R1-B. RSI-LGS continues to achieve the best macro-average performance, outperforming all compared methods in both mIoU and mBIoU on the two benchmarks. Specifically, relative to Dr. Splat, RSI-LGS improves macro mIoU by +20.9 points on LERF-Mask (0.8668 vs. 0.6574) and +16.4 points on 3D-OVS (0.9108 vs. 0.7469), with corresponding mBIoU gains of +20.2 and +18.6 points.

Q1 — Ablation on MLLM size. We ablate the size of the local MLLM and show that our self-improvement mechanism is not merely a benefit of large models — it is an effective, general framework for boosting performance that holds even at the ~1B scale used by prior work. To separate the backbone scale from the architecture, we replace HALCO-Local with Grounded SAM and keep all other settings unchanged. As shown in Table R1-A, RSI-LGS-light still attains a macro 0.7743 mIoU / 0.7361 mBIoU; by contrast, Gaussian Grouping — which uses the same Grounded-SAM detector — scores only 0.6808 / 0.6349, so adopting our framework alone yields +9.4 / +10.1, directly evidencing the effectiveness of the self-improvement loop rather than the local model.

Table R1-A. Combined LERF-Mask results, including RSI-LGS-light with the localizer replaced by Grounded SAM (≈0.4B). Each cell is mIoU / mBIoU; the last row is the equally weighted average over the three scenes.

SceneRSI-LGSRSI-LGS-lightGaussian GroupingReferSplatDr. SplatLangSplatOpenGaussian
figurines0.8690/0.83340.8860/0.86770.7311/0.71420.5445/0.52110.6580/0.63750.2167/0.19350.1739/0.1422
ramen0.8660/0.75950.6884/0.63920.6106/0.53500.7749/0.72670.6676/0.55140.6049/0.53640.4143/0.3271
teatime0.8653/0.79910.7477/0.70230.7006/0.65560.6149/0.56410.6467/0.59850.2822/0.24250.2757/0.1968
Overall0.8668/0.79730.7743/0.73610.6808/0.63490.6448/0.60400.6574/0.59580.3680/0.32410.2879/0.2220

Q2 — Comparison with Dr. Splat. We have supplemented our evaluation with Dr. Splat under the identical protocol used for our method and the other baselines (same queries, test views, ground truth, and metric implementation). The results are reported in Tables R1-A and R1-B. RSI-LGS continues to achieve the best macro-average performance, outperforming all compared methods in both mIoU and mBIoU on the two benchmarks. Specifically, relative to Dr. Splat, RSI-LGS improves macro mIoU by +20.9 points on LERF-Mask (0.8668 vs. 0.6574) and +16.4 points on 3D-OVS (0.9108 vs. 0.7469), with corresponding mBIoU gains of +20.2 and +18.6 points.

Table R1-B. 3D-OVS (5 scenes). Each cell is mIoU / mBIoU; the last row is the equally weighted average over the five scenes.

SceneRSI-LGSGaussian GroupingLangSplatOpenGaussianReferSplatDr. Splat
bed0.9392/0.83030.9424/0.84160.7365/0.55980.2822/0.25530.9379/0.78760.4086/0.3066
covered_desk0.8581/0.79460.4337/0.39130.7937/0.67180.1850/0.16930.8660/0.66840.8607/0.7491
office_desk0.9217/0.75310.7322/0.63770.8091/0.67890.1320/0.12130.4875/0.36680.7336/0.5592
snacks0.8969/0.86500.6588/0.60840.8468/0.78220.1666/0.15030.8743/0.78760.8560/0.7803
sofa0.9379/0.86240.7174/0.65260.7898/0.69690.2358/0.21440.7644/0.64150.8758/0.7800
Overall0.9108/0.82110.6969/0.62630.7952/0.67790.2003/0.18210.7860/0.65040.7469/0.6350

Thank for author's response.

The response clarified question 1 and 2.

However, following weakness are not tackled in the response.

  • slow speed
  • limited technical novelty
  • reliance on MLLMs

So, I keep my score and recommend weak reject on this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.