acceptodds
Under review as a conference paper at ICLR 2027

Ground3R: Streaming 3D Referring Expression Segmentation with Recurrent Multimodal Memory

Abstract

3D referring expression segmentation (3D-RES) aims to segment objects described by natural language in 3D scenes. Existing methods typically rely on preconstructed point clouds or offline collections of RGB images, limiting their applicability to embodied agents that explore unfamiliar scenes through streaming observations. In such scenarios, agents must reconstruct the scene and identify the referred object incrementally as new images arrive, without access to future observations. To address this setting, we introduce streaming 3D referring expression segmentation (Stream3D-RES), which jointly reconstructs the scene and segments the referred object from incoming RGB images using only current and past observations. This task presents two key challenges: (1) identifying the target may require contextual information observed in earlier views, and (2) distinguishing the referent from similar objects becomes difficult as the target leaves and re-enters the field of view. To tackle these challenges, we propose Ground3R, an online framework that combines recurrent multimodal memory with spatially localized object queries. Its Multimodal Memory Branch maintains a language-conditioned scene state across frames, allowing current predictions to leverage historical context. The Spatial Query Localization and Refinement module generates language-independent object proposals and uses the expression and accumulated scene context to identify the referent and refine its mask. Query features are transferred across views through 3D centroid matching, while explicit target-visibility prediction enables empty masks when the target is absent. We further introduce StreamRefer, a benchmark covering four target-visibility patterns and three observation lengths. Extensive experiments demonstrate that Ground3R achieves state-of-the-art performance among online models, substantially outperforming causal 3D baselines and 2D-lifting alternatives across all visibility patterns.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D visual grounding, referring expression segmentation, streaming 3D perception, online multi-view reconstruction, embodied AI, multimodal reasoning, stateful perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iRqeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new task: Streaming 3D Referring Expression Segmentation (Stream3D-RES). The task aims to perform simultaneous 3D scene reconstruction and target object segmentation based on a continuous stream of RGB images and a natural language description, operating in a causal manner—relying solely on current and historical information. Unlike traditional 3D referring expression segmentation methods that depend on complete point clouds or offline multi-view imagery, this task requires the model to perform inference using only current and past observations, while also accounting for dynamic visibility changes.

To address these challenges, the authors propose the "Ground3R" framework. The authors also have developed the "StreamRefer" benchmark, which encompasses four types of target visibility patterns and three sequence lengths, designed to evaluate capabilities in online, language-guided 3D localization.

Critical strengths
  1. The research problem holds clear practical value, and the task formulation is ingeniously conceived.

  2. The designs of MMB and SQLR complement each other.

  3. Experiments demonstrate that the proposed method offers significant performance advantages in online settings.

Critical weaknesses
  1. Lack of direct validation for the language-conditioned memory mechanism.

A central claim of this paper is that the MMB (Multi-Modal Memory Bank) can retain language-relevant scene information from historical observations, specifically spatial references that are no longer visible in the current frame. However, the current experiments (Table 4) assess performance changes primarily by removing either the language conditioning or the recurrent state.

  1. Lack of analysis regarding the quality of 3D geometric reconstruction and sensitivity to geometric errors.

The paper lacks a quantitative analysis of reconstruction quality, geometric uncertainty, and centroid matching errors, leaving the method's robustness against 3D geometric inaccuracies unclear.

  1. Evidence for long-sequence semantic memory and cross-scene generalization remains insufficient.
Questions for authors

See the weaknesses

3 comments

We sincerely thank you for recognizing the practical value of the task, the complementary designs of MMB and SQLR, and Ground3R’s performance in online settings. Your constructive suggestions have helped us strengthen the experimental evidence. We address your three concerns below.

W1: Direct Validation of Language-Conditioned Historical Memory

Comment: Lack of direct validation for the language-conditioned memory mechanism, particularly its ability to retain spatial references that are no longer visible.

A1: Thank you for this valuable comment. To directly test this claim, we added a diagnostic evaluation on out-of-view spatial references. Ground3R relies on historical observations when spatial references leave the view, and its accuracy drops substantially when those earlier reference frames are removed or replaced.

We evaluate Ground3R on 234 16-frame validation episodes from 73 scenes containing same-class distractors, where the spatially referenced object (e.g., "the chair beside the desk") appears only in the first half of the stream and is completely out of view in the second half.

SettingGlobal mIoU (%)
Intact stream39.00
Remove historical reference frames29.04
Replace historical reference frames with irrelevant frames17.86

Removing or replacing the earlier reference frames reduces Global mIoU by 9.96 and 21.14 percentage points, respectively. Together with the MMB ablations in Table 4, these results confirm that Ground3R retains language-relevant spatial references from past observations to disambiguate targets in subsequent views.

W2: Sensitivity to Geometric Reconstruction Errors and Confidence Modulation

Comment: The paper lacks quantitative analysis of reconstruction quality, geometric uncertainty, and centroid matching errors, leaving its robustness against geometric inaccuracies unclear.

A2: Thank you for this valuable suggestion. Following your advice, we added quantitative evaluations of online reconstruction quality, 3D centroid matching, and confidence-modulated inheritance on 120 stratified validation episodes (40 each at 8, 16, and 32 frames). Online CUT3R maintains centimeter-level accuracy on 8–32-frame streams, and replacing predicted geometry with ground-truth geometry for query matching improves Global mIoU by only 0.12 points.

1. Role and quality of online reconstruction. CUT3R is frozen during training: Ground3R performs joint reconstruction and grounding at online inference, without optimizing the geometry backbone; we will clarify this in the manuscript. For evaluation, we align the entire predicted sequence to the ground truth once, correcting its overall scale, rotation, and translation without adjusting individual frames separately [A]. Under this protocol, online CUT3R maintains low scene reconstruction error across 8–32 frames, while exhibiting gradual drift as sequences lengthen.

Sequence length8 frames16 frames32 framesOverall
Median 3D scene error (cm)5.036.289.947.08
Surface coverage @ 10 cm (%)93.7893.2190.2992.42

2. How much do geometric errors affect query matching? Using the same checkpoint on 120 stratified validation episodes, we compared three variants, keeping visual and memory features unchanged:

  • Default: Query matching uses centroids computed from predicted CUT3R pointmaps.
  • Confidence-modulated: CUT3R’s per-pixel confidence weights centroid estimation and modulates historical feature inheritance.
  • GT-geometry routing: Centroids are computed using ground-truth depth and camera poses.
Geometry used for inheritanceInstance matching precision (%)Match coverage (%)Global mIoU (%)
Predicted CUT3R geometry88.2187.7038.09
Confidence-modulated inheritance88.2487.6438.15
GT-geometry routing89.2587.0038.21

Instance matching precision measures the proportion of evaluable matched query pairs corresponding to the same ground-truth object.

Default matching achieves 88.21% precision, only 1.04 points below GT-geometry routing, with a 0.12-point difference in Global mIoU. These results suggest that predicted geometry is sufficiently reliable for query inheritance on this subset. Incorrect matches remain possible; even GT geometry does not eliminate them, since geometric proximity alone does not guarantee object identity.

Our default centroid estimation uses coordinate-wise median pooling over a local (5×55 \times 5) neighborhood, reducing sensitivity to isolated coordinate outliers. These experiments assess naturally occurring reconstruction errors, rather than robustness to arbitrarily large coordinate deviations.

3. Does geometric confidence help? Confidence modulation increases Global mIoU from 38.09% to 38.15% without retraining. The improvement is modest, but suggests that geometric confidence can help control feature inheritance. We appreciate your emphasis on geometric uncertainty and will further explore confidence-aware matching, particularly under larger reconstruction errors.

[A] Unlocking the Power of Critical Factors for 3D Visual Geometry Estimation. CVPR, 2026.

W3: Long-Sequence Semantic Memory and Cross-Scene Generalization

Comment: Evidence for long-sequence semantic memory and cross-scene generalization remains insufficient.

A3: Thank you for encouraging this broader evaluation. We extended accuracy testing to 512 frames and added zero-shot evaluation on RIORefer, complementing the Nr3D and Sr3D results already included in our submission.

1. Long-sequence grounding and target reappearance.

Experimental setup. We selected eight sequences in which Ground3R correctly recognized the target during the initial observations and the target subsequently disappeared and reappeared. This evaluates whether an initially recognized target can be recovered after absence.

We sample one frame every five frames from each original video and evaluate progressively longer prefixes of the same sampled sequence, from 1 to 512 frames. Thus, the 32-frame evaluation uses the first 32 sampled frames, while the 512-frame evaluation extends that same stream to 512 sampled frames.

All runs use the same checkpoint without additional fine-tuning. Each sequence–prefix run starts with independently initialized recurrent states. Frames are then processed chronologically, carrying the states forward without resets within the run and without access to future frames.

Evaluated prefix length (frames)1832128256512
Global mIoU (%)54.7875.5581.0766.3960.2051.00
Target recovery (successful/eligible reappearances)—1/11/12/34/611/20

Global mIoU evaluates the entire prefix, including target-visible and target-absent frames. A recovery succeeds if the model reaches IoU ≥ 0.5 in at least one of the first three consecutive visible frames after the target reappears. Events are identified from ground-truth visibility annotations, and counts are pooled across the eight sequences. Longer prefixes include additional events; “—” indicates no eligible event.

Results and interpretation. Global mIoU first increases and then decreases, rising from 54.78% at 1 frame to 81.07% at 32 frames, before declining to 51.00% at 512 frames. At 512 frames, Ground3R successfully recovers the target in 11 of 20 events.

Additional observations initially provide useful scene context. Over extended sequences, accumulated geometric errors can make cross-view association less reliable. Such drift is a longstanding challenge in online reconstruction [B, C]. CUT3R also acknowledges potential long-sequence drift without global alignment [A]. Ground3R inherits this limitation, which likely contributes to the observed decline. Reducing drift and strengthening long-term memory are important directions for future work.

2. Generalization beyond StreamRefer. Following your suggestion, we additionally evaluated RIORefer, using 88 expressions covering 44 targets in 8 scenes, with chronological 8-frame episodes and no fine-tuning on the new dataset.

MethodGlobal mIoU (%)
OnlineRefer22.94
StreamingRVOS-1B27.35
MVGGT-causal27.20
Ground3R31.33

All results are Global mIoU (%).

These results provide additional evidence of generalization beyond StreamRefer. We sincerely appreciate your constructive feedback, which has helped us strengthen the evaluation and clarify the scope of our conclusions.

[A] Continuous 3D Perception Model with Persistent State. CVPR, 2025.

[B] ElasticFusion: Dense SLAM Without A Pose Graph. Robotics: Science and Systems (RSS), 2015.

[C] BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration. ACM Transactions on Graphics (TOG), 2017.

Reviewer_VaG8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a referring 3D segmentation task using a streaming video. To do this, the authors newly curate the benchmark and propose new evaluation metrics. While the authors provided detailed descriptions about the model / module designs, but it is not that interesting to me, honestly. The paper demonstrates good performance "only" at the proposed benchmark. The authors provided the results in the single benchmark which is even curated by the authors.

Critical strengths

S-1. Task definition.

It is good to design a feed-forward style of 3D referring segmentation from a streaming video. In this perspective, the authors focus on "recurrent" module designs to cope with this in-coming observations. Also, the authors proposed Multimodal Memory Branch (MMB). Those designs are highly similar to SAM-2 which also has the memory attention framework to propagate mask signals across in-coming image frames.

Except this one. I don't know.

Critical weaknesses

W-1. Limited evaluation

The authors only evaluated the proposed method in one benchmark and the benchmark is curated by the authors. Commonly, the authors should provide results at least more 2 benchmarks or 2 datasets. From the given results, I cannot judge whether the proposed method is overfitted to this standalone benchmark or not. Showing results in diverse scenes, benchmarks, and datasets is highly important.

W-2. Limited data curation

There are many recent benchmarks that provide pre-captured videos for 3D reconstruction task or Neural rendering task. I wonder why the authors only used ScanRefer which is also originate from ScanNet dataset. Even ScanNetpp dataset have some data configuration with higher quality images and pointclouds.

My personal opinion is that since the authors address the "referring" 3D segmentation task, the authors need to design how to extract high-quality instructions as well. But here, it seems that authors just naively use the given instructions from the ScanRefer benchmark. If so, I am not sure whether I can determine the dataset as "newly" curated data.

W-3 Revision is needed in Figure 2.

What is input and output of each module? Each module is disconnected without having any connected lines. I am totally lost at this moment.

In Line 212-213, the input is a streaming video and an instruction, the outputs are pointmaps and binary mask maps. It makes sense to me. The authors use the pre-trained CUT3R architecture. Naturally, I wonder how the information has been mixed across multi-view images having different viewpoints. That is somewhat ambiguous to me. Moreover, since the proposed method targets at "referring" 3D segmentation, it is quite important how the given instruction are mixed through the visual tokens. The Figure 2 itself looks similar to MaskFormer architecture that also use ROI queries, but I have no idea. I cannot understand the intention of Figure 2. It is highly confusing. Please refer to the PanSt3r pipeline figure as an example.

Questions for authors

I hope that the authors resolve my concerns that were addressed in the weakness section.

On the other hand, the writing quality is also quite bad. In abstract, the authors wrote as

  • "agents must reconstruct the scene and identify the referred object incrementally as new images arrive, without access to future observations."
  • "We introduce .... incoming RGB images using only current and past observations".

I have no doubt that people in this field know the meaning of "streaming" video. I do understand why the authors intended to say so in the abstract, but paraphrasing the intention with above sentences are highly confusing. This is a personal writing style, but I would prefer "we are targeting at ... using a streaming video. Given an observation at time t, the model recurrently update the predictions with the in-coming observation ... " For myself, the two sentences that I sampled from the abstract is quite distractive.

4 comments

Thank you for your time and feedback on our submission. We appreciate your recognition of our streaming 3D referring segmentation task definition and our recurrent Multimodal Memory Branch (MMB) design for handling incoming observations. Below, we address each of your concerns regarding multi-benchmark evaluation, benchmark curation scope, Figure 2 clarity, and abstract phrasing.


W-1: Evaluation Across Multiple Benchmarks and Unseen Datasets

Comment: The authors only evaluated the proposed method in one benchmark and the benchmark is curated by the authors. Commonly, the authors should provide results at least more 2 benchmarks or 2 datasets. From the given results, I cannot judge whether the proposed method is overfitted to this standalone benchmark or not. Showing results in diverse scenes, benchmarks, and datasets is highly important.

A1: Thank you for this valuable suggestion. Indeed, our submission already includes evaluations on two additional benchmarks, Nr3D and Sr3D, in Appendix A.5 (Table 10), beyond StreamRefer. Following your advice, we have further added a zero-shot evaluation on RIORefer. These experiments extend our evaluation to four benchmarks and strengthen the evidence of generalization.

1. Nr3D and Sr3D: results included in the submission. Ground3R outperforms MVGGT-causal on both benchmarks without benchmark-specific fine-tuning:

MethodNr3D Global mIoU (%)Sr3D Global mIoU (%)
MVGGT-causal21.8025.01
Ground3R35.5840.06
Gain (percentage points)+13.78+15.05

2. RIORefer: new evaluation following your suggestion. To test transfer beyond ScanNet-based benchmarks, we evaluated Ground3R on 88 expressions covering 44 targets in 8 RIORefer scenes, using chronological 8-frame episodes and no fine-tuning on the new dataset.

MethodGlobal mIoU (%)
OnlineRefer22.94
StreamingRVOS-1B27.35
MVGGT-causal27.20
Ground3R31.33

W-2: Benchmark Curation and Contribution

Comment: There are many recent benchmarks that provide pre-captured videos for 3D reconstruction task or Neural rendering task. I wonder why the authors only used ScanRefer which is also originate from ScanNet dataset. Even ScanNetpp dataset have some data configuration with higher quality images and pointclouds. My personal opinion is that since the authors address the "referring" 3D segmentation task, the authors need to design how to extract high-quality instructions as well. But here, it seems that authors just naively use the given instructions from the ScanRefer benchmark. If so, I am not sure whether I can determine the dataset as "newly" curated data.

A2: Thank you for this valuable suggestion. We agree that reusing existing scenes and expressions should be clearly distinguished from collecting new data. Following your advice, we have removed wording such as “newly” from the benchmark description in the main text and moderated the related claims, clarifying that StreamRefer builds on ScanNet observations and ScanRefer’s human-annotated expressions.

We chose these resources because streaming referring segmentation requires RGB sequences, object-linked language descriptions, and instance annotations to determine target visibility across frames. Our work on StreamRefer focuses on adapting these resources to the causal streaming setting: constructing chronological episodes, associating expressions with per-frame target masks, and organizing evaluation across four visibility patterns—Persistent, Emerging, Disappearing, and Intermittent—at three sequence lengths. This adaptation enables evaluation of target identification, absence handling, and reappearance under partial observations. We are particularly grateful for your insightful recommendation to incorporate higher-quality captures and collect expressions tailored to streaming observations. This valuable guidance has helped us identify a clear direction for our future work toward a more comprehensive and practically relevant benchmark.

With this distinction, we would like to clarify the paper’s two core contributions:

  1. A new setting that brings 3D-RES closer to practical deployment. Stream3D-RES enables an agent to reconstruct a scene and segment a language-referred target as RGB observations arrive, without requiring a complete, preconstructed scene. This makes the task more applicable to embodied agents exploring unfamiliar environments and helps narrow the gap between offline research and real-world use.
  2. A strong baseline to support future research in this setting. Ground3R combines recurrent multimodal memory with spatial query localization and refinement, providing an effective starting point for further advances in streaming 3D referring segmentation.

W-3: Revision of Figure 2 and Clarification of Architecture Dataflow

Comment: Revision is needed in Figure 2. What is input and output of each module? Each module is disconnected without having any connected lines. I am totally lost at this moment... Naturally, I wonder how the information has been mixed across multi-view images having different viewpoints... Moreover, since the proposed method targets at "referring" 3D segmentation, it is quite important how the given instruction are mixed through the visual tokens... Please refer to the PanSt3r pipeline figure as an example.

A3: Thank you for taking the time to explain where Figure 2 was difficult to follow and for recommending PanSt3r’s pipeline figure. This closely related work also offers helpful inspiration, and we will cite and discuss it in the revision. Your questions have guided us to revise Figure 2 with explicit module inputs and outputs, directional connections, and recurrent state updates, while distinguishing feature types by color and clearly marking frozen components.

To address your questions directly:

  1. What enters and leaves each module? The geometry backbone takes the current RGB frame and previous geometric state and produces visual features and online 3D geometry. MMB combines visual features, language features, and historical multimodal memory to produce updated memory features. SQLR generates object proposals and uses language and memory features to select and refine the target. The final output is a target mask with corresponding 3D points, or an empty mask when the target is absent.
  2. How is information combined across viewpoints? Previous observations inform the current prediction through CUT3R’s geometric state, MMB’s language-conditioned scene memory, and object-query inheritance via 3D centroid matching. These paths carry geometric, semantic, and object-level information across frames without accessing future observations.
  3. How does language interact with visual features? Language participates at two levels: MMB integrates the expression with visual features and historical scene context, while SQLR uses the expression to identify and refine the target among language-independent object proposals. Beyond single-frame proposal-based segmentation, SQLR transfers query features across views through 3D matching.

We sincerely appreciate your constructive guidance and will continue refining Figure 2 and its accompanying explanation to make the pipeline easier to follow.

Q1: Writing Clarity in the Abstract

Comment: In abstract, the authors wrote as "agents must reconstruct the scene and identify the referred object incrementally as new images arrive, without access to future observations." "We introduce .... incoming RGB images using only current and past observations"... This is a personal writing style, but I would prefer "we are targeting at ... using a streaming video. Given an observation at time t, the model recurrently update the predictions with the in-coming observation ... "

A4: Thank you for this helpful suggestion. We agree that stating the causal constraint twice in consecutive sentences was redundant, and we have revised the Abstract directly following your recommended structure:

"We target streaming 3D referring expression segmentation (Stream3D-RES) from a causal RGB video stream: given an incoming observation at time tt and a language instruction, the model recurrently updates both 3D scene geometry and target segmentation predictions."

We appreciate your detailed feedback, which has helped us significantly improve the clarity and presentation of our paper.

Reviewer_VaG8OpenReview ↗

Thank you for the precise rebuttals.

I have no concerns at all. The authors well addressed the aforementioned issues and it is clear to me. I personally recommend the authors to provide the results in Nr3D / Sr3D / RIORefer in the main paper. This paper can be a unique baseline for the following papers that may target at the same task.

My original score was "weak reject", but the final score is "weak acceptance".

Thanks.

Dear Reviewer VaG8,

Thank you very much for your recognition and for raising your score. We are very glad that our responses have addressed all of your concerns. Your constructive feedback has greatly improved the quality and clarity of our paper.

We will continue refining the presentation and layout of the manuscript, including moving the evaluation results on Nr3D, Sr3D, and RIORefer into the main paper as you suggested. Once again, we sincerely appreciate your time, effort, and valuable suggestions.

Best regards,
Authors

Reviewer_U8JXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces streaming 3D referring expression segmentation, where a model reconstructs a scene and segments a language-referred object from incoming RGB frames without accessing future observations.

Critical strengths
  1. The paper addresses a meaningful setting where grounding requires context from earlier observations and the target may leave or re-enter the view. StreamRefer explicitly evaluates different visibility patterns, providing a useful test of grounding under partial observations rather than assuming access to a complete scene.
  2. The main components have clear empirical support. Removing recurrence reduces Global mIoU from 41.47 to 36.99, while removing language conditioning reduces it to 34.92. The paper also provides a detailed efficiency evaluation, reporting 128 ms per frame and approximately constant GPU memory over sequences of up to 512 frames.
Critical weaknesses
  1. The baseline comparison could be stronger. The paper compares mainly with MVGGT-causal and 2D video segmentation methods, but does not include an online 3D object-memory approach, such as adapted MTU3D, or a GPT-based target selector. The baselines also use released checkpoints, while Ground3R is trained specifically for the streaming setting. A comparison under the same training protocol would help show how much of the improvement comes from the proposed method.
  2. The experiments do not fully test 3D consistency or long-term grounding. The reported IoU scores compare masks in each image, so they do not directly measure whether the reconstructed target aligns correctly across views. Accuracy is tested only on 8–32-frame sequences; the 512-frame experiments report speed and memory, but not segmentation accuracy. Evaluation over longer time spans, especially when the target disappears and returns, would make the conclusions more convincing.
Questions for authors

How accurate is the accumulated target point cloud against ground-truth 3D geometry, and how does grounding performance change over longer temporal spans with extended target absence and reappearance?

3 comments

Thank you very much for your positive and insightful review! We greatly appreciate your recognition of the practical significance of our streaming 3D referring expression segmentation setting, our evaluation across diverse visibility patterns in StreamRefer, and the empirical and efficiency advantages of Ground3R. We address your specific comments and questions in detail below.


W1: Baseline Comparisons (Online 3D Object-Memory & Streaming Training Adaptation)

Comment: The baseline comparison could be stronger. The paper compares mainly with MVGGT-causal and 2D video segmentation methods, but does not include an online 3D object-memory approach, such as adapted MTU3D, or a GPT-based target selector. The baselines also use released checkpoints, while Ground3R is trained specifically for the streaming setting. A comparison under the same training protocol would help show how much of the improvement comes from the proposed method.

A1: Thank you for this valuable suggestion. It prompted us to strengthen both the baseline coverage and the training comparison, improving the paper’s experimental evaluation. We have now added MTU3D[A] as an online 3D object-memory baseline and trained all three baselines—MTU3D, MVGGT, and OnlineRefer—on the ScanRefer/ScanNet training split under causal streaming conditions. Ground3R continues to outperform them on all four grounding metrics.

Results with streaming-trained baselines. All methods receive only current and past RGB observations and are evaluated on the full StreamRefer benchmark (7,835 expressions). MVGGT-streaming refers to our causal streaming adaptation of MVGGT.

MethodInput ModalityTraining SettingGlobal mIoU (%)View mIoU (%)Acc@25 (%)Acc@50 (%)Empty-Frame Acc (%)
OnlineRefer (Adapted)RGB StreamCausal Streaming25.5343.3141.0318.5753.78
MTU3D + CUT3R (Adapted)RGB Stream (Online CUT3R)Causal Streaming27.0455.0143.5624.1079.90
MVGGT-streaming (Adapted)RGB Stream (Streaming-Adapted)Causal Streaming36.9964.0256.0236.2290.25
Ground3R (Ours)RGB Stream (Online CUT3R)Causal Streaming41.4765.1659.4643.6986.54

Compared with the strongest baseline, MVGGT-streaming, Ground3R achieves 4.48 percentage points higher Global mIoU and 7.47 points higher Acc@50, with lower inference latency at frame 32 (128.0 vs. 217.3 ms). MVGGT-streaming achieves higher Empty-Frame Accuracy, while Ground3R leads on the four grounding metrics. The remaining gains after baseline retraining support the contribution of our design beyond access to streaming supervision.

How we adapted and trained the baselines.

  • MTU3D + CUT3R: We supplied MTU3D with depth, camera poses, and intrinsics predicted online by the same frozen CUT3R backbone used in Ground3R, without ground-truth geometry at inference. We then fine-tuned its grounding parameters using Ground3R’s dynamic 8-frame target-window sampling.
  • MVGGT-streaming: We modified MVGGT to process frames causally with bounded memory, avoiding full-prefix recomputation, and trained it with the same dynamic 8-frame target-window sampling.
  • OnlineRefer: We fine-tuned this online 2D baseline under causal 8-frame streaming supervision on the same training split.

These experiments align the training split and causal RGB-only inputs, while retaining model-specific training settings, including MVGGT’s native input resolution and loss. For MTU3D, replacing RGB-D geometry with CUT3R predictions changes its input conditions; accordingly, this comparison evaluates its adaptation to our RGB-only task rather than its performance in its native RGB-D setting.

[A] Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation. ICCV, 2025.

W2 & Q1 (Part 1): 3D Geometric Accuracy and Cross-View Consistency of Accumulated Target Point Clouds

Comment (W2): The experiments do not fully test 3D consistency or long-term grounding. The reported IoU scores compare masks in each image, so they do not directly measure whether the reconstructed target aligns correctly across views.

Question (Q1 - Part 1): How accurate is the accumulated target point cloud against ground-truth 3D geometry...?

A2: Thank you for this valuable suggestion. Following your advice, we added 3D IoU and cross-view consistency evaluations. Ground3R outperforms the compared online baselines in both accumulated 3D target segmentation and cross-view grounding consistency.

(1) 3D IoU [B]. We compare accumulated target predictions against ground-truth observed 3D surfaces on 120 validation episodes from 105 scenes, using one sequence-level Sim(3) alignment without per-frame ICP. Ground3R leads at all three thresholds, exceeding MVGGT-causal by 2.77 percentage points at 10 cm.

Method3D IoU@5 cm (%)3D IoU@10 cm (%)3D IoU@20 cm (%)
OnlineRefer + CUT3R13.0913.4613.49
SSA + CUT3R17.7718.4518.46
StreamingRVOS + CUT3R23.0423.9424.30
ReferDINO + CUT3R23.4024.5324.58
MVGGT-causal28.0128.8229.07
Ground3R30.0031.5931.92

(2) Cross-view consistency. Across the full 7,835 expressions, we measure whether predictions are correct in both frames of consecutive view transitions.

TransitionSuccess criterionMVGGT-causalGround3R
Visible → VisibleIoU > 0.5 in both frames27.01%41.19%
Visible → AbsentIoU > 0.5 in the visible frame and an empty prediction in the absent frame19.02%35.58%

Ground3R improves joint success by 14.18 and 16.56 percentage points, respectively. These results support both more accurate accumulated 3D segmentation and more consistent grounding across consecutive views.

[B] Semantic3D.net: A New Large-Scale Point Cloud Classification Benchmark. ISPRS Annals, 2017

W2 & Q1 (Part 2): Long-Term Grounding Across Short-to-Extended Horizons (1 to 512 Frames) and Target Reappearance

Comment (W2): Accuracy is tested only on 8–32-frame sequences; the 512-frame experiments report speed and memory, but not segmentation accuracy. Evaluation over longer time spans, especially when the target disappears and returns, would make the conclusions more convincing.

Question (Q1 - Part 2): ...and how does grounding performance change over longer temporal spans with extended target absence and reappearance?

A3: Thank you for this valuable suggestion. Following your advice, we extended the accuracy evaluation to 512 frames, including target disappearance and reappearance. Global mIoU initially increases as more observations become available, then decreases over longer sequences, rising from 54.78% at 1 frame to 81.07% at 32 frames, before declining to 51.00% at 512 frames.

We evaluated eight revisit sequences in which the target was initially recognized, using a fixed checkpoint and sampling every fifth frame.

Sequence length (frames)1832128256512
Global mIoU (%)54.7875.5581.0766.3960.2051.00
View mIoU (%)54.7874.3979.4075.3180.8374.20
Target recovery (successful/total events)—1/11/12/34/611/20

A recovery is successful when the model achieves IoU ≥ 0.5 within the first three consecutive visible frames after the target reappears. At 512 frames, Ground3R successfully recovers the target in 11 of 20 events.

Why does accuracy first improve and then decline? Additional observations initially provide more context for identifying the target. Over extended sequences, however, geometric errors can accumulate and make cross-view matching less reliable. Such drift is a longstanding challenge in online 3D reconstruction, motivating correction mechanisms in ElasticFusion [C] and BundleFusion [D]. CUT3R [E], our reconstruction backbone, also explicitly acknowledges potential drift over very long sequences without global alignment. Ground3R inherits this limitation, which likely contributes to the observed decline. Reducing accumulated drift and improving grounding over extended sequences will be an important direction for our future work.

[C] ElasticFusion: Dense SLAM Without A Pose Graph. RSS 2015.

[D] BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration. TOG, 2017.

[E] Continuous 3D Perception Model with Persistent State. CVPR, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.