- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a new task: Streaming 3D Referring Expression Segmentation (Stream3D-RES). The task aims to perform simultaneous 3D scene reconstruction and target object segmentation based on a continuous stream of RGB images and a natural language description, operating in a causal manner—relying solely on current and historical information. Unlike traditional 3D referring expression segmentation methods that depend on complete point clouds or offline multi-view imagery, this task requires the model to perform inference using only current and past observations, while also accounting for dynamic visibility changes.
To address these challenges, the authors propose the "Ground3R" framework. The authors also have developed the "StreamRefer" benchmark, which encompasses four types of target visibility patterns and three sequence lengths, designed to evaluate capabilities in online, language-guided 3D localization.
Critical strengths
-
The research problem holds clear practical value, and the task formulation is ingeniously conceived.
-
The designs of MMB and SQLR complement each other.
-
Experiments demonstrate that the proposed method offers significant performance advantages in online settings.
Critical weaknesses
- Lack of direct validation for the language-conditioned memory mechanism.
A central claim of this paper is that the MMB (Multi-Modal Memory Bank) can retain language-relevant scene information from historical observations, specifically spatial references that are no longer visible in the current frame. However, the current experiments (Table 4) assess performance changes primarily by removing either the language conditioning or the recurrent state.
- Lack of analysis regarding the quality of 3D geometric reconstruction and sensitivity to geometric errors.
The paper lacks a quantitative analysis of reconstruction quality, geometric uncertainty, and centroid matching errors, leaving the method's robustness against 3D geometric inaccuracies unclear.
- Evidence for long-sequence semantic memory and cross-scene generalization remains insufficient.
Questions for authors
See the weaknesses
3 comments
W2: Sensitivity to Geometric Reconstruction Errors and Confidence Modulation
Comment: The paper lacks quantitative analysis of reconstruction quality, geometric uncertainty, and centroid matching errors, leaving its robustness against geometric inaccuracies unclear.
A2: Thank you for this valuable suggestion. Following your advice, we added quantitative evaluations of online reconstruction quality, 3D centroid matching, and confidence-modulated inheritance on 120 stratified validation episodes (40 each at 8, 16, and 32 frames). Online CUT3R maintains centimeter-level accuracy on 8–32-frame streams, and replacing predicted geometry with ground-truth geometry for query matching improves Global mIoU by only 0.12 points.
1. Role and quality of online reconstruction. CUT3R is frozen during training: Ground3R performs joint reconstruction and grounding at online inference, without optimizing the geometry backbone; we will clarify this in the manuscript. For evaluation, we align the entire predicted sequence to the ground truth once, correcting its overall scale, rotation, and translation without adjusting individual frames separately [A]. Under this protocol, online CUT3R maintains low scene reconstruction error across 8–32 frames, while exhibiting gradual drift as sequences lengthen.
| Sequence length | 8 frames | 16 frames | 32 frames | Overall |
|---|---|---|---|---|
| Median 3D scene error (cm) | 5.03 | 6.28 | 9.94 | 7.08 |
| Surface coverage @ 10 cm (%) | 93.78 | 93.21 | 90.29 | 92.42 |
2. How much do geometric errors affect query matching? Using the same checkpoint on 120 stratified validation episodes, we compared three variants, keeping visual and memory features unchanged:
- Default: Query matching uses centroids computed from predicted CUT3R pointmaps.
- Confidence-modulated: CUT3R’s per-pixel confidence weights centroid estimation and modulates historical feature inheritance.
- GT-geometry routing: Centroids are computed using ground-truth depth and camera poses.
| Geometry used for inheritance | Instance matching precision (%) | Match coverage (%) | Global mIoU (%) |
|---|---|---|---|
| Predicted CUT3R geometry | 88.21 | 87.70 | 38.09 |
| Confidence-modulated inheritance | 88.24 | 87.64 | 38.15 |
| GT-geometry routing | 89.25 | 87.00 | 38.21 |
Instance matching precision measures the proportion of evaluable matched query pairs corresponding to the same ground-truth object.
Default matching achieves 88.21% precision, only 1.04 points below GT-geometry routing, with a 0.12-point difference in Global mIoU. These results suggest that predicted geometry is sufficiently reliable for query inheritance on this subset. Incorrect matches remain possible; even GT geometry does not eliminate them, since geometric proximity alone does not guarantee object identity.
Our default centroid estimation uses coordinate-wise median pooling over a local () neighborhood, reducing sensitivity to isolated coordinate outliers. These experiments assess naturally occurring reconstruction errors, rather than robustness to arbitrarily large coordinate deviations.
3. Does geometric confidence help? Confidence modulation increases Global mIoU from 38.09% to 38.15% without retraining. The improvement is modest, but suggests that geometric confidence can help control feature inheritance. We appreciate your emphasis on geometric uncertainty and will further explore confidence-aware matching, particularly under larger reconstruction errors.
[A] Unlocking the Power of Critical Factors for 3D Visual Geometry Estimation. CVPR, 2026.
W3: Long-Sequence Semantic Memory and Cross-Scene Generalization
Comment: Evidence for long-sequence semantic memory and cross-scene generalization remains insufficient.
A3: Thank you for encouraging this broader evaluation. We extended accuracy testing to 512 frames and added zero-shot evaluation on RIORefer, complementing the Nr3D and Sr3D results already included in our submission.
1. Long-sequence grounding and target reappearance.
Experimental setup. We selected eight sequences in which Ground3R correctly recognized the target during the initial observations and the target subsequently disappeared and reappeared. This evaluates whether an initially recognized target can be recovered after absence.
We sample one frame every five frames from each original video and evaluate progressively longer prefixes of the same sampled sequence, from 1 to 512 frames. Thus, the 32-frame evaluation uses the first 32 sampled frames, while the 512-frame evaluation extends that same stream to 512 sampled frames.
All runs use the same checkpoint without additional fine-tuning. Each sequence–prefix run starts with independently initialized recurrent states. Frames are then processed chronologically, carrying the states forward without resets within the run and without access to future frames.
| Evaluated prefix length (frames) | 1 | 8 | 32 | 128 | 256 | 512 |
|---|---|---|---|---|---|---|
| Global mIoU (%) | 54.78 | 75.55 | 81.07 | 66.39 | 60.20 | 51.00 |
| Target recovery (successful/eligible reappearances) | — | 1/1 | 1/1 | 2/3 | 4/6 | 11/20 |
Global mIoU evaluates the entire prefix, including target-visible and target-absent frames. A recovery succeeds if the model reaches IoU ≥ 0.5 in at least one of the first three consecutive visible frames after the target reappears. Events are identified from ground-truth visibility annotations, and counts are pooled across the eight sequences. Longer prefixes include additional events; “—” indicates no eligible event.
Results and interpretation. Global mIoU first increases and then decreases, rising from 54.78% at 1 frame to 81.07% at 32 frames, before declining to 51.00% at 512 frames. At 512 frames, Ground3R successfully recovers the target in 11 of 20 events.
Additional observations initially provide useful scene context. Over extended sequences, accumulated geometric errors can make cross-view association less reliable. Such drift is a longstanding challenge in online reconstruction [B, C]. CUT3R also acknowledges potential long-sequence drift without global alignment [A]. Ground3R inherits this limitation, which likely contributes to the observed decline. Reducing drift and strengthening long-term memory are important directions for future work.
2. Generalization beyond StreamRefer. Following your suggestion, we additionally evaluated RIORefer, using 88 expressions covering 44 targets in 8 scenes, with chronological 8-frame episodes and no fine-tuning on the new dataset.
| Method | Global mIoU (%) |
|---|---|
| OnlineRefer | 22.94 |
| StreamingRVOS-1B | 27.35 |
| MVGGT-causal | 27.20 |
| Ground3R | 31.33 |
All results are Global mIoU (%).
These results provide additional evidence of generalization beyond StreamRefer. We sincerely appreciate your constructive feedback, which has helped us strengthen the evaluation and clarify the scope of our conclusions.
[A] Continuous 3D Perception Model with Persistent State. CVPR, 2025.
[B] ElasticFusion: Dense SLAM Without A Pose Graph. Robotics: Science and Systems (RSS), 2015.
[C] BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration. ACM Transactions on Graphics (TOG), 2017.
We sincerely thank you for recognizing the practical value of the task, the complementary designs of MMB and SQLR, and Ground3R’s performance in online settings. Your constructive suggestions have helped us strengthen the experimental evidence. We address your three concerns below.
W1: Direct Validation of Language-Conditioned Historical Memory
A1: Thank you for this valuable comment. To directly test this claim, we added a diagnostic evaluation on out-of-view spatial references. Ground3R relies on historical observations when spatial references leave the view, and its accuracy drops substantially when those earlier reference frames are removed or replaced.
We evaluate Ground3R on 234 16-frame validation episodes from 73 scenes containing same-class distractors, where the spatially referenced object (e.g., "the chair beside the desk") appears only in the first half of the stream and is completely out of view in the second half.
Removing or replacing the earlier reference frames reduces Global mIoU by 9.96 and 21.14 percentage points, respectively. Together with the MMB ablations in Table 4, these results confirm that Ground3R retains language-relevant spatial references from past observations to disambiguate targets in subsequent views.