acceptodds
Under review as a conference paper at ICLR 2027

TempoGround: State-Aware Streaming Visual Grounding with Vision-Language Models

Abstract

Visual grounding maps language referents to spatial targets and is central to open-vocabulary perception with vision-language models. Existing methods have made substantial progress on single-frame and video-based visual grounding, yet under streaming inputs they still suffer from identity drift, cross-frame inconsistency, and fragile localization under partial occlusion. To address these issues, we present TempoGround, a VLM-native framework that detects cross-frame object correspondence and explicitly models object presence states, thereby enabling accurate and consistent visual grounding under streaming inputs. The key is a curriculum prediction mechanism guided by state-aware cross-frame correspondence: TempoGround resolves 2D instance association, predicts whether each object newly enters, continues in, or leaves the view, decodes the 2D box, and then lifts it to a camera-frame 3D box. As token-level supervision alone cannot capture the geometric objectives of streaming grounding, we further introduce Streaming Grounding Reinforcement (SGR), which optimizes TempoGround with verifiable Grounding, Identity, and Consistency rewards, jointly reinforcing persistent localization and temporally consistent predictions. We carefully design a three-stage training strategy and train TempoGround on large-scale data. We evaluate visual grounding under causally streaming inputs on multiple challenging benchmarks: TempoGround improves F1_2D@0.5 and F1_2D@0.95 by 4.4 and 0.5 on average, and F1_3D@0.25 and AP_3D by 6.2 and 7.5, respectively. These results demonstrate that TempoGround provides a practical foundation for visual grounding under streaming inputs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual grounding, vision-language models, reinforcement learning, open-vocabulary detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1nguOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

see weakness.

Critical strengths

see weakness.

Critical weaknesses

The paper addresses an interesting problem, namely streaming visual grounding under causally arriving image streams, but the literature coverage, technical contribution, empirical rigor, and presentation are insufficient for ICLR.

  1. The literature review is inadequate and shows limited knowledge of visual grounding.
    The paper cites some VLM and 3D perception works but omits many classic and directly relevant visual grounding studies, including TransVG, MDETR, Grounding DINO, ...... It also lacks discussion of open vocabulary grounding, referring expression comprehension and segmentation, video grounding, and tracking by detection, so the problem positioning and novelty are difficult to assess.

  2. The technical contribution is largely an engineering combination and the method is not sufficiently interpretable.
    TempoGround combines cross frame correspondence, discrete presence states, 2D to 3D lifting, and GRPO with verifiable rewards, but each component is familiar and the paper does not isolate why this state aware design is necessary beyond a simple tracker or a 2D prior baseline. It is also unclear how the correspondence cue, presence states, and SGR rewards jointly affect the final prediction, so the mechanism level behavior remains hard to interpret.

  3. The empirical workload is thin and feels rushed.
    The main evaluation uses only 24 test sequences per dataset and reports no multiple seed variance, confidence intervals, or compute matched comparisons, while SGR is trained on only about 4.1K trajectories for two epochs. Several baselines are not adapted to the causal streaming protocol and the paper lacks parameter matched or training budget matched controls, so the reported gains are difficult to attribute to the proposed framework.

  4. The evaluation protocol and metrics are not sufficiently justified.
    The paper claims that presence and identity stability are central, yet the main tables report only F1D and AP3D without separate presence state, identity switch, or stream level diagnostics. The causal streaming protocol is described too briefly and the choice of 24 sequences plus limited referring samples raises concerns about statistical reliability and coverage of enter and exit events.

  5. The presentation is difficult to follow and key details are poorly organized.
    The manuscript contains long line number blocks, inconsistent table and figure formatting, and several dense tables that are hard to parse. Important data filtering, annotation, and reward computation details are deferred to an appendix that is not self contained, and the writing has a template generated flavor with overclaimed phrasing, which makes the paper feel hastily prepared.

Questions for authors

see weakness.

Reviewer_8WpzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce TempoGround, a VLM-based framework designed for streaming visual grounding, where the model tracks and localizes objects over time based on a language query. The model is fine-tuned using a three-stage training strategy that combines supervised fine-tuning (SFT) and reinforcement learning using GRPO. The first two stages use approximately 31 million training samples covering 2D and 3D grounding, while the final stage uses GRPO to improve grounding accuracy and temporal consistency. The main idea is to use information from the previous and current frames to maintain object correspondence over time. The model first uses the previous 2D bounding boxes and object labels as correspondence cues, predicts the object's presence state (new, track, or lost), and then generates the current 2D bounding box followed by a camera-frame 3D bounding box. The presence state helps the model determine whether an object matching the language query has entered, remains in, or has left the current view.

The authors also introduce Streaming Grounding Reinforcement (SGR), which uses grounding, identity, and consistency rewards to improve the model's predictions across frames.

Critical strengths

1.The paper addresses an important problem in streaming visual grounding, particularly maintaining object identity and consistent localization across frames. The motivation is clear, and the proposed approach is easy to understand.

2.The use of correspondence cues and explicit object states (new, track, lost) is a useful approach to maintaining object identity across frames. The model also considers errors in previous predictions by introducing different cue perturbations during training.

3.The three-stage training strategy is well structured, starting with 2D and 3D grounding, followed by streaming supervision and GRPO. The authors also provide ablation studies showing the contribution of the proposed components.

4.The model achieves strong results on multiple benchmarks, particularly for 3D grounding. The authors compare their approach with several existing methods and provide ablation studies to evaluate the effectiveness of correspondence cues, presence-state prediction, and the proposed reinforcement learning rewards.

Critical weaknesses

1.The model is trained on around 31 million samples and requires 109 hours using 128 PPUs. This makes the approach expensive and difficult to reproduce with limited resources.

2.The authors train and evaluate the model on several of the same datasets. It is unclear how well the model would perform on a completely new dataset or with different camera settings.

3.The model uses previous predictions to track objects in the current frame. If the previous bounding box is wrong, the error could affect future predictions. Although the authors test noisy cues, they do not clearly show how errors accumulate over long sequences.

4.The authors mainly evaluate the model using 2D and 3D bounding box accuracy. However, since tracking object identity is one of the main contributions, it would be useful to report tracking metrics such as identity switches and presence-state accuracy.

Questions for authors
  1. How does TempoGround handle geometrically invalid bounding-box predictions, such as negative 3D dimensions, out-of-range 2D coordinates, or physically implausible object sizes?

  2. Since TempoGround predicts camera-frame 3D boxes in meters from RGB inputs, how does it resolve metric scale and depth ambiguity? Are camera intrinsics available at inference?

3.During inference, how does TempoGround determine whether a previously predicted 2D bounding box is reliable, given that ground-truth annotations are unavailable? Does the model estimate any confidence or uncertainty for its predictions?

  1. Also during the inference, if an incorrect box is propagated, what mechanism prevents identity drift or error accumulation over subsequent frames?

4.TempoGround follows a sequential 2D-to-3D prediction curriculum, where the generated 3D box is conditioned on the preceding 2D prediction. How sensitive is 3D grounding to errors in the predicted 2D box?

5.Since the streaming training data include ARKitScenes, ScanNet, ScanNet++, and ASE, which are also used for evaluation, could the authors clarify the train/test separation protocols and whether evaluation sequences are strictly scene-disjoint from training?

Reviewer_yJWYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TempoGround addresses streaming visual grounding by explicitly modeling object correspondence and visibility states across consecutive frames. The method introduces a state-aware prediction curriculum that sequentially performs cross-frame association, presence-state estimation, 2D localization, and camera-frame 3D grounding. To improve temporal robustness, it employs a three-stage training strategy combining large-scale single-frame supervision, streaming correspondence learning, and reinforcement learning with grounding, identity, and consistency rewards. Experiments show improvements in both 2D and 3D localization under causal streaming inputs, particularly in maintaining object presence and reducing error propagation.

Critical strengths
  1. The task is practically meaningful.
  2. The prediction process follows a clear and well-structured design.
  3. The training strategy takes error propagation in real-world streaming scenarios into account.
Critical weaknesses
  1. A more direct evaluation of identity and temporal consistency would be helpful. While the paper emphasizes identity drift and cross-frame stability, improvements in F1 and AP may not fully reflect improvements in identity maintenance and temporal consistency.
  2. Streaming inference efficiency could be further evaluated. Reporting FPS, per-frame latency, throughput, and computational cost as the number of objects increases would help assess the method's real-time applicability.
  3. A broader evaluation of generalization would strengthen the paper. With only 24 test sequences per dataset, it would be valuable to examine performance under rapid egocentric motion, long-term occlusion, object reappearance, and complex outdoor environments.
Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.