acceptodds
Under review as a conference paper at ICLR 2027

Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

Abstract

Spatio-temporal video grounding (STVG) requires a model to identify when a queried event occurs and track the referred target throughout that interval. Applying multimodal large language models (MLLMs) frame by frame produces long visual contexts and autoregressive coordinate sequences. We formulate MLLM-based STVG around second-level trajectory control points and reconstruct dense tracks with deterministic inter-second smoothing. Training combines continued pre-training on broad grounding data, supervised fine-tuning with generated decision traces whose numerical labels are replaced by ground truth, and reinforcement learning with a verifier based on temporal and motion-aware spatial overlap. On 10-second videos, the second-level protocol reduces end-to-end latency by and prefill compute by relative to a 25-FPS frame-level alternative; on 60-second videos it remains executable when 5-FPS and 25-FPS inputs exceed the context window. A 9B task-aligned model improves over its SFT checkpoint on VidSTG and HC-STVG and also improves its base checkpoint on Video-MME and MMVU, demonstrating that compact grounding can coexist with general video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Models, Spatio-Temporal Grounding, Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_wzriOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • This paper addresses efficient spatio-temporal video grounding (STVG) by predicting trajectory control points at one-second intervals and reconstructing dense object tracks via deterministic interpolation.
  • The proposed multi-stage training pipeline combines continued pretraining (CPT), supervised fine-tuning (SFT) on synthesized reasoning chains paired with ground-truth coordinates, and reinforcement learning (RL) driven by spatio-temporal overlap rewards.
  • Experiments on VidSTG and HC-STVG demonstrate improved grounding accuracy, preserved general video understanding, and a 29.3× inference speedup on 10-second videos compared to 25-FPS frame-level processing.
Critical strengths
  • Pragmatic multi-stage data curation that decouples reasoning from geometry. The pipeline scales across 344K CPT, 20K reasoning-SFT, and 20K RL samples. By pairing teacher-generated reasoning chains with ground-truth temporal spans and bounding boxes (supplemented by manual spot-checks), the SFT stage leverages synthetic semantic supervision without propagating teacher localization errors.

  • Consistent gains in both grounding and general video understanding. RL boosts mean tube IoU from 19.47 to 23.79 on VidSTG and from 33.75 to 40.99 on HC-STVG, while simultaneously improving over the base model on Video-MME and MMVU.

Critical weaknesses
  • The writing requires substantial improvement. The manuscript has limited readability and would benefit from clearer, more concise presentation. Although the authors disclose AI assistance with writing, language editing, and figure/table presentation, the remaining clarity and presentation problems suggest insufficient manual revision and proofreading, contributing to poor readability.

  • Lack of comparison against specialized STVG baselines. Table 2 benchmarks the model primarily against general-purpose MLLMs and proposed SFT checkpoint, omitting dedicated STVG architectures discussed in the related work. Standardized comparisons against task-specific baselines are necessary to establish its competitiveness on the target task.

  • Overstated motion-aware spatial reward formulation. Section 3.6 characterizes the spatial reward as "motion-aware," yet Eq. 6 simply computes the mean box IoU over the temporal intersection without explicit velocity, motion, or identity-consistency terms.

  • Overclaimed efficiency gains driven by input subsampling rather than technical innovation. Attributing the 29.3× speedup to the proposed method is misleading, as it stems primarily from downsampling inputs from 25 FPS to 1 FPS rather than algorithmic improvements.

Questions for authors

None

Reviewer_dTCrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a spatio-temporal video grounding pipeline combining second-level trajectory prediction, linear interpolation, and CPT–SFT–RL training. It reports grounding results on VidSTG and HC-STVG and efficiency comparisons with frame-level processing.

Critical strengths

The paper presents a multimodal large-model training and inference pipeline for spatio-temporal video grounding that jointly predicts event intervals and target trajectories while reducing the cost of dense trajectory generation.

Critical weaknesses
  1. Limited theoretical novelty. The proposed method assembles existing components including sparse trajectory representations, linear interpolation, supervised fine-tuning and localization-aware RL rewards. Key ideas have been explored in prior work such as STVG-o1 and LLaVA-ST, and the incremental technical contribution of this work is not clearly articulated.

  2. Incomplete experimental validation and insufficient comparisons with closely related methods. The current experiments mainly verify the benefits of the RL stage and temporal reward, while the individual contributions of CPT and semantic reasoning supervision are not isolated. Ablation studies on output granularity are limited to DanceTrack, and efficiency experiments confound input sampling rate and output granularity without reporting localization performance under matched configurations. Besides, the paper lacks direct comparisons with highly relevant methods including LLaVA-ST and CG-STVG. It is also unclear whether competing models in Table 2 share identical settings for input sampling, resolution, generation budget and post-processing

Questions for authors

Please address the main concerns raised in the Weaknesses section.

Reviewer_19jjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper asks how a large multimodal model should ground a query in a long video without drowning in frames. Its answer is simple and appealing. Predict one box per second and then fill in the dense track by straight line smoothing.

Trained in three steps.

  1. First learn broad grounding.
  2. Then learn a time then space reasoning style from traces whose numbers are replaced by true labels.
  3. Then tune with a reward that scores time overlap and tube overlap together.

On the two main tests the tuned 9B model beats the big general models in the table and it stays fast on long videos.

Critical strengths

The core idea is clear and it solves a real cost problem. At ten seconds the per second route uses far fewer tokens and far less time than the dense frame route and at sixty seconds it still runs while the dense route runs out of context.

The gains from tuning are large and consistent. Tube quality rises from 19.47 to 23.79 on VidSTG and from 33.75 to 40.99 on HCSTVG when the model moves from supervised tuning to reward tuning.

The final model beats Qwen3.5 on HCSTVG by a wide margin on tube quality even though Qwen3.5 is listed at 397B in total size. General video skill does not collapse after grounding training. Scores rise a little on VideoMME and on MMVU compared with the base model.

The supplement is helpful. It gives the reward training settings and the prompts and it states the limits of straight line smoothing in plain terms.

Critical weaknesses

The table is too narrow for the main claim. All five rivals are general models tested without task tuning while this model is tuned on the same test families. The paper does not label the rivals as zero shot and it does not test task specific grounding models or the recent multimodal grounding models that use reward tuning. That leaves the reader unable to tell how much of the win comes from the second level idea and how much comes from tuning alone.

Size is shown in a way that flatters the win. Qwen3.5 at 397B and Kimi k2.5 at 1T are total sizes for sparse models. Only 17B and 32B are active per token in those models. A fair size note would give active size next to total size.

Two rows look odd and need an explanation. Gemini3 Flash has no time score on HCSTVG yet it still has a tube score and a strict tube score. Kimi k2.5 has a decent time score on HCSTVG at 43.9 yet its tube score falls to 10.99. Those patterns look like parsing or identity failures rather than pure grounding skill and the paper gives no valid output rate.

The paper never says which HCSTVG version it uses or whether VidSTG results pool declarative and interrogative queries. That choice changes what the numbers can be compared with in the literature.

Only the final model is described as a three run mean and no spread is given for any model. Small gaps on VidSTG therefore cannot be judged with confidence.

The paper cannot be reproduced from what is given. The supplement only promises a future release with no link and no date. It gives no code and no checkpoint and no training commands for the first two stages and no seeds and no software versions and no evaluation script. The rival model versions and dates and sampling settings are also missing. That leaves the main table unverifiable from the submission alone.

Questions for authors

Can you state the HCSTVG version as v1 or v2 and the split as test or validation and the number of evaluated examples and can you confirm that no test videos from either set appear in the broad pretraining mix?

Were the rival models given the same per second JSON prompt and the same frame rate and thinking settings and what share of their outputs failed to parse?

Why does Gemini3 Flash lack a time score on HCSTVG while still getting tube scores and why does Kimi k2.5 lose so much tube quality on HCSTVG?

Can you add a task specific grounding model and a recent reward tuned multimodal grounding model to the same table so the win can be read against the field and not only against general models?

Can you share the code and checkpoint and training and evaluation commands and seeds and software versions needed to reproduce the main tables and can you say why these are not available for review if they cannot be shared?

Reviewer_GkZaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies efficient spatio-temporal video grounding with multimodal large language models. It represents trajectories using second-level bounding-box control points and reconstructs frame-level predictions through deterministic interpolation. The training pipeline combines continued pretraining, supervised fine-tuning on generated semantic reasoning traces with ground-truth numerical targets, and reinforcement learning using temporal and spatial overlap rewards. The paper reports grounding improvements over its SFT checkpoint and evaluated general-purpose MLLMs, together with computational savings relative to dense frame-level protocols.

Critical strengths
  1. The paper addresses a practical bottleneck: long visual contexts and autoregressive coordinate sequences make dense MLLM-based grounding expensive. Separating compact prediction from deterministic reconstruction is a simple, potentially useful design that does not require an additional learned decoder. Table 3 provides direct measurements of latency, tokens, KV-cache size, and prefill computation using the same checkpoint and measurement environment.

  2. The evaluation provides evidence that the training and output design are useful in the tested settings. Table 2 shows improvements from SFT to RL on both grounding datasets; Table 4 controls visual input when comparing output granularities; and the supplementary reward ablation supports the combined reward. These controls strengthen the empirical case, although they do not fully establish the complete accuracy-efficiency trade-off.

Critical weaknesses
  1. The additional contribution over closely related task-adapted methods is insufficiently established. Table 2 compares general-purpose MLLMs and the authors’ own checkpoints, but omits close STVG methods such as STVG-o1 (Gu et al., Thinking With Bounding Boxes, arXiv:2511.21375) and SpaceVLLM (Wang et al., arXiv:2503.13983), both discussed in the related work. STVG-o1 already combines intermediate reasoning and localization-oriented reinforcement fine-tuning without architectural modifications. Moreover, the SFT-to-RL comparison does not isolate the benefit of corrected reasoning supervision. Consequently, the current evidence does not clearly separate the contribution of the proposed representation and training choices from task adaptation and additional training.

  2. The central accuracy-efficiency claim is not fully supported by a joint evaluation. Table 3 changes both visual sampling density and output representation, but does not report grounding quality alongside cost. Table 4 holds the 1-FPS input fixed and compares output granularities on DanceTrack, so it neither measures the effect of removing visual evidence nor establishes the complete STVG trade-off. Existing task-adapted methods may already use sparse input (e.g., STVG-o1 uses 2 FPS), so the 25-FPS reference alone does not establish an efficiency advantage over these alternatives. This limits how strongly the reported speedup supports the paper’s practical contribution.

Questions for authors
  1. Evaluation clarification: Which HC-STVG version/split and VidSTG query subsets are used in Table 2? Are trajectories reconstructed at the benchmark’s frame-level timestamps and evaluated with the official metrics? Please also distinguish the spatial reward in Eq. (6) from the reported tube-IoU metric.

  2. Evidence for the efficiency contribution: Can you provide STVG accuracy and latency for the same examples under controlled input-sampling and output-granularity configurations, and a comparison with a close task-adapted method? If such a comparison is unavailable, please clarify the constraints. These results would materially affect the assessment of the claimed efficiency benefit, particularly for short events and rapid motion.

  3. Evidence for the training contribution: Is there a matched comparison between corrected reasoning SFT and direct supervision with the same ground-truth spans and trajectories? Such evidence would clarify the independent value of the reasoning-supervision component.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.