acceptodds
Under review as a conference paper at ICLR 2027

VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

Abstract

Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding. Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding. To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval. Our main contributions: (1) Slow-Fast Trajectory Encoding, a dual-branch architecture that establishes trajectory representations by densely sampling low-resolution frames for high-frequency motion cues and sparsely selecting high-resolution frames for scene context, aggregated by a shared LLM backbone; (2) Mask-Guided Contrastive Learning, a training objective built upon these representations that couples an intra-group loss and a batch-level bidirectional loss under a relation mask, excluding unverified relations and preventing confirmed matches from becoming false negatives; (3) Text2Traj-Bench, a trajectory-level text-video benchmark with model-assisted construction and human-verified query–trajectory relations. Experimental results show that VTrajEmbed-5B improves text-to-trajectory and trajectory-to-text R@1 over Qwen3-VL-Embedding-8B by 15.48 and 18.05 percentage points, respectively, and achieves 49.48% average R@1 across five global retrieval benchmarks, compared with 42.86% for Qwen3-VL-Embedding-8B.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal understanding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_92x8OpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VTrajEmbed, a multimodal embedding framework that aligns natural-language descriptions with video object trajectories. It introduces a Slow-Fast trajectory encoder, a mask-guided contrastive learning objective, and the Text2Traj-Bench benchmark. The method achieves substantial improvements in trajectory-level retrieval.

Critical strengths
  1. The formulation of object-level video-text alignment through trajectory embeddings is meaningful and offers a useful alternative to conventional global video representations.

  2. The proposed benchmark incorporates human-verified trajectory-text relations, and the experimental results demonstrate substantial improvements over several multimodal embedding baselines.

Critical weaknesses
  1. The main experiments rely on pre-generated and curated trajectory candidates. The proposed query-conditioned pipeline is not evaluated end to end, leaving its robustness to missed detections, fragmented trajectories, and identity switches unclear.

  2. Text2Traj-Bench uses model-generated queries and admits only queries with a uniquely identifiable target in the curated candidate pool. This may introduce selection bias and does not adequately reflect ambiguous or naturally occurring user queries. Quantitative annotation agreement and benchmark audit statistics would further strengthen its reliability.

  3. Existing embedding models receive videos with overlaid bounding boxes, whereas VTrajEmbed employs a specialized object projector and geometry-aware trajectory representations. Stronger baselines with comparable object-centric inputs and task-specific training would help isolate the proposed method's contributions.

  4. Although the Slow-Fast branches are ablated, the individual contributions of appearance, geometric motion cues, temporal ordering, and semantic frame pruning remain insufficiently investigated. More targeted experiments would substantiate the claimed benefits of spatiotemporal modeling.

  5. Although VTrajEmbed outperforms Qwen3E-8B on average, its global retrieval performance remains below WeMM-4B (49.48% vs. 61.07% average R@1). Moreover, the impact of benchmark-specific adaptation and the comparability of training protocols should be clarified.

Questions for authors

Please refer to Weaknesses.

6 comments
AuthorsOpenReview ↗

Rebuttal(part 1/5)

Thank you for recognizing the value of trajectory-level text–video alignment and the benchmark. In response to your five concerns, we have added a full-pipeline evaluation and tracking-failure diagnostics (W1), an external evaluation using human-written referring expressions (W2), object-centric baselines with matched training budgets (W3), and component ablations (W4). We also clarify the auxiliary role of the global retrieval evaluation (W5). Below, we report the new evidence and its scope for each concern, and welcome any key issues you identify that still affect your overall assessment.

W1/Q1: End-to-End Retrieval and the Effects of Missed Detections, Trajectory Fragmentation, and Identity Switches

Response: To address the limitation that a fixed candidate pool does not evaluate the full pipeline, we have added an end-to-end evaluation with automatically generated candidates and a fragmentation diagnostic. The model from the main paper achieves mean end-to-end R@1 scores of 47.00% and 53.00% under the original parsing and parsing-failure fallback pipelines, respectively, exceeding Qwen8B on the same candidate pools by 9.50 and 10.50 percentage points.

Full-pipeline evaluation. We evaluate the model trained on Text2Traj-Bench in the main paper on 200 queries from 21 source videos in HC-STVG2 that were not included in the benchmark. Training and evaluation are separated by source, with no evaluation-specific adaptation. Qwen3-0.6B and SAM3.1 automatically generate candidates over the full video timeline from each query, without ground truth (GT) assistance or manual curation, and queries are not filtered according to target availability. Parsing, tracking, and ranking failures remain in the denominator. The person fallback is used only when parsing fails. Within each seed, the two embedding models share the candidate pool and both use the complete original query.

A target is considered available if a candidate achieves IoU≥0.4 on at least 80% of the GT-visible frames. All three seeds evaluate the same 200 queries: candidate coverage and end-to-end R@1 are reported as arithmetic means across runs, while conditional R@1 is calculated by dividing pooled hits by pooled covered queries. The number of independent queries remains 200.

Main-paper model evaluation: candidate pipelineMean candidate coverageVTrajEmbed pooled conditional R@1VTrajEmbed mean end-to-end R@1Same-pool Qwen8B mean end-to-end R@1
Original parsing68.50%68.61%47.00%37.50%
Fallback on parsing failure78.50%67.52%53.00%42.50%

The same-pool comparison provides evidence of ranking performance on automatically generated candidates. Fallback increases mean shared candidate coverage by 10.00 percentage points, an upstream improvement that benefits both embedding models. The results account for the effect of candidate-generation failures on final retrieval.

Fragmentation diagnostic. Separately, we compute the union of candidate support under the same geometric criterion for 200 queries from 21 source videos using an independent supplementary candidate protocol. Target support is distributed across multiple trajectories for 13/200 and 19/200 queries under the original and fallback pipelines, respectively. This diagnostic retains parsing failures and empty candidate sets, and fallback applies only to parsing failures. It neither changes the retrieval inputs nor serves as a failure decomposition of the new evaluation in the table above.

Geometric supportOriginal query parsingParsing-failure fallback
At least one qualifying individual trajectory97/200117/200
No individual trajectory qualifies, but the union of support across trajectories meets the criterion13/200, 6.50%19/200, 9.50%
The union across trajectories still provides only partial support32/20035/200
No frame-wise geometric support (including parsing failures)58/20029/200
GT-assisted upper bound on coverage from the union of support110/200, 55.00%136/200, 68.00%

The union across trajectories is only a GT-assisted geometric upper bound. It is not an achieved gain from trajectory stitching and cannot certify identity consistency across fragments. We have not independently measured the naturally occurring identity-switch rate and retain this as an evaluation limitation.

AuthorsOpenReview ↗

Rebuttal(part 2/5)

W2/Q2: Query Selection Bias, Annotation Agreement, and Benchmark Auditing

Response: To address the representativeness of model-generated queries, we have added an evaluation using the official human-written referring expressions from Ref-DAVIS17. Under the open-vocabulary noun strategy, the main-paper model achieves an end-to-end R@1 of 50.82% across all 61 queries, exceeding Qwen8B on the same candidate pools by 4.92 percentage points. This provides external evidence on expressions that were not model-generated.

The evaluation uses the official 30-video validation set, taking only the annotator1 full-video expression for each object, for a total of 61 queries: 25 person queries and 36 non-person queries. The main-paper model is not adapted on Ref-DAVIS17. Queries and strategies are fixed before inference, and all queries are retained without filtering by candidate availability.

The three candidate prompting strategies use the fixed person prompt, the complete original expression, and open-vocabulary nouns extracted by Qwen3-0.6B. Generation uses no GT boxes, time windows, or category labels. Within each strategy, both embedding models share the candidate pool and relevance judgments and use the complete original query; different strategies may produce different candidate pools. Geometric relevance follows the rule in W1.

Ref-DAVIS17: all 61 human-written expressionsCandidate coverageVTrajEmbed end-to-end R@1Same-pool Qwen8B end-to-end R@1
Fixed person20/61 (32.79%)16/61 (26.23%)14/61 (22.95%)
Complete expression32/61 (52.46%)28/61 (45.90%)26/61 (42.62%)
Open-vocabulary nouns (prespecified primary strategy)34/61 (55.74%)31/61 (50.82%)28/61 (45.90%)

Among the 36 non-person queries, the open-vocabulary noun strategy achieves an end-to-end R@1 of 63.89%, compared with 55.56% for Qwen8B on the same candidate pools, a difference of 8.33 percentage points. These results add evidence on human-written expressions and non-person targets. However, the unique-target admission criterion still limits the query scope of the original benchmark. External evaluation cannot eliminate this selection effect or replace annotation-agreement and acceptance/rejection audits of Text2Traj-Bench.

AuthorsOpenReview ↗

Rebuttal(part 3/5)

W3/Q3: Stronger Object-Centric Input Baselines and Training Fairness

Response: To address comparability in input presentation and task training, we have added target-trajectory crop inputs and a comparison with matched training budgets on the Text2Traj-Bench training set used in the main paper. VTrajEmbed achieves a mean ranking R@1 of 69.56%, exceeding the best Qwen8B variant tested here, the crop baseline without fine-tuning at 61.86%, by 7.70 percentage points. It exceeds the crop baseline trained under the matched scheme, at 60.82%, by 8.74 percentage points.

Both task-adapted models follow a prespecified training scheme, matching the training data, candidate supervision, GT perturbations, and the following training budget: at most 4 trajectories and 1 geometric negative per group; a contrastive batch size of 2 groups per GPU and an effective batch of 32 groups across 16 GPUs; 3 training epochs; a learning rate of 5e-5 with a linear schedule; and loss weights of 0.1 for the within-group objective and 1.0 for the bidirectional objective. The physical batch configuration in this new comparison differs from the original main-paper experiments, which used 4 groups per GPU across 8 GPUs, while the effective batch remains 32 groups. The crop baseline freezes the visual backbone and trains LoRA adapters on the query and value (q/v) projections in the last 8 layers (r=8, alpha=16, dropout=0.05) together with the retrieval temperature. Both models use the final epoch-3 checkpoint, without checkpoint selection based on test performance. This comparison evaluates one adaptation scheme under matched training budgets; it does not imply optimal hyperparameters for each model or equal actual computational costs.

Cropping uses candidate trajectory boxes, with at most 32 observations sampled uniformly across the full timeline and a per-frame limit of 200704 pixels, without GT boxes. The crop baseline without fine-tuning uses the same cropping, padding, and preprocessing.

All methods are evaluated repeatedly on the same 200 queries from 21 source videos in HC-STVG2, sharing the candidate pool and target-covered subset within each run, with candidate coverage of 68.50%. R@1 in the table below is first calculated on the shared covered subset in each run and then averaged across runs. Repeated evaluations do not increase the number of independent queries. Target availability follows W1; full-pipeline results are reported in W1.

Model and inputTask adaptationR@1 (mean across repeated evaluations)
Qwen8B, full frames with boxes overlaidNo task training54.64%
Qwen8B, target-trajectory cropsNo task training61.86%
Qwen8B, target-trajectory cropsRetrained on Text2Traj-Bench60.82%
VTrajEmbed, Slow/Fast trajectory inputsRetrained on Text2Traj-Bench69.56%

The crop baseline decreases from 61.86% to 60.82% after this task adaptation, a drop of 1.04 percentage points, showing that the prespecified adaptation scheme did not improve this baseline. We therefore use the crop version without fine-tuning, at 61.86%, as the strongest Qwen8B baseline tested here, while also reporting the matched-budget training results: VTrajEmbed is higher by 7.70 and 8.74 percentage points, respectively. Both are differences between means in this evaluation and support ranking comparisons within the tested setting. We do not claim statistical significance or a consistent advantage across all scenarios.

Reviewer_vjVKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a fine-grained retrieval problem that retrieves between text and video object trajectories. It proposes VTrajEmbed, given a trajectory, combines appearance, motion, and context to encode it into an embedding, and then uses standard cosine similarity for retrieval. It proposes a Slow-Fast encoder that combines low-resolution and high-resolution encoding. It introduces Mask-Guided Contrastive Learning for training. The authors also introduce Text2Traj-Bench for evaluation. Evaluation reports substantial improvements over Qwen3E-8B in text-to-trajectory and trajectory-to-text retrieval.

Critical strengths
  1. This paper focuses on fine-grained retrieval between text and video object trajectories, which is noval.
  2. The Slow-Fast architecture is well designed for this task, and the experiments show that it is effective.
  3. The paper also introduces Text2Traj-Bench, a new benchmark for evaluating object-level video-text alignment, which is a contribution for this realm.
Critical weaknesses
  1. The experimental section provides an ablation on the number of Fast observations in the Fast branch, but there is no similar ablation on the number of Slow observations in the Slow branch.
  2. From the architecture diagram, the ObjEmbed produced by the Projector in the Fast branch is fed into the Obj-Cond Cross-Attn in the Slow branch, but there is no information flowing from the Slow branch back to the Fast branch. Since the Slow branch can provide environmental context, it is unclear whether incorporating Slow-branch information into the Fast branch could further improve performance.
  3. I think the motion of the target object cannot be strictly defined. The displacement of an object in a video may come from two sources: the motion of the object itself and camera motion. The paper does not clearly explain how it handles the cases where the camera moves while the object is static, or where both the camera and the object move.
  4. During dataset construction, a fixed set of category prompts is used for the frozen SAM 3, which may limit generalization.
Questions for authors
  1. Why is there no ablation on the number of Slow observations in the Slow branch?
  2. Have the authors considered feeding information from the Slow branch back into the Fast branch? Could this further improve the model performance?
  3. How does the method handle the cases where the camera moves while the object is static, and where both the camera and the object move?
  4. How does the use of a fixed set of category prompts for the frozen SAM 3 affect the generalization and out of distribution (OOD) performance of the model?

If the authors can address the issues I raised in the weaknesses and questions, I would consider increasing my score.

9 comments
AuthorsOpenReview ↗

Rebuttal(part 1/5)

Thank you for recognizing the value of object-level trajectory retrieval, the Slow-Fast architecture, and Text2Traj-Bench, and for your four specific suggestions. In response, we have added inference-time interventions on the Slow observation budget, comparisons of fusion locations using separately retrained architectures, and external evaluations with motion-based stratification and open-vocabulary prompts. We have also clarified the scope and limitations of the method. We hope these additions address your main concerns, and welcome any remaining issues that affect your overall assessment so that we can clarify them during the discussion.

W1/Q1: Ablation on the Number of Slow Observations

Response: To ablate the number of Slow observations, we conducted inference-time interventions with budgets of 1, 2, 3, and 4, as well as budget 4 without pruning, using a 1000-query ablation evaluation subset of Text2Traj-Bench. We fixed the weights of the model trained on Text2Traj-Bench and reported in the paper, changing only the Slow branch budget and pruning configuration at inference time, without retraining. Budget 4 with semantic pruning achieved the highest R@1 and mAP among the tested settings (69.50% / 83.34%), supporting its use as the default inference configuration for this model.

We follow the paper's text-to-trajectory retrieval (T2Traj) protocol. All settings share the same fixed set of 1000 evaluation queries and query-independent candidate trajectory pools, with R@1 and mAP computed using human-verified relevance labels. The Fast observation budget remains 32, and all other configurations are unchanged, isolating the trained model's dependence on the Slow context budget and semantic pruning at inference time. The default result of 69.50% / 83.34% in this table is obtained on the above 1000-query ablation subset, whereas the paper's 69.67% / 83.72% is obtained on the main Text2Traj-Bench test set. These evaluations use the same dataset but have different scopes and should be interpreted separately.

The "budget" limits the number of preselected Slow frames; the "mean number of retained frames" is the average passed to the high-resolution encoder after semantic pruning:

Slow preselection budgetMean number of frames retained after pruningR@1 / mAPEncoding latency per trajectory (P50 / P95, ms)
11.0065.00% / 80.72%721.0 / 1118.9
21.6667.50% / 82.96%737.2 / 1154.6
32.2568.00% / 83.25%760.3 / 1231.5
42.6769.50% / 83.34%791.1 / 1252.2
4, without pruning-66.50% / 82.23%759.2 / 1202.8

Budget 4 achieves the highest retrieval accuracy in this budget comparison. Relative to budget 1, budget 4 increases R@1 from 65.00% to 69.50% (+4.50 percentage points) and mAP from 80.72% to 83.34% (+2.62 percentage points). Relative to budget 3, it further improves R@1 by 1.50 percentage points and mAP by 0.09 percentage points. With model weights and the Fast observation budget fixed, budget 4 yields the highest values on both retrieval metrics, supporting the use of additional Slow observations to supplement high-resolution context that varies along the trajectory during inference with this model.

The benefit of semantic pruning is reflected in retrieval accuracy after context selection. With the same preselection budget of 4, pruning retains only 2.67 frames on average and achieves an R@1 / mAP of 69.50% / 83.34%. Retaining all preselected frames yields 66.50% / 82.23%, lower by 3.00 and 1.11 percentage points, respectively. Thus, adding unfiltered context frames at inference time does not necessarily improve retrieval for the same trained model, supporting the current semantic pruning configuration. Meanwhile, the pruned version has a P50 / P95 latency of 791.1 / 1252.2 ms, higher than the 759.2 / 1202.8 ms of the unpruned version. This table therefore supports an accuracy benefit, but not an actual encoding speedup: reducing the number of high-resolution frames does not guarantee that the savings outweigh the additional overhead of frame selection and related operations.

The accuracy benefit of budget 4 with semantic pruning comes with a measurable encoding cost. Its P50 encoding latency is 791.1 ms, an increase of 70.1 ms (approximately 9.7%) over budget 1 and 30.8 ms (approximately 4.1%) over budget 3. Taken together, the inference-budget intervention and the pruning comparison at a matched budget show that this combination achieves the best retrieval accuracy among the tested settings, supporting its use as the default inference configuration when retrieval accuracy is the priority for this model. We will add this table to the revision, distinguish preselection budgets, retained frame counts, and measured latency, and clarify that this is an inference-time intervention with fixed model weights.

AuthorsOpenReview ↗

Rebuttal(part 2/5)

W2/Q2: Is Slow Information Fed Back to the Fast Branch?

Response: The current architecture already includes a fusion path from Slow context to Fast temporal-block representations, after Fast temporal-block aggregation and before language-model readout; it does not modify the per-frame Fast region-of-interest (ROI) features. Following your suggestion to examine whether earlier feedback is beneficial, we additionally compared two architectures that inject context before and after aggregation. Their R@1 values on the present 1000 queries are 67.50% and 69.50%, respectively, providing no observed additional benefit from earlier injection.

Specifically, the "Slow-Fast Trajectory Encoding" paragraph and fusion equation in the current revision explicitly describe this information path: target ROI features query high-resolution scene features, the Slow resampler aggregates the resulting context, and a gated residual connection injects it into the Fast temporal-block representations before language-model readout:

zi=LN⁡(fi+αWsc).z_i=\operatorname{LN}(f_i+\alpha W_s c).

Here, fif_i is the feature after Fast temporal-block aggregation, and cc is the scene context aggregated by the Slow branch. This connection transforms existing Fast block features into the fused ziz_i; it does not feed context back into per-frame ROI extraction. Your comment on the illustration indicates that this path needs to be presented more clearly. We will explicitly mark the gated residual connection from Slow context to Fast temporal-block representations in Figure 1.

To directly test whether earlier feedback is beneficial, we separately retrained both architectures using the Text2Traj-Bench training data from the paper. The default injects Slow context after Fast temporal-block aggregation, while the alternative moves the same context residual to before Fast observation aggregation. Both architectures are evaluated on a 1000-query ablation evaluation subset of Text2Traj-Bench. The default result here comes from a retrained model, whereas Q1 fixes the trained model reported in the paper for inference-time interventions. Both tables use 1000 evaluation queries, but the models are obtained differently; their default results should be interpreted with this distinction in mind. The results are as follows:

Separately trained architectureHitsR@1Difference from default (percentage points)
Default: inject Slow context after Fast aggregation695/100069.50%Reference
Earlier: inject Slow context before Fast aggregation675/100067.50%−2.00

The default architecture correctly retrieves the target for 20 more queries than earlier injection, with an R@1 higher by 2.00 percentage points. This small difference in point estimates only indicates that the present comparison does not show an additional benefit from earlier injection. It does not establish a statistically significant advantage for the default fusion location or rule out the effectiveness of other designs that introduce feedback earlier. We therefore retain the current post-aggregation fusion for now.

AuthorsOpenReview ↗

Rebuttal(part 3/5)

W3/Q3: Different Combinations of Camera and Target Motion

Response: Motion variables are defined in the image plane; the model does not explicitly estimate or separate camera ego-motion or recover world-coordinate target velocity. We evaluate the model trained on Text2Traj-Bench in the main paper on fixed MeViS pools. VTrajEmbed exceeds Qwen8B when both camera and target move but remains weaker with a moving camera and low-motion target.

Camera motion can displace a stationary target's box; joint motion combines both effects. VTrajEmbed jointly encodes box positions, displacements, scale changes, and temporally ordered appearance, adds Slow scene context, and aligns the result with query actions. Box displacement alone does not identify true target motion; camera robustness requires grouped evaluation.

The diagnostic uses 250 queries from 163 videos, 50 per group. All targets are human-verified in the fixed pools; both embedding models share candidates and relevance judgments. R@1 measures within-pool ranking on equally sampled groups. Grouping uses background motion and residual target motion relative to background transformations, for evaluation, not as model inputs. Low target motion does not certify world-coordinate stationarity. The model undergoing additional training is not used.

Fixed MeViS candidate pools: motion groupQueries / videosVTrajEmbed R@1 (%)Qwen8B R@1 (%)Uniform-random baseline R@1 (%)
Moving camera, moving target50 / 5068.0050.0022.59
Low-motion camera, moving target50 / 5068.0054.0022.72
Moving camera, low-motion target50 / 3046.0050.0022.40
Low-motion camera, low-motion target50 / 3266.0046.0019.44
Cannot be reliably determined50 / 5034.0040.0036.05
All queries (aggregated by query)250 / 16356.4048.0024.64

Overall R@1 is 56.40% versus 48.00% (+8.40 percentage points), with higher point estimates in three of four determinate groups. Both groups with moving targets reach 68.00%, exceeding Qwen8B by 18.00 and 14.00 percentage points. Equal R@1 does not establish elimination of camera effects: group content and difficulty differ.

Qwen8B is more balanced across groups: 40.00%–54.00%, versus VTrajEmbed's 34.00%–68.00% (ranges of 14.00 and 34.00 percentage points). VTrajEmbed's higher average coexists with difficult groups; these results cannot establish training-data balance without corresponding statistics.

With a moving camera and low-motion target, R@1 is 46.00%, above random (22.40%) but below Qwen8B (50.00%); the indeterminate group scores 34.00%, below Qwen8B (40.00%) and random (36.05%). These remain priorities; we claim neither camera invariance nor consistent superiority across motion combinations.

We remove only additional displacement-magnitude and scale-change inputs at inference time, retaining box positions, time, appearance, and trajectory sequences:

Same fixed candidate pools: inference-time intervention groupDefault R@1 (%)R@1 without dynamic geometry (%)Default − without dynamic geometry (percentage points)
Moving camera, moving target68.0068.00+0.00
Low-motion camera, moving target68.0068.00+0.00
Moving camera, low-motion target46.0044.00+2.00
Low-motion camera, low-motion target66.0064.00+2.00
Cannot be reliably determined34.0034.00+0.00
All queries (aggregated by query)56.4055.60+0.80

Removing these inputs leaves both groups with moving targets at 68.00%; default R@1 is higher by 2.00 percentage points in each determinate group with low target motion and +0.80 overall. This tests inference-time dependence on additional variables; other motion cues remain, so the full advantage cannot be attributed to them.

Additional diagnostics and subsequent validation. Preliminary Text2Traj-Bench training statistics show 85.65% in the two groups with moving targets, versus 0.53% / 2.35% in the difficult groups, respectively. This motivates wider data coverage but does not establish imbalance as the cause of lower performance.

Motion groupProportion of the current training set (%)
Moving camera, moving target57.18
Low-motion camera, moving target28.47
Moving camera, low-motion target0.53
Low-motion camera, low-motion target11.47
Cannot be reliably determined2.35
Total100.00

Annotation and training continue. Later in rebuttal, we will report group proportions after data expansion and compare each group's gains or regressions before and after training with queries, candidate pools, and grouping fixed, testing improvement on difficult groups and preservation on stronger groups.

Reviewer_UasvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Standard multimodal video embedding models compress an entire video into a single global representation, entangling multiple objects and spatiotemporal dynamics. Consequently, while they perform well on global video-text retrieval, they lack the fine-grained granularity necessary for object-level grounding through retrieval. This paper introduces VTrajEmbed, an embedding framework that uses object trajectories as the basic unit of alignment. The proposal-and-ranking pipeline consists of the following components:

  1. Slow-Fast Trajectory Encoding: The Fast branch densely samples low-resolution frames to capture temporal progression, object appearance, and geometry via bounding box encodings and temporal block attention resampling. The Slow branch sparsely selects high-resolution frames using semantic pruning to capture object-conditioned scene context via cross-attention and gated fusion. A shared language model backbone (Qwen3-VL-4B) encodes the trajectory and text separately into a unified embedding space.
  2. Mask-Guided Contrastive Learning: A training objective combining a within-video multi-positive loss to discriminate target objects from background/competing candidates, and a batch-level bidirectional loss under a query-trajectory relation mask.

This paper also collect Text2Traj-Bench, a model-assisted, human-verified benchmark featuring fixed query-independent candidate pools and verified binary relations to evaluate object-centric retrieval settings.

On Text2Traj-Bench, VTrajEmbed-5B achieves 69.67% R@1 on Text-to-Trajectory retrieval (T2Traj) and 66.34% R@1 on Trajectory-to-Text retrieval (Traj2T), outperforming Qwen3-VL-Embedding-8B by 15.48 and 18.05 percentage points, respectively. Furthermore, after task-specific adaptation, VTrajEmbed's trajectory representations transfer effectively to standard global video retrieval, averaging 49.48% R@1 across five datasets versus 42.86% for Qwen3-VL-Embedding-8B.

Critical strengths
  1. This manuscript is well-written. The task and the proposed solution is new and novel.
  2. The proposed method achieves significant gains over the baseline on fine-grained object retrieval, and it achieves the state-of-the-arts performance on Text2Traj-Bench.
  3. The proposed method can be transferred to global video retrieval task. Table 2 shows that VTrajEmbed's trajectory-derived embeddings can be adapted to global video retrieval, achieving higher average performance (49.48% R@1) across 5 benchmark datasets than its 8B baseline Qwen3E-8B (42.86% R@1), demonstrating that object-level specialization does not destroy global context representation capability.
Critical weaknesses
  1. The main trajectory-level evaluation protocol on Text2Traj-Bench freezes the candidate poo generated by SAM 3 prior to query evaluation (Section 3.3, Section 4.1). While this isolates ranking ability from tracking quality, it does not evaluate the complete end-to-end proposal-and-ranking pipeline under open-world query parsing and candidate tracking. This is a design/scope limitation acknowledged by the authors. To address this, the authors should add end-to-end metrics on a validation subset as outlined in their future work section.
  2. The proposed framework relies on a complex stack of frozen and trainable modules: Qwen3-0.6B parser, SAM 3 tracker, SigLIP2-Large vision tower, ObjEmbed-4B ROI projector, and Qwen3-VL-4B language backbone. While Appendix F.5 analyzes visual-encoder cost proxies, the paper lacks detailed empirical wall-clock latency, throughput, and GPU memory overhead comparisons during inference relative to baseline single-vector encoders. The authors should provide end-to-end runtime, memory footprint, and latency metrics in Appendix B or Section 4 to give a complete practical computational profile of VTrajEmbed versus standard global embedding models.
  3. Minor: In Appendix D.2, line 1079 contains an unpopulated reference: "following Appendix ??."
Questions for authors

Please see the weakness part.

3 comments
AuthorsOpenReview ↗

Rebuttal(part 1/2)

Thank you for recognizing the novelty of object-trajectory-level retrieval, our fine-grained retrieval performance, and the potential for transfer to global video retrieval. We also appreciate your specific suggestions on evaluating the complete pipeline, measuring practical inference costs, and correcting the appendix reference. In response, we have added an end-to-end evaluation that includes parsing and tracking failures, together with measurements of latency, throughput, and GPU memory that account for the runtime costs of frozen modules, and have corrected the appendix reference. We hope these additions address your main concerns. We also welcome any remaining issues that affect your overall assessment so that we can clarify them further during the discussion.

W1/Q1: End-to-End Evaluation Including Parsing, Tracking, and Retrieval

Response: To address your concern that a fixed candidate pool does not cover the complete pipeline, we have added an evaluation of automatic candidate generation and retrieval on HC-STVG2. Results pooled across three seed runs show that VTrajEmbed achieves end-to-end R@1 of 47.00% and 53.00% with the original parsing pipeline and the parsing-failure fallback pipeline, respectively, exceeding Qwen8B on the same candidate pools by 9.50 and 10.50 percentage points. These results provide additional evidence on the complete pipeline beyond the fixed-pool evaluation in the main paper.

We selected 200 queries from 21 source videos in HC-STVG2 that were not included in Text2Traj-Bench, with source-level separation between training and evaluation. VTrajEmbed uses the model trained on Text2Traj-Bench reported in the main paper; its parameters remain fixed during this external evaluation, with no HC-STVG2-specific adaptation. Each query is parsed by Qwen3-0.6B, and SAM3.1 automatically generates candidate trajectories over the full video timeline, without ground-truth (GT) assistance or manual curation. Queries are not filtered according to whether the target is proposed. The original pipeline uses the parsed prompts, whereas the fallback pipeline uses the person prompt, appropriate to this setting with known human targets, only when parsing fails. Within each seed run, both embedding models share the same candidate pool and use the complete original query for retrieval.

A target is considered available if at least one candidate trajectory achieves an intersection over union (IoU) of at least 0.4 on at least 80% of the frames in which the GT target is visible. Each of the three seed runs evaluates the same 200 queries. Coverage, conditional R@1, and end-to-end R@1 are calculated after pooling the counts, with parsing, tracking, and retrieval failures retained in the denominator. This yields 600 evaluation attempts, while the number of independent queries remains 200. The results are as follows:

Candidate-generation pipelineEmbedding modelCandidate coverageConditional R@1End-to-end R@1 (hits / 600 evaluation attempts)
Original parsingQwen3-VL-Embedding-8B68.50%54.74%37.50% (225)
Original parsingVTrajEmbed68.50%68.61%47.00% (282)
Parsing-failure fallbackQwen3-VL-Embedding-8B78.50%54.14%42.50% (255)
Parsing-failure fallbackVTrajEmbed78.50%67.52%53.00% (318)

Results on the same automatically generated candidate pools support retrieval gains. On automatically generated candidates outside the benchmark, without manual curation, VTrajEmbed improves conditional R@1 over Qwen8B by 13.87 percentage points and end-to-end R@1 by 9.50 percentage points in the original pipeline. The corresponding improvements in the fallback pipeline are 13.38 and 10.50 percentage points. These differences do not arise from the embedding models receiving different candidates and support VTrajEmbed's trajectory retrieval capability on the automatically generated candidates in this evaluation.

The parsing-failure fallback improves upstream candidate coverage. The fallback increases shared candidate coverage from 68.50% to 78.50%, benefiting both embedding models; this coverage improvement comes from the upstream prompting strategy. The pooled statistics satisfy “end-to-end R@1 = candidate coverage × conditional R@1,” explicitly accounting for the effect of candidate-generation failures on final retrieval. This evaluation therefore adds direct evidence for the complete pipeline from the original query to final trajectory selection, while maintaining a distinction in scope: fixed candidate pools enable controlled retrieval comparisons, whereas the automatic pipeline evaluates final performance in the presence of upstream failures. The geometric trajectory criterion cannot independently certify identity continuity outside the GT interval.

AuthorsOpenReview ↗

Rebuttal(part 2/2)

W2/Q2: Measured Latency, Throughput, and GPU Memory

Response: To address your concern about practical inference costs, we have added measurements comparing complete requests, queries with cached visual embeddings, and native global encoding. The P50 latency of a complete VTrajEmbed object-retrieval request is 161.67 seconds, compared with 1.55 seconds for the native whole-video single-vector reference. Their outputs differ in granularity, and the complete object-level pipeline additionally includes candidate generation and trajectory retrieval. When no new candidate trajectories are needed and their visual embeddings are already cached, VTrajEmbed's P50 latency for text encoding and retrieval is 27.83 milliseconds. Relative to the Qwen8B pipeline performing the same object-level task, VTrajEmbed reduces complete-request P50 latency by 15.39%, increases throughput by 19.88%, and reduces the sampled peak incremental GPU memory attributable to the task by 9.79%.

On A800 80GB GPUs with BF16, we measured the main-paper VTrajEmbed model trained on Text2Traj-Bench and Qwen8B on the same 30 fixed queries, pooling timings across three seed runs and retaining all request attempts, including failures. Frame budgets are Fast32/Slow4 and up to 32 frames, respectively. Both object-level pipelines share the parser and SAM3.1 configuration: parsing runs on CPU, with SAM video/state offloading to CPU. Complete requests include decoding, parsing, tracking, encoding, and retrieval, including frozen-module runtime, but exclude model loading, warm-up, and cleanup. Methods alternate on the same machine with CUDA synchronization, recording each query's first timed complete request per run.

Cached queries apply only when no new candidates are needed and visual embeddings are available. They perform text encoding and similarity search, with 3 warm-ups and 30 measurements per run; additional candidates require parsing, tracking, and visual encoding. The native global reference uses 3 measurements in an independent process per run. Throughput equals pooled attempts divided by total time. NVML samples GPU memory every 50 ms; the baseline before model loading is subtracted; complete-pipeline memory sums simultaneous usage across two GPUs, while other settings use one. CPU RAM is excluded, and percentiles describe this query set.

Inference settingP50 (seconds)P95 (seconds)Sequential throughput (request attempts/second)Sampled peak incremental GPU memory (GiB)
VTrajEmbed: complete request from raw video161.67751.030.00355829.30
Qwen8B: complete request for the same object-level task191.07887.080.00296832.48
VTrajEmbed: trajectory visual embeddings already available0.027830.03302331.9714.73
Qwen8B: trajectory visual embeddings already available0.029620.0339132.1516.49
Qwen8B: native whole-video single-vector reference1.554.230.5316.49

Native global encoding provides a cost reference at a different output granularity. It costs substantially less from raw video but omits object-candidate generation and trajectory retrieval. The table reports both complete object-retrieval costs and query costs when visual representations are reusable.

Efficiency gains are limited to the measured object-level configuration. VTrajEmbed also lowers complete-request P95 by 15.34%. The smaller parameter count helps reduce GPU memory for model weights at matched precision, while Slow-Fast controls visual computation through dense low-resolution and sparse high-resolution encoding. Complete-request costs also depend on parsing, tracking, and candidate count; they cannot be inferred from parameter count alone, and measured differences cannot be attributed entirely to encoder design. Sampled incremental GPU memory does not represent total system memory, including CPU offloading.

Cached visual embeddings primarily improve latency percentiles and GPU memory. VTrajEmbed lowers P50 / P95 by 6.04% / 2.62% and sampled peak incremental GPU memory by 10.67%. Throughput is 31.97 versus 32.15 requests/second, 0.56% lower, showing no throughput advantage. Lower percentiles can coexist with slightly higher mean latency because they describe different aspects of the distribution.

W3/Q3: The Cross-Reference in Appendix D.2

Response: Thank you for pointing out this cross-reference error. We will fix the reference in Appendix D.2 and confirm that the recompiled PDF displays the reference number correctly, with no remaining “??”. We have also reviewed the cross-references and related wording throughout the paper.

If these additional experiments and clarifications sufficiently address your main concerns, we hope you will reconsider your overall assessment in light of the new evidence. We also welcome further discussion of any remaining issues that affect your judgment.

AuthorsOpenReview ↗

Please check whether your concerns are addressed.

Thanks for your review comments, which are valuable to improve the quality of this paper. We provided the rebuttal to those comments of the paper days ago. We kindly request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your timely engagement in the review process and your help in improving the quality of this paper.

Regards from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.