- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes VTrajEmbed, a multimodal embedding framework that aligns natural-language descriptions with video object trajectories. It introduces a Slow-Fast trajectory encoder, a mask-guided contrastive learning objective, and the Text2Traj-Bench benchmark. The method achieves substantial improvements in trajectory-level retrieval.
Critical strengths
-
The formulation of object-level video-text alignment through trajectory embeddings is meaningful and offers a useful alternative to conventional global video representations.
-
The proposed benchmark incorporates human-verified trajectory-text relations, and the experimental results demonstrate substantial improvements over several multimodal embedding baselines.
Critical weaknesses
-
The main experiments rely on pre-generated and curated trajectory candidates. The proposed query-conditioned pipeline is not evaluated end to end, leaving its robustness to missed detections, fragmented trajectories, and identity switches unclear.
-
Text2Traj-Bench uses model-generated queries and admits only queries with a uniquely identifiable target in the curated candidate pool. This may introduce selection bias and does not adequately reflect ambiguous or naturally occurring user queries. Quantitative annotation agreement and benchmark audit statistics would further strengthen its reliability.
-
Existing embedding models receive videos with overlaid bounding boxes, whereas VTrajEmbed employs a specialized object projector and geometry-aware trajectory representations. Stronger baselines with comparable object-centric inputs and task-specific training would help isolate the proposed method's contributions.
-
Although the Slow-Fast branches are ablated, the individual contributions of appearance, geometric motion cues, temporal ordering, and semantic frame pruning remain insufficiently investigated. More targeted experiments would substantiate the claimed benefits of spatiotemporal modeling.
-
Although VTrajEmbed outperforms Qwen3E-8B on average, its global retrieval performance remains below WeMM-4B (49.48% vs. 61.07% average R@1). Moreover, the impact of benchmark-specific adaptation and the comparability of training protocols should be clarified.
Questions for authors
Please refer to Weaknesses.
6 comments
Rebuttal(part 2/5)
W2/Q2: Query Selection Bias, Annotation Agreement, and Benchmark Auditing
Response: To address the representativeness of model-generated queries, we have added an evaluation using the official human-written referring expressions from Ref-DAVIS17. Under the open-vocabulary noun strategy, the main-paper model achieves an end-to-end R@1 of 50.82% across all 61 queries, exceeding Qwen8B on the same candidate pools by 4.92 percentage points. This provides external evidence on expressions that were not model-generated.
The evaluation uses the official 30-video validation set, taking only the annotator1 full-video expression for each object, for a total of 61 queries: 25 person queries and 36 non-person queries. The main-paper model is not adapted on Ref-DAVIS17. Queries and strategies are fixed before inference, and all queries are retained without filtering by candidate availability.
The three candidate prompting strategies use the fixed person prompt, the complete original expression, and open-vocabulary nouns extracted by Qwen3-0.6B. Generation uses no GT boxes, time windows, or category labels. Within each strategy, both embedding models share the candidate pool and relevance judgments and use the complete original query; different strategies may produce different candidate pools. Geometric relevance follows the rule in W1.
| Ref-DAVIS17: all 61 human-written expressions | Candidate coverage | VTrajEmbed end-to-end R@1 | Same-pool Qwen8B end-to-end R@1 |
|---|---|---|---|
Fixed person | 20/61 (32.79%) | 16/61 (26.23%) | 14/61 (22.95%) |
| Complete expression | 32/61 (52.46%) | 28/61 (45.90%) | 26/61 (42.62%) |
| Open-vocabulary nouns (prespecified primary strategy) | 34/61 (55.74%) | 31/61 (50.82%) | 28/61 (45.90%) |
Among the 36 non-person queries, the open-vocabulary noun strategy achieves an end-to-end R@1 of 63.89%, compared with 55.56% for Qwen8B on the same candidate pools, a difference of 8.33 percentage points. These results add evidence on human-written expressions and non-person targets. However, the unique-target admission criterion still limits the query scope of the original benchmark. External evaluation cannot eliminate this selection effect or replace annotation-agreement and acceptance/rejection audits of Text2Traj-Bench.
Rebuttal(part 3/5)
W3/Q3: Stronger Object-Centric Input Baselines and Training Fairness
Response: To address comparability in input presentation and task training, we have added target-trajectory crop inputs and a comparison with matched training budgets on the Text2Traj-Bench training set used in the main paper. VTrajEmbed achieves a mean ranking R@1 of 69.56%, exceeding the best Qwen8B variant tested here, the crop baseline without fine-tuning at 61.86%, by 7.70 percentage points. It exceeds the crop baseline trained under the matched scheme, at 60.82%, by 8.74 percentage points.
Both task-adapted models follow a prespecified training scheme, matching the training data, candidate supervision, GT perturbations, and the following training budget: at most 4 trajectories and 1 geometric negative per group; a contrastive batch size of 2 groups per GPU and an effective batch of 32 groups across 16 GPUs; 3 training epochs; a learning rate of 5e-5 with a linear schedule; and loss weights of 0.1 for the within-group objective and 1.0 for the bidirectional objective. The physical batch configuration in this new comparison differs from the original main-paper experiments, which used 4 groups per GPU across 8 GPUs, while the effective batch remains 32 groups. The crop baseline freezes the visual backbone and trains LoRA adapters on the query and value (q/v) projections in the last 8 layers (r=8, alpha=16, dropout=0.05) together with the retrieval temperature. Both models use the final epoch-3 checkpoint, without checkpoint selection based on test performance. This comparison evaluates one adaptation scheme under matched training budgets; it does not imply optimal hyperparameters for each model or equal actual computational costs.
Cropping uses candidate trajectory boxes, with at most 32 observations sampled uniformly across the full timeline and a per-frame limit of 200704 pixels, without GT boxes. The crop baseline without fine-tuning uses the same cropping, padding, and preprocessing.
All methods are evaluated repeatedly on the same 200 queries from 21 source videos in HC-STVG2, sharing the candidate pool and target-covered subset within each run, with candidate coverage of 68.50%. R@1 in the table below is first calculated on the shared covered subset in each run and then averaged across runs. Repeated evaluations do not increase the number of independent queries. Target availability follows W1; full-pipeline results are reported in W1.
| Model and input | Task adaptation | R@1 (mean across repeated evaluations) |
|---|---|---|
| Qwen8B, full frames with boxes overlaid | No task training | 54.64% |
| Qwen8B, target-trajectory crops | No task training | 61.86% |
| Qwen8B, target-trajectory crops | Retrained on Text2Traj-Bench | 60.82% |
| VTrajEmbed, Slow/Fast trajectory inputs | Retrained on Text2Traj-Bench | 69.56% |
The crop baseline decreases from 61.86% to 60.82% after this task adaptation, a drop of 1.04 percentage points, showing that the prespecified adaptation scheme did not improve this baseline. We therefore use the crop version without fine-tuning, at 61.86%, as the strongest Qwen8B baseline tested here, while also reporting the matched-budget training results: VTrajEmbed is higher by 7.70 and 8.74 percentage points, respectively. Both are differences between means in this evaluation and support ranking comparisons within the tested setting. We do not claim statistical significance or a consistent advantage across all scenarios.
Rebuttal(part 1/5)
Thank you for recognizing the value of trajectory-level text–video alignment and the benchmark. In response to your five concerns, we have added a full-pipeline evaluation and tracking-failure diagnostics (W1), an external evaluation using human-written referring expressions (W2), object-centric baselines with matched training budgets (W3), and component ablations (W4). We also clarify the auxiliary role of the global retrieval evaluation (W5). Below, we report the new evidence and its scope for each concern, and welcome any key issues you identify that still affect your overall assessment.
W1/Q1: End-to-End Retrieval and the Effects of Missed Detections, Trajectory Fragmentation, and Identity Switches
Response: To address the limitation that a fixed candidate pool does not evaluate the full pipeline, we have added an end-to-end evaluation with automatically generated candidates and a fragmentation diagnostic. The model from the main paper achieves mean end-to-end R@1 scores of 47.00% and 53.00% under the original parsing and parsing-failure fallback pipelines, respectively, exceeding Qwen8B on the same candidate pools by 9.50 and 10.50 percentage points.
Full-pipeline evaluation. We evaluate the model trained on Text2Traj-Bench in the main paper on 200 queries from 21 source videos in HC-STVG2 that were not included in the benchmark. Training and evaluation are separated by source, with no evaluation-specific adaptation. Qwen3-0.6B and SAM3.1 automatically generate candidates over the full video timeline from each query, without ground truth (GT) assistance or manual curation, and queries are not filtered according to target availability. Parsing, tracking, and ranking failures remain in the denominator. The
personfallback is used only when parsing fails. Within each seed, the two embedding models share the candidate pool and both use the complete original query.A target is considered available if a candidate achieves IoU≥0.4 on at least 80% of the GT-visible frames. All three seeds evaluate the same 200 queries: candidate coverage and end-to-end R@1 are reported as arithmetic means across runs, while conditional R@1 is calculated by dividing pooled hits by pooled covered queries. The number of independent queries remains 200.
The same-pool comparison provides evidence of ranking performance on automatically generated candidates. Fallback increases mean shared candidate coverage by 10.00 percentage points, an upstream improvement that benefits both embedding models. The results account for the effect of candidate-generation failures on final retrieval.
Fragmentation diagnostic. Separately, we compute the union of candidate support under the same geometric criterion for 200 queries from 21 source videos using an independent supplementary candidate protocol. Target support is distributed across multiple trajectories for 13/200 and 19/200 queries under the original and fallback pipelines, respectively. This diagnostic retains parsing failures and empty candidate sets, and fallback applies only to parsing failures. It neither changes the retrieval inputs nor serves as a failure decomposition of the new evaluation in the table above.
The union across trajectories is only a GT-assisted geometric upper bound. It is not an achieved gain from trajectory stitching and cannot certify identity consistency across fragments. We have not independently measured the naturally occurring identity-switch rate and retain this as an evaluation limitation.