- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge that general-purpose multimodal embedding models often struggle to accurately capture vehicle motion information in the context of video retrieval for autonomous driving. The authors note that while traditional supervised fine-tuning (SFT) can improve cross-modal alignment between text and driving videos, relying solely on textual descriptions as a supervisory signal is insufficient for learning fine-grained representations of vehicle motion.
To this end, the paper introduces TraVEL, a two-stage training framework supervised by vehicle trajectories. First, the framework achieves domain adaptation through contrastive learning between videos and "reasoning-trace" descriptions. Subsequently, it leverages ego-trajectories to construct motion-based similarity rewards between videos. These rewards are combined with text-video instance-matching rewards to further optimize the embedding space via Group Relative Policy Optimization (GRPO). Trajectory similarity is computed by transforming lateral displacement and velocity sequences and applying the Dynamic Time Warping (DTW) algorithm.
Critical strengths
- The research motivation is clear, offering a solution to the limitations of existing "multimodal retrieval models" in practical applications.
This paper addresses a key challenge in video retrieval for autonomous driving: existing embedding models often rely too heavily on visual scenes and static contextual information, failing to adequately capture the dynamic behavior of vehicles. By comparing pre-trained models, models subjected to supervised fine-tuning (SFT), and the proposed TraVEL model, the authors highlight the distinction between "instance-level retrieval performance" and "motion-semantic-based retrieval capability." This finding provides a solid theoretical basis for incorporating additional "motion-based supervision signals."
- The design leverages trajectories as a form of supervision during training, offering significant practical utility.
TraVEL utilizes "ego-trajectories" as "privileged supervision," requiring no additional trajectory data or perception outputs during the inference phase. This design preserves the inherent simplicity and scalability of embedding-based retrieval methods while integrating continuous motion information—data that is difficult to capture through traditional text descriptions alone. Compared to approaches that necessitate structured motion data during inference, this method offers a distinct advantage for real-world deployment.
- The experiments encompass a variety of backbone networks, cross-domain testing, and targeted ablation studies.
The authors conducted experiments using models such as InternVideo2, Cosmos-Embed1, and Qwen3-VL-Embedding, evaluating the model's generalization capabilities on the AV2 dataset (which was not used during training or hyperparameter tuning). The "reward ablation" analysis presented in Table 2 is particularly noteworthy: removing the "video-to-video" reward causes motion-based retrieval performance to degrade to the level of an SFT-only model, while removing the "text-to-video" reward leads to a decline in instance-level retrieval capability. This indicates that the two reward mechanisms serve distinct functions: the former assists the model in learning motion structures, while the latter ensures the maintenance of the alignment between language and video.
Critical weaknesses
- The necessity of GRPO and the innovation of the proposed method are not sufficiently justified.
The core performance gains in this paper stem from motion similarity supervision based on ego-trajectories, while GRPO itself is an existing optimization method. In Table 3, the authors compare TraVEL against methods such as "Trajectory Prediction" and "Trajectory Similarity Matching," showing that their proposed training scheme outperforms these specific implementations.
However, the current comparisons are insufficient to clarify whether the performance improvements arise from GRPO’s "intra-group relative advantage optimization," the trajectory similarity supervision, "Hard Negative Mining," or perhaps a more straightforward ranking-based optimization.
It is worth noting that when using the Qwen3-VL-Embedding-2B model, the "Trajectory Similarity Matching" method achieves mAP scores of 32.2 (lateral) and 52.0 (longitudinal), while TraVEL achieves 35.7 and 55.8, respectively. While this improvement is of some value, its magnitude is relatively limited.
- The approach for constructing trajectory similarity is overly simplistic and may fail to fully capture the complex semantics of driving motion.
The paper characterizes ego-motion using sequences of lateral displacement and velocity, employing Dynamic Time Warping (DTW) to compute similarity-based rewards. Although this design is intuitive and easy to implement, it fails to explicitly incorporate factors such as changes in vehicle heading, road curvature, lane topology, or the relative motion of surrounding traffic participants.
-
The scope of the benchmarks is limited, and support for retrieving complex driving scenarios remains insufficient. Although the paper introduces two evaluation benchmarks, the "motion retrieval" task in nuReasoning includes only 7 lateral motion queries and 5 longitudinal motion queries. Due to an insufficient number of positive samples, certain rare categories—such as rapid acceleration or deceleration—are excluded from the mAP (mean Average Precision) calculation. Queries in the AV2 benchmark are primarily constructed based on predefined ego-motion predicates and text templates; they fail to adequately encompass complex natural language descriptions, interactions between traffic participants, or the broader semantic information present in diverse driving scenarios. Furthermore, the model’s performance on temporal query tasks in AV2 remains weak. Taking the Qwen3-VL-Embedding-2B model as an example, TraVEL achieves a temporal mAP of only 12.5, compared to 7.5 for a random baseline. This indicates that relying solely on supervision based on motion similarity is insufficient to address the challenge of understanding the temporal sequencing of multiple events.
-
There is a lack of systematic analysis regarding statistical stability, training efficiency, and key hyperparameters.
Although Table 1 shows that TraVEL improves motion retrieval performance across all tested backbones, the magnitude of this improvement varies significantly between models. For instance, when using InternVideo2, the lateral and longitudinal mAP scores increased by only 0.5 and 1.0 percentage points, respectively. The paper does not provide standard deviations or confidence intervals for results obtained using multiple random seeds, making it difficult to determine whether these marginal performance gains are statistically robust.
The proposed method introduces an additional GRPO training phase, a candidate group selection process, and the calculation of trajectory similarities; however, the paper fails to provide a comprehensive account of the associated training time, GPU memory overhead, or computational costs relative to traditional contrastive learning.
Furthermore, the authors do not conduct systematic ablation studies on key components—such as group size, reward normalization, the weighting of different trajectory components, or the design of the Dynamic Time Warping (DTW) mechanism. Consequently, the specific contribution of each design choice remains unclear.
Questions for authors
Q1. What are the primary advantages of GRPO compared to traditional ranking optimization methods?
Q2. Why does TraVEL yield significantly greater improvements for turning and stopping maneuvers compared to lane changes or subtle lateral movements?
Q3. How sensitive are the experimental results to the key hyperparameters of GRPO?
Q4. How does the training cost of TraVEL compare to SFT and other trajectory-based supervision methods?
Q5. Does the method provide consistent gains across different random seeds and for more challenging motion categories?
4 comments
Diversity of motion queries
The motion queries are the full set of ego-motion decisions annotated in the nuReasoning test split, i.e., its lateral and longitudinal decision vocabularies, rather than a subset we selected, except for three rare longitudinal categories that we omitted on purpose. "Quickly come to a stop", "Slow down quickly" and "Quickly accelerate" have only 10, 8 and 3 relevant videos, so their AP is highly noisy, and averaging them into the mAP would weight a few videos as heavily as categories with hundreds of positives.
The scope of the benchmarks is limited
To evaluate natural-language queries, we add zero-shot moment retrieval on BDD-X, whose queries are free-form, human-written action descriptions with justifications. Please see Common Question 3.
Weak performance on temporal queries
We agree. Please see Common Question 2.
Stability across seeds and challenging categories
Please see Common Question 1. For the more challenging motion categories, please refer to our response on lane-change performance above.
Training cost
With the same training budget on Qwen3-VL-Embedding-2B (table below), TraVEL costs about as much as contrastive fine-tuning, represented by Trajectory Prediction, which only adds a linear head to the contrastive objective, while giving the largest motion gains. Its additional cost comes from selecting the top-k candidates and a forward pass of the frozen reference model, while computing the trajectory rewards is negligible. Its peak memory is 72.0 GB per GPU with 192 videos per GPU, and since its cost per update depends on the candidate-pool and group sizes rather than on the dataset size, it scales with the number of training clips like contrastive fine-tuning.
| Method | GPU-hours (H100) | Δ LAT / Δ LON vs. SFT |
|---|---|---|
| Trajectory as Text | 1.52 | −0.3 / −0.4 |
| Trajectory Prediction | 0.87 | +4.9 / +1.5 |
| Trajectory Similarity Matching | 0.91 | +3.3 / +3.8 |
| TraVEL | 1.00 | +5.3 / +8.3 |
Sensitivity to hyperparameters
Our hyperparameters are chosen empirically, and we analyze their effect on Qwen3-VL-Embedding-2B (table below, change from the default TraVEL configuration). TraVEL is not sensitive to the group size, the reward normalization or the component weights, as each of these variants changes motion mAP by at most about one point. A smaller group slightly lowers both motion axes, while a larger group lowers retrieval. Changing the component weights shifts the two axes in the intended direction, and replacing DTW with a pointwise distance lowers lateral mAP.
| Variant | Δ LAT | Δ LON | Δ R@10 |
|---|---|---|---|
| Group size k = 12 (default 24) | −0.49 ± 0.05 | −0.86 ± 0.32 | −0.44 ± 0.52 |
| Group size k = 48 | +0.15 ± 0.19 | +0.65 ± 0.56 | −1.16 ± 0.62 |
| No clipping, no scaling | +0.06 ± 0.24 | +0.49 ± 0.24 | −0.36 ± 0.04 |
| Path-length DTW normalization | +0.06 ± 0.19 | −0.51 ± 0.14 | −0.22 ± 0.37 |
| w_lat : w_speed = 1 : 1 (default 1 : 2) | +0.24 ± 0.03 | −1.01 ± 0.25 | −0.29 ± 0.39 |
| w_lat : w_speed = 1 : 4 | −0.09 ± 0.10 | +0.60 ± 0.23 | −0.09 ± 0.21 |
| Pointwise distance instead of DTW | −0.67 ± 0.17 | −0.06 ± 0.26 | −0.24 ± 0.54 |
We hope that we clarified these very helpful comments. We would be glad to engage in discussion if there are any other questions or parts that need further clarification.
[a] Learning to Rank using Gradient Descent, ICML 2005.
Thank you for your response. The additional ablation studies and robustness experiments have addressed most of my concerns. However, the weak performance on temporal queries remains an unresolved issue, and the performance across different backbones in the additional robustness experiments is still a concern. Therefore, I will maintain my original score.
Thank you for your positive evaluation and for acknowledging that the additional ablations and robustness experiments address most of your concerns. We would like to briefly clarify the two remaining points.
Regarding temporal queries, we do not claim temporal generalization, as our objective does not explicitly model the ordering between events. Nevertheless, TraVEL also improves temporal queries on the VLM-based backbones in the zero-shot AV2 evaluation (Fig. 4 of the revised paper), while they remain considerably harder than the other query types. We view this as a concrete direction that our benchmark exposes for future work.
Regarding the different backbones, recent multimodal embedding models are increasingly built on VLMs, such as Qwen3-VL-Embedding and WeMM-Embedding, and TraVEL consistently improves both lateral and longitudinal motion retrieval on all three such backbones across two model families. We therefore believe that TraVEL is effective in the direction where video embedding models are heading. Its smaller gains on InternVideo2 and Cosmos-Embed1 are consistent with their comparatively weak aggregation of video tokens on top of frozen vision encoders, as discussed in Common Question 1.
Thank you again for your constructive feedback, which has helped us improve the paper.
We sincerely appreciate your time and effort in reviewing our paper, and thank you for your expertise and constructive comments. We address your comments and questions one by one below, and the corresponding revisions are highlighted in blue in the revised paper. Unless noted otherwise, all numbers are means ± std over three seeds, with both SFT and TraVEL re-trained for every seed.
We isolate each factor on Qwen3-VL-Embedding-2B by changing one factor at a time (table below). Removing the trajectory supervision removes the motion gain. Replacing the top-k candidates of both groups with random ones makes training destructive and collapses instance retrieval, showing the critical role of hard-negative mining. Replacing GRPO with a pairwise ranking loss (RankNet [a]) on the same groups yields similar motion mAP and slightly lower instance retrieval at the same training budget and cost, since both learn to rank the candidates of the same groups by their trajectory rewards. The motion gains therefore come from the trajectory similarity supervision applied to hard negatives, while GRPO offers a slight advantage in preserving instance retrieval at no extra cost.
Lateral displacement and speed already cover the two motion axes described by the motion queries. Adding the heading change, the yaw rate, or a lane-relative offset as an additional reward component brings little benefit (table below, seed 42). Lane changes are harder to retrieve than turns or stops. A turn or a stop produces a large and sustained change in heading or speed, whereas a lane change is a brief lateral transition across a lane boundary. Retrieving it requires the video embedding to closely align the visual evidence, such as the lane markings, with the ego motion. The smaller gain on lane changes is not caused by the class distribution, since lane changes are about as frequent as turns in both the training and test sets. It is not caused by the DTW-based similarity either, since replacing DTW with a pointwise distance or normalizing it by path length barely changes the lane-change AP. Subtle in-lane shifts are even more difficult, as they involve a lateral movement of only about one meter, which can be hard to discern in the front-camera video. We regard both, as well as interactions with surrounding traffic participants, which are beyond our focus on clip-level ego motion, as remaining challenges for future work.