acceptodds
Under review as a conference paper at ICLR 2027

TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

Abstract

Retrieving scenarios of interest from large driving-video collections is essential for data curation, model development, and safety validation. Program-based systems provide explicit control over well-specified events, but commonly depend on auxiliary structured inputs and an additional language-to-program translation step. Multimodal embedding models offer a more direct interface through single-vector text–video retrieval. However, our evaluation across several pretrained embedding models reveals weak performance on driving videos, particularly for motion-centric queries. Standard supervised fine-tuning substantially narrows the domain gap but remains insufficient for capturing fine-grained driving dynamics. Ego trajectories offer a denser description of motion than captions, and their pairwise similarity can serve as graded motion supervision. Based on this insight, we introduce TraVEL (Trajectory-Guided Video Embedding Learning), a trajectory-aware adaptation framework that uses ego trajectories as privileged training supervision. TraVEL applies Group Relative Policy Optimization with complementary rewards: a graded video-to-video reward organizes the embedding space according to ego-motion similarity, while a text-to-video reward preserves language alignment. Trajectories are used only during training, and inference remains single-vector retrieval without ego poses or auxiliary perception outputs. We construct two complementary driving-video retrieval benchmarks derived from nuReasoning and the Argoverse 2 (AV2) Sensor Dataset for in-domain and zero-shot cross-domain evaluation. Across three embedding-model families, TraVEL consistently improves motion retrieval over supervised fine-tuning while maintaining comparable instance-level performance. On Qwen3-VL-Embedding-2B, it improves lateral and longitudinal motion mAP by 7.5 and 10.3 points, respectively. These improvements also transfer to zero-shot evaluation on the AV2 benchmark, demonstrating cross-dataset generalization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
text-to-video retrieval, multimodal embedding models, autonomous driving, representation learning, reinforcement learning fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ssXmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge that general-purpose multimodal embedding models often struggle to accurately capture vehicle motion information in the context of video retrieval for autonomous driving. The authors note that while traditional supervised fine-tuning (SFT) can improve cross-modal alignment between text and driving videos, relying solely on textual descriptions as a supervisory signal is insufficient for learning fine-grained representations of vehicle motion.

To this end, the paper introduces TraVEL, a two-stage training framework supervised by vehicle trajectories. First, the framework achieves domain adaptation through contrastive learning between videos and "reasoning-trace" descriptions. Subsequently, it leverages ego-trajectories to construct motion-based similarity rewards between videos. These rewards are combined with text-video instance-matching rewards to further optimize the embedding space via Group Relative Policy Optimization (GRPO). Trajectory similarity is computed by transforming lateral displacement and velocity sequences and applying the Dynamic Time Warping (DTW) algorithm.

Critical strengths
  1. The research motivation is clear, offering a solution to the limitations of existing "multimodal retrieval models" in practical applications.

This paper addresses a key challenge in video retrieval for autonomous driving: existing embedding models often rely too heavily on visual scenes and static contextual information, failing to adequately capture the dynamic behavior of vehicles. By comparing pre-trained models, models subjected to supervised fine-tuning (SFT), and the proposed TraVEL model, the authors highlight the distinction between "instance-level retrieval performance" and "motion-semantic-based retrieval capability." This finding provides a solid theoretical basis for incorporating additional "motion-based supervision signals."

  1. The design leverages trajectories as a form of supervision during training, offering significant practical utility.

TraVEL utilizes "ego-trajectories" as "privileged supervision," requiring no additional trajectory data or perception outputs during the inference phase. This design preserves the inherent simplicity and scalability of embedding-based retrieval methods while integrating continuous motion information—data that is difficult to capture through traditional text descriptions alone. Compared to approaches that necessitate structured motion data during inference, this method offers a distinct advantage for real-world deployment.

  1. The experiments encompass a variety of backbone networks, cross-domain testing, and targeted ablation studies.

The authors conducted experiments using models such as InternVideo2, Cosmos-Embed1, and Qwen3-VL-Embedding, evaluating the model's generalization capabilities on the AV2 dataset (which was not used during training or hyperparameter tuning). The "reward ablation" analysis presented in Table 2 is particularly noteworthy: removing the "video-to-video" reward causes motion-based retrieval performance to degrade to the level of an SFT-only model, while removing the "text-to-video" reward leads to a decline in instance-level retrieval capability. This indicates that the two reward mechanisms serve distinct functions: the former assists the model in learning motion structures, while the latter ensures the maintenance of the alignment between language and video.

Critical weaknesses
  1. The necessity of GRPO and the innovation of the proposed method are not sufficiently justified.

The core performance gains in this paper stem from motion similarity supervision based on ego-trajectories, while GRPO itself is an existing optimization method. In Table 3, the authors compare TraVEL against methods such as "Trajectory Prediction" and "Trajectory Similarity Matching," showing that their proposed training scheme outperforms these specific implementations.

However, the current comparisons are insufficient to clarify whether the performance improvements arise from GRPO’s "intra-group relative advantage optimization," the trajectory similarity supervision, "Hard Negative Mining," or perhaps a more straightforward ranking-based optimization.

It is worth noting that when using the Qwen3-VL-Embedding-2B model, the "Trajectory Similarity Matching" method achieves mAP scores of 32.2 (lateral) and 52.0 (longitudinal), while TraVEL achieves 35.7 and 55.8, respectively. While this improvement is of some value, its magnitude is relatively limited.

  1. The approach for constructing trajectory similarity is overly simplistic and may fail to fully capture the complex semantics of driving motion.

The paper characterizes ego-motion using sequences of lateral displacement and velocity, employing Dynamic Time Warping (DTW) to compute similarity-based rewards. Although this design is intuitive and easy to implement, it fails to explicitly incorporate factors such as changes in vehicle heading, road curvature, lane topology, or the relative motion of surrounding traffic participants.

  1. The scope of the benchmarks is limited, and support for retrieving complex driving scenarios remains insufficient. Although the paper introduces two evaluation benchmarks, the "motion retrieval" task in nuReasoning includes only 7 lateral motion queries and 5 longitudinal motion queries. Due to an insufficient number of positive samples, certain rare categories—such as rapid acceleration or deceleration—are excluded from the mAP (mean Average Precision) calculation. Queries in the AV2 benchmark are primarily constructed based on predefined ego-motion predicates and text templates; they fail to adequately encompass complex natural language descriptions, interactions between traffic participants, or the broader semantic information present in diverse driving scenarios. Furthermore, the model’s performance on temporal query tasks in AV2 remains weak. Taking the Qwen3-VL-Embedding-2B model as an example, TraVEL achieves a temporal mAP of only 12.5, compared to 7.5 for a random baseline. This indicates that relying solely on supervision based on motion similarity is insufficient to address the challenge of understanding the temporal sequencing of multiple events.

  2. There is a lack of systematic analysis regarding statistical stability, training efficiency, and key hyperparameters.

Although Table 1 shows that TraVEL improves motion retrieval performance across all tested backbones, the magnitude of this improvement varies significantly between models. For instance, when using InternVideo2, the lateral and longitudinal mAP scores increased by only 0.5 and 1.0 percentage points, respectively. The paper does not provide standard deviations or confidence intervals for results obtained using multiple random seeds, making it difficult to determine whether these marginal performance gains are statistically robust.

The proposed method introduces an additional GRPO training phase, a candidate group selection process, and the calculation of trajectory similarities; however, the paper fails to provide a comprehensive account of the associated training time, GPU memory overhead, or computational costs relative to traditional contrastive learning.

Furthermore, the authors do not conduct systematic ablation studies on key components—such as group size, reward normalization, the weighting of different trajectory components, or the design of the Dynamic Time Warping (DTW) mechanism. Consequently, the specific contribution of each design choice remains unclear.

Questions for authors

Q1. What are the primary advantages of GRPO compared to traditional ranking optimization methods?

Q2. Why does TraVEL yield significantly greater improvements for turning and stopping maneuvers compared to lane changes or subtle lateral movements?

Q3. How sensitive are the experimental results to the key hyperparameters of GRPO?

Q4. How does the training cost of TraVEL compare to SFT and other trajectory-based supervision methods?

Q5. Does the method provide consistent gains across different random seeds and for more challenging motion categories?

4 comments

We sincerely appreciate your time and effort in reviewing our paper, and thank you for your expertise and constructive comments. We address your comments and questions one by one below, and the corresponding revisions are highlighted in blue in the revised paper. Unless noted otherwise, all numbers are means ± std over three seeds, with both SFT and TraVEL re-trained for every seed.

Source of the gains and GRPO vs. ranking objectives

We isolate each factor on Qwen3-VL-Embedding-2B by changing one factor at a time (table below). Removing the trajectory supervision removes the motion gain. Replacing the top-k candidates of both groups with random ones makes training destructive and collapses instance retrieval, showing the critical role of hard-negative mining. Replacing GRPO with a pairwise ranking loss (RankNet [a]) on the same groups yields similar motion mAP and slightly lower instance retrieval at the same training budget and cost, since both learn to rank the candidates of the same groups by their trajectory rewards. The motion gains therefore come from the trajectory similarity supervision applied to hard negatives, while GRPO offers a slight advantage in preserving instance retrieval at no extra cost.

VariantChange from TraVELLAT ↑LON ↑R@10 ↑MnR ↓
SFT—30.8 ± 1.146.8 ± 0.639.2 ± 0.873.9 ± 3.1
w/o trajectory supervisionvideo-to-video reward removed31.0 ± 1.347.1 ± 0.540.0 ± 0.973.9 ± 1.7
w/o hard-negative miningtop-k → random candidates (both groups)27.3 ± 0.137.4 ± 1.313.1 ± 0.8166.3 ± 8.0
Pairwise ranking lossGRPO → RankNet35.7 ± 1.355.0 ± 0.338.2 ± 1.079.1 ± 1.1
TraVEL—36.1 ± 1.355.1 ± 0.639.6 ± 1.177.3 ± 0.3

Simplicity of the trajectory similarity and lane-change performance

Lateral displacement and speed already cover the two motion axes described by the motion queries. Adding the heading change, the yaw rate, or a lane-relative offset as an additional reward component brings little benefit (table below, seed 42). Lane changes are harder to retrieve than turns or stops. A turn or a stop produces a large and sustained change in heading or speed, whereas a lane change is a brief lateral transition across a lane boundary. Retrieving it requires the video embedding to closely align the visual evidence, such as the lane markings, with the ego motion. The smaller gain on lane changes is not caused by the class distribution, since lane changes are about as frequent as turns in both the training and test sets. It is not caused by the DTW-based similarity either, since replacing DTW with a pointwise distance or normalizing it by path length barely changes the lane-change AP. Subtle in-lane shifts are even more difficult, as they involve a lateral movement of only about one meter, which can be hard to discern in the front-camera video. We regard both, as well as interactions with surrounding traffic participants, which are beyond our focus on clip-level ego motion, as remaining challenges for future work.

Reward componentsLAT ↑LON ↑Lane-change AP, left / right ↑
TraVEL (lateral displacement + speed)37.555.711.6 / 11.5
+ heading change38.155.111.5 / 11.7
+ yaw rate38.155.311.4 / 11.5
+ lane-relative offset37.356.011.9 / 11.1

Diversity of motion queries

The motion queries are the full set of ego-motion decisions annotated in the nuReasoning test split, i.e., its lateral and longitudinal decision vocabularies, rather than a subset we selected, except for three rare longitudinal categories that we omitted on purpose. "Quickly come to a stop", "Slow down quickly" and "Quickly accelerate" have only 10, 8 and 3 relevant videos, so their AP is highly noisy, and averaging them into the mAP would weight a few videos as heavily as categories with hundreds of positives.

The scope of the benchmarks is limited

To evaluate natural-language queries, we add zero-shot moment retrieval on BDD-X, whose queries are free-form, human-written action descriptions with justifications. Please see Common Question 3.

Weak performance on temporal queries

We agree. Please see Common Question 2.

Stability across seeds and challenging categories

Please see Common Question 1. For the more challenging motion categories, please refer to our response on lane-change performance above.

Training cost

With the same training budget on Qwen3-VL-Embedding-2B (table below), TraVEL costs about as much as contrastive fine-tuning, represented by Trajectory Prediction, which only adds a linear head to the contrastive objective, while giving the largest motion gains. Its additional cost comes from selecting the top-k candidates and a forward pass of the frozen reference model, while computing the trajectory rewards is negligible. Its peak memory is 72.0 GB per GPU with 192 videos per GPU, and since its cost per update depends on the candidate-pool and group sizes rather than on the dataset size, it scales with the number of training clips like contrastive fine-tuning.

MethodGPU-hours (H100)Δ LAT / Δ LON vs. SFT
Trajectory as Text1.52−0.3 / −0.4
Trajectory Prediction0.87+4.9 / +1.5
Trajectory Similarity Matching0.91+3.3 / +3.8
TraVEL1.00+5.3 / +8.3

Sensitivity to hyperparameters

Our hyperparameters are chosen empirically, and we analyze their effect on Qwen3-VL-Embedding-2B (table below, change from the default TraVEL configuration). TraVEL is not sensitive to the group size, the reward normalization or the component weights, as each of these variants changes motion mAP by at most about one point. A smaller group slightly lowers both motion axes, while a larger group lowers retrieval. Changing the component weights shifts the two axes in the intended direction, and replacing DTW with a pointwise distance lowers lateral mAP.

VariantΔ LATΔ LONΔ R@10
Group size k = 12 (default 24)−0.49 ± 0.05−0.86 ± 0.32−0.44 ± 0.52
Group size k = 48+0.15 ± 0.19+0.65 ± 0.56−1.16 ± 0.62
No clipping, no scaling+0.06 ± 0.24+0.49 ± 0.24−0.36 ± 0.04
Path-length DTW normalization+0.06 ± 0.19−0.51 ± 0.14−0.22 ± 0.37
w_lat : w_speed = 1 : 1 (default 1 : 2)+0.24 ± 0.03−1.01 ± 0.25−0.29 ± 0.39
w_lat : w_speed = 1 : 4−0.09 ± 0.10+0.60 ± 0.23−0.09 ± 0.21
Pointwise distance instead of DTW−0.67 ± 0.17−0.06 ± 0.26−0.24 ± 0.54

We hope that we clarified these very helpful comments. We would be glad to engage in discussion if there are any other questions or parts that need further clarification.

[a] Learning to Rank using Gradient Descent, ICML 2005.

Thank you for your response. The additional ablation studies and robustness experiments have addressed most of my concerns. However, the weak performance on temporal queries remains an unresolved issue, and the performance across different backbones in the additional robustness experiments is still a concern. Therefore, I will maintain my original score.

Thank you for your positive evaluation and for acknowledging that the additional ablations and robustness experiments address most of your concerns. We would like to briefly clarify the two remaining points.

Regarding temporal queries, we do not claim temporal generalization, as our objective does not explicitly model the ordering between events. Nevertheless, TraVEL also improves temporal queries on the VLM-based backbones in the zero-shot AV2 evaluation (Fig. 4 of the revised paper), while they remain considerably harder than the other query types. We view this as a concrete direction that our benchmark exposes for future work.

Regarding the different backbones, recent multimodal embedding models are increasingly built on VLMs, such as Qwen3-VL-Embedding and WeMM-Embedding, and TraVEL consistently improves both lateral and longitudinal motion retrieval on all three such backbones across two model families. We therefore believe that TraVEL is effective in the direction where video embedding models are heading. Its smaller gains on InternVideo2 and Cosmos-Embed1 are consistent with their comparatively weak aggregation of video tokens on top of frozen vision encoders, as discussed in Common Question 1.

Thank you again for your constructive feedback, which has helped us improve the paper.

Reviewer_xh9vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TraVEL, a two-stage framework for motion-centric driving-video retrieval. It combines SFT with trajectory-aware GRPO, using trajectory-similarity and text–video matching rewards to improve motion sensitivity while preserving language alignment. Trajectories are used only during training; inference retains single-vector text-to-video retrieval. Experiments across three model families on nuReasoning show improved motion retrieval with generally comparable instance-level performance, while Qwen-based evaluations demonstrate zero-shot improvements on AV2. Ablations assess both rewards and alternative uses of trajectory supervision.

Critical strengths
  • Clear motivation: the paper addresses an important gap in driving-video retrieval, especially for motion-related queries.
  • The experiments show improved motion retrieval over supervised fine-tuning across three embedding-model families on nuReasoning, with generally comparable instance-level retrieval performance. Zero-shot evaluation with the Qwen models also shows improvements on AV2.
  • The writing is clear and easy to follow. The motivation and the overall method are well explained.
Critical weaknesses
  • The figure design looks a bit too “ChatGPT-style” . I would prefer a more technical presentation.
  • The experiments feel somewhat insufficient. The reward ablations and alternative trajectory baselines are only evaluated on Qwen3-VL-Embedding-2B. Whether these findings also hold for other backbones?
  • In Table 1, TraVEL only shows small improvements over SFT on InternVideo2 and Cosmos-Embed1. The gains are larger on Qwen, but it is still unclear why the benefit varies so much across backbones.
  • The evaluation is limited to two datasets. The zero-shot AV2 results are useful, but broader evaluation would help address concerns about overfitting to dataset-specific patterns and make the generalization claim more convincing.
  • The method still seems weak on temporal queries. On AV2, the temporal mAP is around 12.5, compared with 7.5 for random ranking. This suggests that the improvement in motion retrieval does not yet translate into strong understanding of event order.
Questions for authors

see weakness

6 comments

We sincerely appreciate your time and effort in reviewing our paper, and thank you for your expertise and constructive comments. We address your comments and questions one by one below, and the corresponding revisions are highlighted in blue in the revised paper. All numbers are means ± std over three seeds.

Figure design

Thanks for the feedback. We have revised the figures accordingly. Please see the new figures in our revised paper.

Ablations on other backbones

Since TraVEL is less effective on InternVideo2 and Cosmos-Embed1 (see Common Question 1), ablations on these backbones would be less informative. We therefore repeat both studies on WeMM-Embedding-2B, a single-tower model from a different model family, built on Qwen3.5-2B. Both findings hold (tables below). Removing the video-to-video reward removes the motion gain, whereas removing the text-to-video reward keeps the motion gain but lowers instance retrieval. Among the alternative uses of the trajectory, TraVEL again achieves the highest lateral and longitudinal mAP on every seed, and the alternatives behave as they do on Qwen3-VL-Embedding-2B.

Reward ablation (WeMM-Embedding-2B)

MethodLAT ↑LON ↑R@1 ↑R@10 ↑MnR ↓
SFT31.8 ± 1.753.3 ± 1.811.7 ± 0.641.5 ± 0.370.1 ± 0.3
TraVEL w/o t2v36.2 ± 0.960.8 ± 2.39.3 ± 0.736.7 ± 0.789.8 ± 3.4
TraVEL w/o v2v32.0 ± 1.653.3 ± 1.611.6 ± 0.242.5 ± 0.370.0 ± 1.1
TraVEL35.8 ± 0.761.0 ± 1.911.4 ± 0.541.0 ± 1.275.9 ± 1.9

Alternative uses of the trajectory (WeMM-Embedding-2B)

MethodLAT ↑LON ↑R@1 ↑R@10 ↑MnR ↓
SFT31.8 ± 1.753.3 ± 1.811.7 ± 0.641.5 ± 0.370.1 ± 0.3
Trajectory as Text32.6 ± 0.754.2 ± 1.511.1 ± 0.341.2 ± 0.769.7 ± 1.8
Trajectory Prediction35.0 ± 0.853.4 ± 1.311.1 ± 1.040.4 ± 1.370.3 ± 1.3
Trajectory Similarity Matching34.1 ± 0.758.1 ± 2.010.8 ± 0.341.2 ± 0.672.3 ± 0.5
TraVEL35.8 ± 0.761.0 ± 1.911.4 ± 0.541.0 ± 1.275.9 ± 1.9

Varying benefit across backbones

Please see Common Question 1.

Evaluation beyond two datasets

We add a third benchmark, zero-shot moment retrieval on BDD-X with free-form, human-written queries. Please see Common Question 3.

Weak performance on temporal queries

We agree. Please see Common Question 2.

We hope that we clarified these very helpful comments. We would be glad to engage in discussion if there are any other questions or parts that need further clarification.

Dear Reviewer,

We would like to kindly follow up to see whether our responses have addressed your concerns and whether there is anything else we could further clarify. Thank you once again for your time and valuable feedback.

Area_Chair_v81GOpenReview ↗

Reminder

Dear reviewer,

We’re about halfway through the discussion period. Please take a moment to read their response and leave a brief reply on OpenReview. Please let them know which of your concerns have been addressed and whether you have any follow-up questions, so they have time to respond before the discussion closes.

Thank you for your time and for helping keep the discussion moving!

Reviewer_jhHYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates natural-language retrieval for driving videos, focusing on scenarios where relevant clips must be distinguished by ego-motion instead of visual scene context. It presents TraVEL, which leverages ego-trajectory similarity as training supervision to adapt multimodal embedding models, and evaluates the method on nuReasoning and a rule-constructed Argoverse 2 benchmark. The zero-shot evaluation on the AV2 benchmark also demonstrates cross-dataset generalization.

Critical strengths

(1) The paper identifies a concrete limitation of general-purpose video embedding models: aligning videos with captions does not necessarily yield representations that can distinguish driving behaviors sharing similar visual contexts. Using continuous ego-trajectory similarity as training supervision is a sensible approach to this problem.

(2) The experiments include ablation studies on supervision targets, comparisons with alternative trajectory-supervision strategies, and evaluation on an independent driving dataset. Furthermore, since trajectory information is needed only for training, the method maintains a simple vector-retrieval interface during inference.

Critical weaknesses

(1) The incremental methodological contribution brought by GRPO is not sufficiently established. TraVEL integrates a trajectory-derived video-to-video reward, dynamically sampled top-ranked candidates, and GRPO. Reward ablations in Table 2 indicate that the video-to-video reward accounts for most gains on motion retrieval. Nevertheless, the experiments do not verify whether GRPO yields benefits over standard ranking objectives when trajectory supervision and candidate selection are kept identical. The trajectory-similarity matching baseline in Table 3 differs along two dimensions: its optimization objective and candidate distribution. Since Retrieval-GRPO (Liu et al., 2026) has already adopted GRPO for dense retrieval, a controlled comparison with fixed trajectory rewards and candidate groups is required to isolate the added value of the proposed optimization strategy above the existing framework.

(2) The evaluation offers limited evidence for robust motion retrieval across diverse settings. Improvements relative to SFT are inconsistent: Table 1 shows lateral mAP gains of merely 0.5 and 0.3 points for InternVideo2 and Cosmos-Embed1, respectively, with no accompanying uncertainty estimates. Appendix A.1 omits three infrequent longitudinal categories, and Table 4 reports substantially low AP values for lane-change events and subtle in-lane motion. Cross-domain evaluation on AV2 is restricted to the Qwen3-VL-Embedding model family; for its 2B and 8B variants, temporal-query mAP reaches only 12.5 and 12.4 (Table 5). The manuscript further does not quantify whether annotated events are observable within the sampled video clips, nor does it conduct independent validation of the semantics for rule-synthesized AV2 labels. These shortcomings substantially weaken the evidence supporting reliable retrieval of subtle, rare, and temporally complex driving events.

Questions for authors

(1) Section 3 explicitly follows Retrieval-GRPO (Liu et al., 2026) by framing an embedding model as a retrieval policy. Related approaches also leverage motion cues for driving-scenario retrieval: CITMR (Peng et al., 2026) fuses image, text, and motion encoders, and SMc2f (Chen and Greer, 2026) introduces text–trajectory contrastive learning within a coarse-to-fine scenario-mining pipeline. The manuscript rightly observes that CITMR requires motion inputs during inference, unlike TraVEL. Could the authors clarify which components of the trajectory reward, candidate-group construction, and optimization formulation are novel beyond this inference-time distinction, and supply a component-wise comparison against these prior methods?

(2) Is GRPO necessary under matched supervision and candidate mining? Table 3 reports lateral/longitudinal mAP of 35.7/55.8 for TraVEL versus 32.2/52.0 for Trajectory Similarity Matching. However, the latter regresses similarities over all video pairs in the global batch (Appendix B, Eq. 13), while TraVEL uses top-ranked candidate groups (Eqs. 8–9) and the GRPO objective (Eqs. 1–2, 10–11). The reward ablation in Table 2 tests reward removal, not replacement of GRPO. Can the authors compare a pairwise or listwise ranking loss against GRPO using identical SFT initialization, trajectory rewards, candidate groups, and comparable optimization budgets?

(3) Section 5.1 constructs each eight-second nuReasoning clip around an annotated keyframe, with three seconds before and five seconds after it, and assigns the keyframe motion label as clip-level relevance. Event retrieval does not require that an action persist throughout the entire clip; the pertinent issue is whether that event is actually observable in the encoded input. Appendix B specifies sampling only 4, 8, and 16 frames per eight-second clip for InternVideo2, Cosmos-Embed1, and Qwen3-VL-Embedding, respectively. Can the authors quantify how often the labeled motion event is visible in the sampled frames, especially near maneuver boundaries, and whether label visibility differs across backbones?

(4) Figure 4 and Appendix Table 5 show temporal-query mAP of 12.5 (2B) and 12.4 (8B), compared with a single random-ranking baseline of 7.5 and SFT values of 10.5 and 11.3. Can the authors explain what aspects of compositional retrieval improve and delimit the temporal-generalization claim accordingly?

(5) Appendix B uses wlat = 1 and wspeed = 2, clips normalized distances at one, and scales component distances using a 95th-percentile corpus statistic. Can the authors justify these choices and compare fixed-length versus path-length DTW normalization?

8 comments

We sincerely appreciate your time and effort in reviewing our paper, and thank you for your expertise and constructive comments. We address your comments and questions one by one below, and the corresponding revisions are highlighted in blue in the revised paper. Unless noted otherwise, all numbers are means ± std over three seeds, with both SFT and TraVEL re-trained for every seed.

Comparison to standard ranking objectives

Thank you for this interesting question. We ran this comparison on Qwen3-VL-Embedding-2B, replacing only the GRPO objective on the video-to-video groups with a pairwise ranking loss, RankNet [a], applied to the reward-ordered pairs in each group (table below). Both objectives introduce motion structure into the video embedding and reach similar motion mAP. This is expected, since both learn to rank the candidates of the same groups by their trajectory rewards. GRPO preserves instance retrieval slightly better, as the pairwise loss gives a lower R@10 and a higher mean rank than GRPO. Both use the same training budget at essentially the same cost, since they differ only in the loss computed from the same similarity scores. A similar trend holds on WeMM-Embedding-2B (Appendix C of the revised paper). The motion gains therefore come mainly from the trajectory reward on the on-policy groups rather than from GRPO itself, and GRPO offers a slight advantage in preserving instance retrieval at no extra cost.

MethodLAT ↑LON ↑R@1 ↑R@10 ↑MnR ↓
SFT30.8 ± 1.146.8 ± 0.610.0 ± 1.039.2 ± 0.873.9 ± 3.1
Pairwise ranking loss35.7 ± 1.355.0 ± 0.39.3 ± 0.638.2 ± 1.079.1 ± 1.1
TraVEL (GRPO)36.1 ± 1.355.1 ± 0.69.5 ± 0.939.6 ± 1.177.3 ± 0.3

Limited gains on InternVideo2 and Cosmos-Embed1

Please see Common Question 1.

Omitted infrequent longitudinal categories

We omitted these categories on purpose. "Quickly come to a stop", "Slow down quickly" and "Quickly accelerate" have only 10, 8 and 3 relevant videos among the 1,500 test videos. Their AP is therefore highly noisy, and averaging them into the mAP would weight a few videos as heavily as categories with hundreds of positives.

Low AP on lane changes and subtle in-lane shifts

Lane changes are harder to retrieve than turns or stops. A turn or a stop produces a large and sustained change in heading or speed, whereas a lane change is a brief lateral transition across a lane boundary. Retrieving it requires the video embedding to closely align the visual evidence, such as the lane markings, with the ego motion. Subtle in-lane shifts are even more difficult, as they involve a lateral movement of only about one meter, which can be hard to discern in the front-camera video. We regard both lane changes and in-lane shifts as remaining challenges.

AV2 evaluation beyond Qwen3-VL-Embedding and temporal queries

We additionally evaluate the three other backbones on AV2 (table below, SFT → TraVEL). Consistent with the nuReasoning results, TraVEL clearly improves all four query types on the VLM-based Qwen3-VL-Embedding and WeMM-Embedding, and is less effective on Cosmos-Embed1 and InternVideo2. The relative improvement is largest on simultaneous queries for four of the five backbones. We hypothesize that this is because simultaneous queries are closest to our training data, where each clip is labeled with a longitudinal and a lateral action that occur together.

As for the temporal-generalization claim, please see Common Question 2.

BackboneAtomicUnorderedSimultaneousTemporal
Random ranking31.818.411.27.5
InternVideo2-Stage235.6 → 36.022.6 → 22.617.2 → 17.48.8 → 8.8
Cosmos-Embed1-224p54.1 → 55.433.0 → 34.127.0 → 28.312.8 → 13.2
Qwen3-VL-Embedding-2B48.0 → 55.027.6 → 34.218.1 → 23.110.9 → 13.1
Qwen3-VL-Embedding-8B57.4 → 61.131.6 → 37.125.0 → 28.212.0 → 13.3
WeMM-Embedding-2B54.3 → 60.730.4 → 33.922.0 → 26.212.9 → 14.7

Visibility and reliability of the motion labels

For nuReasoning, each keyframe's driving decision is determined from the ego vehicle's future five-second trajectory (Appendix B.2 of the nuReasoning paper). Our clips include these five seconds plus three seconds of history, so the labeled maneuver falls inside the encoded window by construction, and our manual inspection found it visible in most cases. Visibility could still differ across backbones, since we follow the frame-sampling configurations of their released models (Appendix B). We therefore also train InternVideo2 and Cosmos-Embed1 with 16 frames, the same number as Qwen3-VL-Embedding (table below). Both backbones improve in absolute terms, but TraVEL's gains remain similar to those with their original frame counts, so the sampled frames do not explain the different gains across backbones.

Backbone (seed 42)FramesSFT: LAT / LONTraVEL: LAT / LONΔ LAT / Δ LON
InternVideo2421.6 / 32.621.9 / 34.7+0.3 / +2.1
InternVideo21623.2 / 34.923.2 / 36.6+0.0 / +1.7
Cosmos-Embed1836.7 / 31.436.2 / 33.4−0.5 / +2.0
Cosmos-Embed11637.7 / 32.736.9 / 34.5−0.8 / +1.8

For AV2, the motion events are computed with the atomic functions of HYU_OASIS, the winning entry to the Argoverse 2 Scenario Mining Challenge, which builds on RefAV. These functions are designed and tuned for specific driving actions and are applied to AV2's ground-truth ego poses and lane geometry, so the labels are expected to be reliable, which is also consistent with our manual inspection of 50 AV2 logs.

Comparison with related work

CITMR, SMc2f and TraVEL all use motion information beyond the text–video pair, but in different ways:

How the trajectory is usedExtra trajectory modelNeeded at retrieval
CITMRas an input modality, aligned with image and text by contrastive learninga separately trained GNSS/IMU motion encoderGNSS/IMU signals for every candidate
SMc2fas the search space of LLM-generated mining programs, re-ranked by a text–trajectory matchera separately trained text–trajectory dual encoderobject/ego trajectories, an LLM and a VLM
TraVELonly as a training reward that shapes the video embeddingnonethe video only

TraVEL is the only one of the three that learns motion-aware features in the video embedding itself, so it needs no additional trajectory model and no trajectories at retrieval time.

Compared with Retrieval-GRPO, the on-policy candidate groups and the GRPO formulation follow its framework, which applies them to text retrieval with query–item relevance rewards. The component we introduce is the trajectory reward, a graded similarity between the ego trajectories of two videos, optimized over video-to-video groups. We will add the missing citation of SMc2f in the related work.

Ablation of hyperparameters

Our hyperparameters are chosen empirically, and we provide a sensitivity analysis on Qwen3-VL-Embedding-2B in the table below, reporting the change from the default TraVEL configuration. (1) The 95th-percentile scaling puts the lateral and speed distances on a common scale, and clipping the normalized distance at one is designed to keep potential outlier pairs from dominating the reward. In practice, removing both the clipping and the scaling makes little difference. (2) The component weights adjust the balance between the two components, as reflected by the trade-off between LAT and LON. (3) Normalizing DTW by the path length instead of the fixed length T also makes little difference.

VariantΔ LATΔ LONΔ R@10
No clipping, no scaling+0.06 ± 0.24+0.49 ± 0.24−0.36 ± 0.04
w_lat : w_speed = 1 : 1+0.24 ± 0.03−1.01 ± 0.25−0.29 ± 0.39
w_lat : w_speed = 1 : 4−0.09 ± 0.10+0.60 ± 0.23−0.09 ± 0.21
Path-length DTW normalization+0.06 ± 0.19−0.51 ± 0.14−0.22 ± 0.37

We hope that we clarified these very helpful comments. We would be glad to engage in discussion if there are any other questions or parts that need further clarification.

[a] Learning to Rank using Gradient Descent, ICML 2005.

Dear Reviewer,

We would like to kindly follow up to see whether our responses have addressed your concerns and whether there is anything else we could further clarify. Thank you once again for your time and valuable feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their constructive feedback. We have uploaded a revised version of the paper, in which all changes are highlighted in blue; for updated tables and figures, the caption is highlighted. The main additions are:

  • Multi-seed results. All main results are now reported as means ± std over three seeds, re-training both stages for every seed (Tables 1–3, Fig. 4, and Appendix C).
  • A new model family. We add WeMM-Embedding-2B, built on Qwen3.5-2B, to the main results, the AV2 and BDD-X evaluations, and the ablations (Table 1, Fig. 4, Appendix C).
  • Pairwise ranking loss vs. GRPO. We compare GRPO with a pairwise ranking loss under identical trajectory rewards, candidate groups, and training budget (Table 3, Appendix C).
  • A third benchmark. We add zero-shot moment retrieval on BDD-X with free-form, human-written queries (Sec. 5).
  • Broader transfer evaluation. AV2 results are now reported for all backbones (Fig. 4, Appendix C).
  • Sensitivity analysis. We analyze the group size, reward normalization, component weights, DTW design, and candidate selection (Appendix C).
  • Redrawn figures. Figures 1 and 3 are redrawn in a cleaner style.
AuthorsOpenReview ↗

Common Questions

We summarize the questions shared by several reviewers and address them together below.

CQ1. Why does TraVEL's benefit vary across backbones, and is it stable across seeds? (raised by jhHY, xh9v and ssXm)

All main results are now means ± std over three seeds, re-training both stages for every seed. TraVEL effectively improves the motion retrieval ability of the recent VLM-based Qwen3-VL-Embedding and WeMM-Embedding, improving both lateral and longitudinal mAP on every seed, while it is less effective on InternVideo2 and Cosmos-Embed1 (table below). We attribute this to how the backbones form their video embeddings. Learning a structured video embedding that also benefits retrieval requires strong interactions among video tokens. In our setting, the vision encoder of every backbone is frozen. On top of that, the two dual-tower models build the video embedding with comparatively weak aggregation: attentive pooling with a single cross-attention layer in InternVideo2, and cross-attention followed by mean pooling in Cosmos-Embed1. This limits their capacity to learn such an embedding. In contrast, the video tokens in Qwen3-VL-Embedding and WeMM-Embedding interact through the multiple layers of a VLM, which is better suited to capturing motion dynamics, and TraVEL yields clear gains on both.

BackboneStageLAT ↑LON ↑R@1 ↑R@10 ↑MnR ↓
InternVideo2-Stage2SFT22.4 ± 0.934.1 ± 1.59.9 ± 0.340.6 ± 0.460.4 ± 0.4
TraVEL22.6 ± 0.835.9 ± 1.110.1 ± 0.341.4 ± 0.659.8 ± 0.2
Cosmos-Embed1-224pSFT36.4 ± 0.332.1 ± 1.310.1 ± 0.740.5 ± 0.358.7 ± 0.6
TraVEL35.9 ± 0.334.4 ± 1.69.2 ± 0.739.9 ± 0.362.9 ± 0.3
Qwen3-VL-Embedding-2BSFT30.8 ± 1.146.8 ± 0.610.0 ± 1.039.2 ± 0.873.9 ± 3.1
TraVEL36.1 ± 1.355.1 ± 0.69.5 ± 0.939.6 ± 1.177.3 ± 0.3
Qwen3-VL-Embedding-8BSFT36.4 ± 1.047.4 ± 2.313.1 ± 0.544.6 ± 1.659.1 ± 2.7
TraVEL37.8 ± 0.556.9 ± 1.812.6 ± 0.744.4 ± 1.060.0 ± 0.9
WeMM-Embedding-2B (new)SFT31.8 ± 1.753.3 ± 1.811.7 ± 0.641.5 ± 0.370.1 ± 0.3
TraVEL35.8 ± 0.761.0 ± 1.911.4 ± 0.541.0 ± 1.275.9 ± 1.9

CQ2. Temporal queries remain weak. (raised by jhHY, xh9v and ssXm)

As discussed in Sec. 5 and the Conclusion of the original manuscript, we do not claim temporal generalization, because our objective does not explicitly model temporal relations between events. The query types are part of our benchmark contribution, included to provide a more comprehensive analysis of motion retrieval. This analysis shows that temporal queries, which add an ordering constraint to unordered event pairs, remain considerably harder, which gives future work a concrete direction to pursue.

CQ3. Broader evaluation and natural-language queries: zero-shot moment retrieval on BDD-X. (raised by xh9v and ssXm)

We add a third benchmark, zero-shot moment retrieval on BDD-X [a], a crowd-sourced dashcam dataset in which humans annotated the action descriptions and their time spans. For each query, such as "The car slows down because the light turned red", we split the corresponding video, typically about 40 s long, into ten equal-length, non-overlapping segments of about 4 s, and the model must retrieve the segment in which the action occurs. The ground truth follows BDD-X's human-annotated time spans. We keep only actions no longer than one segment and take the segment containing the midpoint of the span. The benchmark has 601 queries, and chance-level R@1 is 10%. No BDD-X data is used for training or tuning. TraVEL improves over SFT on every seed for Qwen3-VL-Embedding-8B and WeMM-Embedding-2B, and in the same direction for Qwen3-VL-Embedding-2B (table below), indicating that its gains are not specific to the nuReasoning data.

BDD-X moment retrieval (R@1)

BackbonePretrainedSFTTraVEL
Qwen3-VL-Embedding-2B24.530.2 ± 1.831.7 ± 0.5
Qwen3-VL-Embedding-8B24.833.1 ± 0.537.4 ± 0.5
WeMM-Embedding-2B31.932.5 ± 3.136.6 ± 1.4

[a] Textual Explanations for Self-Driving Vehicles, ECCV 2018.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.