- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ReDrive, which uses future prediction to supervise the representations learned by a visual encoder. Compared with conventional future-prediction approaches, the method does not require generating future video frames during inference. Instead, it distills the representational capabilities associated with future generation into the visual encoder during training. The method uses vjpeca2 for pretraining on autonomous driving scenes, followed by multistage training. It achieves strong results on several mainstream benchmarks, demonstrating the feasibility of retaining future-prediction capabilities within the visual encoder.
Critical strengths
The paper offers useful insights to the community: strong performance can be achieved by using future prediction for video representation learning without generating future video frames during inference. The ablation studies clearly show that future prediction itself provides only modest gains, whereas improvements in visual representations yield substantially larger gains. The ablation studies are clear and easy to follow, and the comparative conclusions are well articulated.
Critical weaknesses
The paper analyzes the future predictor’s behavior solely through the similarity of predicted features under different action conditions. This demonstrates sensitivity to actions but is insufficient to establish that the predictor can provide reliable planning gradients for non-expert trajectories.
Questions for authors
Could the authors provide a quantitative comparison between predicted features and actual future features, together with an interpretable analysis of the future predictions? This evidence would help assess prediction reliability beyond sensitivity to action conditions and inform the evaluation of whether the predictor can provide reliable planning gradients for non-expert trajectories.
3 comments
Thank you for the detailed response and additional experiments. The comparison with observed future features and the trajectory-ranking results address my concern about prediction reliability. The one-step experiment also provides direct evidence that the predictor can offer useful gradients for planner-generated trajectories. These results address my main concerns, and I maintain my weak accept recommendation.
We sincerely thank the reviewer for your positive feedback and for acknowledging our clarifications. We greatly appreciate your time and thoughtful consideration.
We sincerely thank the reviewer for the positive evaluation of our work and the valuable suggestions. We have conducted additional experiments to address the remaining concerns and provide detailed responses below.
Question 1:How reliable and interpretable are the future representations predicted by the future predictor?
Response 1: We thank the reviewer for the suggestion. To evaluate the reliability of the predicted future representations, we conduct additional experiments on 1,000 NAVSIM test samples. We compare the predicted future representations conditioned on GT and randomly shuffled trajectories against the actual future representations. As shown in Table R1, GT trajectory conditioning achieves higher cosine similarity, indicating that the future predictor produces representations more consistent with the actual future under the observed trajectory condition.
Table R1. Similarity between predicted and actual future representations under different trajectory conditions.
We further evaluate prediction consistency through a trajectory ranking experiment. For each sample, we rank the GT trajectory and randomly sampled trajectories from other samples based on the cosine similarity between predicted and actual future representations. GT Top-1 measures the proportion of samples where the GT trajectory ranks first, while Pairwise Win Rate measures how often it outperforms shuffled trajectories. As shown in Table R2, the future predictor captures meaningful correspondence between trajectory conditions and actual future representations.
Table R2. Ranking accuracy of the GT trajectory among different trajectory candidates.
Question 2: Can the future predictor provide reliable planning gradients for non-expert trajectories?
Response 2: We thank the reviewer for this question. To evaluate gradient reliability on non-expert trajectories, we conduct a one-step gradient update experiment on 1,000 planner-generated trajectories. As shown in Table R3, gradient descent on the future prediction loss reduces the GT trajectory loss in 70.9% of samples, compared with 46.4% for random directions. These results support the effectiveness of the frozen future predictor in providing useful planning gradients.
Table R3. Effectiveness of future prediction gradients on planner-generated trajectories.