- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Test-Time Learning via Rejection Sampling (TTLR) for adapting driving policies to long-tail scenarios. It samples candidate trajectories, accepts those passing drivable-area, occupancy, and comfort checks, and takes one gradient step toward the accepted set. Candidates may come from a fixed teacher or the adapting policy itself. The geometric verifier combines SAM 3 segmentation with UniDepthV2 depth estimates. A separate experiment learns trajectory selectors from random initialization over kinematically initialized trajectory vocabularies. Results on KITScenes LongTail and Waymo Open E2E show improvements over the initial policies.
Critical strengths
- The method provides a simple way to use existing trajectory checks as learning signals without new expert trajectories at adaptation time. Supporting both fixed-teacher distillation and self-distillation makes the approach applicable to different deployment settings.
- The empirical coverage is useful. Experiments include multiple pretrained policies, two recent long-tail benchmarks, and modified trajectory classifiers trained from scratch. The appendix reports three-seed results, and the paper explicitly discusses perception failures and verifier overfitting.
Critical weaknesses
- The benefit of learning over filtering is not isolated. The fine-tuning experiments sample 120 candidates per scenario, but lacks a frozen-policy baseline using the same candidates and verifier. It is therefore unclear whether weight updates offer an advantage over verifier-based selection alone.
- The strongest performance comparison needs a clearer protocol. It uses final weights after adaptation over the test split, while online predictions represent a different setting. Online KITScenes MMS does not clearly exceed the reported challenge leader. Scenario ordering, reset rules, and comparability with the challenge protocol remain insufficiently specified.
- Limited novelty beyond RAFT. TTLR shares RAFT’s sample–filter–fine-tune framework, using binary safety checks to select training targets. The main distinction appears to be its application to driving at test time. The paper should clarify what methodological advance this introduces beyond the established framework.
Questions for authors
- In the fine-tuning setting, does TTLR outperform a frozen policy using the same 120 candidates and verifier for selection? An unfiltered self-distillation control would also help separate the contributions of the verifier and weight updates.
- How are scenarios ordered, when are weights reset, and does final evaluation revisit the observations used for adaptation? Please clarify how the challenge comparison matches permitted data and adaptation protocols, and how the reported best runs are selected.
- What are the end-to-end latency and memory costs of sampling, perception, the gradient step, and final prediction? These measurements matter because the fine-tuning setting uses 120 candidates per scenario and two large perception models before adaptation is complete.
- Please specify the training loss and updated parameters for each policy, including how multiple accepted trajectories are combined. What happens when no candidate is accepted, especially under the perception failures illustrated in Figure 5?
2 comments
Follow-up
Since many reviewers asked about inference latency, we additionally worked on a real-time version of our verifier and added the results to Table 7. Our version (b) is real-time capable when run in parallel with the driving policy for high-level planning, which typically occurs at 4-5 Hz, matching the recording frequency of the Waymo Open E2E and KITScenes LongTail datasets. Alpamayo 1.5 itself can be accelerated by using quantization, as in the version of Li et al. (2026).
Li et al. (2026), FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving. In arXiv
Thank you for your valuable feedback on our work! We hope to answer all your questions with additional information and new experiments. In addition, we added all new experiments to the rebuttal revision of our paper. All changes in the PDF are highlighted in blue.
R1: We added this baseline as Eval. = w/ verifier to Table 2. Filtering trajectories with our verifier and randomly selecting an accepted trajectory improves MMS, RFS and ADE. Optimizing the weights with TTLR further improves results significantly (e.g., MMS 4.69 to 4.99 (online) or 5.50 (final)). Note that we mixed up MMS and MMS (type) in the initial submission and corrected the order as well, apologies!
R2.1: For each of the random seeds reported in our appendix, we traverse the test splits in a different random order. Each seeded run starts from the base weights of the model. We don’t reset weights during individual runs.
R2.2: Yes. This is permitted by the challenge rules.
R3: We report latency and VRAM for Alpamayo with and without TTLR in Table 6. TTLR increases latency by only 23% and VRAM consumption by 20%. In a production system, the verifier latency can be significantly reduced by using a LiDAR sensor instead of depth estimation and BEV grid generation (see Table 7).
R4.1: We use the default loss functions for each policy and the official code implementations. We added this information to Table 8. For models optimized with batch size one (Alpmamyo 1.5, OneVL) we randomly choose one accepted trajectory to optimize the model, for models that perform trajectory planning as classification (VADv2, CILRS, CoverNet), we optimize backpropagate the loss for all accepted trajectories.
R4.2: We don’t optimize the models on scenarios where all trajectories are rejected and use the current weights for prediction.
This is resolved by fixing a copy-paste error on our side. We copied the MMS scores from the official challenge leaderboard wrong. Specifically, we copied the MMS values for our method and the MMS (type) values of the challenge winners. MMS is a global average, MMS (type) is an average over the per scenario type averages. Apologies for this mistake! We updated Table 2 in the paper to include both scores in the correct ordering. Our TTLR Alpamayo version (self-distillation) evaluated in an online manner significantly outperforms the challenge winner TTVLM in both metrics (MMS: 5.11 vs 4.75, MMS (type): 5.46 vs. 5.15).