acceptodds
Under review as a conference paper at ICLR 2027

RefineDrive: Reliable Failure-Guided Learning for Vision-Language-Action Driving

Abstract

Vision-Language-Action (VLA) models for autonomous driving rely heavily on successful expert demonstrations, leaving model-specific failures underexploited. Learning from these failures is hindered by unreliable diagnoses, poorly matched correction targets, and coarse rewards. We propose RefineDrive, a failure-guided post-training framework that learns from self-generated failures through targeted supervision and safety-aware reinforcement learning. Reliable Diagnosis derives structured, verifiable feedback on collisions and drivable-area violations directly from simulator states. Minimum-Correction Target Retrieval searches a clustered human trajectory bank for nearby corrections that satisfy hard-safety constraints in the current scene, prioritizing preservation of the failed prediction’s motion pattern. Conditioned on the driving context and failed trajectory, Correction SFT learns to generate the diagnosis followed by the retrieved correction as a training-only auxiliary task. We then apply GRPO with a Safety-Layered Reward that strictly prioritizes hard-safe trajectories, retains continuous safety feedback for both unsafe and hard-safe trajectories, and rewards driving progress only after hard safety is satisfied. At inference, the policy directly predicts trajectories from the driving context without an explicit diagnosis or repair stage. On NAVSIM v1, RefineDrive improves the 4B base SFT policy from 87.7 to 91.7 PDMS. Using the same checkpoint without additional training, RefineDrive achieves 89.4 EPDMS on the original NAVTEST scenes evaluated with NAVSIM v2 extended metrics. Controlled ablations support the benefits of structured diagnosis supervision, retrieved corrections, and safety-layered optimization for direct planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action, Autonomous Driving, Trajectory Planning, Failure-Guided Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G1w9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RefineDrive, a failure-guided post-training framework for vision-language-action (VLA) models in autonomous driving. The core concept involves leveraging failure trajectories—generated by the model itself—to construct targeted diagnostic and corrective supervision, followed by safety-prioritized reinforcement learning to enhance the model's direct trajectory planning capabilities. The framework consists of four stages: First, failure cases—such as collisions or departures from the drivable area—are identified within the training scenarios. Second, a simulator’s state information is used to generate structured diagnostics, including the timing of the event, the positions of relevant objects, and the direction of boundary violations; simultaneously, corrective trajectories are retrieved from a human-demonstration dataset—selecting those that align with the model’s failed predictions while satisfying safety constraints for the current scenario. Third, the model undergoes "Correction SFT" (Supervised Fine-Tuning) to learn from these diagnostics and corrective trajectories. Finally, the model is trained using GRPO (Group Relative Policy Optimization) with a hierarchical reward function that integrates hard safety constraints, continuous safety margins, and driving progress. During inference, the model outputs trajectories directly based on the driving context, requiring no additional diagnostic or repair steps. Built upon the Qwen3.5-4B architecture, RefineDrive improves the PDMS (Planning-based Driving Metric Score) on the NAVSIM v1 benchmark from 87.7 to 91.7. The same checkpoint achieves an EPDMS of 89.4 when evaluated on the NAVSIM v2 extended metrics using the original NAVTEST scenarios. Furthermore, the paper presents ablation studies analyzing the individual contributions of the structured diagnostics, the retrieval of corrective targets, and the hierarchical safety-based reward function.

Critical strengths

Rather than continuing to learn from general examples of successful behavior, this paper focuses on addressing the safety flaws exposed by current policies. It establishes a comprehensive pipeline that spans failure detection, the construction of supervisory signals, and policy optimization.

Metrics such as time-to-collision (TTC), interactions with relevant objects, and lane-boundary violations are derived directly from the simulator state, thereby minimizing the reliance on inferences—common in free-text diagnostic feedback—that are difficult to verify. Furthermore, candidate corrective trajectories are re-validated within the specific context of the failure scenario, rather than simply adopting human-demonstrated trajectories from other scenarios as "safe" targets.

Within the defined reward framework, valid trajectories that satisfy strict safety constraints are consistently assigned higher rewards than unsafe ones, while maintaining continuous safety-related feedback. As shown in Table 4, this reward structure—compared to direct optimization of the baseline policy (PDMS)—improves performance across NC (near-collision), DAC (dangerous action), and TTC metrics, all while maintaining an equivalent average driving progress.

The authors conduct several controlled experiments: comparing human ground-truth data against retrieved corrective targets; evaluating the impact of removing diagnostic supervision while keeping the corrective targets fixed; and assessing the effects of different initializations for the "Correction SFT" (Supervised Fine-Tuning) phase followed by GRPO (Group Relative Policy Optimization). These controls help isolate the specific contributions of each component.

The diagnostic and corrective mechanisms are employed exclusively during the training phase; there is no explicit "repair" stage during deployment. The authors also explicitly report the safety-versus-progress trade-offs for the full training approach compared to various ablation scenarios, and they clarify that their extended metric-based evaluation does not constitute the complete, two-stage NAVSIM v2 protocol.

Critical weaknesses

The retrieval process terminates at the first cluster containing a safe candidate and is capped at a budget of 1,000 candidate evaluations; consequently, there is no guarantee that the retrieved safe target is the absolute nearest one within the entire trajectory library. The study currently lacks analysis regarding correction distance, retrieval success rate, and sensitivity to the search budget. Furthermore, there is no direct verification that a smaller feature distance necessarily correlates with a smaller behavioral change.

Cases involving collision objects that cannot be projected onto the current front-view perspective, instances where no safe target can be retrieved, and samples that fail filtering criteria are all excluded. The final dataset of 32,650 correction records may therefore be biased toward failures that are visible and easily remediable. However, as the paper does not report the counts or retention rates at each filtering stage, it is difficult to assess the extent to which challenging scenarios are covered.

The input for the "Correction SFT" (Supervised Fine-Tuning) includes failed trajectories, while the output includes both a diagnosis and a correction; during inference, however, neither the failed trajectory nor the diagnostic output is present. While ablation results support the method's effectiveness, it remains unclear whether the gains stem from the diagnostic content, the additional supervision tokens, or simply the repeated training on difficult scenarios. Further controlled experiments are required.

Some key performance differences are marginal—such as the 0.3 PDMS improvement attributed to diagnostic supervision, or the 0.1 PDMS gap between the final full approach and the "Human-GT" (ground truth) baseline—and these results lack multiple training runs or confidence intervals. Moreover, the absence of a control experiment involving continued SFT with an equivalent training budget makes it difficult to fully rule out the impact of additional computational training effort.

The experiments are centered on a 4B-parameter backbone and the NAVSIM data ecosystem, with extended metric evaluations conducted using the same NAVTEST scenarios. Consequently, the current evidence primarily supports performance improvements within a specific data ecosystem; it is insufficient to demonstrate generalization across different models, datasets, or long-horizon, closed-loop interactions.

While the simulator can verify when a collision occurs, which object is involved, and when a boundary violation takes place, this information does not necessarily reveal the underlying causes of the erroneous decision-making. The appendix further notes that object projection does not explicitly account for occlusions, and that the determination of boundary-violation direction sometimes relies on heuristic rules. Therefore, claims regarding the reliability of the diagnostics should be interpreted strictly within the context of event labeling.

Questions for authors

Could you report the distance distribution relative to human ground truth (Human GT), the retention rate of maneuver types, and performance results across global nearest-neighbor searches, varying cluster counts, and different search budgets?

How many failure cases remain after the filtering process? Please report the initial number of failures, the filtering rate (via visual localization), the retrieval success rate, and the final retention rate. Additionally, analyze whether the excluded cases are concentrated in challenging scenarios, such as those involving occlusions or complex interactions.

Why does diagnostic supervision improve performance compared to direct planning without diagnostics? Could you include ablation studies—such as using generic text of equal length, event-only labels, randomized (shuffled) diagnostics, or a direct mapping from "driving context" to "corrected trajectory"—to isolate the specific contribution of the structured diagnostic content?

Are the performance differences consistent? Could you provide results across multiple random seeds, scene-level paired confidence intervals, and baselines using continued Supervised Fine-Tuning (SFT) with equivalent training steps or computational budgets?

Does the improvement in safety come at the cost of degradation in specific behaviors? Beyond the average EP (Expected Performance/Error metric), could you report the frequency of full stops, the proportion of scenarios with low progress, and performance broken down by maneuver type? In Table 2, the performance for DDC (Direct Driving Control/Command) and LK (Lane Keeping) is relatively lower compared to some other methods; is this because the current reward function does not explicitly cover these behaviors?

Can the method generalize to broader settings? Could you provide evaluations using a different model backbone, an out-of-distribution (OOD) test set, or long-duration closed-loop simulations? Furthermore, please report the total computational cost—covering failure mining, candidate simulation retrieval, and GRPO—to help assess the practical deployment overhead.

Reviewer_FrXgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RefineDrive, a failure-guided post-training framework for autonomous driving VLAs. It uses simulator-grounded failure diagnoses, retrieves safe corrective trajectories, and applies Correction SFT followed by safety-layered GRPO. The method achieves 91.7 PDMS on NAVSIM v1, improving upon the 87.7 PDMS base model.

Critical strengths
  1. The framework is well-motivated. Simulator-grounded diagnoses and scene-verified corrections provide more reliable supervision for learning from driving failures.
  2. The method achieves competitive performance. The ablations demonstrate that the safety-layered reward improves safety metrics without sacrificing mean progress compared with direct PDMS reward optimization.
Critical weaknesses
  1. The necessity of the proposed Correction SFT is unclear. The paper lacks a baseline that directly trains on the same retrieved safe trajectories without failure-conditioned diagnosis and correction. Therefore, it is unclear whether the improvements come from the proposed learning mechanism or simply from additional safety-filtered supervision.
  2. The experimental evidence is insufficient to establish the effectiveness of the correction mechanism. After RL, the full method achieves only 0.1 higher PDMS than Human-GT Correction SFT (91.7 vs. 91.6), while EP decreases from 89.0 to 87.2. Although safety metrics improve, the trade-off is not sufficiently analyzed. Moreover, neither multi-seed results nor correction retrieval success rates are reported.
Questions for authors
  1. How does the proposed Correction SFT compare with directly training on the same retrieved corrections without failed-trajectory conditioning or diagnosis supervision?
  2. What is the retrieval success rate, and how often are corrections retrieved from clusters other than the initially assigned cluster?
  3. How do the authors justify the safety-progress trade-off observed in Table 5?
Reviewer_Cpd3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RefineDrive, a VLA framework that conducts post-training using the model's driving failures. This method generates structured failure diagnoses through simulators, retrieves the minimum correction trajectory that meets safety constraints and is close to the failed trajectory from the human trajectory database, and optimizes it in combination with Correction SFT and Safety-Layered GRPO. Ablation experiments verify the effectiveness of structured diagnosis, minimum correction trajectory, and layered safety rewards, but the method still mainly relies on perception labeling, and the safety improvement comes at the cost of a certain reduction in driving progress.

Critical strengths
  1. This article points out several significant limitations of the existing failure learning methods, including unreliable open-ended diagnosis, possible excessive correction in supervision, and rough reward signals. The proposed framework perfectly aligns with these issues.
  2. Reliable diagnosis, minimized correction retrieval, correction SFT, and Safety-Layered RL together form a logically consistent framework centered on model-specific faults. Importantly, the additional diagnosis and correction processes are only used for training and are not introduced as an explicit repair stage during reasoning.
  3. The experiments separately explores the effects of human-GT, minimum correction, structured diagnostic supervision, and safety layer rewards. These comparisons generally support the design choices and make the sources of performance improvement relatively clear.
  4. When using the 4B model, RefineDrive achieved competitive performance on NAVSIM v1 and was evaluated using the extended metrics of NAVSIM v2 at the same checkpoints without additional training, achieving good results.
Critical weaknesses
  1. The overall framework is well-designed, but its individual components are relatively simple. The main contribution lies in their systematic integration rather than introducing a fundamentally new learning paradigm, and the mechanism of the SFT may be similar to that of ELF-VLA.
  2. This framework highly relies on the perception labeling of the dataset. Object information is easily obtainable, but map information may be difficult to label in other datasets, which may limit the direct application of this framework in environments where perception information is unavailable or unreliable.
  3. Ablation experiments support the proposed architecture, but after the RL stage, the advantages of minimizing corrections and structured diagnosis become less obvious. Stronger evidence helps to more clearly establish their independent contributions to the final strategy.
  4. The current evaluation is mainly conducted within the NAVSIM benchmark. If additional experiments can be carried out on complementary benchmarks such as Bench2Drive, it will provide stronger evidence for the universality and robustness of the proposed framework.
Questions for authors

Please refer to weakness.

Comments on the paper

1 comment
Area_Chair_7jPGOpenReview ↗

Reminder: Author Response and Reviewer Discussion

Dear Reviewers,

We are halfway through the discussion period. Please review the authors’ responses, address any remaining concerns, and engage in the discussion where appropriate. If your assessment remains unchanged, please briefly acknowledge that you have considered the authors’ responses.

Thank you for your time and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.