acceptodds
Under review as a conference paper at ICLR 2027

AutoAgent0: An Agentic Runtime for Safe Closed-loop Driving

Abstract

Safe autonomous driving requires more than accurate planning at a single frame: it demands reliable closed-loop execution in dynamic environments. Although end-to-end driving policies have advanced, the accumulated closed-loop errors, especially in unseen scenarios, can push vehicles into states from which the same policy struggles to recover. We address the challenge by treating closed-loop driving safety as a test-time execution problem and introduce AUTOAGENT0, an agentic runtime and harness framework that augments existing driving experts with complementary detection skills, rule-based action verification, and feedback-driven recovery. First, to identify the two sources of driving failures (inaccurate scene perception and recovery planning), the runtime first introduces an independent detection module that constructs scene context using complementary perception skills, and a rule-based verifier assesses the action feasibility. When the nominal trajectories are unreliable, the orchestrator of the runtime activates Recovery Planning, where an agentic loop constructs alternative maneuvers from action primitives and iteratively refines them using verification feedback. This design combines independent scene assessment with feasibility checks and online recovery, addressing both perception errors and the limited recovery capabilities of individual experts. Evaluations on Fail2Drive and NavSafe- demonstrate improved closed-loop robustness over single-expert execution in long-tail and safety-critical scenarios. These results support agentic runtime as a complementary approach to improving driving safety beyond advances in the underlying policy alone.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Agentic System

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v9j7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents AutoAgent0, an agentic runtime harness to augment frozen pre‑trained driving experts (e.g., SparseDriveV2, UniAD, BridgeDrive) online via a pre-trained VLM. The system consists of three core components:

Scene‑context refinement: Cross‑modal perception together with VLM review to discover obstacles and refine scene representation for verification.

Rule‑based trajectory verifier: A rule‑based verifier checks driving criteria and rejects unsafe nominal trajectories proposed by the base driving expert.

Agentic recovery planning: When trajectories are rejected, a VLM selects maneuver primitives from a predefined action primitive library; parameters are sampled to generate candidate trajectories, which are iteratively refined under verifier feedback until an admissible trajectory is found.

Closed-loop evaluations on Fail2Drive and NavSafe show that augmenting existing driving experts with AutoAgent0 improves driving score (DS) and route success rate (SR), especially for long‑tail behavioral scenarios.

Critical strengths

Novel perspective on introducing agentic workflow for autonomous driving system: The idea of using agentic systems to wrap multiple frozen driving experts is practically inspiring. In real‑world autonomous driving products, the typical solution maintains multiple specialized expert models that handle various driving scenarios. An agentic orchestrator could unify and dispatch these experts for long‑tail corner cases, a promising research direction beyond pure end‑to‑end model training.

Meaningful exploration of bringing LLM capabilities into closed‑loop driving: Most prior works focus on the neural planner itself. This work explores a new paradigm: use a VLM agent for high‑level maneuver reasoning, while keeping deterministic rule‑based verification as the safety gate. It opens new thinking for handling long‑tail failures that traditional brake‑only monitors cannot resolve.

Thorough ablations: The authors perform rich ablations to decompose contributions from scene‑context refinement, rule‑based verification and feedback‑guided recovery. They also conduct well-structured ablations on different scenarios.

Critical weaknesses

Ambiguity on system deployment target leads to potential unfair comparison setting: It remains ambiguous whether AutoAgent0 is designed as an offline cloud‑simulation tool or on‑vehicle online real‑time runtime.

  • If it targets offline simulation experts: comparisons should be made against oracles with perfect perception and exhaustive rule‑based maneuver search baseline, rather than real‑time on‑vehicle driving experts.
  • If it targets on‑vehicle online execution: the current evaluation heavily relies on heavy‑weight remote LLM backends (Opus5) with large millisecond latency per recovery request. If the platform is cut-edge platform like thor-u or 8650, the latency will increase significantly. The paper lacks sufficient validations on small‑size on‑device LLMs (e.g., 2B) to prove the agentic pipeline can work under an on‑vehicle latency budget.

Without clarifying its deployment goal, the comparison between AutoAgent0 and vanilla real‑time driving experts is hard to interpret and may be unfair.

Guarantee of valid recovery trajectory is not well‑addressed: The agentic recovery loop selects primitives and samples parameters within a limited search budget. There is no mechanism / theoretical guarantee that at least one feasible collision‑free trajectory exists within the primitive library and parameter search space. When all sampled candidates get rejected, the system falls back. However, the paper lacks quantitative analysis on how frequently such “complete recovery search failure” happens, and how it impacts end-to-end closed-loop metrics.

Questions for authors

Regarding the Action Primitive Library and recovery search process: Could you elaborate on the Action‑Primitive library workflow? The details on sampling primitive trajectories from a single SparseDriveV2 are not clear. How are the parameters sampled? Moreover, when recovery planning is triggered after all nominal proposals from SparseDriveV2 are rejected, does the recovery loop still sample trajectories from SparseDriveV2?

6 comments
AuthorsOpenReview ↗

Response to W1 (1/3)

W1. Ambiguity on system deployment target leads to potential unfair comparison setting: It remains ambiguous whether AutoAgent0 is designed as an offline cloud-simulation tool or on-vehicle online real-time runtime.

  • If it targets offline simulation experts: comparisons should be made against oracles with perfect perception and exhaustive rule-based maneuver search baseline, rather than real-time on-vehicle driving experts.
  • If it targets on-vehicle online execution: the current evaluation heavily relies on heavy-weight remote LLM backends (Opus5) with large millisecond latency per recovery request. If the platform is cut-edge platform like thor-u or 8650, the latency will increase significantly. The paper lacks sufficient validations on small-size on-device LLMs (e.g., 2B) to prove the agentic pipeline can work under an on-vehicle latency budget.

Without clarifying its deployment goal, the comparison between AutoAgent0 and vanilla real-time driving experts is hard to interpret and may be unfair.

Response: AutoAgent0 targets on-vehicle execution. With a local 2B model and no simulator pause, it still improves SparseDriveV2 by +6.7 DS / +11.3 SR, with a collision rate on par with Opus 5 (Table A).

On-vehicle setting. The VLM is not in the control loop. The ego vehicle brakes as soon as a plan is rejected, so a late VLM answer only means waiting longer. As a result, raising the Opus 5 delay from 1 s to 4 s lowers SR by only 2.9 points, with no more collisions. The 2B model has a collision rate on par with Opus 5, and its lower DS and SR come from resolving fewer blockages.


Table A. Recovery without pausing the simulator on 200 Fail2Drive routes. Delay is set in simulated time, not wall-clock time: traffic keeps moving for this long between a recovery request and its answer.

ProtocolRecovery VLMDelay (sim. time)DS↑SR (%)↑Col.↓
Standalone SparseDriveV2––66.563.40.445
Paused (paper)Opus 5080.083.40.145
Not pausedOpus 51 s79.682.70.152
Not pausedOpus 54 s77.979.80.127
Not pausedQwen3.5-2B (local)1 s73.274.70.140

Offline setting. As suggested for the offline target, both methods use ground-truth perception, and exhaustive search gets a 6× larger budget (Table B). Exhaustive search still trails AutoAgent0. The verifier score does not reflect which maneuver suits the scene, so a larger budget only finds more candidates of the same kind.


Table B. Recovery with ground-truth perception on the 34 threshold-sweep routes (3 seeds). Budget is the parameter-search budget relative to the default.

Recovery selectorBudgetDS↑SR (%)↑Col.↓
Exhaustive search6×80.279.40.147
AutoAgent06×84.991.20.059
AutoAgent0 (default budget)1×85.191.20.059

The comparison with real-time experts is fair. All methods are evaluated under the same protocol, in which the simulator waits for planning at every step. SparseDriveV2 itself is not real-time, since its planning step takes a median of 397 ms against a 50 ms control tick (main paper, Table 7). The only extra wait for AutoAgent0 is the recovery request, and in Table A the simulator is not paused for it.

AuthorsOpenReview ↗

Response to W2 (2/3)

W2. Guarantee of valid recovery trajectory is not well-addressed: The agentic recovery loop selects primitives and samples parameters within a limited search budget. There is no mechanism / theoretical guarantee that at least one feasible collision-free trajectory exists within the primitive library and parameter search space. When all sampled candidates get rejected, the system falls back. However, the paper lacks quantitative analysis on how frequently such "complete recovery search failure" happens, and how it impacts end-to-end closed-loop metrics.

Response: When no recovery candidate passes the verifier, AutoAgent0 falls back to braking and plans again. This happens in about one in five recovery searches and costs at most 3.2 pp SR.

The fallback is a 1 s full brake, after which the runtime plans again. We measured how often it is triggered and what follows (Table C).


Table C. Recovery searches in which no candidate passes the verifier (200 Fail2Drive routes).

MeasureValue
Recovery searches in which no candidate passes21%
After the brake: a recovery maneuver passes64%
After the brake: a SparseDriveV2 plan passes again30%
Routes with contact during the brake1.2%
Routes that time out with the case unresolved2.0%
Simulated time spent in these brakes1.8%

What happens after the brake. Most of these cases occur at speed, when a hazard first comes into range and no maneuver passes. A full stop is then the right response. Once the vehicle has stopped, a later search usually succeeds. Contact during the brake occurs mainly when the hazard is detected inside the stopping distance. No maneuver could avoid it at that point either, so these cases come from late detection.

Search budget and perception. A 6× larger search budget leaves SR unchanged (Table B), whereas ground-truth perception removes about two thirds of these cases.

AuthorsOpenReview ↗

Response to Q (3/3)

Q. Regarding the Action Primitive Library and recovery search process: Could you elaborate on the Action-Primitive library workflow? The details on sampling primitive trajectories from a single SparseDriveV2 are not clear. How are the parameters sampled? Moreover, when recovery planning is triggered after all nominal proposals from SparseDriveV2 are rejected, does the recovery loop still sample trajectories from SparseDriveV2?

Response: SparseDriveV2 is not used during recovery. It supplies only the nominal proposals, and recovery trajectories are computed by the primitive library directly from the current scene geometry.

A recovery decision proceeds as follows.

  1. Trigger. Recovery starts when the verifier rejects all SparseDriveV2 proposals, or when the ego vehicle makes no progress for 5 s although a proposal passes (main paper, Figure 5(c)).
  2. Select primitives. The VLM is given the camera images, the scene context, the verifier's rejection reasons, and the available primitives (main paper, Appendix D.3). It returns up to five primitives with probabilities.
  3. Sample parameters and generate trajectories. Each primitive has predefined parameters (lateral offset, speed, etc.) with bounded ranges (main paper, Table 10). The runtime samples them on a grid within these ranges, with more samples for primitives the VLM rates higher. Each parameter set yields one candidate trajectory, by a kinematic-bicycle rollout or, for lane changes and bypasses, from HD-map lane geometry.
  4. Verify. The same verifier checks every candidate, and the highest-scoring one that passes is executed.
  5. Revise. If none passes, the rejection reasons go back to the VLM, which selects other primitives. After at most three VLM calls without success, the runtime brakes (see our response to W2).
  6. Hand back. SparseDriveV2 is not queried while the maneuver runs. Afterwards it proposes again and takes back control once one of its proposals passes the verifier.
Reviewer_y8hNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoAgent0 augments existing driving policies with an external runtime combining scene-context refinement, rule-based trajectory verification, and VLM-assisted recovery. When nominal trajectories are rejected, the system constructs alternatives using parameterized action primitives and revises them through verifier feedback. The driving expert remains frozen, although additional perception modules are trained for Fail2Drive. Evaluations on Fail2Drive and NavSafe-∞ demonstrate improvements across multiple experts, supported by component ablations, recovery-mechanism analysis, and comparisons with conventional runtime monitors.

Critical strengths
  1. Consistent improvements across multiple driving experts and two closed-loop benchmarks.

The runtime improves performance without retraining the underlying driving policies. For example, on Fail2Drive, SparseDriveV2’s average success rate increases from 63.4% to 83.4%, while BridgeDrive’s generalization success rate increases from 75.0% to 85.3% (Table 1). Improvements on NavSafe-∞ further support the usefulness of runtime augmentation beyond a single expert or benchmark.

  1. A concrete application of frontier LLM agentic harnesses to autonomous driving.

The paper presents a concrete framework for applying frontier multimodal language models within an agentic driving harness. It coordinates perception tools, parameterized recovery maneuvers, and verifier feedback around an existing driving expert. This provides a useful research prototype showing how tool use and iterative correction can translate general-purpose model capabilities into driving recovery.

Critical weaknesses
  1. The practicality and necessity of online runtime supervision are not sufficiently established.

External supervision can complement existing driving policies, but latency is a central constraint for onboard autonomous driving. The reported VLM requests take several seconds, and the Fail2Drive evaluation pauses simulation while recovery requests are pending (Table 8, Appendix B.2). Appendix H.4 distinguishes recovery from immediate safety responses handled by lower-level systems. Nevertheless, the evaluation does not establish whether recovery remains effective when traffic continues evolving during inference. Strong generalization under paused simulation is therefore insufficient evidence of effective supervision under real-time constraints.

Future distillation or model compression could reduce latency, but the benefits of retaining a separate supervisor, beyond avoiding expert retraining, would still need to be established relative to integrating its capabilities into the driving policy or a standalone VLA. The current evidence appears more immediately applicable to offline labeling, simulator/world-model-based recovery-data generation, or policy distillation. The case for online use requires evidence that its benefits survive realistic inference delays.

  1. The methodological distinction from existing slow–fast driving systems is insufficiently established.

Combining a fast driving model with a slower, more capable model has already been explored in SOLVE and ETA, while AdaDrive additionally learns when to invoke the language model and how strongly to incorporate its output. Thus, model complementarity and selective invocation alone do not establish methodological novelty.

AutoAgent0 differs through its frozen-expert runtime, independent scene refinement, shared verification, and feedback-driven recovery. However, the paper does not sufficiently explain which limitations of existing slow–fast systems this particular combination resolves. The comparisons in Section 4.4 primarily concern monitors whose corrective actions are longitudinal; they support the value of richer recovery actions but do not resolve the comparison with existing dual-system approaches. A clearer account of the incremental contribution, supported by comparisons that consider both performance and latency, is needed.

Questions for authors
  1. How does the system perform when the VLM-guided primitive planner serves as the sole planner, while retaining the same perception, verifier, and controller? Comparing driving performance and inference cost would clarify the contribution of the nominal E2E policy and the benefits of combining the two planners.

  2. Which limitations of the underlying driving policies explain the remaining performance differences after applying AutoAgent0, and why does the supervisor fail to compensate for them? Distinguishing missed or delayed interventions, unsuccessful recovery, and failures after returning control to the nominal policy would help clarify the runtime’s capabilities and limitations.

7 comments
AuthorsOpenReview ↗

Response to W1 (1/4)

W1. 1. The practicality and necessity of online runtime supervision are not sufficiently established.

External supervision can complement existing driving policies, but latency is a central constraint for onboard autonomous driving. The reported VLM requests take several seconds, and the Fail2Drive evaluation pauses simulation while recovery requests are pending (Table 8, Appendix B.2). Appendix H.4 distinguishes recovery from immediate safety responses handled by lower-level systems. Nevertheless, the evaluation does not establish whether recovery remains effective when traffic continues evolving during inference. Strong generalization under paused simulation is therefore insufficient evidence of effective supervision under real-time constraints.

Future distillation or model compression could reduce latency, but the benefits of retaining a separate supervisor, beyond avoiding expert retraining, would still need to be established relative to integrating its capabilities into the driving policy or a standalone VLA. The current evidence appears more immediately applicable to offline labeling, simulator/world-model-based recovery-data generation, or policy distillation. The case for online use requires evidence that its benefits survive realistic inference delays.

Response: We thank the reviewer for this very constructive comment! A separate runtime is needed because a policy that integrates these capabilities is still a learned policy and degrades outside its training distribution, whereas the runtime's rule-based check does not share its blind spots. Its benefit survives realistic delay. With a 4 s delay, AutoAgent0 still gains +11.4 DS / +16.4 SR, and collisions do not increase (Table A). We agree that its recoveries can also be used offline to improve the policy, which is our next step.

(1) Why keep a separate supervisor instead of integrating its capabilities into the policy or a VLA. Integration puts these capabilities into a learned policy, and a learned policy degrades outside its training distribution. SimLingo already integrates a language model into the policy and still loses 24.3 SR points from the In-Distribution to the Generalization routes (main paper, Table 1). The runtime's verifier is rule-based and uses a separate scene estimate, so it does not share the policy's blind spots. With it, the same drop for SparseDriveV2 shrinks from 21.3 to 4.7 SR points. Driving has a very long tail of rare cases, and some of them will always fall outside what a policy has learned, so this benefit remains however strong the policy becomes. It also does not depend on whether the policy is retrained.

(2) The benefit survives realistic inference delay. Without pausing the simulator and with the answer arriving 4 s later (about the median Opus 5 request time), AutoAgent0 still improves the expert by +11.4 DS / +16.4 SR, and the collision rate stays at the paused level (Table A). This is because the runtime brakes as soon as a plan is rejected, without waiting for the VLM.


Table A. Recovery without pausing the simulator (200 Fail2Drive routes, SparseDriveV2, Opus 5). Delay is set in simulated time, not wall-clock time: traffic keeps moving for this long between a recovery request and its answer. The first two rows are from the main paper (Tables 1 and 3).

SystemDelay (sim. time)DS↑SR (%)↑Col.↓
Standalone expert–66.563.40.445
Brake-only fallback, no recovery–76.074.00.050
AutoAgent0, paused (paper)080.083.40.145
AutoAgent0, not paused1 s79.682.70.152
AutoAgent0, not paused4 s77.979.80.127

(3) Offline labeling, recovery-data generation, and policy distillation. We agree with the reviewer. The current runtime can support these uses, because each recovery is a verified corrective trajectory at a state where the policy failed. Using it to improve the policy is our next step, and the present work focuses on establishing the value of the runtime itself.

AuthorsOpenReview ↗

Response to W2 (2/4)

W2. 2. The methodological distinction from existing slow–fast driving systems is insufficiently established.

Combining a fast driving model with a slower, more capable model has already been explored in SOLVE and ETA, while AdaDrive additionally learns when to invoke the language model and how strongly to incorporate its output. Thus, model complementarity and selective invocation alone do not establish methodological novelty.

AutoAgent0 differs through its frozen-expert runtime, independent scene refinement, shared verification, and feedback-driven recovery. However, the paper does not sufficiently explain which limitations of existing slow–fast systems this particular combination resolves. The comparisons in Section 4.4 primarily concern monitors whose corrective actions are longitudinal; they support the value of richer recovery actions but do not resolve the comparison with existing dual-system approaches. A clearer account of the incremental contribution, supported by comparisons that consider both performance and latency, is needed.

Response: Learned slow–fast systems (SOLVE, ETA, AdaDrive) combine two learned models and trust the result, so outside the training distribution they can neither tell that the result is wrong nor correct it. AutoAgent0 puts an independent verifier around both models. A rejection invokes the slow model, the verifier checks its answer, and the reason for each rejection is fed back to try another maneuver.

We ran the missing comparison with a slow–fast system (Table B). SOLVE, ETA, and AdaDrive must be trained jointly with the VLM, so we use AESOP [1], the closest slow–fast runtime for a frozen expert (asynchronous Qwen3.5-VL-9B, 1 s latency bound), with and without our primitive library, plus a 2D MPC that can steer.


Table B. Runtime methods around the frozen SparseDriveV2 on the 80 In-Distribution/Generalization route pairs of the monitor comparison (main paper, Figure 5(a)), averaged over the two splits. All share the expert, detector, and verifier. Rows marked new were not in the submission.

RuntimeCorrective actionDS↑SR (%)↑
SparseDriveV2–66.763.8
+ RSSbraking69.555.6
+ Longitudinal MPCspeed along the nominal path67.748.8
+ Uncertainty monitordeceleration67.646.3
+ 2D MPC (new)acceleration and steering62.756.3
+ AESOP (new)nominal or stop branch67.166.0
+ AESOP + our primitives (new)our primitive library72.168.4
+ AutoAgent0verified primitives with feedback80.783.8

Learned slow–fast systems execute their output without monitoring or recovery. In SOLVE, ETA, and AdaDrive, the outputs of the two models are fused and executed directly. AutoAgent0 adds these two steps and closes the loop:

  1. Monitoring. They cannot tell when an output is wrong. AutoAgent0 checks every trajectory by rules on an independent scene context, which removes most collisions (main paper, Table 3).
  2. Recovery. They cannot correct a wrong output. AutoAgent0 feeds the rejection reason back to select another maneuver, and completes more routes than a budget-matched retry without feedback (main paper, Table 4).

Performance and latency. The slow model of AutoAgent0 is slower per call, but it is called rarely and is not on the per-step path. The slow model of AESOP is bounded at 1 s, but AESOP solves a branch MPC at every step.


RuntimeDS↑SR (%)↑Slow model invoked onSlow-model latency per callMean latency per step↓
+ AESOP67.166.0embedding anomaly≤ 1 s (bound)≈ 1.2 s
+ AESOP + our primitives72.168.4embedding anomaly≤ 1 s (bound)≈ 1.1 s
+ AutoAgent080.783.8verification failure, 0.56 times per 100 steps≈ 4 s (median)≈ 0.1 s

The mean latency per step is all extra computation of the runtime, slow-model calls included, amortized over the control steps. The conclusion holds when the delay actually takes effect. Without pausing the simulator and with the same 1 s delay, AutoAgent0 still reaches 82.7 SR, and 79.8 at 4 s (our response to W1, Table A). The median wall-clock time of a whole route increases by only 15.7% (main paper, Table 9). With the same Qwen3.5-VL-9B as AESOP, AutoAgent0 loses less than 3 SR points (main paper, Table 5).

[1] Rohan Sinha, Amine Elhafsi, Christopher Agia, Matthew Foutter, Edward Schmerling, and Marco Pavone. "Real-Time Anomaly Detection and Reactive Planning with Large Language Models." RSS 2024. arXiv:2407.08735.

AuthorsOpenReview ↗

Response to Q1 (3/4)

Q1. How does the system perform when the VLM-guided primitive planner serves as the sole planner, while retaining the same perception, verifier, and controller? Comparing driving performance and inference cost would clarify the contribution of the nominal E2E policy and the benefits of combining the two planners.

Response: As the sole planner, the primitive planner reaches only 8.8 SR, against 88.2 for the full system, and each plan update is about 10× slower. The primitives are short maneuvers designed for local recovery and cannot follow a route. The nominal policy handles normal driving (93.6% of the distance), so the two planners are complementary.

We ran this experiment on the 34 threshold-sweep routes with the same perception, verifier, and controller (Table C).


Table C. VLM-guided primitive planner as the sole planner (34 threshold-sweep routes).

PlannerDS↑SR (%)↑Time per plan updateVLM calls
Primitive planner only31.58.8about 4 severy plan update
Full system81.188.2397 ms (nominal plan)only during recovery

Why primitives alone fail. The primitives are local maneuvers of 1–6 s (main paper, Table 10) that cannot follow a route, and every plan update waits for a VLM request.

How the two combine. In the full system, the nominal policy drives 93.6% of the distance, and the VLM is asked mostly when the ego vehicle is already stopped (64% of requests are issued below 1 m/s).

Reviewer_mB6WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AUTOAGENT 0, a runtime framework that combines scene understanding, trajectory verification, and VLM-based recovery to improve autonomous driving in challenging scenarios.

Critical strengths

The experiments on Fail2Drive and NavSafe-∞ show promising improvements in route completion and driving scores. The problem is important, and the paper is generally well organized.

Critical weaknesses

My main concern is whether the improvements can be attributed to the agentic framework itself. In Table 1, the augmented versions of UniAD and SparseDriveV2 use additional LiDAR information, and the system also benefits from improved pedestrian and obstacle detection. These changes could contribute significantly to the performance gains. Although the paper includes component ablations, I would like to see a more direct comparison under identical perception settings.

I am not fully convinced that VLM-based reasoning is necessary for the recovery process. The comparisons with RSS, MPC, and uncertainty-based monitoring are useful, but these baselines have relatively limited recovery capabilities. AUTOAGENT 0 has access to a broader set of maneuvers, which may explain part of the improvement. It would be interesting to compare the proposed approach with a simpler rule-based or search-based recovery method using the same action primitives.

The safety results deserve further discussion. In Table 3, recovery planning improves the success rate from 74.0% to 83.4%, but the collision rate also increases from 0.050 to 0.145 compared with the conservative fallback. Given that safety is a major motivation of this work, I think this trade-off should be examined more carefully. In particular, it would help to understand what types of collisions are introduced by recovery maneuvers and whether some of them could be avoided.

Another concern is the runtime efficiency. The VLM inference takes several seconds, and the simulation is paused while waiting for recovery decisions. This makes the current evaluation somewhat different from real-world driving, where the surrounding environment continues to change. I understand that the authors discuss this limitation, but more evidence is needed to show how the framework could operate under realistic timing constraints.

Questions for authors
  1. How much of the improvement comes specifically from agentic recovery rather than the additional perception modules? Could you provide a comparison using identical sensor inputs and perception models?

  2. Have you considered a non-VLM recovery baseline that searches over the same action primitive library? I think this would help clarify whether agentic reasoning is actually needed.

  3. Table 3 shows that recovery planning improves task completion but also introduces more collisions than conservative fallback. What are the main reasons for these additional collisions? Are they concentrated in particular scenarios or maneuvers?

  4. How would the proposed system handle the several-second VLM inference delay in a real driving environment, where other vehicles and pedestrians continue moving? Would the reported benefits remain if the simulator were not paused?

  5. Were the routes used to select the verifier threshold excluded from the final evaluation? Also, how consistent are the reported gains across different evaluation seeds?

9 comments
AuthorsOpenReview ↗

Response to W1 / Q1 (1/5)

W1. My main concern is whether the improvements can be attributed to the agentic framework itself. In Table 1, the augmented versions of UniAD and SparseDriveV2 use additional LiDAR information, and the system also benefits from improved pedestrian and obstacle detection. These changes could contribute significantly to the performance gains. Although the paper includes component ablations, I would like to see a more direct comparison under identical perception settings.

Q1. How much of the improvement comes specifically from agentic recovery rather than the additional perception modules? Could you provide a comparison using identical sensor inputs and perception models?

Response: Without any trained perception, using only the expert's own sensors and detections, the agentic framework alone brings a large gain (row (a) of Table A). We fine-tuned the perception modules to make perception as accurate as possible, so that missed detections limit the experiments as little as possible.

Why we fine-tuned the perception modules. The runtime can act only on hazards in its scene context. The expert's own detector misses most pedestrians and has no animal class, so with it the experiment would mostly measure detector coverage. The expert and the base detector stay frozen, and the expert never receives the added perception; only the runtime uses it.

We agree that a direct comparison under identical perception is needed and ran it (Table A). Only the scene context given to the runtime changes. Variant (a) uses SparseDriveV2's own detections and cameras, with no LiDAR and no trained modules, and (b) adds the base BridgeDrive LiDAR detector that the full system builds on, still without the trained modules.


Table A. Gain of the runtime under increasingly accurate perception on Fail2Drive (200 routes, frozen SparseDriveV2). The last two rows are from the main paper (Table 3).

Scene context for the runtimeExtra sensorsDS↑SR (%)↑Col.↓ΔDS / ΔSR
Standalone SparseDriveV2–66.563.40.445–
(a) SparseDriveV2 detectionsnone73.172.70.270+6.6 / +9.3
(b) Base BridgeDrive detectorLiDAR75.876.30.230+9.3 / +12.9
FullLiDAR + trained modules80.083.40.145+13.5 / +20.0
Ground-truth contextoracle81.490.30.110+14.9 / +26.9

The gain grows with each step toward more accurate perception, since the runtime can resolve only hazards that appear in its scene context.

Evidence already in the paper.

  1. With perception fixed, adding recovery planning alone raises SR by 9.4 points (main paper, Table 3).
  2. The RSS, MPC, and uncertainty-based monitors, with the same expert and detector, all stay well below AutoAgent0 (main paper, Figure 5(a)).
  3. NavSafe-∞ uses no trained modules, and all three experts still gain at least 12 DS (main paper, Table 2).
AuthorsOpenReview ↗

Response to W2 / Q2 (2/5)

W2. I am not fully convinced that VLM-based reasoning is necessary for the recovery process. The comparisons with RSS, MPC, and uncertainty-based monitoring are useful, but these baselines have relatively limited recovery capabilities. AUTOAGENT 0 has access to a broader set of maneuvers, which may explain part of the improvement. It would be interesting to compare the proposed approach with a simpler rule-based or search-based recovery method using the same action primitives.

Q2. Have you considered a non-VLM recovery baseline that searches over the same action primitive library? I think this would help clarify whether agentic reasoning is actually needed.

Response: A larger maneuver set does not explain the gain. With the same primitives and verifier, selecting the maneuver by search or by rules instead of the VLM completes fewer routes.

We ran the requested non-VLM baselines on the 80 In-Distribution/Generalization route pairs of the monitor comparison (Table B). Only the selection step differs:

  • Search-based. Searches every primitive and executes the admitted candidate with the highest verifier score.
  • Rule-based. Takes the first admitted primitive in a fixed conservative-first order (brake, wait, creep, then lateral and reverse maneuvers).
  • Random. Picks a primitive uniformly and, unlike the others, uses default parameters instead of parameter search.

Table B. Recovery selection on the 80 In-Distribution/Generalization route pairs of the monitor comparison (main paper, Figure 5(a)), with one rollout per route, averaged over the two splits.

Recovery methodDS↑SR (%)↑Col.↓
Random (default parameters)74.073.00.210
Search-based (exhaustive)74.672.40.202
Rule-based (cascade)76.373.70.070
AESOP + our primitives72.168.40.231
VLM (AutoAgent0)80.783.80.138

Why selection matters. The verifier score only checks whether a trajectory is admissible and how much progress it makes, so it favors creeping forward and lane changes. In the same states, the VLM often waits or reverses first. In a separate run with ground-truth perception, exhaustive search with a 6× larger budget still trails AutoAgent0 by about 12 SR points.

Our primitives in another runtime. AESOP [1], a training-free slow–fast runtime, still trails AutoAgent0 when given our primitive library (Table B).

[1] Rohan Sinha, Amine Elhafsi, Christopher Agia, Matthew Foutter, Edward Schmerling, and Marco Pavone. "Real-Time Anomaly Detection and Reactive Planning with Large Language Models." RSS 2024. arXiv:2407.08735.

AuthorsOpenReview ↗

Response to W3 / Q3 (3/5)

W3. The safety results deserve further discussion. In Table 3, recovery planning improves the success rate from 74.0% to 83.4%, but the collision rate also increases from 0.050 to 0.145 compared with the conservative fallback. Given that safety is a major motivation of this work, I think this trade-off should be examined more carefully. In particular, it would help to understand what types of collisions are introduced by recovery maneuvers and whether some of them could be avoided.

Q3. Table 3 shows that recovery planning improves task completion but also introduces more collisions than conservative fallback. What are the main reasons for these additional collisions? Are they concentrated in particular scenarios or maneuvers?

Response: The fallback's lower collision rate comes from standing still, not from driving more safely. It stops at a blockage until timeout, whereas recovery drives past it. Most of the additional collisions occur on routes that the fallback does not complete, and the full system still collides far less than the expert alone (main paper, Table 3).

We traced every collision of the full system to the action that moved the ego vehicle in the 3 s before contact:

  • 44% during a recovery maneuver: half during a lane-borrowing bypass, mainly with oncoming traffic in the borrowed lane, and half low-speed contacts in the tight space around an obstacle.
  • 33% within 5 s after the expert takes back control, e.g., inside a construction zone.
  • 22% unrelated to recovery, no more than with the fallback.

85% are on routes that the fallback does not complete, and most are in Visual–Lateral scenarios, which require driving around an obstacle.

Avoiding bypass collisions. Collisions with oncoming traffic during a bypass can be partly avoided. The original system checks the bypass only when it starts. We now check it at every tick during the bypass and brake on conflict (Table C).


Table C. Results on the 26 routes that include a bypass.

VariantDS↑SR↑Col.↓Collisions with borrowed-lane traffic (count)↓
Full AutoAgent049.342.30.5005
+ per-tick check during bypass50.242.30.4232

These collisions drop from 5 to 2 at the same SR. In the remaining 2, the ego vehicle is stopped in the borrowed lane and hit by an oncoming vehicle. Avoiding them requires returning to its own lane.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To all reviewers

We thank all reviewers for their thoughtful suggestions, which are very valuable to us! We will incorporate them into the final version of the paper. The weaknesses and questions were laid out very clearly, so we have responded to each of them point by point under the corresponding review. Please see our individual responses for details. Below we briefly summarize our contributions and the new experiments.

[Our Contributions]

We are glad that the reviewers generally acknowledge our contributions:

  • An agentic runtime around frozen driving experts [y8hN, v9j7]
  • VLM maneuver reasoning gated by rule-based verification [y8hN, v9j7]
  • Consistent gains across experts and benchmarks without retraining [mB6W, y8hN]
  • Thorough ablations [v9j7]

[New Experiments]

The new experiments in this rebuttal address the following concerns:

  • Whether the benefit holds under realistic VLM latency and with a small local model [mB6W, y8hN, v9j7]
  • Whether the gain comes from the agentic framework rather than extra perception, and how consistent it is across seeds [mB6W]
  • Whether VLM reasoning is necessary for recovery, compared with rule-based and exhaustive search [mB6W, v9j7]
  • How AutoAgent0 differs from slow–fast systems, and what the nominal policy contributes [y8hN]
  • Where the remaining collisions and failures come from, and how often recovery search fails [mB6W, y8hN, v9j7]

We thank all reviewers again for their time and effort!

AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

Dear AC and all reviewers,

We sincerely thank the AC and all reviewers for the time and effort devoted to our paper. In addition to our point-by-point responses to each reviewer, we summarize the rebuttal period below: the outcome of the discussion, the contributions recognized by the reviewers, the new experiments, and several important clarifications.

[Outcome of the discussion]

All three reviewers have responded:

ReviewerRatingConfidenceStatus
mB6W2 → 33raised, pending update
y8hN2 → 33raised
v9j732maintained

[Contributions recognized by the reviewers]

  • A new direction for closed-loop driving: bringing agentic workflows around existing driving experts [y8hN, v9j7]
  • VLM-based high-level reasoning with rule-based verification as the safety gate [y8hN, v9j7]
  • Consistent improvements across multiple experts and two benchmarks without retraining [y8hN], on an important problem [mB6W]
  • Thorough ablations [v9j7]

[New Experiments & Analyses]

  • Evaluation without pausing the simulator, under different VLM delays, and with a local 2B model [mB6W, y8hN, v9j7]
  • Comparison under identical perception, using only the expert’s own sensors and detections [mB6W]
  • Non-VLM recovery baselines (search-based, rule-based, random) with the same primitives and verifier [mB6W]
  • Exhaustive search with ground-truth perception and a larger search budget [v9j7]
  • Comparison with a slow–fast runtime (AESOP) and a 2D MPC, in both performance and latency [y8hN]
  • Primitive planner as the sole planner [y8hN]
  • Tracing of all collisions, and a per-tick check during bypass maneuvers [mB6W]
  • Classification of all failed routes [y8hN]
  • Frequency and impact of recovery-search failures [v9j7]
  • Results without the threshold-ablation routes, and variance across seeds [mB6W]

[Important Clarifications]

  • Deployment target: AutoAgent0 targets on-vehicle execution; offline use of its recoveries is a natural extension, not the present target [y8hN, v9j7]
  • Effect of VLM latency: without pausing the simulator during recovery VLM requests, the gain is largely preserved and the collision rate stays at the same level [mB6W, y8hN, v9j7]
  • Separate supervisor vs. integrated policy: an integrated policy is still learned and degrades out of distribution, whereas the rule-based verifier does not share its blind spots [y8hN]
  • Novelty over slow–fast systems: AutoAgent0 closes the loop around both models: an independent verifier checks every trajectory, and its rejections drive the recovery [y8hN]
  • Use of the added perception: the expert stays frozen with its original inputs; only the runtime uses the added perception [mB6W]
  • Collision rate of recovery vs. fallback: the fallback collides less because it stays stopped at blockages, not because it drives more safely; the full system still collides far less than the expert alone [mB6W]
  • Choice of the verifier threshold: it follows from the verifier's score formula and was not tuned on evaluation routes [mB6W]
  • Source of recovery trajectories: they are generated by the primitive library, not sampled from the expert; the runtime brakes and plans again if none passes [v9j7]

We thank the AC and all reviewers again for their time, and we are glad that Reviewers mB6W and y8hN found their main concerns addressed and raised their ratings. AutoAgent0 is among the first works to bring an agentic runtime to closed-loop driving. We see runtime supervision as an important direction that is complementary to advances in the driving policy itself, and we hope this work encourages more future research on agentic driving.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.