- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents AutoAgent0, an agentic runtime harness to augment frozen pre‑trained driving experts (e.g., SparseDriveV2, UniAD, BridgeDrive) online via a pre-trained VLM. The system consists of three core components:
Scene‑context refinement: Cross‑modal perception together with VLM review to discover obstacles and refine scene representation for verification.
Rule‑based trajectory verifier: A rule‑based verifier checks driving criteria and rejects unsafe nominal trajectories proposed by the base driving expert.
Agentic recovery planning: When trajectories are rejected, a VLM selects maneuver primitives from a predefined action primitive library; parameters are sampled to generate candidate trajectories, which are iteratively refined under verifier feedback until an admissible trajectory is found.
Closed-loop evaluations on Fail2Drive and NavSafe show that augmenting existing driving experts with AutoAgent0 improves driving score (DS) and route success rate (SR), especially for long‑tail behavioral scenarios.
Critical strengths
Novel perspective on introducing agentic workflow for autonomous driving system:
The idea of using agentic systems to wrap multiple frozen driving experts is practically inspiring. In real‑world autonomous driving products, the typical solution maintains multiple specialized expert models that handle various driving scenarios. An agentic orchestrator could unify and dispatch these experts for long‑tail corner cases, a promising research direction beyond pure end‑to‑end model training.
Meaningful exploration of bringing LLM capabilities into closed‑loop driving:
Most prior works focus on the neural planner itself. This work explores a new paradigm: use a VLM agent for high‑level maneuver reasoning, while keeping deterministic rule‑based verification as the safety gate. It opens new thinking for handling long‑tail failures that traditional brake‑only monitors cannot resolve.
Thorough ablations:
The authors perform rich ablations to decompose contributions from scene‑context refinement, rule‑based verification and feedback‑guided recovery. They also conduct well-structured ablations on different scenarios.
Critical weaknesses
Ambiguity on system deployment target leads to potential unfair comparison setting:
It remains ambiguous whether AutoAgent0 is designed as an offline cloud‑simulation tool or on‑vehicle online real‑time runtime.
- If it targets offline simulation experts: comparisons should be made against oracles with perfect perception and exhaustive rule‑based maneuver search baseline, rather than real‑time on‑vehicle driving experts.
- If it targets on‑vehicle online execution: the current evaluation heavily relies on heavy‑weight remote LLM backends (Opus5) with large millisecond latency per recovery request. If the platform is cut-edge platform like thor-u or 8650, the latency will increase significantly. The paper lacks sufficient validations on small‑size on‑device LLMs (e.g., 2B) to prove the agentic pipeline can work under an on‑vehicle latency budget.
Without clarifying its deployment goal, the comparison between AutoAgent0 and vanilla real‑time driving experts is hard to interpret and may be unfair.
Guarantee of valid recovery trajectory is not well‑addressed:
The agentic recovery loop selects primitives and samples parameters within a limited search budget. There is no mechanism / theoretical guarantee that at least one feasible collision‑free trajectory exists within the primitive library and parameter search space. When all sampled candidates get rejected, the system falls back. However, the paper lacks quantitative analysis on how frequently such “complete recovery search failure” happens, and how it impacts end-to-end closed-loop metrics.
Questions for authors
Regarding the Action Primitive Library and recovery search process:
Could you elaborate on the Action‑Primitive library workflow? The details on sampling primitive trajectories from a single SparseDriveV2 are not clear. How are the parameters sampled? Moreover, when recovery planning is triggered after all nominal proposals from SparseDriveV2 are rejected, does the recovery loop still sample trajectories from SparseDriveV2?
6 comments
Response to W2 (2/3)
W2. Guarantee of valid recovery trajectory is not well-addressed: The agentic recovery loop selects primitives and samples parameters within a limited search budget. There is no mechanism / theoretical guarantee that at least one feasible collision-free trajectory exists within the primitive library and parameter search space. When all sampled candidates get rejected, the system falls back. However, the paper lacks quantitative analysis on how frequently such "complete recovery search failure" happens, and how it impacts end-to-end closed-loop metrics.
Response: When no recovery candidate passes the verifier, AutoAgent0 falls back to braking and plans again. This happens in about one in five recovery searches and costs at most 3.2 pp SR.
The fallback is a 1 s full brake, after which the runtime plans again. We measured how often it is triggered and what follows (Table C).
Table C. Recovery searches in which no candidate passes the verifier (200 Fail2Drive routes).
| Measure | Value |
|---|---|
| Recovery searches in which no candidate passes | 21% |
| After the brake: a recovery maneuver passes | 64% |
| After the brake: a SparseDriveV2 plan passes again | 30% |
| Routes with contact during the brake | 1.2% |
| Routes that time out with the case unresolved | 2.0% |
| Simulated time spent in these brakes | 1.8% |
What happens after the brake. Most of these cases occur at speed, when a hazard first comes into range and no maneuver passes. A full stop is then the right response. Once the vehicle has stopped, a later search usually succeeds. Contact during the brake occurs mainly when the hazard is detected inside the stopping distance. No maneuver could avoid it at that point either, so these cases come from late detection.
Search budget and perception. A 6× larger search budget leaves SR unchanged (Table B), whereas ground-truth perception removes about two thirds of these cases.
Response to Q (3/3)
Q. Regarding the Action Primitive Library and recovery search process: Could you elaborate on the Action-Primitive library workflow? The details on sampling primitive trajectories from a single SparseDriveV2 are not clear. How are the parameters sampled? Moreover, when recovery planning is triggered after all nominal proposals from SparseDriveV2 are rejected, does the recovery loop still sample trajectories from SparseDriveV2?
Response: SparseDriveV2 is not used during recovery. It supplies only the nominal proposals, and recovery trajectories are computed by the primitive library directly from the current scene geometry.
A recovery decision proceeds as follows.
- Trigger. Recovery starts when the verifier rejects all SparseDriveV2 proposals, or when the ego vehicle makes no progress for 5 s although a proposal passes (main paper, Figure 5(c)).
- Select primitives. The VLM is given the camera images, the scene context, the verifier's rejection reasons, and the available primitives (main paper, Appendix D.3). It returns up to five primitives with probabilities.
- Sample parameters and generate trajectories. Each primitive has predefined parameters (lateral offset, speed, etc.) with bounded ranges (main paper, Table 10). The runtime samples them on a grid within these ranges, with more samples for primitives the VLM rates higher. Each parameter set yields one candidate trajectory, by a kinematic-bicycle rollout or, for lane changes and bypasses, from HD-map lane geometry.
- Verify. The same verifier checks every candidate, and the highest-scoring one that passes is executed.
- Revise. If none passes, the rejection reasons go back to the VLM, which selects other primitives. After at most three VLM calls without success, the runtime brakes (see our response to W2).
- Hand back. SparseDriveV2 is not queried while the maneuver runs. Afterwards it proposes again and takes back control once one of its proposals passes the verifier.
Response to W1 (1/3)
Response: AutoAgent0 targets on-vehicle execution. With a local 2B model and no simulator pause, it still improves SparseDriveV2 by +6.7 DS / +11.3 SR, with a collision rate on par with Opus 5 (Table A).
On-vehicle setting. The VLM is not in the control loop. The ego vehicle brakes as soon as a plan is rejected, so a late VLM answer only means waiting longer. As a result, raising the Opus 5 delay from 1 s to 4 s lowers SR by only 2.9 points, with no more collisions. The 2B model has a collision rate on par with Opus 5, and its lower DS and SR come from resolving fewer blockages.
Table A. Recovery without pausing the simulator on 200 Fail2Drive routes. Delay is set in simulated time, not wall-clock time: traffic keeps moving for this long between a recovery request and its answer.
Offline setting. As suggested for the offline target, both methods use ground-truth perception, and exhaustive search gets a 6× larger budget (Table B). Exhaustive search still trails AutoAgent0. The verifier score does not reflect which maneuver suits the scene, so a larger budget only finds more candidates of the same kind.
Table B. Recovery with ground-truth perception on the 34 threshold-sweep routes (3 seeds). Budget is the parameter-search budget relative to the default.
The comparison with real-time experts is fair. All methods are evaluated under the same protocol, in which the simulator waits for planning at every step. SparseDriveV2 itself is not real-time, since its planning step takes a median of 397 ms against a 50 ms control tick (main paper, Table 7). The only extra wait for AutoAgent0 is the recovery request, and in Table A the simulator is not paused for it.