- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents Scenarist, a language-to-policy benchmark that reshapes the long-tail driving data distribution by compiling natural-language failure-scenario intents into rule programs built from reusable map, signal, and interaction primitives. It connects four interdependent links to produce action-aligned supervision for VLA policies via positive-sample SFT and negative-sample RL. On 16,397 interactions across eight failure intents, VLA post-training raises safe rate to 98.58% while preserving general driving on NAVSIM.
Critical strengths
- Compiling language into reusable rule primitives with deterministic validity checks, rather than enumerating a fixed scenario catalog is an advance over catalog-based and pure-adversarial approaches.
- The end-to-end four-link integration has measurable policy gains with collision reduction and safety improvement.
Critical weaknesses
- The VLA evaluation is not a true closed-loop test. The surrounding agents follow fixed logged trajectories and do not react online, and observation updates/replanning are not verified. In open-loop/rollout settings, collision and safe-rate metrics are far easier to game (e.g., by being conservative). The performance is not validated under the realistic closed-loop conditions the benchmark’s motivation demands.
- On WOMD validation (Table 3), all-agent collision only drops from 0.042 to 0.037 and off-road from 0.122 to 0.118 which is very marginal, possibly within noise. The narrative “improves long-tail safety while retaining routine performance” thus rests on long-tail (in-domain, held-out) gains plus NAVSIM parity, with little evidence the synthesized data transfers to genuinely broader general driving.
- The paper does not benchmark against ChatGPT-era scenario generators (e.g., ChatScene, STRIVE) or prior behavior-aware synthesis pipelines on the same long-tail safety metrics. It is unclear whether the gains come from the novel full-chain design or simply from adding more in-distribution adversarial training data.
- The paper did not isolate how rendering quality affects downstream VLA safety, i.e., whether 1.42% collision relies on high-fidelity observations or would hold with weaker rendering is unknown. The rule-based automatic labeler also injects unmeasured label noise.
- Data construction relies on Qwen3-4B generating rule programs and participant behaviors, with LLM-revision loops. The paper reports no success rate, rejection rate, or manual audit of the generated scenarios’ correctness/safety.
Questions for authors
- Would the VLA safety gains survive a genuinely closed-loop evaluation?
- How much of the gain is attributable to the novel four-link design versus simply adding more adversarial data?
- How reliable and controllable is the LLM-based scenario-construction stage?
2 comments
Response to Reviewer Lu6U (Part 2 of 2, Q4–Q5)
Q4. How does Scenarist compare with prior scenario generators on long-tail safety metrics? Do the gains come from the full four-link design, or can they be explained by simply adding more in-distribution adversarial training data?
R4. To compare long-tail supervision sources at a fixed training budget, we randomly sampled 1,000 long-tail examples from the official WOD-E2E training split and 1,000 from Scenarist. Both RAP-DINO SFT runs used the same initialization, input and trajectory preprocessing, prediction horizon, and training settings. We trained each for 80 epochs and evaluated both fine-tuned models and the original RAP-DINO on the same nuScenes validation split.
| Method | Collision rate ↓ | Off-road rate ↓ | Safe rate ↑ | ADE@5s ↓ |
|---|---|---|---|---|
| Original RAP-DINO | 10.10% | 0.43% | 89.49% | 3.007 m |
| WOD-E2E real long-tail SFT (1k) | 11.36% | 6.52% | 82.68% | 4.092 m |
| Scenarist SFT (1k) | 5.72% | 0.22% | 94.06% | 2.406 m |
Relative to the original RAP-DINO, Scenarist SFT reduced the collision rate by 4.38 percentage points. WOD-E2E SFT increased the collision rate by 1.26 percentage points and also increased the off-road rate and ADE@5s. Scenarist SFT achieved a safe rate 11.38 percentage points higher than WOD-E2E SFT. This matched-budget SFT comparison supports the downstream value of the trajectory–observation supervision produced by Scenarist’s four-link pipeline.
We also evaluated Scenarist’s generated traffic rollouts on nuScenes using STRIVE’s collision metrics, with STRIVE’s published results as a reference. These reference values come from STRIVE’s full traffic motion prediction model evaluated on cars and trucks in Table 5 of its supplementary material.
| Method | Vehicle collision rate ↓ | Environment collision rate ↓ |
|---|---|---|
| STRIVE | 5.60% | 10.60% |
| Scenarist | 0.56% | 7.37% |
Relative to STRIVE’s published figures, Scenarist’s vehicle and environment collision rates were 90.0% and 30.5% lower, respectively. We also compared Scenarist with ChatScene on the evaluated CARLA examples. Each method was evaluated for 40 episodes using the same scenarios and the same ego controller.
| Method | Ego collision rate ↓ | Red-light violations ↓ | Stop-sign violations ↓ | Mean off-road distance ↓ |
|---|---|---|---|---|
| ChatScene | 5.00% | 0 | 0 | 0.1880 m |
| Scenarist | 0.00% | 0 | 0 | 0.0286 m |
On these examples, Scenarist reduced the ego collision rate by 5.00 percentage points and the mean off-road distance by 84.8% relative to ChatScene. The reported collision rates correspond to ego collisions in 2/40 episodes for ChatScene and 0/40 for Scenarist. R5 further compares execution pass rates and intent match rates. These results demonstrate the advantages of our method over prior approaches.
Q5. How often do LLM-generated scenarios pass execution checks, and how reliably do they match the requested interaction intent? Does the audit quantify safety-label noise?
R5. We randomly sampled 500 cases with balanced coverage across the eight interaction intents. Scenarist allows up to 30 script revision rounds, and ChatScene follows its original single-pass setting.
| Method | Execution pass / rejection | LLM intent match ↑ | Human intent match ↑ |
|---|---|---|---|
| ChatScene | 22.4% / 77.6% | 0.8% | 0.6% |
| Scenarist 4B | 98.2% / 1.8% | 66.4% | 70.4% |
| Scenarist 27B | 99.8% / 0.2% | 83.6% | 86.6% |
The 4B configuration used in the paper achieved a 98.2% execution pass rate and a 70.4% human intent match rate. With 27B, these rates rose to 99.8% and 86.6%, respectively. The execution pass rate measures whether scripts run, and the intent match rate measures whether the resulting behavior realizes the requested interaction.
For the safety-label audit, we randomly sampled 300 ego trajectories labeled safe before SFT from Scenarist-2K, which contains 2,144 constructed interactions. We manually checked each trajectory for ego collisions and road departures over its full prediction horizon. The observed false-safe rate was 0% (0/300). These results also show that our method substantially outperforms the existing baseline in both execution pass rate and intent match rate.
Response to Reviewer Lu6U (Part 1 of 2, Q1–Q3)
We thank the reviewer for the constructive feedback. Scenarist’s core contribution is a language-to-policy benchmark connecting controllable scenario construction, responsive traffic generation, safe-response exploration, and trajectory-aligned RGB synthesis for VLA post-training, improving long-tail safety while retaining measured routine-driving performance. Anonymous pipeline demo.
Q1. Do VLA safety gains persist with updated observations and closed-loop replanning?
R1. We evaluated RAP-DINO before and after Scenarist post-training in AlpaSim across eight long-tail categories in Nurec_eval. Each setting was evaluated on the same 59 episodes. The ego vehicle replanned from updated observations, while surrounding actors followed their recorded trajectories.
The full Scenarist system combines the GRPO-response SFT checkpoint with a PPO-trained trajectory router. Compared with the model fine-tuned on rule-based responses, the full system increased mean driving progress from 44.99% to 55.59% and reduced the collision rate from 54.24% to 50.85% in the tested ego closed-loop setting. The reported collision rates correspond to collisions in 35/59 episodes for the RAP-DINO baseline, 32/59 for the rule-based SFT model, and 30/59 for Scenarist.
Q2. Do the all-agent WOMD gains exceed seed variation? Does Scenarist post-training transfer to independent real-world scenarios?
R2. We report WOMD validation results from Table 3 alongside ego-level metrics and a variant trained with an environmental safety reward.
Across five WOMD validation seeds, the SceneStreamer baseline’s max–min spreads were 0.000417 for collision and 0.000678 for off-road. The reductions in Table 3, 0.005 for collision and 0.004 for off-road, exceed these baseline spreads. The environmental safety reward variant yields lower point estimates for all-agent and ego-level safety metrics.
For independent real-world transfer, we compared RAP-DINO before and after Scenarist post-training on the same WOD-E2E validation frames, without further fine-tuning.
RFS improves in all four categories, providing independent evidence of VLA transfer.
Q3. How sensitive is the reported 1.42% collision rate to rendering quality?
R3. We ablated RGB resolution against the Table 4 baseline (24.96% collision, 0.27% off-road).
At 624 × 360, collision rises to 2.96%, still 88.1% below baseline, retaining approximately 93.5% of the original-resolution collision-rate reduction; off-road is 0.08% versus 0.27%. All three tested resolutions retain substantial safety gains, showing that most benefits persist with reduced visual detail.