- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
To address the local-to-global (L2G) context mismatch problem in autoregressive traffic simulators, this paper propose CRAFT, a plug-and-play, simulator-agnostic framework that trains a Contextual Preference Evaluator (CPE) on grouped closed-loop rollouts with traffic-rule and driving-quality labels, then reweights candidate actions via one-step lookahead at inference without retraining the base models. On the Waymo Dataset, CRAFT reduces scene-level collision, offroad, and traffic-violation rates across three state-of-the-art simulators (SMART, CAT-K, RLFTSim).
Critical strengths
- The paper formalize the L2G mismatch with a formal derivation showing that fitting incomplete-context demonstrations yields a marginal distribution that necessarily diverges from scene-specific behavior, and validates the causal role of context via a controlled ablation that varies visible agents during training.
- By decoupling behavioral correction (CPE) from the motion prior (base simulator), a single CPE checkpoint improves three architecturally distinct SOTA simulators without retraining or modifying them. This cross-model transferability and low deployment cost is practically valuable in traffic simulation.
Critical weaknesses
- CRAFT is a re-ranking mechanism, not a generator. It can only select among the K=32 candidate actions the base simulator proposes. If the proposal distribution contains no suitable action, CRAFT cannot create it. The improvement is thus bounded by the base model’s coverage and may fail entirely on long-tail extreme scenarios.
- Test-time guidance evaluates single-step outcomes, so it cannot anticipate multi-step cascading effects, which is a conservative action now may steer the scene into a more dangerous interaction later. No experiments vary lookahead depth (2 or 5 steps) to test whether deeper planning would help.
- All experiments run only on the Waymo Open Motion Dataset. No validation on additional dataset such as nuScenes, Argoverse. It is not clear whether CPE’s learned notion of “contextual appropriateness” transfers across geographies.
- CPE’s supervision derives entirely from a hard-coded rule-based auto-labeler. These rules are noisy, e.g., a legitimate yield-to-pedestrian or courtesy stop could be mislabeled as abnormal stopping, and they cannot capture subtler misbehavior (tailgating, unsignaled lane changes). The paper reports no human-verified label accuracy nor an analysis of how label noise affects CPE.
- The default inference latency is very high, while low latency comes from chunked batching of 32 candidates and sacrifices some diversity. For massively parallel simulation (hundreds of agents), running K lookahead evaluations every step still imposes significant aggregate compute.
Questions for authors
- How robust is CRAFT when the base simulator's proposal distribution is itself largely dominated by inappropriate actions. If a base simulator were poor, could CRAFT still provide meaningful improvement, or does it fundamentally inherit the base model’s blind spots?
- What does the learned contextual appropriateness actually capture beyond the six hard-coded rules? Since CPE is trained only on rule-based labels, can it generalize to judge behaviors the auto-labeler never explicitly encodes? Could a human-annotated or VLM-derived preference set change what the evaluator learns?
- How sensitive are the results to the choice of the auto-labeling criteria and the weighting hyperparameters?
3 comments
W4. CPEʼs supervision derives entirely from a hard-coded rule-based auto-labeler. These rules are noisy, e.g., a legitimate yield-to-pedestrian or courtesy stop could be mislabeled as abnormal stopping, and they cannot capture subtler misbehavior (tailgating, unsignaled lane changes). The paper reports no human-verified label accuracy nor an analysis of how label noise affects CPE.
R4. We agree that rule-based annotations inevitably contain some noise. We therefore report the results of a manual audit of 480 randomly sampled rollouts from the 480,000 generated rollouts. Missed abnormalities occurred in 17 rollouts (3.54% of the audited sample): 14 abnormal stops and three traffic-signal violations. The relatively few errors found in the audit, together with the training ablation and closed-loop improvements (Fig. 5 and Table 1), suggest that the observed label noise does not pose a major obstacle to effective CPE training and online guidance.
W5. The default inference latency is very high, while low latency comes from chunked batching of 32 candidates and sacrifices some diversity. For massively parallel simulation (hundreds of agents), running K lookahead evaluations every step still imposes significant aggregate compute.
R5. CRAFT supports low-latency guidance: with a candidate chunk size of 32, lookahead takes only 3.97 ms (Fig. 8(a)), and a full simulation step takes 43.11 ms (Table 13).
Chunking partitions the candidate action pool, not the agents within a scene. Each candidate retains all simulated agents, and all candidates are evaluated and normalized over the complete pool. For the same candidate samples and sampling randomness, different chunk sizes are mathematically equivalent and yield the same action-selection results, without sacrificing diversity.
The full-step latency and memory usage by agent count are:
| Agents per scene | Full-step latency (ms) | Memory (MiB) |
|---|---|---|
| 10–39 | 35.0 | 884.1 |
| 40–79 | 69.6 | 2988.0 |
| 80 or more | 91.0 | 5449.2 |
Even for scenes with 80 or more agents, the average full-step latency is 91 ms, which we consider acceptable for closed-loop simulation. Further runtime analysis is provided in §5.5 and Appendix F.2 (Table 13).
Q2. What does the learned contextual appropriateness actually capture beyond the six hard-coded rules? Since CPE is trained only on rule-based labels, can it generalize to judge behaviors the auto-labeler never explicitly encodes? Could a human-annotated or VLM-derived preference set change what the evaluator learns?
R6. CPE learns continuous, context-dependent preferences from the rule labels, rather than adding new behavioral criteria beyond its supervision. The six rules cover frequent failures observed in our rollouts. Relative preference learning provides finer scores for candidate selection, and Fig. 7(a) shows sensitivity to interaction context.
The preference-learning mechanism is not restricted to this source. Future work could use Human and VLM-generated comparisons to supply interaction judgments that hand-coded rules do not readily express, with human spot checks to verify reliability. Such preferences could change what CPE learns by broadening its supervision.
Q3. How sensitive are the results to the choice of the auto-labeling criteria and the weighting hyperparameters?
R7. The choice of auto-labeling criteria can influence what CPE learns. We selected six criteria to capture the main forms of contextual abnormality observed in our rollouts, enabling CPE to learn the corresponding behavior–context mismatches. The results demonstrate that this choice provides a reasonable implementation of CRAFT, with promising performance in mitigating the L2G gap through contextual correction. Further investigation of the criteria and their thresholds could help refine the supervision provided to CPE and improve CRAFT’s performance.
In terms of weighting hyperparameters, for training, Table 5 reports the loss weights, and Fig. 5 ablates the corresponding objectives. Both intra-trajectory and inter-rollout preference learning improve performance, with their combination achieving the best result. For inference, we analyze two settings: gives the best tested behavioral performance; stronger guidance worsens all three event rates (Fig. 8(b)). CRAFT improves over the matched CAT-K baseline at every tested , with its highest closed-loop score at (Fig. 8(c)).
Request for Reply
Dear Reviewer w4cx,
Thank you for your review and helpful suggestions. We’d like to highlight four results from our revision:
- Re-ranking helps even when the base simulator struggles, with the largest gain in the lowest-scoring scenes.
- One-step lookahead offers the better runtime trade-off, as deeper lookahead yields only modest gains.
- Zero-shot guidance on Argoverse reduces collisions and offroad events.
- Latency and GPU memory are measured across agent counts, with batching preserving the full candidate pool.
If you have any further concerns, please let us know and we are happy to address them. We would welcome any further feedback and would appreciate your taking these results and clarifications into account in your final assessment.
Best regards, Submission25923 Authors
Thank you for the constructive questions about candidate coverage, longer-horizon interactions, transfer, and supervision. We have submitted the revised manuscript, with changes highlighted in blue. And we address each concern below.
R1. Like other proposal-based test-time methods, CRAFT relies on the base model's action distribution. At K=32, CAT-K achieves a 95.14% logged-token recall on the evaluated logged states (Appendix F, Fig. 15), indicating that its candidate pool usually contains demonstrated actions available for selection.
We further examine the distribution of unguided CAT-K closed-loop scores to assess correction when the base simulator performs poorly. The rare, low-scoring group contains 31 of the 1,000 scenes (3.10%). Despite the poor baseline performance in this group, CRAFT improves its average score by 14.20 points, the largest gain among the three groups, while also improving the intermediate and common groups.
We have added this breakdown to Fig. 9(a) and its analysis to §5.6.
Appendix E, Fig. 13, shows CRAFT recovering from an initial perturbation and maintaining road alignment through a sharp turn. By selecting better available actions, CRAFT improves subsequent scene states even when the base rollout is poor. Appendix F, Fig. 14 shows that recovery remains limited when suitable candidates are persistently absent.
R2. We agree that one-step scoring does not explicitly evaluate all later consequences. Following your suggestion, we compare one-, two-, and three-step lookahead with candidate chunk size fixed at 4.
Deeper lookahead adds substantial cost for only marginal benefit: three steps require 2.26 times the runtime for a 0.63-point score gain. With a fixed sampling budget, the longer horizon still covers only a small subset of possible future interactions, limiting the additional information available for choosing the current action. These results support one-step lookahead as our practical default. CRAFT reevaluates candidates after every executed action, allowing repeated correction as the scene evolves.
R3. The WOMD-trained CPE also improves simulation on Argoverse without adaptation. We use WOMD as the primary benchmark because its scale, diverse driving interactions, and dedicated Sim Agents task make it particularly well suited to multi-agent simulation, whereas the relevant Argoverse and nuScenes benchmarks primarily target trajectory forecasting.
Following your suggestion, we evaluate the WOMD-trained CAT-K simulator and CPE on 500 randomly sampled Argoverse scenes, without training or tuning on Argoverse.
Both rates improve under this dataset shift, supporting the transfer of learned contextual preferences beyond WOMD. Additionally, traffic violations are not evaluated because these inputs lack traffic-signal states. We have added the results as Table 3 and discussed this transfer in §5.6.