acceptodds
Withdrawn from ICLR 2027

Hierarchical Safety-Oriented Dense RL with Refined Credit Assignment for Tokenized Multi-Agent Traffic Simulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-fidelity simulation of safety-critical traffic interactions is essential for developing and evaluating autonomous driving systems. Tokenized autoregressive models achieve strong average realism in multi-agent traffic simulation through large-scale supervised pretraining, but safety-critical events such as collisions are rare in natural driving data, providing limited supervision for learning these interactions. Uniformly sampled reinforcement learning (RL) post-training also allocates many updates to ordinary scenarios, agents, and timesteps that provide little safety information, limiting the efficiency of safety-oriented learning. We propose a hierarchical safety-oriented dense learning framework for RL post-training of tokenized multi-agent traffic simulation models. Episode-level densification prioritizes scenarios whose current-policy probe produces a collision, while state-level masks localize updates to vehicle agent-time positions with a collision or physical near miss in at least one same-scenario rollout, retaining aligned safe outcomes for comparison. Our refined credit assignment retains physical rewards at agent-step granularity and compares aligned agent-time positions across same-scenario rollouts, yielding critic-free group-relative advantages with explicit temporal-agent resolution. Experiments with SMART on the Waymo Open Motion Dataset show that, across three training seeds, our method reduces mean collision rate from 3.394% to 2.914% relative to the uniformly sampled RL baseline using the same refined credit assignment, while maintaining a comparable average displacement error, achieving a favorable balance between safety-oriented alignment and per-rollout trajectory fidelity.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Traffic Simulation, Reinforcement Learning Post-Training, Dense Reinforcement Learning, Group Relative Policy Optimization, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_DtshOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a new rl-based post-training method for traffic simulation to reduce collision. Tthe paper proposes aligned agent-time group-relative credit assignment. Rather than computing one scalar reward per rollout and broadcasting its advantage to every token, physical rewards are maintained at agent-step resolution and normalized across same-scenario rollouts at each aligned position. On WOMD with SMART-7M, the three-seed mean collision rate decreases from 3.394% for uniformly sampled refined RL to 2.914% for Dense RL, with ADE increasing only slightly from 2.0805 m to 2.1055 m.

Critical strengths
  1. The code is provided.
  2. The video is clear.
  3. The writing is good and easy to follow.
Critical weaknesses
  1. The primary comparison is against a rather weak RL baseline. Published traffic post-training methods such as SMART-R1, RLFTSim, R1Sim are not directly evaluated under the same CR/OR/ADE protocol.
  2. The improved metrics are limited to collision and ADE. The method substantially degrades overall WOSAC realism relative to the pretrained SMART model. Reducing only collision and ADE is not hard in traffic simulation. It can lead to wired behavior such as off-road or slow driving.
  3. Important related works are missing. The paper should discuss spatial-temporal credit decomposition methods such as STAS (Spatial-Temporal Return Decomposition for Solving Sparse Rewards Problems in MARL), as well as traffic-specific post-training approaches including DecompGAIL and multi-agent AIRL for intersection simulation.
Questions for authors
  1. How accurately does the physical mask approximate the theoretical informative-state set? Is there any empirical relation between mask membership and action-sensitive return variance?
  2. How much overlap exists between the collision-mined scene sets across consecutive epochs?
  3. Can the authors test multiple probe rollouts per scenario and report the stability of collision-based scene labels?
Reviewer_apMwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a hierarchical dense RL framework for post-training tokenized multi-agent traffic simulation models. It proposes a two-level densification strategy: episode-level selection to prioritize scenarios generating collisions under the current policy, and state-level masking to isolate updates to specific agent-time positions involved in collisions or near misses. This is paired with a refined credit assignment mechanism that normalizes local physical rewards across aligned agent-time positions from same-scenario rollouts, mitigating the resolution loss inherent in rollout-level reward broadcasting. The framework is validated using the SMART model on the Waymo Open Motion Dataset, demonstrating a reduction in mean collision rate while preserving overall trajectory fidelity.

Critical strengths

The hierarchical approach effectively bridges scenario-level and agent-step-level rarity without breaking the joint transitions required for coupled multi-agent dynamics. This structural choice is highly consequential, as it allows the model to leverage rare critical events without deteriorating general driving behavior.

Critical weaknesses
  1. The framework's evaluation is narrowly restricted to a single backbone. Furthermore, while the collision rates improve, the official WOSAC Realism Meta Metric decreases compared to the baseline. This performance drop highlights a nontrivial safety-realism trade-off that undermines the practical efficacy of the proposed post-training objective.

  2. The proposed critical-state masks rely on collisions and heuristic near-miss criteria as proxies for informative policy-gradient contributions. As acknowledged in the manuscript, combining these proxies with immediate-reward normalization and PPO clipping yields a biased surrogate objective. Consequently, the theoretical framework for ideal dense learning fails to mathematically guarantee that the practical masking strategy consistently improves the desired policy gradient.

Questions for authors
  1. Can the authors provide results on the official WOSAC leaderboard to enable a comprehensive comparison against other state-of-the-art methods, rather than relying exclusively on the customized validation set?

  2. Can the authors provide direct empirical evidence linking the explicitly selected agent-time positions to the observed safety improvements, to better isolate the effect of the heuristic masks?

  3. How sensitive is the safety-fidelity trade-off to the specific parameterizations of the three-circle geometric collision proxy, and have the authors identified edge-case interactions where this heuristic incorrectly masks out legitimate safety-critical states?

  4. Does the aligned agent-time credit assignment mathematically and practically preserve its credit-localization benefits if adapted for a non-autoregressive or diffusion-based traffic simulator?

Reviewer_8AtPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hierarchical Dense RL, an adaptation of the Dense Learning paradigm to the RL post-training of tokenized multi-agent traffic simulators (specifically SMART). The motivation is the well-known sparsity of safety-critical interactions in naturalistic driving data: under uniform sampling, most RL updates are spent on nominal behavior and provide little safety signal. The method combines three ingredients:

  1. Episode-level densification. Scenarios whose current-policy probe rollouts contain at least one collision are preferentially trained on; probes are refreshed each epoch to combat label staleness.
  2. State-level masking. Within selected rollouts, policy-gradient updates are restricted to vehicle agent-time positions where some rollout in the same group contains a collision or near-miss (TTC/THW criteria), with a union across the G group rollouts so that aligned safe outcomes remain in-group for counterfactual comparison.
  3. Refined group-relative credit assignment. Immediate physical rewards (ADE + collision indicator) are normalized across aligned agent-time positions within same-scenario rollout groups, yielding critic-free advantages at agent-step granularity rather than broadcasting a single rollout scalar across all tokens.

On WOMD under the WOSAC protocol, Dense RL reduces mean collision rate while maintaining a comparable average displacement error, averaged over three seeds. Rollout-level credit and agent-only credit are substantially worse, supporting the chosen credit granularity.

Critical strengths
  • Clear theoretical motivation. §3.3 and Appendix A.2 derive the informative-event mask from a policy-gradient variance-reduction perspective (Eq. 10-11): a non-informative state has zero expected gradient contribution, so masking preserves direction up to a positive scale. The derivation makes the design feel principled.
  • Multi-agent-specific adaptations are careful. The paper correctly notes that D2RL-style state reconnection would break joint multi-agent consistency, and instead keeps full rollouts in the simulator while applying the mask only to loss accumulation. The union-across-group validity mask (Eq. 12) is a nice touch that preserves aligned counterfactual comparisons.
  • Clean ablations. Table 2 decomposes credit granularity and densification level. Reward ablations (Only ADE, Only collision) help defend the chosen reward mix.
  • Rigorous Experiment Dense RL and Uniform-refined are both reported with means and sample standard deviations across three seeds, which is more rigorous than the single-seed convention common in this subarea.
Critical weaknesses
  • Modest absolute improvement. The headline is a reduction about 0.5 percentage points — over an already-tuned uniform-refined RL baseline. In context, the ground- truth token reconstruction CR is 5.378% and the logged-trajectory CR is 2.325% (Table 1), so the method moves within a narrow operating band. The paper would be stronger with downstream evidence that this reduction matters in a planner-in-the-loop evaluation.
  • Narrow evaluation. All results use SMART-7M on WOMD (authors acknowledge in §5). Whether the method helps other tokenized simulators is unknown, and the appropriate credit granularity may be backbone-sensitive. Even one additional backbone would substantially strengthen the generality claim.
  • Collision-only episode mining is coarse. §3.3 explains that near-miss criteria selected ~90% of candidate scenarios so collision-only mining is used. This means mining is binary and sparse; as the policy improves and produces fewer collisions, the mining signal naturally weakens. A discussion of this saturation effect would be useful.
  • State-level masking depends on hand-chosen thresholds. TTC < 1.5s, THW < 0.5s, three-circle collision geometry with 0.2 m near-distance (Table 3). Sensitivity to these thresholds is deferred to Appendix A.5-A.7 and could be summarized in the main text, since the mask's effectiveness depends on them.
  • Qualitative analysis of behavior change is thin. Figure 3 shows one cut-in example. There is no systematic look at how Dense RL agents differ from the baseline on aggregate behavioral statistics (following distance distributions, deceleration profiles, interaction tempo). The WOSAC meta-metric is reported in the appendix but not discussed in the main text.
  • Comparison framing. CAT-K, a cheaper closed-loop supervised fine-tuning method, close to Uniform-refined RL at presumably lower cost. The paper could more clearly explain in what regime Dense RL is preferable to CAT-K-style methods.
Questions for authors
  • Does the result reproduce on a second tokenized backbone (e.g., BehaviorGPT or Trajeglish)? Even one additional backbone would strengthen the generality claim significantly.
  • How sensitive is the method to detector thresholds? If TTC is halved to 0.75s or doubled to 3s, does the CR improvement persist?
  • In Table 2, Episode-only already improves to 3.164% from Uniform refined 3.323%. What portion of the final is attributable to state-level masking specifically, and is that increment within three-seed noise?
  • Can you provide a Dense-RL-on-CAT-K-initialized baseline? CAT-K is weaker than Uniform refined, and it would be informative to see whether Dense RL further improves a stronger supervised starting point.
  • As training progresses and collisions become rare on probe batches, the episode-mining signal naturally weakens. Does the method plateau for this reason, and have you considered an adaptive detector (e.g., tightening TTC/THW over training)?
  • Could you report the WOSAC RMM metric for Dense RL in Table 1? The 0.7694 number is in the body text but a direct table entry would make it easier to compare to published simulators.
Reviewer_dDDmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a hierarchical Dense RL framework for safety-oriented post-training of tokenized multi-agent traffic simulators. It combines agent–time group-relative credit assignment with collision-based scenario mining and critical-interaction masking to concentrate updates on informative samples while preserving complete joint rollouts. Experiments using SMART-7M on WOMD show lower collision rates than uniform RL with the same refined credit assignment, with a small increase in average displacement error. Extensive ablation studies support the effectiveness of the proposed components, while detailed implementation descriptions and additional analyses help readers understand and reproduce the method.

Critical strengths
  1. Clear motivation and reasonable design. This paper clearly explains how safety-relevant signals are sparse across scenarios, agents, and timesteps. Refined credit assignment provides feedback at individual agent–time positions, reducing the indiscriminate assignment of rollout-level rewards to unrelated decisions. Complementarily, Dense RL prioritizes scenarios with observed collisions and concentrates updates on critical interaction steps, while retaining aligned safe outcomes across rollouts for comparison. These components offer a coherent approach to improving both the specificity of feedback and the allocation of training effort.
  2. Extensive experimental analysis. This work provides substantial ablations supporting its design choices. In particular, the comparison with the uniform refined method offers a thorough assessment of the additional benefits of densification beyond fine-grained credit assignment. The appendix provides additional extensive implementation details, helping readers assess both the effectiveness and limitations of the method.
Critical weaknesses
  1. Generalizability across pretrained models remains unverified. All RL experiments start from the same pretrained SMART-7M checkpoint, leaving it unclear whether the improvements generalize across different pretrained policies or primarily compensate for weaknesses specific to SMART. Evaluating the proposed post-training framework on another initialization, such as CAT-K, or a model with a different architecture or tokenization scheme would help establish its broader applicability. Including CAT-K as a standalone baseline does not demonstrate that the proposed method transfers to it.
  2. The behavioral implications of the safety–realism trade-off remain insufficiently characterized. Although Dense RL improves both safety and distributional realism over the uniform refined method, Tables 15–16 report lower kinematic, interaction, and map-based likelihoods and higher minADE than pretrained SMART. The appendix provides useful metric-level and distributional analyses, but it remains unclear which interaction behaviors account for these changes and whether plausible safety-critical behaviors remain adequately represented. The single cut-in example offers limited evidence on this question. Additional analysis by interaction type, supported by representative successes and failures, would help assess the implications for simulation-based training and evaluation.
Questions for authors
  1. Can the safety improvements be demonstrated with another pretrained initialization, such as CAT-K, and does the safety–realism trade-off remain similar?
  2. What behavioral changes explain the lower realism likelihoods and higher minADE relative to SMART in Tables 15–16? To what extent does collision reduction reflect removing unrealistic model-induced collisions versus suppressing plausible safety-critical interactions?
  3. Which types of interactions are most frequently selected by episode-level densification, and which benefit most from the resulting policy? A breakdown of the selected scenarios and performance gains by interaction type, supported by representative examples beyond the single cut-in case, would help clarify where the method is effective. If these concerns are adequately resolved, I would consider the paper suitable for acceptance at ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.