acceptodds
Under review as a conference paper at ICLR 2027

SHRED: Self-Play for Heterogeneous Road-Users without Expert Demonstrations

Abstract

Realistic multi-agent traffic simulation requires modeling the diverse behaviors and dynamics of vehicles, pedestrians, and cyclists interacting in shared environments. Current approaches predominantly rely on imitation learning from large-scale human driving datasets, achieving strong realism at the cost of demonstration-data dependence and computational overhead. We present SHRED, a framework for training heterogeneous populations of vehicles, pedestrians, and cyclists entirely through multi-agent reinforcement learning from random initialization, without trajectory supervision, logged-trajectory replay, or a pretrained imitation-policy anchor. Our approach combines type-specific dynamics, rewards, and procedural scenario generation, including randomized spawning, map-conditioned goal generation, and per-agent randomization of kinematic and reward parameters, enabling decentralized recurrent policies to produce diverse and coordinated behaviors. All agent types are trained jointly within a unified heterogeneous self-play pipeline despite their distinct dynamics and action spaces. Evaluated on the Waymo Open Sim Agents Challenge (WOSAC), SHRED jointly controls vehicles, pedestrians, and cyclists in closed loop. Component ablations favor type-specific objectives as well as dynamics, with uneven benefits across road-user types. SHRED achieves, to our knowledge, state-of-the-art performance on the official InterPlan benchmark, which tests interaction-heavy closed-loop planning. A policy trained on nuPlan map geometry retains comparable realism when evaluated on Waymo, supporting transfer across map sources. Our results suggest that heterogeneous procedural self-play can serve as a scalable and complementary paradigm to data-driven traffic simulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
multi-agent reinforcement learning, self-play, traffic simulation, autonomous driving, heterogeneous agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ugvEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SHRED, a demonstration-free self-play RL framework for jointly learning the behaviors of vehicles, pedestrians, and cyclists. It combines type-specific dynamics, rewards, and policy components with procedural scenario and goal generation, agent-level randomization, and advantage-weighted prioritized replay. The authors evaluate SHRED on WOSAC for traffic simulation and InterPlan for interactive planning, reporting improvements over a demonstration-free baseline and a vehicle-only self-play variant. The paper aims to establish heterogeneous self-play as a unified approach to traffic simulation and autonomous driving planning.

Critical strengths
  1. Learning heterogeneous road-user behaviors without expert demonstrations is an important research direction. SHRED provides a coherent framework for jointly training multiple road-user types through closed-loop interaction.

  2. Evaluation across simulation and planning is promising. The paper investigates both traffic realism and interactive planning, with ablations suggesting that heterogeneous dynamics and rewards contribute to performance.

Critical weaknesses
  1. The distinction between ego planning and agent simulation is insufficiently clarified. During self-play, agents can receive navigation goals, whereas in practical autonomous driving, the ego vehicle's intended route is generally known, but the intentions of surrounding road users are not directly observable. Although the paper introduces goal dropout to reduce dependence on explicit navigation signals, it remains unclear how the policy distinguishes the ego vehicle from other vehicles and how it behaves under these different information conditions. Furthermore, WOSAC and InterPlan have different task setups and evaluation protocols. The paper should clarify how the same framework is configured for simulation and ego planning, what information is available to each agent in each setting, and whether the reported results rely on task-specific modifications.

  2. The simulation evaluation does not sufficiently establish competitiveness against recent methods. The main paper emphasizes WOSAC 2023 and 2024 results, while the WOSAC 2025 evaluation is relegated to the appendix. Given the rapid progress in traffic simulation, the latest benchmark results and comparisons with strong contemporary methods should be central to the evaluation. The reported results appear less competitive against more recent approaches, making it difficult to assess the current state of the art and the practical significance of the proposed method. A more comprehensive comparison using the latest available leaderboard results and consistent evaluation protocols is needed.

Questions for authors
  1. How does SHRED distinguish the ego vehicle, whose navigation goal is known, from surrounding vehicles whose intentions are generally unknown? How are observations, goals, and policy configurations adapted between WOSAC simulation and InterPlan planning?

  2. Why are the WOSAC 2025 results not included in the main paper, and how does SHRED compare with the strongest recent methods under consistent evaluation protocols?

  3. SHRED relies on predefined dynamics models and manually designed reward functions for different road-user types. Although these choices facilitate scalable training, they may constrain the range of learnable behaviors and fail to capture the full distribution of human driving and pedestrian behavior. The paper itself acknowledges a fidelity ceiling imposed by the simulator and reward design. More evidence is needed to establish how sensitive the results are to these assumptions and whether the learned behaviors generalize beyond the capabilities encoded by the dynamics and reward functions. More specifically, how sensitive is the learned behavior to the predefined dynamics and hand-designed rewards? Can the authors demonstrate that the method captures plausible behavioral variation beyond what is explicitly encoded in these choices?

  4. The validity of procedural scenario initialization is insufficiently established. The method introduces randomized heading perturbations to increase interaction diversity and uses rejection sampling to avoid initial collisions. However, collision-free initialization alone does not guarantee physically plausible or traffic-rule-compliant scenes. For example, perturbed headings may produce vehicles or cyclists facing implausible directions or violating lane and road conventions. The authors should clarify how randomized heading perturbations are constrained by road geometry and agent type to ensure physically plausible, collision-free, and traffic-rule-consistent initial scenes, and provide quantitative validation of initial-state plausibility and traffic-rule compliance.

  5. The framework's applicability to end-to-end autonomous driving remains unclear. The policy relies on structured inputs, including map geometry, traffic signals, agent states, and navigation goals. It is therefore unclear whether SHRED is intended as a modular policy operating on structured scene representations or as a framework applicable to end-to-end driving systems. If the authors intend to claim broader relevance to end-to-end autonomous driving, they should discuss how the framework could operate directly from camera observations, or explicitly delimit the current method's scope and perception assumptions.

  6. The planning evaluation is too narrow for the breadth of the claims. InterPlan provides evidence of performance in interaction-heavy scenarios, but the paper presents SHRED as useful for both traffic simulation and autonomous planning. The planning results would be more convincing with evaluation on additional established benchmarks, such as NAVSIM v1/v2, where applicable, or other mainstream planning benchmarks with clearly specified protocols. This would help determine whether the reported gains generalize beyond InterPlan's particular scenario distribution.

4 comments
AuthorsOpenReview ↗

Response for Reviewer ugvE (1/2)

We thank the reviewer for their constructive questions and suggestions. We address each point below:

W1/Q1 — Ego planning versus agent simulation: SHRED uses the same policy architecture for simulation and ego planning. Each agent observes surrounding agents’ states but do not know others' navigation goals. During training, initial states and goals are generated procedurally, goals are renewed on arrival, and goal conditioning is dropped with probability 0.3.

In the WOSAC evaluation, no method receives logged destinations; SHRED uses map-sampled goals and generates a new goal on arrival. In InterPlan, SHRED controls only the ego vehicle and receives goal points along its known route. The distinction is therefore in which agents are controlled and how their goals are supplied.

W2/Q2 — WOSAC 2025 comparison: Imitation-learning methods have been extensively studied in prior work, but direct comparisons are limited because SHRED’s main contribution is achieving realistic heterogeneous multi-agent simulation through self-play without expert demonstrations. Table 3 includes SMART (CAT-K) as an imitation-learning reference. Since prior self-play baselines do not report WOSAC 2025 results, we additionally evaluate SHRED, SPACeR, and HR-PPO using the WOSAC 2025 10k validation set under consistent settings, excluding scenarios unsupported by GPUDrive due to z-axis and agent-count limitations (Table R3). SHRED achieves 0.733, outperforming SPACeR (0.718) and HR-PPO (0.696), consistent with the relative rankings in Table 3.

Table R3 — WOSAC 2025

MethodOverallKinematicInteractiveMap
SHRED0.7330.3850.7780.875
SPACeR0.7180.3710.7640.858
HR-PPO0.6960.3090.7460.852

Q3 — Sensitivity to dynamics and rewards: Table R1 directly evaluates sensitivity to the dynamics and reward design. Sharing dynamics reduces realism from 0.721 to 0.718, while sharing rewards reduces it to 0.714, supporting the benefit of type-specific modeling. In InterPlan and the additional nuPlan evaluations (Table R8 below), SHRED also achieves improved planning performance under nuplan's dynamics different from those used during training. These results provide evidence of transfer beyond the training dynamics.

Table R1 — Heterogeneous modeling

WOSAC 2024 with logged evaluation goals.

VariantOverallΔVehiclePedestrianCyclist
Full SHRED0.721—0.7420.7240.672
Shared dynamics0.718−0.0030.7390.7230.665
Shared rewards0.714−0.0080.7330.7110.663
Shared network0.721≈0.0000.7420.7230.672
Shared normalization0.719−0.0030.7410.7200.669
Fully homogeneous0.686−0.0360.6990.7080.668

Q5 — End-to-end driving scope: We do not claim that SHRED contributes directly to end-to-end autonomous driving. Our focus is on learning interactive behaviors through heterogeneous self-play using structured scene representations. Recent work, such as Scaling Self-Play for End-to-End Driving (Rowe et al., 2026), has demonstrated the potential of extending self-play to pixel-based driving policies, suggesting a promising direction for future work building on SHRED.

Q6 — Broader planning evaluation: We additionally evaluate SHRED on nuPlan Val14 and Test14-hard (Table R8), demonstrating consistent improvements over the vehicle-only baseline across both closed-loop reactive (CL-R) and non-reactive (CL-NR) settings. We also conduct repeated InterPlan-80 evaluations, obtaining 66.41 ± 0.79 for SHRED and 54.20 ± 0.50 for the vehicle-only baseline (mean ± standard deviation).

NAVSIM focuses on end-to-end planning from sensor observations, whereas SHRED operates on explicit object representations, making direct evaluation outside our current scope.

Table R8 — Planning

Benchmark (scenarios)MetricSHREDVehicle-onlyΔ
InterPlan (80)Score66.41 ± 0.7954.20 ± 0.50+12.2
InterPlan full (335)Score70.0166.46+3.6
nuPlan Val14 (1,118)CL-R85.2979.09+6.2
nuPlan Val14 (1,118)CL-NR81.7074.12+7.6
Test14-hard (272)CL-R73.9267.49+6.4
Test14-hard (272)CL-NR68.8959.29+9.6

InterPlan-80 reports mean ± standard deviation over four SHRED and three vehicle-only evaluations.

AuthorsOpenReview ↗

Response for Reviewer ugvE (2/2)

Q4 — Procedural initialization plausibility: Our initialization is designed to produce physically plausible configurations by construction, rather than relying solely on collision rejection. Vehicles spawn on lane centerlines, while cyclists spawn along lanes or road edges, with heading perturbations limited to ±25° relative to lane direction. Pedestrians spawn near road boundaries or crosswalks. Training goals are sampled within type-specific distance ranges: 20–100 m for vehicles, 2–12 m for pedestrians, and 20–35 m for cyclists.

We provide additional ablations separating the effects of initialization and training goals (Table R2). Logged initialization with procedural goals achieves comparable realism to full SHRED (0.724 vs. 0.721), while logged initialization with logged goals achieves 0.716. The latter corresponds to the original “w/o rand. init. & goals” ablation. These results suggest that procedural initialization is a practical alternative to logged initial states, achieving comparable realism without relying on expert demonstrations.

Table R2 — Ablation of initialization and training goals

CellInitializationTraining goalsOverallΔ vs. AVehiclePedestrianCyclist
A — Full SHREDProceduralProcedural0.721—0.7420.7240.672
BLoggedProcedural0.724+0.0030.7380.7190.680
CLoggedLogged0.716−0.0050.7310.7160.675

We additionally quantify initial-state plausibility across 75k procedurally initialized agents (Table R9). Compared with logged initialization, procedural initialization achieves substantially lower overlap rates across all agent types, zero vehicle/cyclist off-road rates, and near-zero wrong-way rates. Pedestrian off-road rates are higher (14.49% vs. 6.87%), partly because valid crosswalk placements are counted as off-road under this metric.

Table R9 — Starting-scene plausibility (% of agents)

TypeOverlap, randomOverlap, logOff-road, randomOff-road, logWrong-way, randomWrong-way, log
Vehicle0.011.140.0014.490.001.71
Cyclist0.013.930.0015.730.1214.04
Pedestrian0.0028.2014.496.87n/an/a

Thank you for your response and the additional experimental results, which address most of my earlier concerns. However, several points remain unclear and require further discussion:

  1. Based on your response to W1/Q1, goals appear to be an essential input in both setups: all agents rely on map-sampled goals in the sim-agent task, and all surrounding agents (excluding the ego vehicle) do so in the planning task. Could you elaborate on how these map-sampled goals are generated? Specifically, how do you ensure the diversity and behavioral plausibility required for realistic simulation, where agents should exhibit diverse driving behaviors?

  2. You position this work as "achieving realistic heterogeneous multi-agent simulation through self-play without expert demonstrations." Given the current performance gap compared to AR-based/diffusion-based approaches (w/ SFT or RFT), could you further justify why avoiding expert demonstrations is practically or scientifically significant for traffic simulation and planning tasks?

  3. Since self-play is driven by rule-based rewards, there is an inherent tension between reward optimization and matching the real-world human driving distribution. Could you share your insights on this trade-off and how your framework mitigates potential distribution shift?

  4. To provide a more comprehensive picture of the self-play paradigm's effectiveness, it is essential to report SHRED's official performance or ranking on the WOSAC 2025 leaderboard (test split).

Lastly, I recommend incorporating these clarifications and the new experimental results into the revised manuscript.

Q1: Map-sampled goals and behavioral diversity: We first clarify that in planning, SHRED controls only the ego vehicle, while surrounding agents are controlled by the simulation environment (e.g., IDM in nuPlan). In traffic simulation, SHRED is trained with 30% goal dropout, allowing agents to act without explicit goal conditioning. Map-sampled goals are generated by traversing directed lane graphs and randomly selecting among valid successor lanes, ensuring route plausibility while introducing diversity through different route choices. Importantly, goals specify destinations rather than trajectories; given the same goal, agents may exhibit different behaviors depending on surrounding traffic, interactions, and reward parameters conditioning.

Q2:SHRED’s core contribution is demonstrating that vehicles, pedestrians, and cyclists can jointly learn realistic interactive behaviors and effective planning policies through decentralized self-play, without expert trajectories or pretrained imitation policies. Agents learn by interacting and adapting to one another, and this reduces reliance on demonstrations and enables exploration of diverse closed-loop interactions, particularly valuable when representative data are scarce, such as in new regions or for underrepresented road-user types. Our simulation and planning results demonstrate the utility of this approach, positioning self-play as a complementary paradigm to imitation learning instead of a replacement

Q3: Our main insight is the importance of diversity during training. SHRED jointly trains different road-user types while randomizing each agent’s reward weights and dynamics, exposing policies to partners with varied preferences and capabilities. Procedural scenarios further diversify their interactions. This helps mitigate reliance on a single reward configuration or narrow interaction distribution, although it does not guarantee matching the frequency of behaviors in human data.

One challenge in imitation learning is distribution shift during long-horizon closed-loop simulation, where errors can accumulate and lead to collisions or off-road behavior. In contrast, self-play can train agents over extended horizons without being constrained by the duration of logged trajectories available. We observe that SHRED agents exhibit robust closed-loop behavior, as they are explicitly trained to avoid such infractions.

We also observe that distributional realism and desirable behavior are not always aligned. WOSAC evaluates similarity to logged behavior, including collision and off-road statistics. If logged trajectories contain off-road events, reproducing their frequency can improve the corresponding realism metric, even though avoiding those events may be preferable for planning. We therefore interpret realism scores alongside behavioral analysis and closed-loop planning performance, rather than treating realism alone as a measure of safe or effective behavior.

Q4 We have reported SHRED’s official WOSAC 2025 test-set results in original paper's Table9, updated pdf Table 11. Since no prior self-play results are available, direct comparison is limited. Leading imitation-based methods, such as Cat-K, achieve realism scores around 0.785, but rely on demonstrations from the benchmark’s training distribution. While a realism gap remains, the methods differ fundamentally in supervision, and part of the gap may reflect the trade-off between distributional fidelity and desirable closed-loop behavior discussed above.

We thank the reviewer for insightful discussions and have updated the final pdf to include the discussions.

Reviewer_zBGEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Traffic simulation usually hinges on recorded data, which is expensive and monolithic (held by a few actors). This method proposes to train traffic agents (cars, bikes, pedestrians) through self-play, without access to existing recordings with a shared LSTM policy but per-type encoders and action/value heads. The paper is positioned as improvement over the related Gigaflow work which differs in modeling all agents homogeneously but otherwise following the same self-play approach. I'm geeeenerally happy with the paper but I'm lacking crucial detail on the implementation. If the authors can provide that, I'm happy to raise my score.

Critical strengths
  • S.1: The paper's core idea is simple, sound (at least on paper) and a meaningful improvement over Gigaflow.
  • S.2: The paper is well-written and easy to follow.
Critical weaknesses
  • W.1: I have big concerns about reproducibility and trust. There are many crucial implementation details missing like: (a) What are the actual encoder networks? Maybe I missed it, but where are all the details about the network architecture? (b) How are other traffic agents sampled and how many are there? (c) If they're sampled by proximity, how do you deal with distant but fast-moving agents vs close but irrelevant ones? (d) What are the error bars on table 1? The realism improvement looks small, and I'd like to know how many seeds that's based on and if the results were cherry-picked from a single seed. (e) how long was this trained for, on which hardware and (f) which hyperparameters?
Questions for authors
  • Q.1: Have you looked into initializing the agents with any imitation learning? You already have that data/framework, and I just feel like it's an oversight.
  • Q.2: Have you done any failure mode analysis, any checks on common error types for each vehicle type?
Reviewer_J9F5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SHRED, a self-play framework for learning interactions among vehicles, pedestrians, and cyclists without relying on human trajectory demonstrations or imitation-policy anchors. The framework uses type-specific dynamics and reward functions. The encoders are shared across agent types, and each type has its own LSTM, actor head, and value head to accommodate heterogeneous agent behaviors. During training, the authors use procedural initialization and goal generation on road maps, together with agent-level randomization of dynamics and reward parameters. On WOSAC 2023, SHRED achieves higher overall realism than Gigaflow. On WOSAC 2024, its realism is comparable to, and slightly higher than, that of self-play methods anchored to demonstration-trained policies (HR-PPO and SPACeR), while remaining below that of the imitation-based SMART (CAT-K). SHRED also reports the highest score among the compared planners on the InterPlan benchmark.

Critical strengths
  1. A meaningful problem setting and architecture for jointly learning heterogeneous traffic agents without demonstrations or imitation-policy anchors.
  • Existing self-play approaches primarily focus on vehicles, share the same dynamics and goal-generation mechanism across agent types, or rely on policies trained on demonstrations. In contrast, SHRED jointly trains vehicles, pedestrians, and cyclists with type-specific dynamics, reward functions, and objectives.
  • Despite not using demonstrations, SHRED achieves realism comparable to demonstration-anchored self-play approaches (Table 3: 0.721, compared with 0.717 for SPACeR and 0.690 for HR-PPO). This suggests that realistic heterogeneous traffic behavior can emerge through self-play without trajectory supervision.
  1. A single learned policy is competitive in both traffic simulation and ego-vehicle planning.
  • SHRED reports the highest score among the compared methods on InterPlan (67.50 on the official 80-scenario split and 70.14 on all 335 scenarios).
  • The same nuPlan-trained policy is used both to simulate background agents (zero-shot WOSAC 2024 realism of 0.715) and as the ego planner on InterPlan. This demonstrates practical potential and suggests that the learned policy can serve different roles in traffic environments.
Critical weaknesses
  1. The evidence for the central claim that heterogeneity drives the performance improvements is insufficient.
  • The definition of the homogeneous baseline is ambiguous and potentially unnatural. It is unclear whether this baseline is a meaningful comparison that isolates the benefit of heterogeneous modeling.
  • The contributions of several important components are not evaluated individually. These include per-type normalization, advantage-weighted replay, goal dropout, asymmetric collision reward shaping, and the architectural separation across agent types. In addition, the effect of procedural initialization and goal generation is not adequately isolated, since there is no comparison with training from logged initial states.
  1. The comparison settings are not sufficiently standardized, and important evaluation and reproducibility details are missing.
  • The baseline results are spread across different WOSAC versions, and their sources are not clearly stated. This makes it difficult to assess whether the comparisons were made under consistent evaluation conditions.
  • The claimed improvement over Gigaflow appears overstated. Apart from the overall realism score, SHRED outperforms Gigaflow on only three of the nine individual metrics. Among the three interaction-related metrics, SHRED improves on only one (time-to-collision), is worse on one (distance to nearest object), and ties on one (collision). The statement of a "particularly strong improvement in interaction-based metrics" is therefore not fully supported by the reported results.
  • Several important implementation details are missing or unclear. These include the inputs and architectures of the encoders, whether the types of neighboring agents are explicitly observed, the source of traffic-light states during training without logged trajectories, the role of the "Latent Encoder," the total number of training steps, and key training hyperparameters.
Questions for authors
  1. Contributions of individual training components. Among per-type PopArt/advantage normalization, advantage-weighted replay, goal dropout, and asymmetric collision reward shaping, which components contribute most to the reported performance? Could the authors provide an ablation in which SHRED is trained from logged initial states instead of procedural initialization? In addition, in Table 2, what replaces randomized initialization and goal generation in the "w/o rand. init. & goals" variant?
  2. Architectural ablation for heterogeneous modeling. Could the authors compare SHRED with the following architectural variants while keeping the type-specific dynamics and reward functions unchanged? (i) A single policy with a shared LSTM and value head that uses type embeddings to distinguish agent types. Since the action spaces differ across types (12, 50, and 49 actions), type-specific action output layers could be retained. (ii) Fully independent networks for each agent type, including separate encoders. Such comparisons would clarify the benefit of the proposed partially shared architecture (Lines 166-168), and show whether it has advantages over both more parameter sharing and complete separation.
  3. Baseline comparability and evaluation settings. What are the sources of the baseline results in Tables 1 and 3, including the diversity values, and which scenario sets were used for each evaluation? Could the authors provide a consolidated comparison in which all baselines are evaluated on the same WOSAC version and scenario set? In addition, why were Gigaflow's pedestrians controlled by a scripted policy during evaluation, given that Section 2.2 describes Gigaflow as training a unified policy for vehicles, pedestrians, and cyclists (Lines 145-146)?
  4. Interpretation of the interaction-related improvements. In Table 1, SHRED is worse than Gigaflow on five of the nine individual metrics, and the three interaction-related metrics show one improvement, one degradation, and one tie. Given these results, what evidence supports the statement of a "particularly strong improvement in interaction-based metrics"? Could the authors also explain why SHRED's distance-to-nearest-object likelihood is substantially lower than Gigaflow's (0.250 vs. 0.316)?
  5. Model inputs. Could the authors provide more details on the inputs and architecture of each encoder, the number of neighboring agents and the observation radius, and whether the types of neighboring agents are explicitly included in the observations? In addition, how are traffic-light states obtained during training without logged trajectories?
  6. Consistency between Figure 2 and the architectural description. Could the authors clarify the following discrepancies between Figure 2 and the main text? (i) How do the six encoders described in Lines 262-263 correspond to the three encoders (Ego, Road, and Partner) shown in Figure 2? (ii) What is the "Latent Encoder" in Figure 2, and is it shared across agent types or separate for each type? (iii) Where are the type-specific LSTMs described in Line 264 represented in Figure 2? (iv) What exactly is meant by "fused heterogeneous LSTM weights"?
  7. Training details, computational cost, and reproducibility. Could the authors report the total number of training steps, the wall-clock training time, the number of agents per scenario, the PPO hyperparameters, the goal dropout rate, and the values of the crosswalk spawn probability p_cw and the goal-distance bounds d_min and d_max (Appendix B)? Could the authors also compare their training compute with that of Gigaflow and SMART? Finally, do the authors plan to release the code?
  8. Evidence for scalability. The paper motivates its approach by pointing to the throughput limitations of centralized models (Lines 46-47) and describes SHRED as "scalable" in the abstract. However, no quantitative results on throughput, inference latency, or scaling behavior relative to existing methods appear to be provided. Could the authors report such measurements to support the scalability claim?
4 comments
AuthorsOpenReview ↗

Response for reviewer J9F5

We thank the reviewer for their constructive comments and suggestions. We address each point below:

W1/Q2 — Heterogeneous modeling and component ablations: Table 2 in the paper reports the shared-dynamics, shared-reward, and fully homogeneous ablations. We now add shared-network and shared-normalization ablations and summarize all results in Table R1. Sharing the LSTM and actor head yields comparable realism to full SHRED (0.721), while sharing normalization reduces realism to 0.719. Together with the existing shared-dynamics (0.718), shared-reward (0.714), and fully homogeneous (0.686) results, these findings support type-specific dynamics, rewards, and normalization as contributors to realism, while suggesting that the gains do not depend on separate policy networks.

Table R1 — Heterogeneous modeling

VariantOverallΔVehiclePedestrianCyclist
Full SHRED0.721—0.7420.7240.672
Shared dynamics0.718−0.0030.7390.7230.665
Shared rewards0.714−0.0080.7330.7110.663
Shared network0.721≈0.0000.7420.7230.672
Shared normalization0.719−0.0030.7410.7200.669
Fully homogeneous0.686−0.0360.6990.7080.668

Each ablation shares one component across agent types while retaining the other SHRED settings:

  • Dynamics: a common unicycle model replaces the type-specific dynamics.
  • Rewards: common goal and collision rewards, retaining type-specific off-road terms.
  • Network: one shared LSTM and actor head; action masks and value outputs remain type-specific.
  • Normalization: shared PopArt statistics and global advantage normalization.
  • Fully homogeneous combines all four changes.

Q1 — Training components: Advantage-weighted replay and asymmetric collision reward shaping build on prior work in Gigaflow, while goal dropout was previously studied in SPACeR. Since these are established results, our additional ablations focus on SHRED’s core contributions to heterogeneous self-play (Table R1).

Q1 — Initialization and goal source: We provide additional ablations separating the effects of initialization and training goals (Table R2). Logged initialization with procedural goals achieves comparable realism to full SHRED (0.724 vs. 0.721), while logged initialization with logged goals achieves 0.716 (Table R2). The latter corresponds to the original “w/o rand. init. & goals” ablation. These results show that procedural initialization achieves comparable realism without relying on logged initial states.

Qualitatively, full SHRED generates a new procedural goal upon arrival, enabling agents to continue navigating and interacting beyond a single destination.

Table R2 —Ablation of log initialization and goals

CellInitializationTraining goalsOverallΔ vs. AVehiclePedestrianCyclist
A — Full SHREDProceduralProcedural0.721—0.7420.7240.672
BLoggedProcedural0.724+0.0030.7380.7190.680
CLoggedLogged0.716−0.0050.7310.7160.675

W2/Q3 — Baseline comparability and evaluation protocols:

We additionally evaluate WOSAC 2025 on 10k validation scenarios in TableR3 extending the original 1k-scenario evaluation in Table 3, excluding those unsupported by GPUDrive (z-axis and agent-count limitations for HR-PPO and SPACeR). Results are consistent with Table 3 (WOSAC 2024).Since GPUDrive does not yet support traffic lights, we assign a traffic-light score of 1 to all methods. Table 1 uses WOSAC 2023 for a direct comparison with GigaFlow, whose code is unavailable. We will include the 2025 results and clarify these protocol differences in the revision.

Table R3 — WOSAC 2025

MethodOverallKinematicInteractiveMap
SHRED0.7330.3850.7780.875
SPACeR0.7180.3710.7640.858
HR-PPO0.6960.3090.7460.852

W2/Q3 — Diversity definition: As described in §4.1, we measure trajectory diversity as the mean pairwise Wasserstein distance between each agent’s occupancy distributions across 32 rollouts, following BITS (Xu et al., 2023). These diversity results appear in Table 3, alongside WOSAC 2024 realism scores, whereas Table 1 focuses on the WOSAC 2023 comparison with GigaFlow and does not report diversity.

W2-3/Q4 — Interpretation of Interaction-Related Improvements: We will update the pdf to specifically highlight SHRED’s substantially better time-to-collision (TTC) likelihood, rather than claiming improvements across all interaction-related metrics.

AuthorsOpenReview ↗

Response for Reviewer J9F5 (2/2)

W2-4/Q6 — Architectural Details and Figure 2: Section 3.3 lists six input groups processed by four shared encoder modules. Ego state, goals, and conditioning variables form one ego vector, encoded by a two-layer MLP to 64 dimensions. Road segments, neighboring agents, and traffic entities have separate Deep-Sets encoders, each applying a two-layer MLP followed by max-pooling, with output widths of 256, 64, and 64, respectively.

The Latent Encoder maps their concatenated 448-dimensional output to a shared 512-dimensional scene embedding. Each road-user type then uses its own 512-unit LSTM, actor head, and value head with per-type PopArt statistics. Figure 2 presents a simplified architectural overview, grouping the LSTMs within the per-type heads and abstracting the traffic encoder for clarity. We will revise Figure 2 and §3.3 to explicitly illustrate these components.

“Fused” refers to stacking the three types’ LSTM weight matrices for batched computation. Each agent selects the output block for its own type; its output and gradient depend only on that type’s parameters. This is equivalent to three separate LSTMs, while avoiding per-type loops at each timestep. Actor and value heads use the same scheme.

W2-4/Q5 — Model Inputs and Observations: Each agent receives four groups of observations during training:

  • Ego: speed, size, steering, acceleration, collision and respawn flags, relative goal position and goal-dropout flag, stop-sign state, agent-type one-hot, and the agent’s own kinematic parameters and reward weights.
  • Neighbors: up to 20 agents within 50 m, represented by relative position, size, relative heading, speed, and agent type. Neighbor types are explicitly observed.
  • Road: up to 200 vectorized segments, with relative position, dimensions, relative orientation, and road type.
  • Traffic control: up to 16 stop lines and signals within 100 m, with entity type, relative position, stop-line endpoints, and signal state.

Training observations are constructed from simulated states. Traffic signals use a stochastic simulator controller, with mean red, green, and yellow durations of 8, 12, and 3 seconds, respectively; in 20% of scenes, signals share one cycle. During WOSAC evaluation, available signal states are replayed from the logs. We will include these input details in the revised manuscript.

Q7 — Training details, computational cost, and reproducibility: SHRED is trained for 10,000 epochs on 32 NVIDIA A100-SXM4-80GB GPUs using distributed bf16 mixed-precision self-play. The measured mean epoch duration is 41.0 s, including checkpoint and evaluation pauses, giving 113.9 hours of wall-clock training and approximately 3,644 A100-hours. Realism reaches 0.720 after 3,000 epochs (approximately 1,093 A100-hours), compared with 0.722 at 10,000 epochs (Table R4).

Each epoch collects 67.1M agent-steps, giving approximately 6.7 × 10¹¹ agent-steps over training. Episodes last 256 steps (25.6 s); the BPTT horizon is 128 steps. Half of pedestrian spawns are on crosswalks (p_cw = 0.5). Goal-distance ranges are 20–100 m for vehicles, 2–12 m for pedestrians, and 20–35 m for cyclists. The main optimization settings are summarized below; dynamics and reward ranges are provided in Appendices A–C.

SettingValue
Optimizer / learning rateAdam / 5 × 10⁻⁴, cosine-annealed over 23,841 epochs
Discount / GAEγ = 0.99 / λ = 0.95
PPO clip ratio0.2
Entropy / value coefficients0.01 / 0.5
Gradient-norm clip0.5
Update epochs / minibatches2 / 16
BPTT horizon / episode length128 / 256 steps
Batch / minibatch per GPU2,097,152 / 131,072 agent-steps
V-trace clippingρ̄ = c̄ = 1
Prioritized samplingα = 0.85, β₀ = 0.85
Per-type PopArt decay0.9997 per step
Goal / LSTM state dropout0.3 / 0.1

Table R4 — Training cost and throughput

32 A100-SXM4-80GB GPUs; measured mean epoch duration 41.0 s, including checkpoint and evaluation pauses. Training to epoch 10k takes 113.9 hours.

EpochsA100-hoursWOSAC 2024
1,000≈3640.710
3,000≈1,0930.720
5,000≈1,8220.721
8,000≈2,9160.723
10,000≈3,6440.722

Q8 — Scalability and computational efficiency: SHRED supports large-scale heterogeneous multi-agent simulation, achieving 1.71M agent-steps/s during training on 32 A100 GPUs. At batch size 4,096, SHRED’s inference latency is 10.65 ms versus 10.16 ms for the homogeneous baseline, corresponding to only 4.8% additional latency from heterogeneous modeling (Table R5).

Table R5 — Policy inference latency

One A100 80GB, bf16, real Waymo observations with LSTM state carried between steps. Latencies are medians over 900 timed steps.

Batch sizeSHRED (ms)Homogeneous (ms)
12.111.85
642.191.94
5122.282.16
4,09610.6510.16

Thank you for the detailed response and the additional results. The clarifications on the model inputs, architecture, and training details are helpful, and I encourage the authors to include them in the revised manuscript.

However, some of my main concerns remain.

First, the evidence that heterogeneous modeling drives the improvements is still limited. Sharing the LSTM and actor head gives the same realism as full SHRED (0.721), and the other single-component ablations change the overall score by only 0.002 to 0.007. The same setting with logged initialization and logged goals is reported as 0.711 in Table 2 and 0.716 in Table R2, which suggests that run-to-run variation is of a similar magnitude to these differences. Without results over multiple seeds, it is difficult to determine which type-specific components are responsible for the gains.

Second, the comparison settings are still not fully consistent. SMART is not included in the WOSAC 2025 evaluation (Table R3), so a comparison of all baselines under the same version and scenario set is still missing. In addition, the response states that Table R1 is evaluated with logged evaluation goals, and the same setting appears to apply to Tables 2 and 3. This is not stated in the paper and should be made explicit, since it affects the comparison with methods that do not use goals, such as SMART. The goal-distance ranges given in the response (2-12 m for pedestrians and 20-35 m for cyclists) also differ from those in Appendix B ([10, 30] m and [20, 60] m), and these should be reconciled in the revision.

Given these remaining concerns, I will maintain my current score.

We thank the reviewer for additional comments:

Inconsistency for Table 2 and Table R2 We identified a reporting inconsistency for this specific ablation: original paper's Table 2 uses the 8k-epoch checkpoint (0.711), while Table R2 uses the 10k-epoch checkpoint (0.716) from the same training run. We have updated Table 3 in the main paper to report 0.716, consistent with the 10k-epoch training budget. We have also verified that the remaining ablations use consistent checkpoints and evaluation protocols.

run to run-variance: Table R6 reports realism of 0.719 ± 0.002 across three independent training seeds, substantially smaller than the 0.035 gap between full SHRED and the fully homogeneous variant. While individual heterogeneous components yield modest improvements, their combined contribution is considerably larger, suggesting that the overall benefit of heterogeneous modeling cannot be explained by run-to-run variation alone.

Baseline comparison: Table R3 provides a consistent WOSAC 2025 comparison among the evaluated self-play methods, rather than a complete comparison with all simulation approaches. SMART remains an imitation-learning reference in our WOSAC 2024 comparison, where it achieves higher overall realism than SHRED. We will include SMART’s WOSAC 2025 results in the final version of the paper. Given the relatively modest differences between the WOSAC 2024 and 2025 evaluation metrics, we do not expect this additional comparison to change our main conclusion regarding the effectiveness of demonstration-free heterogeneous self-play.

Evaluation goals: We apologize for the typo in our previous response. SHRED uses map-sampled goals with 30% goal dropout during WOSAC evaluation, not logged goals. This applies to all reported WOSAC evaluations, including Tables R1, 2, and 3. We have explicitly clarified this protocol in Section 4.1 and the corresponding table captions of the revised manuscript. The goal-distance ranges are 20–100 m for vehicles, 2–12 m for pedestrians, and 20–35 m for cyclists. We have corrected Appendix B in the revised manuscript accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.