acceptodds
Withdrawn from ICLR 2027

R²Drive: Reinforcement-Learned Reactive Agents in a Closed-Loop 3D Driving Simulator

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present R²Drive, a closed-loop multi-agent dynamics framework for sensor-level autonomous driving simulation with reactive surrounding vehicles. R²Drive couples style-specific RL traffic policies with a 3D Gaussian simulator for adaptive behavior and closed-loop planning. The simulator preserves geometric fidelity (34.6 dB PSNR) while supporting distinct safety- and efficiency-oriented policy variants. Each reconstructed circuit is playable in a browser at 12–30 FPS over 1.5–2.0M Gaussians, enabling manual driving, shared-policy multi-agent replay, or an external planner receiving rendered RGB observations. On Waymo, the adapted agents reach 4% collision and 86% goal achievement; on nuPlan, the safety variant transfers zero-shot to 7.6% collision and 98.8% goal achievement. These results bridge photorealistic reconstruction and programmable multi-agent dynamics for closed-loop planner validation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Reinforcement Learning, Driving Simulator, Egocentric Navigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L4vmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces R2Drive, a closed‑loop driving simulator built upon 3D Gaussian Splatting. It combines RL‑driven reactive traffic agents and real‑time photorealistic neural scene reconstruction. The authors reconstruct multi‑kilometer real‑world driving circuits, support interactive manual driving, multi‑agent self‑play, and external planner evaluation. Evaluations are conducted on the Waymo and nuPlan benchmarks, reporting collision rate / goal‑achievement metrics, reconstruction metrics, and closed‑loop planning results after post‑training VAD/UniAD in this simulated environment.

Critical strengths

High practical relevance for end-to-end autonomous driving research: As end‑to‑end models typically take raw image inputs, realistic simulators have become an increasingly important tool within the research community. Such simulation platforms can serve multiple purposes, including closed‑loop evaluation of learned planners, reinforcement‑learning training, and generating training data for imitation‑learning workflows. This released simulator provides potentially useful infrastructure for relevant research work.

Remarkable engineering workload: Building the full pipeline requires heavy engineering investment, including multi‑kilometer real‑world circuit reconstruction, 3DGS asset production, reactive multi‑agent policy implementation, browser‑based interactive frontend, and integration with existing driving planners.

Empirical experiments demonstrate the effectiveness of individual modules: The paper provides benchmark comparisons for multi‑agent traffic behavior, neural rendering quality, and planner post‑training performance. These results offer empirical support for the overall system’s functionality and practical usability.

Critical weaknesses

Limited core algorithmic novelty; this is primarily a system‑assembly‑driven work: The major building blocks (3D Gaussian Splatting for neural rendering, PPO for reinforcement‑learning agent training, off‑the‑shelf planners VAD / UniAD) are existing techniques. While the integration engineering effort is substantial, the paper delivers few new algorithmic innovations. The manuscript reads more like a system demonstration rather than an algorithm‑focused research paper.

Some critical technical and experimental details are insufficiently documented: Key information such as task formulations, full training hyper‑parameters, and ablation studies among different training configurations is missing.

Multiple individual contributions are implied in this paper while lacking deep analysis: This paper mixes three relatively independent contributions: neural scene reconstruction of large‑scale driving circuits, RL‑based reactive traffic‑agent simulation, and closed‑loop planner evaluation infrastructure. For example, the state‑of‑the‑art‑level neural‑rendering claim in Table 2 is a substantial result on its own, yet it cannot receive sufficient depth of discussion within the current paper’s scope.

Unclear source of performance gains and limited actionable takeaways for readers: Given the highly composite nature of R²Drive, it is hard to pinpoint which component or design decision drives the reported quantitative improvements. Moreover, the advantages of this simulator among other published simulators, such as metadrive, carla, are missing.

Overall, I appreciate the authors’ efforts to integrate 3DGS, reinforcement learning, and closed-loop simulation generation in this work. Nevertheless, the omission of key technical details, together with insufficient demonstration of its advantages over existing simulators and prior arts, leads me to lean toward a negative recommendation for now.

Questions for authors

See weaknesses.

Reviewer_E1H3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reactive neural driving simulator for closed-loop evaluation and training of E2E driving models that consume rendered frames. Both these parts (3DGS-based neural reconstruction and reactive policies trained with self-play) have been proposed before, but the authors claim that they are able to combine the two to produce a reactive neural simulator that can render frames at a reasonably fast frame rate making this viable for closed-loop eval and training at scale. The authors show their two policy variants (safe, efficiency) behave differently under certain conditions, slight improvements over OmniRe in terms of reconstruction quality, and that an E2E model fine-tuned with RL in their closed loop simulator sees significant improvements improvements in driving score compared to baseline IL model on Bench2Drive.

Critical strengths
  • This paper proposes a playable, browser-hosted, reactive, neural simulator that can be used to effectively evaluate and train an E2E (or sensor-based) driving model in closed loop. This is very important because many works have shown that a simply IL trained driving model suffers from causal confusion, drift, and other related issues which can only be effectively measured and addressed in closed-loop sim or the real world.
Critical weaknesses
  • This work largely repurposes existing works like GPUDrive (for reactive driving policies) and DriveStudio/OmniRe (for 3D reconstruction) to create a reactive, closed loop sensor simulator. In that sense, despite being a useful tool, the novelty is quite limited. The differences compared to existing self-play driving policies as in GPUDrive seems to be limited to having two reward hyperparams for safe and efficient driving policies. As for the 3DGS pipeline, the only novelty seems to be "joint pose optimization and multi-view consistency constraints during neural rendering". In addition, directly relevant works such as UniSim, NeuroNCAP are not cited and contextualized. Additionally, the RAD and HUGSIM already support reactive rule-based actors like IDM, adversarial actor etc.
  • The evaluation protocol and experiments seem to lack details and are sometimes inconsistent/unfair. For example, Table 1 results do not specify exact evaluation protocol (how many scenarios, log initialized with log goals or otherwide). Table 1 says its nuplan validation set, but text under section5.2 says "WOSAC scenes". Table 2 claims OmniRe's protocol but on "1000 urban scenarios" , which raises the question of whether the baseline numbers were obtained on the same scenes. For Table 3, the paper uses "fixed multi-object tracking protocol on the nuScenes validation set", but which multi object tracker is this and how is it configured. None of this information is provided in the appendix either. Although Table 4 ablations are quite persuasive, we don't ablate for the main contribution, i.e. reactive agents. Ideally, we should compare against RL with log replay agents or RL with IDM/PDM agents. The other issue that is not addressed with regards to Table 4 is the domain gap between neural reconstructions with real data and CARLA's siumulated graphics rendering.
Questions for authors

I would mainly request all the missing details with regards to the experimental setup:

  • For Table 1 numbers, are these self-collisions between multiple agents controlled by the same policy or when only controlling the ego. What is the setup for other agents in the scene, i.e. log replay or the same policy?
  • For Table 2, which Waymo scenes ("1000 urban scenarios") were used, and was this the same setting for all baselines? What component of your renderer accounts for the improvement over OmniRe?
  • For Table 3, what is the detector and tracker used and on what rendered data?
Reviewer_xKMTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents R^2Drive: a photorealistic simulator using 3DGS of real driving scenes, with geometric fidelity of objects (like agents), and plausible (reactive) multi-agent interactions trained with RL with various safety/efficiency thresholds for heterogeneous behaviors. The paper shows improves in traffic-agent safety, and post-training improvements to VAD and UniAD when using this simulator.

Critical strengths
  • The writing is generally clear
  • Great visuals
  • Seems like a useful simulator to run if shipped with the pre-trained traffic agents
  • Nice zero-shot transfer results, e.g. the agents trained on Waymo scenarios, and evaluated on nuPlan scenarios (though using simulator states)
Critical weaknesses

Large weaknesses:

  1. The contributions of this "framework” seems nebulous. While the combination of RL co-trained agents in a 3DGS environment is a nice way to tick all the boxes of having both photorealism and realistic behaviors, and is useful, but seems too much of an implementation of prior methods combined together, without any technical advance in connecting them. For example, Section 3.1 goes into a lot of low level details about representing a simulation state, which could describe any simulator, and disconnected from the RL agent section in section 3.2; which itself was a fairly basic RL setup -- which is fine, but I couldn't see any novelty in the combination.
  2. Table 4: did these gains require reactive traffic, or would IDM agents work just as well? I’m wondering how much of the gains came from the RL method proposed, and if the reactive agents were even necessary.
  3. Line 209: “all reported closed-loop experiments use seed 42” is a single seed enough?

Small weaknesses:

  • Figure 1: the labels “Emerald Green, Night Purple, Titanium” seem distracting. Consider removing or use simpler “Green, Purple, Yellow”?
  • Line 045: “On Waymo” (a company) is unclear, say the specific challenge (WOSAC or WOMD)?
  • Line 117: worth breaking up “Street Simulation” and “Agent Policies” into two paragraphs since they were listed as distinct contributions?
  • Line 138: was does the “static polyline ensemble” have L_{dynamic} elements, what does this mean? Reading Line 144’s “traffic control changes”, does this mean traffic lights? Perhaps it could be reworded to clarify to readers
  • Line 166: where is theta (yaw) in equation 3?
  • Line 166: what is R in equation 3? (can’t see it mentioned in text afterwards)
  • Line 189: “(2), (2)” is a bit difficult to parse, as item2 and equation2; same with “(3), and (3)”; perhaps enumerate with letters?
  • Line 232: “adjusting the sensitivity of the collision signal via ϕ”, was w_c meant instead of ϕ? Is ϕ in any equation? Edit: oh I see later in (7); I was confused up till then
  • Line 236: Should the authors mention PPO if (6) differs from PPO’s loss function, perhaps just stick with “RL”?
Questions for authors
  1. Line 180: “we initialize the policy with imitation learning on recorded expert trajectories” any details on this data?
  2. Line 322: “fixed dynamics of GPUDrive”, what did this mean?
  3. Does safer agent behaviors in the simulator mean this is a better evaluator for an ego policy? If other agents learn to be overly safe (or overly courteous) could this simulator lead an ego policy into a false security?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.