acceptodds
Under review as a conference paper at ICLR 2027

APEX: Robust Driving Self-Play through Adaptive Population Exploration

Abstract

Large-scale driving self-play can learn robust policies through closed-loop interaction without human driving demonstrations. Yet existing methods use shared policies and fixed traffic-behavior distributions, causing driving policies to overfit to training-time behavioral regularities and generalize poorly to unseen traffic behaviors. We introduce Adaptive Population Exploration for Self-Play (APEX), which reframes driving self-play as a population-level multi-objective learning problem. Specifically, APEX separates the target driving policy, termed the focal policy, from a behavior-conditioned traffic policy, while making the distribution of traffic behaviors learnable. While the two policies interact and optimize their respective objectives, a population constructor uses focal-policy feedback to adapt the distribution of traffic conditions, thereby changing which self-interested behaviors the focal policy encounters without directly controlling vehicle actions. To focus this adaptation on policy weaknesses rather than intrinsic scene difficulty, we use the focal critic's initial value estimate as a baseline for realized focal performance. We show that this centering preserves the expected optimization direction of the population objective under our training protocol. By alternating policy learning and population adaptation, APEX tracks evolving weaknesses. Experiments show improved robustness and generalization to unseen traffic behaviors while maintaining competitive performance under standard self-play.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Self-Play Reinforcement Learning, Autonomous Driving, Heterogeneous Traffic, Behavior Distribution Learning, Policy Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NxheOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

APEX aims to improve the robustness of driving policies trained through self-play by adapting the traffic behaviors they encounter during training. It separates a focal driving policy from a behavior-conditioned traffic policy and learns a population constructor that samples traffic conditions, including reward preferences and visibility constraints. The constructor uses the difference between the focal critic’s initial value estimate and the observed return to emphasize challenging traffic populations. The paper analyzes this signal under a first-exposure protocol and evaluates the method on BehaviorBench’s Interactive1k and Random1k splits. The results show stronger improvements on Interactive1k and lower collision rates across several traffic populations, while aggregate performance on Random1k remains close to PPO.

Critical strengths

The paper addresses a useful problem: a driving policy can perform well against its training partners yet generalize poorly to other traffic behaviors. Adapting the distribution of training partners is a plausible approach, and the resulting focal policy can be deployed without the population constructor or behavior-conditioning inputs.

Critical weaknesses
  1. The distinction from existing self-play methods is not sufficiently established. The motivation associates shared policies with homogeneous interactions, although methods such as GigaFlow already produce diverse behavior through agent-specific conditioning. The central contribution is therefore the adaptive population distribution, but the experiments do not adequately establish why this adaptation is necessary or whether it tracks evolving focal-policy weaknesses.

  2. The experimental evidence is too limited to establish the reliability and scope of the gains. Evaluation is restricted to BehaviorBench, and the reported training results use a single seed. On Random1k, the aggregate score is close to PPO despite improvements in collision rates, leaving the broader benefit unclear.

Questions for authors
  1. What numerical collection threshold N triggers a constructor update, and how many policy updates typically occur between constructor updates? Please also specify the constructor optimization settings and exploration-scale schedule.

  2. Can the authors double check Figure 1 (b) & (c)? The symbols are missing there.

  3. Why equation 4 shares the same input for focal policy, conditioned policy, and the population constructor?

1 comment

Thank you for recognizing the motivation for adapting training partners and the simplicity of deploying the focal policy.

R1. Adaptive population construction

Table 2 in the paper adds components in stages. PPO uses one unconditioned self-play policy. BC PPO uses GigaFlow-style per-agent reward and perception conditioning in a shared policy. Fixed dual separates the focal and traffic policies and samples conditions from a fixed distribution. APEX retains both policies and learns the sampling distribution.

Across seven shared traffic settings on Interactive1k, mean collision rates for PPO, BC PPO, Fixed dual, and APEX are 18.23%, 14.74%, 11.57%, and 9.86%, respectively. Mean goal completion is 75.67%, 74.74%, 79.00%, and 80.69%. Fixed dual and APEX both train two policies with a 4B total budget. On Random1k, APEX reduces collisions relative to Fixed dual from 5.40% to 4.26% and raises goal completion from 90.07% to 90.77%. These gains at the same budget support adaptive sampling as a key component of APEX.

R2. Reliability, training seeds, and scope

On Interactive1k, APEX outscored PPO by 2.71 points (95% CI [1.96, 3.46]), with a 6.79 percentage point reduction in at-fault collision rate and a 4.53 point gain in goal completion. On Random1k, aggregate scores differed by 0.34 points, while APEX lowered the at-fault collision rate by 2.44 percentage points and raised goal completion by 2.89 percentage points. We obtained the interval by resampling scenes jointly for both policies, measuring sensitivity to the choice of test scenes for the fixed checkpoints.

We trained APEX twice at seed 42 using an alternative training configuration. Under the main-table evaluation protocol for Interactive1k, both runs lowered overall and at-fault collision rates and improved goal completion in each of the seven shared traffic settings. Across these settings, the mean collision rate fell from 18.23% with PPO to 11.13–11.23% with APEX, and goal completion rose from 75.67% to 79.91–80.69%. For the seed-43 APEX run, we retained the core training settings and evaluated on Interactive1k under IDM traffic. Compared with PPO, APEX reduced overall collisions from 13.9% to 7.1% and at-fault collisions from 9.9% to 4.4%, while increasing goal completion from 75.1% to 77.7%. These results suggest that APEX consistently reduces collisions and improves goal completion across the tested training configurations and seeds.

We deployed the frozen WOMD policies on nuPlan Val14 and interPlan using the same observation interface and controller. APEX led PPO by 1.46 points on non-reactive Val14, 1.06 on reactive Val14, and 7.99 on reactive interPlan. On reactive Val14, the no-at-fault-collision score rose from 64.49 to 66.68. On interPlan, comfort compliance rose from 66.25% to 81.25% and drivable-area compliance from 17.50% to 30.00%.

Q1. Constructor threshold, update frequency, and exploration

N=256N=256 is the minimum number of completed scenario-window records that triggers an update. Each update uses all valid records available at that time. The 268 recorded updates used 2,055–2,329 records each (median 2,194). The first update occurred at driving iteration 15. The next 267 intervals comprised 205 intervals of 14 driving iterations and 62 of 15 (mean 14.23). One driving iteration is a rollout and training cycle.

We use Adam with a learning rate of 0.0003, four optimization epochs per constructor update, minibatch size 64, PPO clipping 0.2, target KL 0.03, and a maximum gradient norm of 1. The exploration standard deviation decays from 0.5 to 0.2 over 600,000 descriptor records, counted separately from driving agent steps.

Q2. Missing symbols in Figure 1(b) and 1(c)

Thank you for pointing this out. Fig. 1(b) labels the traffic policy πω,cCπ_{ω,c}^C. We will use πωCπ_ω^C in both panels, show 𝐜j𝐜_j separately as a policy input in Fig. 1(b), and check the rendered PDF.

Q3. Shared arguments in Equation (4)

In Eq. (4), (θ,ω,φ)(θ,ω,φ) denotes the dependence of JBJ_B, JCJ_C, and JBexpJ_B^{exp} on the three modules' parameters. Both driving policies shape the closed-loop trajectories, and the constructor determines the distribution of training traffic. For example, changing φφ can alter the focal policy's return by changing the traffic it encounters, even with θθ fixed.

For action selection, the focal policy uses its local observation and recurrent state. The conditioned policy additionally receives its vehicle's behavior condition. For each map window, the constructor samples a descriptor from the global distribution qφq_φ that defines how traffic conditions are drawn. Each module updates its own parameters under its own objective. We will clarify the distinction between objective arguments and module inputs in the revision.

Implementation and training details are available in our anonymous repository.

Reviewer_VdeHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores learning an adaptive traffic population curriculum that adapts to the weaknesses of the main driving (focal) policy during multi agent RL training. This is an important problem worth solving because naive self-play often results in self-play bias, i.e. agents excessively optimized to perform well in environments with versions of itself, and fail as soon as the agent behaviors diverge significantly from what is seen during training. With this in mind, the authors propose a novel scheme to disentangle the main driving policy from the learned traffic policy - this then allows then to learn a population constructor trained to maximize a critic-centred signal, which preserves the expected score-function gradient. On BehaviorBench, a single APEX checkpoint beats PPO self-play in all 8 traffic populations on Interactive1k (mean 42.06 → 44.76), and is show to be on par on Random1k.

Critical strengths
  • This is a novel approach that tries to address a real problem that self-play RL for driving suffers from from a population distribution perspective.
  • The problem and the approach itself are well motivated.
  • The proposed approach shows consistent improvements over the baseline on interaction-heavy scenes across all traffic populations on BehaviorBench.
  • The authors provide a clean (yet incomplete) ablation of the critical components.
Critical weaknesses
  • Two key factors in the success for self-play for driving policies is large-scale training on the order of 1e12 steps (see Gigaflow, TerraZero) and a diverse procedural scenario generation recipe which can cover a large distribution of scene initializations, goals, and traffic behaviors. Works that achieve these two factors have achieved state of the art on most closed-loop planning benchmarks in literature. With that in mind, the proposed approach seems to be more of a mitigation for the lack of this, i.e. promising moderate improvement to robustness in medium scale (), log data based self-play training. With this in mind, the paper should frame this work as such. If possible, I would like to see the improvements over baseline self-play with behavior conditioning as we increase the scale of training (cumulative agent steps), say at 2-4x current training budget.
  • The authors show that the largest learned shifts remain in the goal reward and view-cap dimensions, both of which can be achieved in baseline self-play by randomly dropping out agents in observations, and by randomly masking the goal (see SPACeR). Also, the authors themselves note that the results "do not by themselves establish causal tracking of focal-policy weaknesses", which I appreciate. At the very least, we should compare to a baseline PPO setup with random agent dropouts in observations and goal masking that could provide similar levels of robustness.
  • Finally, it would be valuable for the authors to have comparisons on some other established benchmarks like val14 (regular driving) or interplan (interactive driving) so that there is more confidence in the reported gains. This could be zero-shot as has been done before (Gigaflow) and needs no re-training.
Questions for authors
  • What is the number of map records (N) used for the constructor updates? I don't think this is explicitly stated. An ablation on this axis would also be useful considering it might be critical to the approach.
  • What is the value for eps_pop and the number of constructor epochs?
1 comment

Thank you for suggesting a clearer scope, direct perturbation controls, and additional closed-loop benchmarks.

R1. Training scale and scope

APEX targets moderate-budget self-play on logged scenarios. At the original budgets, Table 2 shows that APEX lowers mean collisions from 14.74% to 9.86% and raises goal completion from 74.74% to 80.69% versus 2B-step BC PPO across seven shared Interactive1k traffic settings.

We doubled the budget from 4B to 8B agent steps for APEX (about 2B to 4B for the focal policy) and from 2B to 4B for standard PPO, continuing from the original weights with fresh optimizers. Mean scores across eight traffic settings follow.

MethodInteractive1kRandom1k
APEX44.76 → 43.5956.15 → 55.94
Standard PPO42.06 → 43.0455.81 → 55.42

On Interactive1k, the score gap narrowed from 2.71 to 0.55 points. At the larger budget, APEX retained a lower at-fault collision rate (6.73% vs. 11.21%) and higher goal completion (82.44% vs. 79.03%).

R2. PPO with agent dropout and goal masking

We trained three PPO controls on the same 10k-medium scenarios with seed 42 and about 2B agent steps. Agent dropout independently removes each neighbor from each observer's input with probability 0.5 at each step. Goal masking hides the observer's goal position with probability 0.5 at each step. The joint control applies both perturbations. All five policies use the same evaluation protocol with both perturbations disabled. The table reports mean scores across eight test traffic settings.

MethodInteractive1kRandom1k
Original PPO42.0655.81
PPO + agent dropout41.5854.68
PPO + goal masking42.8956.12
PPO + both43.8055.95
APEX44.7656.15

On Interactive1k, the joint control improves on PPO by 1.74 points (95% CI [0.46, 3.00], adjusted for 14 comparisons). Its at-fault collision rate falls from 12.64% to 7.44%, and goal reaching is 77.65% versus 77.78% for PPO. APEX scores 0.96 points above the joint control (unadjusted 95% CI [0.21, 1.73], adjusted CI [-0.14, 2.09]), with a 5.85% at-fault collision rate and 82.30% goal reaching. On Random1k, goal masking has the highest mean among the PPO controls (56.12), close to APEX (56.15). The score intervals use 10,000 paired bootstrap replicates, sharing scenario indices across all eight traffic settings.

We also compare APEX with a static random two-policy control at the same 4B total budget. It samples four field-of-view caps with equal probability and draws reward conditions and perception radii independently and uniformly. APEX scores 2.18 points higher on Interactive1k (paired 95% CI [1.56, 2.83]) and 0.54 higher on Random1k ([-0.02, 1.11]).

R3. Zero-shot evaluation on Val14 and interPlan

We transfer the fixed WOMD checkpoints with a shared observation interface and controller within each benchmark. Val14 uses the logged segment endpoint as the goal, with goal braking and trajectory padding. For the hybrid methods, the learned trajectory is selected when the best PDM proposal scores below 0.1.

Official closed-loop scores out of 100.

MethodVal14 non-reactiveVal14 reactiveinterPlan reactive
APEX35.9439.8813.65
PPO34.4838.825.66
PDM only92.8492.1341.81
APEX + PDM91.1491.0142.43
PPO + PDM90.8090.9542.69

Under direct deployment, APEX leads PPO by 1.46, 1.06, and 7.99 points, respectively (paired 95% CIs [-0.07, 2.93], [-0.42, 2.49], and [-0.14, 13.66]). In reactive Val14, the no-at-fault-collision score rises from 64.49 to 66.68 (unadjusted 95% CI [0.21, 4.14], with the adjusted interval including zero). Standalone PDM scores highest on Val14. Both hybrids score slightly above PDM on interPlan.

Q1. Constructor collection threshold and update frequency

population_batch_size=256 is the minimum number of completed simulation-window records needed to trigger an update. Each update consumes all valid records available at that time. The 268 archived updates consumed 587,424 records, with 2,055–2,329 records per update and a median of 2,194. The first update occurred at driving iteration 15. Subsequent updates occurred every 14–15 driving iterations (mean 14.23).

Q2. Constructor optimization

The constructor uses PPO clipping with eps_pop=0.2, four optimization epochs per update, minibatches of 64, an Adam learning rate of 0.0003, target KL of 0.03, and a maximum gradient norm of 1. The exploration standard deviation decays from 0.5 to 0.2 over 600,000 descriptor records. The Beta mapping uses a separate epsilon of 0.02.

Implementation and training details are available in our anonymous repository, including the full configuration in TRAINING.md.

Reviewer_raCUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

paper introduce adaptive behavior population exploration. In prior works, method usually adopt same driving policies in different traffic behavior distribution. This cause to overfit the model to specific policies during traning time, lossing generalizability of the model which is important factor in autonomous driving. Thus, paper decompose it into multi-level objective. Model constructor first learn different traffic behavior to distinquish followed by focal and traffic policy to learn action of the vehicle .

Critical strengths
  1. Easy to folow: paper well formulate the problem setup of prior works to current proposed method.

  2. Interesting formulation: It is common that single model adopt to select motion of the vehicle. Yet, the paper proposes real-world adaptable method using constructor to reframe usage of different motion policies in different traffic environment for generalizability of the model which is the most crucial task.

Critical weaknesses
  1. Contructor Overfits: As constructor also learn the traffic behavior from specific dataset, it may not convey actual traffic behavior but overfit specific traffic population or behavior or trained dataset. This seems to be another overfitting problem And these distribution may not be distinctive in certain cases such as similar region with slightly different number of vehicels.

  2. Implementaiton detail: While paper provide proof and explanations, author did not provide code or reproductable sources which make little bit diffcult in understanding what type of model design and strctures are being use to train.

  3. one dataset: connected to weakness 1, to show real genearlizability of the method, I would like to see cross-dataset (trained on Waymo as paper did, and test on nuScenes, or Argoverse, or navsim dataset) to further verify the claim.

Questions for authors

See weakness

1 comment

Thank you for recognizing the problem formulation and the motivation for studying generalization.

R1. Constructor overfitting and distinct traffic behaviors

The constructor learns on the training maps and adjusts the distribution of traffic reward preferences and perception conditions to create challenging interactions for the focal policy. To test for overfitting to the jointly trained traffic, we evaluate the focal policy against seven traffic controllers that APEX never encountered in training. In the Self setting, each method controls traffic with its own policy.

On Interactive1k, APEX raises the mean score across eight traffic settings from 42.06 to 44.76 over standard PPO self-play (paired difference +2.71, 95% CI [1.96, 3.46]). In Table 2, the methods perform comparably under Self. Across all seven unseen traffic settings, APEX reduces collisions and improves goal reaching. The mean collision rate falls from 18.23% to 9.86%, and the mean goal-reaching rate rises from 75.67% to 80.69%. With IDM, these metrics change from 18.2% to 5.8% and from 73.3% to 78.9%. With SMART, they change from 16.9% to 8.1% and from 77.5% to 83.5%. These gains extend beyond the jointly trained traffic.

To test whether traffic behaviors differ at a fixed vehicle count, we ran 18,000 rollouts on 1,000 eligible training maps using two paired sampling seeds and nine condition settings. Each pair shared the scene, vehicle count, and initial state. We froze both policies and varied the traffic condition. When only the traffic goal reward changed between 0 and 2, the paired trajectories of 89.6% of traffic vehicles per map differed by at least 2 m within the first 3 s, on average across 996 maps with active traffic (95% CI [88.0%, 91.1%], adjusted for four comparisons). At matched timesteps in the first episode of both runs, the mean per-vehicle maximum position difference was 9.04 m and the mean absolute speed difference was 3.05 m/s. Changing the field-of-view cap between 90° and 360° yielded trajectory differences of at least 2 m for 24.8% of traffic vehicles (adjusted 95% CI [22.5%, 27.1%]).

R2. Code, architecture, and reproducibility

The anonymous repository provides training and evaluation code, APEX and PPO configurations, and APEX, PPO, and SMART checkpoints. TRAINING.md lists the two methods' training settings and commands side by side. docs/APEX_SETUP.md and BENCHMARK.md cover setup and evaluation. The constructor uses a stateless 12-dimensional Gaussian descriptor distribution with a learned mean and an exploration standard deviation scheduled by the cumulative record count. Each descriptor parameterizes six Beta distributions over traffic conditions.

Both methods use the Drive encoder and a 256-unit LSTM. APEX also trains a separate behavior-conditioned traffic policy and the constructor. PPO uses 2B agent steps. APEX splits 4B steps equally between two driving policies, giving the focal policy a nominal budget of about 2B. Each policy collects 524,288 transitions per rollout, with a minibatch size of 32,768 and 16 updates per iteration. We will add direct links and a hyperparameter table to the paper.

R3. Cross-dataset evaluation

We evaluate frozen WOMD checkpoints zero-shot on nuPlan Val14 and interPlan. Both policies share the observation interface and controller within each benchmark. The Val14 adapter uses the logged segment endpoint as the goal and retains goal braking and trajectory padding. The hybrids use the learned trajectory when the best PDM proposal scores below 0.1.

Official closed-loop scores out of 100.

MethodVal14 non-reactiveVal14 reactiveinterPlan reactive
APEX35.9439.8813.65
PPO34.4838.825.66
PDM only92.8492.1341.81
APEX + PDM91.1491.0142.43
PPO + PDM90.8090.9542.69

Despite billions of reinforcement learning steps, both policies score relatively low when transferred directly, so we also evaluate their combinations with PDM. Standalone PDM has the highest mean on Val14. On interPlan, APEX + PDM and PPO + PDM exceed it by 0.62 and 0.88 points, consistent with the main table's trend toward greater competitiveness of learned policies in interactive scenarios. Under direct deployment, APEX has a higher mean than PPO in all three settings, with the largest gap on interPlan (+7.99 points, paired 95% CI [-0.14, 13.66]).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.