- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
APEX aims to improve the robustness of driving policies trained through self-play by adapting the traffic behaviors they encounter during training. It separates a focal driving policy from a behavior-conditioned traffic policy and learns a population constructor that samples traffic conditions, including reward preferences and visibility constraints. The constructor uses the difference between the focal critic’s initial value estimate and the observed return to emphasize challenging traffic populations. The paper analyzes this signal under a first-exposure protocol and evaluates the method on BehaviorBench’s Interactive1k and Random1k splits. The results show stronger improvements on Interactive1k and lower collision rates across several traffic populations, while aggregate performance on Random1k remains close to PPO.
Critical strengths
The paper addresses a useful problem: a driving policy can perform well against its training partners yet generalize poorly to other traffic behaviors. Adapting the distribution of training partners is a plausible approach, and the resulting focal policy can be deployed without the population constructor or behavior-conditioning inputs.
Critical weaknesses
-
The distinction from existing self-play methods is not sufficiently established. The motivation associates shared policies with homogeneous interactions, although methods such as GigaFlow already produce diverse behavior through agent-specific conditioning. The central contribution is therefore the adaptive population distribution, but the experiments do not adequately establish why this adaptation is necessary or whether it tracks evolving focal-policy weaknesses.
-
The experimental evidence is too limited to establish the reliability and scope of the gains. Evaluation is restricted to BehaviorBench, and the reported training results use a single seed. On Random1k, the aggregate score is close to PPO despite improvements in collision rates, leaving the broader benefit unclear.
Questions for authors
-
What numerical collection threshold N triggers a constructor update, and how many policy updates typically occur between constructor updates? Please also specify the constructor optimization settings and exploration-scale schedule.
-
Can the authors double check Figure 1 (b) & (c)? The symbols are missing there.
-
Why equation 4 shares the same input for focal policy, conditioned policy, and the population constructor?
Thank you for recognizing the motivation for adapting training partners and the simplicity of deploying the focal policy.
R1. Adaptive population construction
Table 2 in the paper adds components in stages. PPO uses one unconditioned self-play policy. BC PPO uses GigaFlow-style per-agent reward and perception conditioning in a shared policy. Fixed dual separates the focal and traffic policies and samples conditions from a fixed distribution. APEX retains both policies and learns the sampling distribution.
Across seven shared traffic settings on Interactive1k, mean collision rates for PPO, BC PPO, Fixed dual, and APEX are 18.23%, 14.74%, 11.57%, and 9.86%, respectively. Mean goal completion is 75.67%, 74.74%, 79.00%, and 80.69%. Fixed dual and APEX both train two policies with a 4B total budget. On Random1k, APEX reduces collisions relative to Fixed dual from 5.40% to 4.26% and raises goal completion from 90.07% to 90.77%. These gains at the same budget support adaptive sampling as a key component of APEX.
R2. Reliability, training seeds, and scope
On Interactive1k, APEX outscored PPO by 2.71 points (95% CI [1.96, 3.46]), with a 6.79 percentage point reduction in at-fault collision rate and a 4.53 point gain in goal completion. On Random1k, aggregate scores differed by 0.34 points, while APEX lowered the at-fault collision rate by 2.44 percentage points and raised goal completion by 2.89 percentage points. We obtained the interval by resampling scenes jointly for both policies, measuring sensitivity to the choice of test scenes for the fixed checkpoints.
We trained APEX twice at seed 42 using an alternative training configuration. Under the main-table evaluation protocol for Interactive1k, both runs lowered overall and at-fault collision rates and improved goal completion in each of the seven shared traffic settings. Across these settings, the mean collision rate fell from 18.23% with PPO to 11.13–11.23% with APEX, and goal completion rose from 75.67% to 79.91–80.69%. For the seed-43 APEX run, we retained the core training settings and evaluated on Interactive1k under IDM traffic. Compared with PPO, APEX reduced overall collisions from 13.9% to 7.1% and at-fault collisions from 9.9% to 4.4%, while increasing goal completion from 75.1% to 77.7%. These results suggest that APEX consistently reduces collisions and improves goal completion across the tested training configurations and seeds.
We deployed the frozen WOMD policies on nuPlan Val14 and interPlan using the same observation interface and controller. APEX led PPO by 1.46 points on non-reactive Val14, 1.06 on reactive Val14, and 7.99 on reactive interPlan. On reactive Val14, the no-at-fault-collision score rose from 64.49 to 66.68. On interPlan, comfort compliance rose from 66.25% to 81.25% and drivable-area compliance from 17.50% to 30.00%.
Q1. Constructor threshold, update frequency, and exploration
N=256 is the minimum number of completed scenario-window records that triggers an update. Each update uses all valid records available at that time. The 268 recorded updates used 2,055–2,329 records each (median 2,194). The first update occurred at driving iteration 15. The next 267 intervals comprised 205 intervals of 14 driving iterations and 62 of 15 (mean 14.23). One driving iteration is a rollout and training cycle.
We use Adam with a learning rate of 0.0003, four optimization epochs per constructor update, minibatch size 64, PPO clipping 0.2, target KL 0.03, and a maximum gradient norm of 1. The exploration standard deviation decays from 0.5 to 0.2 over 600,000 descriptor records, counted separately from driving agent steps.
Q2. Missing symbols in Figure 1(b) and 1(c)
Thank you for pointing this out. Fig. 1(b) labels the traffic policy πω,cC. We will use πωC in both panels, show cj separately as a policy input in Fig. 1(b), and check the rendered PDF.
Q3. Shared arguments in Equation (4)
In Eq. (4), (θ,ω,φ) denotes the dependence of JB, JC, and JBexp on the three modules' parameters. Both driving policies shape the closed-loop trajectories, and the constructor determines the distribution of training traffic. For example, changing φ can alter the focal policy's return by changing the traffic it encounters, even with θ fixed.
For action selection, the focal policy uses its local observation and recurrent state. The conditioned policy additionally receives its vehicle's behavior condition. For each map window, the constructor samples a descriptor from the global distribution qφ that defines how traffic conditions are drawn. Each module updates its own parameters under its own objective. We will clarify the distinction between objective arguments and module inputs in the revision.
Implementation and training details are available in our anonymous repository.