- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a new rl post-training method for traffic simulation. Existing approaches allocate most useful learning signal to scenes that the supervised policy already handles well. Instead, it focus on the hardest scene. The proposed method, RePT, makes two changes to post-training. First, it fixes a pool consisting of the lowest-RMM quartile of the supervised model before training and post-trains only on those scenes. Second, it adds a trajectory-space diversity bonus. Finally, it achives excellent results on WOSAC 2025.
Critical strengths
- The simulation performance is excellent.
- The idea to foucs on the part of the scenes and the trajectory diversity is novel.
- The paper identifies an important failure mode of post-training.
Critical weaknesses
- No code provided for reproducing the result.
- The gradient-mass analysis does not fully establish that optimization is dominated by easy scenes. Figure 2 computes the share of the sum of per-scene gradient norms. However, it (sum of gradient norm) is not equivalent to the contribution of those examples to the actual minibatch update (influenced by the norm of gradient sum). Gradients within one difficulty bin may align or cancel differently.
- Computing the realism score using 32 rollouts is compuationally expensive. The computation cost for post-training is unclear.
Questions for authors
- How does the ppo work compared with the grpo? Does it also have the problem caused by the group-based advantage?
- How does the rolling hardest method work compared with the fix one?
- Can the authors directly show trajectory diversity over training? Does diversity remain after post-training?
4 comments
Response to Reviewer SrpR (2/2)
[Q1] Comparison with PPO
We believe the failure to improve the hardest quartile comes from where the reward is placed, not from the group-based advantage. SMART-R1 (threshold-based) and DecompGAIL (adversarial), which are not group-based, also fail to improve the hardest quartile (Table 3). To test this directly, we trained PPO, which computes the advantage with a value network. We started from the same supervised checkpoint, used the same RMM reward, and trained separately on the full pool and on the hardest-quartile pool.
RMM per quartile:
| Model | Q1 (hardest) | Q2 | Q3 | Q4 (easiest) |
|---|---|---|---|---|
| Base | 0.7162 | 0.7786 | 0.8054 | 0.8483 |
| PPO, full pool | 0.7138 | 0.7779 | 0.8049 | 0.8479 |
| PPO, hardest-quartile pool | 0.7142 | 0.7784 | 0.8051 | 0.8475 |
| Hard (GRPO, no bonus) | 0.7149 | 0.7809 | 0.8059 | 0.8474 |
| RePT | 0.7179 | 0.7810 | 0.8059 | 0.8481 |
Both PPO settings lower the hardest quartile below Base (0.7138 and 0.7142 vs. 0.7162), and the other quartiles are also slightly below Base. The SMART-R1 paper also reports that PPO falls below the supervised model (0.7734 → 0.7611), and its authors attribute this to the value network being hard to train, which makes training unstable. Even when PPO is trained on the hardest-quartile pool, the hardest quartile does not improve, which matches Hard, trained on the same pool without the bonus. So whether the advantage is computed within the group or with a value network, the hard scenes did not improve without the bonus, and our conclusion holds: improving them requires both placing the reward on them and adding the diversity bonus. These results are in Table 16 of Appendix A.11.
[Q2] Comparison with a rolling pool
We fixed the pool to keep the results interpretable. If the hard scenes are re-selected during training, a curriculum effect is mixed in, and the gain can no longer be attributed to where the reward is placed. For comparison, we also trained a rolling version that re-selects the hardest 25% of training scenes, scored with the current policy, every 10k steps (twice during training, at 10k and 20k steps).
Overall metrics:
| Model | RMM ↑ | Kinematic ↑ | Interactive ↑ | Map-based ↑ | minADE ↓ |
|---|---|---|---|---|---|
| RePT (fixed) | 0.7882 | 0.4897 | 0.8171 | 0.9217 | 1.274 |
| RePT (rolling) | 0.7877 | 0.4895 | 0.8165 | 0.9212 | 1.283 |
RMM per quartile:
| Model | Q1 (hardest) | Q2 | Q3 | Q4 (easiest) |
|---|---|---|---|---|
| RePT (fixed) | 0.7179 | 0.7810 | 0.8059 | 0.8481 |
| RePT (rolling) | 0.7162 | 0.7808 | 0.8058 | 0.8480 |
The rolling version is almost the same as the fixed version on the three easier quartiles (Q2–Q4), but it does not improve the hardest quartile: its Q1 is 0.7162, the same as Base, while the fixed version raises Q1 to 0.7179. In this setting, the fixed pool thus worked better on the hard scenes, in addition to keeping the results interpretable. One possible reason is that a rolling pool can drop a scene before the bonus has kept several branches alive in that scene long enough to find a solution. The rolling version also costs more, because it has to re-score the training set during training. These results are in Table 15 of Appendix A.10 and summarized in the last sentence of the "Placement versus pool size" paragraph in Sec. 4.5.
[Q3] Diversity during and after training
We compared Hard and RePT, which differ only in the diversity bonus.
Clusters per group during training (K = 4):
| Step | 0 | 6k | 12k | 18k | 24k | 30k |
|---|---|---|---|---|---|---|
| Hard | 2.47 | 2.25 | 1.92 | 1.84 | 1.73 | 1.68 |
| RePT | 2.47 | 2.41 | 2.35 | 2.39 | 2.47 | 2.51 |
Diversity on the hardest quartile after training (32 rollouts per scene):
| Model | Spread ↑ | # Clusters ↑ |
|---|---|---|
| Base | 4.03 | 10.35 |
| Hard | 3.31 | 7.83 |
| RePT | 4.05 | 10.42 |
Under Hard, the number of clusters drops by about 32% during training, and after training its diversity is still much lower than that of Base. RePT keeps its diversity both during and after training. The results after training are in Table 4 of the main text and Table 9 of Appendix A.8, and the results during training are in Table 10 of Appendix A.8.
Thank the authors for the detailed rebuttal. I have one more question. Do you have some insights on which type of scenario is hard (hard to learn or hard to handle)?
Response to Reviewer SrpR's Follow-up Question
Thank you for the follow-up question. Our answer is "both, and the two are linked": the hard scenes are hard to handle, and this is what makes them hard to learn by post-training.
Hard to handle. From inspecting rollouts (Figures 3–5) and from statistics of the logged scenes, we characterise the hard scenes as those in which agents pass close to one another or turn (hardest vs. easiest quartile: in the log, 3.5 vs. 1.6 pairs of agents per scene, at least one of them moving, pass within 1 m of each other, and 23% vs. 6% of the moving agents turn by more than 30°). By construction, the two quartiles differ in BASE's closed-loop score (RMM 0.7162 vs. 0.8483, Table 3). Given the recorded history, BASE predicts the next action of a moving agent with top-1 accuracy 18.0% on the hardest quartile and 20.6% on the easiest. We read this as the difficulty appearing mainly when the model rolls the scene out.
Hard to learn. With the reward over the whole training set, these scenes receive the weakest signal (17.1% of the summed gradient, against 36.2% for the easiest quartile; Table 13). With the reward on these scenes alone, the signal reaches them, but none of the rollouts scores well there yet, and maximizing the reward keeps the highest-scoring one and stops drawing the others: the rollouts narrow onto fewer behaviors (10.35 → 7.83 clusters, Table 4) and the quartile ends below BASE (Hard in Table 3).
The two components of RePT answer these two problems in turn: placing the reward on the hard scenes brings the signal to them, and the diversity bonus keeps alternative behaviors sampled until the reward can tell which one resolves the scene. With both, the quartile rises above BASE (Table 3) and its collision rate falls from 25.16% to 19.03% (Table 4).
Response to Reviewer SrpR (1/2)
Thank you for your positive comments on the performance of RePT, on the novelty of focusing on a subset of scenes and of using trajectory diversity, and on our identification of how post-training fails.
Following your comments, we:
The new results are in the main text and the appendix of the revised paper, with changes marked in blue. For each change, we give its location in the revision. We will release the code.
[W2] Sum of gradient norms vs. the actual update
This is a fair point. Figure 2 sums the per-scene gradient norms, so it does not reflect how gradients within a quartile align or cancel. We therefore measured, at the same checkpoint, how much each quartile contributes to the total gradient G. For quartile q with summed gradient G_q, the contribution is ⟨G_q, G⟩ / ‖G‖², and the contributions of the four quartiles add up to one.
Measured by the actual update, the imbalance is larger than in Figure 2: the contribution of the hardest quartile drops to 17.1%, and that of the easiest quartile rises to 36.2%. This is because the gradients of easy scenes point in similar directions and add up, while those of hard scenes point in different directions and partly cancel. Taking alignment and cancellation into account thus makes our conclusion clearer: the update leans toward the scenes the model already handles well. In the revision, the caption of Figure 2 now states what is measured (the sum of per-scene gradient norms), and we added one sentence on this result to the "Where the gradient goes" paragraph in Sec. 4.5. The method and the numbers are in Table 13 of Appendix A.10.
[W3] Compute cost
During training, we draw K = 4 rollouts per scene and compute the RMM of each rollout on its own (q_k = M(τ_k)). We compute RMM over 32 rollouts per scene only for pool selection before training and for evaluation.
The pool is selected only once before training, and this one-time cost is shared by all settings. For the same number of steps, RePT takes about 6% (2 hours) longer than Full post-training, because it computes distances among the K = 4 rollouts and clusters them. The costs are in Table 14 of Appendix A.10, which we point to at the end of the "Objective settings" paragraph in Sec. 4.2.