acceptodds
Under review as a conference paper at ICLR 2027

RePT: Re-targeted Post-Training for Multi-Agent Traffic Simulation

Abstract

Traffic agents for closed-loop evaluation are trained by imitating logged driving, and reinforcement learning post-training is meant to repair the interactions imitation gets wrong: the merges and unprotected turns where a driver could yield or push ahead, but the log shows only what one driver did. We show that current post-training does not fix these scenes, and trace the reason to where its reward is placed. Existing methods reward each rollout for resembling recorded driving and apply that reward across the whole training set. Most of the gradient therefore goes to scenes the model already drives well. On the difficult scenes every rollout fails, so rewards are nearly equal, the update is weak, and the little signal left simply reinforces the least bad failure. RePT (Re-targeted Post-Training) makes two changes. It post-trains only on the scenes the supervised model scores lowest, chosen once before training. And because these scenes collapse onto one failing behavior before any alternative can succeed, it adds a diversity bonus: rollouts are clustered by their joint trajectories, since what must stay distinct is whether an agent yields or pushes ahead, each is paid for standing outside the others' clusters, and none is paid once its realism falls below the group. Alternatives thus stay in play until the realism reward can tell which one clears the interaction. On WOSAC 2025, RePT sets the highest realism reported and is the only post-training we test, ours or published, that improves the hardest scenes at all, with no change to architecture or inference cost.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Traffic simulation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SrpROpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new rl post-training method for traffic simulation. Existing approaches allocate most useful learning signal to scenes that the supervised policy already handles well. Instead, it focus on the hardest scene. The proposed method, RePT, makes two changes to post-training. First, it fixes a pool consisting of the lowest-RMM quartile of the supervised model before training and post-trains only on those scenes. Second, it adds a trajectory-space diversity bonus. Finally, it achives excellent results on WOSAC 2025.

Critical strengths
  1. The simulation performance is excellent.
  2. The idea to foucs on the part of the scenes and the trajectory diversity is novel.
  3. The paper identifies an important failure mode of post-training.
Critical weaknesses
  1. No code provided for reproducing the result.
  2. The gradient-mass analysis does not fully establish that optimization is dominated by easy scenes. Figure 2 computes the share of the sum of per-scene gradient norms. However, it (sum of gradient norm) is not equivalent to the contribution of those examples to the actual minibatch update (influenced by the norm of gradient sum). Gradients within one difficulty bin may align or cancel differently.
  3. Computing the realism score using 32 rollouts is compuationally expensive. The computation cost for post-training is unclear.
Questions for authors
  1. How does the ppo work compared with the grpo? Does it also have the problem caused by the group-based advantage?
  2. How does the rolling hardest method work compared with the fix one?
  3. Can the authors directly show trajectory diversity over training? Does diversity remain after post-training?
4 comments
AuthorsOpenReview ↗

Response to Reviewer SrpR (1/2)

Thank you for your positive comments on the performance of RePT, on the novelty of focusing on a subset of scenes and of using trajectory diversity, and on our identification of how post-training fails.

Following your comments, we:

  • W2: measured how much each quartile contributes to the actual update;
  • W3: summarized the compute cost;
  • Q1: trained PPO under the same conditions for comparison;
  • Q2: compared the fixed pool with a rolling pool;
  • Q3: measured diversity during and after training.

The new results are in the main text and the appendix of the revised paper, with changes marked in blue. For each change, we give its location in the revision. We will release the code.

[W2] Sum of gradient norms vs. the actual update

This is a fair point. Figure 2 sums the per-scene gradient norms, so it does not reflect how gradients within a quartile align or cancel. We therefore measured, at the same checkpoint, how much each quartile contributes to the total gradient G. For quartile q with summed gradient G_q, the contribution is ⟨G_q, G⟩ / ‖G‖², and the contributions of the four quartiles add up to one.

QuartileShare of Σ‖g‖ (Figure 2)Contribution to the update
Q1 (hardest)21.6%17.1%
Q223.5%21.9%
Q323.3%24.9%
Q4 (easiest)31.6%36.2%

Measured by the actual update, the imbalance is larger than in Figure 2: the contribution of the hardest quartile drops to 17.1%, and that of the easiest quartile rises to 36.2%. This is because the gradients of easy scenes point in similar directions and add up, while those of hard scenes point in different directions and partly cancel. Taking alignment and cancellation into account thus makes our conclusion clearer: the update leans toward the scenes the model already handles well. In the revision, the caption of Figure 2 now states what is measured (the sum of per-scene gradient norms), and we added one sentence on this result to the "Where the gradient goes" paragraph in Sec. 4.5. The method and the numbers are in Table 13 of Appendix A.10.

[W3] Compute cost

During training, we draw K = 4 rollouts per scene and compute the RMM of each rollout on its own (q_k = M(τ_k)). We compute RMM over 32 rollouts per scene only for pool selection before training and for evaluation.

StageCost
Difficulty pool selection (once, shared by all settings)120 GPU-hours (8 B200 GPUs, 15 h)
Full post-training (30k steps)248 GPU-hours (8 B200 GPUs, 31 h)
RePT post-training (30k steps)264 GPU-hours (8 B200 GPUs, 33 h)

The pool is selected only once before training, and this one-time cost is shared by all settings. For the same number of steps, RePT takes about 6% (2 hours) longer than Full post-training, because it computes distances among the K = 4 rollouts and clusters them. The costs are in Table 14 of Appendix A.10, which we point to at the end of the "Objective settings" paragraph in Sec. 4.2.

AuthorsOpenReview ↗

Response to Reviewer SrpR (2/2)

[Q1] Comparison with PPO

We believe the failure to improve the hardest quartile comes from where the reward is placed, not from the group-based advantage. SMART-R1 (threshold-based) and DecompGAIL (adversarial), which are not group-based, also fail to improve the hardest quartile (Table 3). To test this directly, we trained PPO, which computes the advantage with a value network. We started from the same supervised checkpoint, used the same RMM reward, and trained separately on the full pool and on the hardest-quartile pool.

RMM per quartile:

ModelQ1 (hardest)Q2Q3Q4 (easiest)
Base0.71620.77860.80540.8483
PPO, full pool0.71380.77790.80490.8479
PPO, hardest-quartile pool0.71420.77840.80510.8475
Hard (GRPO, no bonus)0.71490.78090.80590.8474
RePT0.71790.78100.80590.8481

Both PPO settings lower the hardest quartile below Base (0.7138 and 0.7142 vs. 0.7162), and the other quartiles are also slightly below Base. The SMART-R1 paper also reports that PPO falls below the supervised model (0.7734 → 0.7611), and its authors attribute this to the value network being hard to train, which makes training unstable. Even when PPO is trained on the hardest-quartile pool, the hardest quartile does not improve, which matches Hard, trained on the same pool without the bonus. So whether the advantage is computed within the group or with a value network, the hard scenes did not improve without the bonus, and our conclusion holds: improving them requires both placing the reward on them and adding the diversity bonus. These results are in Table 16 of Appendix A.11.

[Q2] Comparison with a rolling pool

We fixed the pool to keep the results interpretable. If the hard scenes are re-selected during training, a curriculum effect is mixed in, and the gain can no longer be attributed to where the reward is placed. For comparison, we also trained a rolling version that re-selects the hardest 25% of training scenes, scored with the current policy, every 10k steps (twice during training, at 10k and 20k steps).

Overall metrics:

ModelRMM ↑Kinematic ↑Interactive ↑Map-based ↑minADE ↓
RePT (fixed)0.78820.48970.81710.92171.274
RePT (rolling)0.78770.48950.81650.92121.283

RMM per quartile:

ModelQ1 (hardest)Q2Q3Q4 (easiest)
RePT (fixed)0.71790.78100.80590.8481
RePT (rolling)0.71620.78080.80580.8480

The rolling version is almost the same as the fixed version on the three easier quartiles (Q2–Q4), but it does not improve the hardest quartile: its Q1 is 0.7162, the same as Base, while the fixed version raises Q1 to 0.7179. In this setting, the fixed pool thus worked better on the hard scenes, in addition to keeping the results interpretable. One possible reason is that a rolling pool can drop a scene before the bonus has kept several branches alive in that scene long enough to find a solution. The rolling version also costs more, because it has to re-score the training set during training. These results are in Table 15 of Appendix A.10 and summarized in the last sentence of the "Placement versus pool size" paragraph in Sec. 4.5.

[Q3] Diversity during and after training

We compared Hard and RePT, which differ only in the diversity bonus.

Clusters per group during training (K = 4):

Step06k12k18k24k30k
Hard2.472.251.921.841.731.68
RePT2.472.412.352.392.472.51

Diversity on the hardest quartile after training (32 rollouts per scene):

ModelSpread ↑# Clusters ↑
Base4.0310.35
Hard3.317.83
RePT4.0510.42

Under Hard, the number of clusters drops by about 32% during training, and after training its diversity is still much lower than that of Base. RePT keeps its diversity both during and after training. The results after training are in Table 4 of the main text and Table 9 of Appendix A.8, and the results during training are in Table 10 of Appendix A.8.

Thank the authors for the detailed rebuttal. I have one more question. Do you have some insights on which type of scenario is hard (hard to learn or hard to handle)?

AuthorsOpenReview ↗

Response to Reviewer SrpR's Follow-up Question

Thank you for the follow-up question. Our answer is "both, and the two are linked": the hard scenes are hard to handle, and this is what makes them hard to learn by post-training.

Hard to handle. From inspecting rollouts (Figures 3–5) and from statistics of the logged scenes, we characterise the hard scenes as those in which agents pass close to one another or turn (hardest vs. easiest quartile: in the log, 3.5 vs. 1.6 pairs of agents per scene, at least one of them moving, pass within 1 m of each other, and 23% vs. 6% of the moving agents turn by more than 30°). By construction, the two quartiles differ in BASE's closed-loop score (RMM 0.7162 vs. 0.8483, Table 3). Given the recorded history, BASE predicts the next action of a moving agent with top-1 accuracy 18.0% on the hardest quartile and 20.6% on the easiest. We read this as the difficulty appearing mainly when the model rolls the scene out.

Hard to learn. With the reward over the whole training set, these scenes receive the weakest signal (17.1% of the summed gradient, against 36.2% for the easiest quartile; Table 13). With the reward on these scenes alone, the signal reaches them, but none of the rollouts scores well there yet, and maximizing the reward keeps the highest-scoring one and stops drawing the others: the rollouts narrow onto fewer behaviors (10.35 → 7.83 clusters, Table 4) and the quartile ends below BASE (Hard in Table 3).

The two components of RePT answer these two problems in turn: placing the reward on the hard scenes brings the signal to them, and the diversity bonus keeps alternative behaviors sampled until the reward can tell which one resolves the scene. With both, the quartile rises above BASE (Table 3) and its collision rate falls from 25.16% to 19.03% (Table 4).

Reviewer_ZbZ3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a new RL post-training method to improve the realism of token-based traffic simulation models trained using IL. Unlike previous RL post training approaches, they carry over insights from RL post-training for LLMs, focus on the hardest quartile of scenarios with lowest realism to mitigate difficulty bias, and optimize the base policy with GRPO to ensure improved realism there. To do this effectively, the authors propose a diversity bonus term with a gate to ensure baseline realism, and show that this setup can achieve SOTA on WOSAC. The experiments are crisp, well-motivated, and the ablations are sensible.

Critical strengths
  • The paper draws a very interesting analogy to issues encountered in RL post-training for LLMs and similar problems that show up in RL post training for traffic simulation models, albeit with a few key differences. This insight, in my opinion is the main contribution of this paper, which highlights the drawbacks of naive RL training to optimize a benchmark's metric, versus their proposed approach which actually addresses the core issue and improves realism in hard, interactive scenarios.
  • Their proposed approach achieves SOTA by a small margin on the WOSAC benchmark with a simple, intuitive RL post-training recipe that does not add any inference cost.
Critical weaknesses
  • The WOSAC benchmark is quite saturated, and the RL post-training approaches that are trained to optimize for the realism meta metric (RMM) are doing just that - optimizing the meta metric. But the metric itself is not a true reflection of the model's utility in actual traffic simulation. Beyond a certain point, I am less interested in the 0.2% improvement in the RMM over previous SOTA, and more interested in how useful it is for simulating traffic in hard scenarios. Any qualitative/quantitative results that the authors can offer beyond the WOSAC benchmark would be highly appreciated. For example, I would love to see metrics that compare diversity and collision rates of rollouts of the proposed approach versus baselines on the hardest quartile (Table 3). Diversity bonus especially is one of their main contributions, so it should be measured and reported. The paper never directly measures whether the bonus keeps several modes alive. For example, number of clusters per group or spread across the 32 rollouts for HARD vs RePT.
  • A lot of the paper is clearly written/translated by AI, which makes it jarring to read and hard to follow in some cases. This is especially obvious in the introduction, related works, and experiments sections. There are no correctness issues as far as I can make out, and the authors do disclose this in their AI use statement. I would suggest the authors re-write all sections where the AI use is heavy to sound more human-like which would make it easier for readers to follow.
Questions for authors
  • Why are there little to know details on the performance of your approach on VRUs (peds and cyclists)? Including VRU-only RMM metrics and qualitative results with discussions would round out the paper.
  • Why is minADE worse for all RL post-trained models compared to base model? A quartile-specific minADE might help explain this better and would provide supporting evidence for the author's claims.
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZbZ3 (1/2)

Thank you for your careful reading and comments. We are glad that you see the link between the problems of RL post-training for language models and those for traffic simulation as the main contribution of the paper, and that you find our approach addresses the core issue and improves realism in hard, interactive scenes, rather than simply optimizing the benchmark metric. Thank you also for noting that the method is simple and intuitive and adds no inference cost.

Following your comments, we:

  • W1: measured the collision rate and diversity on the hardest quartile, and directly compared Hard and RePT to check whether the bonus keeps several modes alive;
  • Q1: added results computed separately for pedestrians and for cyclists;
  • Q2: added minADE by quartile;
  • W2: rewrote the paper, focusing on the sections you pointed out.

The new results are in the main text and the appendix of the revised paper, with changes marked in blue. For each change, we give its location in the revision.

[W1] Metrics beyond RMM and direct measurement of diversity

We agree that RMM, which combines several likelihoods, is not enough to show what RePT actually changes on hard scenes. We measured two things.

(a) Collision rate and diversity on the hardest quartile

For Base, the six training configurations from the paper (Full, Full+Div, Easy+Div, Random+Div, Hard, RePT) and the two baselines (SMART-R1 and DecompGAIL), we measured the following on the hardest quartile (Q1 in Table 3):

  • Collision rate: the fraction of the 32 rollouts with at least one collision.
  • Spread: the mean trajectory distance over all pairs of the 32 rollouts (normalized to a per-agent, per-step displacement, as in Eq. (1)).
  • # Clusters: the number of clusters when the 32 rollouts are clustered with the same rule as in training.
ModelCollision rate ↓Spread ↑# Clusters ↑
Base25.16%4.0310.35
Full25.56%3.569.90
Full+Div25.43%3.9410.22
Easy+Div28.50%3.9810.32
Random+Div25.31%3.8810.12
Hard25.01%3.317.83
RePT19.03%4.0510.42
SMART-R125.24%3.8610.04
DecompGAIL24.42%3.9310.15

On the hardest quartile, RePT lowers the collision rate from 25.16% to 19.03%, a relative reduction of about 24%. All other methods except Easy+Div stay within 0.8 percentage points of Base. Easy+Div, trained only on easy scenes, instead raises the collision rate to 28.50%.

The effect of the bonus on diversity is clear. Hard, trained only on hard scenes without the bonus, has about 24% fewer clusters (10.35 → 7.83) and 18% less spread, so its rollouts concentrate on a few branches. RePT, which differs from Hard only in the bonus, keeps both the number of clusters and the spread at the level of Base.

The other configurations with the bonus (Full+Div, Easy+Div, Random+Div) also mostly keep their diversity, but they do not reduce collisions. Keeping diversity alone therefore does not improve results on hard scenes. When the bonus acts on hard scenes, it keeps several branches, and training selects the one that avoids the collision.

(b) Hard vs. RePT: does the bonus keep several modes alive?

As you suggested, we compared Hard and RePT directly. The two share the training pool and all other settings and differ only in the diversity bonus. During training, we recorded how many clusters each group (the K = 4 rollouts of one scene) splits into.

Clusters per group during training (max 4):

Step06k12k18k24k30k
Hard2.472.251.921.841.731.68
RePT2.472.412.352.392.472.51

Without the bonus, the number of clusters per group under Hard falls steadily during training and ends about 32% lower than at the start (2.47 → 1.68): the rollouts drawn for the same scene gather into fewer and fewer branches. Under RePT, the count also drops a little early on (2.35 at 12k steps), but then recovers and ends slightly above its starting value (2.51). In both settings, reward maximization tends to narrow the rollouts; the diversity bonus works against this and keeps several branches.

The results in (a), measured with 32 rollouts after training, point the same way: Hard has fewer clusters and less spread than Base (−24%, −18%), while RePT stays at the level of Base. In the original paper, we showed the narrowing of the rollouts only indirectly, through the drop in the kinematic score; these measurements show it directly through the number of clusters. In the revision, we added these results for the main models to Table 4 in the main text and discussed them at the end of the "Narrowing on difficult scenes" paragraph in Sec. 4.5. The results for all models are in Table 9, and the number of clusters during training is in Table 10 (both in Appendix A.8).

AuthorsOpenReview ↗

Response to Reviewer ZbZ3 (2/2)

[Q1] Results on VRUs (pedestrians and cyclists)

We computed RMM separately for pedestrians and for cyclists. Since the official WOSAC evaluation aggregates scores over each scene, we ran the same evaluation code on the agents of one type at a time.

ModelPedestrian ↑Cyclist ↑
Base0.77570.6982
SMART-R10.77760.6988
DecompGAIL0.77800.6990
RePT0.77930.7012

RePT gives the largest gain over Base for both pedestrians and cyclists (+0.0036, +0.0030). For cyclists in particular, the other two methods stay close to Base (+0.0006, +0.0008), while RePT gains +0.0030. Cyclists share lanes with vehicles, and yielding and overtaking between them are common, so we think this is where the effect of RePT, which targets hard interactions, shows clearly. These results are in Table 11 of Appendix A.9. We also added two scenes from the hardest quartile to Figure 5: one in which a pedestrian interacts with a vehicle and one in which a cyclist does. In both scenes, Base, SMART-R1, DecompGAIL and Hard produce collisions in some of the 32 rollouts (5–7 of the 32 in the pedestrian scene and 1–10 in the cyclist scene, depending on the model), while RePT has no collision in any of the 32.

[Q2] minADE by quartile

We report minADE on each of the quartiles used in Table 3 of the main text.

ModelQ1 (hardest)Q2Q3Q4 (easiest)
Base1.6791.3621.0760.940
Full1.6961.3701.0860.953
Full+Div1.7091.3791.0940.995
Easy+Div1.7381.3971.1121.002
Random+Div1.7351.4061.1261.005
Hard1.7131.3821.1040.998
RePT1.6781.3651.0810.972
SMART-R11.6851.3511.0420.928
DecompGAIL1.7231.3891.0920.958

minADE is the error of whichever of the 32 rollouts is closest to the logged trajectory, so spreading the rollouts can either lower or raise it, depending on the scenes the bonus is trained on. In a hard scene, the supervised model settles on an average of the possible ways to resolve the scene; when the bonus splits the rollouts into branches, one branch can move closer to the logged trajectory. In an easy scene, the rollouts already sit near the logged trajectory, so a rollout can stand apart only by moving away from it, and the bonus rewards it for doing so as long as it stays plausible.

The quartile results match this. With the hardest-quartile pool, the bonus lowers minADE on all four quartiles: RePT is below Hard, which trains on the same scenes without the bonus (1.678 vs. 1.713 on Q1). With the full pool, which also holds easy scenes, the bonus raises minADE on all four quartiles (Full+Div vs. Full). On the hardest quartile, RePT has the lowest minADE of all models (1.678), about the same as Base (1.679). The increase in the overall minADE of RePT over Base comes mostly from the easiest quartile (0.940 → 0.972), and this increase is already present without the bonus: Hard, trained on the same pool, rises more there (0.998).

SMART-R1 has a lower minADE than Base on quartiles Q2–Q4. We think this comes from its supervised stages rather than from its reward: SMART-R1 trains in three stages (SFT, then RFT, then SFT again), and SFT pulls the rollouts toward the logged trajectories. The SMART-R1 paper shows the same pattern: its RFT stage raises minADE (1.398 → 1.422), while its SFT stages lower it. On the hardest quartile, however, SMART-R1 has a higher minADE than Base (1.685 vs. 1.679) and a lower RMM (Table 3), and it does not reduce collisions (W1). minADE is not part of RMM, and a lower overall minADE does not mean that hard interactions are handled well. The quartile minADE is in Table 12 of Appendix A.9, and we added a summary sentence at the end of the "minADE" paragraph in Sec. 4.3.

[W2] Writing style and readability

We are sorry that the paper was hard to read. We rewrote the paper, mainly the sections you pointed out (Introduction, Related Work and Experiments), and marked the changes in blue:

  • Replaced figurative wording with standard RL terms (e.g., "the bar" → "baseline", "paid" → "rewarded", "siblings" → "the other rollouts in its group", "go silent" → "gives no gradient").
  • Rewrote rhetorical questions, and sentences that address the reader directly, as plain statements.
  • Gave four separate reasons in the Introduction why remedies developed for language models cannot be applied directly (fourth paragraph of the Introduction).
  • Added the supporting numbers to many sentences that state a result in the Experiments section, and gave each paragraph a short heading (Secs. 4.3–4.5).
  • Fixed notation and equation errors (first paragraph of Sec. 3, Sec. 3.2, Eq. (5)).
Reviewer_EzVVOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes Re-targeted Post-Training, a framework for improving multi-agent traffic simulation in challenging driving scenarios. RePT consists of two components: (i) fixed difficulty-based targeting, which focuses post-training on the lowest-scoring quartile of scenes identified by the supervised model before training; and (ii) realism-gated trajectory diversity, which clusters joint multi-agent trajectories and rewards diverse rollouts only when their realism remains sufficiently close to the group mean. Experiments on WOSAC 2025 demonstrate competitive performance.

Critical strengths

RePT achieves the highest RMM among the methods reported in the WOSAC 2025 test-set comparison. It also improves the interactive score without compromising map-based performance relative to the TrajTok baseline.

Critical weaknesses
  1. The method combines difficulty-based data selection, GRPO, and a realism-gated diversity bonus. However, the paper does not sufficiently establish the benefits of the specific trajectory-space clustering and gating design over simpler alternatives. Moreover, the overall performance improvement appears marginal, making the practical significance of the proposed method less clear.

  2. The paper's presentation could be improved in terms of clarity and organization. Some explanations of the proposed method and experimental findings would benefit from more precise and concise writing.

Questions for authors
  1. How essential is the proposed diversity mechanism compared with simpler alternatives? Can the authors provide controlled comparisons with entropy regularization or alternative diversity bonuses under the same hard-scene training set and compute budget? Such comparisons would help isolate the contributions of trajectory-space clustering and realism gating.

  2. How robust are the results to the choice of difficulty threshold and backbone? Would targeting different proportions of the training set or applying RePT to another tokenized traffic simulator yield consistent improvements on difficult scenes? This analysis would help establish whether the approach generalizes beyond the selected quartile and the TrajTok backbone.

  3. Were the baseline methods fairly and adequately trained or reproduced? For baselines such as DecompGAIL and SMART-R1, can the authors clarify whether they used the official implementations and recommended training configurations, and whether sufficient training and hyperparameter tuning were performed? This information is important for ensuring a fair comparison and determining whether RePT's reported gains reflect the proposed method rather than differences in implementation or training quality.

6 comments
AuthorsOpenReview ↗

Response to Reviewer EzVV (1/4)

Thank you for noting that RePT reaches the highest RMM and raises the interactive score without losing map-based performance. Your concrete feedback helped us improve the paper.

Following your comments, we:

  • W1 & Q1: compared RePT with simpler diversity methods, including entropy regularization, under the same conditions;
  • W1: measured the collision rate on the hardest quartile to show the practical significance of the gain;
  • Q2: ran experiments on another backbone (CAT-K) and with other pool sizes (12.5%, 50%);
  • Q3: described how we implemented, reproduced and tuned the baselines;
  • W2: improved the clarity and organization of the paper.

The new results are in the main text and the appendix of the revised paper, with changes marked in blue. For each change, we give its location in the revision.

[W1 & Q1] Why trajectory-space clustering and the realism gate are needed

We agree that a controlled comparison with simpler alternatives is needed to show that these two design choices are necessary. Under the same conditions, we trained three variants that change only the bonus:

  • Hard+Ent: entropy regularization on the token distribution.
  • Hard+Dist: instead of clustering, the bonus uses the mean distance d̄_k from rollout k to the other rollouts in its group, E_k = 1 + (√K − 1)(1 − exp(−d̄_k / τ_div)). Like the RePT bonus, it ranges from 1 to √K. The distance measure, the realism gate and β_div are the same as in RePT, so Hard+Dist differs from RePT only in replacing clustering with the mean distance.
  • RePT w/o gate: RePT with only the realism gate removed.

All variants use the same training pool as RePT (the hardest 25% of training scenes), K = 4, and 30k steps. All numbers below are on the validation subset. The second table gives RMM within each difficulty quartile (Q1 = hardest, Q4 = easiest).

ModelRMM ↑Kinematic ↑Interactive ↑Map-based ↑minADE ↓
Base0.78710.48900.81440.92251.264
Hard0.78730.48830.81610.92101.299
Hard+Ent0.78720.48850.81650.92031.301
Hard+Dist0.78740.48940.81630.92051.303
RePT w/o gate0.78690.48910.81560.92011.309
RePT0.78820.48970.81710.92171.274
ModelQ1 (hardest)Q2Q3Q4 (easiest)
Base0.71620.77860.80540.8483
Hard0.71490.78090.80590.8474
Hard+Ent0.71470.78100.80610.8471
Hard+Dist0.71460.78110.80600.8480
RePT w/o gate0.71440.77960.80640.8472
RePT0.71790.78100.80590.8481

All three variants have about the same RMM as Hard (0.7869–0.7874), below RePT (0.7882). RePT is also the only model that lifts the hardest quartile (Q1) above Base.

Each variant shows why one part of RePT is needed.

First, entropy regularization leaves Q1 below Base (0.7147 vs. 0.7162), as Hard does, and does not improve minADE (1.301 vs. 1.299 for Hard). This indicates that widening each agent's next-token distribution does not create more distinct maneuvers, which supports our point in Sec. 1 that tokens are the wrong unit for measuring how rollouts differ.

Second, the mean-distance bonus works in trajectory space, but it rewards spread itself rather than distinct branches. The rollouts do spread out (Hard+Dist has the highest kinematic score of the three variants, 0.4894), but Q1 stays below Base (0.7146) and minADE gets worse (1.303). More spread does not help the hard scenes unless the rollouts resolve the scene in different ways. Measuring diversity in trajectory space is therefore not enough; we also need clustering, which separates the branches and rewards them.

Third, RePT w/o gate has a higher kinematic score than Hard (0.4891 vs. 0.4883), but its map-based score and Q1 are the lowest in the table (0.9201, 0.7144), and its minADE is the worst (1.309). Without the gate, rollouts are rewarded for being different even when their realism drops.

RePT, in contrast, improves RMM, the interactive score and Q1 over Hard, and also lowers minADE (1.274). These results support our claim that the hard scenes improve only when clustering and the realism gate are used together. In the revision, the three variants are defined in the Configurations paragraph of Sec. 4.1, and their results are added to Tables 2 and 3. We discuss them in the "Simpler diversity terms" paragraph of Sec. 4.4, and give their settings, including the coefficient search range, in Appendix A.6.

AuthorsOpenReview ↗

Response to Reviewer EzVV (2/4)

[W1] Size of the improvement

We understand that the gain in overall RMM looks small. As Reviewer ZbZ3 also notes, WOSAC 2025 is saturated: all seven prior methods in Table 1 lie within a range of 0.005, so differences in overall RMM are small for any method. The gap between RePT and the previous best method (0.0015) is about 30% of this range.

The practical value of our method lies in the hardest scenes rather than in overall RMM. Post-training was introduced to fix the hard interactions that imitation learning gets wrong, yet every other post-training method compared in Table 3 lowers the hardest quartile (Q1) below Base. Only RePT raises it, and we see this as the practical significance of this work.

This gain also shows outside RMM. On the hardest quartile, we measured the collision rate, the fraction of the 32 rollouts with at least one collision:

ModelCollision rate ↓
Base25.16%
Full25.56%
Hard25.01%
SMART-R125.24%
DecompGAIL24.42%
RePT19.03%

RePT lowers the collision rate from 25.16% to 19.03%, a relative reduction of about 24%. Every other model in this table stays within 0.8 percentage points of Base, including Hard, which trains on the same scenes without the bonus. The small gap in overall RMM thus comes with a clear reduction in collisions on the hardest scenes. These results are in Table 4 of the main text; Table 9 of Appendix A.8 adds Full+Div, Easy+Div and Random+Div.

AuthorsOpenReview ↗

Response to Reviewer EzVV (3/4)

[Q2] Robustness to the backbone and the difficulty threshold

We chose TrajTok to test our method in the strictest setting: it has the highest WOSAC score among SMART-based models trained without reinforcement learning. We agree that we should check whether our method works only on TrajTok. We therefore trained Full, Hard and RePT in the same way on a second backbone: SMART-tiny fine-tuned with CAT-K (CVPR 2025), which we call CAT-K below. The difficulty pool and the evaluation quartiles were recomputed from the RMM of CAT-K, and all other settings are the same as in the TrajTok experiments. In the two tables below, Base is the CAT-K checkpoint before post-training.

ModelRMM ↑Kinematic ↑Interactive ↑Map-based ↑minADE ↓
Base0.78100.48680.80830.91391.318
Full0.78310.48590.81250.91501.348
Hard0.78400.48600.81420.91541.365
RePT0.78700.48880.81660.91921.324
ModelQ1 (hardest)Q2Q3Q4 (easiest)
Base0.70900.77540.79930.8402
Full0.70850.77740.80070.8460
Hard0.70880.77990.80320.8442
RePT0.71710.78020.80520.8456

The results agree with those on TrajTok. Full, trained on the full pool, raises RMM by 0.0021, but the gain is concentrated on the easiest quartile (Q4, +0.0058), and the hardest quartile (Q1) even drops below Base (0.7085 vs. 0.7090). Hard, trained only on the hard scenes without the bonus, does not lift Q1 either (0.7088). RePT, in contrast, raises RMM by 0.0060, and Q1 gains the most of the four quartiles (+0.0081). As on TrajTok, the rollouts narrow without the bonus: Full and Hard, which have no bonus, fall below Base on the kinematic score, and only RePT raises it. The overall gain is larger than on TrajTok, which is likely because CAT-K is a weaker backbone with more room to improve.

These results show that the failure of existing post-training on hard scenes, and the effect of RePT, are not limited to TrajTok.

We also trained RePT in the same way on two other pools: the 12.5% and the 50% of training scenes with the lowest Base RMM. In all settings, we evaluate on the same quartiles as in the paper (defined by the RMM of Base).

PoolRMM ↑Kinematic ↑Interactive ↑Map-based ↑minADE ↓
Base0.78710.48900.81440.92251.264
Lowest 12.5%0.78720.48950.81600.92031.286
Lowest 25% (RePT)0.78820.48970.81710.92171.274
Lowest 50%0.78800.48930.81630.92231.279
100% (Full+Div)0.78750.48910.81570.92171.294
PoolQ1 (hardest)Q2Q3Q4 (easiest)
Base0.71620.77860.80540.8483
Lowest 12.5%0.71840.78030.80420.8461
Lowest 25% (RePT)0.71790.78100.80590.8481
Lowest 50%0.71650.78120.80610.8485
100% (Full+Div)0.71360.78010.80740.8488

All three pools that select hard scenes raise the hardest quartile (Q1) above Base (12.5%: +0.0022, 25%: +0.0017, 50%: +0.0003); only with the full pool does Q1 fall below Base (100%: −0.0026). The narrower the pool, the larger the gain on Q1, but also the larger the loss on the easiest quartile, Q4 (12.5%: −0.0022, 25%: −0.0002). Overall RMM is highest at 25% and 50%, where it is about the same (0.7882 and 0.7880).

So the effect of RePT does not depend on a particular pool size. As long as training focuses on hard scenes, those scenes improve; the pool size only sets the balance between the gain on Q1 and the loss on Q4. We added both experiments to Appendix A.7 of the revision (Table 7 for CAT-K, Table 8 for pool size) and summarized them in the last sentence of the "Placement versus pool size" paragraph in Sec. 4.5.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the three reviewers for their careful comments. As strengths, the reviewers noted the strong performance (EzVV, ZbZ3, SrpR), the link to RL post-training for language models, seen as the main contribution (ZbZ3), the analysis of how existing post-training fails (ZbZ3, SrpR), and the novelty of focusing on difficult scenes and of using trajectory diversity (SrpR).

We have uploaded a revised paper, with changes marked in blue. The new experiments support one main finding: among all methods we compared, RePT is the only one that improves RMM on the hardest quartile. This also holds with a different backbone and with different pool sizes.

Below, W# and Q# refer to each reviewer's weaknesses and questions, not to the quartiles Q1–Q4 used in the paper.

EzVV

  • Simpler diversity alternatives (W1, Q1): With the same pool and the same amount of training, the entropy, mean-distance and no-gate variants all score below Base on the hardest quartile. → Tables 2 and 3, Sec. 4.4, App. A.6
  • Size of the gain (W1): The gap in overall RMM is small, but RePT is the only post-training method that raises the hardest quartile, and on that quartile it lowers the collision rate by about 24% (25.16% → 19.03%). → Tables 3 and 4
  • Backbone and pool size (Q2): The same result holds with the CAT-K backbone and with pool sizes from 12.5% to 50%. → App. A.7
  • Fair baselines (Q3): The Table 1 numbers are the official leaderboard entries. For the comparison on TrajTok, we reimplemented SMART-R1 and DecompGAIL, reproduced the results reported in their papers, and then retuned their hyperparameters within the original search ranges.
  • Clarity (W2): We fixed notation errors and reorganized the Introduction and Experiments sections.

ZbZ3

  • Diversity and metrics beyond RMM (W1): On the hardest quartile, RePT is the only model that substantially lowers the collision rate (25.16% → 19.03%). The rollouts of Hard, which has no diversity bonus, become narrower, while RePT keeps its diversity. → Table 4, App. A.8
  • Writing (W2): We rewrote the Introduction, Related Work and Experiments sections.
  • VRUs (Q1): RePT has the highest RMM for both pedestrians and cyclists, and we added qualitative results. → App. A.9
  • minADE (Q2): With the hardest-quartile pool, the bonus lowers minADE on all four quartiles (RePT vs. Hard). Most of RePT's increase over Base comes from easy scenes, and on the hardest quartile its minADE does not increase. → App. A.9

SrpR

  • Code (W1): We will release the code.
  • Gradient analysis (W2): When we re-measure using the contribution to the actual update, the skew toward easy scenes is even larger. → App. A.10
  • Compute (W3): The 32 rollouts are used only for the one-time pool selection and for evaluation. Training costs about 6% more than Full. → App. A.10
  • PPO (Q1): PPO also lowers the hardest quartile, so the problem is not specific to group-based advantages. → App. A.11
  • Rolling pool (Q2): In our setting, the fixed pool worked better: re-selecting the pool every 10k steps did not improve the hardest quartile. → App. A.10
  • Diversity (Q3): RePT keeps its diversity both during and after training. → Table 4, App. A.8
AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewers,

Thank you again for your time and thoughtful feedback on our submission.

As the discussion period is drawing to a close, we would like to gently follow up on our responses and the revised paper. We would be grateful to know whether they have addressed your concerns, and we would be happy to clarify any remaining questions.

If your concerns have been resolved, we would sincerely appreciate it if you could kindly take this into consideration in your final assessment.

AuthorsOpenReview ↗

Closing Thanks to All Reviewers

Dear Reviewers,

Thank you for your careful reviews and for the discussion throughout this period. We believe the paper is now much better than it was before the rebuttal, and we owe this to your comments and questions. It has been a truly enjoyable time for us.

If the paper is accepted, we will keep improving it until the camera-ready version.

Once again, we sincerely thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.