- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes outcome based reward shaping for autonomous driving. It trains a state evaluator, ODSE, using terminal success or failure labels from reference policy trajectories. The frozen evaluator supplies potential differences that augment existing environment rewards, while retaining a terminal term in the learning objective. Experiments on the CARLA benchmark used by VLM RL compare reward formulations, reference policies and learning algorithms, with additional studies across towns.
Critical strengths
- The ablations support using ODSE differences together with local rewards. SAC route success increases from 68.5% to 78.4%, and average distance per episode rises from 877 to 1370 metres. Direct ODSE addition and sparse outcome shaping perform worse, providing useful evidence that the way outcome information enters the reward matters.
- The paper examines applicability across local rewards, reference policies and learning algorithms. With ChatScene rewards, route success improves from 56.7% to 69.1%. A baseline reusing the same reference transitions and training comparisons in additional towns provide useful evidence beyond the main configuration.
Critical weaknesses
- Methodological novelty is limited, and the central motivation is not directly validated. ODSE mainly combines Monte Carlo regression of terminal outcomes with established potential based reward shaping. The paper does not clearly establish a methodological advance beyond this combination. It also lacks an independent evaluation of ODSE as an outcome predictor or a comparison with semantic reward scores. Downstream gains therefore do not establish that ODSE closes the claimed gap between semantic rewards and actual outcomes.
- Evidence for superior final performance and safety remains incomplete. The main comparison averages statistics over the final 10% of training, while the final policy evaluation across towns reports only the proposed method. Compared with VLM RL, route success increases by four percentage points, but episode collision rate rises from 79% to 84% and collision speed from 2.34 to 4.59 km/h. Longer driving distances complicate this comparison, making matched route tests important. The method also depends on informative local rewards, as sparse rewards plus ODSE still lead to stopping.
Questions for authors
- How accurately does ODSE predict and rank terminal outcomes on unseen trajectories compared with semantic reward scores? Does this relationship remain reliable on states visited by the learning policy, including terminal states?
- Can you compare final policies on identical test routes and traffic conditions, including unseen towns? Please also account for reference policy training, reference data collection and ODSE training when assessing sample efficiency and total training cost.
- Can you test zero terminal potential and a control that retains the learned terminal values but replaces intermediate potentials with a constant? These comparisons would help distinguish the contribution of intermediate state information from the additional terminal objective.
7 comments
Rebuttal Part 2
W2 & Q2. Evidence for superior final performance and safety remains incomplete.
We report the final 10% window for accuracy. Averaging over the window avoids the noise of a single checkpoint and of the different training routes. We also have the last-step table, and at the last step alone our advantage is even larger:
| Model | AS (km/h) | RC | TD (m) | CR | CS (km/h) | SR |
|---|---|---|---|---|---|---|
| SAC | 18.91±0.21 | 2.42±0.21 | 889.01±78.83 | 0.93±0.03 | 3.87±1.23 | 0.64±0.04 |
| PPO | 15.02±0.61 | 0.20±0.06 | 20.59±3.12 | 0.12±0.07 | 10.97±0.02 | 0.00±0.00 |
| MEM-SAC | 17.28±0.80 | 3.18±0.13 | 1321.06±75.92 | 0.83±0.02 | 4.29±0.07 | 0.80±0.01 |
| MEM-PPO | 18.42±0.13 | 1.45±0.15 | 483.76±56.96 | 0.98±0.00 | 3.90±0.76 | 0.49±0.02 |
| ChatScene | 16.61±0.05 | 2.17±0.05 | 751.39±35.09 | 0.71±0.16 | 8.68±0.34 | 0.59±0.01 |
| REvolve | 17.98±0.08 | 1.13±0.09 | 328.64±18.95 | 0.68±0.07 | 8.46±1.00 | 0.41±0.05 |
| VLM-RL | 18.86±0.08 | 2.91±0.27 | 1098.78±153.38 | 0.80±0.05 | 2.43±0.01 | 0.73±0.01 |
| LORD | 18.07±1.92 | 2.30±0.23 | 825.52±83.23 | 0.92±0.02 | 9.62±2.94 | 0.67±0.01 |
At the last step MEM-SAC reaches an SR of 0.80 against VLM-RL's 0.73, so the margin over the strongest baseline grows from 0.04 under the windowed protocol to 0.07. Accoridngly, the difference in CR also changes from 0.5 to 0.3. But we don’t think this is the appropriate way to compare the baselines, which may become cherry-picky.
For the evaluation on identical test routes, we evaluated the final checkpoints of the Town 2-trained agents on 100 held-out routes in an unseen town (Town 1). The result is shown below:
| Model | AS (km/h) | RC | CR | CS (km/h) | SR |
|---|---|---|---|---|---|
| MEM-SAC | 20.56±3.75 | 0.99±0.00 | 0.03±0.01 | 1.14±0.66 | 0.97±0.01 |
| VLM-RL | 22.42±2.33 | 0.88±0.07 | 0.12±0.04 | 1.76±0.34 | 0.88±0.04 |
| ChatScene | 17.48±1.66 | 0.83±0.02 | 0.30±0.02 | 9.32±0.72 | 0.70±0.02 |
| REvolve | 18.73±1.70 | 0.70±0.08 | 0.42±0.07 | 9.94±1.22 | 0.58±0.07 |
| LORD | 19.44±2.01 | 0.74±0.03 | 0.49±0.02 | 11.21±0.86 | 0.51±0.02 |
We can easily see that on the same routes, our method completes more and collides less than every baseline. In addition, for the problem about the higher CR comes with a higher SR of MEM-SAC, since a collision ends an episode in this benchmark, each episode contains at most one collision, and collisions per kilometer follow from the Table 6 (the main table in revised version) as CR divided by TD, giving MEM-SAC 0.61 against VLM-RL 0.67, SAC 1.07, and the other baselines above 1. On the other hand, our design aims at route completion, the terminal score our method injects is or for completing the route and contains no direct safety-related information. Compared with vanilla SAC our collision rate does decrease, which we consider as a side effect of the improved completion, since the two metrics are related, rather than a designed safety property.
Then, about the dependence on informative local rewards, we don’t consider this a weakness of the method. Expert reward design is always the first step of creating a reward, and our method fixes a specific problem of such dense rewards, that they contain little direct information about the task outcome. Making the local reward sparse removes the dense signal our method is designed to augment, and we report this behavior as a finding rather than hiding it.
Lastly, about sample efficiency. Each ODSE memory is trained once and shared by all runs that use it, and all methods are trained under the same learner budget. Counting the memory training step for one training is not fair under this setting, and every compared method also carries its own reward-construction cost outside this budget, for example, VLM-RL and LORD need to build the VLM-as-RM, REvolve needs to get the reward function from their evolutionary search. Therefore, we think the learner budget comparison is the fairest way to compare with the baselines.
Rebuttal Part 3
Q3. Can you test zero terminal potential and a control that retains the learned terminal values but replaces intermediate potentials with a constant?
Both controls are in the new Section 4.3.2 of the revision. The zero-terminal variant (ZT-SAC) cancels the outcome term and strictly preserves the original optimum, its collision rate drops (0.94 to 0.70) but its success rate stays at vanilla level (0.699 against 0.685), so the completion improvement requires the outcome term in the objective. The constant-intermediate control is the new TO-SAC, a zero intermediate potential contributes nothing except at the terminal, and it does not improve (0.656), the changed objective is not effective without the dense guidance, which is similar to the result of directly adding sparse terminal reward to the objective. The two parts therefore work at different levels and both are necessary.
We hope these clarifications and the additional experiment address the reviewer's concerns, and we are happy to discuss further during the discussion period.
Rebuttal Part 1
We thank the reviewer for the review. The revision adds the two direct validations the reviewer asks for, an independent evaluation of ODSE as an outcome predictor with a comparison against the local semantic reward, and the decomposition of the terminal and intermediate contributions. We respond point by point.
W1 & Q1. Methodological novelty is limited, and the central motivation is not directly validated.
About the novelty, we admit that the components are standard by design, and our novelty is mainly based on the construction and what it enables. The design move is choosing the setting that makes the value function collapse into a supervised label, non-discounted with a terminal-only score, so the return from every state equals the episode outcome and value estimation becomes plain regression with one label per state. The PBRS use is also a deliberate departure from its standard form. We keep the terminal potential that standard PBRS would set to zero, so the learning objective gains exactly the ϵ-weighted outcome-related term, while the potential form delivers it as dense step-wise signals (Appendix A.2), and the new ablation in Section 4.3.2 shows that both parts are necessary. The revision also adds the controls that isolate this contribution against the natural alternatives in Appendix A.7, the existing reusing the reference's data in the replay buffer of SAC and newly added reusing reference critic as the potential, neither can reproduce our gains.
About the evaluation of ODSE, our revision validates it directly in two ways. A new Appendix A.5 measures ODSE as an outcome predictor on held-out rollouts with (1) the probability that a state from a successful route receives a higher evaluation than a state from a failed route (AUC), (2) terminal predictions V~(sT) and (3) the probability of different states with different value estimation interval ending with success. Our memory ranks states from successful routes above states from failed ones (AUC 0.69 per state and 0.81 per route for the SAC reference, 0.91 and 1.00 for the PPO reference), and the reliability table shows the success probability rising with the predicted value throughout, from 3.6% and 0.1% in the lowest value group to 86.2% and 99.8% in the highest. We also compare ODSE with the semantic local reward, but since the semantic reward is based on different external semantic approximations which are not ruled in [-1, 1], we only evaluate the AUC of the semantic reward. The result shows that semantic scores are not built on the outcome scale, so their magnitudes carry little outcome meanings (0.53 and 0.59 as AUC) but ODSE reaches 0.69 0.81. So the claimed gap is now a measured number, semantic scores describe the current step, not where the route ends.
And for Q1, the relationship remains reliable in states visited by the learning policy. We did the same experiments of Appendix A.5 for the trained MEM-SAC learner (last checkpoint) and evaluated the SAC-ref memory on every visited state. The result is shown in the following table:
The result shows that the ranking quality holds on the learner's own states, and the terminal predictions still separate the two outcomes in the right direction. We keep this measurement in the rebuttal rather than the paper, since the learner changes throughout training and a single checkpoint cannot represent all of them, but we are glad to add it to Appendix A.5 if the reviewer considers it necessary.