acceptodds
Under review as a conference paper at ICLR 2027

Enhancing Reinforcement Learning for Autonomous Driving with Outcome-Related Reward Shaping

Abstract

Designing effective reward functions for autonomous driving remains a major challenge since high-level human driving concepts are difficult to translate into precise numerical signals. While current approaches attempt to address this using expert-designed rewards or semantic signals from language models, these approximations often fail to align with the actual physical outcomes of the driving task. To address this, we propose a reward shaping method utilizing empirical environment outcomes as the evaluation target. Our approach introduces Outcome-Derived State Evaluation (ODSE), a state evaluator learned from reference trajectories by using trajectory-level terminal outcomes as supervised targets for visited states. By integrating these evaluations through the potential-difference formulation of Potential-Based Reward Shaping (PBRS), we derive a dense shaping signal that improves sample efficiency while only adding a controlled, explicitly characterized outcome term to the learning objective. Evaluations on the VLM-RL benchmark show that our method outperforms state-of-the-art reward models, thereby demonstrating that outcome-related signals can successfully enhance training effectiveness and establish a more robust driving strategy across diverse scenarios.

Primary area:
reinforcement learning
Keywords:
Reward Engineering, Autonomous Driving, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AdC3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes outcome based reward shaping for autonomous driving. It trains a state evaluator, ODSE, using terminal success or failure labels from reference policy trajectories. The frozen evaluator supplies potential differences that augment existing environment rewards, while retaining a terminal term in the learning objective. Experiments on the CARLA benchmark used by VLM RL compare reward formulations, reference policies and learning algorithms, with additional studies across towns.

Critical strengths
  1. The ablations support using ODSE differences together with local rewards. SAC route success increases from 68.5% to 78.4%, and average distance per episode rises from 877 to 1370 metres. Direct ODSE addition and sparse outcome shaping perform worse, providing useful evidence that the way outcome information enters the reward matters.
  2. The paper examines applicability across local rewards, reference policies and learning algorithms. With ChatScene rewards, route success improves from 56.7% to 69.1%. A baseline reusing the same reference transitions and training comparisons in additional towns provide useful evidence beyond the main configuration.
Critical weaknesses
  1. Methodological novelty is limited, and the central motivation is not directly validated. ODSE mainly combines Monte Carlo regression of terminal outcomes with established potential based reward shaping. The paper does not clearly establish a methodological advance beyond this combination. It also lacks an independent evaluation of ODSE as an outcome predictor or a comparison with semantic reward scores. Downstream gains therefore do not establish that ODSE closes the claimed gap between semantic rewards and actual outcomes.
  2. Evidence for superior final performance and safety remains incomplete. The main comparison averages statistics over the final 10% of training, while the final policy evaluation across towns reports only the proposed method. Compared with VLM RL, route success increases by four percentage points, but episode collision rate rises from 79% to 84% and collision speed from 2.34 to 4.59 km/h. Longer driving distances complicate this comparison, making matched route tests important. The method also depends on informative local rewards, as sparse rewards plus ODSE still lead to stopping.
Questions for authors
  1. How accurately does ODSE predict and rank terminal outcomes on unseen trajectories compared with semantic reward scores? Does this relationship remain reliable on states visited by the learning policy, including terminal states?
  2. Can you compare final policies on identical test routes and traffic conditions, including unseen towns? Please also account for reference policy training, reference data collection and ODSE training when assessing sample efficiency and total training cost.
  3. Can you test zero terminal potential and a control that retains the learned terminal values but replaces intermediate potentials with a constant? These comparisons would help distinguish the contribution of intermediate state information from the additional terminal objective.
7 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We thank the reviewer for the review. The revision adds the two direct validations the reviewer asks for, an independent evaluation of ODSE as an outcome predictor with a comparison against the local semantic reward, and the decomposition of the terminal and intermediate contributions. We respond point by point.


W1 & Q1. Methodological novelty is limited, and the central motivation is not directly validated.

About the novelty, we admit that the components are standard by design, and our novelty is mainly based on the construction and what it enables. The design move is choosing the setting that makes the value function collapse into a supervised label, non-discounted with a terminal-only score, so the return from every state equals the episode outcome and value estimation becomes plain regression with one label per state. The PBRS use is also a deliberate departure from its standard form. We keep the terminal potential that standard PBRS would set to zero, so the learning objective gains exactly the ϵ\epsilon-weighted outcome-related term, while the potential form delivers it as dense step-wise signals (Appendix A.2), and the new ablation in Section 4.3.2 shows that both parts are necessary. The revision also adds the controls that isolate this contribution against the natural alternatives in Appendix A.7, the existing reusing the reference's data in the replay buffer of SAC and newly added reusing reference critic as the potential, neither can reproduce our gains.

About the evaluation of ODSE, our revision validates it directly in two ways. A new Appendix A.5 measures ODSE as an outcome predictor on held-out rollouts with (1) the probability that a state from a successful route receives a higher evaluation than a state from a failed route (AUC), (2) terminal predictions V~(sT)\tilde{V}(s_T) and (3) the probability of different states with different value estimation interval ending with success. Our memory ranks states from successful routes above states from failed ones (AUC 0.69 per state and 0.81 per route for the SAC reference, 0.91 and 1.00 for the PPO reference), and the reliability table shows the success probability rising with the predicted value throughout, from 3.6% and 0.1% in the lowest value group to 86.2% and 99.8% in the highest. We also compare ODSE with the semantic local reward, but since the semantic reward is based on different external semantic approximations which are not ruled in [-1, 1], we only evaluate the AUC of the semantic reward. The result shows that semantic scores are not built on the outcome scale, so their magnitudes carry little outcome meanings (0.53 and 0.59 as AUC) but ODSE reaches 0.69 0.81. So the claimed gap is now a measured number, semantic scores describe the current step, not where the route ends.

And for Q1, the relationship remains reliable in states visited by the learning policy. We did the same experiments of Appendix A.5 for the trained MEM-SAC learner (last checkpoint) and evaluated the SAC-ref memory on every visited state. The result is shown in the following table:

Routes (+/−)States (+/−)AUC (state)AUC (route)V(sT) on +V(sT) on -
Learner88/1252042/62220.630.860.62±0.19-0.31±0.27

The result shows that the ranking quality holds on the learner's own states, and the terminal predictions still separate the two outcomes in the right direction. We keep this measurement in the rebuttal rather than the paper, since the learner changes throughout training and a single checkpoint cannot represent all of them, but we are glad to add it to Appendix A.5 if the reviewer considers it necessary.

AuthorsOpenReview ↗

Rebuttal Part 2

W2 & Q2. Evidence for superior final performance and safety remains incomplete.

We report the final 10% window for accuracy. Averaging over the window avoids the noise of a single checkpoint and of the different training routes. We also have the last-step table, and at the last step alone our advantage is even larger:

ModelAS (km/h) ↑\uparrowRC ↑\uparrowTD (m) ↑\uparrowCR ↓\downarrowCS (km/h) ↓\downarrowSR ↑\uparrow
SAC18.91±0.212.42±0.21889.01±78.830.93±0.033.87±1.230.64±0.04
PPO15.02±0.610.20±0.0620.59±3.120.12±0.0710.97±0.020.00±0.00
MEM-SAC17.28±0.803.18±0.131321.06±75.920.83±0.024.29±0.070.80±0.01
MEM-PPO18.42±0.131.45±0.15483.76±56.960.98±0.003.90±0.760.49±0.02
ChatScene16.61±0.052.17±0.05751.39±35.090.71±0.168.68±0.340.59±0.01
REvolve17.98±0.081.13±0.09328.64±18.950.68±0.078.46±1.000.41±0.05
VLM-RL18.86±0.082.91±0.271098.78±153.380.80±0.052.43±0.010.73±0.01
LORD18.07±1.922.30±0.23825.52±83.230.92±0.029.62±2.940.67±0.01

At the last step MEM-SAC reaches an SR of 0.80 against VLM-RL's 0.73, so the margin over the strongest baseline grows from 0.04 under the windowed protocol to 0.07. Accoridngly, the difference in CR also changes from 0.5 to 0.3. But we don’t think this is the appropriate way to compare the baselines, which may become cherry-picky.

For the evaluation on identical test routes, we evaluated the final checkpoints of the Town 2-trained agents on 100 held-out routes in an unseen town (Town 1). The result is shown below:

ModelAS (km/h) ↑\uparrowRC ↑\uparrowCR ↓\downarrowCS (km/h) ↓\downarrowSR ↑\uparrow
MEM-SAC20.56±3.750.99±0.000.03±0.011.14±0.660.97±0.01
VLM-RL22.42±2.330.88±0.070.12±0.041.76±0.340.88±0.04
ChatScene17.48±1.660.83±0.020.30±0.029.32±0.720.70±0.02
REvolve18.73±1.700.70±0.080.42±0.079.94±1.220.58±0.07
LORD19.44±2.010.74±0.030.49±0.0211.21±0.860.51±0.02

We can easily see that on the same routes, our method completes more and collides less than every baseline. In addition, for the problem about the higher CR comes with a higher SR of MEM-SAC, since a collision ends an episode in this benchmark, each episode contains at most one collision, and collisions per kilometer follow from the Table 6 (the main table in revised version) as CR divided by TD, giving MEM-SAC 0.61 against VLM-RL 0.67, SAC 1.07, and the other baselines above 1. On the other hand, our design aims at route completion, the terminal score our method injects is +1+1 or −1-1 for completing the route and contains no direct safety-related information. Compared with vanilla SAC our collision rate does decrease, which we consider as a side effect of the improved completion, since the two metrics are related, rather than a designed safety property.

Then, about the dependence on informative local rewards, we don’t consider this a weakness of the method. Expert reward design is always the first step of creating a reward, and our method fixes a specific problem of such dense rewards, that they contain little direct information about the task outcome. Making the local reward sparse removes the dense signal our method is designed to augment, and we report this behavior as a finding rather than hiding it.

Lastly, about sample efficiency. Each ODSE memory is trained once and shared by all runs that use it, and all methods are trained under the same learner budget. Counting the memory training step for one training is not fair under this setting, and every compared method also carries its own reward-construction cost outside this budget, for example, VLM-RL and LORD need to build the VLM-as-RM, REvolve needs to get the reward function from their evolutionary search. Therefore, we think the learner budget comparison is the fairest way to compare with the baselines.

AuthorsOpenReview ↗

Rebuttal Part 3

Q3. Can you test zero terminal potential and a control that retains the learned terminal values but replaces intermediate potentials with a constant?

Both controls are in the new Section 4.3.2 of the revision. The zero-terminal variant (ZT-SAC) cancels the outcome term and strictly preserves the original optimum, its collision rate drops (0.94 to 0.70) but its success rate stays at vanilla level (0.699 against 0.685), so the completion improvement requires the outcome term in the objective. The constant-intermediate control is the new TO-SAC, a zero intermediate potential contributes nothing except at the terminal, and it does not improve (0.656), the changed objective is not effective without the dense guidance, which is similar to the result of directly adding sparse terminal reward to the objective. The two parts therefore work at different levels and both are necessary.


We hope these clarifications and the additional experiment address the reviewer's concerns, and we are happy to discuss further during the discussion period.

Reviewer_sTwrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Outcome-Derived State Evaluation (ODSE), a learned state evaluator trained by assigning each state in a reference trajectory its trajectory's terminal success/failure score. A Transformer over state features and an MLP predict this score. The evaluator is then frozen and used to augment a driving agent's local reward through the potential difference. Because the terminal potential is retained, the shaped return includes an additional discounted terminal-outcome term; the paper explicitly analyzes this objective change rather than asserting policy invariance.

Experiments use the CARLA-based VLM-RL benchmark with PPO and SAC. MEM-SAC increases route success from 68.5% to 78.4% and distance per episode from about 877 m to 1,370 m relative to SAC. Comparisons cover expert-designed and model-generated rewards, alternative shaping formulations, different reference policies, cross-town experiments, and direct reuse of reference transitions.

Critical strengths
  1. The method addresses an important credit-assignment problem with an accessible supervision signal. Learning from realized outcomes provides a concrete alternative to defining every aspect of driving quality through semantic descriptions or manually designed dense rewards.

  2. The distinction between the state evaluator and how it enters the reward is supported by useful ablations. Section 4.3.1 compares direct ODSE addition, potential differences, and sparse-outcome variants, helping establish that the combination matters.

  3. Evaluation covers both on-policy and off-policy learners and multiple reference policies. The three-seed reporting, coefficient analysis, additional training towns, and cross-town evaluation provide more evidence than a single training curve would (Section 4; Appendices A.3–A.4).

  4. The direct reference-replay baseline is valuable. Appendix A.6 uses the same reference data and shows that replay and ODSE offer different completion/safety trade-offs. This is an appropriate control for whether the reference experience itself explains the gains.

Critical weaknesses
  1. The results support improved completion more clearly than improved safety. In Table 3, MEM-SAC has higher collision rate and collision speed than VLM-RL: 84% versus 79%, and 4.59 versus 2.34 km/h. MEM-PPO reaches a 99% episode collision rate. Appendix A.6 also shows SAC with reference replay reaching 60% collision rate, compared with 84% for MEM-SAC. Episode collision rates are affected by distance and alternative termination conditions, as the authors acknowledge, but that makes exposure-normalized evaluation essential. The current metrics do not establish broad safety superiority.

  2. The sample-efficiency comparison needs complete data and compute accounting. Each ODSE model benefits from a pretrained reference policy and 500,000 reference interaction steps before the learner's one-million-step run. Appendix A.3 reports memory-training durations, including approximately ten hours for the selected setting, but the full reference-training, collection, evaluator-training, and learner-training cost is not consolidated. The replay experiment partially controls for extra experience; equal-total-budget learning curves and a longer-trained vanilla learner would clarify the practical advantage.

  3. The learned evaluator's reliability is largely inferred from downstream performance. Section 3.3 correctly identifies ODSE as a reference-policy Monte Carlo outcome estimate. Such an estimate can reflect what the reference policy does after visiting a state, including its weaknesses. Changing reference policies is informative, but does not measure calibration, state coverage, or error on the learner's shifted distribution. The strong result obtained from a PPO reference with only 0.4% route success particularly warrants reporting class balance and the number of independent successful trajectories.

  4. The terminal-outcome interpretation requires additional evidence. Appendix A.2 replaces the learned terminal prediction with the actual terminal score using V~(sT)≈sc(τ)\widetilde V(s_T)\approx sc(\tau). That approximation is not established empirically. A nonzero terminal potential also introduces a duration-dependent factor γT\gamma^T, whose effect depends on the discount and stopping rules. The telescoping identity characterizes the objective change, but does not itself certify driving safety. Terminal prediction errors, potential magnitudes, and sensitivity to episode duration would make this analysis more convincing.

  5. The method's incremental contribution needs sharper isolation. The manuscript already describes ODSE as outcome regression combined with potential-based shaping. It does not show whether the Transformer memory is necessary relative to a small MLP or an explicit nearest-neighbor estimator trained on the same features and labels. Calling the network a learned nearest-neighbor aggregation process is an intuition rather than an established equivalence (Section 3.3).

  6. The strongest reported numbers and transfer results use different evaluation protocols. Table 3 averages metrics over the final 10% of training; Table 2 evaluates a frozen MEM-SAC policy on routes and changes RC to mean route-completion fraction. The latter includes only MEM-SAC, so the comparative generalization benefit remains unclear. Frozen-policy evaluation of all principal baselines on identical held-out routes would better support robustness claims.

Questions for authors
  1. What exactly defines a positive ODSE label: completion of an individual route or the 3,000 m episode? Section 4.1 refers to reference-route labels, while Section 3.3 and Algorithm 1 describe episode labels. Please reconcile these definitions and report success/failure counts for each reference dataset.

  2. Provide an equal-total-interaction comparison including reference-policy pretraining and collection. Extend the existing replay control with a replay-plus-ODSE variant to test whether their complementary benefits can be combined.

  3. Evaluate frozen policies on shared routes and traffic seeds, reporting collisions per kilometer, distance between interventions/failures, off-road and timeout rates, and collision severity alongside completion. Include comparative cross-town results and uncertainty across training seeds.

  4. Report held-out and learner-distribution evaluator error, calibration, and terminal-score accuracy. Specify the state-vector components, whether they include route progress or termination-relevant information, and how states outside reference coverage are handled.

  5. Compare the current terminal convention with zero terminal potential and an explicit outcome-bonus baseline using a matched discounted-return definition. Report the discount, terminal-versus-truncation handling, and the observed contribution of the terminal term.

  6. Add a small-MLP evaluator control and present the contribution as outcome-supervised reward shaping, with claims centered on the demonstrated completion gains and measured safety trade-offs.

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We thank the reviewer for their efforts on reviewing our paper. The revision adds an independent evaluation of ODSE as an outcome predictor, and the decomposition of the terminal and intermediate contributions. We respond to weakness and question points below.


W1. The results support improved completion more clearly than improved safety.

We agree with this point, but we don't consider this as a weakness of our paper. The improvement of completion is the expected behavior of the method rather than an unintended gap. The terminal score our method injects is +1+1 or −1-1 for route completion, and it contains no direct safety-related information, so the design targets completion, and improved completion is exactly what the results should and do support. Although compared with vanilla SAC, MEM-SAC’s collision rate decreases, but that’s more like a side effect from the increase of route completion since these two metrics are related to each other, but not what we specially designed. Therefore, we claim no safety superiority in our paper.

For the exposure-normalized evaluation, in this benchmark a collision ends the episode, so each episode contains at most one collision and the exposure-normalized rate can be directly computed from the printed Table 6 (the main experiment table) as CR divided by TD. Collision episodes per kilometer driven are MEM-SAC 0.61, VLM-RL 0.67, SAC 1.07, ChatScene 1.19, LORD 1.20, MEM-PPO 1.83, REvolve 2.13, PPO 4.06. Under the normalization the reviewer asks for, MEM-SAC is the lowest in the table, and our shaping lowers the rate of each of its base learners (SAC 1.07 to 0.61, PPO 4.06 to 1.83). But we prefer to state this as an observation, not a safety claim.


W2 & Q2. sample-efficiency problem

We didn’t add the memory training steps into comparison to ensure the fairness. Each ODSE memory is trained once and shared by all runs that use it, and all methods are trained under the same learner budget, which Section 4.1 of the revision now states. Counting the memory training step for one training is not fair under this setting, and every compared method also carries its own reward-construction cost outside this budget, for example, VLM-RL and LORD need to build the VLM-as-RM, REvolve needs to get the reward function from their evolutionary search. Therefore, we think the learner budget comparison is the fairest way to compare with the baselines.

On combining replay with ODSE shaping from Appendix A.7 (the replay buffer section now becomes A.7 in the revised version), we agree it is an interesting direction, and A.7's observation that the two mechanisms capture different aspects (replay cuts collisions, shaping drives completion) suggests they may be complementary. However, the replay (the trajectory reuse now) is used to understand whether the trajectory contains sufficient information to support learning something useful, and we think A.7 is already doing its job. On the other hand, a tuned combination is a new method design beyond what this paper claims, and no matter how the combination performs, it won’t change the performance conclusion stated in our paper. Therefore, this is more like a future work instead of one experiment showing whether and why our method is effective.

AuthorsOpenReview ↗

Rebuttal Part 2

W3 & Q4. ODSE reliability, calibration

We acknowledge that the reliability of our ODSE memory is not explored and we think this is an important experiment that should be done for better analysis of our method. The revision now adds a direct evaluation of the ODSE memory in the newly added Appendix A.5, the accuracy of ODSE memory. On 100 held-out routes per reference, we measure (1) the probability that a state from a successful route receives a higher evaluation than a state from a failed route (AUC), (2) terminal predictions V~(sT)\tilde{V}(s_T) and (3) the probability of different states with different value estimation interval ending with success. Our memory ranks states from successful routes above states from failed ones (AUC 0.69 per state and 0.81 per route for the SAC reference, 0.91 and 1.00 for the PPO reference), and the reliability table shows the success probability rising with the predicted value throughout, from 3.6% and 0.1% in the lowest value group to 86.2% and 99.8% in the highest. These results show that ODSE memory can successfully learn the terminal outcome.

On the learner's shifted distribution, we did the same experiments of Appendix A.5 for the trained MEM-SAC learner (last checkpoint) and evaluated the SAC-ref memory on every visited state. The result is shown in the following table:

Routes (+/−)States (+/−)AUC (state)AUC (route)V(sT) on +V(sT) on -
Learner88/1252042/62220.630.860.62±0.19-0.31±0.27

The result shows that the ranking quality holds on the learner's own states, and the terminal predictions still separate the two outcomes in the right direction. We keep this measurement in the rebuttal rather than the paper, since the learner changes throughout training and a single checkpoint cannot represent all of them, but we are glad to add it to Appendix A.5 if the reviewer considers it necessary.

On the PPO reference's class balance, the labels are route-level (we state it in Section 4.1). Under the evaluation routes, the PPO reference completes nearly one route in ten (8 successful routes of 100 in A.5, against the 0.4% training-time rate), so about a quarter of its collected states carry positive labels, and its many failures teach the memory what failing driving looks like. This is why a highly suboptimal reference still provides abundant supervision. We also don’t think the SR difference will affect our conclusion about PPO, since 8% SR on the evaluation route is still highly suboptimal, and we have explicitly stated this in Appendix A.5 in our revision.

About the state vector components, the state is defined by the benchmark environment (carla-gym), not by our method. The memory therefore sees exactly the information every learner sees, no component is added or removed by our method, and all compared methods share the identical state definition.


W4 & Q5. The terminal-outcome interpretation requires additional evidence

We think the reviewer is right that the approximation was not established, and the revision has resolved this. As we stated in W3 & Q4, the new Appendix A.5 measures the terminal predictions. The experiments separate successful from failed endings in the right direction (0.72 and -0.17 for SAC, 0.27 and -0.54 for PPO), so "approximately equal" was too strong. Therefore, we removed the approximation words from Appendix A.2. However, we don’t think this will change our theory, since it was only used to interpret the terminal residual term and no theory depends on it. Equation 8 is an identity in V~\tilde{V}, and the revised derivation runs identity to identity with the terminal evaluation itself. The precise statement is that the objective gains an ϵ\epsilon-weighted predicted outcome, and its accuracy is now measured rather than assumed.

On the duration dependence, the terminal term enters the return from step tt as γT−tV~(sT)\gamma^{T-t}\tilde{V}(s_T). The γ\gamma in the shaping is the learner's own discount factor, which the theoretical analysis requires. The telescoping in Equation 4 and the invariance argument hold only under the learner's discount, which is the default value as 0.98. We also did ablation experiments on the terminal conventions and show the results in the new Section 4.3.2. The zero-terminal variant (ZT-SAC) removes the term and loses the completion gain (0.699 against 0.784 for the full method), and the terminal-only variant (TO-SAC) carries exactly this term and no step-wise signal, with an objective that differs from our method's only by a policy-independent constant, and it does not improve training (SR 0.656 against 0.685 vanilla).

AuthorsOpenReview ↗

Rebuttal Part 3

W5 & Q6. Transformer memory

The memory architecture is also not our contribution. Our insight is the construction, using the outcome as a supervised target under the non-discounted terminal-only setting where value estimation collapses into plain regression on state-label pairs, and this regression can be learned by any function approximator, in the limit even a lookup table implements Equation 1 exactly, only impractically for a high-dimensional state space. The Transformer is our practical choice for the benchmark's high-dimensional states, and we state the reason we use it in Section 3.3. About the K nearest-neighbor sentence, we state that the network functions as a learned nearest-neighbor aggregation, which describes the function the regression performs, generalizing the average-outcome definition of Equation 1 across similar states, and does not claim an equivalence between our memory and K-NN. Since the claims do not rest on the architecture, we consider an architecture comparison outside the scope of this paper, and we note that the memory's sensitivity to its training-data scale is already analyzed in Appendix A.3.2.


W6 & Q3. The strongest reported numbers and transfer results use different evaluation protocols.

The two tables are used for different questions. The main comparison follows the benchmark's training-curve protocol, which shows sample efficiency and effectiveness together, windowed over the final 10% to avoid the noise of a single checkpoint and the different training routes. The cross-town evaluation tests the robustness of our final policy, and its RC definition necessarily changes with the evaluation, since a single-route evaluation cannot share the RC of the 3000 m multi-route training episodes, which the Table 2 caption states. To make the generalization comparison the reviewer asks for, we have evaluated the final checkpoints of the Town 2-trained agents on the same 100 held-out routes in Town 1, identical routes and traffic settings for every method.

ModelAS (km/h) ↑\uparrowRC ↑\uparrowCR ↓\downarrowCS (km/h) ↓\downarrowSR ↑\uparrow
MEM-SAC20.56±3.750.99±0.000.03±0.011.14±0.660.97±0.01
VLM-RL22.42±2.330.88±0.070.12±0.041.76±0.340.88±0.04
ChatScene17.48±1.660.83±0.020.30±0.029.32±0.720.70±0.02
REvolve18.73±1.700.70±0.080.42±0.079.94±1.220.58±0.07
LORD19.44±2.010.74±0.030.49±0.0211.21±0.860.51±0.02

The ordering of the training comparison is preserved and the margin over the strongest baseline widens off-town. Collisions per kilometer on the training-town data, derived from the printed Table 6, are in our W1 response.


Q1. What exactly defines a positive ODSE label

It's based on route-level. In the method section we use episode because episode is a general term for the concept, however, the vlm-rl benchmark uses a 3000m episode containing multiple routes, but each route is independent to others. Therefore, we use route as the ‘episode outcome score’ in Section 3. We have revised Section 4.1 (the experiment setup) to state this more clearly.


We hope these clarifications and the additional experiment address the reviewer's concerns, and we are happy to discuss further during the discussion period.

Dear Reviewer sTwr:

As the rebuttal period is coming to an end, we would like to kindly ask whether our responses address your concerns. We are also glad to provide any further clarification before the period closes.

Reviewer_VdmCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns an outcome evaluator from a reference agent's trajectories by labeling visited states with terminal success or failure. This evaluator supplies the potential for reward shaping in SAC and PPO. On the CARLA-based VLM-RL benchmark, the method reports improved success and distance over the vanilla learners and several reward-design baselines, with additional cross-town evaluations.

Critical strengths
  1. The approach is simple to integrate with existing learners and ties the shaping signal to observed task outcomes. Improvements with both SAC and PPO make the practical result more convincing than a single-learner demonstration.

  2. The ablations examine several useful alternatives: direct reward injection versus PBRS, sparse outcomes, reference replay, different reference policies, and shaping coefficients. In particular, the weak-reference experiment helps establish how much the method depends on reference-policy quality.

Critical weaknesses
  1. The contribution is insufficiently distinguished from learning shaping potentials from demonstrations or prior policies. ODSE is an estimate of a reference policy's sparse-outcome value, yet the comparisons focus on driving reward design. A reference-critic potential and a method that directly reuses the reference policy or data are important controls. Without these, it is difficult to tell whether the proposed memory offers a benefit beyond established ways of exploiting the same prior experience.

  2. The sample-efficiency and theoretical interpretations need stronger support. The comparison excludes reference-policy training, 5×10^5 reference collection steps and memory training from the learner's budget. In addition, the nonzero terminal potential leaves an outcome-dependent boundary term, so the shaping can change the episodic objective. Its behavioral effect and the evaluator's accuracy on learner-visited states are not established. The Monte Carlo argument also needs qualification for the bootstrapped SAC and GAE-based PPO implementations. These issues leave me marginally below acceptance.

Questions for authors
  1. At a matched total interaction budget, how does the method compare with continued SAC and with the reference SAC critic used directly as a potential? Please distinguish reference training, reference-data collection and learner interactions.

  2. How are terminal potentials handled? What evidence supports the assumed terminal-outcome prediction, and how accurate/calibrated is the evaluator on states visited by the learner? Please explain how the theoretical argument applies to the actual SAC/PPO updates.

  3. Were comparison methods re-run under the same protocol? Please clarify whether reported success and collision statistics are from training or fixed held-out routes, how termination is defined, and whether the Transformer takes a single state or a history.

6 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We thank the reviewer for the careful reading, in particular of Appendix A.2, and respond to each point below. The revision contains the new experiments and analysis mainly about the theoretical analysis about SAC and PPO, an independent evaluation of ODSE as an outcome predictor, and the decomposition of the terminal and intermediate contributions.


W1. The contribution is insufficiently distinguished from learning shaping potentials from demonstrations or prior policies.

Our key insight of ODSE is to use the outcome signal as a supervised target rather than a pure reward. As a reward the outcome appears only once per episode, but as a target it can be assigned to every state, so one episode yields one labeled state per step, from failed episodes as much as successful ones. ODSE is then the MC estimate of the reference policy's expected outcome under the non-discounted terminal-only setting, which we choose exactly because in it the return from every state equals the episode outcome and the value estimation collapses into this plain supervised regression. We have revised Section 3.3 and 3.4 to state this insight and construction more clearly.

The requested control of directly reusing the reference data already exists in Appendix A.7 in the revised version. The reference trajectories are added to the SAC learner's replay buffer, with both SAC and PPO reference. This lowers the collision rate but does not reach our method's task completion (SR 0.738 with the SAC buffer against 0.784 for MEM-SAC), and it isn’t applicable to the on-policy PPO learner, while our shaping improves both learners. We also add the new reference-critic experiments into the Appendix A.7 now (the CRI-SAC in Figure 11 of the revision). We extract a state value from the frozen reference SAC's critics, normalize it to replace ODSE as potential, and change nothing else. The critic-potential doesn’t improve over vanilla SAC (SR 0.641 against 0.685), and has the lowest route completion and total distance in the comparison, although it does lower the collision rate to 0.76. Since CRI-SAC and MEM-SAC differ only in the content of the potential, we think this control isolates what our memory adds beyond established ways of exploiting the same prior experience, and shows the effectiveness of our method.


W2,Q1. sample-efficiency needs stronger support

We didn’t add the memory training steps into comparison to ensure the fairness. Each ODSE memory is trained once and shared by all runs that use it, and all methods are trained under the same learner budget, which Section 4.1 of the revision now states. Counting the memory training step for one training is not fair under this setting, and every compared method also carries its own reward-construction cost outside this budget, for example, VLM-RL and LORD need to build the VLM-as-RM, REvolve needs to get the reward function from their evolutionary search. Therefore, we think the learner budget comparison is the fairest way to compare with the baselines.

AuthorsOpenReview ↗

Rebutall Part 2

W2. theory needs stronger support

We write our equations in the Monte Carlo policy-gradient form, but our actual learners are SAC and PPO, so we agree the argument needs a further qualification.

For SAC, its entropy term never touches the shaping rewards, but the critic target changes since rtr_t becomes rt′r'_t. The remaining question is the actor, whose loss is E [αlog⁡π(a∣st)−Q(st,a)]\mathbb{E}\,[\alpha \log \pi(a|s_t) - Q(s_t, a)] with a∼π(⋅∣st)a \sim \pi(\cdot|s_t), so what matters is how the learned Q changes. Our shaping is equivalent to the standard zero-terminal PBRS with ODSE as the potential , applied to the outcome-augmented problem whose final-transition reward includes the residue ϵ∗V~(sT)\epsilon*\tilde{V}(s_T) and whose objective is Equation 8. Let Q+Q^{+} denote this augmented problem’s Q function.By the invariance result of Ng et al. [1], the shaped Q function is Q′(st,a)=Q+(st,a)−ϵV~(st)Q'(s_t, a) = Q^{+}(s_t, a) - \epsilon \tilde{V}(s_t), and the shift carries over to the soft setting since a per-state constant passes through the log-sum-exp backup exactly as through the max. The actor loss therefore becomes E [αlog⁡π(a∣st)−Q+(st,a)+ϵV~(st)]\mathbb{E}\,[\alpha \log \pi(a|s_t) - Q^{+}(s_t, a) + \epsilon \tilde{V}(s_t)], and since ϵV~(st)\epsilon \tilde{V}(s_t) depends on neither the action nor the policy parameters, its gradient vanishes. The state part of the shaping is invisible to the actor, which is the SAC counterpart of the baseline cancellation in Equation 6, and the only surviving change is the outcome term of Equation 8.

For PPO with GAE, the shaping enters through the TD residuals that GAE sums. We have PBRS on the top of GAE as (Please refer to Equation 9 in Appendix A.2 if the equation is not displayed correctly)

δt′=rt+ϵ∗(γ∗V~(st+1)−V~(st))+γ∗V(st+1)−V(st)=rt+γ∗(V+ϵ∗V~)(st+1)−(V+ϵ∗V~)(st),\delta'_t = r_t + \epsilon*(\gamma*\tilde{V}(s_{t+1}) - \tilde{V}(s_t)) + \gamma*V(s_{t+1}) - V(s_t) = r_t + \gamma*(V + \epsilon*\tilde{V})(s_{t+1}) - (V + \epsilon*\tilde{V})(s_t),

so GAE on the shaped rewards with critic VV coincides with GAE on the original rewards with critic V+ϵV~V + \epsilon\tilde{V}, except for the terminal one, which again leaves the outcome term. Therefore, training PPO on the shaped rewards is equivalent to training it on the outcome-augmented problem with the critic warm-started at ODSE.

In conclusion, for both SAC and PPO the shaping changes what is optimized only by the residue, while the state part is absorbed into the critic. This residue is a similar modification of the objective to directly adding the sparse outcome reward, which is the most intuitive way to inject the task outcome. Our ablation shows that the direct sparse addition does not improve and can harm training, while our method improves it (Section 4.3.1), so the benefit requires the dense step-wise ODSE rather than the outcome term alone. We have added this analysis to Appendix A.2 in the revision.

[1] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278–287. Citeseer, 1999.

AuthorsOpenReview ↗

Rebuttal Part 3

W2,Q2. How are terminal potentials handled? What evidence supports the assumed terminal-outcome prediction, and how accurate/calibrated is the evaluator on states visited by the learner?

This is a highly valuable question, and we didn't make it clear in our paper. We intentionally leave the terminal potential not zero. Standard PBRS would set the terminal potential to zero to strictly preserve the policy optimality. We keep the terminal evaluation because the resulting term is exactly the outcome signal our method tries to inject. The new ablation study in Section 4.3.2 separates the step-wise shaping and the terminal residue term and shows the effectiveness by testing both parts in isolation. The zero-terminal variant cancels the term and keeps only the dense guidance, its success rate stays at vanilla level (0.699 against 0.685) while collisions drop. The terminal-only variant keeps the objective change but gives no step-wise signal, and it does not improve (0.656). Our full method reaches 0.784, so the terminal term injection to the objective can empirically improve the effectiveness.

For the evidence on the terminal prediction accuracy, the new Appendix A.5 measures it on held-out rollouts of the SAC and PPO reference agent with (1) the probability that a state from a successful route receives a higher evaluation than a state from a failed route (AUC), (2) terminal predictions V~(sT)\tilde{V}(s_T) and (3) the probability of different states with different value estimation interval ending with success. All three metrics show that ODSE memory can successfully learn the terminal outcome. This experiment also helps us to revise a mistake in our paper, which is the approximation that V~(sT)≈sc(τ)\tilde{V}(s_T) \approx sc(\tau) from the original Appendix A.2 (the theory section) since the terminal term V~(sT)\tilde{V}(s_T) is not close enough to +1 for success and -1 for failure. However, it was only used to interpret the V~(sT)\tilde{V}(s_T) and the correctness of our theory does not depend on it, and we have removed this statement from Appendix A.2. For the details of our new experiments, please refer to Section 4.3.2 and Appendix A.5.

On the learner's own states, we did the same experiments of Appendix A.5 for the trained MEM-SAC learner (last checkpoint) and evaluated the SAC-ref memory on every visited state. The result is shown in the following table:

Routes (+/−)States (+/−)AUC (state)AUC (route)V(sT) on +V(sT) on -
Learner88/1252042/62220.630.860.62±0.19-0.31±0.27

The result shows that the ranking quality holds on the learner's own states, and the terminal predictions still separate the two outcomes in the right direction. We keep this measurement in the rebuttal rather than the paper, since the learner changes throughout training and a single checkpoint cannot represent all of them, but we are glad to add it to Appendix A.5 if the reviewer considers it necessary.


Q3. Were comparison methods re-run under the same protocol?

Yes, all compared methods are trained under the identical setting described in Section 4.1, which is why the setup section exists, and their training curves are shown together in Figure 2. The reported statistics are training statistics, for each seed each metric is averaged over the final 10% of training, then mean and std across seeds, stated in each caption, this windowed training-curve protocol shows sample efficiency and effectiveness together and avoids the noise of a single final checkpoint. Termination is defined in Section 4.1 by its three conditions. The Transformer takes a single state, encodes the features of one state and the MLP predicts the terminal score from that representation, no history is used.


We hope these clarifications and the additional experiment address the reviewer's concerns, and we are happy to discuss further during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of the revision

We thank all reviewers and AC for their efforts on reviewing our paper. Reading the three reviews together, we find several shared concerns that we consider the most important ones. (1) Whether the ODSE memory really learns the task outcome, instead of being assumed to. (2) Whether the improvement comes from our construction or simply from reusing the reference experience, and whether the training budget comparison is fair. (3) How the non-zero terminal potential should be understood, and what the terminal outcome term actually contributes. We address these concerns with new experiments and a more clear presentation, and we have uploaded a revised paper with all changes marked in red. The main changes are:

  1. Section 3.3 states our insight more clearly. The outcome signal is used as a supervised target rather than a reward, and ODSE is formally the MC estimate of the reference policy's expected outcome under a non-discounted terminal-only setting, where the value estimation collapses into a supervised regression on state-label pairs.
  2. Section 3.4 and Appendix A.2 state explicitly that the terminal residue is injected on purpose. We keep the terminal evaluation because this outcome term is exactly the signal our method aims to inject.
  3. The experiment setup (Section 4.1) is more clear. The terminal score is the single route outcome rather than the episode outcome, since one episode contains multiple routes, and we state the reason why the comparison is made under the same learner budget.
  4. A newly added decomposition ablation (Section 4.3.2) separates the dense step-wise guidance and the terminal outcome term, and shows that both parts are necessary.
  5. Appendix A.2 adds the theoretical analysis for the actual SAC and PPO learners.
  6. A newly added Appendix A.5 measures the accuracy of the ODSE memory on held-out rollouts, and compares it with the local reward.
  7. Appendix A.7, the direct reuse of the reference, adds the reference-critic reuse control (CRI-SAC).

We think the revision makes our paper more clear about what we do, why we do it, and why it is working. No findings or conclusions are changed from our previous version, and point-by-point responses are posted under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.