acceptodds
Desk rejected at ICLR 2027

When Learned Rewards Mislead Robotic Reinforcement Learning: Diagnosing and Calibrating Rewards under Distribution Shift

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Dense rewards provide informative feedback for robotic reinforcement learning (RL), but their reliability can deteriorate once online policy optimization drives the state visitation beyond the reward model’s training distribution. In this work, we first diagnose this failure mode and reveal that online RL induces substantial out-of-distribution (OOD) state visitation, where reward prediction errors are markedly larger than on in-distribution states and can misguide policy optimization. Our key insight is to calibrate rewards according to their estimated reliability, preserving high-confidence predictions while suppressing unreliable ones. We therefore propose ***Confidence-Aware Reward Calibration (CRC)***, which estimates reward confidence from feature-space distances to the reward model’s training data and calibrates low-confidence predictions accordingly. In this way, CRC treats the reward model as a frozen black box and can be seamlessly integrated with different learning-based reward models and RL algorithms without retraining it. Across diverse reward models and RL algorithms, CRC consistently improves online policy learning in both simulation and real-world settings, demonstrating its effectiveness and generality under distribution shift. Code and videos are publicly available at: https://crc.k7m2.workers.dev/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reward model, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dtwjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper considers online RL with a pretrained reward model. The authors propose Confidence-Aware Reward Calibration (CRC) to identify and calibrate rewards on out-of-distribution (OOD) state visitation, without retraining the reward model, and it can be applied to different frozen reward models. Experiments across diverse reward models and RL algorithms on simulation and real-world tasks demonstrate the improvement gain from CRC.

Critical strengths
  • The problem is well-motivated with the diagnostic findings. Figure 1 shows the proportion of OOD states, the relationship between reward error and feature-space distance, and the performance increase with corrected rewards. They together identify unreliable reward predictions on OOD states as a key source of error in online policy optimisation.
  • The method is simple to integrate into existing RL pipelines. It only does the calibration without retraining the reward model or additional trajectory collection
  • Its performance in experiments across simulation and real-world tasks is encouraging.
Critical weaknesses
  • The authors argue that the neighbour-guided reward compensation is motivated by the fact that “States that are sparsely represented offline may receive low confidence even when they correspond to meaningful task progress during online interaction”. However, whether the proposed compensation mechanism actually mitigates this issue is unclear. For low-confidence data, the calibrated reward turns to the KNN-based compensation, but it is unclear whether this estimate is more reliable or simply smooths the existing predictions or reference values. Moreover, low standard deviation among neighbours does not necessarily imply high reliability if all neighbours are distant from the measured one. Although the ablation analysis shows the benefit of compensation, it does not imply that the improved performance is due to recovery of meaningful rewards.
  • The experimental comparisons are insufficient to fully validate the proposed design choices. The evaluation of the disagreement penalty effect, comparisons with other baselines considering reward uncertainty, and sensitivity analyses of some hyperparameters are missing.
Questions for authors
  • Could the authors clarify the relationship between the reference set BrefB_\mathrm{ref} and B__\mathrm{test}? Are they the same set? Also, could the authors clarify the size or proportion of offline samples held out for threshold calibration? Would this split influence the final performance?
  • What’s the actual effect of the neighbour-guided reward compensation? Could the authors clarify the causal effect of weighted nearest neighbours and a meaningful reward signal?
  • Additional experimental comparisons would better support the mechanism design. Could the authors also test CRC without the disagreement penalty to isolate its effect? The main experimental comparisons are against uncalibrated reward models, with additional ablations of two proposed components. Comparisons with existing reward uncertainty estimation or calibration methods, even if they involve additional data collection or computational costs, would help better assess the effectiveness of CRC. Moreover, there exist other hyperparameters whose sensitivities are not reported in the ablation studies, e.g., α\alpha, which controls the confidence decay rate, β\beta, which controls the strength of reward compensation, and λr\lambda_r, which controls the penalty for disagreement among neighbouring reference values.
8 comments
AuthorsOpenReview ↗

Response to Reviewer dtwj (Part 1/2)

Thank you for your thoughtful comments and constructive suggestions. We provide detailed responses to your concerns below.

1) Effectiveness and Reliability of Reward Compensation

Reward compensation aims to preserve meaningful reward signals that would otherwise be suppressed by reweighting in low-confidence states. To evaluate its effectiveness, we analyze online RL rollouts on the door-lock task. For states with ct<0.25c_t < 0.25, compensation further reduces the reward prediction MAE by 8.90% compared with reweighting alone, demonstrating improved reward accuracy in low-confidence states.

The disagreement penalty improves compensation reliability by penalizing variations in neighboring reward values, rather than feature-space distances. Even when all neighboring states are distant from the query state, differences in their reward values still incur a penalty that mitigates potential reward overestimation. As shown in the following table, reweighting improves the average success rate from 0.376 to 0.585, and compensation further increases it to 0.635. Removing the disagreement penalty reduces the success rate to 0.614, demonstrating the effectiveness of the compensation design.

ComponentsSuccess Rate
Baseline0.376
Reweighting0.585
Reweighting + Compensation (w/o disagreement penalty)0.614
Reweighting + Compensation0.635

Furthermore, the reward visualizations in Fig. 8 provide qualitative evidence from two real-world OOD cases. In both cases, compensation restores positive reward signals that align with task progress but are otherwise suppressed by reweighting, demonstrating its effectiveness in preserving meaningful reward information.

2) Comparison with More Baselines

We additionally compare CRC against two uncertainty-aware baselines, UWO [1] and RUNE [2]. The results are presented in the following tables. Across the eight Meta-World tasks, CRC improves the average success rates by 39.18%, 43.87%, and 21.83% over the original RM-rule, RM-pre, and RM-pro baselines, respectively, while outperforming both UWO and RUNE by substantial margins.

RM-rule

TaskOriginalUWO [1]RUNE [2]CRC
Door lock0.5440.7650.4070.788
Handle press-side0.5630.7520.5630.798
Door unlock0.2900.3210.5930.613
Sweep into0.1070.4200.4050.339
Faucet close0.6550.7690.7850.795
Hammer0.7970.6590.8150.848
Lever pull0.1420.3240.5000.312
Button press-wall0.7800.7050.8520.908
Average0.4850.5890.6150.675 (↑39.18%)

RM-pre

TaskOriginalUWO [1]RUNE [2]CRC
Door lock0.8740.6780.6380.910
Handle press-side0.8660.7480.9870.987
Door unlock0.4720.5100.2640.509
Sweep into0.0020.2070.0430.226
Faucet close0.2210.5690.3180.947
Hammer0.5220.4860.4060.837
Lever pull0.3550.2560.4680.471
Button press-wall0.7390.8910.7530.938
Average0.5060.5430.4850.728 (↑43.87%)

RM-pro

TaskOriginalUWO [1]RUNE [2]CRC
Door lock0.8870.9150.8890.939
Handle press-side0.6820.9670.7620.978
Door unlock0.5530.2640.3660.752
Sweep into0.2140.3270.3100.436
Faucet close0.9570.9720.9310.991
Hammer0.9940.9800.9970.997
Lever pull0.3800.3740.4750.577
Button press-wall0.7560.9580.7610.937
Average0.6780.7200.6860.826 (↑21.83%)

References

[1] Coste T, et al. Reward Model Ensembles Help Mitigate Overoptimization. ICLR, 2024.

[2] Liang X, et al. Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. ICLR, 2022.

AuthorsOpenReview ↗

Response to Reviewer dtwj (Part 2/2)

3) Clarification and Ablation Analysis of BrefB_\mathrm{ref} and BtestB_\mathrm{test}

BrefB_{\mathrm{ref}} and BtestB_{\mathrm{test}} are two disjoint subsets of trajectories randomly sampled from a pool of offline data BB, i.e., Bref∪Btest=BB_{\mathrm{ref}} \cup B_{\mathrm{test}}=B. More specifically, BrefB_{\mathrm{ref}} is used for OOD distance calculation and reward compensation, while BtestB_{\mathrm{test}} is used to determine the OOD threshold based on its distances to BrefB_{\mathrm{ref}}.

We investigate the impact of different bank split settings of BrefB_{\mathrm{ref}} and BtestB_{\mathrm{test}} across four Meta-World tasks (door-lock, handle-press-side, door-unlock, and sweep-into) in the following table. As observed, CRC consistently outperforms the original baseline across all evaluated bank split settings, with the ratio of 3:2 achieving the best performance. These results demonstrate the robustness of CRC against bank splits.

Bank Split

Bref:BtestB_{\mathrm{ref}}:B_{\mathrm{test}}Average
Baseline0.376
1:40.440
2:30.529
3:2 (Default)0.635
4:10.618

4) Additional Hyperparameter Ablations

Inspired by your comments, we further evaluate the sensitivity of CRC to the confidence decay rate α\alpha, compensation strength β\beta, and disagreement penalty coefficient λr\lambda_r across four Meta-World tasks (door-lock, handle-press-side, door-unlock, and sweep-into), varying each parameter independently. As shown in the following tables, CRC consistently outperforms the original baseline across all tested hyperparameter settings, demonstrating its robustness and effectiveness across different configurations.

Confidence Decay α\alpha

α\alphaSuccess Rate
Baseline0.376
0.60.549
1.2 (Default)0.635
1.80.614

Compensation Strength β\beta

β\betaSuccess Rate
Baseline0.376
0.20.550
0.3 (Default)0.635
0.40.604

Disagreement Penalty λr\lambda_r

λr\lambda_rSuccess Rate
Baseline0.376
0.80.610
1.0 (Default)0.635
1.20.625
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer dtwj,

We appreciate your insightful feedback. We have carefully considered your comments and provided a point-by-point response, and we hope it addresses your concerns.

We are very keen to clarify any remaining points and look forward to your feedback.

Best regards,

Authors

Reviewer_vr6cOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at a common problem with learned robot rewards: the policy changes during online RL, visits unfamiliar states, and gets unreliable scores from a frozen reward model. The authors measure this and use oracle reward replacement to check its effect on learning. CRC then uses feature-space distance to estimate confidence and mixes the model's prediction with a conservative nearest-neighbor reward estimate. It doesn't retrain the reward model or change the RL algorithm. The experiments cover three reward-model types, Meta-World, Robomimic, and three real-robot tasks.

Critical strengths
  1. Replacing rewards on selected states is a useful way to check whether those errors actually hurt learning.
  2. The method is simple enough to add to an existing setup.
  3. The authors are clear that this doesn't recover the true reward. They also explain where the reference values come from when the preference model has no pointwise labels.
Critical weaknesses
  1. I don't see several obvious baselines. What if we just use kNN rewards? What if we mix kNN and model predictions with a fixed weight? What if we remove the disagreement penalty, clip rewards, or add a simple OOD penalty? Without these comparisons, I can't tell how much the confidence rule adds versus simply having the reference bank.
  2. Feature distance isn't the same thing as reward reliability. An unfamiliar-looking state can still get the right reward, and a familiar-looking one can get the wrong reward. Neighbors can also agree on the same mistake, especially when their values come from the model itself. The four-task RM-rule diagnosis is useful, but I'd like to see whether the distance/error relationship also holds for the other reward models and shifts.
  3. What happens with negative rewards? Multiplying a negative prediction by low confidence makes it less negative, which reduces the penalty. That isn't automatically conservative. Also, when the compensation coefficient isn't one, adding the same offset to model and reference rewards adds a confidence-dependent offset after calibration. I'd like the reward ranges and normalization spelled out, plus a fixed-horizon test of sensitivity to these choices.
  4. The diagnostic comparison could use another control. OOD states make up 68.3% of the visited states, so correcting OOD rewards also corrects more rewards than the ID intervention. Matching the number of corrected states would help. The real-robot results also need clear counts of independent training runs and evaluation episodes, with uncertainty on the averages.
  5. I'd like a clearer picture of the practical requirements. This needs training data, hidden features, and neighbor lookup; it won't work as written with an API that only returns a reward score. How big is the reference bank, how much does retrieval cost, and how sensitive is it to the features used?
Questions for authors
  1. How do kNN-only rewards and fixed mixing perform? What extra gain comes from confidence-aware mixing?
  2. Does distance track reward error for RM-pre and RM-pro too? Can you show that directly? What are the output ranges and normalization rules? What happens with negative rewards or a common reward offset in fixed-length episodes?
  3. How many independent robot training runs and evaluation episodes are behind the reported averages? What is the uncertainty?
  4. Can you match the number of corrected states in the ID/OOD intervention?
  5. What if the bank misses an important successful behavior? Does CRC discourage exploring it? Also, how much time does the neighbor lookup add?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vr6c (Part 1/2)

We thank you for your valuable comments and suggestions. Please find our detailed responses below.

1) Comparison with Additional Baselines

Inspired by your comments, we compare the following baselines to further validate the effectiveness of CRC:

  • Baseline: Use the original reward model predictions without additional calibration.
  • kNN-only: Use rewards from the nearest neighbors in the reference bank without the model prediction term.
  • Fixed mixture: Average the model prediction ftf_t and the kNN reward gtg_t, yielding Rt=(ft+gt)/2R_t=(f_t+g_t)/2.
  • RM + OOD penalty: Subtract a normalized distance-based penalty from the model prediction, yielding Rt=ft−max⁡((dt−τ)/τ,0)R_t=f_t-\max((d_t-\tau)/\tau,0), where dtd_t is the distance to the nearest neighbors, and τ\tau is the task-specific OOD threshold.
  • UWO [1]: Subtract an ensemble-variance penalty from the mean reward prediction to discourage reward overoptimization.
  • RUNE [2]: Add a decaying exploration bonus based on ensemble uncertainty in transition rewards.
  • Reweighting only: Apply CRC's confidence-based reweighting to model predictions without kNN compensation or the disagreement penalty.
  • Reweighting + Compensation (w/o disagreement penalty): Combine confidence-weighted model predictions with confidence-dependent kNN compensation while omitting the disagreement penalty.
  • Full CRC: Combine confidence-based reweighting, kNN compensation, and a penalty for disagreement among neighboring reference rewards.

Since the original reward model employs a sigmoid output layer, all predicted rewards are inherently bounded within [0,1][0,1]. Therefore, clipping the predicted rewards to this range has no effect and is mathematically equivalent to the original baseline. The average success rates across four Meta-World tasks (door-lock, handle-press-side, door-unlock, and sweep-into) are summarized below.

MethodSuccess Rate
Baseline0.376
kNN-only0.447
Fixed mixture0.456
RM + OOD penalty0.464
UWO [1]0.565
RUNE [2]0.492
Reweighting only0.585
Reweighting+Compensation (w/o disagreement penalty)0.614
Reweighting+Compensation (Full CRC)0.635

The results show that CRC substantially outperforms the three suggested baselines (i.e., kNN-only, fixed-weight mixing and RM+OOD penalty), as well as the two uncertainty-aware baselines UWO [1] and RUNE [2]. We also note that confidence-based reweighting alone improves the success rate from 0.376 to 0.585, and compensation further increases it to 0.635. Besides, removing the disagreement penalty reduces the success rate to 0.614. These results demonstrate the effectiveness of CRC and its key components.

2) Feature Distance and Reward Prediction Error

We use feature distance as a statistical indicator of reward prediction reliability. Motivated by your comments, we validate the relationship between Feature Distance and Reward Prediction Error by computing the Pearson Correlation Coefficient for RM-rule and RM-pro on four Meta-World tasks. The coefficient ranges from −1-1 to 11, with larger positive values indicating stronger positive linear correlations. Specifically, for RM-rule, prediction errors are calculated against oracle rewards provided by the simulator. For RM-pro, we use an outcome-conditioned progress target, defined as yt=t/Ty_t=t/T for successful trajectories and yt=0y_t=0 for failed trajectories, where t=1,…,Tt=1, \ldots,T. RM-pre is excluded because its preference-based supervision does not provide pointwise reward targets for this analysis.

Reward ModelPearson Correlation Coefficient
RM-rule0.779
RM-pro0.792

As shown in the table, feature distance and reward prediction error exhibit strong positive correlations for both RM-rule and RM-pro, with Pearson coefficients of 0.779 and 0.792, respectively, supporting the use of feature distance as a statistical indicator of reward prediction error.

References

[1] Coste T, et al. Reward Model Ensembles Help Mitigate Overoptimization. ICLR, 2024.

[2] Liang X, et al. Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. ICLR, 2022.

AuthorsOpenReview ↗

Response to Reviewer vr6c (Part 2/2)

3) Reward Scale and Reward Shifts

In our implementation, raw rewards are min-max normalized to [0,1][0,1] before reward model training, and the same normalization is applied to reference rewards. Since reward model predictions are also constrained to [0,1][0,1], the concern regarding negative reward predictions does not arise in our setting.

While we acknowledge that adding a constant offset to both model predictions and reference rewards can introduce a confidence-dependent offset when the compensation coefficient is not one, our normalization pipeline eliminates additive shifts in raw rewards. Specifically, for any constant offset bb, Norm⁡(r+b)=(r+b)−(rmin⁡+b)(rmax⁡+b)−(rmin⁡+b)=Norm⁡(r).\operatorname{Norm}(r+b)=\frac{(r+b)-(r_{\min}+b)}{(r_{\max}+b)-(r_{\min}+b)}=\operatorname{Norm}(r). Therefore, adding a constant offset to raw rewards leaves the training targets and reference rewards unchanged, and consequently does not affect CRC's calibrated rewards under this pipeline.

4) Matching the Number of Corrected States Across ID and OOD Interventions

To account for the different proportions of ID (31.7%) and OOD (68.3%) states, we introduce a subsampled OOD-calibration control to approximately match the number of corrected rewards between ID and OOD interventions. Specifically, each OOD reward is independently replaced with its simulator oracle reward with probability p=0.317/0.683≈0.464p=0.317/0.683\approx0.464, while all remaining rewards retain the original model predictions. We evaluate this control across the same four Meta-World tasks (door-lock, handle-press-side, door-unlock, and sweep-into).

SetupSuccess Rate
Baseline (no repair)0.376
ID calibration (calibrate all ID states, 31.7% of all states)0.427
OOD calibration (calibrate a subset of OOD states, 31.2% of all states)0.522
OOD calibration (calibrate all OOD states)0.604
ID+OOD calibration (calibrate all states)0.686

Under this setting, 31.2% of all visited-state rewards are corrected, exclusively from OOD states, closely matching the 31.7% ID proportion. Despite the comparable numbers of calibrated rewards, subsampled OOD calibration achieves a success rate of 0.522, compared with 0.427 for ID calibration. This result provides strong empirical evidence that OOD reward errors have a greater impact on downstream RL performance than ID reward errors.

5) Practical Requirements and Reference Bank Construction

Following the ReWiND [3] setup, the reward model is jointly trained on data from 50 tasks, with 5 offline trajectories per task. CRC directly reuses these trajectories to construct task-specific reference and calibration banks, BrefB_{\mathrm{ref}} and BtestB_{\mathrm{test}}, for nearest-neighbor retrieval, reward compensation, and OOD threshold estimation. The reference features and rewards are precomputed and stored before online RL, requiring no additional data collection or reward model retraining.

a) Reference Bank Size

For each task, CRC constructs its reference bank using the states from three offline trajectories. The stored features and reward labels require only 192.4 KiB of CPU memory per task.

b) Retrieval Cost

Since online RL training in Meta-World consists of 70,000 steps, we measure the average and total computation time over 70,000 reward evaluations. Baseline includes only reward model inference, whereas CRC additionally incorporates nearest-neighbor retrieval and reward correction. Although CRC introduces additional reward computation overhead, the total increase of 28.67 seconds over 70,000 inferences accounts for only 0.05% of the approximately 15-hour online RL training process, demonstrating that CRC introduces a negligible additional cost compared with the overall RL training time.

MethodAverage time per inferenceTotal time for 70,000 inferences
Baseline2.52ms176.28s
CRC2.93ms (+0.41ms)204.95ms (+28.67s)

c) Sensitivity to the Features Used

To investigate the sensitivity of CRC to feature representations, we compare features extracted from different Transformer layers of the frozen reward model (RM), as well as those from a frozen DINOv2 encoder. Specifically, the middle-layer and last-layer variants use the hidden state from the second and fourth (last) Transformer layers, respectively. As shown in the table, CRC consistently outperforms the baseline across all feature representations and exhibits low sensitivity to the choice of feature extraction layer.

Feature UsedSuccess Rate
Baseline0.376
RM middle layer0.609
RM last layer (Default)0.635
Frozen DINOv20.620

References

[3] Zhang J, et al. ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations. CoRL, 2025.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer vr6c,

We appreciate your insightful feedback. We have carefully considered your comments and provided a point-by-point response, and we hope it addresses your concerns.

We are very keen to clarify any remaining points and look forward to your feedback.

Best regards,

Authors

Reviewer_W9iYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies reward unreliability under policy-induced distribution shift in online robotic RL and analyzes how reward prediction errors on out-of-distribution (OOD) states affect policy optimization. It proposes Confidence-Aware Reward Calibration (CRC), which uses nearest-neighbor distances in feature space to downweight unreliable predictions and compensates for this downweighting using reward values from neighboring training samples. Experimental results demonstrate improvements in task success rates during online policy learning.

Critical strengths
  1. The paper addresses a practical problem: reward prediction errors on OOD states can hinder online policy optimization. The diagnostic analysis provides useful evidence supporting the importance of this problem.
  2. CRC is simple to integrate and requires neither reward-model retraining nor additional annotations.
  3. The empirical results show consistent improvements in policy performance, and the ablation studies demonstrate the benefits of both components of CRC.
Critical weaknesses
  1. CRC uses neighboring reference rewards to compensate for low-confidence predictions. The accuracy of this compensation may depend on how smoothly rewards vary in the learned feature space. In tasks where similar states can have substantially different rewards or task final state, neighboring reference values may be misleading. For example, when a robot builds a five-cube tower, a small displacement between the bottom two cubes can substantially affect the tower’s stability.
  2. The experiments show that CRC improves the success rate of online robotic learning, but they do not directly establish how accurately it estimates rewards for OOD states. The authors could compare the original and calibrated rewards against simulator oracle rewards on the same held-out online trajectories, reporting errors separately for ID and OOD states. This would help clarify whether the performance improvements are accompanied by more accurate reward estimates.
  3. CRC is evaluated primarily against uncalibrated reward models, leaving its advantages over existing uncertainty-aware reward methods unclear. Relevant approaches include ensemble-based conservative rewards (Coste et al., 2024) [1] and lightweight uncertainty penalties based on frozen reward-model features (Zhang et al., 2024) [2].

[1] Coste, Thomas, et al. "Reward model ensembles help mitigate overoptimization.", ICLR 2024

[2] Zhang, Xiaoying, et al. "Mitigating reward overoptimization via lightweight uncertainty estimation.", NeurIPS 2024

Questions for authors

Please answer the points in weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer W9iY

Thank you for your valuable comments and suggestions. We hope the following response can address your concerns.

1) Reliability of Reward Compensation

In CRC, reward compensation does not rely solely on neighboring states with similar features, but also incorporates a penalty based on the variability of their rewards. Specifically, CRC introduces a disagreement penalty into the compensation: RComp=Rknn−λrσknnR_{\mathrm{Comp}}=R_{\mathrm{knn}}-\lambda_r\sigma_{\mathrm{knn}}, where σknn\sigma_{\mathrm{knn}} is the standard deviation of neighboring rewards. When neighboring states have substantially different rewards despite their feature similarity, the resulting higher disagreement increases σknn\sigma_{\mathrm{knn}}, reducing the compensation value and mitigating potential reward overestimation.

To further evaluate the effectiveness of reward compensation, we collect rollouts during online RL on the door-lock task and analyze reward prediction errors for low-confidence states. For states with confidence scores ct<0.25c_t < 0.25, incorporating compensation further reduces the MAE by 8.90% compared with reweighting alone. These results provide empirical evidence that reward compensation improves prediction accuracy for low-confidence states.

2) Accuracy of CRC-Calibrated Rewards

We compare original and CRC-calibrated predictions against simulator oracle rewards on the same recorded online trajectories across four Meta-World tasks. As shown in the following table, CRC reduces the average MAE from 0.163 to 0.129, achieving a 20.86% reduction and demonstrating that CRC effectively improves reward prediction accuracy during online RL.

TaskOriginalCRC
Door lock0.1850.140 (↓24.32%)
Handle press-side0.1400.099 (↓29.29%)
Door unlock0.1870.149 (↓20.32%)
Sweep into0.1400.128 (↓8.57%)
Average0.1630.129 (↓20.86%)

3) Comparison with More Baselines

We additionally compare CRC with UWO[1] and RUNE [2]. Since Zhang et al. [3], as suggested, have not publicly released their code, we instead include RUNE [2] as an additional uncertainty-aware baseline. The results are shown in the following tables. From the obtained average success rates over the eight Meta-World tasks, CRC achieves 39.18%, 43.87%, and 21.83% improvements over the original rewards of RM-rule, RM-pre, and RM-pro, respectively, and outperforms both UWO and RUNE by large margins.

RM-rule

TaskOriginalUWO [1]RUNE [2]CRC
Door lock0.5440.7650.4070.788
Handle press-side0.5630.7520.5630.798
Door unlock0.2900.3210.5930.613
Sweep into0.1070.4200.4050.339
Faucet close0.6550.7690.7850.795
Hammer0.7970.6590.8150.848
Lever pull0.1420.3240.5000.312
Button press-wall0.7800.7050.8520.908
Average0.4850.5890.6150.675 (↑39.18%)

RM-pre

TaskOriginalUWO [1]RUNE [2]CRC
Door lock0.8740.6780.6380.910
Handle press-side0.8660.7480.9870.987
Door unlock0.4720.5100.2640.509
Sweep into0.0020.2070.0430.226
Faucet close0.2210.5690.3180.947
Hammer0.5220.4860.4060.837
Lever pull0.3550.2560.4680.471
Button press-wall0.7390.8910.7530.938
Average0.5060.5430.4850.728 (↑43.87%)

RM-pro

TaskOriginalUWO [1]RUNE [2]CRC
Door lock0.8870.9150.8890.939
Handle press-side0.6820.9670.7620.978
Door unlock0.5530.2640.3660.752
Sweep into0.2140.3270.3100.436
Faucet close0.9570.9720.9310.991
Hammer0.9940.9800.9970.997
Lever pull0.3800.3740.4750.577
Button press-wall0.7560.9580.7610.937
Average0.6780.7200.6860.826 (↑21.83%)

References

[1] Coste T, et al. Reward Model Ensembles Help Mitigate Overoptimization. ICLR, 2024.

[2] Liang X, et al. Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. ICLR, 2022.

[3] Zhang X, et al. Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation. NeurIPS, 2024.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer W9iY,

We appreciate your insightful feedback. We have carefully considered your comments and provided a point-by-point response, and we hope it addresses your concerns.

We are very keen to clarify any remaining points and look forward to your feedback.

Best regards,

Authors

Thank you for the response and additional results. The oracle reward evaluation and comparison with UWO and RUNE address my concerns. Neighbor-based compensation may still be unreliable when rewards change abruptly in feature space, and I encourage the authors to acknowledge this limitation. I will raise my rating.

Dear Reviewer W9iY,

Thank you for your thoughtful feedback and active engagement throughout the rebuttal process. We sincerely appreciate your support and endorsement.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.