- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper considers online RL with a pretrained reward model. The authors propose Confidence-Aware Reward Calibration (CRC) to identify and calibrate rewards on out-of-distribution (OOD) state visitation, without retraining the reward model, and it can be applied to different frozen reward models. Experiments across diverse reward models and RL algorithms on simulation and real-world tasks demonstrate the improvement gain from CRC.
Critical strengths
- The problem is well-motivated with the diagnostic findings. Figure 1 shows the proportion of OOD states, the relationship between reward error and feature-space distance, and the performance increase with corrected rewards. They together identify unreliable reward predictions on OOD states as a key source of error in online policy optimisation.
- The method is simple to integrate into existing RL pipelines. It only does the calibration without retraining the reward model or additional trajectory collection
- Its performance in experiments across simulation and real-world tasks is encouraging.
Critical weaknesses
- The authors argue that the neighbour-guided reward compensation is motivated by the fact that “States that are sparsely represented offline may receive low confidence even when they correspond to meaningful task progress during online interaction”. However, whether the proposed compensation mechanism actually mitigates this issue is unclear. For low-confidence data, the calibrated reward turns to the KNN-based compensation, but it is unclear whether this estimate is more reliable or simply smooths the existing predictions or reference values. Moreover, low standard deviation among neighbours does not necessarily imply high reliability if all neighbours are distant from the measured one. Although the ablation analysis shows the benefit of compensation, it does not imply that the improved performance is due to recovery of meaningful rewards.
- The experimental comparisons are insufficient to fully validate the proposed design choices. The evaluation of the disagreement penalty effect, comparisons with other baselines considering reward uncertainty, and sensitivity analyses of some hyperparameters are missing.
Questions for authors
- Could the authors clarify the relationship between the reference set and B__\mathrm{test}? Are they the same set? Also, could the authors clarify the size or proportion of offline samples held out for threshold calibration? Would this split influence the final performance?
- What’s the actual effect of the neighbour-guided reward compensation? Could the authors clarify the causal effect of weighted nearest neighbours and a meaningful reward signal?
- Additional experimental comparisons would better support the mechanism design. Could the authors also test CRC without the disagreement penalty to isolate its effect? The main experimental comparisons are against uncalibrated reward models, with additional ablations of two proposed components. Comparisons with existing reward uncertainty estimation or calibration methods, even if they involve additional data collection or computational costs, would help better assess the effectiveness of CRC. Moreover, there exist other hyperparameters whose sensitivities are not reported in the ablation studies, e.g., , which controls the confidence decay rate, , which controls the strength of reward compensation, and , which controls the penalty for disagreement among neighbouring reference values.
8 comments
Response to Reviewer dtwj (Part 2/2)
3) Clarification and Ablation Analysis of and
and are two disjoint subsets of trajectories randomly sampled from a pool of offline data , i.e., . More specifically, is used for OOD distance calculation and reward compensation, while is used to determine the OOD threshold based on its distances to .
We investigate the impact of different bank split settings of and across four Meta-World tasks (door-lock, handle-press-side, door-unlock, and sweep-into) in the following table. As observed, CRC consistently outperforms the original baseline across all evaluated bank split settings, with the ratio of 3:2 achieving the best performance. These results demonstrate the robustness of CRC against bank splits.
Bank Split
| Average | |
|---|---|
| Baseline | 0.376 |
| 1:4 | 0.440 |
| 2:3 | 0.529 |
| 3:2 (Default) | 0.635 |
| 4:1 | 0.618 |
4) Additional Hyperparameter Ablations
Inspired by your comments, we further evaluate the sensitivity of CRC to the confidence decay rate , compensation strength , and disagreement penalty coefficient across four Meta-World tasks (door-lock, handle-press-side, door-unlock, and sweep-into), varying each parameter independently. As shown in the following tables, CRC consistently outperforms the original baseline across all tested hyperparameter settings, demonstrating its robustness and effectiveness across different configurations.
Confidence Decay
| Success Rate | |
|---|---|
| Baseline | 0.376 |
| 0.6 | 0.549 |
| 1.2 (Default) | 0.635 |
| 1.8 | 0.614 |
Compensation Strength
| Success Rate | |
|---|---|
| Baseline | 0.376 |
| 0.2 | 0.550 |
| 0.3 (Default) | 0.635 |
| 0.4 | 0.604 |
Disagreement Penalty
| Success Rate | |
|---|---|
| Baseline | 0.376 |
| 0.8 | 0.610 |
| 1.0 (Default) | 0.635 |
| 1.2 | 0.625 |
Gentle Reminder
Dear Reviewer dtwj,
We appreciate your insightful feedback. We have carefully considered your comments and provided a point-by-point response, and we hope it addresses your concerns.
We are very keen to clarify any remaining points and look forward to your feedback.
Best regards,
Authors
Response to Reviewer dtwj (Part 1/2)
Thank you for your thoughtful comments and constructive suggestions. We provide detailed responses to your concerns below.
1) Effectiveness and Reliability of Reward Compensation
Reward compensation aims to preserve meaningful reward signals that would otherwise be suppressed by reweighting in low-confidence states. To evaluate its effectiveness, we analyze online RL rollouts on the door-lock task. For states with ct<0.25, compensation further reduces the reward prediction MAE by 8.90% compared with reweighting alone, demonstrating improved reward accuracy in low-confidence states.
The disagreement penalty improves compensation reliability by penalizing variations in neighboring reward values, rather than feature-space distances. Even when all neighboring states are distant from the query state, differences in their reward values still incur a penalty that mitigates potential reward overestimation. As shown in the following table, reweighting improves the average success rate from 0.376 to 0.585, and compensation further increases it to 0.635. Removing the disagreement penalty reduces the success rate to 0.614, demonstrating the effectiveness of the compensation design.
Furthermore, the reward visualizations in Fig. 8 provide qualitative evidence from two real-world OOD cases. In both cases, compensation restores positive reward signals that align with task progress but are otherwise suppressed by reweighting, demonstrating its effectiveness in preserving meaningful reward information.
2) Comparison with More Baselines
We additionally compare CRC against two uncertainty-aware baselines, UWO [1] and RUNE [2]. The results are presented in the following tables. Across the eight Meta-World tasks, CRC improves the average success rates by 39.18%, 43.87%, and 21.83% over the original RM-rule, RM-pre, and RM-pro baselines, respectively, while outperforming both UWO and RUNE by substantial margins.
RM-rule
RM-pre
RM-pro
References
[1] Coste T, et al. Reward Model Ensembles Help Mitigate Overoptimization. ICLR, 2024.
[2] Liang X, et al. Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. ICLR, 2022.