- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the reliability of cost-value estimation in constrained RL and its impacts on safe exploration. The paper shows that CPO (PIDLag) can satisfy constraints even with inaccurate cost critics. This can underestimate high-cost states and overestimate safe states. The analysis shows that existing bias-correction and regularization techniques do not consistently improve the safety–performance trade-off. Then, the paper proposes successor representations (SR) to decouple long-horizon state prediction from immediate cost estimation. Experiments show that SR-based critics improve early high-risk state discrimination and improve CPO and PIDLag, achieving higher returns with fewer constraint violations.
Critical strengths
-
This paper presents a detailed, comprehensive, and empirical analysis of estimation errors and their impacts on safe exploration. The experiments also systematically investigate the potential causes. The experiments using MICE, clipped double-Q, dropout, and L2 regularization are convincing (for the used environments). They help distinguish the effects of critic estimation from conservative policy optimization.
-
The SR-based experiments report improvements in cumulative training cost and episodic return for both CPO and PIDLag baselines.
Critical weaknesses
-
Limited experiments and algorithm diversity: The paper mainly conducts experiments in the Point-Goal environment and only considers CPO and PIDLag methods. The readers cannot judge whether the conclusions and phenomena hold true for other envs and constrained RL methods with only one env.
-
The source of the improvement of SR for constrained RL is unclear. More ablation studies are needed to explain why, and which key designs of SRs can address the safe exploration issue.
Questions for authors
Q1. Why is only the Point-Goal env evaluated in the empirical analysis? I think that adding more diverse envs is necessary to support the empirical analysis on cost estimation reliability.
Q2. The SR critic introduces contrastive features, shared predictive representations, separate readouts, and additional optimization choices. Which design has the highest contribution to the reported improvement? Can the authors isolate the contribution of different components? Otherwise, it is hard for the readers to figure out why SR works.
Q3. The paper evaluates state-value correlations and high-risk AUROC. However, the optimizer acts on estimated advantages, so better correlation between and does not necessarily mean a more accurate cost policy gradient, which is also important for safe exploration. How would the authors evaluate the impact of cost policy gradient?
2 comments
Dear Authors,
Thank you for your efforts in substantially improving this work. The revised manuscript is much clearer and more complete than the initial version. Its contribution is more on the diagnostic side: the reliability of cost-value estimation in safe RL. In my own safe RL experiments, I indeed observed similar phenomena but did not look into it. So now I think this paper indeed proposes a valuable research question and provides some insights. However, I still have the following concerns:
-
SR is a well-established method and a reasonable choice in this study. The added ablations are somewhat useful. However, it remains unclear whether the improvements come from SR's better representation, additional auxiliary supervision, or the added inductive bias introduced by the linear value structure.
-
I think that the authors need to establish a clear chain as follows: SR imrpoves cost-value estimates, and better cost-value estimates further improve safe policy learning. For the first part, the results and the analysis (page 9) show that SR mainly distinguishes high-cost and low-cost states, instead of better cost-value estimation. For the second part, the policy update depends on advantage estimates, a more direct analysis on cost advantage bias would be more convincing. Value estimation error may not "translate" into advantage estimation error, because errors in the cost-value estimates may either cancel each other out or compound to amplify the advantage errors. I hope that I express the logic clearly.
As the author-reviewer discussion period is ending soon. Adding more experiments may be impractical. In this paper's current form, I would raise my rating to 2.
With a more complete presentation and more comprehensive experiments, this paper has the potential to become a nice study. However, given the limitations of the current version, I believe 2 is a reasonable rating.
Thank you again for your interesting work.
Author Response to Reviewer VGyX
We thank the reviewer for their time and effort in reviewing this paper and their valuable feedback. We have revised the manuscript, and all modifications are highlighted in teal for ease of reference.
[W1 & Q1]: Limited experiments and algorithm diversity
We agree with the reviewer and we've extended our analysis to 3 additional environments and 1 additional baseline in the revised manuscript. Please refer to the General Response to All Authors (above) and Appendix A of the revised manuscript for details.
[W2 & Q2]: SR attribution
We agree with the reviewer that the SR attribution was unclear given the fact that there were multiple components that could impact the performance of the final policy. We've included two ablations in the revised manuscript to clarify this, first we've added an alternate version of SR critic called (SR cost) (in Fig. 6 of the revised manuscript) where only cost value estimate uses the successor representation and reward value is calculated using standard/Base critic. Second, we've also analyzed the effect of the feature map ϕ on the final performance by comparing random feature map (SR (random)) to learned feature map (SR (contrastive)) in Fig. 7 of the revised manuscript. More details in General Response to All Authors (above) and Section 5.1 of the revised manuscript
[Q3] Critic Quality & Cost Policy Gradient
We agree with the reviewer that better value correlation does not necessarily guarantee better cost policy-gradient alignment. However, value estimation is particularly important in the limited-sample setting of on-policy constrained RL, where policy updates rely on relatively few trajectories and bootstrapped advantage estimates. While errors in a state-dependent baseline cancel in expectation, they can increase the variance of finite-sample gradient estimates. Moreover, errors in bootstrapped value estimates can introduce bias into the estimated advantages and resulting policy gradients. Thus, unreliable cost critics can affect policy updates, particularly early in training when data is limited.
Directly evaluating gradient alignment would require a reliable reference cost policy gradient, obtained through extensive sampling of states, actions, and future trajectories. This is significantly more computationally expensive than the critic evaluation, and the resulting gradient estimate is itself subject to sampling error. We therefore focus on value fidelity as an important source of error in advantage estimation, without claiming that improved correlation guarantees better gradient alignment. Importantly, our results show that improving cost-value estimation with SR critics also leads to safer and more efficient exploration, supporting the practical relevance of this analysis.