acceptodds
Under review as a conference paper at ICLR 2027

On the Reliability of Value Estimation for Safe Exploration

Abstract

Safe exploration requires reinforcement learning (RL) agents to effectively trade off task performance against the risk of constraint violations during learning. Constrained RL algorithms use reward and cost value estimates to guide this trade-off, while their theoretical guarantees are stated in terms of the true value functions that these estimates approximate. Yet the reliability of the cost critic is rarely examined directly. We show that cost critics exhibit substantially poorer agreement with Monte Carlo reference values than reward critics. The regression problem for the cost critic spans a spectrum, from heavily bootstrapped targets that are stable but inherit the critic's own errors, to long-horizon targets that contain more information about future cost but are high variance and difficult to generalize from. Neither end yields a critic that correlates well with the reference cost value, though for different reasons. Because cost-critic errors do not exhibit a systematic bias, existing bias-control methods do not recover value fidelity and instead primarily make policy updates more conservative; regularization likewise fails to improve the overall safety–performance trade-off. We therefore explore successor representations as an alternative approach to learning cost value functions and show that they provide a promising direction for safer and more sample-efficient exploration.

Primary area:
reinforcement learning
Keywords:
Safe Exploration, Value Estimation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VGyXOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reliability of cost-value estimation in constrained RL and its impacts on safe exploration. The paper shows that CPO (PIDLag) can satisfy constraints even with inaccurate cost critics. This can underestimate high-cost states and overestimate safe states. The analysis shows that existing bias-correction and regularization techniques do not consistently improve the safety–performance trade-off. Then, the paper proposes successor representations (SR) to decouple long-horizon state prediction from immediate cost estimation. Experiments show that SR-based critics improve early high-risk state discrimination and improve CPO and PIDLag, achieving higher returns with fewer constraint violations.

Critical strengths
  1. This paper presents a detailed, comprehensive, and empirical analysis of estimation errors and their impacts on safe exploration. The experiments also systematically investigate the potential causes. The experiments using MICE, clipped double-Q, dropout, and L2 regularization are convincing (for the used environments). They help distinguish the effects of critic estimation from conservative policy optimization.

  2. The SR-based experiments report improvements in cumulative training cost and episodic return for both CPO and PIDLag baselines.

Critical weaknesses
  1. Limited experiments and algorithm diversity: The paper mainly conducts experiments in the Point-Goal environment and only considers CPO and PIDLag methods. The readers cannot judge whether the conclusions and phenomena hold true for other envs and constrained RL methods with only one env.

  2. The source of the improvement of SR for constrained RL is unclear. More ablation studies are needed to explain why, and which key designs of SRs can address the safe exploration issue.

Questions for authors

Q1. Why is only the Point-Goal env evaluated in the empirical analysis? I think that adding more diverse envs is necessary to support the empirical analysis on cost estimation reliability.

Q2. The SR critic introduces contrastive features, shared predictive representations, separate readouts, and additional optimization choices. Which design has the highest contribution to the reported improvement? Can the authors isolate the contribution of different components? Otherwise, it is hard for the readers to figure out why SR works.

Q3. The paper evaluates state-value correlations and high-risk AUROC. However, the optimizer acts on estimated advantages, so better correlation between V^c(s)\hat V_c(s) and Vcπ(s)V_c^\pi(s) does not necessarily mean a more accurate cost policy gradient, which is also important for safe exploration. How would the authors evaluate the impact of cost policy gradient?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer VGyX

We thank the reviewer for their time and effort in reviewing this paper and their valuable feedback. We have revised the manuscript, and all modifications are highlighted in teal for ease of reference.

[W1 & Q1]: Limited experiments and algorithm diversity

We agree with the reviewer and we've extended our analysis to 3 additional environments and 1 additional baseline in the revised manuscript. Please refer to the General Response to All Authors (above) and Appendix A of the revised manuscript for details.

[W2 & Q2]: SR attribution

We agree with the reviewer that the SR attribution was unclear given the fact that there were multiple components that could impact the performance of the final policy. We've included two ablations in the revised manuscript to clarify this, first we've added an alternate version of SR critic called (SR cost) (in Fig. 6 of the revised manuscript) where only cost value estimate uses the successor representation and reward value is calculated using standard/Base critic. Second, we've also analyzed the effect of the feature map ϕ\phi on the final performance by comparing random feature map (SR (random)) to learned feature map (SR (contrastive)) in Fig. 7 of the revised manuscript. More details in General Response to All Authors (above) and Section 5.1 of the revised manuscript

[Q3] Critic Quality & Cost Policy Gradient

We agree with the reviewer that better value correlation does not necessarily guarantee better cost policy-gradient alignment. However, value estimation is particularly important in the limited-sample setting of on-policy constrained RL, where policy updates rely on relatively few trajectories and bootstrapped advantage estimates. While errors in a state-dependent baseline cancel in expectation, they can increase the variance of finite-sample gradient estimates. Moreover, errors in bootstrapped value estimates can introduce bias into the estimated advantages and resulting policy gradients. Thus, unreliable cost critics can affect policy updates, particularly early in training when data is limited.

Directly evaluating gradient alignment would require a reliable reference cost policy gradient, obtained through extensive sampling of states, actions, and future trajectories. This is significantly more computationally expensive than the critic evaluation, and the resulting gradient estimate is itself subject to sampling error. We therefore focus on value fidelity as an important source of error in advantage estimation, without claiming that improved correlation guarantees better gradient alignment. Importantly, our results show that improving cost-value estimation with SR critics also leads to safer and more efficient exploration, supporting the practical relevance of this analysis.

Dear Authors,

Thank you for your efforts in substantially improving this work. The revised manuscript is much clearer and more complete than the initial version. Its contribution is more on the diagnostic side: the reliability of cost-value estimation in safe RL. In my own safe RL experiments, I indeed observed similar phenomena but did not look into it. So now I think this paper indeed proposes a valuable research question and provides some insights. However, I still have the following concerns:

  1. SR is a well-established method and a reasonable choice in this study. The added ablations are somewhat useful. However, it remains unclear whether the improvements come from SR's better representation, additional auxiliary supervision, or the added inductive bias introduced by the linear value structure.

  2. I think that the authors need to establish a clear chain as follows: SR imrpoves cost-value estimates, and better cost-value estimates further improve safe policy learning. For the first part, the results and the analysis (page 9) show that SR mainly distinguishes high-cost and low-cost states, instead of better cost-value estimation. For the second part, the policy update depends on advantage estimates, a more direct analysis on cost advantage bias would be more convincing. Value estimation error may not "translate" into advantage estimation error, because errors in the cost-value estimates may either cancel each other out or compound to amplify the advantage errors. I hope that I express the logic clearly.

As the author-reviewer discussion period is ending soon. Adding more experiments may be impractical. In this paper's current form, I would raise my rating to 2.

With a more complete presentation and more comprehensive experiments, this paper has the potential to become a nice study. However, given the limitations of the current version, I believe 2 is a reasonable rating.

Thank you again for your interesting work.

Reviewer_1WcvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether learned critics in constrained RL accurately estimate the underlying value functions. Using Monte Carlo reference values on Safety Gym PointGoal1, the authors show that CPO and PIDLag reward critics reach correlation around 0.8 with references while cost critics remain near 0.2 despite satisfying the constraint. Decomposing errors by value regime, they find that near-zero aggregate error conceals underestimation of high-cost states and overestimation of intermediate states, with AUROC around 0.62 for detecting imminent cost. A TD(lambda_c) sweep shows that low-variance bootstrapped targets are easy to fit but inherit critic errors, while high-variance MC targets are faithful but hard to generalize from. Bias-control methods (MICE, CDQ) shift predictions upward without improving correlation, and a biased-cost ablation suggests MICE mainly makes updates conservative. L2/dropout improve correlation but not the return-cost trade-off. Finally, successor representations shared between reward and cost critics are proposed and shown to improve early cost prediction, AUROC, training cost, and return on the same task.

Critical strengths
  1. The paper identifies a genuinely under-examined question: constraint satisfaction during training need not imply accurate cost-value estimation. Its evaluation protocol, based on Monte Carlo reference values with per-regime signed error, AUROC, and split-half ceilings, cleanly separates ranking errors from scale errors and goes well beyond the aggregate MSE used in prior analyses.
  2. The TD(lambda_c) sweep decomposes a familiar bias-variance trade-off into two verifiable claims: critics fit low-variance bootstrapped targets almost perfectly yet remain uncorrelated with the MC reference, whereas faithful MC targets are largely unlearnable from finite on-policy data. Separating fitting the target from estimating the value is a clean and useful analytical device.
  3. The biased-cost ablation is a genuinely non-obvious negative result: a simple decaying bias on the empirical constraint cost reproduces MICE's conservative early behavior, indicating that its benefit comes from biasing CPO's update selection rather than from improved critic fidelity. This directly challenges underestimation correction as a remedy and motivates structural fixes.
Critical weaknesses
  1. The proposed SR critic assumes that reward and cost are linear in learned features, an assumption that is neither validated nor analyzed; no error propagation from feature misspecification to V_c is given, so the method has no theoretical justification, and its claimed gains are demonstrated only on a single sparse-cost task where the linearity assumption is implausible.
  2. The central diagnosis rests on Pearson correlation against a 10-rollout MC reference for sparse, heavy-tailed costs. Pearson is dominated by scale and outliers and is inconsistent with the paper's own AUROC evidence of partial ordering; rank correlations, confidence intervals, and distribution-level diagnostics are absent, so the headline low-correlation claim may partly be an artifact of metric choice.
  3. Generalization evidence is thin: all headline claims come from PointGoal1, SR is compared only with CPO/PIDLag, the significance of SR-versus-base differences is weak (as the authors acknowledge), and locomotion, continuous-cost, multi-constraint, or risk-aware baselines are missing; the evaluation protocol itself also requires simulator reset access, limiting transfer to real systems.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 1Wcv

We thank the reviewer for their time and effort in reviewing the paper. We have revised the manuscript, and all modifications are highlighted in teal for ease of reference.

[W1] SR critic: linearity and scope

We have revised Section 5 in the manuscript to clarify the following points. First, we emphasize that the linearity assumption applies to the instantaneous cost, not the cost-value function. For sparse binary costs, this amounts to a binary classification problem with dd-dimensional inputs, a considerably weaker requirement than directly learning the long-horizon cost-value function VcπV_c^\pi. Second, we now evaluate this assumption empirically rather than simply relying on it: on held-out transitions, the linear readout ϕ⊤wc\phi^\top w_c distinguishes cost-incurring states from the rest with an AUROC of approximately 0.90.9, demonstrating that the feature map supports effective linear discrimination of cost-incurring states. Third, regarding scope, we agree that experiments on a single task provide limited evidence for SR as a general-purpose method, which is not the claim we make. The SR critic is introduced in the paper as a way to test our diagnosis: if cost-value estimation is hindered by sparse long-horizon regression targets, then shifting the sparsity into a one-step regression problem should help, as our experiments demonstrate. Its primary role here is to test this mechanism, and we have revised the text to avoid presenting it as a general-purpose solution. We've also extended our analysis to three additional environments.

[W2] Robustness of Evaluation metrics

We have extended our analysis to examine whether the low correlations arise from differences in scale, outliers, the choice of metric, or noise in the Monte Carlo reference estimates.

  1. low correlation is not a scaling artifact. Pearson correlation is invariant to positive affine transformations of both predictions and reference values, i.e., Corr(X,Y)=Corr(a+bX,c+dY)\mathrm{Corr}(X,Y)=\mathrm{Corr}(a+bX,c+dY) for b,d>0b,d>0. We now also report Spearman rank correlation (Fig. 1, 6 and 7 of the revised manuscript), which is invariant to strictly monotone increasing transformations. The results are consistent: cost critics exhibit an average Spearman correlation of approximately 0.20.2 (against a split-half ceiling of 0.920.92), whereas reward-critic correlation stabilizes around 0.80.8 after 1M steps. This indicates that the critic has not simply learned an incorrectly scaled version of the true cost-value function.

  2. low correlation is not driven by a few outliers To assess the impact of outliers, we systematically removed the three highest-value probe states, which account for approximately 7070% of the between-state variance in the reference values. Rather than improving, Pearson correlation decreases from 0.220.22 to 0.180.18 for CPO and from 0.180.18 to 0.170.17 for PIDLag. Removing the ten highest-value states further reduces these correlations to 0.140.14 and 0.110.11, respectively. Thus, poor cost-value estimation is distributed across the probe set rather than concentrated in a few outliers.

  3. low correlation is not an artifact of noisy Monte Carlo reference estimates. Given the sparse and heavy-tailed cost signal, we explicitly evaluate the reliability of our Monte Carlo estimates V~c\tilde{V}_c (Appendix D and Fig. 17 & 19). We find that a perfect critic would achieve a correlation of 0.9830.983 with V~c\tilde{V}_c for CPO and 0.9750.975 for PIDLag (median across seeds and checkpoints). Accounting for reference noise therefore changes the reported correlations by at most 0.010.01, confirming that the low cost-critic correlations reflect poor agreement with the underlying value function rather than noise in the reference.

  4. We further examine whether these observations depend on the cost distribution by varying the number of hazards in PointGoal1 among 22, 44, 88 (default), and 1010 (Fig. 15, 16). Despite changes in the cost distribution, both Pearson and Spearman correlations remain substantially below their split-half ceilings (below 0.40.4, except for Pearson correlation with 22 hazards, which reaches 0.50.5).

  5. AUROC is consistent with the correlation metrics. Distinguishing high-value states from the rest is a weaker requirement than accurate value regression, and the critic does exhibit above-chance discrimination (AUROC of 0.620.62, averaged over the final 8M steps). However, expressing AUROC as 2,AUROC−12,\mathrm{AUROC}-1 (Somers' DD) gives 0.240.24, against a split-half ceiling of 0.860.86. Thus, Pearson ≈0.2\approx 0.2, Spearman ≈0.2\approx 0.2, and Somers' D≈0.24D \approx 0.24 consistently indicate that the cost critic captures some useful information about state values but remains far from the attainable level of fidelity.

The analysis highlights that the underlying issue is therefore not merely one of scaling or calibration, but a failure to accurately recover the state-dependent structure of the cost-value function.

AuthorsOpenReview ↗

Author Response to Reviewer 1Wcv (Continued)

[W3]: Limited Experiments (Generalization Evidence is Thin)

We agree with the reviewer and we've added results for 3 additional environments and 1 additional baseline to the revised manuscript. Please refer to the General Response to All Authors (above) for details.

Reviewer_HRoGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the reliability of reward and cost critics in safe RL using Monte Carlo reference values. It finds that small aggregate cost-estimation error can hide substantial state-dependent errors, and studies the effects of target selection, bias correction, and regularization. It also evaluates successor representations as an alternative critic architecture.

Critical strengths
  • The distinction between aggregate bias and state-wise prediction quality is useful. The diagnostics show why a small average error can give a misleading impression of the information available to the policy update.
  • The Monte Carlo reference values are checked carefully. The biased-cost ablation also provides an interesting explanation for why a method can improve safety without improving the critic itself.
Critical weaknesses
  • The successor-representation experiments do not isolate the role of cost-value estimation. This method changes both reward and cost critics and introduces additional representation learning. Therefore, the performance gains could come from several sources. The evidence for improved cost fidelity is also mixed. This makes the proposed connection between better cost estimate and safer learning insufficiently established.
  • Testing two policy optimization method is useful but it remains unclear whether the same diagnosis and conclusions about the bias correction hold under these different tasks.
Questions for authors
  • Can the performance improvement be ablated? Specifically, can the impact of changing the cost critic be separated as mentioned in the weakness part?
  • Which findings remain consistent on tasks with different cost structures?
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer HRoG

We thank the reviewer for their thoughtful and constructive feedback. We have revised the manuscript, and all modifications to the original version have been highlighted in teal for ease of reference.

[W1 & Q1] Isolating the Role of Cost-Value Estimation and Other Aspects of SR

  1. We agree with the reviewer that the original version of the manuscript did not isolate the impact of SR on cost-value estimation, since it used the same representation for both reward and cost critics. We have since added an SR cost version of the Base algorithm to the revised manuscript (Fig. 6 and Section 5.1), which uses successor representations only for the cost critic and the standard/Base critic for reward-value estimation. We observe that SR cost outperforms the Base critic in both the safety–efficiency trade-off and cost-critic quality, particularly in terms of AUROC and MAE.

  2. To further understand and analyze the components of the SR critic, we ablate the choice of feature map ϕ\phi, comparing a contrastive ϕ\phi trained using InfoNCE (called SR (contrastive)) to a randomly initialized and frozen ϕ\phi (called SR (random)) (Section 5.1 and Fig. 7 of the revised manuscript). Results show that SR (random) improves critic quality, particularly in terms of AUROC and MAE, and improves performance compared to the Base critic. However, there is a representational advantage to using SR (contrastive), which further improves upon the performance of SR (random).

[W2 & Q2] Extending Results to More Environments and Baselines

We agree with the reviewer, and we have extended our analysis to three additional environments: PointButton1, PointPush1, and PointGoal2, and one additional baseline: CRPO [1], in the revised manuscript (Appendix A and Figs. 9–14). Results show that the findings presented in the original manuscript are consistent across all four environments. Figs. 12, 13, and 14 compare the performance of the SR critic against the Base critic across all four environments for CPO [2], PIDLag [3], and CRPO [1], respectively. We observe that the SR critic improves cost-value estimation quality across multiple metrics and consistently achieves a better safety–efficiency trade-off compared to the Base critic.

[1] Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration (ICML 2025)

[2] Constrained Policy Optimization (ICML 2017)

[3] Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (ICML 2020)

Comments on the paper

2 comments
AuthorsOpenReview ↗

Clarifying the Central Contributions of the paper

We thank the reviewers for their careful reading of the paper. We would like to take this opportunity to explicitly state what we see as the paper’s central contribution.

A common assumption in safe RL is that the cost critic’s primary failure mode is underestimation: estimation errors are assumed to be predominantly one-sided (underestimating cost where it is imminent), suggesting that increasing cost estimates can address the problem. Methods that intervene on cost estimates, including MICE [1], are motivated by this assumption. Our analysis challenges this view. Throughout training, CPO [2] and PIDLag [3] satisfy the constraint despite their cost critics exhibiting a correlation of approximately 0.20.2 with the reference cost values, compared to an attainable 0.980.98. Moreover, estimation errors exhibit a state-dependent pattern rather than a systematic bias: low- and intermediate-value states are overestimated, while high-value states are underestimated. Our analysis shows that this is neither a calibration artifact nor a scaling issue. Spearman rank correlation is no higher than Pearson correlation (0.220.22 vs. 0.220.22 for CPO and 0.200.20 vs. 0.180.18 for PIDLag [3], respectively, against attainable values of 0.920.92 and 0.980.98). The critic has therefore not learned the correct ordering with merely an incorrect scale; no monotonic transformation can recover the correct ordering.

The weak signal the critic does carry is still useful and, through gradient updates, drives the agent toward constraint satisfaction. Deliberately biasing cost estimates to promote more conservative updates (cost-only updates in CPO [2] or a larger Lagrange multiplier in PIDLag [3]) beyond what is required for constraint satisfaction leads to lower total cost at the expense of lower expected return, reproducing the performance of MICE [1]. Leveraging the modest quality of the cost critic through increased conservatism is not the same as improving cost-critic quality or the efficiency of cost-driven updates. Moreover, straightforward approaches to improving the critic itself do not yield substantial gains: across alternative regression targets and regularization schemes, we observe only modest improvements, consistent with the diagnosis above that the difficulty is not merely one of regression tuning.

Finally, we explore representation learning as an alternative direction for improving cost-value estimation. By reformulating cost-value learning through successor representations, we show that cost-reducing updates become more effective without modifying the outer optimization loop, allowing the agent to satisfy constraints with less conservative behavior (fewer cost-oriented updates). These findings suggest that representation learning is a promising direction for future research toward improving cost-critic quality and safe exploration.

[1] Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration (ICML 2025)

[2] Constrained Policy Optimization (ICML 2017)

[3] Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (ICML 2020)

AuthorsOpenReview ↗

General Response to All Reviewers

We thank all the reviewers for providing their valuable feedback on the paper. It has been very helpful in improving the quality and clarity of the manuscript. We have now revised the manuscript, and all modifications to the original version have been highlighted in teal for ease of reference. Below, we address two main concerns raised by all three reviewers. Other specific questions or concerns have been addressed individually.

1. Limited Experiments (Environments & Baselines)

Our initial analysis focused on examining the role of the cost critic in constrained RL, prioritizing depth of analysis in the representative PointGoal1 environment. We agree with the reviewers that the analysis presented in the paper can be greatly strengthened by demonstrating its findings across multiple environments and additional baselines. We have since extended our analysis to three additional environments (PointButton1, PointPush1, and PointGoal2) and added one more baseline, CRPO [1], to our analysis. Due to limited space in the main paper, we have included these results and discussions in the appendix (Appendix A and Figs. 9–14). Our experiments show that the findings presented in the original manuscript are consistent across all four environments and that the Successor Representation (SR) critic improves performance across all four environments and all three baselines, while also improving critic quality, particularly in terms of Spearman correlation, AUROC, and mean absolute error (MAE).

2. SR Attribution

We agree with the reviewers that the original version of our manuscript did not properly isolate the impact of successor representations (SR) on the cost critic, as the same representation was used for both reward- and cost-value prediction. In the revised manuscript, we have added an ablation to address this (Section 5.1 and Fig. 6), where an alternative SR version (named SR cost) is used, in which only the cost critic uses the successor representation for value prediction, while the reward critic uses the standard/Base method for value estimation. Results show that using the SR critic only for cost improves the safety–efficiency trade-off over the Base critic for both PIDLag [3] and CPO [2], while also improving critic quality across the evaluated metrics.

[1] CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee (ICML 2021)

[2] Constrained Policy Optimization (ICML 2017)

[3] Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (ICML 2020)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.