- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how ensemble uncertainty penalties affect action-gradient geometry in safe RL. It decomposes penalized cost gradients into parallel and orthogonal components and evaluates gradient rotation, scaling, and cross-fit consistency. Experiments on three MuJoCo environments show that bootstrap ensembles induce larger gradient rotations, while penalized gradients generally exhibit weaker agreement and local transfer across independently fitted critics.
Critical strengths
- The paper provides a clear geometric characterization of how uncertainty penalties affect action-gradient direction and magnitude.
- The matched experimental design and independently fitted evaluators offer a controlled assessment of gradient stability and local transfer.
Critical weaknesses
- The theoretical analysis primarily relies on standard vector decomposition, and the proposed audit introduces neither a new learning algorithm nor a substantial theoretical result.
- Experiments cover only three MuJoCo environments, two synthetic costs, and one critic architecture. More importantly, the evaluation relies on independently fitted critics rather than ground-truth cost reduction or actual policy performance, limiting the practical significance of the findings for safe RL.
Questions for authors
How does the observed cross-fit gradient agreement relate to actual cost-gradient accuracy or policy safety? Could the authors validate whether the penalized direction achieves better cost reduction using environment rollouts rather than another learned critic?
3 comments
Thank you for the detailed response and the additional experiments. I appreciate the ten-seed rerun and the new environment rollout evaluation.
However, my main concern remains the practical significance of the findings. The paper focuses on local action-gradient geometry, but provides no direct evaluation of policy learning performance or safety outcomes. While the new rollout experiments are a useful addition, all reported confidence intervals include zero, making it difficult to draw meaningful conclusions about the practical consequences of the observed gradient differences.
I understand the motivation for isolating the effect of uncertainty penalties through a controlled experimental design. However, without connecting these diagnostic metrics to actual policy optimization or safety performance, it remains unclear how important the findings are for practical safe RL.
I acknowledge the additional theoretical arguments highlighted by the authors, but I still find the overall contribution limited in scope and practical validation. Therefore, I maintain my current rating.
Thank you for clarifying the remaining concern. We understand that your assessment now centers on the significance of the diagnostic contribution and its connection to policy-level outcomes.
The specific contribution we ask to be considered is that conservative scalar predictions and near-unit gradient norms can conceal a construction-dependent change in the optimization direction. The matched experiments and the ten-seed rerun establish this behavior, and independent refitting shows that the changed direction is less reproducible across fitted critics.
The simulator comparisons address the local environment validation you requested. They show no advantage for the penalized direction in any of the nine comparisons, and we do not interpret them as establishing equivalence, superiority, or improved policy safety.
We agree that policy-level utility is a distinct empirical question. We kept it separate because a training run changes the reward gradient, the multiplier, the policy Jacobian, and the visitation distribution together, which would prevent attributing an outcome to the penalty. The paper's contribution is evidence about the optimization signal that safe-RL methods use, and we see the remaining difference as one of scope rather than of correctness.
Since submission we have added two experiments: a ten-seed rerun of the full study, and environment rollouts that compare the mean and penalized directions on actual cost.
Table A. Original study and ten-seed rerun.
Table B. Rollouts on actual cost. D is the discounted actual cost after a step along the mean direction minus that after a step along the penalized direction, so a positive D favors the penalized direction. Values are means over ten seeds with 95% seed-bootstrap intervals, multiplied by 1,000.
W1. The review describes only the decomposition of Section 3.2. The formal results are in Sections 3.1 and 3.3, which it does not mention. Eq. (3) shows that two ensembles can agree on every member prediction, on the spread, and on the mean gradient and still have arbitrary spread gradients, so no scalar uncertainty estimate determines the direction the actor receives. Eq. (10) shows that scoring a penalized direction on its own fit favors it by construction, which is why the audit requires an independent evaluator. Together they expose an effect that no existing check detects: in HalfCheetah, most actions have their cost gradient rotated by more than 10° while its norm changes by less than 5%.
W2. The narrow design is deliberate. A cost action gradient is one input to an actor update, and a training run changes the reward gradient, the multiplier, the policy Jacobian, and the visitation distribution at the same time, so it cannot attribute an outcome to the penalty. Matched data makes the effect attributable, and the ten-seed rerun confirms it: bootstrap rotates the gradient more than shared data in 59 of 60 paired comparisons.
Q1. Cross-fit agreement is not presented as gradient accuracy or policy safety. It measures whether a direction survives refitting on the same data, and on that measure the mean gradient is more reproducible than the penalized gradient in 18 of 18 groups.
Q2. We ran this test. From identical restored simulator states, we take one step along each direction, keep all later actions identical, and compare the discounted actual cost. The penalized direction shows no cost-reduction advantage in any of the nine comparisons in Table B, and six of the nine point estimates favor the mean direction.
Rating. The weaknesses listed concern novelty and scope. The review identifies no error in the derivations or the measurements, and its own strengths describe the design as a controlled assessment. The rollout test it asked for has now been run, so we ask that the overall rating be reconsidered.