acceptodds
Under review as a conference paper at ICLR 2027

Auditing Action-Gradient Geometry in Ensemble Cost Penalties

Abstract

A common ensemble cost penalty in safe reinforcement learning adds prediction spread to the mean cost. A gradient-based actor follows the action derivative of this penalized estimate. We introduce an action-gradient audit to distinguish changes in direction from changes in signed scale and magnitude. The audit evaluates recorded behavior actions reserved from critic fitting. On matched data from three MuJoCo systems, bootstrap ensembles produce greater rotation than shared-data ensembles. The median increase exceeds 6.5° in HalfCheetah, while median parallel scales and norm ratios remain near one. Across independently seeded fits to the same data, mean gradients agree more closely than penalized gradients in every studied group. The penalized direction also has a nonpositive median advantage over the mean direction in 71 of 72 finite-step comparisons on independently fitted evaluators. These findings show that a penalty can substantially rotate the local cost gradient without yielding a direction that agrees better across fits. Evaluating both scalar estimates and their derivatives makes this distinction visible.

Primary area:
reinforcement learning
Keywords:
safe reinforcement learning, ensemble methods, uncertainty estimation, action gradients, gradient geometry, critic evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vB7HOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how ensemble uncertainty penalties affect action-gradient geometry in safe RL. It decomposes penalized cost gradients into parallel and orthogonal components and evaluates gradient rotation, scaling, and cross-fit consistency. Experiments on three MuJoCo environments show that bootstrap ensembles induce larger gradient rotations, while penalized gradients generally exhibit weaker agreement and local transfer across independently fitted critics.

Critical strengths
  1. The paper provides a clear geometric characterization of how uncertainty penalties affect action-gradient direction and magnitude.
  2. The matched experimental design and independently fitted evaluators offer a controlled assessment of gradient stability and local transfer.
Critical weaknesses
  1. The theoretical analysis primarily relies on standard vector decomposition, and the proposed audit introduces neither a new learning algorithm nor a substantial theoretical result.
  2. Experiments cover only three MuJoCo environments, two synthetic costs, and one critic architecture. More importantly, the evaluation relies on independently fitted critics rather than ground-truth cost reduction or actual policy performance, limiting the practical significance of the findings for safe RL.
Questions for authors

How does the observed cross-fit gradient agreement relate to actual cost-gradient accuracy or policy safety? Could the authors validate whether the penalized direction achieves better cost reduction using environment rollouts rather than another learned critic?

3 comments

Since submission we have added two experiments: a ten-seed rerun of the full study, and environment rollouts that compare the mean and penalized directions on actual cost.

Table A. Original study and ten-seed rerun.

FindingOriginal, 5 seedsRerun, 10 seeds
Bootstrap rotation increase in HalfCheetah, velocity / energy cost6.53° / 6.73°5.60° / 6.47°
Groups in which the mean gradient is more reproducible across fits18 of 1817 of 18
Comparisons in which the penalized direction has no median finite-step advantage71 of 7272 of 72

Table B. Rollouts on actual cost. D is the discounted actual cost after a step along the mean direction minus that after a step along the penalized direction, so a positive D favors the penalized direction. Values are means over ten seeds with 95% seed-bootstrap intervals, multiplied by 1,000.

EnvironmentConstructionMean D95% interval
HopperShared data-0.36[-0.83, 0.06]
HopperBootstrap-0.44[-1.06, 0.17]
HopperRandomized prior-0.35[-1.11, 0.30]
Walker2dShared data0.21[-0.16, 0.57]
Walker2dBootstrap-0.08[-1.38, 1.36]
Walker2dRandomized prior0.34[-1.02, 2.46]
HalfCheetahShared data-1.96[-9.32, 5.99]
HalfCheetahBootstrap2.10[-10.84, 14.05]
HalfCheetahRandomized prior-4.81[-13.24, 3.01]

W1. The review describes only the decomposition of Section 3.2. The formal results are in Sections 3.1 and 3.3, which it does not mention. Eq. (3) shows that two ensembles can agree on every member prediction, on the spread, and on the mean gradient and still have arbitrary spread gradients, so no scalar uncertainty estimate determines the direction the actor receives. Eq. (10) shows that scoring a penalized direction on its own fit favors it by construction, which is why the audit requires an independent evaluator. Together they expose an effect that no existing check detects: in HalfCheetah, most actions have their cost gradient rotated by more than 10° while its norm changes by less than 5%.

W2. The narrow design is deliberate. A cost action gradient is one input to an actor update, and a training run changes the reward gradient, the multiplier, the policy Jacobian, and the visitation distribution at the same time, so it cannot attribute an outcome to the penalty. Matched data makes the effect attributable, and the ten-seed rerun confirms it: bootstrap rotates the gradient more than shared data in 59 of 60 paired comparisons.

Q1. Cross-fit agreement is not presented as gradient accuracy or policy safety. It measures whether a direction survives refitting on the same data, and on that measure the mean gradient is more reproducible than the penalized gradient in 18 of 18 groups.

Q2. We ran this test. From identical restored simulator states, we take one step along each direction, keep all later actions identical, and compare the discounted actual cost. The penalized direction shows no cost-reduction advantage in any of the nine comparisons in Table B, and six of the nine point estimates favor the mean direction.

Rating. The weaknesses listed concern novelty and scope. The review identifies no error in the derivations or the measurements, and its own strengths describe the design as a controlled assessment. The rollout test it asked for has now been run, so we ask that the overall rating be reconsidered.

Thank you for the detailed response and the additional experiments. I appreciate the ten-seed rerun and the new environment rollout evaluation.

However, my main concern remains the practical significance of the findings. The paper focuses on local action-gradient geometry, but provides no direct evaluation of policy learning performance or safety outcomes. While the new rollout experiments are a useful addition, all reported confidence intervals include zero, making it difficult to draw meaningful conclusions about the practical consequences of the observed gradient differences.

I understand the motivation for isolating the effect of uncertainty penalties through a controlled experimental design. However, without connecting these diagnostic metrics to actual policy optimization or safety performance, it remains unclear how important the findings are for practical safe RL.

I acknowledge the additional theoretical arguments highlighted by the authors, but I still find the overall contribution limited in scope and practical validation. Therefore, I maintain my current rating.

Thank you for clarifying the remaining concern. We understand that your assessment now centers on the significance of the diagnostic contribution and its connection to policy-level outcomes.

The specific contribution we ask to be considered is that conservative scalar predictions and near-unit gradient norms can conceal a construction-dependent change in the optimization direction. The matched experiments and the ten-seed rerun establish this behavior, and independent refitting shows that the changed direction is less reproducible across fitted critics.

The simulator comparisons address the local environment validation you requested. They show no advantage for the penalized direction in any of the nine comparisons, and we do not interpret them as establishing equivalence, superiority, or improved policy safety.

We agree that policy-level utility is a distinct empirical question. We kept it separate because a training run changes the reward gradient, the multiplier, the policy Jacobian, and the visitation distribution together, which would prevent attributing an outcome to the penalty. The paper's contribution is evidence about the optimization signal that safe-RL methods use, and we see the remaining difference as one of scope rather than of correctness.

Reviewer_tRPJOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a diagnostic that measures how adding an ensemble-disagreement penalty to a safe-RL cost estimate changes the direction of the action gradient, finding on three MuJoCo tasks that the penalty mainly rotates the gradient, and that the rotated direction is less consistent across independently retrained networks than the unpenalised one.

Critical strengths
  • Good experimental design (though it is limited), and measurements are defined clearly. Comparison of ensemble types appears to be fair, as the same data and seeds are used.
  • Many robustness checks and an honest limitations section.
Critical weaknesses
  • The paper is hard to follow for non-specialist readers. It does not clearly state a real problem or failure with current methods that the paper attempts to address. The contributions of the paper are also not clear and appear to be not very significant.
  • No practical impact is shown. No agent is trained, so it’s unclear whether the rotation affects safety or performance. The authors acknowledge this in the limitations section. Findings are interesting, but their impact (what do they mean in practice) is not obviously clear.
  • The setup is limited: random behaviour data, small networks, synthetic costs, three environments, and five seeds. It’s unclear whether the findings carry over to real constrained RL training.
Questions for authors

I would appreciate further clarification on the weaknesses I raised above.

1 comment

Since submission we have added two experiments: a ten-seed rerun of the full study, and environment rollouts that compare the mean and penalized directions on actual cost.

Table A. Original study and ten-seed rerun.

FindingOriginal, 5 seedsRerun, 10 seeds
Bootstrap rotation increase in HalfCheetah, velocity / energy cost6.53° / 6.73°5.60° / 6.47°
Groups in which the mean gradient is more reproducible across fits18 of 1817 of 18
Comparisons in which the penalized direction has no median finite-step advantage71 of 7272 of 72

W1. The penalty is added to make a cost estimate more cautious, but the actor uses the slope of that estimate with respect to the action, and the penalty turns that slope while leaving its length almost unchanged. In HalfCheetah, most actions are rotated by more than 10° while the gradient norm changes by less than 5%, so inspecting predicted cost or gradient magnitude reveals nothing, and Section 3.1 proves that scalar predictions never can.

W2. No agent is trained by design, because a training run changes the reward gradient, the multiplier, and the policy Jacobian together and cannot attribute an outcome to the penalty. The practical consequence is measured directly instead. The rotated direction is less reproducible across refits in 18 of 18 groups, and in new simulator rollouts on actual cost it shows no advantage over the mean direction in any of nine comparisons. The penalty therefore redirects the actor without a measurable benefit, which matters to anyone who uses it.

W3. The setup is a matched comparison: every construction sees the same data, targets, and evaluated actions, which is what allows a difference in direction to be attributed to ensemble construction. The seed concern is resolved by the ten-seed rerun, in which bootstrap rotates more than shared data in 59 of 60 paired comparisons.

Reviewer_e4XFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the mean-plus spread cost penalty widely used in safe RL methods. While the cost penalty U is what's being calculated, what's actually consumed by the models is the gradient of U. The paper explores the question of whether the penalty only rescales the gradient, or also rotates it.

The paper does not propose an algorithm, but rather an analysis. It analyzes spread derivative, geometric decomposition and cross-fit transfer. The experiments use a five-member ensemble of MLPs on three MuJuCo enviroments. The findings of the paper are:

  1. Bootstrap increases rotation
  2. mean gradients are in agreement more than penalized gradients
  3. The penalized direction has nonpostive median finite-step advantage in 71 out of 72 comparisons.

The main (and only) contribution claimed by the paper is linking ensemble construction to the cost derivative.

Critical strengths
  1. An interesting and relevant question - safe RL is an important field with many applications. A better understanding of the underlying processes is always welcome

  2. A well-executed evaluation - shred buffers and paired seeds, reporting the specific seeds, sensitivity checks

Critical weaknesses

Most of the weaknesses I mention below have a common thread: the paper analyzes an interesting aspect of safe RL, but it stops at the analysis. There are no deeper experiments, no recommendations, and the analysis is performed on three "toy" problems. From a paper with limited novelty (this is not a criticism, just an observation), I would expect a much more robust experimental setup.

  1. No link to the problem the penalty is meant to solve - the aithors don't train any policies, and no RL algorithms are run. Returns, constraints, or safety outcomes are also not reported. The paper shows that gradients rotate, but does not show what are the implications.

  2. No actionable recommendations - the observation is presented fairly well, but what does one do with it?

  3. Small evaluation set - there are only three (very small) locomotion tasks, all from MuJuCo. This is limited to begin with, but is exacerbated by showing most examples on Cheetah. All environments also only use smooth dynamics.

  4. No ground truth - while imperfect, the simulator could have been used to determine whether using the appraoch leads to improvement in performance. This was not done.

  5. Lack of statistical rigor - reporting statistical significance tests, or at least standard deviations, would have helped to increase confidence in the results. Five seeds is not a lot, and the variance is not reported.

Questions for authors

I invite the authors to address my concerns above.

2 comments

Since submission we have added two experiments: a ten-seed rerun of the full study, and environment rollouts that compare the mean and penalized directions on actual cost.

Table A. Original study and ten-seed rerun.

FindingOriginal, 5 seedsRerun, 10 seeds
Bootstrap rotation increase in HalfCheetah, velocity / energy cost6.53° / 6.73°5.60° / 6.47°
Groups in which the mean gradient is more reproducible across fits18 of 1817 of 18
Comparisons in which the penalized direction has no median finite-step advantage71 of 7272 of 72

Table B. Rollouts on actual cost. D is the discounted actual cost after a step along the mean direction minus that after a step along the penalized direction, so a positive D favors the penalized direction. Values are means over ten seeds with 95% seed-bootstrap intervals, multiplied by 1,000.

EnvironmentConstructionMean D95% interval
HopperShared data-0.36[-0.83, 0.06]
HopperBootstrap-0.44[-1.06, 0.17]
HopperRandomized prior-0.35[-1.11, 0.30]
Walker2dShared data0.21[-0.16, 0.57]
Walker2dBootstrap-0.08[-1.38, 1.36]
Walker2dRandomized prior0.34[-1.02, 2.46]
HalfCheetahShared data-1.96[-9.32, 5.99]
HalfCheetahBootstrap2.10[-10.84, 14.05]
HalfCheetahRandomized prior-4.81[-13.24, 3.01]

W1. The implication is stated in Section 6, and the new rollouts test it on the simulator. The penalty moves the actor's cost direction by a margin that magnitude checks cannot see, the moved direction is less reproducible across refits, and it reduces actual cost no better than the mean direction. No policy is trained by design, because a training run changes the reward gradient, the multiplier, the policy Jacobian, and the visitation distribution together and cannot isolate the penalty.

W2. Section 6 states the use, and it amounts to four recommendations.

  1. Check the rotation angle, because predicted cost and gradient norm do not reveal it.
  2. Treat ensemble construction as a design choice that changes direction: bootstrap rotates more than shared data in every environment and cost.
  3. Report the angle when tuning the penalty weight or selecting members, since both change it.
  4. Evaluate a penalized direction on an independently fitted critic, because scoring it on its own fit favors it by construction.

W3. The paper does not show most examples on HalfCheetah. Tables 1 and 2 and Figures 2 to 4 give every environment equal space, and the text highlights HalfCheetah because its effect is the largest. The dynamics are not smooth either: Hopper and Walker2d involve ground contact and terminate on falling. The construction effect is positive in all six environment and cost groups in both studies.

W4. We used the simulator to test whether the penalized direction reduces actual cost more than the mean direction. It does not in any of the nine comparisons in Table B.

W5. Variance is reported in the submission: Figures 2 to 4 plot every seed, and Table 2 gives the minimum and maximum across seeds for every entry. On the seed count, the ten-seed rerun gives a positive bootstrap effect in 59 of 60 paired comparisons, and a post hoc exact sign test with Holm correction is significant in all six environment and cost groups, with adjusted p of at most 0.021. The seed SD of the HalfCheetah effect is about 0.5° in the original study and 1.0° in the rerun, against medians of at least 5.6°.

Reviewer_e4XFOpenReview ↗

Response to Authors

I thank the authors for their response and the additional experimental results. While the new results do make the experiments more robust, my main concern was not addressed. Without actionable (and proven) recommendations how to act on the paper's insights, it is unclear whether one can (or should) do regarding this insight. Should practitioners aim for more or less rotation, for example?

My concern about statistical rigor was mostly addressed, but important details (like the rollout method) are still not provided. I also think that the authors could have done more about the training policy with ablations.

To conclude, I am maintaining my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.