acceptodds
Under review as a conference paper at ICLR 2027

WONI: Weight-Space Orbit structured Natural Gradient for Scalable Policy Optimization via Inverse-Fisher Preconditioner

Abstract

In deep RL with multi-objective and constrained settings, the natural gradient provides a principled way to handle the correlations among gradients within the policy geometry, while its computation, which grows with the number of gradients, remains a bottleneck. In this paper, we study curvature approximations for inverting the Fisher information matrix, which can be applied to a set of gradients to obtain multiple natural gradients efficiently. We propose a curvature approximation structured by the weight-space symmetry of the policy network and calibrated with its spectrum estimated from samples. Empirical results show that applying the proposed method in place of the conjugate gradient method used in various second-order algorithms achieves comparable or better performance with substantially less computation.

Primary area:
reinforcement learning
Keywords:
weight space, symmetry, multi-objective reinforcement learning, curvature

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Qp23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WONI, a curvature approximation method for computing natural policy gradients in settings with multiple objectives and constraints. Using weight-space symmetry, the FIM is averaged over symmetry-equivalent parameterizations of the policy network. WONI then replaces the identity and projection matrices induced by this averaging with weighted second moments to recover part of the lost anisotropy. For Gaussian policies with a linear mean head and state-independent standard deviations, it uses an exact output-layer Fisher block. The resulting block-diagonal curvature approximation combines Kronecker-factored blocks with full-moment blocks. In the single-objective experiments, WONI is compared with HF and K-FAC and its variants. In the multi-objective settings, it is compared with the other curvature approximations on the constrained legged-locomotion tasks, and with Euclidean updates on the MO-Gymnasium tasks.

Critical strengths
  1. The derivation is clearand well presented. Extending weight-space symmetry from the Hessian to the policy Fisher is a useful idea. The derivation is well written and easy to follow.

  2. Clear computational advantage over HF in multi-objective settings. In the constrained multi-objective experiments on Laikago and Cassie, WONI substantially reduces the reported GFLOPs while maintaining competitive performance compared with HF.

Critical weaknesses
  1. The empirical impact of the differences between WONI and TKFAC is unclear. WONI’s curvature approximation is closely related to TKFAC in certain settings, but the effects of their differences have not been isolated experimentally. In particular, WONI and TKFAC use different damping schemes, although the specific damping formulation does not appear to be required by WONI’s curvature derivation. Controlled comparisons under matched damping schemes would help clarify whether the observed performance differences arise from the curvature approximation itself.

  2. The advantage of WONI in multi-objective settings remains unclear. K-FAC and its variants can also estimate curvature once per update and reuse it across multiple gradients. Table 4 reports the same computational cost for WONI and K-FAC/TKFAC under signed permutations, and additional costs for WONI under ordinary permutations. In multi-objective settings, WONI is compared with K-FAC and its variants on only Laikago and Cassie. In MO-Gymnasium tasks, WONI is compared only with Euclidean updates. Including these curvature-based baselines in the six MO-Gymnasium tasks would provide stronger evidence of WONI’s specific benefits in multi-objective optimization.

Questions for authors
  1. What advantages does WONI’s curvature approximation have over TKFAC, and do its additional components improve approximation accuracy?

  2. What specific advantages does WONI offer over K-FAC methods in multi-objective tasks?

5 comments

We thank the reviewer for the clear summary and the helpful comments on the relation to TKFAC and the multi-objective comparisons. Our responses to each point are given below.

W1/Q1: Relation to TKFAC and component wise ablation

We agree that WONI's weight blocks are algebraically equivalent to TKFAC under signed permutations, and that the two methods coincide when bias handling, Gaussian output blocks, and damping are treated identically. The fundamental difference arises under the permutation group. We therefore conducted an ablation study with single and multi constraint tasks, that starts from the original TKFAC and adds one component at a time: (1) the Gaussian output block, (2) additive damping, and (3) the permutation-group decomposition, which gives WONI.

For the single-objective setting, we report the IQM normalized by TKFAC following [1]. For the multi-constrained setting on Cassie with EMA, we follow [2] and report the updates required to satisfy all constraints, together with the Fisher cosine between the update direction and the reference direction obtained by solving the QP with the natural gradients computed by HF until the residual fell below 10−810^{-8}, before and after all constraints are first satisfied.

The first table shows that the IQM increases as the components are progressively added to the original TKFAC, with WONI achieving the highest IQM in both setting. Although additive damping gives the largest increase among the three components, (3) consistently raises the IQM further. In second table, the fisher cosine similarity also improves as components are added. Compared with +(1,2), WONI shows a higher similarity before satisfied and a lower one afterward. Before the satisfaction, the QP combines multiple violated constraints, whereas afterward, the update involves only one or two constraints or the reward alone. This indicates WONI captures the relations among multiple gradients better, which is also reflected in the fewer updates required to satisfy all constraints .

MethodBaseEMA
TKFAC (original)0.963 [0.872, 1.068]0.948 [0.859, 1.057]
TKFAC + (1)0.993 [0.905, 1.102]0.955 [0.862, 1.070]
TKFAC + (1, 2)1.036 [0.921, 1.154]1.097 [0.953, 1.276]
TKFAC + (1, 2, 3) (WONI)1.088 [0.975, 1.203]1.116 [1.003, 1.249]
MethodBeforeAfterRequired updates
TKFAC (original)0.578 ± 0.1900.498 ± 0.0931716 ± 95 (3)
TKFAC + (1)0.578 ± 0.1900.498 ± 0.0931726 ± 95 (2)
TKFAC + (1, 2)0.576 ± 0.1930.534 ± 0.0941478 ± 89 (5)
TKFAC + (1, 2, 3) (WONI)0.584 ± 0.1890.526 ± 0.0921421 ± 196 (5)

Additionally shown in Table 8 of Appendix C.1, WONI achieves the highest IQM under the permutation group beyond TKFAC among all methods with and without the Gaussian output block (1), and as EKFAC and TEKFAC also use additive damping, the gap to EKFAC and TEKFAC reflects the contribution of the permutation-group decomposition.

W2/Q2: Advantages over K-FAC methods in multi-objective tasks

In the evaluation of the constrained multi-objective setting, the distillation of the preference-conditioned policy obscured the differences among the updates, despite the same constraints as in the multi-constrained setting. In the multi-constrained setting on the same tasks, where each update is applied to the policy directly, the differences appear in the satisfaction of the constraints. On Cassie, WONI, EKFAC, and TEKFAC satisfy all constraints in every seed in both modes, whereas KFAC satisfies them in four and one of the five seeds and TKFAC in five and three of the five seeds. WONI reaches this level of constraint satisfaction with a slightly lower cost than EKFAC and TEKFAC (shown in Table 2).

We also agree that further comparisons on the MO-Gymnasium tasks would provide stronger evidence for the benefits of WONI, and we are currently running the additional experiments As noted in our response to Reviewer G1Aa (Q2), a fair comparison requires measuring the step size with the exact FIM, which is costly in CoMOGA as it computes about 2 to 3 million natural gradients over a run of one million steps. Given our limited computational resources, these experiments require additional time, and we kindly ask for the reviewer's understanding.

Correction. We found that the TRPO experiments with width 64 used a critic learning rate of 0.001, which differed from the value of 0.0003 used in the other settings. We reran these experiments with the same critic learning rate as the other settings and updated the corresponding tables in the revised paper. The correction does not change the main conclusions.

[1] Agarwal, et al. "Deep reinforcement learning at the edge of the statistical precipice." Neurips 2021. [2] Kim, et al. "Trust region-based safe distributional reinforcement learning for multiple constraints." Neurips 2023.

Thank you for the detailed response and the new ablation. It clarifies how WONI relates to TKFAC, and I appreciate the correction of the critic learning rate. However, the ablation confirms my concern: additive damping gives the largest gain, while the increment from the permutation group decomposition lies within the reported uncertainty. On Cassie, EKFAC and TEKFAC also satisfy all constraints in every seed, and the MO-Gymnasium comparisons are still pending. The advantage of the orbit structure over existing K-FAC variants therefore remains unclear, so I will keep my current rating for now.

Thank you for your further assessment. We have completed additional comparisons with K-FAC and its variants on MO-Gymnasium. To ensure a fair comparison, we applied the same Gaussian block processing (1) to all methods, minimizing potential performance differences arising from implementation choices. The table below summarizes the results using average ranks and IQM normalized by K-FAC performance, while full results provided in Table 9 of Appendix C.1. Across six tasks, WONI achieves the best average rank in HV and the joint-best average rank in SP, as well as the best aggregate IQMs for both metrics (1.078 for HV and 0.717 for SP).

MethodAvg rank (HV) ↓Avg rank (SP) ↓IQM (HV) ↑IQM (SP) ↓
KFAC3.173.500.988 [0.92, 1.04]0.917 [0.80, 1.06]
TKFAC2.833.330.990 [0.93, 1.02]0.946 [0.81, 1.08]
EKFAC3.832.670.982 [0.91, 1.04]0.854 [0.71, 1.01]
TEKFAC3.172.831.067 [1.00, 1.18]0.783 [0.67, 0.95]
WONI2.002.671.078 [0.99, 1.17]0.717 [0.63, 0.88]

These results are also consistent with the single-objective results under the permutation group, shown in the table below and provided in Table 8 of Appendix C.1, where WONI achieves the highest IQM (1.286) regardless of whether Gaussian block processing is applied to EKFAC and TEKFAC, both of which also employ additive damping.

MethodOriginal+ (1)
HF0.984 [0.903, 1.065]0.984 [0.903, 1.065]
KFAC1.183 [1.141, 1.228]1.183 [1.134, 1.233]
TKFAC1.167 [1.124, 1.212]1.180 [1.135, 1.224]
EKFAC1.267 [1.214, 1.320]1.261 [1.213, 1.309]
TEKFAC1.275 [1.227, 1.324]1.260 [1.208, 1.310]
WONI1.286 [1.239, 1.334]1.286 [1.239, 1.334]

Since K-FAC variants and WONI share the same block-diagonal assumption, the achievable improvements from different curvature approximations may be limited, particularly in the two-hidden-layer networks commonly used in RL benchmarks. Moreover, as policy updates continuously change the state distribution and optimization landscape, improvements in curvature estimation may not always translate into clearly distinguishable final performance. Nevertheless, despite the overlapping confidence intervals, we believe that the consistent improvements observed under the permutation group and the favorable aggregate results provide meaningful evidence of WONI's effectiveness.

Beyond empirical performance, we hope the methodological contribution will also be considered. Weight-space symmetry-based curvature estimation is still in its early stages, and to the best of our knowledge, this is its first application to RL. Furthermore, no prior work has investigated efficiently estimating and reusing a common curvature approximation across multiple natural gradients in multi-objective and multi-constraint RL. We believe that demonstrating the feasibility and effectiveness of these approaches provides a valuable foundation for future research.

We respectfully hope these additional results and methodological contributions will be considered in reassessing our work.

Reviewer_Qv4EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes WONI, a curvature approximation for natural policy gradients based on weight-space symmetries of neural networks. The method starts from an orbit-averaged Fisher information matrix (FIM), whose structure is characterized for permutation and signed-permutation symmetries. To recover information lost through orbit averaging, WONI replaces the resulting isotropic components with weighted second-moment estimates, leading to a block-structured approximation that can be efficiently inverted and shared across multiple gradients. The method is evaluated on single-objective, multi-constrained, and multi-objective reinforcement learning tasks against Hessian-free natural gradients and several K-FAC variants. The results suggest that WONI can achieve competitive policy performance while substantially reducing the computational cost relative to Hessian-free methods.

Critical strengths
  • The paper addresses an important computational bottleneck in natural-gradient methods, particularly when several objectives or constraints require distinct gradient computations.

  • The use of weight-space symmetries to motivate the curvature structure is interesting. The derivation of the orbit-averaged FIM and its characterization under different symmetry groups provide a principled starting point for constructing the approximation.

  • The experimental evaluation covers several settings and compares WONI against relevant curvature approximations, including K-FAC, EKFAC, TKFAC, and TEKFAC. The computational comparisons also indicate substantial savings relative to Hessian-free methods.

Critical weaknesses
  • Limited distinction from existing curvature approximations. The relationship between WONI and existing Kronecker-factored methods needs to be clarified. In particular, Section 3.2 explicitly states that, under signed permutations, the resulting blocks coincide with TKFAC blocks computed from the same vectors. This raises questions about the novelty of the resulting approximation and which components are responsible for the observed differences in performance.

  • Limited theoretical justification for the final approximation. Lemmas 1 and 2 characterize the orbit-averaged FIM, but WONI subsequently replaces its identity and projection matrices with weighted second moments. These replacements generally no longer preserve the exact orbit-invariant structure. While the resulting approximation is computationally convenient, the paper provides no quantitative guarantee on its accuracy relative to the original FIM, nor on the quality of the resulting natural-gradient direction. A clearer distinction between properties of the orbit average and those of the final estimator would strengthen the theoretical contribution.

  • The empirical improvements are not consistently significant. In Table 1, WONI achieves the highest mean return in only three of the eight configurations. In Table 2, other curvature approximations achieve higher hypervolume on both constrained multi-objective tasks. Although WONI performs favorably on several metrics, the results do not establish a systematic advantage over existing K-FAC variants. A more detailed discussion of the settings in which WONI is preferable would be useful.

  • Insufficient evidence for the claimed scalability. A central motivation is the ability to reuse the same curvature approximation for multiple objectives and constraints. However, the experiments do not systematically vary the number of gradients to quantify the corresponding computational savings. Moreover, while WONI is considerably cheaper than HF in Table 2, its reported GFLOPs are higher than those of several K-FAC variants. An explicit complexity analysis and experiments showing computation time as a function of the number of objectives would better support the scalability claims.

  • Missing ablation studies. The paper does not clearly isolate the contributions of orbit averaging, weighted second moments, and the choice of symmetry group. In particular, comparing the final estimator against the unmodified orbit-averaged approximation would help establish whether the symmetry-derived structure itself provides an advantage. The sensitivity to damping and curvature-estimation choices also deserves investigation.

Questions for authors
  • Since the signed-permutation version of WONI yields the same blocks as TKFAC when computed from the same vectors, what is the precise methodological difference between the two approaches? Can the authors isolate this difference experimentally?

  • Can the authors characterize the approximation error introduced by replacing the orbit-averaged identity and projection matrices with weighted second moments? Are there conditions under which the resulting preconditioner approximates the true natural-gradient direction accurately?

  • How does the computational cost scale with the number of objectives or constraints? In particular, can the authors provide wall-clock comparisons against K-FAC variants and HF as this number increases?

  • What explains the relatively weaker performance of WONI on certain single-objective tasks, and how sensitive are the results to the damping parameter and the curvature estimation procedure?

3 comments

Thank you for the careful reading of our paper and the valuable suggestions. We apologize as the additional experiments took longer than expected.l. We respond to each point below.

W1,5/Q1: Relation to TKFAC and component wise ablation We agree that the weight blocks of WONI are algebraically equivalent to TKFAC under signed permutations, and that the two methods coincide when the bias handling, the Gaussian output block, and the damping are treated identically. We have added the corresponding ablation study to Appendix C.1 of the revised paper, and a brief summary of the results is provided in our response to Reviewer Qp23.

For the choice of symmetry group, Appendix C.1 (Table 8) reports the IQM of the HF-normalized return separately for the signed permutation and permutation group. while for the contributions of orbit averaging and weighted second moments, Appendix C.2 (Table 9) compares the unmodified orbit-averaged FIM with WONI under the same setting as Table 1, where WONI shows higher return except the two rows of LunarLander-v3.

W3/Q4* Settings where WONI is preferable or weaker Appendix C.1 (Table 8) reports the IQM of the HF-normalized return for each symmetry group [1]. Under the permutation group, WONI shows the highest IQM, with its confidence interval above those of HF, KFAC, and TKFAC. Under the signed permutation group, the K-FAC variants and WONI show similar values within their confidence intervals, all above HF.

The weaker result of WONI on LunarLander-v3 in Table 1 is explained in the last paragraph of Appendix C.2. On this task, a batch of 2048 steps contains only four to five episodes in the later stage of training, and the FIM computed on so few states underestimates the curvature over the state distribution of the policy. A method that follows the curvature of the batch more closely along each direction suffers more: KFAC and TKFAC add the damping to each Kronecker factor and follow the batch curvature less closely, which is consistent with their advantage on this task in the Exact mode, and the EMA mode reduces this dependence by accumulating the curvature over the batches.

W2/Q2 Replacing the orbit-averaged identity

Lemmas 1 and 2 characterize the orbit-averaged FIM F⋆F^\star, whereas the final estimator departs from the orbit-invariant structure by replacing the identity and projection matrices with the weighted second moments. Under the signed permutation group, the weighted second moments are the partial traces of the true block Fℓℓ=E[aℓaℓ⊤⊗δℓδℓ⊤]F_{\ell\ell}=\mathbb E[a_\ell a_\ell^\top\otimes\delta_\ell\delta_\ell^\top], and the WONI block becomes

F^=Tr(2)(F)⊗Tr(1)(F)tr F,\hat F=\frac{\mathrm{Tr}^{(2)}(F)\otimes\mathrm{Tr}^{(1)}(F)}{\mathrm{tr}\,F},

The WONI block therefore matches both partial traces of the true block and recovers the block exactly whenever the block is a single Kronecker product A⊗GA\otimes G. Unlike K-FAC, which requires the independence between aℓa_\ell and δℓ\delta_\ell for this exactness, the property holds without the assumption, as also in TKFAC under the signed permutation group. The same holds for each pair of zero-mean components under the permutation group. As these properties do not bound the error for arbitrary blocks, we measured the relative block error to the exact FIM and the Fisher cosine to the exact natural gradient on the two-layer actors of the single-objective setting, compared with F⋆F^\star.

ActivationBlock error (W1W_1 / W2W_2), F⋆F^\starBlock error (W1W_1 / W2W_2), WONIFisher cosine, F⋆F^\starFisher cosine, WONI
Tanh0.993 / 1.0000.345 / 0.3300.1040.643
ReLU0.986 / 0.9970.458 / 0.6420.0960.597
LeakyReLU0.986 / 0.9970.439 / 0.6330.0890.600

As the orbit average is isotropic on the hidden side of each block, F⋆F^\star gives block errors close to one and Fisher cosines of about 0.1, whereas the weighted second moments reduce the error to 0.33–0.64 and raise the Fisher cosine to 0.60–0.64.

Q3: Computational cost scale with the number of gradients We have added the comparison of the FLOPs and the wall-clock time to Section 4.4 of the revised paper. The cost of HF grows with every additional gradient, and with 16 gradients HF takes 56 to 122 times the wall-clock time of WONI. The once-per-update cost of WONI is comparable to KFAC and TKFAC under signed permutations and to EKFAC and TEKFAC under permutations, while its per-gradient cost matches EKFAC and TEKFAC.

Correction. We found that the TRPO experiments with width 64 used a critic learning rate of 0.001, which differed from the value of 0.0003 used in the other settings. We reran these experiments with the same critic learning rate as the other settings and updated the corresponding tables in the revised paper. The correction does not change the main conclusions.

I thank the authors for their detailed response and the additional experiments. I appreciate the effort they have made to address my comments. In particular, the new ablation studies and the computational comparisons substantially improve the paper and address most of my initial concerns regarding the experiments.

My main remaining concern is the novelty of the proposed approximation relative to existing methods. As clarified in Appendix B.3, WONI coincides with TKFAC at the weight-block level under signed permutations. The new ablation studies in Appendix C.1 are helpful in understanding the differences between the methods. However, they suggest that most of the improvement over TKFAC comes from the choice of damping, while the additional benefit of the permutation-group decomposition is relatively modest. Moreover, WONI achieves performance comparable to EKFAC and TEKFAC, without a clear systematic advantage.

I also appreciate the new computational experiments, which convincingly demonstrate substantial savings over HF. However, these benefits are largely shared by existing K-FAC variants. I thus remain unconvinced that the proposed approximation represents a sufficiently significant methodological advance over existing approaches. I therefore maintain my initial rating of weak rejection.

Thank you for your further assessment and for acknowledging the additional experiments. We understand your remaining concern regarding the methodological novelty of WONI relative to existing K-FAC variants.

As clarified in Appendix B.3, WONI coincides with TKFAC at the weight-block level under signed permutations. However, we would like to emphasize that the contribution of WONI extends beyond this particular setting. To further distinguish the effects of curvature approximation from implementation choices, we conducted additional comparisons under the permutation group, applying the same Gaussian block processing to EKFAC and TEKFAC, both of which also employ additive damping. As reported in Table 8 of Appendix C.1, WONI achieves the highest IQM (1.286) regardless of this implementation choice.

We also extended the evaluation to six multi-objective tasks in MO-Gymnasium, applying the same Gaussian block processing to all methods for a fair comparison. WONI achieves the best average rank in HV, the joint-best average rank in SP, and the best aggregate IQMs for both metrics (1.078 for HV and 0.717 for SP), as reported in Table 9 of Appendix C.1. Although the confidence intervals overlap, we believe these consistent improvements under the permutation group and favorable aggregate results provide additional evidence that WONI's performance is not solely attributable to additive damping.

(A summary of both results is provided in our second response to Reviewer Qp23.)

Regarding methodological novelty, weight-space symmetry-based curvature estimation is still in its early stages, and to the best of our knowledge, WONI represents its first application to RL. Furthermore, to the best of our knowledge, no prior work has investigated efficiently estimating and reusing a common curvature approximation across multiple natural gradients in multi-objective and multi-constraint RL. We believe these contributions extend beyond computational savings shared with existing K-FAC variants and provide a valuable foundation for further research.

We respectfully hope these additional results and clarifications will help address your remaining concerns regarding the significance of our methodological contribution.

Reviewer_G1AaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WONI, a structured approximation to the inverse Fisher information matrix for efficient natural-gradient computation in deep reinforcement learning, particularly in multi-objective and multi-constrained settings. The method leverages weight-space symmetries of neural networks to derive orbit-averaged Fisher structures under permutation and signed-permutation groups. It then replaces the isotropic components of these structures with weighted second moments to retain anisotropic curvature information. The resulting approximation allows multiple natural gradients to be computed efficiently using shared matrix decompositions. Experiments on single-objective TRPO, multi-constrained robotic locomotion, and constrained multi-objective RL compare WONI against Hessian-free conjugate gradient and several K-FAC variants. The results demonstrate competitive policy performance and substantially lower computational cost than Hessian-free optimization.

Critical strengths
  1. Natural-gradient methods can become computationally expensive when several objectives and constraints require distinct gradient directions. The proposed use of a shared curvature preconditioner is well motivated for multi-objective and safe RL.

  2. The derivation of the Fisher structure under permutation and signed-permutation groups is mathematically interesting. The paper makes an explicit connection between neural-network parameter symmetries and structured Fisher approximations.

Critical weaknesses
  1. The most important concern is the relationship between WONI and existing Kronecker-factored curvature approximations. Appendix B.3 explicitly establishes that, under the signed-permutation group, WONI's weight blocks coincide with those of TKFAC when computed from the same samples. Specifically, consider: $ B=\mathbb E[xx^\top\otimes yy^\top]. $ Define $ X=\mathbb E[||y||^2xx^\top], Y=\mathbb E[||x||^2yy^\top]. $ Then the proposed block becomes $ \hat B_{\mathrm{WONI}} =\frac{X\otimes Y}{\operatorname{tr}(B)} =\hat B_{\mathrm{TKFAC}}. $ Thus, an important part of the proposed curvature construction is algebraically equivalent to an existing method. The paper does identify differences in bias handling, Gaussian output blocks, damping, and permutation-group component decompositions. However, the relative contribution of these differences is not experimentally isolated. A targeted ablation study is needed to determine which components account for the improvement. This concern is particularly important because the paper presents symmetry-structured curvature approximation as its primary technical contribution.

  2. The explanation for HF's inferior performance is not sufficiently validated. The authors argue that different gradients generate different Krylov subspaces and thus incur inconsistent approximation errors, which adversely affect the quadratic program used to combine constraint gradients. This explanation is plausible, but it is not directly verified by the presented experiments. According to Appendix C, HF uses a fixed number of CG iterations: 15 for the single-objective TRPO experiments and 10 for the multi-constrained and multi-objective experiments. However, the paper does not establish whether these iteration budgets are sufficient for accurate convergence of the linear solves. Increasing the CG iteration budget could potentially improve both natural-gradient accuracy and constraint handling, although at additional computational cost. Thus, the observed performance difference may partly reflect the selected approximation budget rather than an inherent disadvantage of HF.

Questions for authors
  1. Since the signed-permutation weight blocks are identical to TKFAC, can the authors quantify the separate contributions of bias treatment, output-layer handling, damping, and the permutation-group decomposition? In particular, does WONI retain a meaningful advantage when these implementation choices are matched?

  2. Can the authors clarify the mixed multi-objective results? In particular, why does WONI underperform Euclidean updates on Ant and HalfCheetah in terms of HV, and why does it yield much worse mean sparsity on Swimmer? Is this related to the curvature structure, trust-region sizing, or other optimization effects?

2 comments

We sincerely thank the reviewer for the careful review and constructive comments, and we apologize for the delayed response, which was due to the additional experiments. Below, we have done our best to address the reviewer's questions and concerns.

W1/Q1: Relation to TKFAC and component wise ablation We agree that the weight blocks of WONI are algebraically equivalent to TKFAC under signed permutations, and that the two methods coincide when the bias handling, the Gaussian output block, and the damping are treated identically. We have added the corresponding ablation study to Appendix C.1 of the revised paper, and a brief summary of the results is provided in our response to Reviewer Qp23.

W2 We agree that the relation between the performance of HF and its iteration budget needs to be clarified, while we would like to note that the budgets of 15 and 10 iterations follow the default values in the original implementations of the RL algorithms. To examine the effect of the iteration budget, we measured the accuracy of the natural gradients with 10, 30, and 50 iterations in the multi-constrained setting while conducting the ablation study in Appendix C.1 , as well as the returns of HF trained with 15, 30, and 50 iterations in the single-objective setting.

The first table shows the Fisher cosine between the natural gradient of each method and the exact natural gradient on Cassie, as the median over training. The accuracy of HF differs across the gradients, and more iterations raise the accuracy of every gradient and narrow the gap, although the gap still exceeds 0.15 with 50 iterations. With different degrees of convergence, HF multiplies the gradients by different approximations of the inverse FIM, which distorts the Fisher inner products in the QP. WONI, in contrast, keeps the relations among the gradients consistent in the QP by applying the same approximation to every gradient, with its error coming mainly from the discarded off-diagonal blocks. Running the iterations to convergence, as done for the reference, could provide accurate updates, although some of the solves required more than 500 iterations. Together with the single-objective results below, full convergence may not be a practical solution for HF.

The second table shows the returns of HF under the permutation group in the single-objective setting. More iterations do not consistently improve the return of HF, which even decreases on LunarLander. On LunarLander, a batch of 2048 steps contains only four to five episodes in the later stage of training, and the FIM computed on the batch underestimates the curvature over the state distribution of the policy. More iterations enlarge the update along the directions of the underestimated curvature, which drives the updated policy beyond the trust region.

MethodRewardCost 0Cost 1Cost 2
HF (10)0.6720.4700.5270.805
HF (30)0.8150.6620.6730.892
HF (50)0.8680.7520.7620.918
WONI0.6320.6690.6480.674
TaskHF (15)HF (30)HF (50)WONI
Hopper2,044 ± 2962,145 ± 2702,342 ± 2832,978 ± 209
LunarLander172 ± 77175 ± 15137 ± 28237 ± 7
HalfCheetah1,101 ± 1781,098 ± 1771,181 ± 1891,466 ± 194
Walker2d934 ± 184958 ± 178909 ± 1651,169 ± 187

Q2 . Mixed multi-objective results

We found that, in our original implementation, WONI measured its step size with its own approximate FIM. We have therefore added WONI with the step size set to meet the KL bound measured with the exact FIM to Table 3 and the related discussion of the revised paper. With the same direction, setting the step size with the exact FIM closes the gap on HalfCheetah, narrows it on Ant, and gives the lowest SP on Swimmer. The remaining gap on Ant comes from a single seed of the Euclidean update with a large HV, as reflected in its standard deviation.

MetricTaskEuclideanWONI (A)WONI (KL)
HVHalfCheetah2.15 ± 0.162.07 ± 0.252.14 ± 0.29
HVAnt3.19 ± 3.992.42 ± 2.942.83 ± 2.11
SPSwimmer28.2 ± 9.135.8 ± 14.026.3 ± 10.9

Correction. We found that the TRPO experiments with width 64 used a critic learning rate of 0.001, which differed from the value of 0.0003 used in the other settings. We reran these experiments with the same critic learning rate as the other settings and updated the corresponding tables in the revised paper. The correction does not change the main conclusions.

Dear Reviewer,

Thank you again for your review. As the discussion period is approaching its end, we would appreciate it if you could take a moment to consider our response and the additional experimental results. We hope these additions address your concerns and would welcome any further feedback you may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.