- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes WONI, a curvature approximation method for computing natural policy gradients in settings with multiple objectives and constraints. Using weight-space symmetry, the FIM is averaged over symmetry-equivalent parameterizations of the policy network. WONI then replaces the identity and projection matrices induced by this averaging with weighted second moments to recover part of the lost anisotropy. For Gaussian policies with a linear mean head and state-independent standard deviations, it uses an exact output-layer Fisher block. The resulting block-diagonal curvature approximation combines Kronecker-factored blocks with full-moment blocks. In the single-objective experiments, WONI is compared with HF and K-FAC and its variants. In the multi-objective settings, it is compared with the other curvature approximations on the constrained legged-locomotion tasks, and with Euclidean updates on the MO-Gymnasium tasks.
Critical strengths
-
The derivation is clearand well presented. Extending weight-space symmetry from the Hessian to the policy Fisher is a useful idea. The derivation is well written and easy to follow.
-
Clear computational advantage over HF in multi-objective settings. In the constrained multi-objective experiments on Laikago and Cassie, WONI substantially reduces the reported GFLOPs while maintaining competitive performance compared with HF.
Critical weaknesses
-
The empirical impact of the differences between WONI and TKFAC is unclear. WONI’s curvature approximation is closely related to TKFAC in certain settings, but the effects of their differences have not been isolated experimentally. In particular, WONI and TKFAC use different damping schemes, although the specific damping formulation does not appear to be required by WONI’s curvature derivation. Controlled comparisons under matched damping schemes would help clarify whether the observed performance differences arise from the curvature approximation itself.
-
The advantage of WONI in multi-objective settings remains unclear. K-FAC and its variants can also estimate curvature once per update and reuse it across multiple gradients. Table 4 reports the same computational cost for WONI and K-FAC/TKFAC under signed permutations, and additional costs for WONI under ordinary permutations. In multi-objective settings, WONI is compared with K-FAC and its variants on only Laikago and Cassie. In MO-Gymnasium tasks, WONI is compared only with Euclidean updates. Including these curvature-based baselines in the six MO-Gymnasium tasks would provide stronger evidence of WONI’s specific benefits in multi-objective optimization.
Questions for authors
-
What advantages does WONI’s curvature approximation have over TKFAC, and do its additional components improve approximation accuracy?
-
What specific advantages does WONI offer over K-FAC methods in multi-objective tasks?
5 comments
Thank you for the detailed response and the new ablation. It clarifies how WONI relates to TKFAC, and I appreciate the correction of the critic learning rate. However, the ablation confirms my concern: additive damping gives the largest gain, while the increment from the permutation group decomposition lies within the reported uncertainty. On Cassie, EKFAC and TEKFAC also satisfy all constraints in every seed, and the MO-Gymnasium comparisons are still pending. The advantage of the orbit structure over existing K-FAC variants therefore remains unclear, so I will keep my current rating for now.
Thank you for your further assessment. We have completed additional comparisons with K-FAC and its variants on MO-Gymnasium. To ensure a fair comparison, we applied the same Gaussian block processing (1) to all methods, minimizing potential performance differences arising from implementation choices. The table below summarizes the results using average ranks and IQM normalized by K-FAC performance, while full results provided in Table 9 of Appendix C.1. Across six tasks, WONI achieves the best average rank in HV and the joint-best average rank in SP, as well as the best aggregate IQMs for both metrics (1.078 for HV and 0.717 for SP).
| Method | Avg rank (HV) ↓ | Avg rank (SP) ↓ | IQM (HV) ↑ | IQM (SP) ↓ |
|---|---|---|---|---|
| KFAC | 3.17 | 3.50 | 0.988 [0.92, 1.04] | 0.917 [0.80, 1.06] |
| TKFAC | 2.83 | 3.33 | 0.990 [0.93, 1.02] | 0.946 [0.81, 1.08] |
| EKFAC | 3.83 | 2.67 | 0.982 [0.91, 1.04] | 0.854 [0.71, 1.01] |
| TEKFAC | 3.17 | 2.83 | 1.067 [1.00, 1.18] | 0.783 [0.67, 0.95] |
| WONI | 2.00 | 2.67 | 1.078 [0.99, 1.17] | 0.717 [0.63, 0.88] |
These results are also consistent with the single-objective results under the permutation group, shown in the table below and provided in Table 8 of Appendix C.1, where WONI achieves the highest IQM (1.286) regardless of whether Gaussian block processing is applied to EKFAC and TEKFAC, both of which also employ additive damping.
| Method | Original | + (1) |
|---|---|---|
| HF | 0.984 [0.903, 1.065] | 0.984 [0.903, 1.065] |
| KFAC | 1.183 [1.141, 1.228] | 1.183 [1.134, 1.233] |
| TKFAC | 1.167 [1.124, 1.212] | 1.180 [1.135, 1.224] |
| EKFAC | 1.267 [1.214, 1.320] | 1.261 [1.213, 1.309] |
| TEKFAC | 1.275 [1.227, 1.324] | 1.260 [1.208, 1.310] |
| WONI | 1.286 [1.239, 1.334] | 1.286 [1.239, 1.334] |
Since K-FAC variants and WONI share the same block-diagonal assumption, the achievable improvements from different curvature approximations may be limited, particularly in the two-hidden-layer networks commonly used in RL benchmarks. Moreover, as policy updates continuously change the state distribution and optimization landscape, improvements in curvature estimation may not always translate into clearly distinguishable final performance. Nevertheless, despite the overlapping confidence intervals, we believe that the consistent improvements observed under the permutation group and the favorable aggregate results provide meaningful evidence of WONI's effectiveness.
Beyond empirical performance, we hope the methodological contribution will also be considered. Weight-space symmetry-based curvature estimation is still in its early stages, and to the best of our knowledge, this is its first application to RL. Furthermore, no prior work has investigated efficiently estimating and reusing a common curvature approximation across multiple natural gradients in multi-objective and multi-constraint RL. We believe that demonstrating the feasibility and effectiveness of these approaches provides a valuable foundation for future research.
We respectfully hope these additional results and methodological contributions will be considered in reassessing our work.
We thank the reviewer for the clear summary and the helpful comments on the relation to TKFAC and the multi-objective comparisons. Our responses to each point are given below.
W1/Q1: Relation to TKFAC and component wise ablation
We agree that WONI's weight blocks are algebraically equivalent to TKFAC under signed permutations, and that the two methods coincide when bias handling, Gaussian output blocks, and damping are treated identically. The fundamental difference arises under the permutation group. We therefore conducted an ablation study with single and multi constraint tasks, that starts from the original TKFAC and adds one component at a time: (1) the Gaussian output block, (2) additive damping, and (3) the permutation-group decomposition, which gives WONI.
For the single-objective setting, we report the IQM normalized by TKFAC following [1]. For the multi-constrained setting on Cassie with EMA, we follow [2] and report the updates required to satisfy all constraints, together with the Fisher cosine between the update direction and the reference direction obtained by solving the QP with the natural gradients computed by HF until the residual fell below 10−8, before and after all constraints are first satisfied.
The first table shows that the IQM increases as the components are progressively added to the original TKFAC, with WONI achieving the highest IQM in both setting. Although additive damping gives the largest increase among the three components, (3) consistently raises the IQM further. In second table, the fisher cosine similarity also improves as components are added. Compared with +(1,2), WONI shows a higher similarity before satisfied and a lower one afterward. Before the satisfaction, the QP combines multiple violated constraints, whereas afterward, the update involves only one or two constraints or the reward alone. This indicates WONI captures the relations among multiple gradients better, which is also reflected in the fewer updates required to satisfy all constraints .
Additionally shown in Table 8 of Appendix C.1, WONI achieves the highest IQM under the permutation group beyond TKFAC among all methods with and without the Gaussian output block (1), and as EKFAC and TEKFAC also use additive damping, the gap to EKFAC and TEKFAC reflects the contribution of the permutation-group decomposition.
W2/Q2: Advantages over K-FAC methods in multi-objective tasks
In the evaluation of the constrained multi-objective setting, the distillation of the preference-conditioned policy obscured the differences among the updates, despite the same constraints as in the multi-constrained setting. In the multi-constrained setting on the same tasks, where each update is applied to the policy directly, the differences appear in the satisfaction of the constraints. On Cassie, WONI, EKFAC, and TEKFAC satisfy all constraints in every seed in both modes, whereas KFAC satisfies them in four and one of the five seeds and TKFAC in five and three of the five seeds. WONI reaches this level of constraint satisfaction with a slightly lower cost than EKFAC and TEKFAC (shown in Table 2).
We also agree that further comparisons on the MO-Gymnasium tasks would provide stronger evidence for the benefits of WONI, and we are currently running the additional experiments As noted in our response to Reviewer G1Aa (Q2), a fair comparison requires measuring the step size with the exact FIM, which is costly in CoMOGA as it computes about 2 to 3 million natural gradients over a run of one million steps. Given our limited computational resources, these experiments require additional time, and we kindly ask for the reviewer's understanding.
Correction. We found that the TRPO experiments with width 64 used a critic learning rate of 0.001, which differed from the value of 0.0003 used in the other settings. We reran these experiments with the same critic learning rate as the other settings and updated the corresponding tables in the revised paper. The correction does not change the main conclusions.
[1] Agarwal, et al. "Deep reinforcement learning at the edge of the statistical precipice." Neurips 2021. [2] Kim, et al. "Trust region-based safe distributional reinforcement learning for multiple constraints." Neurips 2023.