- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
I believe this paper addresses a real-world problem in multi-objective reinforcement learning (MORL): Should the critic directly regress preference-weighted scalar Q-values, or should each objective have its own independent Q-output? Should these outputs share a backbone, or be trained completely separately? The authors compare three designs—SC, MH, and SP—and attempt to explain how they influence the learning of preference-conditioned actors. I agree with this research topic because critic architectures are often treated as the default implementation, yet they can actually alter a policy’s performance under different preferences.
Critical strengths
S1: I particularly appreciate their experiments across five MORL algorithms. SP achieves higher EUM in many environments, which at least demonstrates that the degree of parameter sharing is a factor that cannot be ignored. However, I would like to see more than just which curve is highest in the end; I also want to know exactly which value estimation errors it improves.
S2: I find it insightful that the paper analyzes the contrast between shared value components and the objective separately. For preference-conditioned policies, differences in action value objectives can influence action selection under different preferences; if the critic learns slowly from these differences, the actor may also learn slowly. However, I was not originally convinced that the authors had proven that such errors necessarily lead to a poorer final policy.
Critical weaknesses
W1: I still have reservations about the theoretical novelty. Given a fixed preference , as long as the scalarized is estimated accurately, certain biases in the target components may not necessarily affect strategy improvement for that preference. Theorem 3 shares similarities with the gradient conflicts discussed in PCGrad and CAGrad; Theorem 4, in turn, relies on additional error alignment conditions. I am more interested in knowing: Do these theorems yield predictions that can be tested during training, or do they primarily serve to explain phenomena in hindsight?
W2: I am particularly concerned about the fairness of the experiments. The fact that SP achieves a higher EUM does not automatically imply that the cause is gradient interference from the shared trunk. In particular, GPI-LS and MO-MPO already incorporate cross-policy Q evaluation, objective-specific E-steps, temperature, and KL constraints; if the policy-improvement procedure is modified when switching to SC, it is no longer a pure comparison of critics. EUM is also merely a metric at the policy set level; I would like the authors to directly examine the performance of the policies output by the actor itself after inputting a specific preference.
Questions for authors
Q1: I’d like to see whether MH combined with PCGrad/CAGrad can still keep up with SP when the number of parameters and training steps are matched.
Q2: I would like the authors to clearly specify the critic inputs, TD targets, Q-query methods, and actor-update steps for the SC variants of GPI-LS and MO-MPO.
Q3: I care more about , which evaluates the actor at each input preference, than EUM, which selects the best policy from a set after the fact.
9 comments
| Prediction | Observed |
|---|---|
| Contrast contraction factors 0.9833 / 0.95 / 0.8 (SC/MH/SP) | Measured trajectories match to 10-15 (A.1) |
| One-step split–shared slope =1.5 | 1.4999906 as the probe step → 0 (A.5) |
| Sharing is faster on the common mode | MH/SC 1.8110-4 vs. SP 1.1510-2 (A.3) |
| MH = SP when =0; gap grows with | Confirmed across the sweep (A.4) |
| Tracking-lag ordering under moving targets | Matches the recurrence: 0.209 / 0.072 / 0.018 (A.6) |
In nonlinear critics, Theorem 4's sufficient condition holds in 19 seeded pairs, and SP has lower held-out scalarized MSE in all those 19 experiments. (Appendix D.2).
(d) Relation to Momba (Štafa et al., 2026, Appendix B)
Momba's contribution is a modernized network backbone for MORL. Its Appendix B is an ablation showing that a vector-valued critic with shared heads performs nearly identically to its scalarized critic, so the scalarized version can be used where vector values are needed. It does not evaluate split critics or analyze critic design. Our paper studies the question this ablation leaves open:
- Agreement. End-to-end, MH does not consistently outperform SC in our experiments either (MH higher in 13 of 24 cells).
- Explanation. Objective-wise supervision removes the contrast attenuation of scalarization, but a shared trunk reintroduces cross-objective interference (Theorems 3–5). The two effects can offset each other, which is what Momba's ablation observes.
- The critic design not evaluated in Momba is the one that demonstrates the strongest gains SP has the highest mean EUM in 29 of 30 MuJoCo comparisons across five algorithms (not only Momba), and the advantage extends to the three- and five-objective tasks (Momba and MORL/D on both). Within Momba itself, SP exceeds the native scalarized critic in all six MuJoCo environments, at both 200k and 400k transitions.
We will discuss Momba's Appendix B in the related work, and we will make the relation of Theorem 3 to gradient-conflict analyses explicit.
Response to W2: Isolating the mechanism
We thank the reviewer for this question. (a) First, our mechanism is cross-objective coupling through shared parameters: the updates ct that each shared block receives from the other objectives. Negative gradient inner products are one sign of this coupling. Representation and optimization dynamics are not separate from it: with matched losses and cloned initialization, Theorem 4 shows that the entire MH–SP prediction difference is generated by ct, while actor, replay, and target feedback enter through a separate term st (Appendix C.6). The evidence isolates this progressively:
- Capacity is matched: SP widths are reduced to MH's total critic parameter count.
- Only the trunk is untied: with a frozen actor, identical data and minibatches, and SP cloned from MH, SP has lower held-out contrast RMSE in all 10 configurations (Appendix D.1).
- The effect has the predicted signature: untying helps contrasts far more than the common component (Swimmer: 33–37% vs. 9–10%, Table 8), and sharing is faster on the common mode (Appendix A.3). A generic "split critics are better representations" account predicts neither asymmetry.
- It occurs in real training: negative trunk-gradient interactions appear in 44 of 80 probes from end-to-end checkpoints, and the split update is better in all 44 (Appendix E.1).
We agree that attributing each end-to-end gain to this mechanism remains an inference, and we will state this explicitly.
(b) GPI-LS and MO-MPO. See answer for Q2.
(c) EUM vs. direct responsiveness. We agree that EUM permits selecting the best policy for each preference. Beyond EUM, we report empirical Pareto fronts for every algorithm and environment (Figure 2a; Figures 7, 9, 11, 13, 15, 18–19), which show the distribution and extent of the trade-offs achieved rather than only their average utility. Both are set-level views; however, Appendix D.5 defines the direct metric , which evaluates the actor at each preference without selection. When the policy set is the actor evaluated on W, , EUM, with equality exactly when each preference's own policy is the best for it, so the gap measures preference mismatch.
We computed for Momba at 200k (mean ± SD over three seeds):
| Environment | SC | MH | SP | SP vs. best alternative |
|---|---|---|---|---|
| Ant | 1876.05 ± 931.66 | 1633.32 ± 460.43 | 2337.24 ± 403.79 | +24.6% |
| HalfCheetah | 2398.61 ± 134.30 | 2757.16 ± 83.75 | 2822.97 ± 443.01 | +2.4% |
| Walker2d | 1730.13 ± 56.63 | 1606.11 ± 185.58 | 2147.87 ± 188.02 | +24.1% |
| Hopper | 3240.06 ± 368.68 | 3067.59 ± 496.45 | 3473.72 ± 95.08 | +7.2% |
| Swimmer | 17.88 ± 0.78 | 10.27 ± 5.74 | 18.05 ± 1.02 | +1.0% |
| Humanoid | 3893.74 ± 490.77 | 4865.36 ± 76.44 | 4863.56 ± 94.00 | −0.04% |
is below EUM for every critic, confirming the reviewer's point that retrospective selection matters. SP is the only critic that is best or tied for best in all three environments: it attains the highest mean in Ant and Swimmer and matches MH in Humanoid, where both objective-wise critics clearly exceed SC. In Humanoid, MH and SP also retain 96–97% of their EUM without retrospective selection, versus 79% for SC, consistent with our claim that scalarized supervision weakens preference-dependent learning.
Q1&W1(b): Can an MH critic with PCGrad or CAGrad recover SP's gains at matched parameters and budget?
Our analysis predicts that it cannot recover SP’s gains on the objective contrasts, which is where SP’s advantage arises. Gradient surgery changes the direction of a single shared update; it cannot give shared parameters different values for different objectives. On an objective-contrast error, the objectives' shared-trunk gradients are exactly opposite (g1=-g2): MH's summed update is zero, PCGrad's projections are zero, and CAGrad's average gradient, and hence its update, is zero. All three leave the shared parameters unchanged on that mode, whereas SP moves each objective's copy along its own gradient (rates 1- vs. 1-(1-); Corollary 6).
Controlled check. In the three-action problem of Appendix A.1 (parameter-matched, 20 updates, =0.75, =0.2):
| Critic | Scalarized RMSE ↓ | Decision regret ↓ | Suboptimal preferences ↓ |
|---|---|---|---|
| SP | 0.0054 | 6.110-6 | 0.35% |
| MH | 0.1690 | 0.0141 | 16.76% |
| MH + PCGrad | 0.1690 | 0.0141 | 16.76% |
| MH + CAGrad | 0.1690 | 0.0141 | 16.76% |
To further address your concern, we additionally trained Momba with PCGrad and CAGrad applied to the MH critic's shared trunk, at matched critic parameter count and training budget (three seeds; run separately from Table 2):
| Critic | Ant EUM | Ant Udirect | Swimmer EUM | Swimmer Udirect |
|---|---|---|---|---|
| MH | 3054.13 ± 822.92 | 2278.85 ± 989.67 | 20.76 ± 0.36 | 17.47 ± 1.04 |
| MH + PCGrad | 2283.58 ± 719.17 | 1477.45 ± 431.85 | 21.01 ± 0.91 | 14.61 ± 7.27 |
| MH + CAGrad | 3082.14 ± 868.10 | 2270.55 ± 433.62 | 20.49 ± 0.38 | 15.26 ± 2.77 |
| SP | 3119.89 ± 381.09 | 2381.98 ± 744.14 | 21.11 ± 0.06 | 18.05 ± 1.02 |
Based on the results, we can see that both CAGrad and PCGrad degrades MH in both env. This matches our analysis: surgery can reduce conflict within a shared update, but only separate parameters let objectives move in opposite directions on contrasts. By 200k the remaining margins are small relative to seed variation. We therefore read these results as showing that gradient surgery does not substitute for parameter separation, not as an additional significance claim for SP.
Q2&W2(b): Critic inputs and policy-update equations of the SC variants of GPI-LS and MO-MPO
Thank you for your question. GPI-LS (Appendix D.3). The SC variant is a structured scalar-query critic. "Scalarized" refers to the supervision, not to removing the component-accessible representation that GPI requires.
-
Critic input and output. The network receives [s,a,z], where z indexes the continuation policy (,z). The evaluation utility u is applied at the readout:
-
Scalarized TD target and loss. With a' drawn from the smoothed target policy conditioned on z, the target twin is and No objective-wise losses are added; MH and SP instead regress each component of F on objective-wise TD targets.
-
Actor update (unchanged from the baseline):
-
Generalized policy improvement (unchanged): Because , this is exactly the query that MH and SP evaluate, over the same candidate actions and continuation policies . Cross-policy evaluation is therefore preserved, and the SC–MH comparison changes only the supervision.
We are grateful to the reviewer for their rigorous and constructive review. We address each point below, with new experiments where needed.
Response to W1: Theoretical novelty and implications for policy improvement
We thank the reviewer for this detailed assessment. Our contribution is a two-axis account of MORL critic design that separates what supervision each objective receives from how much of the critic is shared. This question differs from those addressed by gradient-conflict methods and by Momba and focuses on preference-conditioned critics. We address the four points in turn.
(a) Proposition 2 and policy improvement
For a fixed preference, an accurate scalarized value suffices, and masking alone does not establish harm. Our claim concerns how quickly that accuracy is reached under the limited training budgets of practical implementations: on the components that preference-conditioned policies depend on, the objective contrasts, scalarized critics converge more slowly than objective-wise critics.
(b) A different problem from gradient-conflict methods (see our answer to Q2 for additional experiments)
PCGrad (Yu et al., 2020) and CAGrad (Liu et al., 2021) are multi-task learning methods: given separate tasks that share a network, they modify the shared update direction so that one model performs well on every task. Our setting is MORL, which differs in three ways that change the critic-design problem (Appendix A.7):
Our paper therefore studies the mechanism by which critic design shapes MORL learning. Theorem 3 shares the first-order expansion of the gradient-conflict literature and serves as a building block for the trajectory-level results in Theorems 4–5 and Corollary 7.
(c) Theorems 4 and 5
Theorem 4 isolates the part of a shared critic's prediction error caused by sharing, for nonlinear critics, without realizability, convexity, or contraction assumptions. It also yields testable statements:
The quantitative predictions come from Theorem 5 and Corollary 6, and each was tested, including predictions that favor sharing: