acceptodds
Under review as a conference paper at ICLR 2027

What Should Critic Learn? Rethinking Critic Design for Preference-Conditioned Multi-Objective Reinforcement Learning

Abstract

Multi-objective reinforcement learning (MORL) increasingly relies on preference-conditioned policies to adapt decisions across competing objectives, yet a basic architectural question remains unresolved: how should critics represent and estimate objective-wise values? Existing methods variously use scalarized critics, shared multi-head critics, or fully separated critics, often without a principled account of what design is appropriate. We study this question by separating two sources of critic error that are typically conflated: the loss of objective-specific information induced by scalarized supervision and the cross-objective interference induced by shared trainable representations. We first show that scalarized supervision can mask substantial objective-wise errors, even when the scalarized prediction is accurate. We then characterize the interference that remains in multi-head critics through their shared representation, deriving a local gradient-interaction result and a finite-trajectory decomposition of the prediction residual introduced by parameter sharing. Guided by our proposed theory, we first conduct a controlled toy experiment designed to directly validate the predicted behavior. We then empirically evaluate scalarized, shared multi-head, and fully split critic architectures across five representative MORL algorithms—Momba, CAPQL, GPI-LS, MORL/D, and MO-MPO—on eight continuous-control environments spanning two, three, and five objectives. Across the evaluated settings, our results show that critic architecture materially affects MORL performance, with fully split critics providing a robust default across diverse evaluated settings. More broadly, our findings establish critic architecture as a fundamental design choice in MORL.

Primary area:
reinforcement learning
Keywords:
Preference-Conditioned, Multi-Objective Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3uXKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

I believe this paper addresses a real-world problem in multi-objective reinforcement learning (MORL): Should the critic directly regress preference-weighted scalar Q-values, or should each objective have its own independent Q-output? Should these outputs share a backbone, or be trained completely separately? The authors compare three designs—SC, MH, and SP—and attempt to explain how they influence the learning of preference-conditioned actors. I agree with this research topic because critic architectures are often treated as the default implementation, yet they can actually alter a policy’s performance under different preferences.

Critical strengths

S1: I particularly appreciate their experiments across five MORL algorithms. SP achieves higher EUM in many environments, which at least demonstrates that the degree of parameter sharing is a factor that cannot be ignored. However, I would like to see more than just which curve is highest in the end; I also want to know exactly which value estimation errors it improves.

S2: I find it insightful that the paper analyzes the contrast between shared value components and the objective separately. For preference-conditioned policies, differences in action value objectives can influence action selection under different preferences; if the critic learns slowly from these differences, the actor may also learn slowly. However, I was not originally convinced that the authors had proven that such errors necessarily lead to a poorer final policy.

Critical weaknesses

W1: I still have reservations about the theoretical novelty. Given a fixed preference ww, as long as the scalarized QwQ_w is estimated accurately, certain biases in the target components may not necessarily affect strategy improvement for that preference. Theorem 3 shares similarities with the gradient conflicts discussed in PCGrad and CAGrad; Theorem 4, in turn, relies on additional error alignment conditions. I am more interested in knowing: Do these theorems yield predictions that can be tested during training, or do they primarily serve to explain phenomena in hindsight?

W2: I am particularly concerned about the fairness of the experiments. The fact that SP achieves a higher EUM does not automatically imply that the cause is gradient interference from the shared trunk. In particular, GPI-LS and MO-MPO already incorporate cross-policy Q evaluation, objective-specific E-steps, temperature, and KL constraints; if the policy-improvement procedure is modified when switching to SC, it is no longer a pure comparison of critics. EUM is also merely a metric at the policy set level; I would like the authors to directly examine the performance of the policies output by the actor itself after inputting a specific preference.

Questions for authors

Q1: I’d like to see whether MH combined with PCGrad/CAGrad can still keep up with SP when the number of parameters and training steps are matched.

Q2: I would like the authors to clearly specify the critic inputs, TD targets, Q-query methods, and actor-update steps for the SC variants of GPI-LS and MO-MPO.

Q3: I care more about UdirectU_{direct}, which evaluates the actor at each input preference, than EUM, which selects the best policy from a set after the fact.

9 comments

We are grateful to the reviewer for their rigorous and constructive review. We address each point below, with new experiments where needed.

Response to W1: Theoretical novelty and implications for policy improvement

We thank the reviewer for this detailed assessment. Our contribution is a two-axis account of MORL critic design that separates what supervision each objective receives from how much of the critic is shared. This question differs from those addressed by gradient-conflict methods and by Momba and focuses on preference-conditioned critics. We address the four points in turn.

(a) Proposition 2 and policy improvement

For a fixed preference, an accurate scalarized value suffices, and masking alone does not establish harm. Our claim concerns how quickly that accuracy is reached under the limited training budgets of practical implementations: on the components that preference-conditioned policies depend on, the objective contrasts, scalarized critics converge more slowly than objective-wise critics.

  • Mechanism. Averaged over preferences, scalarized supervision supplies only 1/(m+1) of the curvature of objective-wise supervision on the contrast subspace (Eq. 6, Corollary 6), so contrasts are learned more slowly.
  • Why contrasts matter for the actor. By Lemma 1, ∇aQw=1m1⊤∇aQ+wˉ⊤P−∇aQ\nabla_a Q_w = \tfrac1m\mathbf 1^\top\nabla_a Q + \bar w^\top P_-\nabla_a Q with wˉ=w−1m1\bar w = w - \tfrac1m\mathbf 1. The first term is the same for every preference, so the only preference-dependent part of the actor's value gradient is the contrast component. Slower contrast learning is therefore slower learning of how to trade objectives off.
  • Decision-level evidence. In the controlled three-action problem, after 20 updates SC selects the balanced action for every preference, giving a true regret of 0.245, versus 0.014 for MH and 6.110-6 for SP (Appendix A.1.1). With per-architecture learning-rate tuning, SC still chooses suboptimally for 5.72% of preferences vs. 0.318% for MH (A.1.2).
  • End-to-end. SP attains a higher mean EUM than SC in all 24 MuJoCo comparisons where both exist (Table 2).

(b) A different problem from gradient-conflict methods (see our answer to Q2 for additional experiments)

PCGrad (Yu et al., 2020) and CAGrad (Liu et al., 2021) are multi-task learning methods: given separate tasks that share a network, they modify the shared update direction so that one model performs well on every task. Our setting is MORL, which differs in three ways that change the critic-design problem (Appendix A.7):

  • Different object. In multi-task learning, each task has its own data and loss, and each task's performance is an end in itself. In MORL, the objectives are components of one return, evaluated under the same policy on the same transitions, with bootstrapped targets that move with the policy, and decisions depend only on wQ for every preference w. Errors shared by all objectives therefore do not change which action is preferred, while differences between objectives do. Sharing accelerates the first (common) mode and slows the second (contrast) mode (Corollary 6), a structure with no counterpart in multi-task gradient conflict.
  • Different design axis. A scalarized critic trains a single loss, so it has no inter-task gradients to manipulate. Its cost is weaker learning of contrasts when averaged over preferences (Prop. 2, Eq. 6), which gradient-surgery methods do not address.
  • Different intervention. We compare parameterizations (scalarized, shared multi-head, split) rather than modifying gradients within one. The approaches are complementary: gradient surgery could be applied inside an MH critic, and our analysis identifies the interactions (contrast modes) it would need to correct.

Our paper therefore studies the mechanism by which critic design shapes MORL learning. Theorem 3 shares the first-order expansion of the gradient-conflict literature and serves as a building block for the trajectory-level results in Theorems 4–5 and Corollary 7.

(c) Theorems 4 and 5

Theorem 4 isolates the part of a shared critic's prediction error caused by sharing, for nonlinear critics, without realizability, convexity, or contraction assumptions. It also yields testable statements:

  • an exact criterion: sharing increases error if and only if ⟨eTSP,bTshare⟩>−12∥bTshare∥22\langle e_T^{\mathrm{SP}}, b_T^{\mathrm{share}}\rangle > -\tfrac12\|b_T^{\mathrm{share}}\|_2^2, plus a second sufficient condition, ∥bTshare∥2>2∥eTSP∥2\|b_T^{\mathrm{share}}\|_2 > 2\|e_T^{\mathrm{SP}}\|_2 (Appendix C.4);
  • a magnitude bound linking bTshare to the accumulated cross-objective trunk updates ct, which can be measured during training (Appendix C.5).

The quantitative predictions come from Theorem 5 and Corollary 6, and each was tested, including predictions that favor sharing:

PredictionObserved
Contrast contraction factors 0.9833 / 0.95 / 0.8 (SC/MH/SP)Measured trajectories match to 10-15 (A.1)
One-step split–shared slope =1.51.4999906 as the probe step → 0 (A.5)
Sharing is faster on the common modeMH/SC 1.8110-4 vs. SP 1.1510-2 (A.3)
MH = SP when =0; gap grows withConfirmed across the sweep (A.4)
Tracking-lag ordering under moving targetsMatches the recurrence: 0.209 / 0.072 / 0.018 (A.6)

In nonlinear critics, Theorem 4's sufficient condition holds in 19 seeded pairs, and SP has lower held-out scalarized MSE in all those 19 experiments. (Appendix D.2).

(d) Relation to Momba (Štafa et al., 2026, Appendix B)

Momba's contribution is a modernized network backbone for MORL. Its Appendix B is an ablation showing that a vector-valued critic with shared heads performs nearly identically to its scalarized critic, so the scalarized version can be used where vector values are needed. It does not evaluate split critics or analyze critic design. Our paper studies the question this ablation leaves open:

  • Agreement. End-to-end, MH does not consistently outperform SC in our experiments either (MH higher in 13 of 24 cells).
  • Explanation. Objective-wise supervision removes the contrast attenuation of scalarization, but a shared trunk reintroduces cross-objective interference (Theorems 3–5). The two effects can offset each other, which is what Momba's ablation observes.
  • The critic design not evaluated in Momba is the one that demonstrates the strongest gains SP has the highest mean EUM in 29 of 30 MuJoCo comparisons across five algorithms (not only Momba), and the advantage extends to the three- and five-objective tasks (Momba and MORL/D on both). Within Momba itself, SP exceeds the native scalarized critic in all six MuJoCo environments, at both 200k and 400k transitions.

We will discuss Momba's Appendix B in the related work, and we will make the relation of Theorem 3 to gradient-conflict analyses explicit.

Response to W2: Isolating the mechanism

We thank the reviewer for this question. (a) First, our mechanism is cross-objective coupling through shared parameters: the updates ct that each shared block receives from the other objectives. Negative gradient inner products are one sign of this coupling. Representation and optimization dynamics are not separate from it: with matched losses and cloned initialization, Theorem 4 shows that the entire MH–SP prediction difference is generated by ct, while actor, replay, and target feedback enter through a separate term st (Appendix C.6). The evidence isolates this progressively:

  • Capacity is matched: SP widths are reduced to MH's total critic parameter count.
  • Only the trunk is untied: with a frozen actor, identical data and minibatches, and SP cloned from MH, SP has lower held-out contrast RMSE in all 10 configurations (Appendix D.1).
  • The effect has the predicted signature: untying helps contrasts far more than the common component (Swimmer: 33–37% vs. 9–10%, Table 8), and sharing is faster on the common mode (Appendix A.3). A generic "split critics are better representations" account predicts neither asymmetry.
  • It occurs in real training: negative trunk-gradient interactions appear in 44 of 80 probes from end-to-end checkpoints, and the split update is better in all 44 (Appendix E.1).

We agree that attributing each end-to-end gain to this mechanism remains an inference, and we will state this explicitly.

(b) GPI-LS and MO-MPO. See answer for Q2.

(c) EUM vs. direct responsiveness. We agree that EUM permits selecting the best policy for each preference. Beyond EUM, we report empirical Pareto fronts for every algorithm and environment (Figure 2a; Figures 7, 9, 11, 13, 15, 18–19), which show the distribution and extent of the trade-offs achieved rather than only their average utility. Both are set-level views; however, Appendix D.5 defines the direct metric UdirectU_{direct}, which evaluates the actor at each preference without selection. When the policy set is the actor evaluated on W, UdirectU_{direct}, EUM, with equality exactly when each preference's own policy is the best for it, so the gap measures preference mismatch.

We computed UdirectU_{direct} for Momba at 200k (mean ± SD over three seeds):

EnvironmentSCMHSPSP vs. best alternative
Ant1876.05 ± 931.661633.32 ± 460.432337.24 ± 403.79+24.6%
HalfCheetah2398.61 ± 134.302757.16 ± 83.752822.97 ± 443.01+2.4%
Walker2d1730.13 ± 56.631606.11 ± 185.582147.87 ± 188.02+24.1%
Hopper3240.06 ± 368.683067.59 ± 496.453473.72 ± 95.08+7.2%
Swimmer17.88 ± 0.7810.27 ± 5.7418.05 ± 1.02+1.0%
Humanoid3893.74 ± 490.774865.36 ± 76.444863.56 ± 94.00−0.04%

UdirectU_{direct} is below EUM for every critic, confirming the reviewer's point that retrospective selection matters. SP is the only critic that is best or tied for best in all three environments: it attains the highest mean in Ant and Swimmer and matches MH in Humanoid, where both objective-wise critics clearly exceed SC. In Humanoid, MH and SP also retain 96–97% of their EUM without retrospective selection, versus 79% for SC, consistent with our claim that scalarized supervision weakens preference-dependent learning.

Q1&W1(b): Can an MH critic with PCGrad or CAGrad recover SP's gains at matched parameters and budget?

Our analysis predicts that it cannot recover SP’s gains on the objective contrasts, which is where SP’s advantage arises. Gradient surgery changes the direction of a single shared update; it cannot give shared parameters different values for different objectives. On an objective-contrast error, the objectives' shared-trunk gradients are exactly opposite (g1=-g2): MH's summed update is zero, PCGrad's projections are zero, and CAGrad's average gradient, and hence its update, is zero. All three leave the shared parameters unchanged on that mode, whereas SP moves each objective's copy along its own gradient (rates 1- vs. 1-(1-); Corollary 6).

Controlled check. In the three-action problem of Appendix A.1 (parameter-matched, 20 updates, =0.75, =0.2):

CriticScalarized RMSE ↓Decision regret ↓Suboptimal preferences ↓
SP0.00546.110-60.35%
MH0.16900.014116.76%
MH + PCGrad0.16900.014116.76%
MH + CAGrad0.16900.014116.76%

To further address your concern, we additionally trained Momba with PCGrad and CAGrad applied to the MH critic's shared trunk, at matched critic parameter count and training budget (three seeds; run separately from Table 2):

CriticAnt EUMAnt UdirectSwimmer EUMSwimmer Udirect
MH3054.13 ± 822.922278.85 ± 989.6720.76 ± 0.3617.47 ± 1.04
MH + PCGrad2283.58 ± 719.171477.45 ± 431.8521.01 ± 0.9114.61 ± 7.27
MH + CAGrad3082.14 ± 868.102270.55 ± 433.6220.49 ± 0.3815.26 ± 2.77
SP3119.89 ± 381.092381.98 ± 744.1421.11 ± 0.0618.05 ± 1.02

Based on the results, we can see that both CAGrad and PCGrad degrades MH in both env. This matches our analysis: surgery can reduce conflict within a shared update, but only separate parameters let objectives move in opposite directions on contrasts. By 200k the remaining margins are small relative to seed variation. We therefore read these results as showing that gradient surgery does not substitute for parameter separation, not as an additional significance claim for SP.

Q2&W2(b): Critic inputs and policy-update equations of the SC variants of GPI-LS and MO-MPO

Thank you for your question. GPI-LS (Appendix D.3). The SC variant is a structured scalar-query critic. "Scalarized" refers to the supervision, not to removing the component-accessible representation that GPI requires.

  • Critic input and output. The network receives [s,a,z], where z indexes the continuation policy (,z). The evaluation utility u is applied at the readout: qj(s,a,z;u)=uFj(s,a,z),j1,2.qj(s,a,z;u)=uFj(s,a,z),j{1,2}.

  • Scalarized TD target and loss. With a' drawn from the smoothed target policy conditioned on z, the target twin is j=jqj(s′,a′,z;z)j=jqj(s',a',z;z) and y(u)=u⊤r+γ(1−d) qϕˉj⋆(s′,a′,z;u),LQSC=ED,u[∑jℓ(qϕj(s,a,z;u), sg[y(u)])].y(u) = u^\top r + \gamma(1-d)\, q_{\bar\phi_{j^\star}}(s',a',z;u), \qquad \mathcal L_Q^{\mathrm{SC}} = \mathbb E_{\mathcal D,u}\Big[\textstyle\sum_{j}\ell\big(q_{\phi_j}(s,a,z;u),\ \mathrm{sg}[y(u)]\big)\Big]. No objective-wise losses are added; MH and SP instead regress each component of F on objective-wise TD targets.

  • Actor update (unchanged from the baseline): Lπ=− Es,w[12∑jqϕj(s,μθ(s,w),w;w)].\mathcal L_\pi = -\,\mathbb E_{s,w}\Big[\tfrac12\textstyle\sum_{j} q_{\phi_j}\big(s, \mu_\theta(s,w), w; w\big)\Big].

  • Generalized policy improvement (unchanged): aGPI(s;w)∈arg⁡max⁡a∈AM(s) max⁡z∈M qϕ1(s,a,z;w).a_{\mathrm{GPI}}(s;w) \in \arg\max_{a\in A_M(s)}\ \max_{z\in M}\ q_{\phi_1}(s,a,z;w). Because qϕ1(s,a,z;w)=w⊤Fϕ1(s,a,z)q_{\phi_1}(s,a,z;w) = w^\top F_{\phi_1}(s,a,z), this is exactly the query that MH and SP evaluate, over the same candidate actions AM(s)A_M(s) and continuation policies MM. Cross-policy evaluation is therefore preserved, and the SC–MH comparison changes only the supervision.

Reviewer_e1uROpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how different critic architectures affect learning in actor-critic MORL, comparing scalarized, multi-head, and fully separate critics. It also presents a theoretical analysis of how these architectures affect prediction error under limited training budgets, focusing on optimization before convergence. Its comprehensive experiments include a small controlled terminal decision problem to isolate finite-budget critic learning and compare scalarized, multi-head, and split critics under matched optimization conditions as well as evaluate the same critic designs inside several MORL algorithms across MuJoCo, GreenLight, and energy-scheduling benchmarks, using fixed training budgets and Expected Utility Metric to assess the returned policy sets.

Critical strengths

I agree with the paper’s statement that prior work does not give recommendations on the selection of the objective value estimation strategy which has a significant effect on MORL performance, and their contribution in doing so is relevant. Also, the experiments are comprehensive, they first investigate finite-budget critic optimization in a controlled setting and then evaluate whether the architectural effects persist in practical multi-objective reinforcement learning (MORL) under limited training budgets.

Critical weaknesses

The finite-budget setting is relevant for studying optimization, but it may favor architectures such as SP that benefit from reduced interference during learning. The comparability of the experimental results from this viewpoint should be discussed.

Questions for authors

Are the improvements observed specific to limited training budgets, or do they indicate more general advantages of the architecture?

3 comments

The authors sincerely thank the reviewer for their generous review. Your questions helped us strengthen the paper, and we address each below.

W1&Q1: Finite-budget evaluation and persistence of SP's gains

We thank the reviewer for this question.

Our theory predicts that a finite budget can favor either design. Sharing accelerates learning on the common (aligned) mode, where =1+(m-1)>1, and slows it only on contrast modes, where =1-<1 (Corollary 6). Appendix A.3 confirms the first case: with common-mode error, shared critics reach a relative RMSE of 1.8110-4 after 20 updates vs. 1.1510-2 for SP. We also claim no asymptotic advantage on fixed targets, which all critics fit to numerical precision (Appendix A.2). In actor–critic training, however, the target moves with the policy, so slower fitting becomes persistent lag rather than a transient (Corollary 7, Appendix A.6). Finite-budget efficiency is the regime critics operate in throughout training, not an artifact of stopping early.

In our experiments, all architectures share the transition budget, actor, policy-improvement procedure, training protocol, paired training seeds, and evaluation episodes. For Momba, SP widths are reduced to match MH's total critic parameter count, so SP's gains do not come from extra capacity.

To show the impact of budget, we also condition trained Momba trained for up to 800k transitions. Entries are the mean EUM ± sample standard deviation over three seeds; Δ is SP relative to the strongest shared critic.

EnvBudgetSCMHSPΔ
Swimmer100k20.54 ± 0.8519.92 ± 0.2520.60 ± 0.25+0.3%
200k21.34 ± 0.9020.31 ± 0.9321.34 ± 0.310.0%
400k22.17 ± 0.5221.10 ± 1.2822.83 ± 0.76+3.0%
800k22.79 ± 0.3421.39 ± 1.0926.00 ± 4.08+14.1%
Hopper100k3135.73 ± 231.422979.01 ± 244.643177.87 ± 307.17+1.3%
200k3575.59 ± 19.463407.29 ± 223.223715.73 ± 81.15+3.9%
400k3749.71 ± 16.343710.91 ± 69.693824.99 ± 45.40+2.0%
800k3875.49 ± 18.313882.03 ± 63.663887.20 ± 24.26+0.1%
Walker2d100k1757.09 ± 894.541901.15 ± 231.611962.48 ± 373.49+3.2%
200k2588.71 ± 272.622470.98 ± 179.032806.92 ± 147.89+8.4%
400k3051.39 ± 52.012854.37 ± 217.583080.71 ± 169.24+1.0%
800k3072.46 ± 285.563311.31 ± 70.543324.22 ± 65.35+0.4%
HalfCheetah100k2221.85 ± 185.862462.73 ± 63.262399.33 ± 553.89−2.6%
200k2540.40 ± 242.422938.16 ± 60.413037.73 ± 602.18+3.4%
400k3107.23 ± 601.983596.83 ± 745.463848.16 ± 121.83+7.0%
800k3966.88 ± 952.124795.44 ± 34.254733.54 ± 74.44−1.3%
Ant100k779.04 ± 135.621108.20 ± 234.761274.45 ± 236.82+15.0%
200k2450.49 ± 942.302295.21 ± 614.073291.30 ± 178.54+34.3%
400k4222.14 ± 776.964287.95 ± 588.794462.94 ± 77.79+4.1%
800k5090.26 ± 691.725335.97 ± 269.685418.38 ± 126.51+1.5%
Humanoid100k1237.80 ± 211.621190.81 ± 271.721099.86 ± 101.01−11.1%
200k4930.90 ± 45.815012.81 ± 20.885054.65 ± 59.44+0.8%
400k5086.55 ± 40.655126.56 ± 65.555320.33 ± 34.98+3.8%
800k5495.83 ± 215.015341.32 ± 30.505865.54 ± 167.13+6.7%

SP has the highest mean EUM in 21 of 24 cells (11 of 12 at 400k–800k). From 200k onward, SP is never more than 1.3% behind the strongest shared critic (HalfCheetah at 800k, within seed variation). The gap narrows in some tasks, consistent with a finite-budget efficiency effect: in Ant, Walker2d and Hopper, SP's lead peaks at 200k and shared critics close most of it by 800k (Ant: +34.3% → +1.5%). in Humanoid it grows from +0.8% (200k) to +3.8% (400k) and +6.7% (800k); in Swimmer it reaches +14.1% at 800k, with higher SP variance.

The improvements are not specific to the original budget, but their size depends on the budget, as our analysis anticipates. SP's advantage is best described as a learning-efficiency advantage that is retained, not traded away, at longer budgets; With three seeds, many individual cells are within seed variation, so the claim rests on the consistency of the pattern across environments and budgets. We will add these results and this discussion to the revised paper.

Thank you again for your positive assessment and constructive feedback. We have added further experiments and clarifications in response to your question about training budgets, which we briefly summarize below.

W1: Comparability under finite budgets. We clarified that all variants use matched interaction budgets, actor architectures, policy-improvement procedures and evaluation protocols, with paired training seeds. For Momba, SP and MH also have matched total critic parameter counts.

Q1: Do the gains persist with more training? We extended the Momba experiments to 100k, 200k, 400k and 800k transitions, with three seeds per configuration. SP achieves the highest mean EUM in 11 of the 12 comparisons at 400k–800k. The gaps narrow in Hopper, Walker2d and Ant, while persisting or increasing in other environments. These results show that the observed benefits extend beyond the original training budget, although their magnitude depends on the task and budget. We have qualified our conclusions accordingly and do not claim asymptotic superiority.

We hope these additional results help address your concerns about comparability and budget dependence. We would be happy to hear your thoughts and sincerely appreciate your time, encouragement, and constructive feedback.

Thank you for the reply and explanations. I will keep my recommendation as is and will support the acceptance of the paper.

Reviewer_9zbmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the impact of critic design in preference-conditioned multi-objective reinforcement learning with linear utility functions (determined by weights ww). It compares three critic designs all conditioned on (s,a,w)(s,a,w): a scalarized critic (SC) that directly predicts the expected scalarized return; a multi-head critic (MH) that predicts objective-wise expected returns using a shared trunk and objective-specific heads; and split critics (SP) that use a separate function approximator for each objective. The theoretical analysis examines three aspects: ** At a fixed preference, scalarized supervision can conceal objective-specific prediction errors even when the scalarized prediction is correct. ** With the policy and training targets fixed, conflicting gradients in the shared trunk can cause MH to achieve less one-step reduction in total critic loss than SP, under matched initial parameters and sufficiently small updates. ** Over a finite training trajectory, the authors characterize the accumulated prediction difference induced by parameter sharing and provide sufficient conditions under which it increases MH’s prediction error relative to SP.

Empirically, the authors first compare critic prediction errors under a fixed policy, reporting lower errors when moving from SC to MH to SP. They then evaluate the three critic designs across five preference-conditioned reinforcement learning algorithms in multi-objective control environments.

Critical strengths

** It offers a way to analysis the critic bias in MORL in general.

** The paper is well-written; it is easy to follow the whole story.

** Code is available and experiments details are clear.

Critical weaknesses

** The scalarized-supervision analysis has limited implications for policy quality. Hidden component-wise errors can coexist with a correct scalarized prediction, which is sufficient for decision-making at a fixed preference. Thus, this result alone does not establish a disadvantage of SC. Errors in experiments are all quite small.

** The interference results do not establish worse final predictors. Theorem 3 concerns one-step training-loss reduction. Theorem 4 extends the analysis to a finite trajectory, but its error comparison depends on an additional alignment condition involving SP’s residual error and the sharing-induced prediction difference. This condition provides limited guidance before training, and its prevalence in practice needs to be established.

** The connection to end-to-end RL remains indirect. The paired analysis uses matched data and frozen targets at each step; but the training for AC framework is usually more complicated. Moreover, greater critic-loss reduction does not automatically imply better policy performance.

Questions for authors

My concern is primarily the significance and applicability of the results, rather than their mathematical correctness. Below are my thoughts and also my questions: The scalarized-supervision analysis identifies hidden component-wise errors even when the scalarized prediction is correct; such errors are not inherently problematic for optimization under a fixed linear utility (don't forget the target for Pareto Optimal policies and Pareto Frontier). The shared-trunk analysis establishes a conditional one-step optimization disadvantage, but this does not demonstrate persistent interference or inferior converged predictors(in other domain, for instance the scaling law or multi-task learning somehow shows tangle everything together have some sort of advantage). Although the finite-trajectory analysis goes further, its sufficient condition depends on the alignment between the final residual error and the sharing-induced prediction difference. This limits its usefulness as an a priori explanation of when splitting should help.

6 comments

We are grateful to the reviewer for their rigorous and constructive review. We address each point below, with new experiments where needed.

W1&Q1: Policy relevance of scalarized supervision

We thank the reviewer for this precise comment. It holds for a fixed linear preference: a correct scalarized value is sufficient for decision-making, and component-wise errors that cancel under w() do not by themselves degrade the policy. We have revised the paper to state it more directly below:

The cost of scalarized supervision is not hidden error at one preference. It is slower learning of the value differences that determine which action is optimal as the preference varies.

Below we (1) explain the mechanism, (2) show that it changes decisions, (3) show that it survives architecture-specific learning-rate tuning, (4) address the magnitude of the errors, and (5) connect it to end-to-end policy quality.

1. Why correctness at a fixed preference does not settle the question

A preference-conditioned critic must rank actions correctly for every w. The component that changes the ranking as w moves is the objective contrast (e.g., Q1-Q2); the common component shifts all actions equally and never changes the argmax. Averaged over preferences, scalarized supervision provides much weaker curvature in exactly this contrast direction. Under uniform preferences with normalization c=m (Proposition 2, Eq. 6; Corollary 6, Appendix B.6):

HSC=P++1m+1 P−,Hobj=Im,H_{\mathrm{SC}} = P_+ + \tfrac{1}{m+1}\,P_-, \qquad H_{\mathrm{obj}} = I_m,

so the decision-relevant contrast subspace P- receives only 1/(m+1) of the correction signal that objective-wise supervision provides. The (3,1) vs. (2,2) masking example is the pointwise view; Eq. 6 is its preference-averaged consequence, and it is this quantity that governs learning speed.

2. The effect changes decisions, not only component errors

We did additional analysis to measure regret using the true environment rewards in the three-action problem with r(a+)=(3,1), r(a-)=(1,3), r(a0)=(2.3,2.3). All three actions are Pareto-optimal; the optimal action switches at t=0.35 and t=0.65 for w=(t,1-t):

RTA=Ew[max⁡aw⊤r(a)−w⊤r(a^TA(w))],R_T^A = \mathbb{E}_w\Big[\max_a w^\top r(a) - w^\top r\big(\hat a_T^A(w)\big)\Big],

a^TA(w)∈arg⁡max⁡aQ^w,TA(a).\hat a_T^A(w) \in \arg\max_a \hat Q^A_{w,T}(a).

All critics have 12 trainable parameters and identical initial scalarized predictions, and all can represent the target exactly (=0.75, =0.2; Appendix A.1.1, Table 3).

UpdatesCriticScalarized RMSE ↓Decision regret ↓Suboptimal preferences ↓
20SC0.33680.245070.00%
20MH0.16900.0140516.76%
20SP0.005436.12e-60.35%
100SC0.08782.36e-36.87%
100MH0.002791.60e-60.18%
100SP9.60e-11<1e-12≈0%

After 20 updates, SC selects the balanced action a0 for every preference, MH switches at (0.266,0.734), and SP at (0.348,0.652), close to the optimal (0.35,0.65). Because SC and MH share the same effective parameterization, the SC–MH comparison isolates supervision: replacing scalarized with objective-wise targets reduces regret by 17× (0.245 → 0.014). These are not component errors that cancel; they are scalarized-value errors that reorder actions.

3. Robustness to learning-rate tuning and initialization

To address your concern, we therefore added a control (Appendix A.1.2, Table 4) with mixed-mode initialization (equal common and contrast error) and per-architecture learning-rate selection under an identical budget (9 candidates × 20 updates), selected by validation RMSE rather than regret.

CriticScalarized RMSE ↓Decision regret ↓Suboptimal preferences ↓
SC0.07912.23e-35.72%
MH0.002995.52e-60.318%
SP<1e-12<1e-120%

This ordering is expected to survive tuning because the mechanism concerns conditioning, not step size. From Corollary 6 (m=2, =0.75), the ratio of common-mode to contrast-mode curvature is 21 for SC, 7 for MH, and 1 for SP. A single step size must remain stable on the fast common mode, so it cannot remove the slow contrast mode: tuning narrows the gap but cannot close it. SP's exact fit after tuning is the same fact seen from the other side, since its update is perfectly conditioned in this instance.

4. On the magnitude of the errors

The error magnitude indirect proxy: on its own, it is not conclusive evidence of policy degradation. We therefore now also report regret. Three points:

  • The scale is set by construction. The instance is deliberately small and warm-started with the common component already correct, so absolute regret is small for every method. Decision frequency is the scale-free readout: even after tuning, SC selects a suboptimal action for 5.72% of preferences vs. 0.318% for MH (18×). At the common step size, SC collapses to a single action for all preferences after 20 updates.
  • In actor–critic training, targets move. When the evaluated policy changes between critic updates, slower fitting becomes persistent lag rather than a transient (Corollary 7; Appendix A.6). After 200 policy changes, the contrast lag is 0.209 (SC), 0.072 (MH), and 0.018 (SP).
  • The claim is finite-budget, not asymptotic. All three critics fit a fixed target to numerical precision within 2,000 updates (Appendix A.2). We claim slower learning of decision-relevant contrasts, which matters in RL, where critics receive a limited number of updates against moving targets.

We do not present the tuned SC regret in this toy (expected utility 2.5428 vs. optimum 2.5450) as a large practical loss; the end-to-end results below speak to practical magnitude.

5. End-to-end policy quality

The main paper evaluates the returns of learned policy sets (EUM, over a fixed set of preferences, of the best weighted return any policy in the returned set achieves for that preference), not critic losses, across five MORL algorithms. On MuJoCo (Table 2), SP attains the highest mean EUM in all 24 method–environment comparisons that include an SC variant (29/30 overall), with large gains in some settings (CAPQL on Walker2d: 2.47× the best shared variant). On three-objective GreenLight, SP improves Momba (0.2593 vs. 0.2397 for SC) and MORL/D (0.1907 vs. 0.1815); CAPQL and GPI-LS learn poorly there under all critics. On the five-objective IEEE 33-bus task, SP attains the highest mean EUM for Momba, MORL/D, and CAPQL. We are explicit about one pattern: end-to-end, MH does not consistently beat SC (13 of 24 cells). This is consistent with our two-axis analysis. MH gains objective-wise supervision but retains shared-trunk interference (Theorems 3–4), whose net effect depends on gradient alignment. SP removes both sources and is the design that consistently improves on SC. The controlled SC–MH comparison isolates the supervision effect; the end-to-end results establish that critic design materially affects policy quality.

6. Pareto coverage vs. preference-to-action matching

After tuning, all three critics select each of the three Pareto-optimal actions for some preferences, so the differences do not concern Pareto-front coverage. They concern choosing the right Pareto-optimal action for a given preference, which is what preference-conditioned MORL is evaluated on (EUM).

W2&Q2: : Persistence of interference and predictor quality

We thank the reviewer for separating local interference from final predictor quality.

Sharing induces a sign-dependent coupling that accumulates over the whole training trajectory, slows learning specifically on objective contrasts, and measurably degrades finite-budget predictors in nonlinear critics.

1. The theory is not one-step

  • Theorem 3 alone is a local, one-step statement; parameter sharing can produce positive transfer; and with realizable fixed targets, shared critics need not converge to worse predictors (all three critics fit our controlled target to numerical precision, Appendix A.2).
  • Theorem 4 is a whole-trajectory statement. The sharing-induced residual bTshare=-t<TtJTT,t+1ct accumulates the cross-objective trunk updates ct from every step, transported through the nonlinear training dynamics.
  • Theorem 5 gives an exact multi-step ordering. On an error mode with coupling eigenvalue 0<<1, ETSP=(1−ησ)TE0<(1−ησκ)TE0=ETMH\mathcal E_T^{\mathrm{SP}} = (1-\eta\sigma)^T\mathcal E_0 < (1-\eta\sigma\kappa)^T\mathcal E_0 = \mathcal E_T^{\mathrm{MH}} for all T1. Corollary 6 shows this holds on every contrast mode of the shared-plus-private family, where =1-.
  • With moving targets, the gap does not decay. In actor–critic training, the target changes whenever the policy changes. Corollary 7 shows that slower fitting then becomes a persistent tracking lag: after 200 policy changes, the contrast lag is 0.072 for MH vs. 0.018 for SP, a 4× difference (Appendix A.6).

The relevant question is therefore not whether MH eventually reaches the same fixed-target predictor (it does), but how much error it carries at the finite budgets and moving targets under which MORL critics are actually trained.

2. Interference persists in nonlinear critics

Late in training. Among 80 probes taken from MuJoCo checkpoints at 100k and 200k steps, 44 have a negative inter-objective trunk-gradient inner product. In all 44, the split update reduces the critic loss more than the shared update, as Theorem 3 predicts (Appendix E.1). Interference is present late in training, not only at initialization. After thousands of updates. In a paired intervention (Appendix D.1, Table 5), SP is created by cloning the MH trunk and heads, and both critics evaluate the same frozen actor with the same data and minibatch sequence for 3,000 updates. SP achieves lower held-out objective-contrast RMSE in all 10 environment–actor-source configurations:

EnvironmentMH actor sourceSP actor source
Swimmer36.7%33.2%
Hopper22.3%8.7%
Walker2d13.1%6.6%
HalfCheetah2.6%19.8%
Ant3.2%3.2%

SP's reduction in held-out contrast RMSE relative to MH.

Consistent with Corollary 6, the advantage concentrates on contrasts: in Swimmer, SP reduces contrast RMSE by 33–37% but common-component RMSE by only 9–10% (Table 8).

3. Theorem 4's condition: exact criterion and measured prevalence

Theorem 4 is an exact identity, MTMH−MTSP=2⟨eTSP,bTshare⟩+∥bTshare∥22M_T^{\mathrm{MH}} - M_T^{\mathrm{SP}} = 2\langle e_T^{\mathrm{SP}}, b_T^{\mathrm{share}}\rangle + \|b_T^{\mathrm{share}}\|_2^2, so the alignment term is an exact accounting, not an assumption. Sharing increases error if and only if ⟨eTSP,bTshare⟩>−12∥bTshare∥22\langle e_T^{\mathrm{SP}}, b_T^{\mathrm{share}}\rangle > -\tfrac12\|b_T^{\mathrm{share}}\|_2^2 (Appendix C.4). Nonnegative alignment is one sufficient condition. Another is ∥bTshare∥2>2∥eTSP∥2\|b_T^{\mathrm{share}}\|_2 > 2\|e_T^{\mathrm{SP}}\|_2: whenever the sharing-induced difference is large relative to the SP residual, MH is worse regardless of alignment. We measured the condition across 30 independently seeded paired evaluations (Appendix D.2, Table 6):

EnvironmentNonneg. alignment holdsSP lower scalarized MSE
Ant4/65/6
HalfCheetah2/62/6
Walker2d3/64/6
Hopper4/64/6
Swimmer6/66/6
Total19/3021/30

SP has lower scalarized MSEs in 21 of 30 pairs (one-sided sign test, p0.02), including two pairs outside the sufficient condition. The variation across environments (6/6 in Swimmer, 2/6 in HalfCheetah) is what a conditional result predicts. It is also consistent with Table 5: scalarized MSE mixes common and contrast components, and sharing can help the common component (Corollary 6), so a seed can favor MH on scalarized MSEs while SP still estimates contrasts better. Thus, Theorem 4's role is to attribute exactly the part of MH's error caused by sharing. Observable guidance comes from Theorem 3, whose interaction term ∑i<j⟨gi,gj⟩\sum_{i<j}\langle g_i, g_j\rangle can be computed during training from per-objective trunk gradients (as in Appendix E.1), and from Theorem 5, whose condition depends on the coupling structure rather than on final residuals.

4. Positive transfer, multi-task learning, and scale

Our theory predicts positive transfer. On the common mode, =1+(m-1)>1, so sharing accelerates learning (Corollary 6). Appendix A.3 confirms this: after 20 updates, the relative RMSE is 1.8110-4 for MH vs. 1.1510-2 for SP. Sharing is not harmful in general; it helps the common component and slows the contrasts. Why MORL differs from multi-task learning (Appendix A.7). In multi-task learning, sharing pools data across tasks. In MORL, every objective critic is trained on the same transitions under the same policy, so SP critics still use all of the data. What parameter sharing adds is coupling, and the contrasts it slows are exactly what determines how the optimal action changes with the preference. Gains are not from extra capacity. In the end-to-end comparisons, SP critic widths are reduced to match MH's total critic parameter count (Momba, GPI-LS, MO-MPO; reduced SP sizes for MORL/D). In the linear analysis, a parameter-count-matched construction leaves the dynamics unchanged (Appendix B.8). The cloned diagnostic in D.1 is deliberately not count-matched, because it isolates the effect of untying the trunk.

Summary

We agree that Theorem 3 is local, that sharing can transfer positively, and that shared critics need not reach worse converged predictors on fixed targets. Our results instead show that sharing-induced coupling accumulates over the full trajectory (Theorems 4–5), persists under moving targets (Corollary 7), is present late in nonlinear training (E.1), and degrades held-out contrast estimation after 3,000 updates in all 10 configurations (D.1). Theorem 4's condition is an exact criterion whose prevalence we now report (D.2); we do not present it as an a priori selection rule. We will make these distinctions explicit in §4.3–4.4 and reference Appendices D.1, D.2 and E.1 from the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.