acceptodds
Under review as a conference paper at ICLR 2027

Jacobian Policy Optimization: Geometry-Aware Learning from Structured Rewards

Abstract

Reinforcement learning for language-model reasoning increasingly uses multiple reward signals to evaluate the consequences and properties of generated responses. Training a shared policy to respond to different priorities requires learning behaviors whose outcomes reflect the requested emphasis among these signals. We introduce **Jacobian Policy Optimization (JPO)**, a preference-conditioned reinforcement learning method that uses objective-specific gradient geometry to construct policy updates. JPO independently normalizes each reward dimension, constructs separate clipped policy losses, and differentiates them to form a Jacobian of policy gradients. Preference-weighted UPGrad modifies conflicting gradient components before aggregation, allowing both the requested preference and the relationships among learning signals to determine the update. We evaluate JPO in two complementary settings. On MaZE, where rewards describe completion, resource collection, and hazard avoidance, JPO achieves substantially higher matched-preference utility and stronger preference specialization than GRPO and GDPO on held-out mazes under the training preference set. On Level-5 MATH, where rewards assess correctness, conciseness, and rigour, JPO achieves the highest individual-sample accuracy and Pass@8 among the evaluated methods, while remaining competitive on Majority@8. It also achieves the highest Majority@8 after Recursive Self-Aggregation. These results support geometry-aware multi-reward policy optimization as an approach to improving both preference-conditioned behavior and reasoning performance under multiple quality criteria.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Multi-Objective Reinforcement Learning, Multi-Reward Optimization, Policy Optimization, Jacobian Descent, Gradient Conflict, Preference-Conditioned Reinforcement Learning, LLM Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_F3pZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Jacobian Policy Optimization (JPO), which aims primarily to address the problem of how language models should adjust their behavior according to user preferences when faced with multiple rewards simultaneously. The paper first normalizes each reward individually, then constructs separate policy losses and gradients for each, and finally utilizes UPGrad to resolve gradient conflicts between different objectives, applying the results to model updates. The authors utilized Qwen3.5-4B and conducted tests on maze navigation and MATH Level-5 math problems. In both the maze and math experiments, the model achieved higher single-sample accuracy.

Critical strengths

The issues addressed in this paper are of practical significance. The training of language models often takes into account correctness, conciseness, and other requirements simultaneously, and these objectives may influence one another. The authors attempt to preserve the gradient information for each objective prior to updating the model, which offers a promising direction for research into multi-objective training. The paper demonstrates notable improvements on certain metrics. JPO achieves an average preference-matching utility of 0.595 on the maze task, which is higher than GRPO’s 0.288 and GDPO’s 0.271, with improvements observed across all four preference conditions. JPO’s single-sample mathematical accuracy stands at 78.60 per cent, which is also higher than GRPO’s 74.64 per cent and GDPO’s 73.60 per cent.

Critical weaknesses
  1. The contribution of UPGrad, a central component of the method, has not been independently validated. The authors themselves acknowledge in sec 5, lines 471–472 that an ablation isolating UPGrad’s contribution remains future work. For a method whose central claim concerns gradient geometry, this is a question the current paper should address.

  2. The paper already discusses D³PO, which constructs a separate clipped loss for each objective before applying a weighted sum. This baseline would better isolate the incremental contribution of JPO’s gradient processing than GRPO or GDPO, yet it is not included in the experiments.

  3. The method requires computing separate gradients for multiple objectives. Although the authors acknowledge the additional computational and memory overhead, they do not report training time, peak memory usage, or comparisons under the same compute budget. The experiments are also largely limited to LoRA training of Qwen3.5-4B, leaving its practicality for full-parameter fine-tuning or larger models insufficiently established.

  4. Does delaying scalarization offer any particular benefits? The paper first acknowledges that fixed weighted aggregation commutes with differentiation (lines 56–58). However, it later attributes improvements to preserving objective-specific signals through differentiation (lines 445–447 and 465–471). Delaying aggregation alone does not change the update; the substantive differences lie in objective-specific clipping and nonlinear UPGrad projection. Ablations are needed to distinguish their contributions and support this interpretation.

Questions for authors

See Weaknesses.

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer F3pZ [1/5]: Important Correction to the Original MaZE Baselines

We thank the reviewer for their thoughtful assessment and for recognizing the practical significance of multi-objective LLM post-training and the encouraging results on MaZE and MATH. We agree that isolating UPGrad's contribution and quantifying its computational overhead are essential. To address these concerns, we have conducted additional controlled experiments with linear weighted aggregation, D3PO-LSW, PCGrad, and P-GAPO, alongside corrected GRPO and GDPO baselines. We also introduce quantitative measures of gradient conflict and preference-conditioned behavioral responsiveness.

Correction to the original GRPO/GDPO MaZE baselines

During the additional experiments, we identified a silent numerical inconsistency affecting the original GRPO and GDPO MaZE training runs.

The issue arose when PPO old-policy log-probabilities were recomputed using a different microbatch shape from the JD actor forward computation. In our Qwen3.5/veRL configuration, token log-probabilities were batch-shape-dependent, causing a spurious difference between the old and current policies even though the model weights were unchanged.

We fixed the issue by setting score_batch_size equal to the rollout group size (G=8G=8), ensuring consistent batch shapes. This restored the expected PPO probability ratio of 1 for an unchanged policy, up to numerical precision.

We reran the affected MaZE baselines using the corrected configuration. These results supersede the original GRPO and GDPO MaZE comparisons, which understated baseline performance. The baseline comparisons below use the corrected results. This issue did not affect our MATH experiments, whose previously reported results remain unchanged.

AuthorsOpenReview ↗

Official Response to Reviewer F3pZ [2/5]: UPGrad Ablation and Gradient Conflict Analysis (Weaknesses 1 & 4)

1. Isolating UPGrad's contribution (Weaknesses 1 and 4)

We agree that the original experiments did not fully disentangle objective-specific loss construction from gradient aggregation. Our additional experiments now directly address this distinction. UPGrad's contribution is that it almost entirely eliminates meaningful first-order gradient conflicts.

Given objective-specific losses LiL_i, gradients gi=∇ϕLig_i=\nabla_\phi L_i, and preference weights pip_i, ordinary weighted aggregation produces

dlinear=∑ipigi.d_{\mathrm{linear}}=\sum_i p_i g_i.

JPO instead projects each gradient onto the common non-conflicting cone before aggregation:

dJPO=∑ipiΠC∗(J)(gi),d_{\mathrm{JPO}}=\sum_i p_i\Pi_{\mathcal{C}^*(J)}(g_i),

where

C∗(J)={d:gi⊤d≥0, ∀i}.\mathcal{C}^*(J)=\{d:g_i^\top d\geq 0,\ \forall i\}.

These directions coincide when the objective gradients are mutually non-conflicting. Otherwise, UPGrad can produce a different direction that avoids first-order interference between objectives.

New ablation: Linear weighted gradients. We trained a baseline using exactly the same objective-wise normalization, independently clipped PPO losses, preferences, and training configuration as JPO, replacing only UPGrad with an ordinary preference-weighted gradient sum. This directly isolates the gradient aggregation mechanism.

Quantifying gradient conflicts. We analyzed the Gramian of the five objective-specific LoRA gradients (completion, gold, diamond, lava avoidance, and response length), pooled across runs. The strongest conflict occurs between gold and diamond:

Objective pairMean gradient cosineNegative cosine frequency
Gold–Diamond−0.52199.0%
Gold–Length+0.03637.9%
Diamond–Length+0.06434.6%
Completion–Lava avoidance+0.9980.0%

Negative cosine indicates that descending along one objective gradient would locally increase the other objective's loss. Gold–diamond conflict is therefore substantial and persistent, rather than merely hypothetical.

We next measured whether the combined gradient direction conflicts with any active objective gradient. Specifically, we calculate the minimum cosine between the combined direction and each nonzero objective gradient, and report the fraction of training steps where this minimum is below −0.01.

Aggregation methodGradient stepsConflicting direction (%)
JPO (UPGrad)1,1420.1%
Linear weighted sum (same losses)1,16311.8%
p-GAPO1,16611.3%
PCGrad1,1632.0%

Thus, UPGrad's contribution is that it almost entirely eliminates meaningful first-order gradient conflicts, compared with 11.8% of steps for the identical-loss linear baseline. As a further controlled comparison, we observed that applying an ordinary weighted sum to the same gradients recorded during the JPO run would produce conflicting directions in 11.2% of steps.

AuthorsOpenReview ↗

Official Response to Reviewer F3pZ [3/5]: D3PO-LSW Comparison and Preference Steering (Weakness 2)

2. D3PO comparison and preference-conditioned behavior (Weakness 2)

We agree that D3PO is a particularly relevant baseline. Its Late-Stage Weighting (LSW) mechanism independently constructs clipped surrogate losses before preference-weighted aggregation. We therefore implemented a D3PO-LSW-style baseline within our LLM training framework.

Unlike a full reproduction of D3PO, which additionally uses a multi-head critic and diversity regularization, this comparison focuses on its separate-loss and late-weighting mechanism. Together with the identical-loss linear ablation, it provides a direct assessment of JPO's incremental gradient processing.

To evaluate preference-conditioned behavior beyond weighted utility, we introduce two complementary metrics for the empirically conflicting gold and diamond objectives.

(a) Gold–diamond preference steering.

Let rˉ(k,c)\bar{r}(k,c) denote the mean reward for objective kk when conditioned on preference cc. (G=Gold, D=Diamond, C=Completion, L=Lava)

Gold steering:

S(G)=rˉ(G,G)−rˉ(G,C)+rˉ(G,D)+rˉ(G,L)3S(G) = \bar{r}(G,G) - \frac{\bar{r}(G,C) + \bar{r}(G,D) + \bar{r}(G,L)}{3}

Diamond steering:

S(D)=rˉ(D,D)−rˉ(D,C)+rˉ(D,G)+rˉ(D,L)3S(D) = \bar{r}(D,D) - \frac{\bar{r}(D,C) + \bar{r}(D,G) + \bar{r}(D,L)}{3}

Combined steering:

Ssteer=S(G)+S(D)2S_{steer} = \frac{S(G) + S(D)}{2}

This measures whether requesting gold increases gold reward, and requesting diamonds increases diamond reward, relative to the three alternative conditioning preferences.

(b) Direct gold–diamond switching.

Since gold and diamond exhibit the strongest gradient conflict, we introduce a targeted metric comparing only these two preferences:

Sswitch=12[(rˉ(G,G)−rˉ(G,D))+(rˉ(D,D)−rˉ(D,G))]S_{switch} = \frac{1}{2}\left[(\bar{r}(G,G)-\bar{r}(G,D))+(\bar{r}(D,D)-\bar{r}(D,G))\right]

This measures the change in the gold-minus-diamond reward balance when switching the requested preference from gold-heavy to diamond-heavy.

Both metrics measure responsiveness to the requested preference. We additionally report completion and absolute rewards as controls.

Held-out evaluation. All models were evaluated on the same 50 held-out mazes under all four preference conditions. The following table reports the final checkpoint at training step 1,200 for every method:

MethodCompletionGold when requested / otherwiseDiamond when requested / otherwiseSteeringDirect switching
JPO (UPGrad)0.8590.767 / 0.2130.781 / 0.1920.5720.773
D3PO-LSW0.8490.732 / 0.1920.802 / 0.1950.5730.766
PCGrad0.8640.731 / 0.2170.793 / 0.2000.5540.762
p-GAPO0.8550.717 / 0.1630.798 / 0.1850.5830.756
GRPO (corrected)0.8590.721 / 0.1850.767 / 0.1990.5530.741
Linear (same losses)0.8280.698 / 0.1780.756 / 0.1890.5430.727
GDPO (corrected)0.7980.660 / 0.1930.653 / 0.1850.4680.654

JPO achieves the highest direct gold–diamond switching score (0.773), improving over the identical-loss linear baseline (0.727) and corrected GRPO (0.741) and GDPO (0.654). It also improves completion, gold-when-requested, and diamond-when-requested rewards over the identical-loss linear baseline.

However, the additional baselines are competitive: D3PO-LSW achieves essentially identical overall steering, while p-GAPO achieves slightly higher steering. We interpret these results as supporting JPO's ability to learn preference-responsive behavior, particularly under explicit gold–diamond conflict. The results do not establish superiority across every metric.

Training behavior. We additionally evaluated preference responsiveness over the final 200 training steps (1,001–1,200), using the mean rewards from eight rollouts per training step:

MethodCompletionSteeringDirect switching
JPO (UPGrad)0.8500.6410.821
p-GAPO0.8480.6070.773
Linear0.8360.5930.773
PCGrad0.8500.5900.781
D3PO-LSW0.8310.5880.767
GRPO (corrected)0.8380.5710.750
GDPO (corrected)0.7840.5240.679

JPO achieves the highest steering and direct-switching scores in this training window. These training statistics measure conditional rewards across different training steps, whereas held-out evaluation compares preferences within the same mazes. We therefore place greater weight on the held-out comparisons.

Overall, the additional experiments establish a clear geometric difference between UPGrad and ordinary weighted aggregation, together with evidence of improved preference responsiveness in several comparisons.

Reviewer_5YfjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

JPO constructs separate clipped policy losses for multiple reward dimensions and combines their gradients using preference-weighted UPGrad. The method is evaluated on MaZE and Level-5 MATH against GRPO and GDPO.

Critical strengths

The method is clearly described. The substantial MaZE improvements and higher direct-sampling accuracy on MATH are encouraging.

Critical weaknesses
  • The novelty is limited, and the closest comparisons are missing. JPO combines existing objective-wise normalization, separate clipped losses, and UPGrad, but does not compare against D³PO or the same separate-loss setup without UPGrad. These controls are needed to establish what the proposed gradient aggregation actually contributes.

  • The additional computational cost is not evaluated. JPO requires objective-specific gradients, yet reports no runtime, peak memory, or equal-compute comparison. Without these measurements, it is difficult to judge whether the improvements justify the extra work.

Questions for authors
  • How does JPO compare with identical objective-specific losses combined using ordinary weighted gradients or PCGrad?
  • What are the runtime and memory overheads, and do the improvements hold under a matched compute budget?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer 5Yfj [1/4]: Correction to the Original MaZE Baselines

We thank the reviewer for their assessment and for recognizing the encouraging MaZE and MATH results. We agree that isolating UPGrad's contribution and quantifying computational overhead are important. We have completed additional experiments addressing these concerns.

Important correction to the original MaZE baselines

During our additional experiments, we identified a silent implementation issue affecting the original GRPO/GDPO MaZE runs. PPO old log-probabilities were recomputed using a different microbatch shape from the actor forward pass. In our Qwen3.5/veRL configuration, this produced spurious old/current policy ratios despite unchanged model weights. Setting score_batch_size to the rollout group size (G=8) aligned both forwards and restored the expected ratio of 1 up to numerical precision. We reran GRPO and GDPO with this correction. The MATH experiments were unaffected.

The corrected GRPO and GDPO MaZE baselines perform substantially better than those reported in the original submission. We therefore withdraw the original MaZE baseline comparisons and replace them with the corrected results presented. Importantly, the corrected comparisons show that several baselines are considerably more competitive with JPO than the original results suggested. We have adjusted our empirical conclusions accordingly. All MaZE comparisons below use the corrected runs.

AuthorsOpenReview ↗

Official Response to Reviewer 5Yfj [2/4]: UPGrad Novelty, Controlled Ablations, and Gradient Conflicts (Weakness 1)

1. Novelty and UPGrad's contribution (Weakness 1, Question 1)

JPO combines objective-wise normalization (GDPO), separately clipped PPO losses (D3PO), and UPGrad (Jacobian Descent) for preference-conditioned LLM training. Its distinguishing feature is conflict-aware gradient aggregation, not merely delayed scalarization. Indeed, fixed linear weighting commutes with differentiation:

∇ϕ(∑ipiLi)=∑ipi∇ϕLi.\nabla_\phi\left(\sum_i p_iL_i\right)=\sum_i p_i\nabla_\phi L_i.

We ran four additional MaZE baselines, all using the same Qwen3.5-4B model, LoRA configuration, data, preferences and 1,200 training steps:

  • Linear: Same objective-specific losses and preferences as JPO, but ordinary weighted gradients. This is the direct UPGrad ablation.
  • PCGrad: Alternative gradient-conflict correction.
  • p-GAPO: MGDA-based gradient aggregation.
  • D3PO-LSW: Separate clipped losses followed by linear weighting, adapted to our critic-free setup (without D3PO's diversity regularizer).

GRPO, GDPO and D3PO-LSW aggregate before differentiation (at the reward, advantage and loss levels, respectively). Linear, PCGrad, p-GAPO and JPO calculate separate gradients; only the latter three perform geometry-aware aggregation. This isolates the contributions of objective decomposition and gradient aggregation.

Preference responsiveness metrics. To assess whether conflict resolution translates into preference-conditioned behavior, we introduce two complementary metrics.

Let rˉi∣j\bar r_{i|j} denote the mean reward for objective ii when the policy is conditioned on preference jj, where G,D,C,LG,D,C,L denote gold-heavy, diamond-heavy, completion-heavy, and lava-avoidance-heavy preferences, respectively.

The first metric, preference steering, measures whether emphasizing an objective increases its corresponding reward relative to the three other conditioning preferences.

For gold and diamond separately:

\frac{\bar r_{G|C}+\bar r_{G|D}+\bar r_{G|L}}{3},$$ $$S_D=\bar r_{D|D}- \frac{\bar r_{D|C}+\bar r_{D|G}+\bar r_{D|L}}{3}.$$ The combined preference steering score is $$\boxed{ S_{\mathrm{steering}}=\frac{S_G+S_D}{2}. }$$ The second metric, **direct gold–diamond switching**, focuses specifically on the two conflicting collection objectives. It measures how strongly the policy changes its gold-versus-diamond reward balance when switching directly between gold and diamond-heavy instructions: $$\boxed{ S_{\mathrm{direct}}=\frac12\left[ (\bar r_{G|G}-\bar r_{G|D})+ (\bar r_{D|D}-\bar r_{D|G}) \right]. }$$ The first term measures the increase in gold collection when gold rather than diamonds is prioritized; the second measures the corresponding increase in diamond collection when diamonds are prioritized. **Table 1. Held-out results at step 1,200 (higher is better).** Gold/diamond columns show reward when requested / under the other three preferences. | Method | Completion | Gold / otherwise | Diamond / otherwise | Steering | Direct | |---|---:|---:|---:|---:|---:| | **JPO** | 0.859 | **0.767** / 0.213 | 0.781 / 0.192 | 0.572 | **0.773** | | D³PO-LSW | 0.849 | 0.732 / 0.192 | **0.802** / 0.195 | 0.573 | 0.766 | | PCGrad | **0.864** | 0.731 / 0.217 | 0.793 / 0.200 | 0.554 | 0.762 | | p-GAPO | 0.855 | 0.717 / 0.163 | 0.798 / 0.185 | **0.583** | 0.756 | | GRPO (corrected) | 0.859 | 0.721 / 0.185 | 0.767 / 0.199 | 0.553 | 0.741 | | Linear | 0.828 | 0.698 / 0.178 | 0.756 / 0.189 | 0.543 | 0.727 | | GDPO (corrected) | 0.798 | 0.660 / 0.193 | 0.653 / 0.185 | 0.468 | 0.654 | JPO improves on the identical-loss Linear control by **0.029 steering** and **0.046 direct switching**, and has the highest direct-switching score; p-GAPO slightly leads in steering (0.583 vs 0.572). The next table shows the same metrics averaged in the last 200 training steps. **Table 2. Training rollouts, final 200 steps.** | Method | Completion | Steering | Direct | |---|---:|---:|---:| | **JPO** | **0.850** | **0.641** | **0.821** | | p-GAPO | 0.848 | 0.607 | 0.773 | | Linear | 0.836 | 0.593 | 0.773 | | PCGrad | **0.850** | 0.590 | 0.781 | | D3PO-LSW | 0.831 | 0.588 | 0.767 | | GRPO (fixed) | 0.838 | 0.571 | 0.750 | | GDPO (fixed) | 0.784 | 0.524 | 0.679 | **Gradient-conflict evidence.** Across runs logging the Gramian, gold–diamond gradient cosine averages **−0.521** and is negative in **99.0% of observations**. We also measure the share of gradient steps whose aggregated direction has cosine below −0.01 against any active objective **before AdamW**: | Aggregator | Steps | Conflicting direction | |---|---:|---:| | **JPO (UPGrad)** | 1,142 | **0.1%** | | Linear | 1,163 | 11.8% | | PCGrad | 1,163 | 2.0% | | p-GAPO | 1,166 | 11.3% | Together, the controlled ablations and gradient diagnostics establish that UPGrad produces a meaningfully different, substantially less conflicting update direction, with modest improvements in held-out preference responsiveness relative to ordinary weighted gradients.
AuthorsOpenReview ↗

Official Response to Reviewer 5Yfj [3/4]: Runtime, Memory Complexity, and Equal-Compute Comparisons (Weakness 2)

2. Runtime and memory overhead (Weakness 2, Question 2)

We agree that objective-specific gradient computations introduce overhead. We therefore measured training-step latency, actor-update time, peak allocated GPU memory, and full-run wall-clock time.

Table 4. Computational costs over 1,200 training steps.

MethodStep (s)Generation (s)Actor update (s)Peak GPU memory (GB)Wall time (h)
JPO (UPGrad)113.774.333.119.137.4
Linear113.774.332.319.136.0
PCGrad115.376.333.319.136.6
p-GAPO107.672.330.119.132.9
D3PO-LSW90.574.39.914.428.6
GRPO (corrected)86.972.39.022.427.0
GDPO (corrected)85.270.38.822.425.2

Step, and actor-update times are medians; peak memory is the observed maximum, and wall time covers the complete run. All methods use H100 hardware. These measurements reflect our specific LoRA-based implementation and hardware conditions, rather than implementation-independent efficiency guarantees.

Computational overhead primarily arises from objective-specific gradient computation, rather than UPGrad aggregation itself. Compared with the identical-loss Linear baseline, JPO increases actor-update time from 32.3s to 33.1s and total wall time from 36.0h to 37.4h (3.9% overhead). Both methods execute 40 backward passes per actor update. Relative to GDPO, however, JPO requires 48.4% more wall time.

Memory overhead is comparatively modest. JPO, Linear, PCGrad, and p-GAPO all reach 19.1 GB peak allocated GPU memory, versus 14.4 GB for D3PO-LSW and 22.4 GB for GRPO/GDPO. Thus, UPGrad introduces no observed additional peak-memory cost relative to the closest multi-gradient baselines.

Theoretical memory complexity. Let MM be the number of objectives and PP the number of trainable parameters. Our implementation explicitly stores the Jacobian J∈RM×PJ\in\mathbb{R}^{M\times P} and Gram matrix G=JJ⊤∈RM×MG=JJ^\top\in\mathbb{R}^{M\times M}, requiring O(MP+M2)O(MP+M^2) gradient-related storage, approximately linear in MM when P≫MP\gg M.

However, storing the full Jacobian is not an inherent requirement JPO. The Gramian formulation of Jacobian Descent (Quinton & Rey, 2025) allows GG to be accumulated incrementally over parameter blocks. UPGrad coefficients can then be computed from GG alone, and the aggregated gradient recovered through a weighted backward pass. This reduces persistent aggregation storage to O(M2)O(M^2), excluding the final O(P)O(P) gradient and intermediate activations. While not implemented in our experiments, this offers a route to lower memory requirements for larger models.

3. Equal-compute comparison (Weakness 2, Question 2)

We agree that equal optimizer steps do not imply equal computational cost. We therefore compare each baseline's final checkpoint (step 1,200) against an earlier JPO checkpoint trained for approximately the same wall-clock duration. This allows us to assess whether faster methods can achieve comparable performance by completing more training steps.

For consistency, checkpoint times are estimated by scaling JPO's measured full-run wall time:

TJPO(s)≈37.4s1200 hours.T_{\mathrm{JPO}}(s)\approx 37.4\frac{s}{1200}\text{ hours}.

For each baseline, we select the latest available JPO checkpoint within its measured full-run wall-clock budget. All comparisons use the same 50 held-out mazes.

Table 5. Approximately compute-matched held-out MaZE comparisons (higher is better).

ComparisonJPO / baseline time (h)Steering (JPO / baseline)Direct switching (JPO / baseline)
JPO step 800 vs GDPO step 1,20024.9 / 25.20.488 / 0.4680.674 / 0.654
JPO step 800 vs GRPO step 1,20024.9 / 27.00.488 / 0.5530.674 / 0.741
JPO step 900 vs D3PO-LSW step 1,20028.1 / 28.60.530 / 0.5730.727 / 0.766
JPO step 1,000 vs p-GAPO step 1,20031.2 / 32.90.539 / 0.5830.732 / 0.756
JPO step 1,100 vs Linear step 1,20034.3 / 36.00.593 / 0.5430.785 / 0.727
JPO step 1,100 vs PCGrad step 1,20034.3 / 36.60.593 / 0.5540.785 / 0.762

JPO outperforms GDPO, Linear, and PCGrad under approximately matched wall-clock budgets, achieving higher steering and direct-switching scores despite using slightly less estimated training time. However, GRPO, D3PO-LSW, and p-GAPO outperform the corresponding earlier JPO checkpoints. Checkpoint times are estimated from full-run wall time assuming roughly uniform per-step cost, and evaluations are available at 100-step intervals.

Overall, JPO demonstrates measurable benefits from geometry-aware gradient aggregation, particularly against its identical-loss Linear control, while the increased cost of objective-specific gradients remains an important trade-off.

Reviewer_TG6KOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies LLM reinforcement learning with multiple rewards. The authors point out that existing methods collapse multiple rewards into a single scalar too early, and that directly taking a weighted sum makes the objectives interfere with each other. To address this, the paper proposes JPO (Jacobian Policy Optimization), which lets each reward produce its own gradient and then merges the gradients according to preferences while accounting for conflicts. Specifically, the authors normalize each reward dimension within the group separately and construct an independent clipped PPO loss for each dimension. They compute the gradient of each loss to obtain a Jacobian matrix, and then apply preference-weighted UPGrad to remove the conflicting components among the gradients and aggregate them into a single update direction. Experiments use Qwen3.5-4B with LoRA. On the MaZE maze navigation task, JPO achieves higher matched-preference utility than GRPO and GDPO. On MATH Level-5, with correctness, conciseness, and rigor as rewards, JPO reaches a single-sample accuracy of 78.6%, compared with 74.6% for GRPO.

Critical strengths
  1. The paper clearly compares how prior methods such as GRPO, GDPO, and D³PO compute multi-objective rewards, which is simple and easy to follow.
  2. The method is relatively simple to implement and easy to transfer to other settings.
Critical weaknesses
  1. Limited novelty. Work on gradient allocation for multi-reward objectives already existed as early as 2025 [1], yet the authors neither discuss nor compare against it.
  2. The motivation lacks a quantitative metric. The paper argues that rewards from different dimensions conflict when multiple objectives are combined, but it provides no quantitative metric for how large this conflict actually is. This makes it difficult for readers to clearly understand how much can be gained by resolving the conflict.
  3. The experimental design does not match the paper’s claims. Since this is a multi-objective optimization method, there should be metrics showing that the proposed method is effective across multiple dimensions. However, on MATH the paper reports only accuracy, and on the maze task it reports only two weighted scalar metrics, which are largely determined by the completion rate. These do not reflect the metrics of the other dimensions. The motivation of the authors’ method is therefore not supported by the experiments.

[1] Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models. ACL 2025.

Questions for authors

See above

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer TG6K [1/5]: Important Correction to MaZE Baselines

We thank the reviewer for their detailed assessment and constructive feedback. We appreciate their recognition of the method's simplicity and transferability. We agree that the original submission would benefit from stronger comparisons with related methods, quantitative analysis of gradient interference, and more direct evaluation of individual reward dimensions. We have conducted extensive additional experiments to address these concerns.

Important correction to the original MaZE baseline results

During our additional experiments, we identified an implementation issue affecting the original GRPO and GDPO MaZE training runs. Specifically, the recomputation of PPO old-policy log-probabilities used a different microbatch shape from the actor forward pass. In our Qwen3.5/veRL configuration, this caused numerical differences in token log-probabilities even when the model parameters were unchanged, producing spurious deviations from the expected PPO probability ratio of one.

We corrected this issue by matching score_batch_size to the rollout group size G=8G=8, ensuring consistent forward batch shapes. We verified that the old-versus-current probability ratio returned to one within numerical precision, and reran the affected GRPO and GDPO experiments using the corrected configuration. The MATH experiments were unaffected.

The corrected GRPO and GDPO MaZE baselines perform substantially better than those reported in the original submission. We therefore withdraw the original MaZE baseline comparisons and replace them with the corrected results presented. Importantly, the corrected comparisons show that several baselines are considerably more competitive with JPO than the original results suggested. We have adjusted our empirical conclusions accordingly.

AuthorsOpenReview ↗

Official Response to Reviewer TG6K [2/5]: Novelty, GAPO Comparison, and Additional Baselines (Weakness 1)

1. Novelty and relationship to GAPO (ACL 2025) (Weakness 1)

We thank the reviewer for highlighting Gradient-Adaptive Policy Optimization (GAPO; Li et al., ACL 2025). We acknowledge that this is relevant prior work and that its omission from our related-work discussion was an oversight.

Although GAPO and JPO share the motivation of addressing multi-objective gradient interference, they differ in both their gradient aggregation mechanisms and policy-learning formulations.

First, the gradient aggregation mechanisms differ. GAPO extends Multiple Gradient Descent (MGDA) by normalizing individual objective gradients before solving a minimum-norm optimization problem. Its preference-aware extension, P-GAPO, instead uses a preference-weighted sum of normalized gradients:

=\sum_i p_i\frac{g_i}{\|g_i\|_2}.$$ While MGDA identifies a common descent direction when one exists, P-GAPO's preference-weighted sum does not retain this guarantee. The resulting direction can still conflict with individual objective gradients. In contrast, JPO uses UPGrad, which projects individual gradients onto their common dual cone before preference-weighted aggregation: $$d_{\mathrm{JPO}} = \sum_i p_i \Pi_{\mathcal{K}}(g_i).$$ Here, the common dual cone is defined as $$\mathcal{K} = \{ d \mid g_i^\top d \geq 0,\ \forall i \}.$$ Since the cone is convex, nonnegative combinations of the projected gradients remain within it. Consequently, $$g_i^\top d_{\mathrm{JPO}} \geq 0,\qquad \forall i.$$ Since the dual cone is convex, nonnegative combinations of the projected gradients remain within it. In exact arithmetic, this gives $$g_i^\top d_{\mathrm{JPO}}\geq0,\qquad\forall i.$$ Thus, unlike P-GAPO, JPO incorporates preference weighting while retaining the non-conflicting property of the aggregated direction with respect to the individual surrogate gradients. This is a first-order property and does not imply monotonic improvement in realized rewards after finite AdamW updates. **Second, the policy-learning formulations differ.** P-GAPO evaluates separately optimized policies under different preference vectors. JPO trains a single shared policy explicitly conditioned on the requested preference, allowing it to switch between preferences at inference time without retraining. Furthermore, JPO combines objective-wise group-relative advantage estimation, independently clipped PPO losses, and geometric aggregation over the trainable LoRA parameters. GAPO's reported implementation uses last-layer gradients as a computational proxy. We therefore do not claim novelty in multi-objective gradient optimization itself. Our contribution is the integration of UPGrad's non-conflicting aggregation into preference-conditioned LLM policy optimization, together with its empirical evaluation on structured reasoning tasks. **Additional baseline experiments.** To address the missing comparisons, we trained additional baselines: ordinary preference-weighted gradient aggregation (Linear), P-GAPO, PCGrad, and D3PO-LSW, alongside GRPO and GDPO. These baselines can be divided into two categories based on how they aggregate the objectives. **GRPO, GDPO, and D3PO-LSW scalarize before differentiation**, combining rewards, normalized advantages, or separately clipped PPO losses, respectively, into a single scalar objective before computing its gradient. In contrast, **Linear, PCGrad, P-GAPO, and JPO explicitly compute objective-specific gradients before aggregation**. Among these, Linear performs an ordinary preference-weighted sum, while PCGrad, P-GAPO, and JPO use gradient-level information to modify the update direction. This distinction enables us to evaluate both the benefit of retaining objective-specific gradients and the additional contribution of UPGrad's conflict-aware aggregation. The **Linear baseline directly isolates UPGrad's contribution**, since the objective-specific losses remain unchanged and only the aggregation rule differs. The held-out comparisons are presented below (in part 3), and the gradient diagnostics below quantify the different levels of interference produced by these methods.
AuthorsOpenReview ↗

Official Response to Reviewer TG6K [3/5]: Quantifying Gradient Conflicts and UPGrad's Effectiveness (Weakness 2)

2. Quantifying gradient conflicts (Weakness 2)

We agree that the motivation would be strengthened by direct measurements of gradient interference during training.

To address this limitation, we logged the pairwise cosine similarities of the gradients in our additional MaZE experiments, allowing us to measure conflict frequencies and the alignment of the final aggregated direction with individual objective gradients.

Pairwise gradient conflicts.

The following table summarizes selected pairs across the gradient-logging runs (JPO, Linear, P-GAPO, and two PCGrad runs):

Objective pairMean cosineNegative cosine (co-active observations)
Gold – Diamond−0.52199.0%
Gold – Length+0.03637.9%
Diamond – Length+0.06434.6%
Completion – Gold+0.7340.6%
Completion – Diamond+0.7340.7%
Completion – Lava avoidance+0.9980.0%

Co-active observations are training steps where both objective gradients are nonzero.

These measurements confirm that gradient conflict occurs in practice and is concentrated in specific objective pairs. In particular, gold and diamond gradients are strongly opposed whenever both are active, consistent with the deliberate trade-off in the MAZE environment. Conversely, completion and lava avoidance are nearly perfectly aligned. The percentages in the rightmost column are measured on training steps where both gradients are non-zero. At a fixed step, some objectives can have zero gradients when all eight sampled responses receive identical rewards, and these are excluded from the denominator in the percentage calculation.

Conflict in the aggregated direction.

We also measured whether the final aggregated direction dd is negatively aligned with any active objective gradient:

cmin⁡=min⁡i:∥gi∥2>0gi⊤d∥gi∥2∥d∥2.c_{\min} = \min_{i:\|g_i\|_2>0} \frac{g_i^\top d}{\|g_i\|_2\|d\|_2}.

MethodValid stepsAny negative cosineCosine below −0.01
JPO (UPGrad)1,1422.5%0.1%
PCGrad1,1633.3%2.0%
P-GAPO1,16611.7%11.3%
Linear1,16312.3%11.8%

These measurements concern aggregated gradients before AdamW, not the actual optimizer updates. The small residual violations for UPGrad reflect numerical approximation.

JPO nearly eliminates meaningful negative alignment with individual objectives. In comparison, ordinary weighted aggregation and P-GAPO produce conflicting directions substantially more frequently. PCGrad also reduces interference, but its observed directions exhibit more substantial negative alignment than UPGrad.

These diagnostics provide direct quantitative evidence for the motivation of JPO and demonstrate that its gradient aggregation mechanism substantially reduces interference compared with conventional weighted aggregation.

Reviewer_X2nQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author propose Jacobian Policy Optimization which trains a LLM to balance multiple rewards according to requested preferences. It constructs a separate policy loss for each reward, uses the existing UPGrad method to project away conflicts between their gradients, and combines the adjusted gradients with preference weights. On Qwen3.5-4B, it substantially improves held-out maze utility and raises individual-sample math accuracy from GRPO’s 74.64% to 78.60%.

Critical strengths

The idea make intuitive sense, and math derivation is concrete.

The experiment and evaluation setup is sound.

The improvements seems consistent in multiple task settings.

Critical weaknesses

The experiment does not isolate why JPO works. JPO changes both per-objective loss construction and gradient aggregation. There is no comparison using the same separately clipped losses with an ordinary weighted gradient sum.

Extra training cost is not measured and compared rigorously.

Questions for authors

The paper says that compute and memory grow with the number of objectives. Would JPO actually lead to a better performance vs cost Pareto frontier?

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer X2nQ [1/4]: Corrected Baselines and UPGrad Ablations (Weakness 1)

We thank the reviewer for their positive assessment of the motivation, mathematical formulation, and experimental results, as well as their constructive suggestions. We agree that isolating UPGrad's contribution and quantifying its computational overhead are important for evaluating JPO.

1. Correction to the MaZE baselines and additional ablations to isolate why JPO works (weakness 1)

During our reproduction experiments, we identified a silent issue in the Qwen3.5/veRL training stack affecting the original GRPO and GDPO MaZE runs.

The issue arose because PPO old log-probabilities were recomputed using a different microbatch shape from the actor forward pass. Under this configuration, Qwen3.5/veRL produced batch-shape-dependent token log-probabilities, resulting in spurious old-versus-current policy probability ratios even when the model weights were unchanged. Setting score_batch_size equal to the rollout group size (G=8) made the forward-pass batch shapes consistent and restored the expected probability ratio of 1 (up to numerical precision). We have corrected this issue in the new MaZE reruns. The original GRPO and GDPO MaZE results are therefore superseded by the corrected results reported below, which we will incorporate into the revised manuscript. The MATH experiments were not affected.

We also conducted additional experiments with four new baselines (Linear, D3PO-LSW, PCGrad, and P-GAPO). All 6 baselines are as follows:

  1. Linear: Identical objective-specific advantages, separately clipped PPO losses, and Jacobian computation to JPO, replacing UPGrad with the ordinary weighted gradient sum (d=J⊤pd=J^\top p).
  2. D3PO-LSW: The late-stage weighting component of D3PO, combining separately clipped losses using a single scalar backward pass. This implementation excludes D3PO's diversity regularizer and multi-head critic, and differs from JPO in advantage normalization.
  3. PCGrad: Replaces UPGrad with sequential gradient conflict projections.
  4. P-GAPO: Uses preference-weighted, unit-normalized objective gradients, without conflict projection.
  5. GRPO: Scalarize rewards before differentiation.
  6. GDPO: Normalizes advantages before differentiation.

These baselines can be divided into two categories based on how they aggregate the objectives. GRPO, GDPO, and D3PO-LSW scalarize before differentiation. They do so by combining rewards, normalized advantages, or separately clipped PPO losses, respectively, into a single scalar objective before computing its gradient. In contrast, Linear, PCGrad, P-GAPO, and JPO-UPGrad explicitly compute objective-specific gradients before aggregation. This distinction enables us to evaluate both the benefit of retaining objective-specific gradients and the additional contribution of UPGrad's conflict-aware aggregation. The Linear baseline directly isolates UPGrad's contribution, since the objective-specific losses remain unchanged and only the aggregation rule differs.

AuthorsOpenReview ↗

Official Response to Reviewer X2nQ [2/4]: Gradient Conflicts and Preference Responsiveness

2. Gradient conflicts and preference responsiveness

The MaZE environments were specifically constructed to introduce competing objectives. Gold and diamond collection are symmetric in their generation and reward definitions: both are placed in regions near opposite corners, each contains 3–5 collectible items. The environment imposes a movement budget that permits an exit-reaching detour through either the gold or diamond region, but not through both designated collection corners. Thus, gold and diamond are symmetric but deliberately conflicting objectives: prioritizing one requires sacrificing collection opportunities for the other.

Our gradient measurements confirm this conflict. Across runs with recorded objective-gradient Gramians, the gold and diamond gradients have a mean cosine similarity of −0.521, with negative similarity in 99.0% of observations where both are active. In comparison, completion and lava-avoidance gradients are nearly perfectly aligned (mean cosine +0.998).

Objective pairMean cosineNegative cosine (%)
Gold – Diamond−0.52199.0%
Gold – Length+0.03637.9%
Diamond – Length+0.06434.6%
Completion – Gold+0.7340.6%
Completion – Diamond+0.7340.7%
Completion – Lava avoidance+0.9980.0%

Preference responsiveness metrics. To assess whether conflict resolution translates into preference-conditioned behavior, we introduce two complementary metrics.

Let rˉi∣j\bar r_{i|j} denote the mean reward for objective ii when the policy is conditioned on preference jj, where G,D,C,LG,D,C,L denote gold-heavy, diamond-heavy, completion-heavy, and lava-avoidance-heavy preferences, respectively.

The first metric, preference steering, measures whether emphasizing an objective increases its corresponding reward relative to the three other conditioning preferences.

For gold and diamond separately:

\frac{\bar r_{G|C}+\bar r_{G|D}+\bar r_{G|L}}{3},$$ $$S_D=\bar r_{D|D}- \frac{\bar r_{D|C}+\bar r_{D|G}+\bar r_{D|L}}{3}.$$ The combined preference steering score is $$\boxed{ S_{\mathrm{steering}}=\frac{S_G+S_D}{2}. }$$ The second metric, **direct gold–diamond switching**, focuses specifically on the two conflicting collection objectives. Rather than comparing against all three alternative preferences, it measures how strongly the policy changes its gold-versus-diamond reward balance when switching directly between gold-heavy and diamond-heavy instructions: $$\boxed{ S_{\mathrm{direct}}=\frac12\left[ (\bar r_{G|G}-\bar r_{G|D})+ (\bar r_{D|D}-\bar r_{D|G}) \right]. }$$ The first term measures the increase in gold collection when gold rather than diamonds is prioritized; the second measures the corresponding increase in diamond collection when diamonds are prioritized. Because the two collection objectives are symmetric and intentionally conflicting, this second metric provides a targeted measure of the policy's ability to respond to competing priorities. Both scores are higher when the requested preference produces a stronger corresponding behavioral change. They quantify preference responsiveness rather than overall utility or Pareto-front quality. **Table 2: Final-checkpoint held-out evaluation.** Results are averaged over 50 held-out mazes, each evaluated under all four conditioning preferences. | Method | Steering ↑ | Direct switching ↑ | Completion ↑ | |---|---:|---:|---:| | **JPO (UPGrad)** | 0.572 | **0.773** | 0.859 | | Linear | 0.543 | 0.727 | 0.828 | | D3PO-LSW | 0.573 | 0.766 | 0.849 | | PCGrad | 0.554 | 0.762 | **0.864** | | P-GAPO | **0.583** | 0.756 | 0.855 | | GRPO (corrected) | 0.553 | 0.741 | 0.859 | | GDPO (corrected) | 0.468 | 0.654 | 0.798 | Compared with Linear, JPO improves held-out direct switching from 0.727 to 0.773 and completion from 0.828 to 0.859, providing evidence that UPGrad contributes beyond objective-specific loss construction. JPO also achieves the highest direct switching score among the evaluated final checkpoints. However, P-GAPO obtain a slightly higher steering score, while D3PO-LSW is comparable.
AuthorsOpenReview ↗

Official Response to Reviewer X2nQ [3/4]: Computational Overhead, Memory, and Scalability (Weakness 2)

3. Computational and memory overhead

We instrumented the training runs to measure actor-update time, GPU memory allocation, and end-to-end training time.

Table 3: Observed training costs over 1,200 optimization steps. Step and actor-update times are reported as medians across training steps, while peak memory is the maximum logged CUDA allocated memory. All methods were trained on H100 GPUs.

MethodStep time (s)Actor update (s)Peak memory (GB)Total wall time (h)
JPO (UPGrad)113.733.119.137.4
Linear113.732.319.136.0
D3PO-LSW90.59.914.428.6
PCGrad115.333.319.136.6
P-GAPO107.630.119.132.9
GRPO86.99.022.427.0
GDPO85.28.822.425.2

Computational overhead. We distinguish between the cost of computing objective-specific gradients and the additional cost of UPGrad aggregation. Compared with Linear, which constructs the same Jacobian but applies ordinary preference-weighted aggregation, JPO increases median actor-update time only from 32.3 to 33.1 seconds (approximately 2.5%). This suggests that UPGrad's geometry-aware aggregation introduces relatively little computational overhead once the objective-specific gradients have been computed.

The primary overhead therefore arises from Jacobian construction. Compared with D3PO-LSW, which combines separately clipped objective losses before differentiation and requires only a single backward pass, JPO's actor update is approximately 3.3 times slower. However, the increase in end-to-end training time is substantially smaller, at approximately 31% (37.4 versus 28.6 hours), since rollout generation and other operations account for a considerable fraction of training time.

Memory overhead. Despite explicitly constructing the Jacobian, JPO does not exhibit uniformly higher peak GPU memory consumption than the scalarized baselines. JPO, Linear, PCGrad, and P-GAPO all have identical observed peak allocations of 19.1 GB, suggesting that the choice of gradient aggregation method introduces little additional peak memory beyond their shared Jacobian construction.

Relative to D3PO-LSW, JPO exhibits approximately 32.6% higher peak allocated memory (19.1 versus 14.4 GB), consistent with the additional storage requirements of objective-specific gradients. Conversely, its observed peak allocation is approximately 14.7% lower than that of GRPO and GDPO (22.4 GB). These results indicate that Jacobian-based optimization does not necessarily incur a larger GPU memory footprint than scalarized methods under the evaluated configurations.

Theoretical scalability. Let MM denote the number of optimization objectives and PP the number of trainable parameters. Our current implementation explicitly accumulates the objective-gradient Jacobian J∈RM×PJ \in \mathbb{R}^{M \times P}, requiring O(MP)O(MP) memory, together with its Gram matrix G=JJ⊤∈RM×MG = JJ^\top \in \mathbb{R}^{M \times M}, requiring O(M2)O(M^2) memory. Thus, the objective-dependent gradient storage complexity is O(MP+M2)O(MP + M^2), which is approximately linear in MM when P≫MP \gg M.

Importantly, materializing the full Jacobian is not an inherent requirement of JPO. As briefly noted in the limitations section. The Gramian formulation of Jacobian Descent (Quinton & Rey, 2025), implemented in the TorchJD library, permits computing the Gram matrix incrementally from parameter-block contributions without storing the complete Jacobian. Since UPGrad determines its aggregation coefficients using only the Gramian matrix, the final aggregated direction can subsequently be recovered through a weighted backward pass. This reduces persistent aggregation storage to O(M2)O(M^2), although intermediate gradient computations, model activations, and the final parameter gradient still require additional memory. Consequently, the O(MP)O(MP) storage overhead of our current implementation is not a fundamental requirement of the underlying algorithm.

While our experiments do not establish scalability to substantially larger numbers of objectives or full-parameter fine-tuning, they demonstrate that the additional cost of UPGrad aggregation is modest relative to Jacobian construction in our LoRA-based setting. Moreover, the Gramian-based formulation provides a potential route to substantially reducing JPO's memory requirements in future implementations, particularly for larger models.

Finally, we observed variation in GPU operating conditions, including power and clock frequencies, alongside possible implementation-level efficiency differences. Accordingly, the reported timings and memory measurements characterize resource consumption under our specific hardware and software configurations, rather than implementation-independent algorithmic complexity.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary for reviewers and AC

We sincerely thank all reviewers for their valuable comments and constructive suggestions. To facilitate discussion among the reviewers and the area chair, we have summarized the reviewers' feedback in the table below:

ReviewerClear / Intuitive Method (strength 1)Encouraging Empirical Results (strength 2)Practical Relevance (strength 3) / Transferability (strength 4)Sound Experimental Setup (strength 5)Novelty / Related Work (weakness 1)Missing Ablations / Attribution (weakness 2)Gradient Conflict Quantification (weakness 3)Per-Objective Evaluation (weakness 4)Computational Cost / Scalability (weakness 5)
X2nQ✅✅✅❌❌
TG6K✅✅❌❌❌
5Yfj✅✅❌❌❌
F3pZ✅✅❌❌

We also provide a short summary of what we did to address each of the weaknesses brought up:

Correction to the original MaZE results: (all) During our reproduction experiments, we discovered a batch-shape-dependent numerical issue in the Qwen3.5/veRL training stack that affected the original GRPO and GDPO MaZE runs, producing incorrect PPO probability ratios. We have reran all our experiments, corrected this issue, and are superseding the original GRPO/GDPO MaZE results with verified reruns. This is in addition to our four additional baselines (Linear, D3PO-LSW, PCGrad, and P-GAPO). The MATH experiments were unaffected.

Weakness 1 - Novelty / Related Work: (TG6K, 5Yfj) We acknowledged the previously omitted GAPO (ACL 2025) work, and implemented preference weighted GAPO as an additional baseline. We also clarified JPO's distinguishing contribution, which is applying UPGrad's non-conflicting gradient aggregation to preference-conditioned LLM training (rather than introducing gradient-based multi-objective optimization itself) and showing that this leads to increased steerability of preferences during inference time. We also compare against other close baselines using different scalarization methods (D3PO-LSW, GRPO, GDPO) and other gradient aggregation methods (PCGrad, GAPO, Linear).

Weakness 2 - Missing Ablations / Attribution: (X2nQ, 5Yfj, F3pZ) Our four additional baseline experiments (Linear, D3PO-LSW, PCGrad, and P-GAPO) resolves this. We included a controlled Linear ablation that uses identical objective-specific losses to JPO but replaces UPGrad with ordinary weighted gradient aggregation. On MaZE, JPO improves direct gold–diamond switching from 0.727 to 0.773 over Linear, providing evidence for improved preference steerability during inference time.

Weakness 3 - Gradient Conflict Quantification: (TG6K) We recorded objective-gradient Gramians during MaZE training and found substantial conflict between gold and diamond gradients (mean cosine similarity −0.521). We further demonstrate that UPGrad reduces the proportion of steps where the aggregated direction conflicts with an active objective gradient from 11.8% for Linear to just 0.1% for JPO.

Weakness 4 - Per-Objective Evaluation: (TG6K) We expanded our MaZE evaluation to report individual reward outcomes and introduced two additional metrics, preference steering and direct gold–diamond switching, to measure the responsiveness of the model to different preferences during inference time. JPO achieves the highest final-checkpoint direct switching score, while P-GAPO achieves slightly higher steering. We also added MATH experiments reporting individual correctness, rigour, conciseness, and format rewards. We found that that JPO achieves the highest final-window correctness and rigour rewards during training, while GRPO leads on conciseness and format. This is consistent with earlier results in the GDPO paper (Liu et al., 2026) that both GRPO and GDPO tend to focus on the easier rewards (conciseness and format) first. JPO offers a more balanced approach.

Weakness 5 - Computational Cost / Scalability: (X2nQ, 5Yfj, F3pZ) We measured training time, actor-update latency, and peak GPU memory across all seven methods, finding that UPGrad adds only approximately 2.5% actor-update overhead relative to the identical-Jacobian Linear baseline (33.1 vs. 32.3 seconds), with identical observed peak memory (19.1 GB). On an equal training-time comparison, JPO compares favourably against GDPO, PCGrad, and Linear weighting baselines, and slightly worse against GRPO, D3PO, and P-GAPO baselines. Unfortunately we do not have the GPU compute resources to conduct experiments with full fine tuning (instead of LoRA) and with larger models. Instead, we also provide a short theoretical analysis on the memory complexity of JPO. We also highlight the fact that storing the full Jacobian in memory is not a requirement of JPO, and this offers a potential route to substantially reduce JPO's memory requirements when applied with full fine tuning on larger models.

AuthorsOpenReview ↗

Summary of Changes to the Revised Manuscript

We sincerely thank all reviewers and the Area Chair for their valuable feedback throughout the rebuttal process.

We have uploaded an updated manuscript incorporating the additional experiments, analyses, and clarifications discussed in our responses. All changes are highlighted in red in the revised manuscript.

For ease of reference, we provide a consolidated changelog below. All line numbers refer to the revised manuscript.

1. Abstract and Introduction

  • Abstract (Lines 22-24): Removed mention of the matched preference utility metric. Added in P-GAPO comparison.

  • Related methods (Lines 54-55): Briefly mentioned P-GAPO.

  • MaZE results (Lines 87-90): Changed reported metrics to switching and steering. Finding remains unchanged.

  • Mathematical reasoning experiments (Lines 93–98): Added a summary of our new DeepScaleR-Preview training experiments

  • Contributions (Lines 107-115): Revised the contributions to emphasize preference-aware gradient conflict resolution, updated the MaZE claims to reflect comparisons against the expanded set of baselines, and incorporated the findings from DeepScaleR-Preview training.

2. Related Work

  • GAPO and P-GAPO (Lines 139-144): Expanded the discussion of Gradient-Adaptive Policy Optimization (GAPO, ACL 2025)

3. Methodology and Experiments

Gradient conflict analysis

  • New Table 1 (Lines 324-335): Added quantitative measurements of pairwise cosine similarity between objective-specific policy gradients.

  • Gradient conflict discussion (Lines 375-405): Added an analysis of the observed gradient conflicts during MaZE training.

Clarification of the two experimental settings

  • Experimental overview (Lines 349-357): Expanded the explanation of how MaZE and MATH investigate different aspects of multi-reward optimization.

Revised MaZE experiments

  • Updated MaZE results (Lines 378-390): Replaced the original MaZE result tables and figures with updated experimental results, incorporating the corrected GRPO/GDPO runs and four additional baselines: Linear, D3PO-LSW, PCGrad, and P-GAPO.

  • Evaluation metrics and results (Lines 406-421): Rewrote the MaZE evaluation and results discussion around individual reward outcomes, preference steering, and direct gold–diamond switching.

Additional MATH experiments

  • DeepScaleR-Preview training (Lines 490-498): Added a new experiment training JPO, GRPO, and GDPO for one pass over DeepScaleR-Preview, evaluating correctness, rigour, conciseness, and formatting rewards.

Computational costs

  • Computational cost analysis (Lines 500-506): Added a discussion of computational overhead and detailed empirical measurements of training runtime and GPU memory consumption. Approximately compute-matched comparisons are also provided in Appendix D.

4. Discussion, Conclusion, and Future Work

  • UPGrad ablation: Removed the previous statement identifying isolation of UPGrad's contribution as future work, since this experiment has now been completed through the controlled Linear ablation.

  • Future Work (Lines 528–539): Revised and expanded the previous limitations discussion. In addition to computational scalability, we now discuss the challenge of identifying and designing meaningful, independently informative reward dimensions. While JPO addresses how multiple reward signals can be optimized jointly, determining which reward dimensions best describe desirable behavior remains an important direction for future research.

5. Appendix Revisions

  • Appendix B.7 — Strong-Preference Evaluation (Lines 993–1059): Replaced the original MaZE metric definitions with the new evaluation framework, including formal definitions and explanations of preference steering and direct gold–diamond switching.

  • Appendix C.11 — DeepScaleR-Preview Training (Lines 1500–1550): Added detailed experimental results on the evolution of individual reward dimensions during training. This includes early, middle, and final training-window comparisons, response-length and truncation statistics, and held-out mathematical reasoning accuracy.

  • Appendix D — Computational Cost and Scalability: Expanded the computational analysis to include empirical training runtime and GPU memory measurements, approximately compute-matched comparisons, and theoretical memory-complexity analysis.

We thank all reviewers and the Area Chair again for their constructive feedback, which has been invaluable in improving the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.