- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Jacobian Policy Optimization (JPO), which aims primarily to address the problem of how language models should adjust their behavior according to user preferences when faced with multiple rewards simultaneously. The paper first normalizes each reward individually, then constructs separate policy losses and gradients for each, and finally utilizes UPGrad to resolve gradient conflicts between different objectives, applying the results to model updates. The authors utilized Qwen3.5-4B and conducted tests on maze navigation and MATH Level-5 math problems. In both the maze and math experiments, the model achieved higher single-sample accuracy.
Critical strengths
The issues addressed in this paper are of practical significance. The training of language models often takes into account correctness, conciseness, and other requirements simultaneously, and these objectives may influence one another. The authors attempt to preserve the gradient information for each objective prior to updating the model, which offers a promising direction for research into multi-objective training. The paper demonstrates notable improvements on certain metrics. JPO achieves an average preference-matching utility of 0.595 on the maze task, which is higher than GRPO’s 0.288 and GDPO’s 0.271, with improvements observed across all four preference conditions. JPO’s single-sample mathematical accuracy stands at 78.60 per cent, which is also higher than GRPO’s 74.64 per cent and GDPO’s 73.60 per cent.
Critical weaknesses
-
The contribution of UPGrad, a central component of the method, has not been independently validated. The authors themselves acknowledge in sec 5, lines 471–472 that an ablation isolating UPGrad’s contribution remains future work. For a method whose central claim concerns gradient geometry, this is a question the current paper should address.
-
The paper already discusses D³PO, which constructs a separate clipped loss for each objective before applying a weighted sum. This baseline would better isolate the incremental contribution of JPO’s gradient processing than GRPO or GDPO, yet it is not included in the experiments.
-
The method requires computing separate gradients for multiple objectives. Although the authors acknowledge the additional computational and memory overhead, they do not report training time, peak memory usage, or comparisons under the same compute budget. The experiments are also largely limited to LoRA training of Qwen3.5-4B, leaving its practicality for full-parameter fine-tuning or larger models insufficiently established.
-
Does delaying scalarization offer any particular benefits? The paper first acknowledges that fixed weighted aggregation commutes with differentiation (lines 56–58). However, it later attributes improvements to preserving objective-specific signals through differentiation (lines 445–447 and 465–471). Delaying aggregation alone does not change the update; the substantive differences lie in objective-specific clipping and nonlinear UPGrad projection. Ablations are needed to distinguish their contributions and support this interpretation.
Questions for authors
See Weaknesses.
6 comments
Official Response to Reviewer F3pZ [2/5]: UPGrad Ablation and Gradient Conflict Analysis (Weaknesses 1 & 4)
1. Isolating UPGrad's contribution (Weaknesses 1 and 4)
We agree that the original experiments did not fully disentangle objective-specific loss construction from gradient aggregation. Our additional experiments now directly address this distinction. UPGrad's contribution is that it almost entirely eliminates meaningful first-order gradient conflicts.
Given objective-specific losses , gradients , and preference weights , ordinary weighted aggregation produces
JPO instead projects each gradient onto the common non-conflicting cone before aggregation:
where
These directions coincide when the objective gradients are mutually non-conflicting. Otherwise, UPGrad can produce a different direction that avoids first-order interference between objectives.
New ablation: Linear weighted gradients. We trained a baseline using exactly the same objective-wise normalization, independently clipped PPO losses, preferences, and training configuration as JPO, replacing only UPGrad with an ordinary preference-weighted gradient sum. This directly isolates the gradient aggregation mechanism.
Quantifying gradient conflicts. We analyzed the Gramian of the five objective-specific LoRA gradients (completion, gold, diamond, lava avoidance, and response length), pooled across runs. The strongest conflict occurs between gold and diamond:
| Objective pair | Mean gradient cosine | Negative cosine frequency |
|---|---|---|
| Gold–Diamond | −0.521 | 99.0% |
| Gold–Length | +0.036 | 37.9% |
| Diamond–Length | +0.064 | 34.6% |
| Completion–Lava avoidance | +0.998 | 0.0% |
Negative cosine indicates that descending along one objective gradient would locally increase the other objective's loss. Gold–diamond conflict is therefore substantial and persistent, rather than merely hypothetical.
We next measured whether the combined gradient direction conflicts with any active objective gradient. Specifically, we calculate the minimum cosine between the combined direction and each nonzero objective gradient, and report the fraction of training steps where this minimum is below −0.01.
| Aggregation method | Gradient steps | Conflicting direction (%) |
|---|---|---|
| JPO (UPGrad) | 1,142 | 0.1% |
| Linear weighted sum (same losses) | 1,163 | 11.8% |
| p-GAPO | 1,166 | 11.3% |
| PCGrad | 1,163 | 2.0% |
Thus, UPGrad's contribution is that it almost entirely eliminates meaningful first-order gradient conflicts, compared with 11.8% of steps for the identical-loss linear baseline. As a further controlled comparison, we observed that applying an ordinary weighted sum to the same gradients recorded during the JPO run would produce conflicting directions in 11.2% of steps.
Official Response to Reviewer F3pZ [3/5]: D3PO-LSW Comparison and Preference Steering (Weakness 2)
2. D3PO comparison and preference-conditioned behavior (Weakness 2)
We agree that D3PO is a particularly relevant baseline. Its Late-Stage Weighting (LSW) mechanism independently constructs clipped surrogate losses before preference-weighted aggregation. We therefore implemented a D3PO-LSW-style baseline within our LLM training framework.
Unlike a full reproduction of D3PO, which additionally uses a multi-head critic and diversity regularization, this comparison focuses on its separate-loss and late-weighting mechanism. Together with the identical-loss linear ablation, it provides a direct assessment of JPO's incremental gradient processing.
To evaluate preference-conditioned behavior beyond weighted utility, we introduce two complementary metrics for the empirically conflicting gold and diamond objectives.
(a) Gold–diamond preference steering.
Let denote the mean reward for objective when conditioned on preference . (G=Gold, D=Diamond, C=Completion, L=Lava)
Gold steering:
Diamond steering:
Combined steering:
This measures whether requesting gold increases gold reward, and requesting diamonds increases diamond reward, relative to the three alternative conditioning preferences.
(b) Direct gold–diamond switching.
Since gold and diamond exhibit the strongest gradient conflict, we introduce a targeted metric comparing only these two preferences:
This measures the change in the gold-minus-diamond reward balance when switching the requested preference from gold-heavy to diamond-heavy.
Both metrics measure responsiveness to the requested preference. We additionally report completion and absolute rewards as controls.
Held-out evaluation. All models were evaluated on the same 50 held-out mazes under all four preference conditions. The following table reports the final checkpoint at training step 1,200 for every method:
| Method | Completion | Gold when requested / otherwise | Diamond when requested / otherwise | Steering | Direct switching |
|---|---|---|---|---|---|
| JPO (UPGrad) | 0.859 | 0.767 / 0.213 | 0.781 / 0.192 | 0.572 | 0.773 |
| D3PO-LSW | 0.849 | 0.732 / 0.192 | 0.802 / 0.195 | 0.573 | 0.766 |
| PCGrad | 0.864 | 0.731 / 0.217 | 0.793 / 0.200 | 0.554 | 0.762 |
| p-GAPO | 0.855 | 0.717 / 0.163 | 0.798 / 0.185 | 0.583 | 0.756 |
| GRPO (corrected) | 0.859 | 0.721 / 0.185 | 0.767 / 0.199 | 0.553 | 0.741 |
| Linear (same losses) | 0.828 | 0.698 / 0.178 | 0.756 / 0.189 | 0.543 | 0.727 |
| GDPO (corrected) | 0.798 | 0.660 / 0.193 | 0.653 / 0.185 | 0.468 | 0.654 |
JPO achieves the highest direct gold–diamond switching score (0.773), improving over the identical-loss linear baseline (0.727) and corrected GRPO (0.741) and GDPO (0.654). It also improves completion, gold-when-requested, and diamond-when-requested rewards over the identical-loss linear baseline.
However, the additional baselines are competitive: D3PO-LSW achieves essentially identical overall steering, while p-GAPO achieves slightly higher steering. We interpret these results as supporting JPO's ability to learn preference-responsive behavior, particularly under explicit gold–diamond conflict. The results do not establish superiority across every metric.
Training behavior. We additionally evaluated preference responsiveness over the final 200 training steps (1,001–1,200), using the mean rewards from eight rollouts per training step:
| Method | Completion | Steering | Direct switching |
|---|---|---|---|
| JPO (UPGrad) | 0.850 | 0.641 | 0.821 |
| p-GAPO | 0.848 | 0.607 | 0.773 |
| Linear | 0.836 | 0.593 | 0.773 |
| PCGrad | 0.850 | 0.590 | 0.781 |
| D3PO-LSW | 0.831 | 0.588 | 0.767 |
| GRPO (corrected) | 0.838 | 0.571 | 0.750 |
| GDPO (corrected) | 0.784 | 0.524 | 0.679 |
JPO achieves the highest steering and direct-switching scores in this training window. These training statistics measure conditional rewards across different training steps, whereas held-out evaluation compares preferences within the same mazes. We therefore place greater weight on the held-out comparisons.
Overall, the additional experiments establish a clear geometric difference between UPGrad and ordinary weighted aggregation, together with evidence of improved preference responsiveness in several comparisons.
Official Response to Reviewer F3pZ [1/5]: Important Correction to the Original MaZE Baselines
We thank the reviewer for their thoughtful assessment and for recognizing the practical significance of multi-objective LLM post-training and the encouraging results on MaZE and MATH. We agree that isolating UPGrad's contribution and quantifying its computational overhead are essential. To address these concerns, we have conducted additional controlled experiments with linear weighted aggregation, D3PO-LSW, PCGrad, and P-GAPO, alongside corrected GRPO and GDPO baselines. We also introduce quantitative measures of gradient conflict and preference-conditioned behavioral responsiveness.
Correction to the original GRPO/GDPO MaZE baselines
During the additional experiments, we identified a silent numerical inconsistency affecting the original GRPO and GDPO MaZE training runs.
The issue arose when PPO old-policy log-probabilities were recomputed using a different microbatch shape from the JD actor forward computation. In our Qwen3.5/veRL configuration, token log-probabilities were batch-shape-dependent, causing a spurious difference between the old and current policies even though the model weights were unchanged.
We fixed the issue by setting
score_batch_sizeequal to the rollout group size (G=8), ensuring consistent batch shapes. This restored the expected PPO probability ratio of 1 for an unchanged policy, up to numerical precision.We reran the affected MaZE baselines using the corrected configuration. These results supersede the original GRPO and GDPO MaZE comparisons, which understated baseline performance. The baseline comparisons below use the corrected results. This issue did not affect our MATH experiments, whose previously reported results remain unchanged.