acceptodds
Under review as a conference paper at ICLR 2027

SteerDPO: Steerable Multi-Objective Alignment via Implicit Reward Scalarization

Abstract

Aligning large language models (LLMs) with diverse, often conflicting human preferences is a fundamental challenge in building versatile, controllable AI assistants. While multi-objective alignment aims to address such trade-offs, existing approaches typically require costly online training or multiple explicit reward models, or rely on heuristic parameter-level aggregation with limited guarantees about the resulting preference trade-offs. We introduce SteerDPO, a single-stage, reward-model-free offline multi-objective alignment method that learns a single steerable policy directly from per-objective pairwise preference data. SteerDPO enables inference-time steering across trade-offs by explicitly conditioning on arbitrary preference weights, without additional training or online data collection. To this end, we derive the optimal policy for any preference weight as a normalized weighted geometric mean of the one-hot optimal policies under KL-regularized linear scalarization. SteerDPO uses this identity to train a single preference-conditioned policy through offline self-distillation. Two- and three-objective synthetic bandits and LLM experiments evaluate reward quality, steering, and front geometry separately. SteerDPO leads the compared methods on requested reward and steerability in the three-seed Llama-3.2-3B and single-seed Qwen3-8B comparisons.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Objective Alignment, Multi-Objective Optimization, Offline Preference Optimization, Inference-Time Steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XFzAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SteerDPO as an offline, reward-model-free method for multi-objective alignment that enables a single language model to adjust its behavior according to user-specified preference weights, such as the relative importance of helpfulness and harmlessness. The authors derive a theoretical result showing that, under a KL-regularized linear scalarization objective, the optimal policy for any mixture of objectives can be expressed as a normalized weighted geometric mean of the optimal single-objective policies. They leverage this identity to construct training targets through offline self-distillation, learning a single preference-conditioned model from objective-specific pairwise preference data without requiring additional training for each preference mixture. Experiments on synthetic bandits and Llama-3.2-3B and Qwen3-8B report improvements over several existing methods in reward performance and steerability.

Critical strengths

Derives a closed-form characterization of the optimal policy for arbitrary preference weights as a normalized weighted geometric mean of single-objective optimal policies.

Practical training framework: Introduces an offline self-distillation approach that learns a single preference-conditioned policy directly from objective-specific pairwise preference data, without requiring explicit reward models or separate training for every preference mixture.

Enables users to adjust trade-offs between objectives, such as helpfulness and harmlessness, by changing preference weights without further training.

Critical weaknesses

The central geometric-mean identity is related to existing policy-composition results, and the paper should more clearly establish what is fundamentally new beyond applying this identity to multi-objective DPO. [1] Decoding-Time Language Model Alignment with Multiple Objectives, Du et al Neurips 2024 [2] Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback Gao et al ICML 2024

Errors or biases in the single-objective policies can propagate into the targets for intermediate preference weights, potentially degrading the resulting trade-offs.

The Qwen3-8B evaluation uses a single random seed, making it difficult to assess the stability and reproducibility of the reported improvements.

The experiments do not fully establish how its advantages scale with the number of objectives or how it compares in computational cost and training efficiency

Questions for authors

How do the authors ensure that SteerDPO generalizes to responses not seen during training?

Also see weaknesses

3 comments
AuthorsOpenReview ↗

Response to XFzA — Contribution beyond decoding-time composition (1/3)

Thank you for the careful review. Key comparisons are reproduced below.

[W1. What is new beyond the geometric-mean identity]

The central geometric-mean identity is related to existing policy-composition results, and the paper should more clearly establish what is fundamentally new beyond applying this identity to multi-objective DPO. [1] Decoding-Time Language Model Alignment with Multiple Objectives, Du et al NeurIPS 2024 [2] Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback, Gao et al ICML 2024

While MOD derives the geometric-mean identity over full responses, it uses a token-level approximation at decoding time (Shi et al., 2024, Sec. 3.3), and does not account for function approximation error (Shi et al., 2024, Sec. 7). Therefore, the practical question is whether geometric composition provides useful training supervision for a single neural policy.

With optimal basis policies, the sequence-level geometric mean gives the optimal policy at each interpolated weight. During training, Eq. (13) computes the exact composition of the learned basis policies restricted to the observed response pair. SteerDPO trains the conditioned policy directly against the sequence-level targets, not the token-wise proxy proposed in MOD.

Our consistency ablation directly tests the usefulness of this supervision. With the architecture and preference labels held fixed, adding the distillation loss raises steerability from 0.304 to 0.591 on Llama and from 0.507 to 0.739 on Qwen (Table 6 and Table R5 in the global response). At α=0\alpha=0, only the one-hot conditions receive direct supervision; the distillation loss supervises mixture conditions within the same policy. The three-objective HH results further demonstrate useful steering from these offline composition targets (Table 5 and Figure 5).

SteerDPO achieves the highest mean scalarized reward and steerability on both backbones. Its hypervolume exceeds that of every baseline except MOD. Even with BT-sampled labels, SteerDPO achieves higher SR and steerability than MOD trained on hard oracle labels on both backbones, with a single policy evaluation per token and lower measured decoding latency (Tables 3–4 and R8).

Tables 3–4. MOD and SteerDPO on HH at 4,664 matched prompts. Values are mean±SE\text{mean} \pm \mathrm{SE}. SteerDPO uses BT-sampled labels.

MethodLlama SRLlama steerabilityQwen SRQwen steerability
MOD0.336±0.0030.336 \pm 0.0030.505±0.0010.505 \pm 0.0010.661±0.0010.661 \pm 0.0010.713±0.0010.713 \pm 0.001
SteerDPO0.344±0.001\mathbf{0.344} \pm 0.0010.568±0.004\mathbf{0.568} \pm 0.0040.664±0.011\mathbf{0.664} \pm 0.0110.743±0.008\mathbf{0.743} \pm 0.008

SteerDPO evaluates one conditioned policy per token, whereas MOD evaluates KK objective-specific experts.

Table R8. Training cost and median latency over 24 requests per decoding setting on HH/Llama-3.2-3B (one NVIDIA B200, batch size 1). Natural stopping can emit EOS before 64 tokens, so the higher cap of 256 can still yield shorter outputs and lower latency.

MethodPolicies trainedGPU-hoursPolicy evaluations per decoding step
MODPO11≈11.21\approx 11.21 (est.)1
MORLHF11≈25.65\approx 25.65 (est.)1
MO-ODPO16.251
Rewarded Soups212.141
RiC17.111
SteerDPO16.421
DecodingSteerDPOMOD (two experts)
Fixed 64 output tokens0.616 s1.155 s
Natural stopping, at most 256 tokens0.400 s0.567 s

Composing SteerDPO's own basis conditions at decoding time gives similar scores to the distilled policy (Q1).

Linear Alignment [2] follows a prompted principle at inference without training. SteerDPO learns a policy conditioned on weights over distinct objectives. We cite [2] in the revised Related Work.

AuthorsOpenReview ↗

Response to XFzA — Basis errors, response generalization, and Qwen seeds (2/3)

[W2. Errors in the basis policies propagate to interior targets]

Errors or biases in the single-objective policies can propagate into the targets for intermediate preference weights, potentially degrading the resulting trade-offs.

Each basis condition receives direct per-objective DPO supervision throughout joint training. For a given response pair, each basis contributes to the teacher’s log odds in proportion to its preference weight (Eq. 13). On HH/Llama, adding consistency supervision raises steerability from 0.304 to 0.591 with the architecture and preference labels held fixed (Table 6). This shows that the learned bases provide useful supervision at intermediate weights.

The experiments evaluate steering with these learned bases. In the bandit, basis policies learn from finite data, and comparison with the empirical reference front gives IGD+ 0.028 and HV 2.129 for SteerDPO, versus 0.062 and 2.032 for MOD (App. F). On Llama, increasing α\alpha from 0 to 2 raises steerability while anchor average changes; at α=10\alpha=10, the anchor falls.

Table 6. Oracle-label HH/Llama ablation, 384/512 limits.

Consistency weightSteerabilityAnchor Avg
α=0\alpha=00.3040.871
α=2\alpha=20.5910.845
α=10\alpha=100.5130.722

With BT-sampled labels, SteerDPO still exceeds all five original baselines on both backbones. These results show useful steering with estimated bases and stochastic supervision.

[Q1. Generalization to responses not seen during training]

How do the authors ensure that SteerDPO generalizes to responses not seen during training?

The pairwise targets train the parameters of an autoregressive policy, which generates responses freely at inference. We evaluate the learned policy on held-out HH prompts and compare its generated-response performance with decoding-time composition of its own basis conditions (Table 6).

Table 6. Generated-response comparison under the same Llama ablation setup as W2.

DecodingSRSteerability
Distilled policy0.4000.591
PtargetP_{\mathrm{target}}, two passes per token0.4090.582

[W3. Qwen3-8B uses a single seed]

The Qwen3-8B evaluation uses a single random seed, making it difficult to assess the stability and reproducibility of the reported improvements.

We extended the Qwen3-8B evaluation of SteerDPO to 3 seeds.

SteerDPO trained on BT-sampled labels has the highest mean SR and steerability.

Table R6. HH/Qwen3-8B on 4,664 matched prompts, using the manuscript Qwen normalization. Values are mean±SE\text{mean} \pm \mathrm{SE} over 3 seeds. SteerDPO uses BT-sampled labels.

MethodSRSteerabilityAnchor AvgHV
MODPO0.298±0.0130.298 \pm 0.0130.532±0.0030.532 \pm 0.0030.711±0.0140.711 \pm 0.01415.730±0.56215.730 \pm 0.562
MORLHF−0.114±0.002-0.114 \pm 0.0020.139±0.0030.139 \pm 0.003−0.016±0.010-0.016 \pm 0.01011.538±0.04311.538 \pm 0.043
MO-ODPO0.139±0.0030.139 \pm 0.0030.105±0.0060.105 \pm 0.0060.260±0.0040.260 \pm 0.00414.405±0.02714.405 \pm 0.027
Rewarded Soups0.104±0.0030.104 \pm 0.0030.436±0.0110.436 \pm 0.0110.362±0.0040.362 \pm 0.00413.355±0.01413.355 \pm 0.014
RiC0.475±0.0080.475 \pm 0.0080.635±0.0080.635 \pm 0.0080.854±0.0020.854 \pm 0.00217.876±0.07317.876 \pm 0.073
MOD0.661±0.0010.661 \pm 0.0010.713±0.0010.713 \pm 0.0011.046±0.003\mathbf{1.046} \pm 0.00318.885±0.012\mathbf{18.885} \pm 0.012
Panacea0.220±0.0490.220 \pm 0.0490.522±0.0540.522 \pm 0.0540.609±0.0610.609 \pm 0.06115.167±0.58315.167 \pm 0.583
COS-DPO0.269±0.0040.269 \pm 0.0040.349±0.0090.349 \pm 0.0090.404±0.0050.404 \pm 0.00515.348±0.05615.348 \pm 0.056
SteerDPO0.664±0.011\mathbf{0.664} \pm 0.0110.743±0.008\mathbf{0.743} \pm 0.0080.968±0.0120.968 \pm 0.01218.634±0.08518.634 \pm 0.085
AuthorsOpenReview ↗

Response to XFzA — Scaling to three objectives and computational cost (3/3)

[W4. Scaling with the number of objectives; computational cost]

The experiments do not fully establish how its advantages scale with the number of objectives or how it compares in computational cost and training efficiency.

We now provide a three-objective LLM comparison on Anthropic-HH with harmlessness, helpfulness, and verbosity. SteerDPO uses one conditioned-policy evaluation per decoding step. We fully fine-tune Llama-3.2-3B. SteerDPO leads all four reported metrics.

Table 5. Four methods on the same 4,664 held-out prompts and 15 preference vectors. Reward normalization and the hypervolume reference are shared across methods.

MethodSR ↑\uparrowSteerability ↑\uparrowAnchor Avg ↑\uparrowHV ↑\uparrow
MO-ODPO-0.0400.1560.06223.02
RiC0.1480.2120.28941.02
MOD0.3060.3450.62447.02
SteerDPO0.5400.3870.90059.63

SteerDPO trains its basis and mixture conditions jointly within one policy. At inference, a mixture preference requires one conditioned-policy evaluation per decoding step, whereas MOD composes predictions from three objective-specific experts. The K=3 experiment therefore provides empirical evidence that offline distillation can turn geometric-composition targets into useful three-objective control while retaining single-policy decoding. Section 5.5 and Figure 5 present the three-dimensional reward comparison, with numerical results in Table 5 and settings in Appendix L.6.

In the three-objective bandit, SteerDPO leads all four metrics. SR / steerability are 0.329 / 0.742 for SteerDPO and 0.276 / 0.678 for MOD (second, Table 2).

For any KK, SteerDPO defines the teacher over the whole simplex, adds one basis forward pass per objective during training, and uses one pass per token at inference. MOD evaluates KK experts per token. MODPO and MORLHF train one policy per evaluated weight, and the grid grows combinatorially with KK.

The measured training times and decoding latency are from the two-objective HH comparison. On Llama, SteerDPO trains in 6.42 GPU-hours (MO-ODPO 6.25, RiC 7.11). Median latency at 64 output tokens is 0.616 s for SteerDPO and 1.155 s for two-expert MOD (B200, batch size 1, Table R8).

[Q2. "Also see weaknesses"]

See W1–W4 above.

Reviewer_qs4hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies steerable multi-objective alignment, where one model should follow any user-given weight between objectives such as harmlessness and helpfulness. The key observation is that, under KL-regularized linear scalarization with a shared reference policy and β\beta, the optimal policy for any weight λ\lambda is a normalized weighted geometric mean of the per-objective optimal policies. The authors propose SteerDPO. It writes λ\lambda into the prompt and trains one model with two losses: a per-objective DPO loss at one-hot weights, and a self-distillation loss that pushes the model at weight λ\lambda toward the geometric mean of its own one-hot outputs, computed on the two responses in each preference pair. The method is offline and needs no reward model. Experiments use two- and three-objective synthetic bandits, and Anthropic-HH with Llama-3.2-3B and Qwen3-8B with LoRA. SteerDPO gives the best scalarized reward, steerability, anchor performance and hypervolume against MODPO, MORLHF, MO-ODPO, Rewarded Soups and RiC.

Critical strengths
  1. The method comes from first principles

I like the idea. The derivation needs no assumptions beyond the standard DPO setting, with a shared reference policy and β\beta across objectives. The method also needs no explicit reward model.

  1. Clear evidence on the tested weights

The synthetic bandits check the method against ground-truth rewards. The Llama results use three seeds, and SteerDPO leads on all four main metrics by a clear margin.

Critical weaknesses
  1. No test on unseen weights

The paper claims that one model can follow any preference weight. But training only uses 11 weight values. The LLM evaluation uses the same 11 values. So every result is on a weight the model has already seen. The weights are written as text with two decimals, so the second decimal is always 0 in training. It is unclear how the model handles a weight like 0.75. I would like to see SR and steerability on unseen weights, such as 0.25, 0.33 and 0.75. This would show whether the model really interpolates between the training weights.

  1. Missing baselines

The geometric-mean identity is not new. MOD (Shi et al., 2024) already derives it and uses it at decoding time, as the paper says. So the new part is distilling this identity into one conditioned model. The fair baselines are then (a) MOD itself, and (b) offline conditioned DPO methods, such as Panacea and COS-DPO.

Questions for authors
  1. How does SteerDPO perform on weights not used in training?

  2. Can the authors add MOD, Panacea and COS-DPO to the HH comparison?

4 comments
AuthorsOpenReview ↗

Response to qs4h — Unseen weights and decoding-time composition (1/2)

Thank you for the constructive review. We evaluated unseen preference weights and added comparisons with MOD, Panacea, and COS-DPO. Key comparisons are reproduced below; tables added to the revised PDF are marked in blue.

[W1. Unseen weights]

No test on unseen weights. The paper claims that one model can follow any preference weight. But training only uses 11 weight values. The LLM evaluation uses the same 11 values. So every result is on a weight the model has already seen. The weights are written as text with two decimals, so the second decimal is always 0 in training. It is unclear how the model handles a weight like 0.75. I would like to see SR and steerability on unseen weights, such as 0.25, 0.33 and 0.75. This would show whether the model really interpolates between the training weights.

We further evaluated six unseen weights on Qwen3-8B, 0.15, 0.25, 0.33, 0.67, 0.75, and 0.85, including MOD (Table 25). We reuse the existing checkpoints without additional training and retain the original SR and steerability definitions and fixed reward normalization.

Table R2. SR and steerability at the six unseen weights a∈{0.15,0.25,0.33,0.67,0.75,0.85}a \in \{0.15, 0.25, 0.33, 0.67, 0.75, 0.85\}, excluding a=0,1a=0,1. All rows use 4,664 prompts and the main HH normalization.

MethodQwen SRQwen steerability
MO-ODPO0.054−0.003
Rewarded Soups0.0680.396
RiC0.4540.667
MOD0.4480.656
COS-DPO−0.1120.052
SteerDPO0.4940.678

Figure 10 shows the six unseen weights alongside the existing seen-weight curves. Across the six unseen weights, SteerDPO achieves the highest SR and steerability among the compared methods (Table R2).

[Q1. Performance on unseen weights]

How does SteerDPO perform on weights not used in training?

See W1.

[W2. Distilling the geometric-mean composition]

Missing baselines. The geometric-mean identity is not new. MOD (Shi et al., 2024) already derives it and uses it at decoding time, as the paper says. So the new part is distilling this identity into one conditioned model. The fair baselines are then (a) MOD itself, and (b) offline conditioned DPO methods, such as Panacea and COS-DPO.

(a) While MOD derives the geometric-mean identity over full responses, it uses a token-level approximation at decoding time (Shi et al., 2024, Sec. 3.3), and does not account for function approximation error (Shi et al., 2024, Sec. 7). Therefore, the practical question is whether geometric composition provides useful training supervision for a single neural policy.

SteerDPO showed that sequence-wise composition and locally normalized token-wise composition generally induce different response distributions. Our consistency ablation directly tests the usefulness of this supervision. At α=0\alpha=0, only the one-hot conditions receive direct supervision. The three-objective HH results further demonstrate useful steering from these offline composition targets (Table 5 and Figure 5).

MOD evaluates KK objective-specific experts per token; SteerDPO distills the composition during training and evaluates one conditioned policy per token. SteerDPO achieves the highest mean scalarized reward and steerability on both backbones. Its hypervolume exceeds that of every baseline except MOD. Even with BT-sampled labels, SteerDPO achieves higher SR and steerability than MOD trained on hard oracle labels on both backbones, with a single policy evaluation per token (Tables 3–4).

On HH/Llama with one B200 and batch size 1, median latency at 64 output tokens is 0.616 s for SteerDPO and 1.155 s for two-expert MOD (Table R8). Composing SteerDPO's own basis conditions at decoding time also gives similar scores to the distilled policy.

Table R8. Training cost and median latency over 24 requests per decoding setting on HH/Llama-3.2-3B (one NVIDIA B200, batch size 1). Natural stopping can emit EOS before 64 tokens, so the higher cap of 256 can still yield shorter outputs and lower latency.

MethodPolicies trainedGPU-hoursPolicy evaluations per decoding step
MODPO1111.171
MORLHF1125.631
MO-ODPO16.251
Rewarded Soups212.141
RiC17.111
SteerDPO16.421
DecodingSteerDPOMOD (two experts)
Fixed 64 output tokens0.616 s1.155 s
Natural stopping, at most 256 tokens0.400 s0.567 s
AuthorsOpenReview ↗

Response to qs4h — Offline DPO baselines and HH comparisons (2/2)

(b) Panacea and COS-DPO use preference-weighted DPO objectives. Panacea’s Pareto-optimality guarantee assumes sufficient model expressivity and global optimization. Whether these conditions hold for its trained SVD-LoRA policies remains unverified. SteerDPO directly supervises intermediate weights using the exact sequence-level geometric composition of its learned basis policies on each response pair (Eq. 13). Adding this supervision improves steerability with the architecture and preference labels held fixed (Table 6 and Table R5).

[Q2. MOD, Panacea, and COS-DPO on HH]

Can the authors add MOD, Panacea and COS-DPO to the HH comparison?

We added MOD, Panacea, and COS-DPO experiment on HH, and included on revision.

We conducted additional comparisons with COS-DPO on Llama-3.2-3B and Qwen3-8B. Tables 3–4 report the results.

We reproduced Panacea using SVD-LoRA conditioning, as suggested on the paper.

Tables 3–4. HH at 512/128 limits and 4,664 matched prompts, using each backbone's main normalization. Values are mean±SE\text{mean} \pm \mathrm{SE}. SteerDPO uses BT-sampled labels.

MethodLlama SRLlama steerabilityQwen SRQwen steerability
MOD0.336±0.0030.336 \pm 0.0030.505±0.0010.505 \pm 0.0010.661±0.0010.661 \pm 0.0010.713±0.0010.713 \pm 0.001
Panacea0.257±0.0190.257 \pm 0.0190.370±0.0130.370 \pm 0.0130.220±0.0490.220 \pm 0.0490.522±0.0540.522 \pm 0.054
COS-DPO0.242±0.0060.242 \pm 0.0060.284±0.0080.284 \pm 0.0080.269±0.0040.269 \pm 0.0040.349±0.0090.349 \pm 0.009
SteerDPO0.344±0.001\mathbf{0.344} \pm 0.0010.568±0.004\mathbf{0.568} \pm 0.0040.664±0.011\mathbf{0.664} \pm 0.0110.743±0.008\mathbf{0.743} \pm 0.008

Table R7. COS-DPO comparison on HH/Llama-3.2-3B. Values are mean ±\pm SE over 3 seeds.

MethodSRSteerabilityAnchor AvgHV
COS-DPO0.242±0.0060.242 \pm 0.0060.284±0.0080.284 \pm 0.0080.390±0.0130.390 \pm 0.01320.658±0.12020.658 \pm 0.120
SteerDPO0.344±0.001\mathbf{0.344} \pm 0.0010.568±0.004\mathbf{0.568} \pm 0.0040.723±0.0040.723 \pm 0.00423.070±0.02223.070 \pm 0.022

Table R5. Oracle-label consistency and baseline comparison on Qwen3-8B at the five shared weights using 4,664 matched prompts. All rows use the same reward normalization and HV reference.

MethodSRSteerabilityHV
MODPO0.4470.60616.851
Rewarded Soups0.4240.66016.658
SteerDPO0.7000.73920.789
SteerDPO (α=0\alpha=0)0.6840.50719.624

The consistency ablation separately measures the effect of interior-weight supervision with SteerDPO's architecture held fixed.

Reviewer_qs4hOpenReview ↗

Thank you

Thank you for your detailed response. I will keep my positive score.

Thank you for reading our response and for the positive assessment. If any point remains unclear, we are happy to clarify.

Reviewer_NbHWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors address sparse supervision over preference weights in multi-objective alignment. They propose SteerDPO, a single-stage, reward-model-free offline method that learns a single preference-conditioned policy. Specifically, SteerDPO (1) learns one-hot-conditioned policies using per-objective DPO and (2) constructs weighted geometric-mean targets on two offline responses for self-distillation at other preference weights. On Anthropic-HH, it achieves SR of 0.499 versus 0.372 for MO-ODPO on Llama-3.2-3B, and 0.681 versus 0.582 for RiC on Qwen3-8B.

Critical strengths

Offline self-distillation enables practical preference steering. SteerDPO constructs geometric-mean targets from learned one-hot-conditioned policies and trains a single preference-conditioned policy (§4.2). This avoids collecting comparisons for every mixture or composing basis policies at inference.

The consistency loss improves steering in the reported ablation. In Table 5, alpha=2 increases steerability from 0.510 to 0.634 compared with alpha=0. This comes with a small Anchor Avg decrease, from 0.871 to 0.845.

Critical weaknesses

Oracle–proxy differences complicate the LLM comparison. SteerDPO uses oracle-relabeled preferences, whereas reward-based baselines rely on proxy scores (§I.2). Proxy–oracle agreement of 76.97–83.16% on generated-response pairs (Table 10) leaves unclear how much of the reported gain comes from the alignment method.

A closely related offline baseline is missing. COS-DPO already learns a weight-conditioned model from per-objective preferences without an explicit reward model (Ren et al., 2025). Comparing with it would clarify the added value of geometric-mean self-distillation.

Questions for authors

Unseen preference weights: Which 11 unique vectors in Table 16 are used during training, and are any evaluation weights excluded from that set? Please report SR and steerability at weights not used during training to assess the claim of steering across arbitrary preference weights.

Training lengths: SteerDPO uses maximum prompt/completion lengths of 384/512, while the baselines in Tables 17–21 use 512/128. Does the performance ranking persist when these limits are matched?

More than two objectives: The three-objective results are limited to synthetic bandits. Is there evidence that the same offline self-distillation approach retains effective steering with three LLM objectives?

2 comments
AuthorsOpenReview ↗

Response to NbHW — Oracle and proxy supervision, training lengths, and COS-DPO (1/2)

Thank you for your thoughtful review.

[W1. Oracle–proxy differences]

We retrained SteerDPO on the original Anthropic-HH human preferences to test its dependence on oracle relabeling. With the rest of the Llama training setup and the evaluation oracles held fixed, SR rose from 0.400 to 0.497 and steerability from 0.591 to 0.704 (Table 32).

In the revised main comparison, SteerDPO uses the same Bradley-Terry-sampled labels used to train the proxies (Eq. (2)). The reward-based baselines retain their original training signals. SteerDPO still exceeds all five original baselines on SR, HV, and steerability on both backbones (Table R1). Section 4 of the global response reports the external judges' comparisons on Qwen.

Table R1. The hard-label row is shown for reference. Bold marks the best value among the other rows.

MethodLlama SRLlama SteerabilityLlama HVQwen SRQwen SteerabilityQwen HV
MODPO0.084±0.0030.084 \pm 0.0030.177±0.0010.177 \pm 0.00119.977±0.15719.977 \pm 0.1570.298±0.0130.298 \pm 0.0130.532±0.0030.532 \pm 0.00315.730±0.56215.730 \pm 0.562
MORLHF0.287±0.0070.287 \pm 0.0070.408±0.0290.408 \pm 0.02921.793±0.13421.793 \pm 0.134−0.114±0.002-0.114 \pm 0.0020.139±0.0030.139 \pm 0.00311.538±0.04311.538 \pm 0.043
MO-ODPO0.318±0.0060.318 \pm 0.0060.444±0.0050.444 \pm 0.00521.794±0.06021.794 \pm 0.0600.139±0.0030.139 \pm 0.0030.105±0.0060.105 \pm 0.00614.405±0.02714.405 \pm 0.027
Rewarded Soups0.090±0.0020.090 \pm 0.0020.186±0.0020.186 \pm 0.00219.593±0.03219.593 \pm 0.0320.104±0.0030.104 \pm 0.0030.436±0.0110.436 \pm 0.01113.355±0.01413.355 \pm 0.014
RiC0.276±0.0030.276 \pm 0.0030.501±0.0040.501 \pm 0.00421.876±0.03821.876 \pm 0.0380.475±0.0080.475 \pm 0.0080.635±0.0080.635 \pm 0.00817.876±0.07317.876 \pm 0.073
MOD0.336±0.0030.336 \pm 0.0030.505±0.0010.505 \pm 0.00123.491±0.011\mathbf{23.491} \pm 0.0110.661±0.0010.661 \pm 0.0010.713±0.0010.713 \pm 0.00118.885±0.012\mathbf{18.885} \pm 0.012
Panacea0.257±0.0190.257 \pm 0.0190.370±0.0130.370 \pm 0.01321.363±0.14821.363 \pm 0.1480.220±0.0490.220 \pm 0.0490.522±0.0540.522 \pm 0.05415.167±0.58315.167 \pm 0.583
COS-DPO0.242±0.0060.242 \pm 0.0060.284±0.0080.284 \pm 0.00820.658±0.12020.658 \pm 0.1200.269±0.0040.269 \pm 0.0040.349±0.0090.349 \pm 0.00915.348±0.05615.348 \pm 0.056
SteerDPO, BT sampled labels0.344±0.001\mathbf{0.344} \pm 0.0010.568±0.004\mathbf{0.568} \pm 0.00423.070±0.02223.070 \pm 0.0220.664±0.011\mathbf{0.664} \pm 0.0110.743±0.008\mathbf{0.743} \pm 0.00818.634±0.08518.634 \pm 0.085
SteerDPO, hard labels0.516±0.0030.516 \pm 0.0030.603±0.0040.603 \pm 0.00423.393±0.02623.393 \pm 0.0260.656±0.0100.656 \pm 0.0100.696±0.0060.696 \pm 0.00618.806±0.07718.806 \pm 0.077

[Q2. Training lengths]

Yes. We retrained SteerDPO with the baselines' 512/128 prompt/completion limits while retaining hard preference labels. Across 3 seeds on Llama-3.2-3B, it achieves SR 0.516±0.0030.516 \pm 0.003, steerability 0.603±0.0040.603 \pm 0.004, and HV 23.393±0.02623.393 \pm 0.026 (Table 30). Under these matched limits, SteerDPO outperforms all five reward-based baselines on SR, HV, and steerability (Table R1 above).

[W2. Comparison with COS-DPO]

We conducted additional comparisons with COS-DPO on Llama-3.2-3B and Qwen3-8B. Tables 3–4 report the results.

Table R7. COS-DPO comparison on HH/Llama-3.2-3B. Values are mean ±\pm SE over 3 seeds.

MethodSRSteerabilityAnchor AvgHV
COS-DPO0.242±0.0060.242 \pm 0.0060.284±0.0080.284 \pm 0.0080.390±0.0130.390 \pm 0.01320.658±0.12020.658 \pm 0.120
SteerDPO0.344±0.001\mathbf{0.344} \pm 0.0010.568±0.004\mathbf{0.568} \pm 0.0040.723±0.0040.723 \pm 0.00423.070±0.02223.070 \pm 0.022

Table R5. Oracle-label consistency and baseline comparison on Qwen3-8B at the five shared weights using 4,664 matched prompts. All rows use the same reward normalization and HV reference.

MethodSRSteerabilityHV
MODPO0.4470.60616.851
Rewarded Soups0.4240.66016.658
SteerDPO0.7000.73920.789
SteerDPO (α=0\alpha=0)0.6840.50719.624

SteerDPO distills a target over complete responses on each candidate pair. COS-DPO supervises interior weights with a λ\lambda-weighted sum of per-objective losses. Removing consistency lowers Qwen steerability from 0.739 to 0.507 with little change in SR. The Llama ablation shows the same improvement in steering: adding consistency raises steerability from 0.304 to 0.591 (Table 6).

AuthorsOpenReview ↗

Response to NbHW — Unseen weights and three-objective alignment (2/2)

[Q1. Unseen preference weights]

In the paper, SteerDPO is trained and evaluated on the same 11 preference weights, λ(a)=(a,1−a)\boldsymbol{\lambda}(a) = (a, 1-a) with a∈{0,0.1,…,1}a \in \{0, 0.1, \ldots, 1\}, so all evaluation weights were seen during training. As the reviewer requested, we additionally evaluate SteerDPO, MOD, RiC, MO-ODPO, Rewarded Soups, and COS-DPO at 6 weights that never appear in training, a∈{0.15,0.25,0.33,0.67,0.75,0.85}a \in \{0.15, 0.25, 0.33, 0.67, 0.75, 0.85\}. We reuse the existing checkpoints without additional training and retain the original SR and steerability definitions and fixed reward normalization. Table R2 (Table 25 in the revision) reports metrics on these unseen weights alone, including MOD. Figure 10 retains the seen-weight curves and marks the six unseen weights. Across the six unseen weights on Qwen3-8B, SteerDPO's mean harmlessness increases and mean helpfulness decreases as aa increases, demonstrating directional reward control beyond the training grid.

Table R2. SR and steerability at the six unseen weights a∈{0.15,0.25,0.33,0.67,0.75,0.85}a \in \{0.15, 0.25, 0.33, 0.67, 0.75, 0.85\}, excluding a=0,1a=0,1. All rows use 4,664 prompts and the main HH normalization.

MethodQwen SRQwen steerability
MO-ODPO0.054−0.003
Rewarded Soups0.0680.396
RiC0.4540.667
MOD0.4480.656
COS-DPO−0.1120.052
SteerDPO0.4940.678

[Q3. More than two objectives]

More than two objectives: The three-objective results are limited to synthetic bandits. Is there evidence that the same offline self-distillation approach retains effective steering with three LLM objectives?

We now provide a three-objective LLM comparison on Anthropic-HH with harmlessness, helpfulness, and verbosity as an additional preference objective. We fully fine-tune Llama-3.2-3B.

Table 5. Four methods on the same 4,664 held-out prompts at 15 preference vectors covering the simplex vertices, edges, and interior. Reward normalization and the hypervolume reference are shared across methods.

MethodSR ↑\uparrowSteerability ↑\uparrowAnchor Avg ↑\uparrowHV ↑\uparrow
MO-ODPO-0.0400.1560.06223.02
RiC0.1480.2120.28941.02
MOD0.3060.3450.62447.02
SteerDPO0.5400.3870.90059.63

SteerDPO achieves the highest requested reward, steerability, anchor average, and hypervolume among the compared methods. Section 5.5 and Figure 5 present the new comparison; Appendix L.6 provides the full results and setup (Tables 5 and 31).

Comments on the paper

4 comments
AuthorsOpenReview ↗

Global response (1/4)

Thank you for the reviews. SteerDPO uses a sequence-level self-supervised signal to train one policy at mixture weights without an explicit reward model. Per-objective preference pairs supervise the policy at one-hot weights. For each mixture weight, we take the weighted geometric mean of the probabilities that these one-hot conditions assign to each complete response, normalize over the two responses in the pair, and train the same policy to match that distribution.

MOD derives the geometric-mean identity over full responses and uses a token-level approximation at decoding time (Shi et al., 2024, Section 3.3). MOD also acknowledges that its analysis is primarily tabular and does not account for function approximation error (Shi et al., 2024, Section 7). We evaluate consistency, compare with MOD, Panacea, and COS-DPO, and test stochastic labels, original human annotations, unseen weights, and three-objective alignment. We also match training lengths and extend SteerDPO to 3 seeds on both backbones. Tables added to the revised PDF are marked in blue.

[Additional experiments]

1. Offline distillation, baseline comparisons, and inference cost (NbHW W2/Q2, qs4h W2/Q2, XFzA W1/W2/W3). The consistency loss supplies supervision at interior preference weights. COS-DPO and Panacea use λ\lambda-weighted sums of per-objective DPO losses; SteerDPO distills a target over complete responses on each offline candidate pair. At α=0\alpha=0, only the one-hot conditions receive direct supervision.

On Qwen3-8B, adding consistency raises steerability from 0.507 to 0.739, while SR changes from 0.684 to 0.700 (Table R5). On Llama-3.2-3B, it raises steerability from 0.304 to 0.591 and SR from 0.282 to 0.400 (Table 6). Figure 6 shows the corresponding reward fronts, which collapse toward one preference region without consistency.

The pairwise targets train an autoregressive policy that generates responses freely at inference. On held-out HH prompts, the distilled policy scores close to decoding-time composition of its own basis conditions (Table 6). SteerDPO uses one conditioned-policy evaluation per token, while composition requires evaluations under both basis conditions.

In the separate Qwen3-8B oracle-label comparison at 384/512 prompt/completion limits, SteerDPO obtains SR / steerability of 0.700 / 0.739, compared with 0.447 / 0.606 for MODPO and 0.424 / 0.660 for Rewarded Soups formed from two DPO experts (Table R5).

Table R5. Oracle-label consistency and baseline comparison on Qwen3-8B at the five shared weights using 4,664 matched prompts. All rows use the same reward normalization and HV reference.

MethodSRSteerabilityHV
MODPO0.4470.60616.851
Rewarded Soups0.4240.66016.658
SteerDPO0.7000.73920.789
SteerDPO (α=0\alpha=0)0.6840.50719.624

Table 6. Oracle-label consistency ablation on HH/Llama.

SettingSRSteerabilityAnchor Avg
α=0\alpha = 00.2820.3040.871
α=2\alpha = 2 (SteerDPO)0.4000.5910.845
α=10\alpha = 100.3280.5130.722
PtargetP_{\mathrm{target}} decoding, two passes per token0.4090.5820.868

Table R8 reports training times and decoding latency for the two-objective HH comparison. SteerDPO trains one policy in 6.42 GPU-hours. On HH/Llama-3.2-3B, median latency at 64 output tokens is 0.616 s for SteerDPO and 1.155 s for two-expert MOD, measured on one B200 with batch size 1.

Table R8. Training cost and median latency over 24 requests per decoding setting on HH/Llama-3.2-3B (one NVIDIA B200, batch size 1). Natural stopping can emit EOS before 64 tokens, so the higher cap of 256 can still yield shorter outputs and lower latency.

MethodPolicies trainedGPU-hoursPolicy evaluations per decoding step
MODPO1111.171
MORLHF1125.631
MO-ODPO16.251
Rewarded Soups212.141
RiC17.111
SteerDPO16.421
DecodingSteerDPOMOD (two experts)
Fixed 64 output tokens0.616 s1.155 s
Natural stopping, at most 256 tokens0.400 s0.567 s
AuthorsOpenReview ↗

Global response (2/4)

MOD composes independently trained objective-specific experts at decoding time. At matched 512/128 prompt/completion limits, SteerDPO trained on BT-sampled labels achieves the highest mean scalarized reward and steerability on both backbones. Its hypervolume exceeds that of every baseline except MOD. Even with BT-sampled labels, SteerDPO achieves higher SR and steerability than MOD trained on hard oracle labels on both backbones, with a single policy evaluation per token (Tables 3 and 4, Table R6). On Llama, MOD obtains SR 0.336±0.0030.336 \pm 0.003 and steerability 0.505±0.0010.505 \pm 0.001, compared with 0.344±0.0010.344 \pm 0.001 and 0.568±0.0040.568 \pm 0.004 for SteerDPO. On Qwen, MOD obtains 0.661±0.0010.661 \pm 0.001 and 0.713±0.0010.713 \pm 0.001, compared with 0.664±0.0110.664 \pm 0.011 and 0.743±0.0080.743 \pm 0.008 for SteerDPO. The main HH table comparisons use eleven preference weights a∈{0,0.1,…,1}a\in\{0,0.1,\ldots,1\}.

Table R6. HH/Qwen3-8B on 4,664 matched prompts, using the manuscript Qwen normalization. Values are mean±SE\text{mean} \pm \mathrm{SE} over 3 seeds. SteerDPO uses BT-sampled labels.

MethodSRSteerabilityAnchor AvgHV
MODPO0.298±0.0130.298 \pm 0.0130.532±0.0030.532 \pm 0.0030.711±0.0140.711 \pm 0.01415.730±0.56215.730 \pm 0.562
MORLHF−0.114±0.002-0.114 \pm 0.0020.139±0.0030.139 \pm 0.003−0.016±0.010-0.016 \pm 0.01011.538±0.04311.538 \pm 0.043
MO-ODPO0.139±0.0030.139 \pm 0.0030.105±0.0060.105 \pm 0.0060.260±0.0040.260 \pm 0.00414.405±0.02714.405 \pm 0.027
Rewarded Soups0.104±0.0030.104 \pm 0.0030.436±0.0110.436 \pm 0.0110.362±0.0040.362 \pm 0.00413.355±0.01413.355 \pm 0.014
RiC0.475±0.0080.475 \pm 0.0080.635±0.0080.635 \pm 0.0080.854±0.0020.854 \pm 0.00217.876±0.07317.876 \pm 0.073
MOD0.661±0.0010.661 \pm 0.0010.713±0.0010.713 \pm 0.0011.046±0.003\mathbf{1.046} \pm 0.00318.885±0.012\mathbf{18.885} \pm 0.012
Panacea0.220±0.0490.220 \pm 0.0490.522±0.0540.522 \pm 0.0540.609±0.0610.609 \pm 0.06115.167±0.58315.167 \pm 0.583
COS-DPO0.269±0.0040.269 \pm 0.0040.349±0.0090.349 \pm 0.0090.404±0.0050.404 \pm 0.00515.348±0.05615.348 \pm 0.056
SteerDPO0.664±0.011\mathbf{0.664} \pm 0.0110.743±0.008\mathbf{0.743} \pm 0.0080.968±0.0120.968 \pm 0.01218.634±0.08518.634 \pm 0.085

We conducted additional comparisons with COS-DPO on Llama-3.2-3B and Qwen3-8B. Tables 3–4 report the results.

Across 3 seeds on Llama, COS-DPO obtains SR 0.242±0.0060.242 \pm 0.006 and steerability 0.284±0.0080.284 \pm 0.008, compared with 0.344±0.0010.344 \pm 0.001 and 0.568±0.0040.568 \pm 0.004 for SteerDPO (Table 3).

We added Panacea with SVD-LoRA conditioning and 3 seeds on each backbone at the same 512/128 limits. On Llama, Panacea obtains SR 0.257±0.0190.257 \pm 0.019 and steerability 0.370±0.0130.370 \pm 0.013, compared with 0.344±0.0010.344 \pm 0.001 and 0.568±0.0040.568 \pm 0.004 for SteerDPO. On Qwen, Panacea obtains 0.220±0.0490.220 \pm 0.049 and 0.522±0.0540.522 \pm 0.054, compared with 0.664±0.0110.664 \pm 0.011 and 0.743±0.0080.743 \pm 0.008 for SteerDPO (Tables 3 and 4).

Table R7. COS-DPO comparison on HH/Llama-3.2-3B. Values are mean ±\pm SE over 3 seeds.

MethodSRSteerabilityAnchor AvgHV
COS-DPO0.242±0.0060.242 \pm 0.0060.284±0.0080.284 \pm 0.0080.390±0.0130.390 \pm 0.01320.658±0.12020.658 \pm 0.120
SteerDPO0.344±0.001\mathbf{0.344} \pm 0.0010.568±0.004\mathbf{0.568} \pm 0.0040.723±0.0040.723 \pm 0.00423.070±0.02223.070 \pm 0.022
AuthorsOpenReview ↗

Global response (3/4)

2. Stochastic preference labels (NbHW W1, XFzA W2). We trained SteerDPO on labels sampled from Bradley-Terry probabilities of the oracle reward differences. Under these stochastic labels, SteerDPO still exceeds all five original baselines on SR, HV, and steerability on both backbones, while the baselines retain their original training signals.

Table R1. The hard-label row is shown for reference. Bold marks the best value among the other rows.

MethodLlama SRLlama SteerabilityLlama HVQwen SRQwen SteerabilityQwen HV
MODPO0.084±0.0030.084 \pm 0.0030.177±0.0010.177 \pm 0.00119.977±0.15719.977 \pm 0.1570.298±0.0130.298 \pm 0.0130.532±0.0030.532 \pm 0.00315.730±0.56215.730 \pm 0.562
MORLHF0.287±0.0070.287 \pm 0.0070.408±0.0290.408 \pm 0.02921.793±0.13421.793 \pm 0.134−0.114±0.002-0.114 \pm 0.0020.139±0.0030.139 \pm 0.00311.538±0.04311.538 \pm 0.043
MO-ODPO0.318±0.0060.318 \pm 0.0060.444±0.0050.444 \pm 0.00521.794±0.06021.794 \pm 0.0600.139±0.0030.139 \pm 0.0030.105±0.0060.105 \pm 0.00614.405±0.02714.405 \pm 0.027
Rewarded Soups0.090±0.0020.090 \pm 0.0020.186±0.0020.186 \pm 0.00219.593±0.03219.593 \pm 0.0320.104±0.0030.104 \pm 0.0030.436±0.0110.436 \pm 0.01113.355±0.01413.355 \pm 0.014
RiC0.276±0.0030.276 \pm 0.0030.501±0.0040.501 \pm 0.00421.876±0.03821.876 \pm 0.0380.475±0.0080.475 \pm 0.0080.635±0.0080.635 \pm 0.00817.876±0.07317.876 \pm 0.073
MOD0.336±0.0030.336 \pm 0.0030.505±0.0010.505 \pm 0.00123.491±0.011\mathbf{23.491} \pm 0.0110.661±0.0010.661 \pm 0.0010.713±0.0010.713 \pm 0.00118.885±0.012\mathbf{18.885} \pm 0.012
Panacea0.257±0.0190.257 \pm 0.0190.370±0.0130.370 \pm 0.01321.363±0.14821.363 \pm 0.1480.220±0.0490.220 \pm 0.0490.522±0.0540.522 \pm 0.05415.167±0.58315.167 \pm 0.583
COS-DPO0.242±0.0060.242 \pm 0.0060.284±0.0080.284 \pm 0.00820.658±0.12020.658 \pm 0.1200.269±0.0040.269 \pm 0.0040.349±0.0090.349 \pm 0.00915.348±0.05615.348 \pm 0.056
SteerDPO, BT sampled labels0.344±0.001\mathbf{0.344} \pm 0.0010.568±0.004\mathbf{0.568} \pm 0.00423.070±0.02223.070 \pm 0.0220.664±0.011\mathbf{0.664} \pm 0.0110.743±0.008\mathbf{0.743} \pm 0.00818.634±0.08518.634 \pm 0.085
SteerDPO, hard labels0.516±0.0030.516 \pm 0.0030.603±0.0040.603 \pm 0.00423.393±0.02623.393 \pm 0.0260.656±0.0100.656 \pm 0.0100.696±0.0060.696 \pm 0.00618.806±0.07718.806 \pm 0.077

3. Original human annotations (Table 32 in App. L.7, NbHW W1). We retrained SteerDPO on Llama-3.2-3B with the original Anthropic-HH annotations, replacing the oracle-generated preference labels while retaining the same training setup. SR increases from 0.400 to 0.497 and steerability from 0.591 to 0.704 (Table 32).

All four reported metrics improve without oracle-generated preference labels during policy training. The oracle-relabeled setting is therefore conservative for SteerDPO in this ablation. MODPO's margins and RiC's scoring require both objective labels on each pair.

4. External LLM judges (Tables R3 and R4, NbHW W1). Both judges prefer both SteerDPO policies over MODPO, MORLHF, Rewarded Soups, and RiC on helpfulness and harmlessness. Against MO-ODPO, the original policy scores 52.7–53.6% on helpfulness and 46.9–47.9% on harmlessness across the two judges; the BT-label policy scores 50.7–51.0% and 46.4–47.0%, respectively.

Muse-Glimmer-30B and Claude Haiku 5.5 compared the original Qwen3-8B policies with each baseline at the five shared weights. The judges scored helpfulness and harmlessness separately, saw each pair in both A/B orders, and never saw λ\lambda. We pool the five weights, 1,500 pairs per cell, and count ties as half a win.

Table R3. Judge win rates, original Qwen3-8B policy against each baseline.

OpponentMuse helpfulnessMuse harmlessnessHaiku helpfulnessHaiku harmlessness
MODPO77.4%61.0%81.1%63.6%
MORLHF72.2%59.1%75.8%61.6%
MO-ODPO52.7%47.9%53.6%46.9%
Rewarded Soups74.0%59.3%78.6%62.6%
RiC56.9%59.1%57.7%59.7%
AuthorsOpenReview ↗

Global response (4/4)

Table R4. Judge win rates, BT-label Qwen3-8B policy against each baseline.

OpponentMuse helpfulnessMuse harmlessnessHaiku helpfulnessHaiku harmlessness
MODPO75.6%59.9%79.6%63.2%
MORLHF70.7%58.8%74.7%60.2%
MO-ODPO50.7%47.0%51.0%46.4%
Rewarded Soups72.7%59.3%77.4%62.4%
RiC54.4%58.1%55.6%58.6%

5. Steering at unseen preference weights (NbHW Q1, qs4h W1/Q1). We train on (a,1−a)(a,1-a) for a∈{0,0.1,…,1}a\in\{0,0.1,\ldots,1\}, with each weight written to two decimal places. We evaluated the existing Qwen3-8B checkpoints at six unseen weights, a∈{0.15,0.25,0.33,0.67,0.75,0.85}a\in\{0.15,0.25,0.33,0.67,0.75,0.85\}. On these six weights alone, SteerDPO obtains SR 0.494 and steerability 0.678 using the original reward normalization (Table R2). As the requested harmlessness weight increases, mean harmlessness rises and mean helpfulness falls. Figure 10 retains the seen-weight curves and adds all six unseen weights, including the MOD evaluations.

Table R2. SR and steerability at the six unseen weights a∈{0.15,0.25,0.33,0.67,0.75,0.85}a \in \{0.15, 0.25, 0.33, 0.67, 0.75, 0.85\}, excluding a=0,1a=0,1. All rows use 4,664 prompts and the main HH normalization.

MethodQwen SRQwen steerability
MO-ODPO0.054−0.003
Rewarded Soups0.0680.396
RiC0.4540.667
MOD0.4480.656
COS-DPO−0.1120.052
SteerDPO0.4940.678

6. Three-objective LLM alignment and inference cost (NbHW Q3, XFzA W4). We extend Anthropic-HH with verbosity alongside harmlessness and helpfulness, and compare fully fine-tuned Llama-3.2-3B policies on 4,664 matched held-out prompts at 15 preference vectors. With shared reward normalization and an HV reference, SteerDPO has the highest SR (0.540), steerability (0.387), anchor average (0.900), and HV (59.63) among SteerDPO, MOD, MO-ODPO, and RiC. MOD obtains 0.306, 0.345, 0.624, and 47.02, respectively (Table 5, reproduced below).

Table 5. Four methods on the same 4,664 held-out prompts at 15 preference vectors covering the simplex vertices, edges, and interior. Reward normalization and the hypervolume reference are shared across methods.

MethodSR ↑\uparrowSteerability ↑\uparrowAnchor Avg ↑\uparrowHV ↑\uparrow
MO-ODPO-0.0400.1560.06223.02
RiC0.1480.2120.28941.02
MOD0.3060.3450.62447.02
SteerDPO0.5400.3870.90059.63

In the three-objective bandit, SteerDPO also leads all four metrics, with SR / steerability of 0.329 / 0.742 versus 0.276 / 0.678 for MOD (Table 2).

For KK objectives, SteerDPO adds one basis forward pass per objective during training and uses one conditioned-policy evaluation per token at inference. MOD evaluates KK experts per token. MODPO and MORLHF train one policy per evaluated weight, so covering a finer preference grid requires more policies. Table R8 gives the measured training and inference costs for the two-objective HH setting.

Code and evaluation models. We will publicly release our code and oracle reward model checkpoints after the rebuttal period, including the training and evaluation scripts needed to reproduce our results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.