acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Representation Alignment in Preference Optimization

13 views

Abstract

Direct preference optimization (DPO) has become a simple and effective paradigm for aligning large language models (LLMs) with human preferences. However, DPO-style methods are fundamentally constrained by a mismatch between static offline preference data and the evolving behavior of the current policy. Recent works attempt to mitigate this mismatch by constructing preference supervision in the reward space or through data-side strategies. However, these constraints provide only a compressed view of the internal generation process and do not explicitly ensure that the current policy encodes sufficient preference information during generation. In this paper, we revisit this limitation from a representation perspective and propose On-Policy Representation Alignment (OPRA), which extends preference optimization from reward-level response ranking to representation-level preference alignment. Specifically, for an offline preference pair, we sample a short response prefix from the current policy as an anchor and encourage its response-level representation to be closer to that of the chosen response than to that of the rejected response. By jointly optimizing output-level preference consistency and on-policy representation alignment, OPRA provides a direct representation-level bridge between offline preference supervision and the policy's own autoregressive generation. Experiments on ten benchmarks demonstrate consistent improvements across preference-data settings and LLM backbones. Further analyses show that OPRA learns more preference-consistent representations and generalizes well across diverse settings.

Primary area:
optimization
Keywords:
representation learning, preference optimization, implicit rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Cd18OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OPRA adds a representation loss to DPO. It generates a short prefix with the model being trained, then pulls its hidden representation closer to the preferred response and away from the rejected response. A filter removes pairs whose preference labels disagree with the model's implicit reward ranking. The knowledge and reasoning experiments also train on the preferred responses with a likelihood loss. Results across several models and datasets suggest that the representation loss helps.

Critical strengths
  1. The ablations show that the representation loss helps. Removing it lowers the seven-task average by 1.30 percentage points on Qwen and 1.77 on Gemma. Removing the filter also reduces performance (Table 5). Table 4 further compares representation-based and reward-based supervision.
  2. The experiments cover a useful range of settings. The paper tests two preference datasets, several model families and sizes, and different sources of preference labels. Training takes 1.32 times as long as DPO in the reported setup (Table 9).
Critical weaknesses
  1. It is unclear how much the method gains from using the current model's outputs. The ablations do not keep the representation loss fixed while changing the prefix source to a frozen model or previously saved outputs. The improvement could therefore come from the added loss more generally. Proposition 1 and Theorem 1 also do not show that the representation ordering leads to better responses or reduces the mismatch with offline data.
  2. Some gains over the strongest baselines are small. The Gemma average is only 0.05 percentage points above HyPO, and MT-Bench is 0.04 points above TIDPO. AlpacaEval LC is slightly below HyPO (Tables 1 and 3). Without uncertainty estimates, it is hard to judge these differences. The generation tests also use a different loss setup and custom judges or reference models, so the results need to be interpreted within those settings (Appendices C.1 and C.5).
Questions for authors
  1. What happens if the prefixes come from a frozen model or previously saved outputs, with the same representation loss? Does using the current model give better generation results?
  2. Can you report variation across runs or confidence intervals for the strongest-baseline comparisons, including generation results without the representation loss?
  3. How were the loss settings and hyperparameters selected? Did OPRA and the baselines receive comparable tuning budgets, especially for generation tasks?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the constructive comments. We have added controlled anchor-source ablations, expanded generation evaluations, and clarified the training configurations and statistical evidence.

Q1: Contribution of Current-Policy Anchors Beyond Representation Alignment.

A1: We isolate the contribution of current-policy anchoring by keeping the representation alignment objective fixed and varying the anchor source (revised Section 4.3, Table 5).

MethodAnchor sourceSeven general tasks Avg. (%)AlpacaEval2.0 LC (%)
OPRACurrent policy77.3545.01
w/ Chosen PrefixOffline chosen response76.9044.06
w/ SFT AnchorFrozen SFT model76.3943.67
w/o RANone76.0544.79

Current-policy anchors outperform both frozen-SFT anchors (+0.96/+1.34 pp) and offline chosen prefixes (+0.45/+0.95 pp) on the seven-task average and AlpacaEval2.0 LC, respectively. Removing RA also reduces performance, separating the contributions of representation supervision and anchor selection. These results support the benefit of aligning representations encountered by the evolving policy rather than relying solely on fixed offline or frozen-model prefixes.

Held-out representation preference transfer. We further evaluate preference-relation transfer on 1,437 held-out pairs correctly ranked by both DPO and OPRA under the knowledge and reasoning configuration (revised Table 6).

Shared held-out pairsDPO mean gapOPRA mean gapΔ gapPaired-bootstrap 95% CIOPRA larger gap (%)
1,437−0.05692−0.00076+0.05615[0.04399, 0.06779]65.14

OPRA improves the representation preference gap on shared held-out pairs, providing complementary evidence that offline preference relations are more consistently reflected in current-policy-generated representations (revised Section 4.3, Table 6; Appendix E).

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Q2: Statistical Reliability and Generation Quality Without Representation Alignment.

A2: We rerun the generation benchmarks using GPT-4o-mini as the judge, extend the evaluation to Qwen3-14B, and add generation ablations without representation alignment.

1. Instruction following and open-ended generation. We evaluate MT-Bench and AlpacaEval2.0 using GPT-4o-mini as the judge/annotator and GPT-4-Turbo as the AlpacaEval reference response source. AlpacaEval2.0 win rates are reported with standard errors (revised Section 4.2, Table 4).

BackboneMethodMT-BenchAlpacaEval2.0 LC (%)AlpacaEval2.0 WR (%) ± SE
Qwen2.5-7B-InstructDPO7.4344.7943.10 ± 1.75
DPO w/ NLL7.2634.3126.34 ± 1.55
HyPO7.3837.5736.40 ± 1.70
TI-DPO7.3440.7238.93 ± 1.72
DPO w/ Online KL7.4244.0242.42 ± 1.74
OPRA7.5345.0143.42 ± 1.75
Qwen3-14BDPO7.4762.6166.27 ± 1.66
HyPO7.3749.6348.07 ± 1.76
TI-DPO7.2362.4164.72 ± 1.68
DPO w/ Online KL7.3462.9366.77 ± 1.66
OPRA7.4964.7968.26 ± 1.64

OPRA achieves the highest MT-Bench and AlpacaEval2.0 scores among the evaluated methods on both backbones. Relative to DPO, its AlpacaEval2.0 LC/WR gains are +0.22/+0.32 pp on Qwen2.5-7B-Instruct and +2.18/+1.99 pp on Qwen3-14B. The additional 14B evaluation extends the generation-quality comparison beyond the original 7B backbone.

2. Generation ablation without representation alignment. We evaluate the contribution of RA and alternative anchor sources while retaining the underlying preference-training objectives (revised Section 4.3, Table 5).

MethodSeven general tasks Avg. (%)AlpacaEval2.0 LC (%)
OPRA77.3545.01
w/o RA76.0544.79
w/ Chosen Prefix76.9044.06
w/ SFT Anchor76.3943.67
w/ Reward Alignment72.8241.89

Removing RA reduces the seven-task average by 1.30 pp and AlpacaEval2.0 LC by 0.22 pp. The anchor-source and reward-alignment controls further support the contributions of current-policy anchoring and representation-level supervision, although the generation gain over the w/o RA variant is modest.

3. Held-out policy-generation quality. We further evaluate 2,000 held-out UltraFeedback prompts using two reward models and a pairwise preference judge. The pairwise win rates against DPO are summarized below (revised Section 4.2, Table 3).

MethodPairwise win rate vs. DPO (%)
DPO50.00
HyPO44.15
TI-DPO43.65
DPO w/ Online KL55.70
OPRA57.53

OPRA achieves a 57.53% pairwise win rate against DPO (reported p < 0.001), while obtaining comparable reward-model scores (revised Table 3). This provides additional evidence for the quality of policy-generated responses.

Statistical reliability and revised claims. We report AlpacaEval2.0 win-rate standard errors and a paired-bootstrap 95% confidence interval of [0.04399, 0.06779] for the held-out representation-gap improvement (revised Table 6). These estimates characterize evaluation uncertainty and representation-level preference transfer, respectively, rather than variability across independent training runs. We have therefore revised the Abstract and Introduction to distinguish strong improvements on knowledge and reasoning tasks from competitive performance on instruction-following and open-ended generation benchmarks, avoiding overly broad claims about generation gains.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q3: Hyperparameter Selection and Fairness of Comparisons.

A3: To ensure fair comparisons, all methods are reproduced under the same experimental environment and shared training settings, except for their method-specific objectives and hyperparameters. Specifically, we use the same preference data and filtering protocol, LoRA fine-tuning configuration (rank 64, scaling factor 128, dropout 0.05), AdamW optimizer, cosine learning-rate schedule with 10% warmup, and one training epoch. We tune each method's hyperparameters following its original paper and report the best-performing configuration, rather than using arbitrarily fixed baseline settings.

Hyperparameter Selection. The following table summarizes the optimization objectives and hyperparameter ranges or settings considered for each method.

MethodObjectiveHyperparameter range or settings
SFT−log⁡πθ(y+∣x)-\log\pi_\theta(y^+\mid x)—
DPO−log⁡σ[β(Δθ−Δref)]-\log\sigma[\beta(\Delta_\theta-\Delta_{\mathrm{ref}})]β∈{0.1,0.5}\beta\in\{0.1,0.5\}
DPO w/ NLLLDPO+αLNLL\mathcal L_{\mathrm{DPO}}+\alpha\mathcal L_{\mathrm{NLL}}β∈{0.1,0.5}\beta\in\{0.1,0.5\}; α∈{0.01,0.02,0.05}\alpha\in\{0.01,0.02,0.05\}
SimPO−log⁡σ(βΔavg−γ)-\log\sigma(\beta\Delta_{\mathrm{avg}}-\gamma)β∈{0.1,2,10}\beta\in\{0.1,2,10\}; γ/β∈{0,0.25,0.5,0.75}\gamma/\beta\in\{0,0.25,0.5,0.75\}
ORPOLNLL−λlog⁡σ(log⁡(oθ+/oθ−))\mathcal L_{\mathrm{NLL}}-\lambda\log\sigma(\log(o_\theta^+/o_\theta^-))λ∈{0.1,0.5,2.0}\lambda\in\{0.1,0.5,2.0\}
AlphaDPO−log⁡σ[β(Δθ−Δ^ref)]-\log\sigma[\beta(\Delta_\theta-\widehat\Delta_{\mathrm{ref}})]β∈{0.1,2,10}\beta\in\{0.1,2,10\}; γ/β∈{0,0.5}\gamma/\beta\in\{0,0.5\}; α∈{10−3,10−2,10−1}\alpha\in\{10^{-3},10^{-2},10^{-1}\}
HyPO−log⁡σ(β(Δθ−max⁡(γ,Δref)))-\log\sigma(\beta(\Delta_\theta-\max(\gamma,\Delta_{\mathrm{ref}})))β∈{0.1,0.5}\beta\in\{0.1,0.5\}; γ∈{0,0.1}\gamma\in\{0,0.1\}
TI-DPOLweighted DPO+γLtriplet\mathcal L_{\mathrm{weighted\ DPO}}+\gamma\mathcal L_{\mathrm{triplet}}β∈{0.1,0.5}\beta\in\{0.1,0.5\}; λ∈{0.3,0.5,0.7}\lambda\in\{0.3,0.5,0.7\}; α∈{0.1,0.5}\alpha\in\{0.1,0.5\}; γ∈{0.1,0.5}\gamma\in\{0.1,0.5\}; K∈{8,64}K\in\{8,64\}
DPO w/ Online KL [1]LDPO+λ Ex∼ρ,y∼πθ[log⁡(πθ(y∣x)/πref(y∣x))]\mathcal L_{\mathrm{DPO}}+\lambda\,\mathbb E_{x\sim\rho,y\sim\pi_\theta}[\log(\pi_\theta(y\mid x)/\pi_{\mathrm{ref}}(y\mid x))]β∈{0.1,0.5}\beta\in\{0.1,0.5\}; λ∈{10−4,2×10−4}\lambda\in\{10^{-4},2\times10^{-4}\}; RLOO samples k=2k=2
OPRA (Ours)LDPO+λLRA+αLNLL\mathcal L_{\mathrm{DPO}}+\lambda\mathcal L_{\mathrm{RA}}+\alpha\mathcal L_{\mathrm{NLL}}β=0.1\beta=0.1; λ∈{10−4,10−3,10−2,10−1}\lambda\in\{10^{-4},10^{-3},10^{-2},10^{-1}\}; α∈{0,0.01,0.02}\alpha\in\{0,0.01,0.02\}; K∈{4,8}K\in\{4,8\}

Task-specific Settings. For knowledge and reasoning benchmarks, OPRA uses length-normalized DPO with chosen-response NLL supervision. For generation and instruction-following benchmarks, we use standard DPO without length normalization or NLL. Specifically, we set λ=10−4\lambda=10^{-4} and α=0\alpha=0, with K=4K=4 without preference-consistency gating for Qwen2.5-7B-Instruct and K=8K=8 with gating for Qwen3-14B.

Comparison Fairness. All methods share the same training-data protocol and principal fine-tuning settings, while their method-specific hyperparameters are tuned according to the corresponding original papers. We report the best-performing configurations under these shared experimental conditions. The objectives, implementations, and final training settings are detailed in revised Section 4.1 and Appendix C.4.

[1] The Importance of Online Data: Understanding Preference Fine-Tuning via Coverage. NeurIPS 2024.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewer,

Thank you again for your reviews. As the discussion period is nearing its end, we would greatly appreciate your feedback and reconsideration of our submission.

We have made the following revisions:

  1. Added controlled frozen-SFT and offline chosen-prefix comparisons to verify the contribution of on-policy anchoring (Table 5).
  2. Expanded generation evaluations to Qwen3-14B, and added RA ablations and uncertainty estimates (Tables 3–6).
  3. Clarified shared training settings, hyperparameter selection, and comparison fairness (Appendix C.4).

Importantly, OPRA achieves the best average performance across all backbone and dataset settings, covering 4 model families and 2 preference datasets, with substantial improvements on the 7 knowledge and reasoning benchmarks. We also revised our claims to distinguish these stronger gains from the more modest improvements on generation benchmarks.

Thank you for your time and consideration.

Reviewer_CrPrOpenReview ↗
Rating
3(+1)
Confidence
1(−2)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OPRA, which adds a representation-level loss to DPO. The model generates a short prefix and learns to make its hidden representation closer to the preferred answer than to the rejected one. Experiments across several models and datasets show gains, with useful ablations.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The reported empirical results are strong.
Critical weaknesses
  1. The novelty seems a little incremental. TI-DPO[1] already uses on-policy generated anchors, and S-SPPO[2] regularizes policy hidden states. OPRA’s combination is distinct and natural, though its contribution should be positioned more clearly against these methods.
  2. The proposed representation signal is under-explored. For example, I’m not sure what the eight-token anchor actually learns. If the chosen answer is better because of reasoning or facts introduced later, how does aligning such a short prefix help the model learn that difference?

Reference

  1. Token-Importance Guided Direct Preference Optimization. ICLR 2026.
  2. S-SPPO: Semantic-Calibrated Self-Play Preference Optimization. ICML 2026.
Questions for authors
  1. Could the authors position OPRA more clearly against these related methods?
  2. A deeper analysis of the representation signal would be great. For example, I am concerned about the eight-token anchor choice.
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for recognizing the empirical contributions of OPRA. We have clarified its distinction from closely related methods and further analyzed the role of short current-policy anchors.

Q1: Novelty and Distinction from TI-DPO and S-SPPO.

A1: The key novelty of OPRA is transferring offline chosen–rejected preference relations to representations induced by current-policy generations, rather than simply combining on-policy sampling with representation regularization.

Clarification of Novelty. TI-DPO uses on-policy anchors for token-level implicit-reward supervision, whereas S-SPPO calibrates chosen–rejected representations to improve preference separation and preserve latent diversity. In contrast, OPRA explicitly aligns current-policy-generated anchor representations toward chosen and away from rejected representations, using both offline representations as stop-gradient targets. The key distinction lies in where preference supervision is applied: OPRA directly constrains representations encountered by the evolving policy, without requiring additional preference annotations or an external reward model during training.

AspectTI-DPOS-SPPOOPRA (Ours)
On-policy anchor generation✓✗✓
Preference optimization objectiveReward-levelReward + RepresentationReward + Representation
Explicit constraints on on-policy representations✗✗✓
Explicit constraints on preference representations✗✓✗ (stop-gradient targets)
External reward model during training✗✓ (PairRM)✗
Addressing offline-to-on-policy mismatchReward-level✗Representation-level

Controlled evidence for the design distinction (revised Table 5). We evaluate the effects of alignment level and anchor source.

MethodSeven general tasks Avg. (%)AlpacaEval2.0 LC (%)
OPRA (current-policy anchor)77.3545.01
w/ Reward Alignment72.8241.89
w/ Chosen Prefix76.9044.06
w/ SFT Anchor76.3943.67

Replacing representation alignment with reward-level alignment reduces the seven-task average from 77.35% to 72.82%, while changing the anchor source also reduces performance. These ablations support the contributions of representation-level preference supervision and current-policy anchoring (revised Section 4.3, Table 5).

The comparison with S-SPPO is methodological rather than experimental. We have added its discussion and methodological comparison to revised Appendix A.2. S-SPPO was published in July 2026, shortly before our September 2026 submission.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)


Q2: Interpretation and Effectiveness of the Short On-Policy Anchor.

A2: The short anchor represents the current policy's early generation state rather than a truncated version of the complete chosen response. Although it may not contain the reasoning or factual information appearing later, its representation is aligned with the preference relation encoded by the complete chosen–rejected responses. This transfers offline response-level preference supervision to early representations encountered during current-policy generation (revised Section 3.2.1).

Anchor-source evidence. We compare different anchor sources under the same representation alignment objective.

AnchorSourceSeven general tasks Avg. (%)AlpacaEval2.0 LC (%)
On-policy anchorCurrent policy77.3545.01
Offline chosen prefixChosen response76.9044.06
Frozen-SFT anchorInitial SFT policy76.3943.67

Current-policy anchors achieve the best results in both evaluations (revised Section 4.3, Table 5). The knowledge and reasoning experiments use K=8K=8. For generation, Qwen2.5-7B-Instruct uses K=4K=4 without preference-consistency gating, whereas Qwen3-14B uses K=8K=8 with gating (revised Appendix C.4).

Representation-layer evidence. We additionally examine the effect of applying representation alignment at different hidden layers. As shown in revised Figure 3, deeper layers generally yield stronger knowledge and reasoning performance, with the final hidden layer achieving the best overall results. This provides complementary evidence for the choice of alignment representations, while the anchor-source ablations directly support the use of current-policy-generated prefixes.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewer,

Thank you again for your valuable reviews. As the discussion period is nearing its end, we would like to briefly highlight our revisions addressing your two main concerns:

  1. Novelty: We clarified OPRA's distinction from TI-DPO and S-SPPO. Unlike token-level reward supervision in TI-DPO or preference representation calibration in S-SPPO, OPRA explicitly transfers offline chosen–rejected preference relations to current-policy-generated representations (revised Sections 3.1–3.2 and Appendix A.2).
  2. Short on-policy anchors: We clarified how short prefixes receive supervision from complete chosen–rejected response representations (Section 3.2.1). Additional anchor-source ablations (Table 5) show that current-policy anchors outperform frozen-SFT and offline chosen-prefix alternatives. We also included hidden-layer analysis (Figure 3).

We hope these clarifications and additional experiments address your concerns regarding the methodological contribution and the role of short anchors. Thank you for your time and consideration.

Reviewer_AWqoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes OPRA, which develops a contrastive representation loss in addition to preference optimization using short prefixes sampled from the current policy. The method is straightforward, and the ablations provide some evidence that the representation loss helps beyond the standard training objective.

Critical strengths

The method is straightforward, and the ablations provide some evidence that the representation loss helps beyond the standard training objective.

Critical weaknesses

My main concern is the motivation. The fact that preference scores do not uniquely determine hidden representations does not explain why those representations need to satisfy the proposed cosine-similarity constraint. I am not convinced that the paper identifies a concrete failure of output-level preference optimization that this constraint addresses. A comparison with appropriately tuned output-space regularization, including KL controls, would help. More importantly, some difference between the offline data distribution and the learned policy is expected during policy improvement. I think the paper needs to explain which aspect of this difference is harmful and show how OPRA mitigates it.

The theory does not resolve this concern. Proposition 1 establishes representation underdetermination, while Theorem 1 bounds violations of the representation ordering imposed by the loss itself. Neither connects this ordering to obtain better responses.

The empirical gains over strong baselines are modest such as the average improvement over HyPO on Gemma and the MT-Bench improvement over TIDPO. Without reported training variability or confidence intervals, it is difficult to assess whether these differences are reliable. The appendix includes models up to 14B, but generation-quality evaluation remains limited to one 7B backbone.

Questions for authors

see comments

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the constructive assessment. We have clarified the motivation and theoretical scope, added output-space regularization controls, and expanded the ablations and held-out generation evaluations.

Q1: Motivation for Representation Alignment and Offline-to-On-Policy Mismatch.

A1: Our initial observation is that correct offline preference rankings do not necessarily imply consistent preference relations in representations induced by current-policy generations (revised Sections 2 and 3.1). OPRA addresses this discrepancy by transferring offline chosen–rejected preference relations to current-policy anchor representations through relative alignment (Section 3.2).

Clarification of Motivation. The key challenge is not the distribution shift itself, but extending preference supervision from fixed offline responses to representations encountered by the evolving policy. While DPO optimizes preference rankings on annotated pairs, it does not explicitly constrain representations of current-policy generations. Our diagnostics (revised Appendix E, Table 12) reveal this supervision gap.

Output–representation diagnostics. We evaluate 1,997 held-out UltraFeedback preference pairs, measuring whether correct offline preference rankings correspond to preference-consistent representations of current-policy-generated anchors.

ModelOutput Acc. (%) ↑Conditional Rep. Violation (%) ↓Mean Rep. Gap ↑
SFT61.3440.33+0.04391
DPO72.6144.90+0.02163
DPO w/ NLL73.7160.26−0.02421

Conditional violations are measured among correctly ranked pairs. SFT uses likelihood rankings; DPO variants use reference-relative implicit rewards.

Compared with DPO, DPO w/ NLL achieves higher offline ranking accuracy (73.71% vs. 72.61%) but exhibits more representation violations (60.26% vs. 44.90%). This discrepancy motivates complementary supervision on current-policy representations, rather than relying solely on fixed offline preference rankings.

Why relative representation alignment? Our objective is to transfer annotated preference ordering, rather than impose a theoretically necessary cosine geometry. The loss ablations (revised Table 7) show that relative alignment outperforms positive-only alignment. On Qwen2.5-7B-Instruct, positive-only alignment achieves 76.53%, compared with 77.35% for NCE and 77.09% for hinge triplet. On Gemma2-9B-it, the corresponding results are 77.43%, 79.57%, and 78.16%. These findings support relative preference supervision beyond simply matching chosen representations, without requiring a unique similarity constraint.

Comparison with output-space regularization. We additionally compare OPRA with DPO w/ Online KL [1]. Both use current-policy generations, but DPO w/ Online KL regularizes output probability distributions through reverse KL, whereas OPRA directly aligns current-policy anchor representations with offline chosen–rejected preference targets.

Preference dataBackboneDPO Avg. (%)DPO w/ Online KL Avg. (%)OPRA Avg. (%)
UltraFeedbackQwen2.5-7B73.7773.5977.35
UltraFeedbackGemma2-9B76.6076.4179.57
ORPOQwen2.5-7B72.9072.8774.54
ORPOGemma2-9B75.8275.4776.50

OPRA outperforms DPO w/ Online KL in all four backbone–dataset settings (revised Tables 1–2). The offline-versus-on-policy data comparison (revised Table 8) further shows that OPRA remains effective under both data settings, suggesting that representation-level preference supervision is complementary to changes in the training-data distribution.

Generation and instruction-following comparison. We compare DPO, DPO w/ Online KL, and OPRA on MT-Bench and AlpacaEval2.0 using Qwen2.5-7B-Instruct and Qwen3-14B (revised Table 4). AlpacaEval2.0 win rates are reported with standard errors. Unlike the original submission, we consistently use GPT-4o-mini as the judge and GPT-4-Turbo as the AlpacaEval reference response source.

BackboneMethodMT-BenchAlpacaEval2.0 LC (%)AlpacaEval2.0 WR (%) ± SE
Qwen2.5-7B-InstructDPO7.4344.7943.10 ± 1.75
DPO w/ Online KL7.4244.0242.42 ± 1.74
OPRA7.5345.0143.42 ± 1.75
Qwen3-14BDPO7.4762.6166.27 ± 1.66
DPO w/ Online KL7.3462.9366.77 ± 1.66
OPRA7.4964.7968.26 ± 1.64

OPRA achieves higher MT-Bench and AlpacaEval2.0 scores than DPO w/ Online KL on both backbones, providing additional empirical support for representation-level preference supervision beyond output-distribution regularization.

[1] The Importance of Online Data: Understanding Preference Fine-Tuning via Coverage. NeurIPS 2024.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Q2: Theoretical Justification and Connection to Response Quality.

A2: Proposition 1 shows that scalar preference statistics do not uniquely determine representation-level relations, while Theorem 1 bounds violations of the proposed representation ordering. We evaluate the downstream effects of representation alignment and current-policy anchoring through controlled ablations.

Representation supervision and anchor-source ablation (revised Table 5). We vary the alignment level and anchor source while retaining the underlying preference-training objectives.

MethodAlignment levelAnchor sourceSeven-task Avg. (%)AlpacaEval2.0 LC (%)
OPRARepresentationCurrent policy77.3545.01
w/o RA——76.0544.79
w/ Chosen PrefixRepresentationOffline chosen76.9044.06
w/ SFT AnchorRepresentationFrozen SFT76.3943.67
w/ Reward AlignmentRewardCurrent policy72.8241.89

Removing RA reduces the seven-task average by 1.30 percentage points. Replacing current-policy anchors with offline chosen or frozen-SFT anchors also reduces performance. Moreover, replacing representation alignment with reward-level alignment yields 72.82%, compared with 77.35% for OPRA. These results provide empirical evidence for the contributions of representation-level supervision and current-policy anchoring.

Held-out representation preference transfer (revised Table 6). We compare representation preference gaps on the 1,437 held-out pairs correctly ranked by both DPO and OPRA under the knowledge and reasoning configuration.

Shared pairsDPO mean gapOPRA mean gapΔ gapPaired-bootstrap 95% CIOPRA larger gap (%)
1,437−0.05692−0.00076+0.05615[0.04399, 0.06779]65.14

OPRA improves the mean representation preference gap by +0.05615, with a paired-bootstrap confidence interval excluding zero. Together, Table 5 establishes the downstream benefits under the evaluated settings, while Table 6 directly measures improved preference relations in policy-generated representations. These analyses complement the theoretical characterization in revised Section 3.1 and Appendix B.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q3: Statistical Reliability, Strong Baselines, and Generation Generalizability.

A3: We expand the evaluation across model families, preference datasets, and generation benchmarks. OPRA outperforms HyPO in all six evaluated backbone–dataset settings, although the improvement on UltraFeedback using Gemma2-9B-it is small. We also add generation evaluations on Qwen3-14B and report AlpacaEval2.0 win-rate standard errors.

Comparison with HyPO across backbones and datasets. The following results evaluate knowledge and reasoning performance across multiple model families and preference datasets.

Preference dataBackboneDPO Avg. (%)HyPO Avg. (%)OPRA Avg. (%)OPRA − HyPO (pp)
UltraFeedbackQwen2.5-7B-Instruct73.7774.5477.35+2.81
UltraFeedbackGemma2-9B-it76.6079.5279.57+0.05
ORPOQwen2.5-7B-Instruct72.9071.0574.54+3.49
ORPOGemma2-9B-it75.8275.6876.50+0.82
UltraFeedbackLlama3-8B-Instruct71.0672.4773.40+0.93
UltraFeedbackMistral-7B-Instruct-v0.371.0672.0273.37+1.35

The gains over HyPO vary across model families, but OPRA achieves the highest seven-task average in all six reported settings (revised Tables 1–2 and Appendix D.5).

Comparison across preference-data conditions (Qwen2.5-7B). We further vary data filtering, dataset choice, and preference annotation.

Preference-data settingDPO Avg. (%)HyPO Avg. (%)OPRA Avg. (%)OPRA − HyPO (pp)
UltraFeedback, unfiltered73.3475.3977.14+1.75
UltraFeedback, filtered73.7774.5477.35+2.81
ORPO, unfiltered72.9471.6875.81+4.13
ORPO, filtered72.9071.0574.54+3.49
UltraFeedback, GPT-4o annotation73.7774.9577.31+2.36

OPRA outperforms HyPO under all five tested preference-data conditions (revised Section 4.4 and Appendix D.4), supporting the robustness of its knowledge and reasoning performance across data sources and annotations.

Instruction following and open-ended generation. We evaluate MT-Bench and AlpacaEval2.0 using GPT-4o-mini as the judge and GPT-4-Turbo as the AlpacaEval reference response source (revised Table 4).

BackboneMethodMT-BenchAlpacaEval2.0 LC (%)AlpacaEval2.0 WR (%) ± SE
Qwen2.5-7B-InstructDPO7.4344.7943.10 ± 1.75
DPO w/ NLL7.2634.3126.34 ± 1.55
HyPO7.3837.5736.40 ± 1.70
TI-DPO7.3440.7238.93 ± 1.72
DPO w/ Online KL7.4244.0242.42 ± 1.74
OPRA7.5345.0143.42 ± 1.75
Qwen3-14BDPO7.4762.6166.27 ± 1.66
HyPO7.3749.6348.07 ± 1.76
TI-DPO7.2362.4164.72 ± 1.68
DPO w/ Online KL7.3462.9366.77 ± 1.66
OPRA7.4964.7968.26 ± 1.64

Relative to DPO, OPRA improves AlpacaEval2.0 LC/WR by +0.22/+0.32 percentage points on Qwen2.5-7B-Instruct and +2.18/+1.99 points on Qwen3-14B. The added 14B results extend the generation-quality evaluation beyond the original 7B backbone.

Held-out policy-generation quality (Qwen2.5-7B). We additionally evaluate 2,000 held-out UltraFeedback prompts using a pairwise preference judge, reporting win rates against DPO (revised Table 3).

MethodPairwise win rate vs. DPO (%)
SFT41.20
DPO50.00
HyPO44.15
TI-DPO43.65
DPO w/ Online KL55.70
OPRA57.53

OPRA achieves a 57.53% pairwise win rate against DPO (reported p < 0.001), providing additional evidence of its open-ended generation quality.

Statistical reliability and revised claims. We report AlpacaEval2.0 win-rate standard errors and a paired-bootstrap 95% confidence interval for representation preference transfer (revised Table 6). These estimates do not characterize variation across independent training runs. Accordingly, we have revised the Abstract and Introduction to emphasize strong improvements on knowledge and reasoning tasks and competitive performance on instruction-following and open-ended generation benchmarks, avoiding overly broad claims about generation gains.

I appreciate the authors’ response and their careful answers to my questions. I have increased the presentation and soundness scores. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Area Chair

Dear Reviewers and Area Chair,

We thank the reviewers for their constructive feedback. We have clarified the motivation, added experiments, and detailed the experimental settings. All changes are highlighted in blue in the revised manuscript. The key points are summarized below.

1. Motivation and Effectiveness (Q1–A1 and Q2–A2 for Reviewer AWqo). DPO learns from fixed offline preference pairs, while the policy generates different responses during training and inference. Our diagnostics on held-out pairs (Table 12) show that correct offline preference rankings do not guarantee consistent preference relations in on-policy representations. OPRA extends preference supervision to current-policy representations through on-policy representation alignment. Additional analyses show improved representation preference consistency (Table 6) and downstream performance (Table 5).

2. Methodological Contribution (Q1–A1 and Q2–A2 for Reviewer CrPr; Q1–A1 for Reviewer Cd18). We clarified the differences between OPRA and TI-DPO[1], S-SPPO[2], and Online KL[3]. OPRA transfers offline chosen–rejected preference relations to current-policy-generated representations without additional preference annotations. New controlled ablations demonstrate the contributions of representation alignment and current-policy anchoring (Table 5). We also clarified how short on-policy prefixes receive preference supervision from complete responses.

[1] Token-Importance Guided Direct Preference Optimization. ICLR 2026. [2] S-SPPO: Semantic-Calibrated Self-Play Preference Optimization. ICML 2026. [3] The Importance of Online Data: Understanding Preference Fine-Tuning via Coverage. NeurIPS 2024.

3. Experimental Results and Reliability (Q3–A3 for Reviewer AWqo; Q2–A2 and Q3–A3 for Reviewer Cd18). We expanded generation evaluations to Qwen3-14B, added uncertainty estimates and controlled ablations, and clarified the training settings and hyperparameter selection. OPRA achieves the best seven-task knowledge and reasoning average across all evaluated backbone and dataset settings, covering 4 model families and 2 preference datasets, while remaining competitive on generation benchmarks.

We believe these revisions address the reviewers' main concerns and provide stronger evidence for OPRA's contributions. We welcome further feedback on any remaining concerns.

The Authors of Submission 22983

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.