- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OPRA adds a representation loss to DPO. It generates a short prefix with the model being trained, then pulls its hidden representation closer to the preferred response and away from the rejected response. A filter removes pairs whose preference labels disagree with the model's implicit reward ranking. The knowledge and reasoning experiments also train on the preferred responses with a likelihood loss. Results across several models and datasets suggest that the representation loss helps.
Critical strengths
- The ablations show that the representation loss helps. Removing it lowers the seven-task average by 1.30 percentage points on Qwen and 1.77 on Gemma. Removing the filter also reduces performance (Table 5). Table 4 further compares representation-based and reward-based supervision.
- The experiments cover a useful range of settings. The paper tests two preference datasets, several model families and sizes, and different sources of preference labels. Training takes 1.32 times as long as DPO in the reported setup (Table 9).
Critical weaknesses
- It is unclear how much the method gains from using the current model's outputs. The ablations do not keep the representation loss fixed while changing the prefix source to a frozen model or previously saved outputs. The improvement could therefore come from the added loss more generally. Proposition 1 and Theorem 1 also do not show that the representation ordering leads to better responses or reduces the mismatch with offline data.
- Some gains over the strongest baselines are small. The Gemma average is only 0.05 percentage points above HyPO, and MT-Bench is 0.04 points above TIDPO. AlpacaEval LC is slightly below HyPO (Tables 1 and 3). Without uncertainty estimates, it is hard to judge these differences. The generation tests also use a different loss setup and custom judges or reference models, so the results need to be interpreted within those settings (Appendices C.1 and C.5).
Questions for authors
- What happens if the prefixes come from a frozen model or previously saved outputs, with the same representation loss? Does using the current model give better generation results?
- Can you report variation across runs or confidence intervals for the strongest-baseline comparisons, including generation results without the representation loss?
- How were the loss settings and hyperparameters selected? Did OPRA and the baselines receive comparable tuning budgets, especially for generation tasks?
4 comments
Official Comment by Authors (2/3)
Q2: Statistical Reliability and Generation Quality Without Representation Alignment.
A2: We rerun the generation benchmarks using GPT-4o-mini as the judge, extend the evaluation to Qwen3-14B, and add generation ablations without representation alignment.
1. Instruction following and open-ended generation. We evaluate MT-Bench and AlpacaEval2.0 using GPT-4o-mini as the judge/annotator and GPT-4-Turbo as the AlpacaEval reference response source. AlpacaEval2.0 win rates are reported with standard errors (revised Section 4.2, Table 4).
| Backbone | Method | MT-Bench | AlpacaEval2.0 LC (%) | AlpacaEval2.0 WR (%) ± SE |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct | DPO | 7.43 | 44.79 | 43.10 ± 1.75 |
| DPO w/ NLL | 7.26 | 34.31 | 26.34 ± 1.55 | |
| HyPO | 7.38 | 37.57 | 36.40 ± 1.70 | |
| TI-DPO | 7.34 | 40.72 | 38.93 ± 1.72 | |
| DPO w/ Online KL | 7.42 | 44.02 | 42.42 ± 1.74 | |
| OPRA | 7.53 | 45.01 | 43.42 ± 1.75 | |
| Qwen3-14B | DPO | 7.47 | 62.61 | 66.27 ± 1.66 |
| HyPO | 7.37 | 49.63 | 48.07 ± 1.76 | |
| TI-DPO | 7.23 | 62.41 | 64.72 ± 1.68 | |
| DPO w/ Online KL | 7.34 | 62.93 | 66.77 ± 1.66 | |
| OPRA | 7.49 | 64.79 | 68.26 ± 1.64 |
OPRA achieves the highest MT-Bench and AlpacaEval2.0 scores among the evaluated methods on both backbones. Relative to DPO, its AlpacaEval2.0 LC/WR gains are +0.22/+0.32 pp on Qwen2.5-7B-Instruct and +2.18/+1.99 pp on Qwen3-14B. The additional 14B evaluation extends the generation-quality comparison beyond the original 7B backbone.
2. Generation ablation without representation alignment. We evaluate the contribution of RA and alternative anchor sources while retaining the underlying preference-training objectives (revised Section 4.3, Table 5).
| Method | Seven general tasks Avg. (%) | AlpacaEval2.0 LC (%) |
|---|---|---|
| OPRA | 77.35 | 45.01 |
| w/o RA | 76.05 | 44.79 |
| w/ Chosen Prefix | 76.90 | 44.06 |
| w/ SFT Anchor | 76.39 | 43.67 |
| w/ Reward Alignment | 72.82 | 41.89 |
Removing RA reduces the seven-task average by 1.30 pp and AlpacaEval2.0 LC by 0.22 pp. The anchor-source and reward-alignment controls further support the contributions of current-policy anchoring and representation-level supervision, although the generation gain over the w/o RA variant is modest.
3. Held-out policy-generation quality. We further evaluate 2,000 held-out UltraFeedback prompts using two reward models and a pairwise preference judge. The pairwise win rates against DPO are summarized below (revised Section 4.2, Table 3).
| Method | Pairwise win rate vs. DPO (%) |
|---|---|
| DPO | 50.00 |
| HyPO | 44.15 |
| TI-DPO | 43.65 |
| DPO w/ Online KL | 55.70 |
| OPRA | 57.53 |
OPRA achieves a 57.53% pairwise win rate against DPO (reported p < 0.001), while obtaining comparable reward-model scores (revised Table 3). This provides additional evidence for the quality of policy-generated responses.
Statistical reliability and revised claims. We report AlpacaEval2.0 win-rate standard errors and a paired-bootstrap 95% confidence interval of [0.04399, 0.06779] for the held-out representation-gap improvement (revised Table 6). These estimates characterize evaluation uncertainty and representation-level preference transfer, respectively, rather than variability across independent training runs. We have therefore revised the Abstract and Introduction to distinguish strong improvements on knowledge and reasoning tasks from competitive performance on instruction-following and open-ended generation benchmarks, avoiding overly broad claims about generation gains.
Official Comment by Authors (3/3)
Q3: Hyperparameter Selection and Fairness of Comparisons.
A3: To ensure fair comparisons, all methods are reproduced under the same experimental environment and shared training settings, except for their method-specific objectives and hyperparameters. Specifically, we use the same preference data and filtering protocol, LoRA fine-tuning configuration (rank 64, scaling factor 128, dropout 0.05), AdamW optimizer, cosine learning-rate schedule with 10% warmup, and one training epoch. We tune each method's hyperparameters following its original paper and report the best-performing configuration, rather than using arbitrarily fixed baseline settings.
Hyperparameter Selection. The following table summarizes the optimization objectives and hyperparameter ranges or settings considered for each method.
| Method | Objective | Hyperparameter range or settings |
|---|---|---|
| SFT | — | |
| DPO | ||
| DPO w/ NLL | ; | |
| SimPO | ; | |
| ORPO | ||
| AlphaDPO | ; ; | |
| HyPO | ; | |
| TI-DPO | ; ; ; ; | |
| DPO w/ Online KL [1] | ; ; RLOO samples | |
| OPRA (Ours) | ; ; ; |
Task-specific Settings. For knowledge and reasoning benchmarks, OPRA uses length-normalized DPO with chosen-response NLL supervision. For generation and instruction-following benchmarks, we use standard DPO without length normalization or NLL. Specifically, we set and , with without preference-consistency gating for Qwen2.5-7B-Instruct and with gating for Qwen3-14B.
Comparison Fairness. All methods share the same training-data protocol and principal fine-tuning settings, while their method-specific hyperparameters are tuned according to the corresponding original papers. We report the best-performing configurations under these shared experimental conditions. The objectives, implementations, and final training settings are detailed in revised Section 4.1 and Appendix C.4.
[1] The Importance of Online Data: Understanding Preference Fine-Tuning via Coverage. NeurIPS 2024.
Follow-up on Our Rebuttal and Revised Manuscript
Dear Reviewer,
Thank you again for your reviews. As the discussion period is nearing its end, we would greatly appreciate your feedback and reconsideration of our submission.
We have made the following revisions:
- Added controlled frozen-SFT and offline chosen-prefix comparisons to verify the contribution of on-policy anchoring (Table 5).
- Expanded generation evaluations to Qwen3-14B, and added RA ablations and uncertainty estimates (Tables 3–6).
- Clarified shared training settings, hyperparameter selection, and comparison fairness (Appendix C.4).
Importantly, OPRA achieves the best average performance across all backbone and dataset settings, covering 4 model families and 2 preference datasets, with substantial improvements on the 7 knowledge and reasoning benchmarks. We also revised our claims to distinguish these stronger gains from the more modest improvements on generation benchmarks.
Thank you for your time and consideration.
Official Comment by Authors (1/3)
We thank the reviewer for the constructive comments. We have added controlled anchor-source ablations, expanded generation evaluations, and clarified the training configurations and statistical evidence.
Q1: Contribution of Current-Policy Anchors Beyond Representation Alignment.
A1: We isolate the contribution of current-policy anchoring by keeping the representation alignment objective fixed and varying the anchor source (revised Section 4.3, Table 5).
Current-policy anchors outperform both frozen-SFT anchors (+0.96/+1.34 pp) and offline chosen prefixes (+0.45/+0.95 pp) on the seven-task average and AlpacaEval2.0 LC, respectively. Removing RA also reduces performance, separating the contributions of representation supervision and anchor selection. These results support the benefit of aligning representations encountered by the evolving policy rather than relying solely on fixed offline or frozen-model prefixes.
Held-out representation preference transfer. We further evaluate preference-relation transfer on 1,437 held-out pairs correctly ranked by both DPO and OPRA under the knowledge and reasoning configuration (revised Table 6).
OPRA improves the representation preference gap on shared held-out pairs, providing complementary evidence that offline preference relations are more consistently reflected in current-policy-generated representations (revised Section 4.3, Table 6; Appendix E).