acceptodds
Under review as a conference paper at ICLR 2027

Multi-Level Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a predominant approach for aligning artificial intelligence systems with human preferences, demonstrating exceptional and measurable efficacy in instruction following tasks; however, it exhibits insufficient compliance capabilities when confronted with complex multi-instruction tasks. Conventional approaches rely heavily on human annotation or more sophisticated large language models, thereby introducing substantial resource expenditure or potential bias concerns. Meanwhile, alternative synthetic methods that augment standard preference datasets often compromise the model's semantic quality. Our research identifies a critical oversight in existing techniques, which predominantly focus on comparing responses while neglecting valuable latent signals embedded within prompt inputs, and which only focus on preference disparities at the intra-sample level, while neglecting to account for the inter-sample level preference differentials that exist among preference data. To leverage these previously neglected indicators, we propose a novel Multi-level Preference Learning (MPL) framework, capable of enhancing multi-instruction capabilities. Specifically, for any given response in original preference data pairs, we construct varied prompts with a preference relation under different conditions, in order to learn intra-sample level preference disparities. Furthermore, for any given original preference pair, we synthesize multi-instruction preference pairs to capture preference discrepancies at the inter-sample level. Building on the two datasets constructed above, we consequently devise two sophisticated training objective functions. Subsequently, our framework integrates seamlessly into both Reward Modeling and Direct Preference Optimization paradigms. Through rigorous evaluation across multiple benchmarks, we empirically validate the efficacy of our framework.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model, Reinforcement Learning, Multi-instruction Following

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8r1SOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper augments existing preference data with programmable format constraints, constructing two types of preference relationships: prompt-side (intra-sample) and preference gap-side (inter-sample), and designs corresponding losses. These can be integrated with RM training and DPO to improve the ability to follow multi-constraint instructions.

Critical strengths

Strengths

The motivation is intuitive. Labels are programmatically verifiable, do not rely on stronger LLMs, and are low-cost.

The method is applicable to both RM and DPO, and is simple to implement. It covers 6 backbones, including equal-volume ablation, OOD instruction types, and PPO downstream experiments.

Critical weaknesses

Weaknesses

Limited novelty; the claim of being "overlooked" is too strong.

The Intra-sample objective directly follows Rc-BT.

"Input-side preferences" have been systematically studied by IOPO (Input-Output Preference Optimization, 2024); the paper merely lists them without comparison.

The data construction is of the same origin as LIFT/M-LIFT.

The "preference gap ranking" in the Inter-sample is a new element, but its assumption (adding constraints necessarily widens or narrows the gap) is heuristic and unverified.

The baseline is weak, lacking key baselines.

The OpenAssistant data used by Vanilla DPO is itself unrelated to instruction compliance; any augmentation targeting format constraints will beat it. M-LIFT is the authors' own extension.

Since there are already procedural verification functions for each type of constraint, the most natural baseline is reinforcement learning based on verifiable rewards, such as RLVR for instruction constraints in Tulu 3, or directly using GRPO.

Comparisons with IOPO, Conifer, RPO, etc., are also lacking.

The evaluation and training metrics are highly homospersed.

The IF metric of RM is measured on D^if_eval constructed using the same method as training; an accuracy of 91–98% does not indicate generalization.

The prompt-level metric of IFEval includes 10 constraints seen during training and uses the same validation code.

The improvement in Inst (OOD), which truly reflects generalization, is significantly smaller; for example, Llama-3.1-8B only improves by 1.2.

Gains on new models are small and lack significance testing.

Qwen3-4B's Prompt metric: 77.41 vs. M-LIFT 77.04; Qwen3-14B: 81.30 vs. MuSC 80.19.

IFEval has approximately 541 prompts; one prompt is approximately equal to 0.18%; except for Qwen3-8B, all are single-run tests.

The ComplexBench benchmark, which only tested one model, scored 0.36 higher than Vanilla DPO, insufficient to support the conclusion that it is "transferable to natural instructions."

There are questionable values.

In Table 1, the Vanilla RM for Qwen3-8B achieved a SQ accuracy of 99.35% on OpenAssistant rank0 versus rank1, while Qwen2 and Llama only achieved approximately 61–62%.

OASST's ranking itself is quite noisy; a 99% accuracy rate suggests data contamination or a testing bug. This could affect trust in the entire RM evaluation process.

Questions for authors

See Weaknesses

5 comments

We sincerely thank you for your insightful and constructive comments. In response, we have thoroughly revised the manuscript to enhance its clarity and rigor. Below, we address each of your comments in turn, and the corresponding revisions will be included in the revised paper.

Q1: Limited novelty, and the claim of being "overlooked" is too strong.

A1: Thank you for the candid assessment. We address each point below.

(1) The claim of being "overlooked" is too strong.

We are sorry for the lack of clarity. "Overlooked" referred to multi-instruction following, where human annotation, LLM-based augmentation (e.g., Conifer and MuSC), and programmatic augmentation (e.g., LIFT) all compare responses under a fixed prompt and do not use prompt-side preference signals. The exception is IOPO, which relies on teacher-generated data. In contrast, MPL models intra-sample and inter-sample relations jointly, from existing human preference pairs and without a teacher model. We will state this precisely in the Abstract and Introduction of the revised paper.

(2) The intra-sample objective directly follows Rc-BT.

We are sorry for the lack of clarity. The functional form of the intra-sample objective follows Rc-BT (Eqn. 3), and our contribution at this level is the data construction and its extension to multiple instructions.

  • Multiple heterogeneous instructions. Rc-BT uses a single word-count constraint, while MPL composes instructions from 10 verifiable types and generalizes to 15 held-out types.
  • Partial satisfaction. MPL contrasts two augmented prompts with opposite parameter settings and defines preferences under partial satisfaction (Algorithm 1), which has no counterpart with a single constraint.

The intra-sample level alone raises Prompt from 49.26 to 56.25 on Qwen2-7B (Table 6).

(3) Input-side preferences have been studied by IOPO, which the paper lists without comparison.

We are sorry for the lack of clarity. We now discuss IOPO in Section 2. IOPO's loss and our inter-sample term share the same algebraic form, but they differ in data and relation.

  • Data. IOPO relies on TRACE, 120K samples generated by a 72B teacher. MPL appends verifiable instructions to the original human-ranked pairs programmatically, with no LLM call.
  • Relation. In IOPO the preference between the two responses reverses across prompts. MPL keeps ywy_w preferred and only orders the size of the gap.

IOPO's training data has not been released (the repository contains a single example), and its construction is not described in enough detail for an exact reproduction. We therefore re-implemented its grouping and loss on our data under the same settings as MPL-DPO on Qwen2-7B, which may differ from the original in some details.

IFEval PromptInst (OOD)
Vanilla DPO49.2658.90
IOPO grouping + IOPO loss (re-implemented on our data)53.1560.12
BT + inter-sample level (Ours w/o intra-sample level)54.0460.67
Ours57.5464.18

The IOPO-style objective improves over Vanilla DPO and matches our BT + inter-sample variant, while full MPL is about 4 points higher, showing the benefit of supervising all three levels jointly.

(4) The data construction is of the same origin as LIFT/M-LIFT.

The construction differs from LIFT in three respects.

  • Scope. LIFT adds a single length constraint. M-LIFT is our multi-instruction extension of it, used as the closest programmatic baseline.
  • Labels. LIFT and M-LIFT flip the preference when ywy_w violates the added constraint. MPL never flips labels.
  • Effect. Flipping hurts semantic quality. On Qwen2-7B, M-LIFT lowers the AlpacaEval SQ win rate from 54.86 to 44.51, while MPL raises it to 58.73 (Table 5).

(5) The gap-ordering assumption of the inter-sample level is heuristic and unverified.

The gap ordering is not merely heuristic. It follows from a simple condition on the reward, and human judgments confirm it.

Theoretically, let Δ(x)=r∗(x,yw)−r∗(x,yl)\Delta(x) = r^*(x, y_w) - r^*(x, y_l) be the preference gap under prompt xx, and suppose the reward under an augmented prompt x′x' is the original reward plus a compliance term, r∗(x′,y)=r∗(x,y)+c(x′,y)r^*(x', y) = r^*(x, y) + c(x', y), where cc is higher for a response that satisfies all added instructions than for one that violates at least one. Under the gap-expanding prompt xwx_w, ywy_w satisfies all added instructions and yly_l violates at least one, so Δ(xw)−Δ(x)=c(xw,yw)−c(xw,yl)>0\Delta(x_w) - \Delta(x) = c(x_w, y_w) - c(x_w, y_l) > 0. Under the gap-reducing prompt xlx_l the roles are swapped, so Δ(x)−Δ(xl)=c(xl,yl)−c(xl,yw)>0\Delta(x) - \Delta(x_l) = c(x_l, y_l) - c(x_l, y_w) > 0. These are exactly the two orderings MPL uses, and neither requires yly_l to be preferred over ywy_w.

Empirically, we checked whether humans agree with the constructed orderings. We randomly sampled 50 gap-expanding and 50 gap-reducing relations without filtering. Three annotators, blind to the construction and to our hypothesis, saw the original and the augmented prompt with the two responses in random order, and judged whether the better response's advantage becomes larger, stays the same, or becomes smaller under the augmented prompt. We report the majority vote.

NAgreeNo changeOppositeFleiss' κ
Gap-expanding5092%8%0%0.68
Gap-reducing5078%18%4%0.60

Humans almost never contradict the constructed direction (0/50 and 2/50). The remaining "no change" cases are label noise without a sign error.

These revisions are in Section 2 (L091-L100), Section 3.2 (L152-L155), Section 3.3 (L212-L223), and Section 4.5 and Table 9 (L441-L467). We will also update the Abstract and Introduction accordingly in the revised paper.

Q2: The baselines are weak, lacking key baselines.

A2: Thank you for the suggestions. We address each point below.

(1) Vanilla DPO on OpenAssistant is unrelated to instruction compliance, and M-LIFT is the authors' own extension.

Vanilla DPO serves as the reference for semantic quality, since methods that add constraints can lose it (e.g., M-LIFT and MuSC lower the Qwen2-7B SQ win rate from 54.86 to 44.51 and 43.9, Table 5). For constraint following, the augmentation-based baselines (M-LIFT, IOPO, RPO, and our BT + inter-sample variant) use the same constraint-augmented prompts and reach 53.15 to 56.30 Prompt on Qwen2-7B, while MPL reaches 57.54. MPL also outperforms M-LIFT at equal data volume (53.49 vs. 50.37, Table 7), and MuSC is an independent published method.

(2) The most natural baseline is RLVR with the verification functions, e.g., Tulu 3-style RLVR or GRPO.

We added GRPO on Qwen2-7B with the same augmented prompts and verification functions (the reward is the fraction of satisfied instructions, with 8 rollouts per prompt on the same hardware).

IFEval PromptInst (OOD)SQ Win RateGPU-hours
Vanilla DPO49.2658.9054.862.6
RLVR (GRPO)60.7462.4249.8146.8
Ours57.5464.1858.7314.3
Ours → RLVR (GRPO initialized from Ours)62.2265.6456.1561.1
  • RLVR is strongest on the seen instruction types, since its reward uses the IFEval verifier.
  • MPL generalizes better and preserves semantic quality, at under a third of the GPU-hours.
  • The two are complementary. Since a key contribution of MPL lies in its data, the same data can also be used in GRPO. Our augmented prompts serve as the RLVR training prompts above, and initializing RLVR from MPL gives the best IFEval results.

(3) Comparisons with IOPO, Conifer, RPO, etc. are lacking.

We added all three on Qwen2-7B. IOPO and RPO are re-implemented on our data (RPO reverses the constraints that ywy_w violates).

IFEval PromptInst (OOD)
Vanilla DPO49.2658.90
IOPO (re-implemented on our data)53.1560.12
RPO (re-implemented on our data)54.6360.28
Ours57.5464.18
Conifer SFT → Vanilla DPO54.0762.73
  • MPL outperforms both re-implementations on the same data by 2.9 to 4.4 points.

These baselines will be added to Section 4.1 and Section 4.3 in the revised paper.

Reviewer_UJmiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Multi-Level Preference Learning (MPL), a framework designed to improve the complex multi-instruction following capabilities of large language models through preference learning. The authors argue that conventional RLHF methods primarily focus on response-level preference comparisons while overlooking preference signals embedded in prompts and discrepancies across preference pairs.

Critical strengths

The paper is well-written, with clear motivation and a simple, easy-to-follow methodology.

The framework efficiently constructs preference data using programmatically verifiable instructions without requiring stronger LLMs.

Extensive experiments across different model families demonstrate consistent improvements in instruction-following performance.

Critical weaknesses

The proposed intra-sample objective closely resembles the Response-Conditioned Bradley–Terry (Rc-BT) framework [1], raising concerns about methodological novelty.

A strong mathematical similarity exists between the inter-sample objective and IOPO [2], yet their relationship is not sufficiently discussed, and direct experimental comparisons are missing.

Assuming a particular ordering of inter-sample preference gaps requires stronger theoretical or empirical justification.

Performance gains on stronger models and naturally occurring complex instructions remain relatively modest, especially on ComplexBench.

Further analysis of computational overhead and statistical significance is needed to establish the practical value and robustness of the reported improvements.

References

[1] Cai et al., 2025. Disentangling Length Bias in Preference Learning via Response-Conditioned Modeling. https://arxiv.org/abs/2502.00814

[2] Zhang et al., 2025. IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization. ACL 2025. https://aclanthology.org/2025.acl-long.1079/

Questions for authors

Could the authors clarify the key differences between MPL and prior works such as Rc-BT and IOPO, particularly regarding their similar preference learning objectives, and explain why IOPO is not included as a baseline?

What theoretical or empirical evidence supports the assumption that adding constraints necessarily increases or decreases the preference gap between two responses?

Given the modest improvements on ComplexBench and the fivefold expansion of training data, could the authors further investigate the generalization and computational efficiency of MPL?

4 comments

We sincerely thank you for your insightful and constructive comments. In response, we have thoroughly revised the manuscript to enhance its clarity and rigor. Below, we address each of your comments in turn, and revisions made specifically in response to your comments are highlighted in green\color{green}{\text{green}} in the revised manuscript for your convenience. Two sets of new experiments were also requested by Reviewer zvFj and therefore carry that color: the human study (Q3) and the multi-seed results on Qwen3 and ComplexBench (Q4) are highlighted in red\color{red}{\text{red}}.

Q1: The intra-sample objective closely resembles Rc-BT, raising concerns about methodological novelty. What are the key differences?

A1: Thank you for raising this. We agree that the functional form of the intra-sample objective follows Rc-BT, which we credit (Eqn. 3, Section 3.2). The differences lie in what it is applied to and what is built around it:

  • Multiple heterogeneous instructions instead of one length constraint. Rc-BT uses a single word-count constraint. MPL composes several instructions per prompt from 10 verifiable types in four categories and generalizes to 15 held-out types (Inst (OOD)).
  • Partial satisfaction. MPL compares two augmented prompts with opposite parameter settings and defines the preference under partial satisfaction ("satisfies all vs. violates at least one" for ywy_w, "satisfies at least one vs. violates all" for yly_l; Algorithm 1). This has no counterpart with a single constraint.
  • The inter-sample level is new. Rc-BT compares prompts for a fixed response and does not model how the gap between two responses changes across prompts.

Both levels contribute (Table 6). This comparison is added to Section 3.2 (L152-L155).

Q2: The inter-sample objective is mathematically similar to IOPO, yet their relationship is not discussed and a direct comparison is missing. Why is IOPO not included as a baseline?

A2: Thank you for pointing this out. We address each point below.

(1) What is the relationship between the inter-sample objective and IOPO?

The differences between the inter-sample objective and IOPO are:

  • Data. IOPO uses a teacher LLM (Qwen2-72B-Instruct) to rewrite x1x_1 into x2x_2 and to generate a new response y2y_2. MPL keeps the original human-ranked pair and only appends verifiable instructions to the prompt programmatically, with no LLM call.
  • Relation. In IOPO, the preference between y1y_1 and y2y_2 reverses from x1x_1 to x2x_2. MPL keeps ywy_w preferred under every prompt and only orders the size of the gap, anchored at xx, in both directions.
  • Training. IOPO folds all comparisons into one group-level logit. MPL supervises the BT, intra-sample, and inter-sample relations separately, for both reward modeling and DPO.

(2) Why is IOPO not a baseline, and how does MPL compare with it?

IOPO as published is trained on TRACE (120K samples built by a 72B teacher), so comparing with it directly would conflate the objective with the data. In addition to the data mismatch, the full generated preference training set is not available in the released repository, which contains only a single training example. The paper describes the construction pipeline but does not specify the prompt or acceptance criteria for the final validation of preference groups. These gaps prevent an exact reconstruction of the original training data, so our reimplementation may differ from the original IOPO setup. We re-implemented IOPO's grouping and loss on our data, with {(xw,yw),(xl,yl)}\{(x_w, y_w), (x_l, y_l)\} as the matched group and {(xw,yl),(xl,yw)}\{(x_w, y_l), (x_l, y_w)\} as the mismatched group on Qwen2-7B:

IFEval PromptInst (OOD)
Vanilla DPO49.2658.90
IOPO grouping + IOPO loss (re-implemented on our data)53.1560.12
BT + inter-sample level (Ours w/o intra-sample level)54.0460.67
Ours57.5464.18

The IOPO-style objective improves over Vanilla DPO and is comparable to our BT + inter-sample variant, as the shared form predicts. Full MPL is about 4 points higher, so the gain comes from supervising the three levels jointly.

The discussion is added to Section 2 (L091-L100) and the comparison to Section 4.3 (L389-L421).

Q3: Assuming a particular ordering of inter-sample preference gaps requires stronger theoretical or empirical justification.

A3: Thank you for the question. We provide both.

A sufficient condition. Suppose the reward under an augmented prompt x′x' decomposes into the original reward plus a compliance term, r∗(x′,y)=r∗(x,y)+c(x′,y)r^*(x', y) = r^*(x, y) + c(x', y), where cc is higher for a response that satisfies all added instructions than for one that violates at least one. Then Δ(xw)−Δ(x)=c(xw,yw)−c(xw,yl)>0\Delta(x_w) - \Delta(x) = c(x_w, y_w) - c(x_w, y_l) > 0 and Δ(x)−Δ(xl)=c(xl,yl)−c(xl,yw)>0\Delta(x) - \Delta(x_l) = c(x_l, y_l) - c(x_l, y_w) > 0, which are exactly our two orderings. The objective only orders the size of the gap and never uses a reversed label.

Human validation. Three annotators, blind to the construction, judged 50 randomly sampled gap-expanding and 50 gap-reducing relations (majority vote):

NAgreeNo changeOppositeFleiss' κ
Gap-expanding5092%8%0%0.68
Gap-reducing5078%18%4%0.60

Humans almost never contradict the constructed direction (0/50 and 2/50). The remaining cases are "no change" judgments, where the added constraint is minor next to the semantic difference; they act as label noise without a sign error.

The sufficient condition is added to Section 3.3 (L216-L221, in green\color{green}{\text{green}}) and the human study to Section 4.5 and Table 9 (L441-L467, in red\color{red}{\text{red}}).

Q4: Performance gains on stronger models and on naturally occurring complex instructions remain modest, especially on ComplexBench.

A4: Thank you for raising this. We extended every Qwen3 result to 5 seeds (mean ± std):

4B Prompt4B Inst (OOD)8B Prompt8B Inst (OOD)14B Prompt14B Inst (OOD)
Vanilla DPO75.07 ± 0.6581.50 ± 0.5477.00 ± 0.4882.64 ± 0.7079.41 ± 0.3085.28 ± 0.38
M-LIFT76.96 ± 0.4882.98 ± 0.3477.70 ± 0.7283.16 ± 0.6680.00 ± 0.6885.06 ± 0.52
SimPO76.70 ± 0.5882.52 ± 0.7477.93 ± 0.4283.53 ± 0.5078.59 ± 0.6285.03 ± 0.48
MuSC76.56 ± 0.7782.85 ± 0.7378.52 ± 0.5783.99 ± 0.3480.41 ± 0.5185.46 ± 0.50
Ours77.89 ± 0.5683.50 ± 0.4380.63 ± 0.8785.58 ± 0.7381.48 ± 0.5286.13 ± 0.37

Differences between Ours and each baseline, with p-values from a Welch's t-test over the 5 seeds:

Ours minusQwen3-4BQwen3-8BQwen3-14B
Vanilla DPO, Prompt+2.82 (p < 0.001)+3.63 (p < 0.001)+2.07 (p < 0.001)
Vanilla DPO, Inst (OOD)+2.00 (p < 0.001)+2.94 (p < 0.001)+0.85 (p = 0.007)
Best baseline, Prompt+0.93 (M-LIFT, p = 0.023)+2.11 (MuSC, p = 0.003)+1.07 (MuSC, p = 0.011)
Best baseline, Inst (OOD)+0.52 (M-LIFT, p = 0.069)+1.59 (MuSC, p = 0.005)+0.67 (MuSC, p = 0.045)

On ComplexBench (Qwen3-8B), with 5 training seeds and 3 evaluation repeats:

DRFR (5 training seeds)Std over 3 evaluation repeats
Vanilla DPO64.08 ± 0.170.07
M-LIFT63.56 ± 0.300.10
Ours64.74 ± 0.220.09
  • The improvements are statistically significant. Over 5 seeds, MPL is significantly better than Vanilla DPO in all 6 comparisons and than the best baseline in 5 of 6 (p < 0.05).
  • Stronger models leave less headroom. Vanilla DPO fails 50.7% of IFEval prompts on Qwen2-7B but only 21–25% on Qwen3, so the absolute gains are naturally smaller. Relative to this headroom, MPL still removes 10–16% of the remaining failures on Qwen3 (16.3% on Qwen2-7B).
  • On ComplexBench, the gain is small but significant (+0.66; Welch: 95% CI [+0.37, +0.95], p < 0.001; paired bootstrap over instructions: p = 0.017), well above the evaluation noise (std ≤ 0.10).

Section 4.4 (L430-L459) now states the smaller gain on ComplexBench explicitly, and Table 3 (L341-L350) and Table 8 (L432-L438) report the 5-seed results (in red\color{red}{\text{red}}, shared with Reviewer zvFj).

Q5: Further analysis of computational overhead and statistical significance is needed, given the fivefold expansion of training data.

A5: Thank you for the suggestion. Significance is reported in A4. For the overhead, we measured GPU-hours on Qwen2-7B (4×A100-80GB, 5 epochs, batch size 32); accuracies are from Tables 2 and 7, where "@1n" downsamples to the same number of training pairs as Vanilla DPO:

Training pairsGPU-hoursIFEval PromptInst (OOD)
Vanilla DPOn2.649.2658.90
M-LIFT@1nn2.750.3756.55
M-LIFT3n7.856.3059.05
Ours@1nn3.053.4962.20
Ours5n14.357.5464.18
  • At equal data volume, MPL already beats both baselines at nearly the same cost (3.0 vs. 2.6 GPU-hours).
  • The full setting scales roughly linearly: 5.5× the GPU-hours of Vanilla DPO for 5× the pairs, with no teacher-model inference or online sampling.
  • Data construction is negligible: about 2 minutes on a single CPU core.

This is added to Section 4.3 (L399-L426).

Thank you for the detailed response and the new experiments.

The human study, the multi-seed results with significance tests, and the cost analysis address my concerns about the gap-ordering assumption, robustness and overhead. The equal-data result (Ours@1n) is particularly helpful, and I appreciate the added discussion of Rc-BT and IOPO in the revision.

However, my main concerns (W1 and W2) remain. The response confirms that the intra-sample objective follows Rc-BT, and the revised Section 4.3 shows that each inter-sample term has the same form as IOPO's Eqn. 9. In your own comparison, IOPO re-implemented on your data performs on par with BT + inter-sample (53.15 vs. 54.04), so the added discussion clarifies the relationship but does not change my assessment.

In addition, the gain on natural complex instructions is still small (+0.66 DRFR on ComplexBench over Vanilla DPO), while full MPL uses 5× the training pairs.

Reviewer_zvFjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Multi-level Preference Learning (MPL), a framework to improve the multi-instruction following capabilities of LLMs during post-training. MPL augments existing preference data by comparing different prompts for a fixed response and comparing preference gaps between the same responses under different prompts. The authors incorporate these additional training signals into reward modeling and DPO, evaluate PPO using the trained reward models, and report improvements on instruction-following benchmarks.

Critical strengths
  • The paper addresses a valid and important problem, which is improving instruction following on multi-instruction tasks.
  • The paper proposes a method to perform synthetic augmentations to prompts while keeping the answer unchanged, which allows easy scaling of data without additional data curation costs/compute.
  • The authors evaluate MPL through both DPO and PPO using reward models trained with their proposed objective, and report improvements over the evaluated baselines.
  • The experiments cover multiple model families and sizes, including Qwen2, Qwen3, and Llama-3.1.
Critical weaknesses
  • The gains of the method over Vanilla DPO diminish with more recent and stronger models. Qwen3-4B and Qwen3-14B show relatively modest gains over the baselines.
  • The results are stronger on tasks with programmatically verifiable constraints, including IFEval and MathIF. Although the IFEval evaluation includes a separate subset of held-out instruction types, the improvement on natural complex instructions is less convincing. On ComplexBench, MPL improves over vanilla DPO by only 0.36 %.
  • Doing DPO on the second type of preference relations (e.g., the yl−conditionedy_{l-conditioned} intra-sample pairs) might increase the likelihood of the rejected answer under a regular question.
  • The paper programmatically constructs the modified DPO pairs, and these do not guarantee human preference gaps. For example, consider the prompt "What is 5 * 6?" with the preferred response "30" and the rejected response "60." Now append the instruction "End your answer with a period." A program can verify that the rejected response satisfies the added instruction while the preferred response violates it. However, a human may still prefer the correct answer just as strongly, because the punctuation error is negligible compared with the arithmetic error. Constraint satisfaction alone, therefore, may not establish that the preference gap strictly decreases, and the assumption should be human-validated, even on a small scale.
Questions for authors
  • Report how the likelihood of rejected answers on the original preference pairs changes after DPO with the paper’s method, e.g. log-probs of ywy_w and yly_l.
  • Validate how often do human judgments agree with the constructed preference-gap orderings?
  • Are the ComplexBench improvements reliable under repeated training and evaluation, or by chance given the minor improvements?
2 comments

We sincerely thank you for your insightful and constructive comments. In response, we have thoroughly revised the manuscript to enhance its clarity and rigor. Below, we address each of your comments in turn, and all associated revisions are highlighted in red\color{red}{\text{red}} in the revised manuscript for your convenience.

Q1: The gains over Vanilla DPO diminish on more recent and stronger models; Qwen3-4B and Qwen3-14B show only modest gains over the baselines.

A1: Thank you for raising this. We extended every Qwen3 result to 5 seeds (mean ± std):

4B Prompt4B Inst (OOD)8B Prompt8B Inst (OOD)14B Prompt14B Inst (OOD)
Vanilla DPO75.07 ± 0.6581.50 ± 0.5477.00 ± 0.4882.64 ± 0.7079.41 ± 0.3085.28 ± 0.38
M-LIFT76.96 ± 0.4882.98 ± 0.3477.70 ± 0.7283.16 ± 0.6680.00 ± 0.6885.06 ± 0.52
SimPO76.70 ± 0.5882.52 ± 0.7477.93 ± 0.4283.53 ± 0.5078.59 ± 0.6285.03 ± 0.48
MuSC76.56 ± 0.7782.85 ± 0.7378.52 ± 0.5783.99 ± 0.3480.41 ± 0.5185.46 ± 0.50
Ours77.89 ± 0.5683.50 ± 0.4380.63 ± 0.8785.58 ± 0.7381.48 ± 0.5286.13 ± 0.37

Differences between Ours and each baseline, with p-values from a Welch's t-test over the 5 seeds:

Ours minusQwen3-4BQwen3-8BQwen3-14B
Vanilla DPO, Prompt+2.82 (p < 0.001)+3.63 (p < 0.001)+2.07 (p < 0.001)
Vanilla DPO, Inst (OOD)+2.00 (p < 0.001)+2.94 (p < 0.001)+0.85 (p = 0.007)
Best baseline, Prompt+0.93 (M-LIFT, p = 0.023)+2.11 (MuSC, p = 0.003)+1.07 (MuSC, p = 0.011)
Best baseline, Inst (OOD)+0.52 (M-LIFT, p = 0.069)+1.59 (MuSC, p = 0.005)+0.67 (MuSC, p = 0.045)
  • The improvements are statistically significant. Over 5 seeds, MPL is significantly better than Vanilla DPO in all 6 comparisons and than the best baseline in 5 of 6 (p < 0.05).
  • Stronger models leave less headroom. Vanilla DPO fails 50.7% of IFEval prompts on Qwen2-7B but only 21–25% on Qwen3, so the absolute gains are naturally smaller. Relative to this headroom, MPL still removes 10–16% of the remaining failures on Qwen3 (16.3% on Qwen2-7B).

Table 3 (L341-L350) now reports these 5-seed results.

Q2: The improvement on natural complex instructions is less convincing (only +0.36% on ComplexBench). Is it reliable under repeated training and evaluation?

A2: Thank you for the question. On Qwen3-8B, we trained 5 seeds per method and repeated the full evaluation 3 times:

DRFR (5 training seeds)Std over 3 evaluation repeats
Vanilla DPO64.08 ± 0.170.07
M-LIFT63.56 ± 0.300.10
Ours64.74 ± 0.220.09

The gain is statistically significant: +0.66 over Vanilla DPO (Welch's t-test over seeds: 95% CI [+0.37, +0.95], p < 0.001; paired bootstrap over instructions: p = 0.017), well above the evaluation noise (std ≤ 0.10). Section 4.4 (L430-L459) now claims a small but significant gain with no loss of natural-instruction ability, and Table 8 (L432-L438) reports the 5-seed results.

Q3: DPO on the intra-sample pairs might increase the likelihood of the rejected answer under a regular question. How do the log-probs of the rejected answers on the original preference pairs change?

A3: Thank you for the suggestion. The intra-sample pair for yly_l contrasts two augmented prompts and never involves log⁡π(yl∣x)\log\pi(y_l \mid x) directly. Following your suggestion, we measured the log-probabilities on 2,000 training triples with Qwen2-7B (summed over response tokens):

Mean log⁡π(yw∣x)\log\pi(y_w \mid x)Mean log⁡π(yl∣x)\log\pi(y_l \mid x)Samples where log⁡π(yl∣x)\log\pi(y_l \mid x) rose above πref\pi_{\text{ref}}
πref\pi_{\text{ref}}-327.6-306.3—
Vanilla DPO-335.8-322.421.0%
Ours w/o intra-sample level-334.1-323.619.3%
Ours-333.0-323.419.6%
  • The intra-sample level does raise log⁡π(yl∣x)\log\pi(y_l \mid x) slightly. Compared with Ours w/o intra-sample level, the mean rises by 0.2 nats and the share of samples above πref\pi_{\text{ref}} by 0.3 points (6 of 2,000 samples). This small spillover is expected: the intra-sample pair favors yly_l under xw2x^2_w, which is xx with added instructions.
  • The increase is negligible and does not weaken the preference.
    • It is about 1% of the 17.3-nat drop in log⁡π(yl∣x)\log\pi(y_l \mid x) that training produces without this level. Overall, MPL still lowers log⁡π(yl∣x)\log\pi(y_l \mid x) more than Vanilla DPO (17.1 vs. 16.1 nats), and both the mean and the share stay below Vanilla DPO's.
    • The intra-sample level raises log⁡π(yw∣x)\log\pi(y_w \mid x) even more (+1.1 nats), so the margin log⁡π(yw∣x)−log⁡π(yl∣x)\log\pi(y_w \mid x) - \log\pi(y_l \mid x) keeps growing: +11.7 nats under MPL vs. +10.8 without the intra-sample level and +7.9 under Vanilla DPO.
    • Accordingly, MPL matches or exceeds Vanilla DPO in semantic quality (Table 5).

This analysis is added to Section 4.5 and Table 10 (L448-L479).

Q4: The programmatically constructed pairs do not guarantee human preference gaps, and the assumption should be human-validated.

A4: Thank you for the concrete example, which lets us state precisely what MPL assumes and what it does not. We address each point below.

(1) In the "What is 5 * 6?" example, a human may still prefer the correct answer just as strongly, so constraint satisfaction alone may not establish that the gap decreases.

In your example, xx = "What is 5 * 6?", ywy_w = "30", yly_l = "60.", and xlx_l appends "End your answer with a period." The inter-sample objective (Eqn. 6) encodes only Δ(x)>Δ(xl)\Delta(x) > \Delta(x_l): "30" still beats "60." under xlx_l, only by a smaller margin. It never requires "60." to be preferred over "30", and no term in the loss uses a reversed label, unlike preference-flipping augmentation. The constraint is also purely ordinal, so any small decrease satisfies it. We agree that even the direction is uncertain when the added constraint is negligible next to the semantic gap, so we ran the human validation you asked for.

(2) How often do human judgments agree with the constructed preference-gap orderings?

We randomly sampled 50 gap-expanding and 50 gap-reducing relations. Three annotators, blind to the construction and to our hypothesis, judged whether the better response's advantage is larger, the same, or smaller under the augmented prompt (majority vote):

NAgreeNo changeOppositeFleiss' κ
Gap-expanding5092%8%0%0.68
Gap-reducing5078%18%4%0.60

Humans almost never contradict the constructed direction (0/50 and 2/50). Your case does occur, mainly as "no change" among gap-reducing relations. There the objective asks for a positive difference where the human difference is about zero, i.e., label noise without a sign error, consistent with MPL preserving semantic quality (Table 5).

This clarification is added to Section 3.3 (L212-L223), and the human study to Section 4.5 and Table 9 (L441-L467).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.