- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary: This paper augments existing preference data with programmable format constraints, constructing two types of preference relationships: prompt-side (intra-sample) and preference gap-side (inter-sample), and designs corresponding losses. These can be integrated with RM training and DPO to improve the ability to follow multi-constraint instructions.
Critical strengths
Strengths
The motivation is intuitive. Labels are programmatically verifiable, do not rely on stronger LLMs, and are low-cost.
The method is applicable to both RM and DPO, and is simple to implement. It covers 6 backbones, including equal-volume ablation, OOD instruction types, and PPO downstream experiments.
Critical weaknesses
Weaknesses
Limited novelty; the claim of being "overlooked" is too strong.
The Intra-sample objective directly follows Rc-BT.
"Input-side preferences" have been systematically studied by IOPO (Input-Output Preference Optimization, 2024); the paper merely lists them without comparison.
The data construction is of the same origin as LIFT/M-LIFT.
The "preference gap ranking" in the Inter-sample is a new element, but its assumption (adding constraints necessarily widens or narrows the gap) is heuristic and unverified.
The baseline is weak, lacking key baselines.
The OpenAssistant data used by Vanilla DPO is itself unrelated to instruction compliance; any augmentation targeting format constraints will beat it. M-LIFT is the authors' own extension.
Since there are already procedural verification functions for each type of constraint, the most natural baseline is reinforcement learning based on verifiable rewards, such as RLVR for instruction constraints in Tulu 3, or directly using GRPO.
Comparisons with IOPO, Conifer, RPO, etc., are also lacking.
The evaluation and training metrics are highly homospersed.
The IF metric of RM is measured on D^if_eval constructed using the same method as training; an accuracy of 91–98% does not indicate generalization.
The prompt-level metric of IFEval includes 10 constraints seen during training and uses the same validation code.
The improvement in Inst (OOD), which truly reflects generalization, is significantly smaller; for example, Llama-3.1-8B only improves by 1.2.
Gains on new models are small and lack significance testing.
Qwen3-4B's Prompt metric: 77.41 vs. M-LIFT 77.04; Qwen3-14B: 81.30 vs. MuSC 80.19.
IFEval has approximately 541 prompts; one prompt is approximately equal to 0.18%; except for Qwen3-8B, all are single-run tests.
The ComplexBench benchmark, which only tested one model, scored 0.36 higher than Vanilla DPO, insufficient to support the conclusion that it is "transferable to natural instructions."
There are questionable values.
In Table 1, the Vanilla RM for Qwen3-8B achieved a SQ accuracy of 99.35% on OpenAssistant rank0 versus rank1, while Qwen2 and Llama only achieved approximately 61–62%.
OASST's ranking itself is quite noisy; a 99% accuracy rate suggests data contamination or a testing bug. This could affect trust in the entire RM evaluation process.
Questions for authors
See Weaknesses
5 comments
(5) The gap-ordering assumption of the inter-sample level is heuristic and unverified.
The gap ordering is not merely heuristic. It follows from a simple condition on the reward, and human judgments confirm it.
Theoretically, let be the preference gap under prompt , and suppose the reward under an augmented prompt is the original reward plus a compliance term, , where is higher for a response that satisfies all added instructions than for one that violates at least one. Under the gap-expanding prompt , satisfies all added instructions and violates at least one, so . Under the gap-reducing prompt the roles are swapped, so . These are exactly the two orderings MPL uses, and neither requires to be preferred over .
Empirically, we checked whether humans agree with the constructed orderings. We randomly sampled 50 gap-expanding and 50 gap-reducing relations without filtering. Three annotators, blind to the construction and to our hypothesis, saw the original and the augmented prompt with the two responses in random order, and judged whether the better response's advantage becomes larger, stays the same, or becomes smaller under the augmented prompt. We report the majority vote.
| N | Agree | No change | Opposite | Fleiss' κ | |
|---|---|---|---|---|---|
| Gap-expanding | 50 | 92% | 8% | 0% | 0.68 |
| Gap-reducing | 50 | 78% | 18% | 4% | 0.60 |
Humans almost never contradict the constructed direction (0/50 and 2/50). The remaining "no change" cases are label noise without a sign error.
These revisions are in Section 2 (L091-L100), Section 3.2 (L152-L155), Section 3.3 (L212-L223), and Section 4.5 and Table 9 (L441-L467). We will also update the Abstract and Introduction accordingly in the revised paper.
Q2: The baselines are weak, lacking key baselines.
A2: Thank you for the suggestions. We address each point below.
(1) Vanilla DPO on OpenAssistant is unrelated to instruction compliance, and M-LIFT is the authors' own extension.
Vanilla DPO serves as the reference for semantic quality, since methods that add constraints can lose it (e.g., M-LIFT and MuSC lower the Qwen2-7B SQ win rate from 54.86 to 44.51 and 43.9, Table 5). For constraint following, the augmentation-based baselines (M-LIFT, IOPO, RPO, and our BT + inter-sample variant) use the same constraint-augmented prompts and reach 53.15 to 56.30 Prompt on Qwen2-7B, while MPL reaches 57.54. MPL also outperforms M-LIFT at equal data volume (53.49 vs. 50.37, Table 7), and MuSC is an independent published method.
(2) The most natural baseline is RLVR with the verification functions, e.g., Tulu 3-style RLVR or GRPO.
We added GRPO on Qwen2-7B with the same augmented prompts and verification functions (the reward is the fraction of satisfied instructions, with 8 rollouts per prompt on the same hardware).
| IFEval Prompt | Inst (OOD) | SQ Win Rate | GPU-hours | |
|---|---|---|---|---|
| Vanilla DPO | 49.26 | 58.90 | 54.86 | 2.6 |
| RLVR (GRPO) | 60.74 | 62.42 | 49.81 | 46.8 |
| Ours | 57.54 | 64.18 | 58.73 | 14.3 |
| Ours → RLVR (GRPO initialized from Ours) | 62.22 | 65.64 | 56.15 | 61.1 |
- RLVR is strongest on the seen instruction types, since its reward uses the IFEval verifier.
- MPL generalizes better and preserves semantic quality, at under a third of the GPU-hours.
- The two are complementary. Since a key contribution of MPL lies in its data, the same data can also be used in GRPO. Our augmented prompts serve as the RLVR training prompts above, and initializing RLVR from MPL gives the best IFEval results.
(3) Comparisons with IOPO, Conifer, RPO, etc. are lacking.
We added all three on Qwen2-7B. IOPO and RPO are re-implemented on our data (RPO reverses the constraints that violates).
| IFEval Prompt | Inst (OOD) | |
|---|---|---|
| Vanilla DPO | 49.26 | 58.90 |
| IOPO (re-implemented on our data) | 53.15 | 60.12 |
| RPO (re-implemented on our data) | 54.63 | 60.28 |
| Ours | 57.54 | 64.18 |
| Conifer SFT → Vanilla DPO | 54.07 | 62.73 |
- MPL outperforms both re-implementations on the same data by 2.9 to 4.4 points.
These baselines will be added to Section 4.1 and Section 4.3 in the revised paper.
We sincerely thank you for your insightful and constructive comments. In response, we have thoroughly revised the manuscript to enhance its clarity and rigor. Below, we address each of your comments in turn, and the corresponding revisions will be included in the revised paper.
A1: Thank you for the candid assessment. We address each point below.
We are sorry for the lack of clarity. "Overlooked" referred to multi-instruction following, where human annotation, LLM-based augmentation (e.g., Conifer and MuSC), and programmatic augmentation (e.g., LIFT) all compare responses under a fixed prompt and do not use prompt-side preference signals. The exception is IOPO, which relies on teacher-generated data. In contrast, MPL models intra-sample and inter-sample relations jointly, from existing human preference pairs and without a teacher model. We will state this precisely in the Abstract and Introduction of the revised paper.
We are sorry for the lack of clarity. The functional form of the intra-sample objective follows Rc-BT (Eqn. 3), and our contribution at this level is the data construction and its extension to multiple instructions.
The intra-sample level alone raises Prompt from 49.26 to 56.25 on Qwen2-7B (Table 6).
We are sorry for the lack of clarity. We now discuss IOPO in Section 2. IOPO's loss and our inter-sample term share the same algebraic form, but they differ in data and relation.
IOPO's training data has not been released (the repository contains a single example), and its construction is not described in enough detail for an exact reproduction. We therefore re-implemented its grouping and loss on our data under the same settings as MPL-DPO on Qwen2-7B, which may differ from the original in some details.
The IOPO-style objective improves over Vanilla DPO and matches our BT + inter-sample variant, while full MPL is about 4 points higher, showing the benefit of supervising all three levels jointly.
The construction differs from LIFT in three respects.