acceptodds
Under review as a conference paper at ICLR 2027

Learning Soft Constraints from Reversed and Recoverable Preference Pairs

Abstract

Soft constraints shape how a response should be written, asking for a writing style, a situation to respect, or a scope of content. Unlike hard constraints, they cannot be verified by code, so existing pipelines build preference data for them by asking an LLM judge which response is better. Whether a response follows a soft constraint is a subjective call, so the judge's verdict carries its own biases, and a soft constraint does not always change the response enough to give the judge anything to compare. We present REVERB, whose principle is to plant, between the two responses of a pair, a difference that is easy to tell apart and to learn from. Reversing one soft constraint into its contrary gives two mirrored prompts, and each response is labeled as preferred under the prompt that produced it. Whether the two responses really separate along the reversed constraint is then verified: an LLM sees only the two responses and must recover the reversed constraint, a test with a known answer, and pairs that fail it are discarded. A single open-weight model runs the whole pipeline and produces 7,949 verified pairs. Direct preference optimization (DPO) on this data improves constraint following on two benchmarks by up to 16.4 points on two student models, outperforming every compared baseline, beating the judge-labeled pipeline by up to 12.0 points with fewer pairs, and preserving general abilities across five benchmarks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
instruction following, soft constraints, synthetic data generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VVbNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces REVERB, a method for constructing preference data to improve soft-constraint following in language models. It reverses one constraint in an instruction, generates responses under the original and reversed prompts, and retains pairs only when a model can recover the intended constraint difference in both directions. The accepted responses form preference examples whose labels depend on the corresponding prompt. Using a single open-weight teacher model, the authors construct 7,949 training examples and apply DPO to two student models. Experiments on FollowBench and CFBench show consistent improvements over SFT and a reproduced judge-labeled baseline, with additional validation through alternative teacher and evaluator models, small-scale human audits, and general-capability benchmarks.

Critical strengths
  1. Well-motivated construction of informative preference pairs. Reversing one soft constraint provides an explicit contrast that constraint removal may fail to produce, while bidirectional recovery filters for whether that contrast is observable in the responses. This offers a practical approach to improving preference-data quality for requirements that lack straightforward programmatic checks.

  2. Consistent empirical gains with useful robustness evidence. REVERB outperforms both SFT and the reproduced judge-labeled baseline across all reported constraint-following metrics on two student models. The improvements are particularly substantial for Llama-3.1-8B: FB-soft HSR and CFBench PSR exceed SFT by 6.7 and 9.0 percentage points, respectively. Additional experiments with a different teacher pipeline and a second evaluator strengthen the evidence that the benefits extend beyond a single model configuration.

Critical weaknesses
  1. Some work like AIR (arXiv:2504.03612) finds that absolute response quality matters for preference learning, even when preference margins are controlled. REVERB checks whether the intended constraint difference is recognizable, but how does it ensure that the chosen response also answers the underlying task well? Does the human audit assess task correctness and overall quality, or only which response better follows the target constraint?

  2. Missing ablations leave the central methodological contribution unclear. REVERB combines constraint reversal, recovery filtering, repeated sampling, and training under both original and reversed prompts, but the experiments primarily evaluate this combination. In particular, a reversal-only baseline is needed to establish what recovery adds beyond generating more contrasting responses. Without this separation, the overall gains are encouraging, but the value of the proposed recovery mechanism remains unclear.

Questions for authors

Clarification on the recovery check: Does successful recovery establish only that the responses differ along the intended constraint, or do you claim it also establishes the correctness of the preference label? How does the method handle pairs where this difference is clear but one response gets the task wrong or violates another requirement?

Evidence needed to assess the recovery mechanism: How does REVERB compare with training on reversed-constraint pairs without recovery filtering, using a comparable training-data size? This comparison would help determine whether recovery is essential to the gains or whether constraint reversal alone provides most of the benefit.

Reviewer_jCVcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles preference learning for soft constraints on content, style, and situation, which are difficult to check with fixed rules. REVERB creates preference pairs by flipping one constraint in a prompt to its opposite (for example, formal to casual), generating a response under each version, and treating each response as preferred under its corresponding prompt. It then applies a recovery check, keeping a pair only if the same LLM can identify the flipped constraint from the two responses in both directions. Using Qwen3.5-397B-A17B throughout this process, the authors build 7,949 DPO training examples.

The experiments on Qwen3.5-9B and Llama-3.1-8B-Instruct show that REVERB outperforms the base models, SFT on the chosen responses, and a judge-labeled pipeline (SbSM) built with the same prompts and generator. One notable result is on Llama, where REVERB reaches 81.8 FB-soft HSR, compared with 69.8 for SbSM (Tab. 1).

Overall, I find the idea intuitive and promising. Instead of relying on a judge to directly compare responses under a subjective constraint, REVERB constructs a controlled contrast and checks whether that difference is reflected in the responses. The improvements over SbSM are encouraging and suggest that this approach can produce useful preference data without direct pairwise judging. My main reservation is that the entire pipeline, including the recovery check, relies on the same LLM, leaving some uncertainty about the independence of the quality check and how well the approach generalizes. Still, I think the paper offers an interesting and practical direction for preference learning, with empirical results that make it worth exploring further.

Critical strengths

Two strengths I particularly appreciate in this paper are:

1. A simple and reusable idea. By introducing a known contrast into the prompts, REVERB avoids relying on a judge's subjective preference for labeling and turns filtering into a check with a known answer. The pipeline is also relatively cheap (one open-weight model, about 5.3 generations per flip) and straightforward for others to adopt.

2. A fair and convincing main comparison. SbSM is rebuilt using the same prompts and generator, yet REVERB achieves better results with fewer training pairs (7,949 vs. 8,520). The improvement is particularly clear on the weaker student model, with a 12.0-point gain in FB-soft HSR on Llama. REVERB also performs best across all metrics when evaluated by a second judge (Tab. 4), which further supports the effectiveness of the approach.

Critical weaknesses

Two weaknesses I would like the authors to address are:

1. The key design choice is not tested independently. The paper describes the recovery check as "the core design of REVERB" (Sec. 3.3), but does not include an ablation without it. The comparison with SbSM also changes several components at once: reversal instead of removal, labels by construction instead of a judge, the recovery and length filters, and the mirrored export. This makes it difficult to tell which components actually drive the improvements. On the stronger student model, SFT on the same filtered responses already comes close to REVERB on most metrics (within 2.2 points on six of seven), suggesting that the quality of the filtered data may be more important than the contrastive training itself.

2. Limited baselines and no variance estimates. All results are reported as single numbers, without multiple seeds or confidence intervals. On Qwen3.5-9B, many improvements are only one or two points on a benchmark of 285 prompts, making it difficult to judge their reliability. SbSM is also the only earlier soft-constraint method included in the comparison. In particular, RPO (Huang et al., 2025) similarly rewrites constraints into their opposites, but is not evaluated against REVERB. Another concern is that the training constraint types are drawn from FollowBench, which is also the main evaluation benchmark, making FB-soft relatively close to the training distribution.

Overall, I lean toward acceptance because the idea is simple and the main comparison is reasonably fair. However, I would particularly like to see the ablations in Q1 and the variance estimates in Q2, as these would help me decide whether to maintain my current score.

Questions for authors
  1. Can the authors provide ablations to isolate the key design choices? Specifically, (i) removing the recovery filter and using either all valid flips or a random subset matched in size to the accepted set; (ii) using only one side of each mirrored pair; and (iii) replacing reversal with removal while keeping labels by construction and the same filtering process?

  2. Can the authors report results across at least three training seeds, or provide bootstrap confidence intervals over evaluation prompts, for Tab. 1? This would be particularly helpful for the Qwen3.5-9B results, where several improvements are relatively small.

  3. How does REVERB compare with RPO under the same prompts and evaluation setup? If the main difference is RPO's reliance on a judge, could the authors also test a REVERB variant that replaces the recovery check with a judge verdict? This would help clarify where REVERB's improvements come from.

Reviewer_oAfVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Soft constraints (e.g., style, scope, situation) are hard to evaluate: Whether a response follows a soft constraints may subject to the judge's biases, and a soft constraint does not always change the response enough to tell apart.

This paper introduces REVERB, a method to construct preference data pairs by planting a known difference with reversed soft constraints. Starting from a instruction II with several constraints, one constrain (c) will be reversed to get Iˉ\bar{I}, and responses will be sampled for both versions: RR and Rˉ\bar R. And this process gives two preference pairs: (I, R > Rˉ\bar R) and (Iˉ\bar{I}, Rˉ\bar R > R). Pair quality is then gated by a "recovery" (back-translation) check with a known answer: an LLM shown only the two responses must describe and match the planted constraint in both directions; failing pairs are resampled or discarded, and a length-comparability filter removes pairs separable by length alone. The full pipeline is run by a single open-weight model (Qwen3.5-397B) and yields 7,949 DPO pairs.

DPO on this data improves FollowBench and CFBench for both Qwen3.5-9B and Llama-3.1-8B, beating the base models, an SFT baseline trained on the same chosen responses, and SbSM. Additionally, the trained model largely keeps general capabilities (GSM8K, MMLU, HumanEval, AlpacaEval, MT-Bench), beating the base model in 6 of 10 settings.

Critical strengths
  • The idea of REVERB with reversed preference pair generation and quality checking with back-translation is nice and clean towards the soft constraint challenges.
  • The quality control of the data construction are carefully described together with human validation.
  • The results show REVERB outperforms all baselines evaluated and that it generally keep the general capability of the trained models compared to the base model.
Critical weaknesses
  • The baselines evaluated are a bit narrow (only SFT and SbSM). For example, RPO [1] also uses reverses constraints, and Crab [2] also uses constraints back translation. These methods share similar ideas to REVERB, but are not compared against.
  • The margins over SFT and SbSM are small on some metrics (Table 1). And lacking of significance tests make it hard to assess the real benefits of REVERB.

[1] Reverse Preference Optimization for Complex Instruction Following. arxiv 2505.22172
[2] Constraint Back-translation Improves Complex Instruction Following of Large Language Models. arxiv 2410.24175

Questions for authors
  • Could you compare with RPO, Crab or similar methods or explain why they might not be suitable/why REVERB is better while they share similar ideas?
  • A significance tests comparing REVERB against baselines maybe helpful to understand the strengths of the proposed method.
1 comment

Thank you for the responses; I will keep the positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.