- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces REVERB, a method for constructing preference data to improve soft-constraint following in language models. It reverses one constraint in an instruction, generates responses under the original and reversed prompts, and retains pairs only when a model can recover the intended constraint difference in both directions. The accepted responses form preference examples whose labels depend on the corresponding prompt. Using a single open-weight teacher model, the authors construct 7,949 training examples and apply DPO to two student models. Experiments on FollowBench and CFBench show consistent improvements over SFT and a reproduced judge-labeled baseline, with additional validation through alternative teacher and evaluator models, small-scale human audits, and general-capability benchmarks.
Critical strengths
-
Well-motivated construction of informative preference pairs. Reversing one soft constraint provides an explicit contrast that constraint removal may fail to produce, while bidirectional recovery filters for whether that contrast is observable in the responses. This offers a practical approach to improving preference-data quality for requirements that lack straightforward programmatic checks.
-
Consistent empirical gains with useful robustness evidence. REVERB outperforms both SFT and the reproduced judge-labeled baseline across all reported constraint-following metrics on two student models. The improvements are particularly substantial for Llama-3.1-8B: FB-soft HSR and CFBench PSR exceed SFT by 6.7 and 9.0 percentage points, respectively. Additional experiments with a different teacher pipeline and a second evaluator strengthen the evidence that the benefits extend beyond a single model configuration.
Critical weaknesses
-
Some work like AIR (arXiv:2504.03612) finds that absolute response quality matters for preference learning, even when preference margins are controlled. REVERB checks whether the intended constraint difference is recognizable, but how does it ensure that the chosen response also answers the underlying task well? Does the human audit assess task correctness and overall quality, or only which response better follows the target constraint?
-
Missing ablations leave the central methodological contribution unclear. REVERB combines constraint reversal, recovery filtering, repeated sampling, and training under both original and reversed prompts, but the experiments primarily evaluate this combination. In particular, a reversal-only baseline is needed to establish what recovery adds beyond generating more contrasting responses. Without this separation, the overall gains are encouraging, but the value of the proposed recovery mechanism remains unclear.
Questions for authors
Clarification on the recovery check: Does successful recovery establish only that the responses differ along the intended constraint, or do you claim it also establishes the correctness of the preference label? How does the method handle pairs where this difference is clear but one response gets the task wrong or violates another requirement?
Evidence needed to assess the recovery mechanism: How does REVERB compare with training on reversed-constraint pairs without recovery filtering, using a comparable training-data size? This comparison would help determine whether recovery is essential to the gains or whether constraint reversal alone provides most of the benefit.
Thank you for the responses; I will keep the positive score.