acceptodds
Under review as a conference paper at ICLR 2027

LIFT: Scoped Constraints for Evaluating and Training Instruction Following

Abstract

Common instruction following (IF) benchmarks, such as IFEval, evaluate the ability of language models (LMs) to follow simple constraints that can easily be verified programmatically. These same constraints, in turn, are used as reward signals for reinforcement learning. This design may lead to reward hacking of constraints, pushing models away from the main task. Recent attempts to address this resulted in benchmarks with richer constraints, but required expert labor, remained closed-access, and cannot be easily extended. We present LIFT-Bench, an IF benchmark that asks a model to draft a document given a topic and a structured outline. Constraints are scoped to certain document sections, providing finer control over constraint density. LIFT-Bench includes both verifiable and judged constraints, as well as a training counterpart, LIFT-Train. Importantly, LIFT-Bench provides both main task and instruction-following constraints, mitigating shallow reward hacking. On experiments with multiple models, we find that frontier models struggle with the harder subsets of our benchmark and analyze model performance along multiple axes. We also find that training on LIFT-Train is much less susceptible to reward hacking than training on the common IFTrain. LIFT-Bench and LIFT-Train are publicly available, as well as the data generating pipeline.

Primary area:
datasets and benchmarks
Keywords:
instruction following, language models, benchmark, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QhzNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose LIFT-Bench, an instruction following eval (with an accompanying training set). It is constructed through pulling seed topics from wikipedia, creating a report sketch, and then annotating it with constraints (both verifiable and LM-judged). Evaluation shows that models struggle on flat-style tasks especially, and further analyses shows that smaller models struggle more with higher numbers of constraints, along with constraint-type analyses. The authors also create a train set equivalent of LIFT-Bench, LIFT-Train, and show it generalizes better to other instruction following benchmarks than IFTrain, prior work.

Critical strengths
  1. The evaluation setup and analysis are interesting and impactful, in particular the document size, constraint type failure analyses are nice to see, and the distinguishing of Kimi K3 and GPT 5.5 in the analysis is interesting.
  2. I think the training results showing OOD performance does not drop is solid, and this work seems pretty clearly to be an improvement over IFTrain, its predecessor, as well as being a reasonably 'new' shape of IF benchmark by focusing more on report writing.
Critical weaknesses
  1. The LM judge quality is poorly validated. There is no details (even in the appendix) on how annotators were recruited, how many there were, or inter-annotator agreement. This is especially important as the judge is much weaker than many of the models tested in the evaluation, which themselves could be 'hacking' the judge. The study in Appendix C only has 78 invalid samples, and the judge accepts 51 of them.
  2. The benchmark is fairly saturated already for frontier models, with GPT 5.5 already almost at 100%. Benchmarks solvable by frontier models are still valid, but it would be more interesting and exciting if this work used its synthetic nature to really push to document sizes or constraint difficulties that frontier models struggle with - or spend more time on analysis of existing smaller models (e.g. focusing on training or on the GPT vs GPT-nano analysis) then proposing this benchmark.
  3. The reward hacking analysis is a little lacking: it would be nice to see more analysis (e.g. human annotations) on if the LM judge (which is deciding most of the score!) is being reward hacked in this scenario, and to what degree it is (cheaper/easier alternatives could be to use a set of model judges, and/or use much larger, newer, stronger models). Given LM judges are known to be biased, do the benchmark scores reflect real human scores even for the GPT 5.5 outputs? Over the course of RL training, does the LM judged quality diverge from human ratings? You could argue that hacking an LM judge is harder, and there is some evidence that this setup keeps the model from collapsing (e.g., from the length analysis), but it's tricky to say that it isn't reward hacking in different ways. It would also be nice to compare the reward hacking to other work, e.g. does training on the MOSAIC set generalize worse/better to OOD evals than LIFT-Train?

Right now, I am slightly negative due to the lack of details, but answering my questions and the concerns above could make me lift my score.

nit: Appendix H seems vacous, it references two tables already in the main text. I assume it would refer to Figure 8 but it doesn't state so in the text, and the floating nature of the figures makes it difficult to determine.

Questions for authors
  1. Is a wikipedia snapshot used for generation, or is wikipedia just directly queried?
  2. How were human annotators gathered, and how did you ensure they provided quality annotations?
  3. What do non-IF evaluations look like when training on LIFT-Train, and how does it perform when trained on in multi-task scenarios? These can be alternative ways to avoid reward hacking, e.g. Olmo 3 finds that mixing domains seems to prevent IF reward hacking to some extent (https://arxiv.org/abs/2512.13961).
3 comments

W1 / Q2. The LM judge is poorly validated; how were annotators gathered and their quality ensured?

Please see "Judge reliability" above.

Annotators. The meta-evaluation set was labelled by in-house annotators, who were given only a broad satisfaction criterion. The re-labeling showed this was too loose: annotators and judges were never given the grading standard. We corrected 56 of 305 labels (18%) and wrote the standard down. We will add these details and the standard to Appendix C.

False accepts. For the adopted judge, Appendix C reports 45 false accepts among the 78 invalid cases (scoped Gemma-4-31B). On the revised labels, out of 24 invalid labels it accepts 3 and scores 0.81 macro-F1.

W2. The benchmark is fairly saturated for frontier models.

We respectfully disagree. When considering the structured splits, once more constraints are added, then even frontier models are not saturated: GPT-5.5 drops from 0.97–0.98 on the few splits to 0.93 (short) and 0.91 (long) on numerous. Density and document size are generation parameters, so harder splits can be generated by raising them, with no new annotation.

In addition, flat splits exhibit much lower performances (0.71 on average for GPT-5.5).

W3. The reward-hacking analysis is lacking.

We agree that judge-based rewards need closer scrutiny.

Is the LM judge being reward hacked? We do not claim the policy is perfect. Our claim is that a judged content reward is much harder to exploit than the purely programmatic checkers common in prior instruction-following RL work, which reward surface compliance regardless of answer quality. Under checker-only rewards (IFTrain), median response length collapses (from 114 words to 28.5 by step 180 and to 18 by step 390); under LIFT-Train it stays (between 389 and 533.5 words). The share of short, high-reward completions reaches 28.5% at step 180 under IFTrain, and stays at 0% under LIFT-Train.

Stronger judges and judge ensembles. We followed this suggestion for the meta-evaluation: with the grading standard stated, frontier and open judges agree with the revised human labels at 0.83–0.88 macro-F1. An ensemble, or a fine-tuned rubric judge as in AdvancedIF, would further reduce exposure to any single judge's biases. We mention this as future work in the conclusion.

We did not collect human ratings of trained-model outputs over the course of RL training, as this would be very expensive. So we cannot say whether judged quality diverges from human ratings during training. We also did not train on MOSAIC, so we cannot compare its out-of-distribution generalization with LIFT-Train's. We will state these as limitations.

Q1. Is a Wikipedia snapshot used for generation, or is Wikipedia queried directly?

Wikipedia is a snapshot, and we only use it for titles. Topics are seeded from a historical snapshot of the Wikipedia Vital Articles list (level 5, about 40,000 topics, out of which we use 4,000). Article text is never used and Wikipedia is not queried during generation. The topic is consumed only as free text, so the list can be swapped for any topic source.

Q3. What do non-IF evaluations look like after training on LIFT-Train, and how does it perform in multi-task training?

We have not run non-IF evaluations or multi-task training. We agree that mixing domains is a complementary way to limit reward hacking, as Olmo 3 reports, alongside strengthening the reward signal as we do here. Mixing would also help preserve non-IF capabilities such as math and coding. We will add this to the discussion as potential future work.

Hi, thanks for the response! The details are interesting and address some of my concerns. Considering the revised grader could change model rankings and potentially some of the analysis, I’d like to see that rerun before acceptance.

As for saturation, I still think 0.93 leaves limited headroom. The flat split is more convincing in having headroom, but then do we need the full pipeline just for the flat tasks? I think my original point stands that it would be interesting to push the structured tasks harder to see if there is more headroom to be had in the benchmark.

Finally, I think the wikipedia setup is still not fully reproducible - it would be good to get a promise of shared code/prompts, or just access dates/revision IDs for the article snapshots.

Overall, I’m keeping my current score, considering the issues around the LM judge changes.

Thank you for the follow-up, and for engaging with the details.

Regarding task saturation, we agree that the structured splits currently leave limited headroom for frontier models. Pushing them harder is possible within the same pipeline, since density and document size are generation parameters, and adding further constraint types is another direction; both need further experiments, which we intend to do, but didn’t have enough time to explore. On whether the full pipeline is needed: the structured tasks are far from solved for smaller models. These models still struggle on the structured splits and improve substantially when trained on LIFT-Train. We expect this to matter more once harder structured tasks open up the headroom, as you suggest: the same pipeline generates both the harder benchmark and its training data.

For rerunning with the revised grader, We agree. We are currently rescoring all models and splits with the revised judge setup, and the final version will report the updated results and model rankings, with any affected analysis revised accordingly.

Regarding Wikipedia reproducibility, our code, dataset and the Wikipedia topics we used are already released; the links are anonymized in the submission for double-blind review and will appear in the final version. The release includes the exact list of topic titles we sampled from. Since only titles are used, and no article text, this list is everything needed to reproduce the seeding step. The source list is also cited in the paper (Wikipedia: Vital Articles, Level 5, 2023; a historical snapshot of about 40,000 unique topics).

Reviewer_5MLuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LIFT, a framework for evaluating and training instruction following through section-scoped constraints. LIFT-Bench constructs document-drafting tasks with structured outlines and combines programmatically verifiable constraints with LLM-judged content requirements. LIFT-Train further uses these constraints as rewards for GRPO training. The training results show substantial improvements on LIFT-Bench and reduced reward-hacking behavior compared with IFTrain.

Critical strengths
  • Fine-grained instruction-following evaluation.

    Scoped constraints enable precise evaluation and controlled analysis of constraint density.

  • Comprehensive evaluation and useful training findings.

    The paper evaluates 17 models and provides a scalable data-generation pipeline. The GRPO experiments demonstrate reduced reward-hacking behavior compared with IFTrain.

Critical weaknesses
  • Insufficient isolation of the scoping contribution.

    Structured and flat tasks differ in document structure, constraint generation, and prompt formulation. Controlling for constraint count alone does not isolate the effect of scoping.

  • Insufficient isolation of the content-reward contribution.

    LIFT-Train and IFTrain differ in task distributions and reward formulations. Without controlled ablations, the reduced reward hacking cannot be confidently attributed to content-aware rewards.

  • Limited judge reliability and external generalization.

    The adopted judge incorrectly accepts 45 of 78 invalid cases, while improvements on AdvancedIF and ComplexBench are only 0.9 and 0.3 percentage points, respectively.

Questions for authors
  1. Could the authors provide a paired comparison that holds topics, constraint sets, and prompt formulations as consistent as possible while varying only the constraint scope? How are conflicting constraints handled?

  2. Could the authors conduct a controlled ablation on identical training data, comparing rewards with and without the content component while keeping the reward formulation comparable?

  3. Appendix C reports 45 false positives among 78 invalid cases. How sensitive are the benchmark scores and model rankings to judge selection and threshold calibration?

  4. Section 3.2 assigns 90% weight to body constraints, but Appendix A shows that these also include lexical and formatting constraints. What proportion of the reward actually comes from semantic content requirements? Can the authors verify that satisfying only verifiable constraints cannot yield a high reward?

  5. Appendix I controls for the number of constraints in a target section, but not necessarily their identities. Could the observed performance degradation reflect changes in local constraint difficulty rather than interference from other sections? A paired experiment fixing the target section's exact constraints would help clarify this.

2 comments

W1 / Q1. Insufficient isolation of the scoping contribution; a paired comparison varying only scope.

We agree that the structured-versus-flat gap compares two task regimes and is not an isolated effect of scoping. We point out the judge meta-evaluation (Appendix C), where we hold the judge model, the responses and the human labels fixed, and change only what the judge sees: the full request and full response (unscoped), or the single criterion with the text it governs (scoped). Here, judging with scoping is worth 11-19 macro-F1 points per judge.

We did not run a paired structured-versus-flat comparison with identical constraint sets, since this is not feasible. A section-scoped constraint set cannot in general be moved to a single global scope as is: sections may carry different language or casing constraints, which contradict each other once they share a scope.

Q1 (second part). How are conflicting constraints handled?

Conflicts are prevented by construction, as explained in the data pipeline appendix. In particular:

  • A scope receives at most one language, one casing and one emoji-count constraint, since two of the same type could contradict each other. This holds for flat tasks as well, where the whole response is one scope.
  • All other constraints are additive. Judged constraints require specific content or an artifact (table, list, code snippet) to be present, and lexical constraints induce constraints on keyword frequency. Therefore there is no expected conflict with such constraints.
  • IFTrain-style constraints use the source's own conflict map. In experiments with IFTrain-style constraints, we use the conflict map from IFTrain to prevent conflicts. This is only done in our ablation and these constraints are not part of the released splits.

W2 / Q2. Insufficient isolation of the content-reward contribution; a controlled ablation on identical training data.

We agree the submission lacked this ablation. We have run one, and it supports the contribution to overall response quality.

We generated three training sets from the same pipeline, with the same topics and outlines, that differ only in which constraints are attached and scored:

  • Verifiable-only: each constrained section gets one programmatically checked constraint (keyword count, casing, emoji count, language). This is the IFTrain-style reward on our tasks.
  • Judged-only: each constrained section gets one LM-judged content constraint.
  • Mixed: each constrained section gets one of each. This is the LIFT-Train design.

We trained the same base model on each and evaluated every trained model on the held-out test set of all three variants. We found that the mixed reward gives a balanced configuration, while each single-family reward produces its own pathology: “Verifiable-only” develops IFTrain-like reward hacking - response length shrank to one sentence satisfying its own reward, while score degrades on judged content constraints; “Judged-only” inflates its responses while neglecting verifiable constraints and producing the longest responses, padded with unnecessary lists, tables and code-blocks; the “Mixed” setting is balanced, it improves on both constraint families with a reasonable response length between the other two.

Since topics, outlines and prompt format are identical across the three, the collapse under the verifiable-only reward is attributable to the reward lacking a content component.

W3. Limited judge reliability and external generalization.

On the judge, see "Judge reliability" in the general comment.

On external generalization, we agree, and we do not claim broad transfer. LIFT-Train and LIFT-Bench come from one generator and share a scoring function, so the large gain on LIFT-Bench is in-distribution adaptation. The gains on out-of-distribution datasets are indeed very small.

However, we emphasize that LIFT-Train is more robust to distribution shift than IFTrain, since training on IFTrain degrades the performance out of distribution, while training on LIFT-Train increases performance (albeit slightly).

We’ll revise the training section and the conclusion, stating that the gains are concentrated on LIFT-Bench and that transfer to external benchmarks is marginal.

Q3. How sensitive are scores and rankings to judge selection and threshold calibration?

See "Judge reliability" above.

Q4. What proportion of the reward comes from semantic content requirements? Can verifiable constraints alone yield a high reward?

The wording in Section 3.2 was imprecise, and we thank the reviewer for catching it. The 90% weight covers all section-level constraints: programmatically verified ones (lexical, case, emoji, language) and LM-judged semantic ones (content, lists, tables, code). Averaged over the dataset, the reward is 10% structure, about ~45% programmatic body constraints, and about ~45% judged semantic constraints.

Every example contains at least one judged constraint. A response that satisfies all verifiable constraints and fails all semantic ones is bounded at about 0.55-0.61 average reward. We will add this breakdown to the paper.

Q5. Could the degradation in Appendix I reflect local constraint difficulty rather than interference from other sections?

Recap: Table 3 holds a section's own constraint count N fixed and asks whether its score drops as the rest of the document gets more constraints. The concern is that denser tiers (few → some → many → numerous) might give those sections harder constraints, which would look like interference.

Answer: We re-ran the section-load analysis with the target section's constraint composition held fixed, and the slopes move by at most 0.5 points,mostly unchanged. The sampling also rules this out by design: every tier draws a section's constraints uniformly from the same fixed pool.

We will add both the re-analysis and this argument to Appendix I.

Reviewer_7biMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LIFT introduces synthetic document-drafting tasks with constraints attached to specific sections or artifacts. Its generation pipeline selects Wikipedia topics, constructs outlines, generates constraint pools, and samples tasks with different document sizes, constraint densities, and local or global scopes. Deterministic checkers evaluate verifiable constraints, while an LLM judge assesses semantic requirements. The resulting weighted score is used both for evaluation and as the reward for GRPO training. The paper presents LIFT-Bench with 784 examples and LIFT-Train with 34,258 examples, evaluates existing models, and compares training on LIFT-Train with IFTrain. It reports substantial gains on LIFT-Bench, small improvements on external instruction-following benchmarks, and fewer observed reward-hacking behaviors than in its IFTrain training runs.

Critical strengths

Scoping constraints to particular document sections gives a useful way to control and study local constraint density, rather than relying only on the total number of requirements in a prompt. The structured and flat task variants make this distinction concrete.

Including task-content requirements alongside formatting constraints addresses an important weakness of rewards that only check surface compliance. The training comparison with IFTrain, including its flat-only control and analysis of generated responses, provides useful evidence about this issue.

Critical weaknesses

The evidence for generalization is limited. LIFT-Train and LIFT-Bench use the same data-generation pipeline, and the benchmark's scoring function is reused unchanged as the training reward. The large gain on LIFT-Bench may therefore mainly reflect adaptation to this task distribution and scoring system. In Table 4, Qwen2.5-7B improves by 39.9 points on LIFT-Bench after 180 training steps, but only by 1.1 on IFEval, 0.8 on IFBench, 0.9 on AdvancedIF, and 0.3 on ComplexBench. The external evaluations are valuable, but these much smaller gains do not establish a substantial improvement in broadly transferable instruction-following ability.

The paper does not sufficiently investigate whether the limited transfer comes from the scoring design or the training-data distribution. The scoring criteria may cover only a narrow part of instruction following, while Wikipedia-based document drafting may offer limited task diversity even when many constraint combinations are generated. These are different possible explanations. Evaluating with additional independent measures would help test the coverage of the reward, while expanding training to other task types under the same scoring approach would help test the role of data diversity. The current experiments do not adequately separate these factors.

Questions for authors

Can you evaluate the trained models using additional independent measures that do not reuse LIFT's training reward, to clarify whether the gains extend beyond its own scoring criteria? Please discuss why the large improvement on LIFT-Bench is accompanied by only small improvements on the external benchmarks in Table 4.

Can you distinguish limitations of the scoring design from limitations of the training data? For example, compare scoring approaches on the same training tasks, and separately broaden the training task types beyond Wikipedia-based document drafting while keeping the scoring approach fixed. Which change improves performance on independent tasks?

2 comments

W1. The evidence for generalization is limited.

We agree, and we do not claim broad transfer. LIFT-Train and LIFT-Bench come from one generator and share a scoring function, so the large gain on LIFT-Bench is in-distribution adaptation. The gains on out-of-distribution datasets are indeed very small.

However, we emphasize that LIFT-Train is more robust to distribution shift than IFTrain, since training on IFTrain degrades the performance out of distribution, while training on LIFT-Train increases performance (albeit slightly).

We'll revise the training section and the conclusion, stating that the gains are concentrated on LIFT-Bench and that transfer to external benchmarks is marginal.

W2. Does the limited transfer come from the scoring design or from the training-data distribution?

We understand the concern, but we think Wikipedia is itself a wide domain, and that document drafting is an important task in its own right.

Our score is composed of many kinds of constraints: programmatically verified ones (keyword counts, casing, emoji, language) and judged ones (required content, lists, tables, code). We therefore think it represents instruction following quite well.

We do agree that LIFT does not cover instruction following in general, so generalization in that broader sense is expected to be limited. We will add a discussion of this to the paper.

Q1. Can you evaluate with independent measures that do not reuse LIFT's training reward, and discuss why the external gains are small?

Independent measures. We agree this is not yet well measured, and what we have is a noisy start. The external benchmarks use their own scoring (Table 4): training on LIFT-Train improves them slightly, while training on IFTrain degrades them. Response length, which the reward does not score, stays stable under LIFT-Train (median 389–533.5 words) but collapses under IFTrain (see R3, W3). The meta-evaluation set in Appendix C, drawn from outputs of fine-tuned training checkpoints, gives a further signal: our reward judge agrees with the revised human labels at 0.81 macro-F1, and judges from other families agree at 0.83–0.88. None of these is a controlled evaluation of the final model, a limitation we state explicitly in the Conclusion.

Q2. Does the limited transfer come from the scoring design or from the training-data distribution?

We think it might be related to either one. LIFT does not cover instruction following in general, so generalization in that broader sense is expected to be limited.

Thank you for the detailed responses and clarifications. I appreciate your efforts to address my concerns. However, the response does not sufficiently resolve the main issues raised in my review, so I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all

We thank the reviewers for their thoughtful and constructive feedback. We are glad they found the paper's contributions valuable across the benchmark design, the evaluation, and the training results.

Scoped constraints. Reviewers found that scoping constraints to document sections "gives a useful way to control and study local constraint density" (R1) and enables "precise evaluation and controlled analysis of constraint density" (R2). R3 sees the benchmark as "a reasonably 'new' shape of IF benchmark" through its focus on report writing.

Evaluation and analysis. R2 highlighted the "comprehensive evaluation" of 17 models and the scalable data-generation pipeline. R3 found the evaluation setup and analysis "interesting and impactful", in particular the document-size and constraint-type failure analyses.

Training findings. R1 noted that including task-content requirements "addresses an important weakness of rewards that only check surface compliance", and that the comparison with IFTrain "provides useful evidence about this issue". R2 noted that the GRPO experiments "demonstrate reduced reward-hacking behavior compared with IFTrain". R3 called the result that OOD performance does not drop "solid", and the work "pretty clearly" an improvement over IFTrain.

The reviewers' main concerns were judge reliability, how far the training gains generalize, and how well the scoping and content-reward contributions are isolated. We address the concern about the judge here since it is shared by two reviewers. We discuss other concerns in the relevant responses to each review.

Judge reliability (R2, R3)

R2 and R3 both questioned the judge's reliability, citing the 45 of 78 invalid cases it accepts in Appendix C. We re-audited both the judge and the meta-evaluation set, and found that most of the gap came from the labels and from an unstated grading standard, not from the judge.

Following the reviewers' suggestion, we re-ran the meta-evaluation with frontier judges (Claude Haiku/Sonnet/Opus 5.5, GPT-6 Luna/Sol) and several open models. On the original labels, our adopted judge (Gemma-4-31B) was better or comparable to the others (0.73 macro-F1 for Gemma versus 0.65-0.72 for the others). Given these results, we re-examined the invalid labels together with the judges' written reasoning. This resulted in correcting 56 of 305 labels (18%).

The disagreements between the judges and the original labels had mostly resulted from the lack of a clear grading standard. We found three main issues that were missing from the instructions given to both the human annotations and the LLM judges: (a) that the response is one extracted part of a longer document to be graded on its own; (b) that "such as" lists are examples rather than requirements, so it is ok to include other items in the lists; and (c)that the judge should not check facts. We then append these standards to the judge’s existing context, and re-ran the meta evaluation. On the revised labels, the Gemma judge scores 0.81 macro-F1 and falsely accepts 3 of the 24 invalid cases. With the grading guidance, frontier and most open judges score 0.83-0.88 macro-F1, within 5 points of each other. Thus, once the labels were corrected and the judging task was stated more clearly, we observe high quality LLM judge results.

We have not rescored the full benchmark under alternative judges and thresholds, so we cannot yet report the effect on model rankings; we state this as a limitation. We will add the revised meta-evaluation, the per-judge results, the threshold analysis and the grading standard to Appendix C.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.