- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a method to overcome the decrease in response quality despite higher constraint satisfaction, which is common when training for reinforcement learning with verified rewards (with instruction following) and evaluated on IFBench as well as a few other benchmarks (e.g. FollowBench). This method augments verifier reward with a calibrated likelihood-based bonus to preserve response quality.
Critical strengths
- This paper raises an important limitation of existing Instruction Following evaluations that only measures constraint fulfillment and not response quality. The problem of overfitting to instruction following benchmark is real (at the cost of response quality) and this paper does a good job of raising awareness for model trainers to not only care about constraint satisfaction.
- The choice of evaluation benchmarks and starting models are diverse and the ablations regarding KL regularization and lowerbound/upperbound are designed carefully.
Critical weaknesses
- An important missing baseline is simply just RL with rubrics. For instance, it can have one rubric that represents response quality (e.g. with an LLM judge) and another that represents constraint satisfaction (with a functional verifier). Then, reward can be based on r_judge + alpha * r_constraint. This requires an LLM judge/reward model but might be optimal for joint optimization, especially if the initial model doesn’t have a good response quality so keeping to the unconditional response quality is not that important.
Questions for authors
- In figure 1, there’s a negative correlation between response quality and constraint satisfaction. Specifically, models that are typically known to be strong models (e.g. Claude Opus 5 and GPT-5.6-Sol) generally have higher quality but lower constraint satisfaction while it’s the opposite with smaller open-weight models (e.g. Qwen 3.8 27B and Muse Glimmer). This seems counter-intuitive since we expect the proprietary models to be better on both response quality and constraint satisfaction. Is this an artifact of IFBench? For instance, if a constraint is to say write a response within 10 words, while some prompts (e.g. explain a topic) requires much more to write in a high quality. More importantly, are the constraint in IFBench useful for a high response quality? Figure 2 and 4 suggests that the constraints in IFBench seem arbitrary (including exactly 5 numbers has nothing to do with the original question). The root cause of this problem is not well-established in the current paper.
- Looking at Table 6, it seems like the inference to draw is that IF-RLVR generally hurts performance on OOD benchmarks (outside of IFBench) maybe with the exception of ComplexBench/Format. If IF-RLVR is from the same paper as IFBench, it could suggest that there are certain patterns in there that is less useful for models to learn. In this case, it would be good to do a brief analysis of IFBench/IFEval prior to solving the problem.
7 comments
Rebuttal - Part II
Q1. Understanding the negative correlation between response quality and constraint satisfaction
We conjecture that the observed negative correlation arises because the constraints are not always useful for improving response quality and can even be difficult to satisfy while producing a high-quality response, as the reviewer pointed out; however, we view this as a limitation of verifiable instruction following (VIF) arising from its emphasis on practical convenience, rather than an issue specific to IFBench.
Ideally, it’s desirable to assign constraints highly relevant to improving response quality; in that case, we would indeed expect frontier models such as Claude Opus 5 to perform well on both dimensions. However, in practice, identifying such constraints for diverse user requests is challenging, and their effects on quality are difficult to assess. These challenges are exactly what motivate VIF, enabling scalable data construction and efficient programmatic verification. Yet these advantages come at a cost: constraints that are easy to construct and verify programmatically are not necessarily those that capture response quality.
Importantly, when evaluation checks only the added constraints, producing text that passes these checks can be easier than producing a complete answer that also satisfies them. This exclusive focus on constraint satisfaction creates what we call an “evaluation blind spot” in this work. Figure 1b is consistent with this explanation: the models shown answer well without the added constraints, but those with higher constraint satisfaction tend to exhibit larger quality drops when the constraints are introduced.
However, this does not make such constraints in VIF invalid. For example, a length limit or required format can be a valid user requirement without making the content more informative. In addition, IFBench’s authors report that each request was manually checked for compatibility with its constraints.
To further examine this pattern, we have added qualitative examples to the revised manuscript (Appendix D.10, Tables 11–15), focusing on prompts for which Grok 4.3 satisfies the constraints but Opus 5 does not. In a task requesting a 2,000-word review, the only added constraint requires a particular word "malabathrum" in the 27th sentence. Grok satisfies this constraint but writes only 677 words, omits all five requested hyperlinks, and refers to sections it never provides. Opus writes 1,994 words and includes the requested links but misses the required keyword position. In another example, Grok meets the keyword-count requirements with repetitions such as “bridge bridge bridge bridge bridge,” while Opus integrates the keywords into an analysis but misses two required counts by one occurrence each. These examples illustrate how passing the checks can coexist with failing to fulfill the original request.
Finally, these observations do not imply that the trade-off between constraint satisfaction and response quality is inevitable. By accounting for response quality through a reference-calibrated likelihood region, our method, CaLIF, improves constraint satisfaction while maintaining response quality near or above pre-RL levels in our experiments. In contrast, our IF-RLVR results show that rewarding only the checked constraints can reinforce incomplete answers.
Q2. Understanding OOD performance degradation through IFBench/IFEval characteristics
While we used IF-RLVR from IFBench, its limited generalization on OOD benchmarks does not necessarily indicate the limited usefulness for learning; in fact, it is more likely to indicate the limitation of constraint-only rewards, considering the effectiveness of CaLIF on the same data.
Specifically, IF-RLVR directly rewards satisfaction of the verifiable constraints without explicitly assessing whether the response fulfills the underlying user request. Optimizing this signal can therefore reinforce constraint compliance at the expense of broader instruction fulfillment. In contrast, CALIF addresses this issue by incorporating response quality into the training signal, using likelihood as a lightweight proxy that requires no external judge.
Our experimental results also support this. Under matched base policies, training data, and training settings, CaLIF improves over the base policies in 40 of 45 rubric-based model–metric comparisons, compared with 14 of 45 for IF-RLVR. These results demonstrate that the IFBench training data can support broader instruction-following gains when the reward also accounts for the underlying request.
Reviewer response
Hi thanks for your response!
W1: To clarify, my question is why not just try where is a tunable hyper-param instead of $r_{judge(x, c, y)} = r_{ver(c, y)} + α 1[r_{ver(c, y)} > 0 ∧ s_{judge(x, c, y)} > τ ] $. Understand time during discussion period is tight so no need to run expts if it can't finish in time but wondering why we want to set the reward function in a complex manner when a simpler one could potentially work.
Q1/2: Thanks for adding the examples. I read tables 11-15 and found the constraints to be not very useful in the real world. For instance, in Table 13, when try to derive some math formula, the constraint is to ask it to "Ensure that the ratio of sentence types (declarative, interrogative, exclamatory) in your response is balanced. Ensure each word in your response has at least one consonant cluster (two or more consonants together).". I honestly don't think regular people ask LLMs to constrain their response in this way. Similarly, I think for Table 11, the instruction is to use a keyword xxx n times but didn't say that it has to be exactly n times or >= n times (i.e. it doesn't state the term "only" n times). In this sense, opus actually got it correct even if it used a word 6 instead of 5 times. Similarly, Table 12 ask it to use a random word "malabathrum" (a word that I've never heard of until today) in a particular sentence position (27). This is not an issue with this work but with the IFBench work that it builds upon, which could limit the usefulness of this work in the real world.
Rebuttal - Part I
We are delighted that the reviewer recognizes the importance of assessing response quality alongside constraint satisfaction and appreciates the diversity of our evaluations and the careful design of our ablations. We also thank the reviewer for the constructive feedback, which has helped us strengthen the paper. We hope the following additional comparisons and discussions can address the reviewer’s remaining concerns.
W1. Comparison with a rubric-based RL baseline
We agree that combining explicit quality feedback with verifier rewards is an important comparison, and that comparison is already presented in the original manuscript. In Section 5.4 (Table 3), we evaluated baselines that combine these two signals, with CaLIF outperforming them on most evaluated metrics.
We further added preference-model reward [1] and intent-check [2] baselines, both of which combine verifier rewards with explicit feedback on response quality, under matched training settings on Tülu 3 8B DPO (Appendix D.4 of the revised manuscript). The table below summarizes IFBench Constraint Satisfaction (CS) and Response Quality (RQ).
Both Preference RM and IntentCheck mitigate the RQ degradation observed under IF-RLVR, but explicit quality feedback does not necessarily lead to more effective joint optimization of CS and RQ. Preference RM achieves its CS gains at a substantial cost to RQ, whereas IntentCheck preserves RQ but yields little improvement in CS. By contrast, CaLIF substantially improves CS while maintaining RQ near the base-policy level. It also outperforms both baselines on most rubric-based metrics; see Table 6 for the full results.
Regarding the initial policy’s quality, we agree that additional quality supervision can be valuable when the policy cannot adequately answer the user’s request. Our concern is that verifier-based RL can degrade answering capabilities already acquired in earlier training stages, as illustrated by the DPO–RLVR comparisons in Figure 3. CaLIF therefore considers the setting where the pre-RL policy can already provide useful answers to the underlying user request, and aims to preserve this ability as it learns to satisfy additional constraints.
[1] Pyatkin et al. (2025). Generalizing Verifiable Instruction Following. NeurIPS 2025.
[2] Guo et al. (2025). IFDecorator: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards. arXiv:2508.04632.