acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

Abstract

Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model–metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
verifiable instruction following, RLVR, reward hacking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cHjvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a method to overcome the decrease in response quality despite higher constraint satisfaction, which is common when training for reinforcement learning with verified rewards (with instruction following) and evaluated on IFBench as well as a few other benchmarks (e.g. FollowBench). This method augments verifier reward with a calibrated likelihood-based bonus to preserve response quality.

Critical strengths
  1. This paper raises an important limitation of existing Instruction Following evaluations that only measures constraint fulfillment and not response quality. The problem of overfitting to instruction following benchmark is real (at the cost of response quality) and this paper does a good job of raising awareness for model trainers to not only care about constraint satisfaction.
  2. The choice of evaluation benchmarks and starting models are diverse and the ablations regarding KL regularization and lowerbound/upperbound are designed carefully.
Critical weaknesses
  1. An important missing baseline is simply just RL with rubrics. For instance, it can have one rubric that represents response quality (e.g. with an LLM judge) and another that represents constraint satisfaction (with a functional verifier). Then, reward can be based on r_judge + alpha * r_constraint. This requires an LLM judge/reward model but might be optimal for joint optimization, especially if the initial model doesn’t have a good response quality so keeping to the unconditional response quality is not that important.
Questions for authors
  1. In figure 1, there’s a negative correlation between response quality and constraint satisfaction. Specifically, models that are typically known to be strong models (e.g. Claude Opus 5 and GPT-5.6-Sol) generally have higher quality but lower constraint satisfaction while it’s the opposite with smaller open-weight models (e.g. Qwen 3.8 27B and Muse Glimmer). This seems counter-intuitive since we expect the proprietary models to be better on both response quality and constraint satisfaction. Is this an artifact of IFBench? For instance, if a constraint is to say write a response within 10 words, while some prompts (e.g. explain a topic) requires much more to write in a high quality. More importantly, are the constraint in IFBench useful for a high response quality? Figure 2 and 4 suggests that the constraints in IFBench seem arbitrary (including exactly 5 numbers has nothing to do with the original question). The root cause of this problem is not well-established in the current paper.
  2. Looking at Table 6, it seems like the inference to draw is that IF-RLVR generally hurts performance on OOD benchmarks (outside of IFBench) maybe with the exception of ComplexBench/Format. If IF-RLVR is from the same paper as IFBench, it could suggest that there are certain patterns in there that is less useful for models to learn. In this case, it would be good to do a brief analysis of IFBench/IFEval prior to solving the problem.
7 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We are delighted that the reviewer recognizes the importance of assessing response quality alongside constraint satisfaction and appreciates the diversity of our evaluations and the careful design of our ablations. We also thank the reviewer for the constructive feedback, which has helped us strengthen the paper. We hope the following additional comparisons and discussions can address the reviewer’s remaining concerns.


W1. Comparison with a rubric-based RL baseline

We agree that combining explicit quality feedback with verifier rewards is an important comparison, and that comparison is already presented in the original manuscript. In Section 5.4 (Table 3), we evaluated baselines that combine these two signals, with CaLIF outperforming them on most evaluated metrics.

We further added preference-model reward [1] and intent-check [2] baselines, both of which combine verifier rewards with explicit feedback on response quality, under matched training settings on Tülu 3 8B DPO (Appendix D.4 of the revised manuscript). The table below summarizes IFBench Constraint Satisfaction (CS) and Response Quality (RQ).

MethodConstraint Satisfaction ↑Response Quality ↑
Base23.785.25
IF-RLVR41.78 (+18.00)2.93 (−2.32)
CaLIF (ours)29.78 (+6.00)5.23 (−0.02)
Preference RM39.00 (+15.22)4.35 (−0.90)
IntentCheck24.78 (+1.00)5.30 (+0.05)

Both Preference RM and IntentCheck mitigate the RQ degradation observed under IF-RLVR, but explicit quality feedback does not necessarily lead to more effective joint optimization of CS and RQ. Preference RM achieves its CS gains at a substantial cost to RQ, whereas IntentCheck preserves RQ but yields little improvement in CS. By contrast, CaLIF substantially improves CS while maintaining RQ near the base-policy level. It also outperforms both baselines on most rubric-based metrics; see Table 6 for the full results.

Regarding the initial policy’s quality, we agree that additional quality supervision can be valuable when the policy cannot adequately answer the user’s request. Our concern is that verifier-based RL can degrade answering capabilities already acquired in earlier training stages, as illustrated by the DPO–RLVR comparisons in Figure 3. CaLIF therefore considers the setting where the pre-RL policy can already provide useful answers to the underlying user request, and aims to preserve this ability as it learns to satisfy additional constraints.

[1] Pyatkin et al. (2025). Generalizing Verifiable Instruction Following. NeurIPS 2025.
[2] Guo et al. (2025). IFDecorator: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards. arXiv:2508.04632.

AuthorsOpenReview ↗

Rebuttal - Part II

Q1. Understanding the negative correlation between response quality and constraint satisfaction

We conjecture that the observed negative correlation arises because the constraints are not always useful for improving response quality and can even be difficult to satisfy while producing a high-quality response, as the reviewer pointed out; however, we view this as a limitation of verifiable instruction following (VIF) arising from its emphasis on practical convenience, rather than an issue specific to IFBench.

Ideally, it’s desirable to assign constraints highly relevant to improving response quality; in that case, we would indeed expect frontier models such as Claude Opus 5 to perform well on both dimensions. However, in practice, identifying such constraints for diverse user requests is challenging, and their effects on quality are difficult to assess. These challenges are exactly what motivate VIF, enabling scalable data construction and efficient programmatic verification. Yet these advantages come at a cost: constraints that are easy to construct and verify programmatically are not necessarily those that capture response quality.

Importantly, when evaluation checks only the added constraints, producing text that passes these checks can be easier than producing a complete answer that also satisfies them. This exclusive focus on constraint satisfaction creates what we call an “evaluation blind spot” in this work. Figure 1b is consistent with this explanation: the models shown answer well without the added constraints, but those with higher constraint satisfaction tend to exhibit larger quality drops when the constraints are introduced.

However, this does not make such constraints in VIF invalid. For example, a length limit or required format can be a valid user requirement without making the content more informative. In addition, IFBench’s authors report that each request was manually checked for compatibility with its constraints.

To further examine this pattern, we have added qualitative examples to the revised manuscript (Appendix D.10, Tables 11–15), focusing on prompts for which Grok 4.3 satisfies the constraints but Opus 5 does not. In a task requesting a 2,000-word review, the only added constraint requires a particular word "malabathrum" in the 27th sentence. Grok satisfies this constraint but writes only 677 words, omits all five requested hyperlinks, and refers to sections it never provides. Opus writes 1,994 words and includes the requested links but misses the required keyword position. In another example, Grok meets the keyword-count requirements with repetitions such as “bridge bridge bridge bridge bridge,” while Opus integrates the keywords into an analysis but misses two required counts by one occurrence each. These examples illustrate how passing the checks can coexist with failing to fulfill the original request.

Finally, these observations do not imply that the trade-off between constraint satisfaction and response quality is inevitable. By accounting for response quality through a reference-calibrated likelihood region, our method, CaLIF, improves constraint satisfaction while maintaining response quality near or above pre-RL levels in our experiments. In contrast, our IF-RLVR results show that rewarding only the checked constraints can reinforce incomplete answers.


Q2. Understanding OOD performance degradation through IFBench/IFEval characteristics

While we used IF-RLVR from IFBench, its limited generalization on OOD benchmarks does not necessarily indicate the limited usefulness for learning; in fact, it is more likely to indicate the limitation of constraint-only rewards, considering the effectiveness of CaLIF on the same data.

Specifically, IF-RLVR directly rewards satisfaction of the verifiable constraints without explicitly assessing whether the response fulfills the underlying user request. Optimizing this signal can therefore reinforce constraint compliance at the expense of broader instruction fulfillment. In contrast, CALIF addresses this issue by incorporating response quality into the training signal, using likelihood as a lightweight proxy that requires no external judge.

Our experimental results also support this. Under matched base policies, training data, and training settings, CaLIF improves over the base policies in 40 of 45 rubric-based model–metric comparisons, compared with 14 of 45 for IF-RLVR. These results demonstrate that the IFBench training data can support broader instruction-following gains when the reward also accounts for the underlying request.

Reviewer_cHjvOpenReview ↗

Reviewer response

Hi thanks for your response!

W1: To clarify, my question is why not just try rjudge(x,c,y)=rver(c,y)+αsjudge(x,c,y)r_{judge(x, c, y)} = r_{ver(c, y)} + \alpha s_{judge(x, c, y)} where α\alpha is a tunable hyper-param instead of $r_{judge(x, c, y)} = r_{ver(c, y)} + α 1[r_{ver(c, y)} > 0 ∧ s_{judge(x, c, y)} > τ ] $. Understand time during discussion period is tight so no need to run expts if it can't finish in time but wondering why we want to set the reward function in a complex manner when a simpler one could potentially work.

Q1/2: Thanks for adding the examples. I read tables 11-15 and found the constraints to be not very useful in the real world. For instance, in Table 13, when try to derive some math formula, the constraint is to ask it to "Ensure that the ratio of sentence types (declarative, interrogative, exclamatory) in your response is balanced. Ensure each word in your response has at least one consonant cluster (two or more consonants together).". I honestly don't think regular people ask LLMs to constrain their response in this way. Similarly, I think for Table 11, the instruction is to use a keyword xxx n times but didn't say that it has to be exactly n times or >= n times (i.e. it doesn't state the term "only" n times). In this sense, opus actually got it correct even if it used a word 6 instead of 5 times. Similarly, Table 12 ask it to use a random word "malabathrum" (a word that I've never heard of until today) in a particular sentence position (27). This is not an issue with this work but with the IFBench work that it builds upon, which could limit the usefulness of this work in the real world.

Reviewer_mRtBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines a failure of verifiable instruction following: a response can pass all the output checks while giving a poor answer to the user's actual question. The authors show examples of this behavior in current models and after verifier-reward training. They then propose CALIF, which uses likelihoods from a frozen reference model to decide when to add a bonus to the verifier reward. Experiments are reported on IFBench and three rubric-based instruction-following

Critical strengths
  1. The problem is well motivated. The example in Figure 2, where the constrained response is simply “No, 0 1 2 3 4”, captures the weakness of constraint-only scoring much better than a correlation plot would. I can see this failure happening in practice when a model is trained against format checkers.
  2. The ablations around the likelihood bonus are among the best parts of the paper. Directly rewarding likelihood drives the Qwen3-1.7B model to 1,677 tokens per response, on average, while its response-quality score falls to 1.91. Removing either boundary also changes the outcome considerably. These experiments explain the design of CALIF and show why the seemingly simpler reward fails.
  3. There is some practical appeal to reusing the frozen reference policy instead of calling another LLM judge during every training rollout. The results on rubric-based benchmarks are also useful because performance on IFBench alone would leave the main motivation unresolved.
Critical weaknesses
  1. I am less convinced by the response-quality evaluation than by the verifier results. Figure 1 uses three judges for the frontier-model comparison, but most of the subsequent training analysis relies on GPT-5.6 Sol. An answer that says almost nothing is easy to recognize; the more interesting cases are responses that partially answer a difficult request while satisfying its formal constraints. Human ratings on a sample of those cases would help establish how much quality CALIF actually preserves. The additional pairwise judge experiment is useful, although it still depends on model judgments.
  2. The cost in constraint satisfaction is sizable. Averaged across the five configurations, CALIF reaches 34.64 on IFBench, whereas ordinary IF-RLVR reaches 44.93. The practical value of this tradeoff depends on how much constraint compliance users are willing to give up for better answers. Figure 8 compares KL tuning on Qwen3-1.7B. I would be interested in seeing a similar quality-matched comparison on one of the larger configurations, where the balance might be different.
  3. I also have a question about the upper likelihood cutoff. CALIF gives zero reward above that boundary, even to a response that passes the verifier. How often do generated responses actually fall into this case? The paper studies different ways of estimating the boundaries, yet the frequency of this reward assignment would make the mechanism easier to understand.
Questions for authors
  1. Could the authors provide a small human evaluation of verifier-passing responses, focusing on borderline cases of answer completeness rather than obvious failures?
  2. For a larger model, what does the constraint-satisfaction versus response-quality curve look like when CALIF and a KL-tuned IF-RLVR baseline are compared at similar quality?
  3. What proportion of training rollouts receive zero reward because they exceed the upper likelihood boundary?
5 comments

We are delighted that the reviewer finds the problem well motivated and appreciates the likelihood-reward ablations and the practical appeal of reusing the reference policy. We also thank the reviewer for the constructive feedback, which has helped us strengthen the paper. We hope the following additional experiments and clarifications can address the reviewer’s remaining concerns.


W1 & Q1. Human evaluation

We agree that human judgments would provide an important check on our LLM-based evaluation of response quality. Following prior work [1], we conducted a pairwise human evaluation to reduce variability arising from differences in how raters use numerical scales.

We randomly sampled 100 response pairs where both CaLIF and IF-RLVR responses passed the verifier, excluding pairs with clearly uninformative or irrelevant responses. Each pair was evaluated by one of five non-author human raters, with each rater comparing 20 pairs and selecting the better response.

Human judgments showed 79% agreement with the GPT-5.6 Sol judge. Human raters also preferred CaLIF over IF-RLVR in 80% of comparisons, compared with 75% for the LLM judge. These results further support our LLM-based evaluation and CaLIF’s response-quality advantage. We have included these results in Appendix D.6 of the revised manuscript.

[1] Li et al. (2019). ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons. arXiv:1909.03087.


W2 & Q2. KL-Tuned Comparison on a Larger Model

To examine whether CaLIF’s advantage persists at similar response quality on a larger model, we extended the KL analysis to Qwen3-4B. We have added the resulting trade-off curves in Figure 11, Appendix D.3 of the revised manuscript.

The table below compares Constraint Satisfaction (CS) and Response Quality (RQ) across KL coefficients.

MethodConstraint Satisfaction ↑Response Quality ↑
Base27.674.60
CaLIF (ours)35.11 (+7.44)4.69 (+0.09)
IF-RLVR (β = 0.001)41.33 (+13.66)2.91 (−1.69)
IF-RLVR (β = 0.02)38.89 (+11.22)3.88 (−0.72)
IF-RLVR (β = 0.05)29.89 (+2.22)4.43 (−0.17)

As β increases, IF-RLVR recovers RQ but sacrifices much of its CS gain. At β = 0.05, its RQ approaches the base-policy level, yet its CS gain is only +2.22. CaLIF achieves a larger CS gain of +7.44 while also attaining higher RQ, exceeding this baseline by 5.22 CS and 0.26 RQ.

This comparison extends the evidence beyond the original 1.7B setting, showing that CaLIF’s more favorable trade-off persists on a larger model when stronger KL regularization brings IF-RLVR closer to base-policy quality.


W3 & Q3. Upper Likelihood Cutoff Frequency

To quantify how often the upper likelihood cutoff assigns zero reward to verifier-passing responses, we analyzed training rollouts on Qwen3-1.7B. The likelihood-region distributions are shown in Figure 12 of Appendix D.5 in the revised manuscript.

Under CaLIF, approximately 5% of verifier-passing rollouts exceed the upper likelihood boundary and receive zero reward, with this proportion remaining roughly stable throughout training. In contrast, under the single-bound variant (w/o upper bound; Section 6.2), the proportion exceeding the same boundary increases to approximately 17%. These results illustrate the role of the upper likelihood cutoff in limiting likelihood inflation.

Reviewer_mRtBOpenReview ↗

Additional Questions of Submission1255 by Reviewer mRtB

Q1. For your newly‑added human evaluation: you filter out “clearly uninformative or irrelevant responses” before pairwise comparison. But the core motivating failure case of this paper is borderline partially‑correct answers that formally pass verifier checks yet provide limited real information. By pre‑filtering these borderline samples, are you systematically removing exactly the cases your method is supposed to fix? Could you report human results on this unfiltered full sample set, or at minimum give statistics on how many borderline samples were discarded in this filtering step?

Q2. In your Qwen3‑4B KL‑tuned IF‑RLVR comparison, CaLIF yields CS=35.11, while IF‑RLVR β=0.02 yields CS=38.89 with only a modest RQ drop (3.88 vs CaLIF’s 4.69). The gap in constraint‑satisfaction is still ~3.78 points. Given many real‑world production systems prioritize constraint compliance, can you quantify how much downstream task utility is lost from this persistent CS deficit? Simply reporting RQ and CS numbers does not tell us whether the CS drop is practically tolerable for real‑world deployments.

Q3. You report that ~5% of verifier‑passing rollouts hit the upper likelihood cutoff and get zero reward. However, this statistic is only measured on Qwen3‑1.7B. Larger models have different likelihood distributions over generated text. Could you share the same cutoff‑hit fraction for your Qwen3‑4B training runs? Also, what ablation results show that 5% zero‑reward rollouts do not hurt final convergence or stability of RL training?

Q4. CALIF relies entirely on a frozen reference model’s likelihood distribution to calibrate bonus rewards. What happens when there is a substantial distribution shift between training prompts and test‑time real‑world prompts? The likelihood boundaries learned on training data may no longer be valid at test time. Have you evaluated out‑of‑distribution prompt sets to test robustness of your likelihood‑boundary mechanism?

AuthorsOpenReview ↗

Follow-Up Responses to the Reviewer

We thank the reviewer for the continued engagement with our rebuttal and the thoughtful follow-up questions. We provide additional experimental results and clarifications below to address the remaining concerns.


Q1. Human evaluation

We are not systematically removing the borderline cases raised in your question. Our phrase “clearly uninformative or irrelevant responses” referred to the obvious failures described in your original review. Following your suggestion to focus on borderline cases of answer completeness, an author manually screened the pairs and excluded 29 pairs containing obvious failures.

We subsequently conducted a pairwise human evaluation on all 29 pairs excluded for obvious failures. Human judgments showed 100% agreement with the GPT-5.6 Sol judge on these pairs. Combined with the original 100-pair evaluation, agreement across the 129 evaluated pairs was 83.7%.


Q2. Practical implications of the constraint-satisfaction gap

Below, we address (1) the CS–RQ scales and (2) production-relevant constraint compliance.

(1) Magnitude of the CS–RQ trade-off

To clarify the relative magnitude of the reported gaps, we remark that CS uses a 0–100 scale, whereas RQ uses a 1–10 scale. The 3.78-point CS gap and 0.81-point RQ gap therefore correspond to 3.78% and 9.0% of their respective scale ranges. Namely, RQ drop is not modest, but more significant compared to CS drop.

(2) Structured-output compliance

To examine whether the IFBench CS deficit extends to output requirements encountered in production workflows, we report results on IFStruct [1], which programmatically verifies JSON/YAML validity and schema compliance. The table reports the percentage of responses passing all required checks.

MethodQwen3-1.7BQwen3-1.7B (Thinking)Qwen3-4BQwen3-4B (Thinking)Tülu 3 8B DPO
CaLIF49.0359.0271.4576.6857.07
IF-RLVR (β = 0.02)48.1052.6072.8575.3054.40

IF-RLVR achieves higher IFBench CS in all five configurations, but higher IFStruct compliance in only one. These results indicate that its IFBench CS gains at the expense of RQ do not consistently translate into better structured-output compliance.

[1] Liquid AI (2026). IFStruct: Measuring structured-output compliance. https://www.liquid.ai/blog/ifstruct-v1.0


Q3. Upper likelihood cutoff: model scale and training stability

Following your suggestion, we measured the same cutoff-hit fraction on Qwen3-4B and found that approximately 5% of verifier-passing rollouts receive zero reward from the upper likelihood cutoff, consistent with Qwen3-1.7B.

Regarding convergence and stability, Figure 12 compares CaLIF with the variant without the upper cutoff on Qwen3-1.7B. Under CaLIF, the cutoff-hit fraction remains around 5% throughout training, while the fraction within the desired likelihood region increases from roughly 50% to 70% and then remains stable. Without the upper cutoff, the fraction exceeding the same boundary rises to approximately 17%, indicating increasing likelihood inflation.

The corresponding ablation in Section 6.2 (Table 4) shows that CaLIF achieves a final Response Quality of 4.01, compared with 3.51 without the upper cutoff. These results suggest that the upper cutoff helps stabilize training by limiting likelihood inflation.


Q4. Generalization to out-of-distribution prompts

CaLIF’s likelihood boundaries are computed separately for each training request using the frozen reference policy and are used only to calibrate rewards during RL training. At test time, the trained policy generates responses without reference-model scoring or likelihood-boundary checks.

In addition, regarding the evaluation with out-of-distribution prompts, Section 5.2 (Table 1) of the original manuscript already reports those OOD evaluations of the resulting policies on AdvancedIF [1], FollowBench [2], and ComplexBench [3]. Across five model configurations, CaLIF outperforms IF-RLVR in 41 of 45 model–metric comparisons and improves over the base policies in 40 of 45 comparisons. These results provide empirical evidence that the instruction-following gains from training with CaLIF’s likelihood-boundary mechanism extend to OOD prompts.

[1] He et al. (2026). AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following. ACL 2026.
[2] Jiang et al. (2024). FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models. ACL 2024.
[3] Wen et al. (2024). Benchmarking Complex Instruction-Following with Multiple Constraints Composition. NeurIPS 2024.

Reviewer_DtzEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reward hacking in verifiable instruction following. Programmatic verifiers can determine whether a response satisfies explicit output constraints, but they do not determine whether the response meaningfully answers the underlying request. The paper first documents this evaluation blind spot in frontier models and shows a corresponding training failure: reinforcement learning with verifier rewards (IF-RLVR) can increase constraint satisfaction while substantially reducing response quality.

The proposed method, Calibrated Likelihood for Instruction Following (CALIF), supplements the verifier reward with a request-conditioned likelihood signal from the frozen pre-RL reference policy. For each training prompt, the reference policy generates one response from the request with its constraints and another from the request alone. Both responses are scored under the request alone, defining a prompt-specific likelihood interval. During RL, a verifier-passing rollout receives a fixed bonus when its reference likelihood lies inside this interval; the verifier reward is retained without the bonus below the interval and suppressed above it. The lower boundary is intended to discourage degradation relative to the pre-RL constrained policy, while the upper boundary prevents direct likelihood maximization from favoring repetitive, unusually predictable responses. CALIF reuses the reference policy already maintained for KL regularization and therefore requires no separate quality judge or learned reward model.

The method is evaluated on five configurations spanning Qwen3-1.7B, Qwen3-4B, and Tulu 3 8B, including thinking and non-thinking modes. On IFBench, standard IF-RLVR raises average constraint satisfaction from 26.98 to 44.93 but reduces response quality from 4.51 to 2.96. CALIF raises constraint satisfaction to 34.64 while preserving response quality at 4.59. CALIF also improves over the corresponding base policies in 40 of 45 model-metric comparisons on AdvancedIF, FollowBench, and ComplexBench. Comparisons with LLM-judge rewards and ablations of the likelihood reward, both interval boundaries, bound estimation, and KL regularization provide additional validation.

Critical strengths
  1. The paper identifies and experimentally isolates a failure that affects both model evaluation and model optimization. The contribution goes beyond observing that a benchmark is incomplete. The paper shows that the same omission can become an optimization target: IF-RLVR learns to improve verifier scores by sacrificing substantive answers, including on examples for which verification changes from failure to success. This is consequential to my assessment because it challenges how verifier-based instruction-following scores should be interpreted and how the corresponding rewards should be used during post-training.

  2. CALIF is a practical two-sided correction whose design is supported by targeted ablations. The method does not simply maximize request-only likelihood; the paper demonstrates that doing so creates a second shortcut through repetitive, highly predictable outputs. Its lower and upper likelihood boundaries address distinct failure modes, and the ablations show that removing either boundary degrades response quality. Reusing the existing reference policy also makes the method substantially cheaper than maintaining a separate judge model. This combination of an operationally simple method, mechanistic ablations, five model configurations, and transfer to three rubric-based benchmarks is consequential because it suggests that the improvement is not limited to one model or the exact training verifiers.

Critical weaknesses
  1. The current endpoint comparisons do not fully establish that CALIF improves the achievable trade-off between constraint satisfaction and response quality. CALIF preserves quality but obtains lower average IFBench constraint satisfaction than final-checkpoint IF-RLVR. Comparing these two final points cannot determine whether CALIF provides a better reward signal or merely prevents the policy from optimizing the verifier as aggressively. Standard IF-RLVR might reach a similar compliance-quality point through early stopping, a different number of RL updates, or different regularization. Figure 8 provides encouraging evidence through a KL-coefficient sweep for Qwen3-1.7B, but training trajectories and matched operating-point comparisons are not reported across the other configurations. This is consequential because CALIF's central algorithmic value depends on whether it achieves a better Pareto frontier than carefully tuned standard training.

  2. The practical and empirical advantage over the closest existing mitigation methods is not yet fully established. Prior work on IF-RLVR already supplements verifier rewards with preference-model supervision, and IFDecorator uses an intent check to prevent verifier-compliant responses that fail the underlying request. CALIF offers an appealing efficiency advantage because it avoids another model, but the paper compares against self-judge and stronger-judge reward variants only on two configurations and does not directly compare with these closest published methods under matched models, data, compute, and evaluation. This is consequential because the diagnosis itself substantially overlaps prior work; the acceptance case therefore rests mainly on whether CALIF provides a better performance-cost trade-off than existing corrections.

Questions for authors
  1. Can the authors provide constraint-satisfaction versus response-quality curves over training for CALIF and IF-RLVR across all five model configurations? At matched constraint satisfaction, does CALIF consistently provide higher response quality than early-stopped or KL-tuned IF-RLVR? Conversely, at matched response quality, does it consistently provide higher constraint satisfaction? This would determine whether CALIF improves the Pareto frontier rather than selecting a less aggressively optimized operating point.

  2. How does CALIF compare with the preference-reward approach accompanying IF-RLVR and with IFDecorator under matched models, training data, and compute? If a direct comparison is infeasible, please clarify the precise methodological and computational differences and provide enough cost information to substantiate CALIF's claimed practical advantage.

  3. Do the reported means and standard deviations represent independent RL training runs or repeated stochastic evaluations of a single trained checkpoint? If they are evaluation repeats, how stable are the principal IFBench and transfer results across independent training seeds?

5 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We are delighted that the reviewer recognizes the importance of the evaluation blind spot and finds CaLIF’s two-sided correction practical and supported by targeted ablations. We also thank the reviewer for the constructive feedback, which has helped us strengthen the paper. We hope the following additional experiments and clarifications can address the reviewer’s remaining concerns.


W1 & Q1. Training Dynamics and Quality Preservation

To test whether CaLIF’s advantage can be recovered through early stopping or stronger KL regularization, we evaluated CaLIF and IF-RLVR after each epoch across all five model configurations and added stronger-KL comparisons (β = 0.02; Figure 10). We further compared IF-RLVR across β ∈ {0.001, 0.02, 0.05} on Qwen3-1.7B and Qwen3-4B (Figure 11). Both analyses are included in Appendix D.3 of the revised manuscript.

Note: Due to the higher computational cost of thinking-mode training, the additional β = 0.02 runs for Qwen3-1.7B and Qwen3-4B in thinking mode are still ongoing. Figure 10 includes the available intermediate-epoch results, and we will add the remaining results during the rebuttal period.

Across the five configurations, IF-RLVR gains Constraint Satisfaction (CS) while losing Response Quality (RQ), whereas CaLIF improves CS while largely preserving base-policy RQ. The following Qwen3-4B results illustrate the comparison with both alternatives.

MethodEpochConstraint Satisfaction ↑Response Quality ↑
Base–27.674.60
CaLIF (ours)435.11 (+7.44)4.69 (+0.09)
IF-RLVR (β = 0.001)136.89 (+9.22)3.72 (−0.88)
IF-RLVR (β = 0.02)234.22 (+6.55)4.30 (−0.30)
IF-RLVR (β = 0.05)429.89 (+2.22)4.43 (−0.17)

Early-stopped IF-RLVR reaches comparable CS but already falls 0.88 RQ points below the base policy. Combining early stopping with stronger KL (β = 0.02) reduces this loss, yet CaLIF achieves both higher CS and higher RQ. With still stronger KL (β = 0.05), IF-RLVR approaches base-policy quality, but CaLIF exceeds it by 5.22 CS and 0.26 RQ.

These comparisons show that early stopping and stronger KL regularization do not recover CaLIF’s combination of constraint satisfaction and response quality, supporting a more favorable trade-off beyond the original final-checkpoint results.

AuthorsOpenReview ↗

Rebuttal - Part II

W2 & Q2. Comparison with existing mitigation methods and computational efficiency

We agree that direct comparisons with existing mitigation methods are important for assessing CaLIF’s practical advantage. We therefore compared CaLIF with the preference-model reward baseline [1] and the intent-check baseline [2], using the same Tülu 3 8B DPO base model, training data, and number of RL training steps (Appendix D.4 of the revised manuscript).

The table below summarizes IFBench performance and reward-computation costs.

MethodConstraint Satisfaction ↑Response Quality ↑Additional model params ↓Reward computation (s/step) ↓
Base23.785.25––
IF-RLVR41.78 (+18.00)2.93 (−2.32)0–
CaLIF (ours)29.78 (+6.00)5.23 (−0.02)053.87
Preference RM39.00 (+15.22)4.35 (−0.90)8B54.74
IntentCheck24.78 (+1.00)5.30 (+0.05)32B1,020.81

Both Preference RM and IntentCheck mitigate the RQ degradation observed under IF-RLVR. However, Preference RM still achieves its CS gains at a substantial cost to RQ, whereas IntentCheck preserves RQ but improves CS by only 1.00 point over the base policy. CaLIF improves CS by 6.00 points while maintaining RQ near the base-policy level. It also outperforms both mitigation baselines on most rubric-based metrics, showing benefits beyond the training verifiers; see Table 6 for the full results.

Moreover, CaLIF achieves these results without an additional scoring model, whereas Preference RM and IntentCheck require separate 8B and 32B models, respectively. CaLIF also computes rewards approximately 19× faster than IntentCheck (Table 7).

[1] Pyatkin et al. (2025). Generalizing Verifiable Instruction Following. NeurIPS 2025.
[2] Guo et al. (2025). IFDecorator: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards. arXiv:2508.04632.


Q3. Independent training runs and seed robustness

The reported means and standard deviations reflect three stochastic evaluations of a single trained checkpoint. To assess variability across training seeds, we conducted three additional independent RL training runs of CaLIF on Qwen3-1.7B, each evaluated three times.

Training RunConstraint Satisfaction ↑Response Quality ↑
Seed 1 (Original)28.78 ± 0.194.01 ± 0.02
Seed 229.00 ± 0.333.96 ± 0.03
Seed 328.67 ± 1.154.00 ± 0.06
Seed 428.89 ± 0.513.97 ± 0.04
Across-seed28.84 ± 0.143.99 ± 0.02

Each run reports the mean and standard deviation across evaluations; the final row reports these statistics across the four run-level means.

All four runs improve CS over the base policy (22.22) while maintaining RQ near its baseline (3.98). These results show that the quality-preserving gains are consistent across independent training seeds. We have added these results to Appendix D.7 of the revised manuscript.

AuthorsOpenReview ↗

Update on Additional Experiments and Gentle Reminder

Dear Reviewer DtzE,

The additional β = 0.02 runs for Qwen3-1.7B and Qwen3-4B in thinking mode are now complete. We have updated Figure 10 in Appendix D.3 of the revised manuscript with the full training trajectories.

We would appreciate your feedback on whether our responses and the completed experiments address your concerns. Please let us know if any points require further clarification.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.