acceptodds
Under review as a conference paper at ICLR 2027

Precision Bounds Diversity: Reward Engineering for Instruction Following via RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a dominant paradigm for improving instruction following (IF), with recent work assuming that scaling constraint diversity is essential for generalization. We challenge this assumption and show that it is fundamentally misguided. By decoupling training data into hard-only, soft-only, and mixed subsets, we find that models trained exclusively on code-verifiable hard constraints outperform those trained on LLM-judged soft constraints on average, including on soft-constraint benchmarks, and match mixed training despite using fewer constraint types. Diagnosis reveals that LLM-based reward models suffer from critically low recall and a systematic inability to penalize violations, with reliability degrading sharply as constraint count increases. Controlled noise-injection experiments further confirm that reward precision, not constraint diversity, is the primary driver of RLVR performance. Motivated by these insights, we propose H1S, a data-composition strategy that retains all hard constraints while pruning soft constraints to at most one per instruction, confining LLM evaluation to its most reliable single-constraint regime. On five IF benchmarks, H1S achieves average gains of 11.3% (7B) and 6.7% (32B) over base models, surpassing mixed-constraint training by 5.5% at 7B while reducing per-step training latency by 58% and preserving general capabilities.

Primary area:
reinforcement learning
Keywords:
LLM, Instruction Following, RL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3H2QOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares hard-only, soft-only, and mixed-constraint RLVR training, arguing that reward reliability matters more than constraint diversity. Human-annotated verifier audits and noise-injection experiments motivate H1S, which retains hard constraints while limiting soft constraints to one per instruction. Experiments report improved instruction following on five benchmarks, largely preserved general capabilities, and lower training latency.

Critical strengths
  1. H1S is simple and practically useful. Table 3 reports average scores of 62.81 and 68.65 for the 7B and 32B models, exceeding the corresponding mixed-training results. The measured reduction from 19.63 to 8.23 minutes per step at 32B is also substantial.

  2. The human-annotated verifier audit and comparisons across model families usefully highlight reward reliability as a confounder in instruction-following research. The additional judge-consistency analysis strengthens the evidence that the reported soft-constraint gains are not specific to one evaluation judge.

Critical weaknesses
  1. The central causal claim is not established. Section 4.2 varies training-data volume and per-instruction constraint count when studying “diversity,” so these factors are not isolated. Its noise calibration is also inconsistent with the stated intervention: overriding rewards to 1 with probability pp gives negative-response recall rnoisy=(1−p)rcoder_{\mathrm{noisy}}=(1-p)r_{\mathrm{code}}. Matching recall therefore requires p=1−rjudge/rcodep=1-r_{\mathrm{judge}}/r_{\mathrm{code}}, rather than the recall difference used in the paper. Matching recall alone also does not match reward precision or systematic judge errors. These experiments demonstrate sensitivity to reward corruption, but do not establish the claimed primacy of precision over diversity.

  2. The robustness analysis contains consequential reporting errors. In Table 8, the five VerlIF-Pro Hard-only scores average to 61.38, not the reported 64.98, reducing its advantage over the reported Mix average from 4.04 to approximately 0.44 points. Section F.1.1 also claims a significant Hard-only lead over Mix on every evaluated benchmark, although the VerInstruct-Pro* rows show Mix ahead on CFBench and FollowBench. A significant difference cannot establish superiority in the opposite direction. These inconsistencies weaken confidence in the quantitative support for the main conclusion.

Overall, the conclusions of this paper (Precision Bounds Diversity) seem limited to the specific setting and are difficult to generalize. It is widely acknowledged that data diversity is crucial.

Questions for authors

After correcting the noise calibration, does the central conclusion hold when training-set size and per-instruction constraint count are controlled and constraint-type diversity is varied separately?

5 comments

We thank the reviewer for the rigorous review. We address the two weaknesses below.

W1-1: Confounded factors in the Section 4.2 diversity experiment

We agree this weakness. To address this, we fix the training-data volume and vary only the number of constraints retained per example. Starting from the Mix training set, we keep 0, 1/4, 2/4, 3/4, or 4/4 of each example's constraints, and train all five variants under the identical recipe.

Constraints retained(a) IFEval(b) CFBench
00.7340.44
1/40.8230.53
2/40.8050.53
3/40.8020.52
4/40.7840.51

Test performance does not increase monotonically with the number of constraints. The 1/4 variant is the strongest on IFEval (0.823) and ties the 2/4 variant on CFBench (0.53); the full-constraint (4/4) variant is no better than these low-constraint variants. Because the number of training examples is held fixed across all five runs, the comparison isolates constraint count from data scale.

This is consistent with the original conclusion of Section 4.2: once a minimal level of constraint diversity is reached, adding more constraints yields no consistent gain.

W1-2: Noise calibration

We agree that the previous noise level was miscalibrated and thank the reviewer for the correction. Under the stated forced-positive intervention, the negative-response recall of the corrupted reward is r(p) = (1 − p)·r_code. Matching the judge's negative-response recall of 30.6% therefore requires

  • Soft-only: p = 1 − 30.6/81.2 = 62.3%;
  • Mix (77.7% soft-only, 22.3% hard-only): p = 0.777 × 62.3% = 48.4%.

We reran the matched conditions at these levels and updated Figure 4 (https://ibb.co/XrHmHCDY). We observe:

  1. On IFEval, Hard-only + noise reaches 0.775 / 0.769, versus 0.783 / 0.778 for Mix / Soft-only — gaps of +0.8 / +0.9 points. The reference models are thus slightly better, and Hard-only + noise no longer matches them exactly.
  2. On CFBench, the gaps are 7.8 points (Mix) and 4.0 points (Soft-only), essentially unchanged (previously 7.0 and 4.0).

Relative to the original calibration, the gaps change only marginally: 0.5–0.8 → 0.8–0.9 points on IFEval, and 4–7 → 4–7.8 points on CFBench. The qualitative conclusion is preserved: noise closes about 70% of the in-distribution gap (2.5–3.0 points at p = 0 → 0.8–0.9 points at the matched level) and erases the Hard-only OOD advantage on CFBench. This is not change the original conclusion.

W2: Reporting errors in Table 8 and Appendix F.1.1

We thank the reviewer for catching this error and apologize for the oversight. The five per-benchmark VerIF-Pro Hard-only scores in Table 8 average 61.38, not 64.98; We will correct the Table 8 average accordingly.

We will also revise the claim in Appendix F.1.1 to: "Hard-only training outperforms Soft-only training on hard-constraint test sets and performs comparably on soft-constraint test sets; overall, it performs comparably to Mix training." The revised claim is consistent with Table 1 (lines 178–181): "Hard-only training outperforms Soft-only on average and matches the Mix variants, including on soft IF benchmarks."

The corrected result supports the two conclusions of the main paper: (i) training with higher-precision rewards generalizes better than training with lower-precision rewards; and (ii) a high-precision reward with limited diversity, e.g., Hard-only, can match the generalization of larger and more diverse data, e.g., Mix training.

Narrowing this claim does not weaken the main paper's conclusions.

AuthorsOpenReview ↗

Follow-Up on Responses

Dear Reviewer,

Thank you again for your review.

With the discussion period closing soon, we would like to respectfully follow up and inquire if you have the opportunity to review our responses to your comments.

We sincerely look forward to your feedback.

Thank you for your reply. I will maintain my score (Weak rejection) and stand by my view: Overall, the conclusions of this paper (Precision Bounds Diversity) seem limited to the specific setting and are difficult to generalize. It is widely acknowledged that data diversity is crucial.

Reviewer_n3o9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reward design for instruction following with reinforcement learning using verifiable rewards. It distinguishes hard constraints evaluated by code from soft constraints evaluated by model-based judges. Through training ablations and verifier audits, the paper argues that unreliable soft-constraint rewards can limit the benefits of broader constraint coverage. It proposes H1S, a reward construction strategy that retains all hard constraints and at most one uniformly selected soft constraint. Experiments with 7B and 32B models report improved instruction-following performance and reduced training-step latency relative to the evaluated alternatives.

Critical strengths
  1. H1S is a simple and actionable reward construction strategy. Across the two evaluated model scales, it improves over the within-paper ablation alternatives in nine of ten benchmark-scale combinations. For the 32B model, the reported training-step time decreases from 19.63 minutes under Mix to 8.23 minutes under H1S, corresponding to a reduction of approximately 58 percent.

  2. The paper examines both policy performance and verifier behavior. The evaluation includes five instruction-following benchmarks, human-annotated verifier audits, multiple judge models, and additional analyses of the number of retained soft constraints. These components provide useful evidence about the practical limitations of model-based rewards.

Critical weaknesses
  1. The boundary between prior findings and the paper's new contributions is insufficiently established.

TinyJudge already reports hard-only, soft-only, and mixed-training diagnostics across several model families, together with verifier-precision comparisons [1]. Several numerical results appear to coincide with those reported in the present paper. For example, the 7B Hard configuration has the same IFEval score of 80.78 and FollowBench score of 68.96. Several verifier-precision values also coincide. Although TinyJudge is cited in the verifier-audit discussion, the paper does not clearly identify which results are newly obtained, reproduced, or adapted.

VerIF already studies hybrid code-based and model-based verification and discusses the difficulty and cost of evaluating multiple soft constraints [2]. Qwen-IF already evaluates constraint-level verification and averaged constraint rewards [3]. Consequently, the contribution should be positioned more precisely around the specific H1S selection rule and the evidence for its incremental benefit. IF-RLVR also demonstrates benefits from diversity under code-verifiable rewards [4], so its findings do not establish that expanding soft constraints or using unreliable model judges is necessary.

  1. The causal interpretation and quantitative reporting need correction or clarification.

For the stated intervention that replaces a reward by one with probability pp, negative-response recall satisfies Rp=R0−pR0R_p = R_0 - pR_0. Here, RR denotes the probability of correctly detecting a noncompliant response. Matching a recall of 0.812 to 0.306 therefore requires p=1−0.306/0.812p = 1 - 0.306/0.812, approximately 62.3 percent. The reported difference-based value of 50.6 percent instead produces recall of approximately 0.401. Matching recall alone also does not match precision. This limits the interpretation of the noise experiment as a controlled reproduction of the weaker judge.

The diversity experiments change data scale and constraint count without fully isolating semantic diversity. H1S also changes the number of evaluated constraints and the sparsity of the strict all-pass reward. Its gains could therefore reflect changes in optimization difficulty as well as verifier reliability. A limited comparison using pointwise judging of all soft constraints and a clarified reward-aggregation control would help distinguish these explanations.

There are also material reporting inconsistencies. Under an unweighted mean, the five VerlIF-Pro Hard scores in Table 8 average approximately 61.38, while the reported average is 64.98. Some statements that Hard consistently outperforms Mix conflict with the listed benchmark values. I also cannot reconstruct several Table 9 significance values from the stated five-run means and standard deviations using an independent run-level t-test. The aggregation rule, statistical unit, test specification, and per-run results are needed to resolve these discrepancies.

References

[1] TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles. ACL 2026. [2] VerIF: Verification Engineering for RL in Instruction Following. EMNLP 2025. [3] Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following. ACL 2026. [4] Generalizing Verifiable Instruction Following. NeurIPS 2025.

Questions for authors
  1. Which results in Tables 1 and 2 are new experiments, reproductions, or adaptations of previously published results? Please clarify their relationship to the corresponding training ablations and verifier-precision results in TinyJudge, including any changes to data, annotations, or evaluation procedures.

  2. Are constraints excluded by H1S also removed from the prompt, or only from reward evaluation? Is the selected soft constraint fixed for each example or resampled during training? Please provide existing evidence or a limited controlled comparison that separates the effects of pointwise judging and reward aggregation from the effect of retaining only one soft constraint.

  3. Please clarify the noise-matching calculation. Under the stated forced-positive intervention, matching negative-response recall requires p=1−0.306/0.812p = 1 - 0.306/0.812, approximately 62.3 percent. Would the conclusions remain under the corrected intervention? Please also clarify what the diversity experiment establishes when verifier quality and constraint count are controlled, particularly in relation to the code-verifiable diversity results of IF-RLVR.

  4. Please explain the Table 8 averaging rule, reconcile the textual claims with the listed Hard and Mix scores, and specify the statistical unit and test used in Table 9. Per-run scores would help verify these results. Please also clarify the mapping between the datasets and verifier configurations used in the main experiments and the appendix.

4 comments
AuthorsOpenReview ↗

Response to Weakness #1

Thank you for your valuable feedback. We will address your confusion one by one.

W1 (Boundary with prior work).

We agree the boundary is insufficiently explicit and will make it precise. The following works are cited as baselines or for their conclusions; below we clarify our new contributions relative to them.

  • TinyJudge [1]. We cite [1] to establish the Hard-only/Soft-only/Mix diagnostic and the verifier-precision comparison. New in this submission: Building on it, we conduct a deeper investigation via controlled precision–diversity analysis (recalibrated noise injection and fixed-volume constraint-count control), uncovering deeper causes of reward precision degradation. We further propose the H1S selection rule and the operating map, provide cross-domain, multi-judge, and efficiency evidence, and introduce a cheaper, more effective method to mitigate low-precision rewards.

  • VerIF [2]. [2] builds hybrid code/model verifiers and evaluates multi-soft verification costs. Our findings motivate reducing soft constraints rather than building a better verifier; H1S requires no new verifier.

  • Qwen-IF [3]. [3] derives dense, averaged constraint rewards. We study which constraints enter the reward; aggregation is orthogonal, and H1S is composable with constraint-level verification.

  • IF-RLVR [4]. [4] establishes the benefit of diversity under code-verifiable rewards. Our claim is scoped to soft-constraint diversity under noisy LLM judges: the k-ablation fixes all hard constraints and varies only the soft budget—a regime [4] does not cover. We do not claim diversity is unimportant in general, only that its benefit is bounded by reward precision.

AuthorsOpenReview ↗

Response to Weakness #2

Thank you for your valuable feedback. We will address your confusion one by one in weakness #2.

W2.1: Noise calibration

We agree that the previous noise level was miscalibrated and thank the reviewer for the correction. Under the stated forced-positive intervention, the negative-response recall of the corrupted reward is r(p) = (1 − p)·r_code. Matching the judge's negative-response recall of 30.6% therefore requires

  • Soft-only: p = 1 − 30.6/81.2 = 62.3%;
  • Mix (77.7% soft-only, 22.3% hard-only): p = 0.777 × 62.3% = 48.4%.

We reran the matched conditions at these levels and updated Figure 4 (https://ibb.co/XrHmHCDY). We observe:

  1. On IFEval, Hard-only + noise reaches 0.775 / 0.769, versus 0.783 / 0.778 for Mix / Soft-only — gaps of +0.8 / +0.9 points. The reference models are thus slightly better, and Hard-only + noise no longer matches them exactly.
  2. On CFBench, the gaps are 7.8 points (Mix) and 4.0 points (Soft-only), essentially unchanged (previously 7.0 and 4.0).

Relative to the original calibration, the gaps change only marginally: 0.5–0.8 → 0.8–0.9 points on IFEval, and 4–7 → 4–7.8 points on CFBench. The qualitative conclusion is preserved: noise closes about 70% of the in-distribution gap (2.5–3.0 points at p = 0 → 0.8–0.9 points at the matched level) and erases the Hard-only OOD advantage on CFBench. This is not change the original conclusion.

W2.2: Confounded factors in the Section 4.2 diversity experiment

We agree this weakness. To address this, we fix the training-data volume and vary only the number of constraints retained per example. Starting from the Mix training set, we keep 0, 1/4, 2/4, 3/4, or 4/4 of each example's constraints, and train all five variants under the identical recipe.

Constraints retained(a) IFEval(b) CFBench
00.7340.44
1/40.8230.53
2/40.8050.53
3/40.8020.52
4/40.7840.51

Test performance does not increase monotonically with the number of constraints. The 1/4 variant is the strongest on IFEval (0.823) and ties the 2/4 variant on CFBench (0.53); the full-constraint (4/4) variant is no better than these low-constraint variants. Because the number of training examples is held fixed across all five runs, the comparison isolates constraint count from data scale.

This is consistent with the original conclusion of Section 4.2: once a minimal level of constraint diversity is reached, adding more constraints yields no consistent gain.

W2.3: Reporting errors in Table 8 and Appendix F.1.1

We thank the reviewer for catching this error and apologize for the oversight. The five per-benchmark VerIF-Pro Hard-only scores in Table 8 average 61.38, not 64.98; We will correct the Table 8 average accordingly.

We also revise the claim in Appendix F.1.1 to: "Hard-only training outperforms Soft-only training on hard-constraint test sets and performs comparably on soft-constraint test sets; overall, it performs comparably to Mix training." The revised claim is consistent with Table 1 (lines 178–181): "Hard-only training outperforms Soft-only on average and matches the Mix variants, including on soft IF benchmarks."

The corrected result supports the two conclusions of the main paper: (i) training with higher-precision rewards generalizes better than training with lower-precision rewards; and (ii) a high-precision reward with limited diversity, e.g., Hard-only, can match the generalization of larger and more diverse data, e.g., Mix training.

Narrowing this claim does not weaken the main paper's conclusions.

AuthorsOpenReview ↗

Response to Question 1-4

Questions

Q1 (Which results in Tables 1–2 are new, reproduced, or adapted?).

We did cite some of its results, and we went further based on it [1]. See Weakness 1 for response.

Q2 (Prompt vs. reward, selection rule, and the source of H1S's gain).

(1) H1S rewrites each prompt so that it contains only the retained constraints. The removed soft constraints are neither shown to the policy nor scored by the reward; pruning is not reward-only.

(2) The soft constraint is sampled once during dataset construction and then fixed for each example for the entire training run.

(3) Component control. To separate pointwise judging and reward aggregation from constraint retention, we add the following ablation:

Table Q2:

ArmSoft constraintsJudgingAggregationIFEvalCFBench
A: Mix (baseline)allbatchstrict all-pass78.3751.2
B: Full + pointwise + all-passallpointwisestrict all-pass82.4553.6
C: Full + pointwise + averagedallpointwisemean pass78.6151.94
D: H1Sonepointwisestrict all-pass85.2555.1

Protocol. All arms use Qwen2.5-7B-Instruct, the same RL training set and instance count, the same training judge (Qwen3-32B), and the same number of optimization steps. Pointwise denotes one judge call per soft constraint; batch denotes all soft constraints in a single call. Averaged denotes R = 1[all hard constraints pass] × mean(soft-constraint verdicts).

Observation. Pointwise judging alone improves IFEval from 78.37 to 82.45 and CFBench from 51.2 to 53.6, and replacing strict all-pass with an averaged soft reward adds nothing (78.61 / 51.94). Retaining a single soft constraint adds a further +2.8 / +1.5 over the strongest full-constraint arm.

Conclusion. H1S's gain is not explained by reward aggregation; it comes from limiting the LLM judge to one soft constraint per instruction, i.e., to the regime in which its verdicts are most reliable.

Q3 (Corrected noise matching and the scope of the diversity experiment).

See Weakness 2 for response.

Q4 (Table 8/9 reporting and dataset–verifier mapping).

See W2.3 for the corrected Table 8 average (61.38, not 64.98) and the revised claim in Appendix F.1.1 (see weakness 2.3 for details);

We response the remaining points here:

(1) Averaging rule. the Average column is the unweighted mean of the five benchmark means. The statistical unit throughout is the independent training run (n = 5), not the benchmark instance.

(2) Statistical unit and test used in Table 9. The statistical unit is the independent training run (n = 5), not the benchmark instance, and Table 8 reports their mean and standard deviation. Table 9 compares Hard-only with Soft-only and with Mix using a two-sided Welch's t-test (independent samples, unequal variances) on these five run-level scores; we report their exact p-value.

(3) Dataset–verifier mapping. We add following Table, which maps the datasets to the verifier configurations used in the main experiments and the appendix:

DatasetSizeHard / softHard verifierSoft judgeUsed in & purpose
VerInstruct22k22.3% / 77.7%LLM re-implementation code verifierQwen3-32BMain experiments, e.g., Table 1
VerInstruct-Pro*9.8k balanced (hard = soft)50% / 50%LLM re-implementationQwen3-32BAppendix; ablates the effect of data size (see Table 8)
VerIF20k40% / 60%Original codeQwen3-32BAppendix; ablates dataset generalization (Table 8, VerIF)
VerIF-Pro20k40% / 60%Refined codeQwen3-32BAppendix; ablates dataset generalization (Table 8, VerIF-Pro)
AuthorsOpenReview ↗

Follow-Up on Responses

Dear Reviewer,

Thank you again for acknowledging the contributions of this work.

With the discussion period closing soon, we would like to respectfully follow up and inquire if you have the opportunity to review our responses to your comments.

We sincerely look forward to your feedback.

Reviewer_wyY1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the interaction between constraint diversity and reward reliability in reinforcement learning for instruction following. The authors compare hard-only, soft-only, and mixed-constraint training, evaluate verifier reliability against human annotations, and conduct reward-noise and constraint-budget experiments. They argue that unreliable LLM judgments, particularly missed violations under multiple constraints, limit the benefits of soft-constraint training. Based on these observations, they propose Hard + 1 Soft (H1S), which retains all hard constraints and uniformly selects at most one soft constraint per instruction.

Critical strengths
  1. The paper identifies a practically relevant issue in instruction-following RL: expanding constraint coverage can also introduce unreliable supervision.

  2. The empirical diagnosis combines human-annotated verifier evaluation, controlled noise injection, and a soft-constraint-budget ablation. The initial hard/soft comparison also covers several policy architectures and sizes, providing evidence beyond a single model configuration.

Critical weaknesses
  1. My main concern is the incremental conceptual contribution. The central message that RL performance depends strongly on reward reliability, and that imperfect rewards can encourage exploitation, is conceptually unsurprising. Demonstrating this clearly in instruction following has value, particularly the hard-to-soft transfer results, but H1S itself is a simple constraint-pruning heuristic. I am not yet convinced that the current explanation establishes enough additional insight for a top-tier conference.

  2. The underlying source of the large H1S gains over Mix remains insufficiently explained. Figure 3 shows a much larger change in violation-detection recall than in reward precision when moving between one and two constraints. However, this observation alone does not explain the downstream gains in Table 3, including improvements from 78.37 to 85.25 on IFEval and from 29.59 to 38.13 on IFBench at 7B. These benchmarks use rule-based constraints, so their improvements is hard to be attributed to a more favorable soft-constraint evaluation judge.

  3. The soft-constraint selection procedure needs more detail and validation. Equation (4) specifies uniform sampling, but it is unclear whether the retained constraint is sampled once during dataset construction or resampled across epochs or rollouts. Section 6.4 reports averages over three seeds, but does not isolate sensitivity to the selected constraints or report its variability. Could different retained subsets materially change the results? As the limitations section stated, uniform selection is also not necessarily the strategy that maximizes violation-detection recall.

  4. The interpretation of Figure 5 is stronger than the evidence supports. The statement "No consistent improvement is observed as constraints increase" is compatible with the plotted non-monotonic curves, but the figure also contains clear improvements as more constraints are used.

Questions for authors

Frontier long cot models are not considered. Would they follow the same pattern as short cot, or would it be different?

3 comments
AuthorsOpenReview ↗

Response to Weakness #1-3

Thank you for your valuable feedback. We will address your confusion one by one.

W1 (Incremental conceptual contribution).

Our contribution is a specific, falsifiable account of how reliability shapes the value of constraint diversity in IF-RLVR, and the data-composition rule that follows from it. Four points are not implied by the generic statement.

  1. The effect is asymmetric across constraint types. Hard-only matches mixed training at all four model scales and beats soft-only even on soft benchmarks (Table 1). The optimal allocation is therefore asymmetric: keep all hard constraints, prune soft to at most one.

  2. The failure mode is recall, and it is compositional. The judge's bottleneck is recall, not precision (30.6% vs 81.2% for code), and adding a second soft constraint cuts recall by 15.6% while precision drops only 2.5% (Table 2, Fig. 3). This explains why multi-constraint diversity backfires and predicts the inverted-V.

  3. The relationship is quantitative and predictive, not a slogan. Training reward rises monotonically with the number of soft constraints while test performance follows an inverted V peaking at k = 1 (Figs. 1, 7). This reward-hacking signature is what H1S is read off from, not a standalone heuristic.

  4. The transfer result is the most non-obvious part. Training only on code-verifiable hard constraints still improves soft-constraint benchmarks (Table 1), and corrupting the hard reward at the judge-matched recall level closes most of the gap (Fig. 4).

Here, we summary these new contributions:

QuestionPrevailing viewThis paper
Why does more diversity help?Broader coverage → better generalizationOnly when the verifier reliably scores each added constraint; otherwise it adds reward noise
Which reliability failure binds?False positives / reward hacking in generalRecall: judges fail to penalize violations; recall drops 15.6% with a second soft constraint
How much soft diversity?More is betterInverted-V; k = 1 is optimal under noisy judges
How to improve?Better verifier or more dataAsymmetric allocation: keep all hard, prune soft to one; no new verifier, 58% faster

We also agree that H1S is simple, but simplicity is intentional: it is the minimal intervention that isolates the mechanism, it requires no new verifier, and it cuts per-step training latency while outperforming more complex baselines.

W2 (Source of the hard-constraint gains).

The hard-benchmark gain is not evaluation-side; it comes from removing a training reward-hacking channel: in Mix, many soft constraints are scored by a low-recall judge, so the policy can exploit judge biases rather than satisfy constraints (training reward +3.4%, test performance −2.7%; Fig. 2), and this corruption degrades the shared policy on hard constraints as well.

H1S keeps all hard constraints plus exactly one soft constraint, the regime in which the judge is more reliable (Table 2, Fig. 3), so the resulting high-precision reward teaches a general IF skill that transfers across constraint types (Table 1/Section 3; Table 3;). The noise-injection result (Fig. 4) and the attention shift in Appendix E corroborate this mechanism.

W3 (Selection protocol and subset sensitivity).

(1)The retained soft constraint is sampled once during dataset construction and fixed for each example for the entire training run; it is not resampled per epoch or per rollout, so the reward is stationary.

(2) It should be clarified that, the three seeds in §6.4 vary initialization and rollout stochasticity, not the retained subset; to isolate subset sensitivity, we will add three independent subset draws, each trained with two seeds, and report across-draw and across-seed variance.

(3) We agree that uniform sampling is not the optimal selection rule for maximizing violation-detection recall; it is a neutral default that avoids selection bias, and we state adaptive selection as future work.

The main claim does not depend on picking the best constraint: H1S's gain comes from reducing the soft budget to the judge's reliable single-constraint regime, so any retained constraint benefits from that regime.

AuthorsOpenReview ↗

Response to Weakness #4 & Q1

Weakness 4 (Interpretation of Figure 5).

We agree the original sentence is a little too broad: Figure 5 does show a clear initial rise from 0 to a low constraint count. The precise pattern is an initial rise followed by a plateau or slight decline, and in Figure 5 that rise co-varies with training-data volume, so it cannot be attributed to constraint diversity alone.

As mentioned in Reviewer n3o9 (W2) & 3H2Q (W1), 'confounded factors in the Section 4.2 diversity experiment', To address this, we fix the training-data volume and vary only the number of constraints retained per example. Starting from the Mix training set, we keep 0, 1/4, 2/4, 3/4, or 4/4 of each example's constraints, and train all five variants under the identical recipe.

Constraints retained(a) IFEval(b) CFBench
00.7340.44
1/40.8230.53
2/40.8050.53
3/40.8020.52
4/40.7840.51

Test performance does not increase monotonically with the number of constraints; performance rises from 0 to 1/4 and then plateaus or declines (IFEval 0.734 → 0.823 → 0.784; CFBench 0.44 → 0.53 → 0.51); This is consistent with the original conclusion: once a minimal level of constraint diversity is reached, adding more constraints yields no consistent gain.

Question 1 (Frontier long-CoT judges).

Table 2 already includes long-CoT-capable judges (Gemini-2.5-Pro and Qwen3-235B-Inst). To address the concern directly, we have added two recent frontier long-CoT models, Gemini-3.1-Pro and DeepSeek-4.1-Flash in Table 2:

Reward StrategyHard Constraint<br>Prec.Hard Constraint<br>Rec.Soft Constraint<br>Prec.Soft Constraint<br>Rec.
Code Verifier96.081.2––
Gemini-3.1-Pro88.069.389.165.4
DeepSeek-4.1-Flash89.572.785.364.6
Gemini-2.5-Pro86.065.786.363.8
Qwen3-235B-Inst73.223.675.663.8
Qwen3-32B76.530.674.520.9
QwQ-32B80.744.178.233.1
Qwen-2.5-32B71.59.271.812.3

The pattern is the same qualitatively, with one quantitative shift. Frontier long-CoT judges substantially improve both precision and recall over OSS judges, but they still trail the code verifier, and the residual gap is largest in recall.

AuthorsOpenReview ↗

Follow-Up on Responses

Dear Reviewer,

Thank you again for acknowledging the contributions of this work.

With the discussion period closing soon, we would like to respectfully follow up and inquire if you have the opportunity to review our responses to your comments.

We sincerely look forward to your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reviews. The reviews converge on four main concerns: 1 the noise calibration in §4.2 (n3o9, 3H2Q); 2 reporting errors in Table 8 and §F.1.1(n3o9, 3H2Q); 3 insufficient control in the §4.2 diversity experiment (wyY1, n3o9, 3H2Q); 4 the boundary to prior work (wyY1, n3o9); We correct the first two and add new controls for the latter two:

(1) Noise calibration. We corrected the injection-rate derivation and reran the matched conditions at the properly calibrated rate. The recalibrated gaps change only marginally, so the paper's conclusion is preserved rather than reversed.

(2) Controlled diversity experiment. We added a fixed-volume control that varies only the number of retained constraints per example. The new results show no consistent gain from more constraints, confirming the original conclusion once constraint count is isolated from data scale.

(3) Reporting errors. We corrected the Table 8 average, narrowed the §F.1.1 claim to a per-benchmark statement. The corrected numbers now agree with the main text's claims and leave the main results unchanged.

(4) Mechanism and novelty. We added a component control separating constraint retention from reward aggregation, which traces H1S's gain to the judge's reliable single-constraint regime, and we clarified the relation to prior work and scoped our claim to the noisy-judge setting. This makes the contribution and its boundary explicit without overclaiming generality.

Long-CoT judges. We extended the reliability audit with two frontier long-CoT judges. They narrow but do not close the gap to the code verifier, which corroborates the precision bottleneck rather than weakening it.

Summary. Taken together, all main concerns raised by reviews are resolved; the main results and conclusions are unchanged and now rest on a cleaner experimental basis. We thank them again for helping us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.