acceptodds
Under review as a conference paper at ICLR 2027

Do LLMs Know Where Constraints Bite? Towards Fine-Grained Reward Signals for Instruction Following

Abstract

Reinforcement learning with verifiable rewards has become the dominant post-training paradigm for instruction following. Current pipelines compress all constraint verdicts into a scalar, and broadcast the advantage uniformly across every token, while such coarse-grained credit assignment fails to localize responsible tokens for each constraint. In this paper, we investigate whether the policy's own generation distribution can be utilized for finer-grained credit. Through a pilot experiment, we find that for constraints within the model's capability, removing a constraint from the prompt causes the probabilities of exactly the tokens that implement it to drop sharply. However, for constraints beyond capability, no sampled response could satisfy them and the same contrast becomes unreliable. Based on this dichotomy, we propose LoCo, which performs constraint-level credit assignment: for constraints within the model's capability, we localize the responsible tokens by the leave-one-out contrast of each token; for constraints beyond it, we additionally provide the policy with the verbalized failure reason, so that the probability difference could localize the offending tokens. Moreover, when no sampled response satisfies all constraints, demonstration-conditioned self-distillation provides the policy with a golden response minimally edited from its best attempt to expand the model's capability boundary. Comprehensive experiments across eight instruction-following tasks confirm the effectiveness of our proposed LoCo.

Primary area:
reinforcement learning
Keywords:
large language model, instruction following, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_69v2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies token-level credit assignment for RLVR-based instruction following and proposes LoCo, a capability-aware counterfactual method that uses leave-one-out constraint ablation for in-capability constraints and failure-reason-conditioned contrast plus demonstration-conditioned self-distillation for out-of-capability constraints. Experiments on two Qwen models and eight benchmarks show consistent gains over GRPO and strong RLVR baselines under CSR/AON rewards, supported by ablations, hyperparameter, efficiency, and judge-robustness analyses.

Critical strengths
  • The capability dichotomy is empirically motivated and clearly supported by leave-one-out contrast and CAD analyses.

  • LoCo avoids an external token-level discriminator and integrates constraint-level token credit conservatively with the outcome advantage.

  • The empirical evaluation is broad across models, benchmarks, reward aggregations, RLVR algorithms, judges, and ablations.

Critical weaknesses
  • Per-constraint ablation/repair forward passes and external editing make training costly and potentially hard to scale to many constraints.

  • Failure verbalization, semantic judging, and minimal-edit repair rely on external models/judges, raising bias, reward-hacking, cost, and reproducibility concerns.

  • the main tables lack variance/significance tests and a CAD baseline.

  • The claimed equivalence to CAD relies on simplified assumptions, while multi-turn, long-context, and constraint-interaction settings remain unverified.

Questions for authors

See weakness

3 comments

Thank you for your constructive suggestions.

Response to Q1. As you pointed out, our method add cost as the number of constraints grows. To handle this, a simple option is to cluster the constraints by similarity or by constraint type with an external model. Since HIR already labels each constraint (e.g., length_constraints, keyword, language, etc.), we group same-type constraints into one atomic constraint for advantage and credit assignment, and we name this variant C-cluster. Experiments below are conducted on Qwen3-4B-Instruct with CSR as the outcome reward.

MethodIFEvalIFBMulDimID-AvgCFBInFoFollowBAdvIFRoleMOOD-Avg
LoCo87.140.083.470.250.550.871.745.144.052.4
C-cluster87.340.382.670.150.051.270.944.644.252.2

C-cluster shows no clear drop from LoCo. So when there are many constraints, they can be clustered first and then fed into our method, and LoCo remains effective after clustering.

Response to Q2. Semantic judging is needed here, since other baselines and their papers also use external APIs to check constraint satisfaction. Moreover, constraint checking is fairly objective, so bias or reward hacking problems are minor. For the concerns on failure verbalization and minimal-edit repair, we add an ablation where Qwen3-4B-Instruct itself gives the failure reasons and the repair answers for SDFT (LoCo-self), which reduces both API cost and potential bias.

MethodIFEvalIFBMulDimID-AvgCFBInFoFollowBAdvIFRoleMOOD-Avg
LoCo87.140.083.470.250.550.871.745.144.052.4
LoCo-self86.739.782.869.749.950.071.744.242.951.7

Even with a small drop, LoCo-self still leads the baselines by a clear margin, which shows our method can reduce its reliance on external models.

Response to Q3. We run three seeds on Qwen3-4B for the main GRPO-vs-LoCo comparison, and report the mean with the standard deviation:

RewardMethodIFEvalIFBenchMulDimIFCFBenchInfoBenchFollowBenchAdvIFRoleMRC
CSRGRPO84.2±0.636.0±0.980.8±0.846.3±0.346.2±0.269.2±0.341.8±0.341.7±0.3
LoCo87.2±0.840.1±0.983.1±0.750.2±0.450.7±0.671.7±0.445.3±0.444.1±0.5
AONGRPO84.6±0.636.1±1.080.4±0.546.7±0.247.7±0.369.9±0.241.7±0.441.1±0.4
LoCo86.7±0.540.7±0.781.9±0.749.0±0.449.5±0.471.4±0.544.6±0.543.8±0.2

For the CAD baseline, we have reported it in Appendix B, and have now added it to the Table 1 in the revised manuscript.

Response to Q4. As you pointed out, our theory use simplified assumptions. But our analysis is not to claim strict equivalence; it is only a first-order analysis to motivate gradient-direction alignment, which we place in the appendix as a preliminary discussion. Our experiments also indicates that training with LoCo's credit outperforms the training-free CAD decoding.

We thank the reviewer again for these suggestions, and have added the corresponding additional experiments to the revised manuscript, which we believe will meaningfully strengthen the paper!

Thank you for your reply. I will maintain my positive attitude.

Thank you for your recognition. We really appreciate the time and effort you put into reviewing our work.

Reviewer_tMsHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fine-grained credit assignment for reinforcement learning with verifiable rewards (RLVR) in instruction following. The key observation is that, for a constraint that is already within the policy's capability, removing that constraint from the prompt often causes a sharp decrease in the probabilities of the tokens that implement it. Based on this observation, the paper proposes LOCO (Leave-one-out Constraint for Credit Assignment). LOCO first computes a constraint-wise advantage, then routes that advantage to individual tokens using a leave-one-out log-probability contrast. For constraints for which none of the sampled rollouts succeeds, LOCO conditions the policy on a verbalized failure reason to repair the contrast. It further uses demonstration-conditioned self-distillation: an external editor minimally repairs the best rollout, and the frozen reference policy conditioned on this repaired demonstration provides a top-K distillation target. The method is evaluated on Qwen3-4B-Instruct and Qwen3.5-9B using HIR-16K and eight instruction-following benchmarks. Across CSR and all-or-nothing rewards, LOCO consistently improves over GRPO, Replay, and R2T. The paper also evaluates transfer to PPO/DAPO/GSPO, general-capability retention, judge robustness, component ablations, hyperparameter sensitivity, and training cost.

Critical strengths

S1. LOCO offers a principled token-level credit signal for RLVR by exploiting constraint-ablation probability shifts from the policy itself, avoiding a separately trained attribution model. The capability-aware routing further gives the method a coherent treatment of both satisfiable and currently unsatisfied constraints.

S2. The evaluation is broad and well controlled: two backbones, eight IF benchmarks, two reward formulations, several RL algorithms, judge robustness, ablations, hyperparameter sweeps, and wall-clock analysis. The consistency of the gains substantially strengthens the empirical case.

Critical weaknesses

W1. The capability split in Eq. 8 is a high-variance hard decision: one success among G=6 rollouts changes the entire credit route. Sensitivity to G or softer capability estimates is not studied.

W2. The key localization hypothesis is validated mainly on constraints with explicitly identifiable responsible tokens. Evidence for semantic, stylistic, compositional, or long-range constraints remains indirect.

W3. LOCO uses richer external supervision for hard constraints, including failure explanations and minimally edited demonstrations from a stronger model. Without matched-supervision baselines, the gain attributable specifically to the proposed credit assignment is unclear.

W4. Main results appear to be single-run numbers, with no variance or confidence intervals. This is important because some gains are modest and AON training is visibly hyperparameter-sensitive.

W5. Constraint-wise rescoring introduces cost that should grow with the number of constraints. The reported 1.52-1.66x step overhead is useful, but scaling with constraint count, sequence length, memory, and batching is not characterized.

Questions for authors

Q1. How stable is the Eq. 8 capability partition across rollout groups and training? Please report sensitivity to G and, if possible, a soft satisfaction-rate-based alternative.

Q2. Can you quantitatively validate token localization on semantic or judge-based constraints, beyond cases with manually identifiable responsible tokens?

Q3. Could you add matched-supervision baselines that receive the same failure reasons or repaired demonstrations without LOCO's token-credit mechanism?

Q4. Were Tables 1-4 and Figure 5 obtained from multiple random seeds? Please report uncertainty for the main GRPO-vs.-LOCO comparisons.

Q5. How does runtime and memory scale with the number of constraints K and response length, and which constraint-wise forward passes can be cached or batched?

1 comment

Thank you for your constructive suggestions.

Response to Q1. To check the stability of the Eq. (8) capability partition, we run four rollouts with group size 6, and mark the set of beyond-capability constraints as n1–n4n_1–n_4, and report the pairwise Jaccard values:

pairn1n_1 vs n2n_2n1n_1 vs n3n_3n1n_1 vs n4n_4n2n_2 vs n3n_3n2n_2 vs n4n_4n3n_3 vs n4n_4
Jaccard0.8460.8500.8460.8470.8450.846

The detected constraints stay ~0.85 across rollout groups, so the partition is relatively stable. We also add a soft satisfaction-rate-based alternative LoCo-soft=kk, where a constraint is judged beyond capability if no more than kk rollouts satisfy it. Experiments below are conducted on Qwen3-4B-Instruct with CSR as the outcome reward.

MethodIFEvalIFBMulDimID-AvgCFBInFoFollowBAdvIFRoleMOOD-Avg
LoCo87.140.083.470.250.550.871.745.144.052.4
LoCo-soft=186.740.783.570.350.750.672.143.943.952.2
LoCo-soft=387.439.382.169.649.751.070.944.643.852.0

Overall, kk=1 keeps performance almost unchanged, and kk=3 drops a little. Since expanding the set of out-of-capability constraints require extra API calls and additional prefill overhead, our original design remains the better choice.

Response to Q2. We take semantic constraints for further analysis, and use deepseek-v4-pro to annotate the responsible tokens. We then compute the score distribution for within-capability constraints according to Eq.(3), with p05p_{05}–p99p_{99} denoting percentiles.

Categorymeanp05p_{05}p25p_{25}p75p_{75}p95p_{95}p99p_{99}
unrelated0.053−0.0030.0040.0470.1650.842
related1.2410.0120.1141.8654.7687.355

Responsible tokens have clearly higher scores than unrelated ones. Note that the p99p_{99} of our unrelated group reaches high. This is because in our proposed LoCo, tokens that are not literally labeled as responsible but still help satisfy the constraint can be also rewarded, while a strict localization method would miss them.

Response to Q3. As suggested, we add a matched-supervision baseline. Specifically, we equip GRPO with 2way-SDFT: besides the SDFT on repaired demonstrations (Section 4.2), we collect failure evidence and let the model run extra SDFT with teacher provided with the failed reasons for refinement.

MethodIFEvalIFBMulDimID-AvgCFBInFoFollowBAdvIFRoleMOOD-Avg
LoCo87.140.083.470.250.550.871.745.144.052.4
2way-SDFT85.837.381.768.346.848.069.442.542.849.9

Even with the same reasons and demonstrations, GRPO still underperform LoCo, which shows the importance of LoCo's token-credit mechanism.

Response to Q4. We run three repeats on Qwen3-4B-Instruct for the main GRPO-vs-LoCo comparison, under both CSR and AON. Overall, GRPO's upper bound never exceeds LoCo's lower bound, so LoCo steadily performs better.

RewardMethodIFEvalIFBenchMulDimIFCFBenchInfoBenchFollowBenchAdvIFRoleMRC
CSRGRPO84.2±0.636.0±0.980.8±0.846.3±0.346.2±0.269.2±0.341.8±0.341.7±0.3
LoCo87.2±0.840.1±0.983.1±0.750.2±0.450.7±0.671.7±0.445.3±0.444.1±0.5
AONGRPO84.6±0.636.1±1.080.4±0.546.7±0.247.7±0.369.9±0.241.7±0.441.1±0.4
LoCo86.7±0.540.7±0.781.9±0.749.0±0.449.5±0.471.4±0.544.6±0.543.8±0.2

Response to Q5. We split instructions by constraint number into K=2, 6, 10 (K=2 is built by truncation, only for cost), and further split each by response length, with max length 256 (short) or 512 (long). We report the average time of one training step (mean of 3 runs).

Time and memory behave differently. LoCo's constraint-level forwards are packed into a single batch that the worker processes in micro-batches of 8. The total number of sequences grows linearly with K, so the per-step time also rises with K. GPU Memory is opposite: the number of sequences on the GPU is fixed by the micro-batch size regardless of K, so peak memory stay unchanged.

Kavg len (max len)GRPO (s)LoCo (s)×timeGRPO (GB)LoCo (GB)×mem
2212 (256)49.369.01.40×69.783.31.20×
2433 (512)96.5134.11.39×71.786.41.23×
6229 (256)53.679.21.48×69.987.11.25×
6345 (512)71.0110.11.55×70.786.81.23×
10244 (256)57.398.11.71×70.085.11.22×
10390 (512)81.1136.41.68×71.290.01.26×

We include additional experiments in appendix G to show that the time overhead can be reduced through constraint clustering.

We thank the reviewer again for these suggestions, and have added the corresponding additional experiments to the revised manuscript, which we believe will meaningfully strengthen the paper!

Reviewer_KfawOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LOCO, a fine-grained credit assignment method for instruction following with RLVR. It uses leave-one-out probability contrasts and constraint-wise advantages to construct token-level reward signals. For difficult constraints, LOCO introduces failure feedback and self-distillation. Experiments on two Qwen models across eight instruction following benchmarks show consistent improvements in average performance over GRPO and other evaluated baselines.

Critical strengths
  • Interesting motivation and empirical findings. The paper shows that leave-one-out probability contrasts behave differently depending on the model's ability to satisfy a constraint. The pilot study and additional analyses provide useful evidence for the capability-aware design.

  • Comprehensive experiments. LOCO uses the policy's own probability distributions without requiring a separately trained token discriminator. It achieves the best in-domain and out-of-domain average performance across both model backbones and reward formulations, with additional evaluations using different RL algorithms and judge models.

Critical weaknesses
  • Insufficient isolation of the token-level contribution. Table 4 does not isolate the contribution of token localization. A control using uniform or shuffled token weights while retaining the other components is needed to verify whether the improvements genuinely come from accurate token attribution.

  • Insufficient validation of attribution accuracy. Figures 2 and 6 analyze contrast scores on identified tokens, but do not quantify how reliably the method distinguishes responsible from unrelated tokens. Furthermore, the reliability of attribution under interacting or complex semantic constraints remains unclear.

Questions for authors
  1. Could you include a control that retains constraint-wise advantages and the other training components but replaces contrast-based token weights with uniform or shuffled weights? This would help determine whether accurate token localization is responsible for the improvements.

  2. Beyond the mean contrast analyses in Figures 2 and 6, could you quantify how well the token scores distinguish responsible from unrelated tokens, particularly for complex semantic constraints?

  3. For jointly satisfiable interacting constraints, each constraint may be satisfied by some rollouts while no rollout satisfies all constraints simultaneously. Under Eq. (8), all constraints would be classified as within-capability, so self-distillation would not activate. Is this intended, and how does LOCO address such joint failures?

  4. For constraints beyond the model's capability, could you hold the previous response and rewriting instructions fixed while replacing the failure reason with generic or shuffled feedback, to determine whether the improvement comes from constraint-specific failure evidence?

3 comments

Thank you for your constructive suggestions.

Response to Q1. To isolate the contribution of the token-level credit assignment, we compare LoCo with two variants as suggested: 1) w-uniform, where all token weights are set uniformly to 1, to verify that our gains are not simply due to amplifying the reward signal; 2) w-shuffle, where token weights are shuffled at the sequence level, to verify that RL performance only arises when credit is assigned to the correct tokens. Experiments below are conducted on Qwen3-4B-Instruct with CSR as the outcome reward.

MethodIFEvalIFBMulDimID-AvgCFBInFoFollowBAdvIFRoleMOOD-Avg
LoCo87.140.083.470.250.550.871.745.144.052.4
w-uniform84.737.082.167.946.947.069.742.542.049.6
w-shuffle83.734.780.566.345.246.267.542.640.648.4

Both variants clearly underperform LoCo, indicating that accurate token localization is indeed responsible for the improvements.

Response to Q2. Following your suggestion, beyond Figure 2 and 6, we take semantic constraints for further analysis. We use deepseek-v4-pro to annotate the responsible tokens in the answers. We then compute the score distribution for within-capability constraints according to Eq.(3), with p05p_{05}–p99p_{99} denoting percentiles.

Categorymeanp05p_{05}p25p_{25}p75p_{75}p95p_{95}p99p_{99}
unrelated0.053−0.0030.0040.0470.1650.842
related1.2410.0120.1141.8654.7687.355

As shown, responsible tokens have a clearly higher scores distribution than unrelated tokens overall, which confirms that responsible and unrelated tokens are effectively distinguished. Moreover, some of the unrelated tokens have very high scores. This is because in our proposed LoCo, tokens that are not literally labeled as responsible but still help satisfy the constraint can be also rewarded. For example, when a constraint asks for a rare keyword, the model often sets up the context in advance so the keyword can be generated naturally. The generation probability of this setup text increases given the constraint, and LoCo can find and boost them, while a strict localization method would miss them.

Response to Q3. We thank the reviewer for this valuable suggestion, which points out a meaningful extension of our method. As discussed in Limitation section, our design is based on constraint-independence assumption, where "at least one constraint is beyond capability" equals "the whole instruction is beyond capability". However, when constraints truly conflict, the two become different, and this is exactly the gap you point out.

Such cases are rare in our training datasets HIR: of 16325 instructions, 7055 have no rollout satisfying all constraints in 6 rollouts, and only 272 match the case above (3.8% of the 7055, ~1.7% overall). Raising rollouts from 6 to 12, and 144 of these 272 already satisfy all constraints, so most are not true conflicts but insufficient sampling. However, for datasets where constraints conflict substantially, the approach you suggested would undoubtedly be the better choice.

Following your suggestion, we add an experiment that also triggers SDFT under joint failure, denoted LoCo-joint. The difference between the two is well within the range of variance, which is consistent with the nature of the dataset.

MethodIFEvalIFBMulDimID-AvgCFBInFoFollowBAdvIFRoleMOOD-Avg
LoCo87.140.083.470.250.550.871.745.144.052.4
LoCo-joint87.439.783.370.150.251.272.244.644.352.5

Response to Q4. To verify whether the improvement comes from constraint-specific failure evidence, we add two ablations following your suggestion: 1) fb-generic, where every failure reason is replaced by a generic default one: "An automated check of the response determined this constraint is NOT satisfied"; 2) fb-shuffled, where the failure reasons are shuffled within the same batch.

MethodIFEvalIFBMulDimID-AvgCFBInFoFollowBAdvIFRoleMOOD-Avg
LoCo87.140.083.470.250.550.871.745.144.052.4
fb-generic86.338.783.069.349.449.071.242.844.251.3
fb-shuffled85.634.082.067.246.148.866.841.742.349.1

As shown, a generic feedback cannot help the model precisely localize the error, and shuffling the feedback within a batch severely disrupts the model's ability to localize errors, both can cause a drop in performance. This indicates that constraint-specific failure evidence is indeed a key source of the improvement.

We thank the reviewer again for these suggestions, and have added the corresponding additional experiments to the revised manuscript, which we believe will meaningfully strengthen the paper!

Thank the author for addressing my concerns and I will maintain my rating.

Thank you for your recognition. We really appreciate the time and effort you put into reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.