- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies token-level credit assignment for RLVR-based instruction following and proposes LoCo, a capability-aware counterfactual method that uses leave-one-out constraint ablation for in-capability constraints and failure-reason-conditioned contrast plus demonstration-conditioned self-distillation for out-of-capability constraints. Experiments on two Qwen models and eight benchmarks show consistent gains over GRPO and strong RLVR baselines under CSR/AON rewards, supported by ablations, hyperparameter, efficiency, and judge-robustness analyses.
Critical strengths
-
The capability dichotomy is empirically motivated and clearly supported by leave-one-out contrast and CAD analyses.
-
LoCo avoids an external token-level discriminator and integrates constraint-level token credit conservatively with the outcome advantage.
-
The empirical evaluation is broad across models, benchmarks, reward aggregations, RLVR algorithms, judges, and ablations.
Critical weaknesses
-
Per-constraint ablation/repair forward passes and external editing make training costly and potentially hard to scale to many constraints.
-
Failure verbalization, semantic judging, and minimal-edit repair rely on external models/judges, raising bias, reward-hacking, cost, and reproducibility concerns.
-
the main tables lack variance/significance tests and a CAD baseline.
-
The claimed equivalence to CAD relies on simplified assumptions, while multi-turn, long-context, and constraint-interaction settings remain unverified.
Questions for authors
See weakness
3 comments
Thank you for your reply. I will maintain my positive attitude.
Thank you for your recognition. We really appreciate the time and effort you put into reviewing our work.
Thank you for your constructive suggestions.
Response to Q1. As you pointed out, our method add cost as the number of constraints grows. To handle this, a simple option is to cluster the constraints by similarity or by constraint type with an external model. Since HIR already labels each constraint (e.g., length_constraints, keyword, language, etc.), we group same-type constraints into one atomic constraint for advantage and credit assignment, and we name this variant C-cluster. Experiments below are conducted on Qwen3-4B-Instruct with CSR as the outcome reward.
C-cluster shows no clear drop from LoCo. So when there are many constraints, they can be clustered first and then fed into our method, and LoCo remains effective after clustering.
Response to Q2. Semantic judging is needed here, since other baselines and their papers also use external APIs to check constraint satisfaction. Moreover, constraint checking is fairly objective, so bias or reward hacking problems are minor. For the concerns on failure verbalization and minimal-edit repair, we add an ablation where Qwen3-4B-Instruct itself gives the failure reasons and the repair answers for SDFT (LoCo-self), which reduces both API cost and potential bias.
Even with a small drop, LoCo-self still leads the baselines by a clear margin, which shows our method can reduce its reliance on external models.
Response to Q3. We run three seeds on Qwen3-4B for the main GRPO-vs-LoCo comparison, and report the mean with the standard deviation:
For the CAD baseline, we have reported it in Appendix B, and have now added it to the Table 1 in the revised manuscript.
Response to Q4. As you pointed out, our theory use simplified assumptions. But our analysis is not to claim strict equivalence; it is only a first-order analysis to motivate gradient-direction alignment, which we place in the appendix as a preliminary discussion. Our experiments also indicates that training with LoCo's credit outperforms the training-free CAD decoding.
We thank the reviewer again for these suggestions, and have added the corresponding additional experiments to the revised manuscript, which we believe will meaningfully strengthen the paper!