- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Soft-RLVR, which creates a checklist from each instruction and scores a response by the fraction of checklist items approved by an LLM verifier. It analyzes the variance reduction and partial-credit bias introduced by this reward. The authors also develop Soft-SVeRL, where the policy and verifier share parameters, with verifier co-training and an additional penalty intended to prevent overly permissive judgments. The evaluation uses a controlled instruction-following dataset and Command R7B.
Critical strengths
- Checklist rewards are a sensible choice for instructions containing several requirements. The discussion of how checklist errors affect the reward is especially useful. In particular, the discussion of partial-credit bias makes the limitation of the formulation explicit.
- The comparison across verifier strengths is informative. With GPT-OSS-20B, checklist training reaches 84.20 on IFEval, versus 80.80 for holistic verification. The gap is large enough to take seriously. I also found the self-verification collapse and the effect of replay and verifier votes interesting; Figure 2 gives a concrete reason for the stabilization terms.
Critical weaknesses
- The core application motivation is broader than the experiments. The training prompts have IFEval-style constraints for which a rule-based checker exists, although the policy is trained with an LLM verifier. The case that interests me is a semantic instruction such as preserving the argument of a paragraph while changing its tone. A checklist generator might omit the important part of that request, and the learned verifier would then reward the wrong objective. There is little evidence here about that failure mode. Even a small manually checked set of open-ended instructions would be valuable.
- The benefit of decomposition depends heavily on the verifier. With GPT-OSS-120B, holistic verification obtains 85.58 on IFEval and checklist verification obtains 85.00. This is an important result, since the paper is partly motivated by the limitations of holistic judgments. I would like a sharper discussion of where checklist grading is worth the additional per-item verification work. Reporting the number of checklist items and verifier calls per response would make the comparison more concrete.
- The self-verification results are much more preliminary than the external-verifier results. All self-verification variants run for 100 steps, while the main Soft-RLVR experiments are reported after 500. The strongest reported self-verification setting reaches 77.02 on IFEval, still well short of the 85.00 external-verifier result. Moreover, self-verification includes training on gold prompt-response-criterion labels. Gold supervision helps stabilize the shared judge, and the practical benefit of replacing an external verifier depends on the amount of labeled data this requires. A matched training-budget comparison with a self-judging holistic reward would help establish what is gained by sharing the policy parameters.
- The theory estimates verifier sensitivity using ground-truth constraint items in the controlled evaluation. The actual training checklists are generated by another model. It would be useful to measure how often errors in checklist generation dominate errors in item verification, since the variance analysis is most favorable when the decomposition itself is reliable.
Questions for authors
- Could the authors manually audit generated checklists on semantic instructions and report the rate of missing or misleading criteria?
- How many labeled verifier examples are used for Soft-SVeRL, and what happens when this set is reduced? A comparison with holistic self-grading under the same training budget would help clarify the benefit of the proposed shared-verifier design.
- I wonder why the overview of the proposed method (Figure 4) is placed in the appendix. I think it would be more helpful to have this figure in the main paper.