acceptodds
Under review as a conference paper at ICLR 2027

Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewards

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) has improved language models in domains such as mathematics and code, where correctness can be checked automatically. However, many important tasks are only partially verifiable: prompts contain multiple requirements, responses may satisfy some but not all of them, or no single reference answer might exist. We introduce Soft-RLVR, a framework for reinforcement learning from decomposed, learned verification signals. Soft-RLVR converts each prompt into a checklist of atomic requirements, scores candidate responses item by item with an LLM verifier, and trains on the resulting soft reward. Checklist-based rewards turn sparse pass/fail supervision into a denser partial-credit signal, but they also introduce a tradeoff: averaging item-level judgments can reduce verifier noise, while partial credit can reward incomplete responses. We formalize this tradeoff and identify conditions under which checklist-based verification gives a more reliable RL training signal than holistic verification. We further introduce Soft-SVeRL, a self-verifying variant of Soft-RLVR in which the policy also acts as the verifier. We show that self-verification is prone to reward inflation from overly permissive self-judgments, and that explicit stabilization is needed to prevent this collapse. In a controlled instruction-following setting with rule-based ground-truth evaluation, checklist-based Soft-RLVR improves IFEval by up to 11.1 points using only learned verifier rewards. Our experiments further show that verifier quality and checklist quality both affect downstream RL outcomes, and that explicit stabilization is essential for effective self-verification.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL, LLMs, self-verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4dWuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Soft-RLVR, which creates a checklist from each instruction and scores a response by the fraction of checklist items approved by an LLM verifier. It analyzes the variance reduction and partial-credit bias introduced by this reward. The authors also develop Soft-SVeRL, where the policy and verifier share parameters, with verifier co-training and an additional penalty intended to prevent overly permissive judgments. The evaluation uses a controlled instruction-following dataset and Command R7B.

Critical strengths
  1. Checklist rewards are a sensible choice for instructions containing several requirements. The discussion of how checklist errors affect the reward is especially useful. In particular, the discussion of partial-credit bias makes the limitation of the formulation explicit.
  2. The comparison across verifier strengths is informative. With GPT-OSS-20B, checklist training reaches 84.20 on IFEval, versus 80.80 for holistic verification. The gap is large enough to take seriously. I also found the self-verification collapse and the effect of replay and verifier votes interesting; Figure 2 gives a concrete reason for the stabilization terms.
Critical weaknesses
  1. The core application motivation is broader than the experiments. The training prompts have IFEval-style constraints for which a rule-based checker exists, although the policy is trained with an LLM verifier. The case that interests me is a semantic instruction such as preserving the argument of a paragraph while changing its tone. A checklist generator might omit the important part of that request, and the learned verifier would then reward the wrong objective. There is little evidence here about that failure mode. Even a small manually checked set of open-ended instructions would be valuable.
  2. The benefit of decomposition depends heavily on the verifier. With GPT-OSS-120B, holistic verification obtains 85.58 on IFEval and checklist verification obtains 85.00. This is an important result, since the paper is partly motivated by the limitations of holistic judgments. I would like a sharper discussion of where checklist grading is worth the additional per-item verification work. Reporting the number of checklist items and verifier calls per response would make the comparison more concrete.
  3. The self-verification results are much more preliminary than the external-verifier results. All self-verification variants run for 100 steps, while the main Soft-RLVR experiments are reported after 500. The strongest reported self-verification setting reaches 77.02 on IFEval, still well short of the 85.00 external-verifier result. Moreover, self-verification includes training on gold prompt-response-criterion labels. Gold supervision helps stabilize the shared judge, and the practical benefit of replacing an external verifier depends on the amount of labeled data this requires. A matched training-budget comparison with a self-judging holistic reward would help establish what is gained by sharing the policy parameters.
  4. The theory estimates verifier sensitivity using ground-truth constraint items in the controlled evaluation. The actual training checklists are generated by another model. It would be useful to measure how often errors in checklist generation dominate errors in item verification, since the variance analysis is most favorable when the decomposition itself is reliable.
Questions for authors
  1. Could the authors manually audit generated checklists on semantic instructions and report the rate of missing or misleading criteria?
  2. How many labeled verifier examples are used for Soft-SVeRL, and what happens when this set is reduced? A comparison with holistic self-grading under the same training budget would help clarify the benefit of the proposed shared-verifier design.
  3. I wonder why the overview of the proposed method (Figure 4) is placed in the appendix. I think it would be more helpful to have this figure in the main paper.
Reviewer_gVQUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Soft-RLVR, a framework that decomposes complex instructions into checklists and uses LLM-based verification to provide soft rewards for reinforcement learning. It further introduces Soft-SVeRL, a self-verifying variant with mechanisms to prevent reward inflation and stabilize training. The authors also provide a theoretical analysis of the bias-variance tradeoff in checklist-based verification. Experiments on instruction-following benchmarks show improvements of up to 11.1 percentage points on IFEval, while ablation studies examine the effects of verifier quality, checklist quality, and self-verification mechanisms.

Critical strengths
  1. The paper addresses the challenge of constructing reliable rewards for partially verifiable tasks, which is important for extending RLVR beyond domains with exact verification.

  2. The paper provides a theoretical analysis of checklist-based rewards and demonstrates substantial improvements on IFEval. The ablation studies further support the effectiveness of the proposed approach.

Critical weaknesses
  1. Checklist-based rewards are closely related to existing rubric-based evaluation methods, making the technical novelty somewhat limited.
  2. The strongest improvements come from external verifiers, while the self-verifying variant shows relatively modest gains and still relies on labeled verification data. This limits the evidence supporting its effectiveness and generalizability.
Questions for authors
  1. How does the method handle incomplete, redundant, or incorrectly generated checklist items, and how do these errors affect RL training?
  2. Can the authors explain why the self-verifying variant achieves substantially smaller improvements than the external-verifier approach?
  3. How sensitive is Soft-SVeRL to the amount and quality of labeled verification data used for verifier co-training?
  4. Can the authors provide stronger evidence that Soft-SVeRL remains stable over longer training periods without reward inflation?
Reviewer_WQbmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies RLVR for instruction-following tasks where single holistic judgments are noisy or too sparse. The authors propose SOFT-RLVR, which decomposes each prompt offline into a checklist of atomic yes/no criteria, evaluates each criterion with an LLM verifier, and uses the fraction of satisfied items as a soft reward for GRPO. They analyze when checklist averaging reduces gradient MSE relative to holistic verification, and further propose SOFT-SVERL, a self-verifying variant where the generator shares parameters with the verifier and is stabilized via repeated voting, verifier co-training, and a Yes-rate penalty. On Command R7B trained on Llama-Nemotron and evaluated on IFEVAL, external checklist verification improves accuracy by up to 11.1 points, while self-verification improves by about 3.1 points over 100 steps.

Critical strengths
  1. Decomposing multi-constraint instructions into atomic checklist items to provide dense partial-credit rewards is intuitive, and the ablations in Figure 1 cleanly separate the effects of checklist quality and verifier quality.

  2. Analyzing the bias-variance trade-off between variance reduction from item averaging and partial-credit relaxation gap is a useful perspective, and the diagnosis of "always-yes" collapse in naive shared-parameter self-verification is informative.

Critical weaknesses
  1. There is a major disconnect between the title/framing ("Self-Verified RL") and what actually drives the empirical gains, and the self-verifying variant relies on privileged assumptions while removing the efficiency advantage. First, the headline +11.1 gain on IFEVAL (Table 1, 73.89 to 85.00) comes entirely from SOFT-RLVR using an external DeepSeek-R1 checklist generator and a frozen 20B/120B external verifier, not from self-verification. When the initial policy itself is used as a frozen verifier (Figure 1), IFEVAL is 73.83 (vs. 73.89 baseline, zero gain). Even with the full SOFT-SVERL stabilization recipe (Figure 2), it only reaches 77.02 (+3.13 points) and is only run for 100 steps rather than 500. Second, SOFT-SVERL is not truly self-supervised: it still uses checklists generated offline by DeepSeek-R1 and requires gold-labeled prompt-response-criterion triples (x,y,c)(x, y, c) to co-train the verifier. Moreover, scoring KK checklist items with J=8J=8 verifier votes per item requires 8K8K verifier generations per rollout, which undermines the paper's claim that self-verification reduces the inference footprint of RL training.

  2. The core benefits of checklist decomposition are restricted to weak verifiers and synthetic rule-based tasks, and the related work omits several closely related studies on self-rewarding, meta-rewarding, generative verification, and checklist-based evaluation, not limited to [1-5]. In Figure 3b, when using the stronger GPT-OSS-120B verifier, single-call holistic verification (85.58) already matches or slightly outperforms checklist verification (85.00) on IFEVAL while avoiding KK-fold verification calls. Furthermore, the training and evaluation prompts (Llama-Nemotron and IFEVAL) are explicitly constructed from atomic, rule-checkable constraints (e.g., word counts, exact ending phrases), which artificially favors atomic yes/no checklists and raises doubts about whether Theorem 1 (which assumes conditional independence across checklist judgments) holds on realistic open-ended instructions. Finally, only a single base model (Command R7B) is evaluated, and no existing self-rewarding or generative-verifier baselines [1-5] are empirically compared.

[1] Self-Rewarding Language Models

[2] Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge

[3] Self-Taught Evaluators

[4] Generative Verifiers: Reward Modeling as Next-Token Prediction

[5] CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists

Questions for authors
  1. Why is SOFT-SVERL only trained for 100 steps in Section 6.2 while Table 1 trains for 500 steps? Does SOFT-SVERL remain stable up to 500 steps, and what is its accuracy if the checklists are also generated by Command R7B itself rather than DeepSeek-R1?

  2. Where do the gold-labeled prompt-response-criterion triples (x,y,c)(x, y, c) for verifier co-training come from in Section 4.2? If they are generated by the ground-truth rule-based verifier or a stronger teacher, doesn't this defeat the purpose of self-verification?

  3. In Figure 3b, holistic verification with GPT-OSS-120B achieves 85.58 on IFEVAL (slightly beating 85.00 for checklist verification) using only 1 verifier call per rollout instead of KK calls. Under a matched verifier-token or compute budget, when is checklist verification preferable to holistic verification?

  4. Can the authors discuss and compare with prior self-rewarding and meta-rewarding RL frameworks [1-5], and evaluate the method on at least one open-ended instruction benchmark beyond IFEVAL (e.g., AlpacaEval 2 or Arena-Hard)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.