acceptodds
Under review as a conference paper at ICLR 2027

The Wisdom of Wrong Crowds: Learning from Disagreement Among Failed Rollouts

Abstract

Reinforcement learning with verifiable rewards (RLVR) trains language models from verifiable outcomes, but with binary rewards, the outcome-based advantage in group-relative methods vanishes when every rollout for a problem is incorrect. Many recent self-distillation methods provide token-level supervision by modulating verifier-derived advantages, introducing external correct information, or constructing reference-free signals for failed rollouts. However, relations among failed rollouts within an all-wrong group remain underexplored as a source of token-level supervision. We observe that all-wrong groups need not be information-free: failed rollouts can make different mistakes, and disagreement in their answers provides a coarse cue to differences in their reasoning. We introduce **SibCredit**, which converts these answer relations into selective token-level negative credit by re-scoring a target's sampled tokens under contexts constructed from other failed rollouts in the same group. Different-answer rollouts provide candidate negative evidence, but their opposition can also reflect shared conditioning effects, such as shifts in reasoning style. Same-answer rollouts serve as a control, retaining penalties only when different-answer opposition sufficiently exceeds same-answer opposition. Because every conditioning rollout is incorrect, SibCredit uses only induced probability decreases as candidate negative evidence and discards probability increases. The resulting token-level signal is constructed entirely from the current all-wrong group, while other groups use standard GRPO. Experiments on Qwen3-1.7B and Qwen3-4B across mathematical reasoning and code generation show that SibCredit consistently improves over GRPO and recent self-distillation baselines. Token-level analysis further shows that same-answer calibration retains a larger fraction of opposition strength on annotated error tokens than on epistemic expressions.

Primary area:
reinforcement learning
Keywords:
LLM Reasoning, Reinforcement Learning with Verifiable Rewards (RLVR), Token-Level Credit Assignment, On-Policy Self-Distillation (OPSD)

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sSy7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors address the problem that in RLVR with binary rewards, all-wrong rollout groups produce zero advantage and thus no learning signal. They introduce SibCredit, which extracts token-level negative supervision from disagreement among failed siblings: different-answer siblings provide candidate opposition (tokens whose probability drops under sibling conditioning), while same-answer siblings serve as a control to filter shared conditioning effects (e.g., suppression of epistemic expressions). The method is purely negative (no positive credit from wrong siblings), bounded in [-1, 0], and added as an auxiliary objective to GRPO. Experiments on Qwen3-1.7B and Qwen3-4B across math reasoning and code generation show consistent improvements over GRPO (+2.3/+1.7 pp math, +0.8/+0.7 pp code), outperforming self-distillation baselines that require external correct references.

Critical strengths
  1. Same-answer calibration is well-motivated and well-validated. The core insight that different-answer opposition conflates error-related signal with shared conditioning effects (e.g., style shifts, epistemic suppression) is demonstrated both through ablation (removing calibration drops accuracy to 14.6%, below the 17.0% base model, Table 2) and through token-level analysis showing calibration retains 55% of opposition on error tokens vs. 10% on epistemic expressions.

  2. Ablations isolate each design choice convincingly. Four ablations (randomized answer relation, no same-answer control, positive credit allowed, shuffled token positions) each degrade performance in a way consistent with the stated design principles; notably, shuffled positions eliminate all gains over GRPO (18.0% vs 18.2%), confirming the signal is position-specific rather than aggregate.

  3. No external correct references required. Unlike OPSD and CPO which need ground-truth solutions, SibCredit constructs its signal entirely from the current failed group, yet outperforms both on domain-average accuracy in all four settings (Table 1), which is a practically meaningful advantage since correct references are often unavailable for hard problems.

Critical weaknesses
  1. The "different answer ⟹ different error" assumption is unexamined. The method assumes that two rollouts reaching different wrong answers must have diverged due to different reasoning errors, so the divergence point is informative about where errors are. But two rollouts can share the same critical early error (e.g., misreading the problem) yet diverge later for unrelated reasons, producing different final answers. In this case, the method penalizes tokens at the superficial divergence point rather than the actual error. The token-level analysis (Section 4.4, 60 groups / 209 rollouts) is too small to assess how often this occurs at scale.

  2. Add a positive baseline: rejection-sample in all-wrong groups until a correct rollout is obtained, then apply standard self-distillation

Questions for authors
  1. How does the fraction of eligible groups evolve over training?

  2. Why are the code generation gains so small (+0.7–0.8 pp), and can you analyze this further?

1 comment
AuthorsOpenReview ↗

A Gentle Follow-Up

Dear Reviewer sSy7,

Thank you again for your time and constructive feedback, which has been very helpful in improving our work.

As the discussion period nears its end (November 18 AoE), we wanted to leave a brief note to let you know we are following this thread closely. We hope our rebuttal has addressed your initial concerns. Should you have any remaining questions or need further clarification, we would be happy to discuss them further.

Thank you for your continued time and effort!

Best regards,

The Authors

Reviewer_hxvFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper targets reinforcement learning for LLMs when group-relative methods with binary rewards fail because all rollouts for a problem are incorrect, and recent self-distillation methods often require external knowledge. It proposes SibCredit, which enables token-level supervision in all-wrong rollout groups by contrasting the effects of same-answer and different-answer rollouts from the same problem, assuming sufficient token-level learning signal remains. The main experiments use Qwen3-1.7B and Qwen3-4B on four mathematical reasoning and four code generation tasks, comparing against the base model, GRPO, OPSD, RLCSD, CPO, and SibCredit. SibCredit achieves the best average performance in both domains, and the appendix results with SmolLM3-3B provide further validation.

Critical strengths

The paper presents an interesting and somewhat novel idea. The proposed method attempts to recover token-level supervision from all-wrong rollout groups, and the experimental results provide some preliminary evidence that it can work. That said, the underlying mechanism is somewhat counter-intuitive, and the current validation is not yet sufficient to fully establish its effectiveness.

Critical weaknesses
  1. The theoretical assumption and experimental validation are not fully convincing. The effectiveness of SibCredit relies on the assumption that the target and its same-answer sibling share a similar reasoning process. However, identical final answers do not necessarily imply similar reasoning traces. If a same-answer sibling reaches the same answer through a different reasoning process, it may behave more like a different-answer sibling and thus lose its role as a control for filtering common noise. The paper does not adequately address this concern or analyze its potential failure modes. In addition, SibCredit is built on GRPO and only activates when all rollouts are wrong; otherwise, it reduces to GRPO. The main experiments use small fixed group sizes (G=12 for GRPO and G=8 for other methods), making it difficult to determine whether the gains truly come from SibCredit or merely reflect GRPO being disadvantaged by a small group size, since GRPO requires at least one differing reward to produce a nonzero advantage. More experiments with larger models, more domains, and varied group sizes are needed. The SmolLM3-3B results in the appendix are insufficient, as they only compare SibCredit, GRPO, and the base model.
  2. Several hyperparameter choices lack sufficient justification. In Jtrain(θ) = JGRPO(θ) + αJaux(θ), the auxiliary weight α = 0.5 for mathematics and 0.2 for code without explanation or experiment. It remains unclear why different values are used across domains, whether α is domain-sensitive, and why it would not be task-sensitive. The default choices λ=10 and K=2 are also insufficiently supported. For λ, only 1, 3, and 10 are tested, which is too limited to claim that λ=10 is optimal. For K, only 1, 2, and 4 are tested; since 2 and 4 perform similarly, the authors should at least evaluate K=3 before fixing K=2 as the default throughout the paper.
Questions for authors
  1. How to get hyperparameters like λ as well as K via insufficient experiments, and the auxiliary weight α.
  2. Why only used small Qwen model family as the main experiment setting without other models belong to other families or larger models. And why choose only two domains?
2 comments
AuthorsOpenReview ↗

A Gentle Follow-Up

Dear Reviewer hxvF,

Thank you again for your time and constructive feedback, which has been very helpful in improving our work.

As the discussion period nears its end (November 18 AoE), we wanted to leave a brief note to let you know we are following this thread closely. We hope our rebuttal has addressed your initial concerns. Should you have any remaining questions or need further clarification, we would be happy to discuss them further.

Thank you for your continued time and effort!

Best regards,

The Authors

I have reviewed the authors' responses and the revised manuscript. The additional experiments and analyses effectively address my primary concerns. Given these substantial revisions, I am raising my rating to 3: Weak acceptance. I appreciate the authors' thorough and constructive engagement.

Reviewer_6FbUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SibCredit, an auxiliary token-level learning signal for all-wrong rollout groups, where binary-reward GRPO provides no outcome-based policy-gradient signal. For an eligible failed rollout, the method scores its sampled tokens under contexts containing other failed rollouts with the same or different answers. Different-answer opposition supplies candidate negative credit; same-answer opposition acts as a weighted control. The resulting signal is bounded and nonpositive and is added to the GRPO objective only when both sibling types are available. Experiments cover mathematical reasoning and code generation with Qwen3-1.7B and Qwen3-4B, plus a SmolLM3-3B mathematics experiment. Ablations and an offline token analysis investigate the roles of answer relations, calibration, and token-level localization.

Critical strengths
  1. A concrete way to exploit otherwise unused failed groups, with reasonably broad empirical support.

  2. The ablations target the proposed mechanism rather than merely removing the entire auxiliary objective.

Critical weaknesses
  1. The comparative evidence is incomplete for the setting that motivates the paper, and uncertainty is not quantified. The main baselines either require correct privileged context or, as the paper explains for RLCSD, produce no update on all-wrong groups. The manuscript discusses but does not experimentally compare against CAST or Negative Self-Distillation, which are particularly relevant reference-free alternatives: CAST explicitly supplies nonzero feedback to zero-variance groups, while Negative Self-Distillation constructs selective negative supervision without correct references.

  2. The error-specific interpretation remains suggestive rather than fully validated. Identical prompting controls the reference format, but same-answer and different-answer siblings can also differ in semantic similarity, intermediate assumptions, and reasoning strategy. Consequently, their contrast need not isolate incorrect reasoning.

Questions for authors
  1. Are matched-setup results available for CAST, Negative Self-Distillation, or another strong reference-free method that learns from all-wrong groups?

  2. How many rollouts and spans were manually audited for the reported 95% confirmation rate? What fraction of tokens receiving a nonzero final penalty are actual errors?

  3. What precisely is encoded in each component of the test-outcome vector: pass/fail, returned output, or exception status?

1 comment
AuthorsOpenReview ↗

A Gentle Follow-Up

Dear Reviewer 6FbU,

Thank you again for your time and constructive feedback, which has been very helpful in improving our work.

As the discussion period nears its end (November 18 AoE), we wanted to leave a brief note to let you know we are following this thread closely. We hope our rebuttal has addressed your initial concerns. Should you have any remaining questions or need further clarification, we would be happy to discuss them further.

Thank you for your continued time and effort!

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. As the author–reviewer discussion period draws to a close (November 18 AoE), we are posting this general response to recap SibCredit’s core contributions and summarize the additional experiments, analyses, and manuscript revisions made during the rebuttal period.


Core contributions

  • Learning from all-wrong rollout groups. SibCredit recovers token-level supervision from groups where binary-reward GRPO assigns zero group-relative advantage. It uses only the original failed rollouts and requires no correct reference solution. Reviewers 6FbU and hxvF highlighted the potential of learning from these groups, and sSy7 emphasized the reference-free design.
  • Same-answer calibration for token-level credit. SibCredit uses different-answer siblings to propose token penalties and same-answer siblings to filter shared conditioning effects. Reviewer sSy7 highlighted the motivation and evidence for this calibration, and the reviewer also highlighted the token-shuffling ablation as evidence that penalty placement matters for the observed gains.
  • Empirical evaluation. SibCredit improves average accuracy over GRPO on mathematical reasoning and code generation at both model sizes in the main evaluation. Reviewer 6FbU noted the breadth of the evaluation, and reviewers 6FbU and sSy7 highlighted that the ablations test individual design choices.

Updated experiments in the revised manuscript

  • Reference-free baseline (6FbU, Section 4.2 and Appendix D.5). Added a comparison with CAST under matched training and evaluation settings.
  • Additional rollout sampling (hxvF and sSy7, Section 4.2 and Appendix D.6). Evaluated larger-group GRPO and rejection-sampled OPSD to test whether extra sampling can recover supervision for difficult problems.
  • Model scale and family (hxvF, Appendix D.1). Extended the evaluation to Qwen3-8B and added RLCSD as a baseline on SmolLM3-3B.
  • Hyperparameter sensitivity (hxvF, Section 4.5 and Appendix D.4). Expanded the analysis of the auxiliary weight, calibration weight, and number of conditioning siblings.

Additional analyses and clarifications

  • Calibration and error localization (6FbU, hxvF, and sSy7). We analyzed how calibration behaves when same-answer siblings follow different reasoning paths and the error annotation procedure (Section 4.4 and Appendices B.1–B.2). We also discuss why the signal can miss early errors shared by failed rollouts (Conclusion).
  • Other clarifications and analyses (6FbU and sSy7). We clarified the definition of the test-outcome vector (Section 3.2 and Appendix C.1). Our point-by-point replies also discuss how group eligibility evolves during training and possible reasons for the smaller gains in code generation. We also revised the wording throughout the manuscript to improve clarity and readability. Key revisions in the main text are highlighted in blue.

We hope the additional experiments and analyses address the main concerns raised in the reviews and clarify what SibCredit can learn from all-wrong groups. We are grateful for the feedback that guided these revisions and look forward to the reviewers’ and Area Chairs’ consideration of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.