- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors address the problem that in RLVR with binary rewards, all-wrong rollout groups produce zero advantage and thus no learning signal. They introduce SibCredit, which extracts token-level negative supervision from disagreement among failed siblings: different-answer siblings provide candidate opposition (tokens whose probability drops under sibling conditioning), while same-answer siblings serve as a control to filter shared conditioning effects (e.g., suppression of epistemic expressions). The method is purely negative (no positive credit from wrong siblings), bounded in [-1, 0], and added as an auxiliary objective to GRPO. Experiments on Qwen3-1.7B and Qwen3-4B across math reasoning and code generation show consistent improvements over GRPO (+2.3/+1.7 pp math, +0.8/+0.7 pp code), outperforming self-distillation baselines that require external correct references.
Critical strengths
-
Same-answer calibration is well-motivated and well-validated. The core insight that different-answer opposition conflates error-related signal with shared conditioning effects (e.g., style shifts, epistemic suppression) is demonstrated both through ablation (removing calibration drops accuracy to 14.6%, below the 17.0% base model, Table 2) and through token-level analysis showing calibration retains 55% of opposition on error tokens vs. 10% on epistemic expressions.
-
Ablations isolate each design choice convincingly. Four ablations (randomized answer relation, no same-answer control, positive credit allowed, shuffled token positions) each degrade performance in a way consistent with the stated design principles; notably, shuffled positions eliminate all gains over GRPO (18.0% vs 18.2%), confirming the signal is position-specific rather than aggregate.
-
No external correct references required. Unlike OPSD and CPO which need ground-truth solutions, SibCredit constructs its signal entirely from the current failed group, yet outperforms both on domain-average accuracy in all four settings (Table 1), which is a practically meaningful advantage since correct references are often unavailable for hard problems.
Critical weaknesses
-
The "different answer ⟹ different error" assumption is unexamined. The method assumes that two rollouts reaching different wrong answers must have diverged due to different reasoning errors, so the divergence point is informative about where errors are. But two rollouts can share the same critical early error (e.g., misreading the problem) yet diverge later for unrelated reasons, producing different final answers. In this case, the method penalizes tokens at the superficial divergence point rather than the actual error. The token-level analysis (Section 4.4, 60 groups / 209 rollouts) is too small to assess how often this occurs at scale.
-
Add a positive baseline: rejection-sample in all-wrong groups until a correct rollout is obtained, then apply standard self-distillation
Questions for authors
-
How does the fraction of eligible groups evolve over training?
-
Why are the code generation gains so small (+0.7–0.8 pp), and can you analyze this further?
A Gentle Follow-Up
Dear Reviewer sSy7,
Thank you again for your time and constructive feedback, which has been very helpful in improving our work.
As the discussion period nears its end (November 18 AoE), we wanted to leave a brief note to let you know we are following this thread closely. We hope our rebuttal has addressed your initial concerns. Should you have any remaining questions or need further clarification, we would be happy to discuss them further.
Thank you for your continued time and effort!
Best regards,
The Authors