- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the RLVR setting for training LLMs. Instead of studying improvements to the reliability of verifiers, it frames the verifier as fundamentally noisy, which allows to study current training methods' robustness. The paper proposes two methods with reasonable assumptions how to obtain a good training signal, with theoretical grounding. Empirical results on synthetic FP/FN rates support the claims.
Critical strengths
- A new perspective to RLVR -- instead of improving verifiers, our training methods can be made more robust against imperfect verifiers. A synergy of improved verifiers and improved training methods may be the right direction for RLVR.
- Theoretical results showing the efectiveness of the two methods under reasonable assumptions, and Table 2 showing what happens when assumptions break, i.e., the sensitivity of the method to the FP/FN rates. It illustrates the methods are effective both in theory and in simulated practice.
Critical weaknesses
- I was expecting to see results of the full pipeline: what happens when the verifier noise is unknown? How much possible it is to estimate & ? Can the proposed training method with these estimates perform well? The method is well motivated and supported by theory, although a full end-to-end check is missing. Perhaps some of these questions can already be answered based on previous work?
Questions for authors
- Figure 4: are there any variance / min-max values / confidence intervals for these results? Do they overlap? Are these the average result across all benchmarks?
- Did the authors identify any limitations of their method?