- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Reliability-Aware Latent Solvability Inference (RLSI) for group-relative reinforcement learning with verifiable rewards. It models verifier outcomes as noisy observations of latent response success, using response-specific reliability to estimate prompt solvability through EM. The resulting estimate replaces the group reward mean and normalization scale in GRPO-style advantages. Under the proposed observation model, the analysis explains why reward counts alone can miss reliability information. Experiments on mathematical reasoning benchmarks report improvements over GRPO and other evaluated baselines.
Critical strengths
- Treating prompt solvability as a latent-variable inference problem gives response reliability a direct role in the group baseline rather than using it only to reweight rewards.
- The EM update is simple, and the analysis shows how reliability changes each response's contribution under the assumed observation model while recovering the GRPO baseline when reliability is perfect.
- Results improve over the evaluated baselines across two model scales and six math benchmarks. A reliability-weighted control and a heterogeneity ablation help probe the source of the gains.
Critical weaknesses
- The theoretical observation model interprets the response reliability as the probability that the verifier outcome agrees with the latent success state. However, the experiments instantiate using length-normalized sequence probability without validating this interpretation. Under the proposed E-step, would reverse the evidential meaning of a correct verifier outcome, not merely downweight it.
- RLSI changes both reward centering and advantage normalization relative to GRPO. However, the paper does not separately quantify the contribution of the EM-derived baseline versus the Bernoulli-variance normalization.
- The paper does not study sensitivity to rollout group size. With only responses per prompt, the EM solvability estimate may be noisy.
- The comparison omits uncertainty-aware GRPO methods such as SEED-GRPO[1] and UARM[2], which also use uncertainty to modify advantages.
[1] Chen M, Chen G, Wang W, et al. SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization[J]. arXiv:2505.12346, 2025.
[2] Pan L, Yang H, Li H, et al. Uncertainty-Aware Reward Modeling for Stable RLHF[J]. arXiv:2606.19818, 2026.
Questions for authors
- What is the empirical distribution of , particularly the fraction of scores below , throughout training? What evidence supports interpreting it as , rather than as a generic confidence score?
- Can the authors separately ablate the EM baseline and the new advantage normalization under same training settings?
- Since the online method performs only one EM step initialized from the previous prompt-specific estimate, how sensitive are results to , , and historical warm starts?
8 comments
[W3] Missing group-size sensitivity experiments ().
Thank you for raising this concern. To investigate the sensitivity of RLSI to rollout group size, we conducted an additional ablation study with on Qwen3-1.7B-Base, while keeping the remaining training and evaluation configurations unchanged.
| Method | MATH500 (Avg@1) | Minerva Math (Avg@1) | OlympiadBench (Avg@1) | AIME24 (Avg@32) | AIME25 (Avg@32) | AMC23 (Avg@32) | Average |
|---|---|---|---|---|---|---|---|
| RLSI () | 69.8 | 32.0 | 33.6 | 9.0 | 7.8 | 43.9 | 32.7 |
| RLSI () | 71.2 | 33.5 | 33.0 | 9.6 | 5.2 | 46.1 | 33.1 |
| RLSI () | 73.2 | 32.7 | 36.3 | 9.7 | 10.2 | 47.3 | 34.9 |
As shown in the table, the average performance improves from 32.7 at to 33.1 at and 34.9 at , corresponding to an overall improvement of 2.2 percentage points. The overall trend indicates that RLSI benefits from additional rollouts, which provide more response-level information for solvability estimation.
These results demonstrate that RLSI performs effectively with a relatively small group size of and achieves further improvements as the number of rollouts increases. We have included this additional ablation study in the revised manuscript.
[W4] Missing uncertainty-aware baselines (SEED-GRPO and UARM).
We sincerely appreciate your comments. Following your suggestion, we conducted additional experiments comparing our method with SEED-GRPO on six mathematical reasoning benchmarks using Qwen3-1.7B-Base and Qwen3-4B. As shown in the table below, RLSI consistently outperforms SEED-GRPO across all six benchmarks, further demonstrating the effectiveness of our reliability-aware solvability estimation.
The improvement can be attributed to the different ways the two methods incorporate response information into policy optimization. SEED-GRPO uses semantic entropy to measure prompt-level uncertainty and adjusts the magnitude of policy updates accordingly, while retaining the empirical reward mean as the baseline. In contrast, RLSI uses response-specific reliability to infer latent prompt solvability through EM, providing a more informative baseline for advantage construction. This allows RLSI to distinguish response groups with identical empirical accuracy but different reliability patterns, which cannot be captured by uncertainty-based advantage scaling alone. We have also added discussions of SEED-GRPO and UARM in the Related Work section of the revised paper.
Qwen3-1.7B-Base
| Method | MATH500 (Avg@1) | Minerva Math (Avg@1) | OlympiadBench (Avg@1) | AIME24 (Avg@32) | AIME25 (Avg@32) | AMC23 (Avg@32) |
|---|---|---|---|---|---|---|
| SEED-GRPO | 68.6 | 28.7 | 30.1 | 7.9 | 5.6 | 38.6 |
| RLSI | 69.8 | 32.0 | 33.6 | 9.0 | 7.8 | 43.9 |
Qwen3-4B
| Method | MATH500 (Avg@1) | Minerva Math (Avg@1) | OlympiadBench (Avg@1) | AIME24 (Avg@32) | AIME25 (Avg@32) | AMC23 (Avg@32) |
|---|---|---|---|---|---|---|
| SEED-GRPO | 86.8 | 45.6 | 55.3 | 29.0 | 27.8 | 73.9 |
| RLSI | 88.2 | 46.3 | 58.8 | 32.9 | 29.4 | 79.5 |
[Q1] What is the distribution of , especially the proportion below ? What evidence supports interpreting it as ?
Thank you for raising these questions. We have added a new Figure 3 to the revised manuscript to illustrate the empirical distribution of response reliability scores during training. As shown in Figure 3, 11.6% of the scores fall below 0.5, while the remaining 88.4% are above or equal to 0.5. Furthermore, 80.1% of the scores fall within the range , indicating that the distribution is concentrated toward higher confidence values. For , the observation model introduces a signed correction to the latent-state posterior, as characterized in Theorem 4.4, rather than changing the original verifier reward.
Regarding the interpretation of response reliability, our formulation follows the general principle of latent-variable inference from imperfect observations. Classical EM-based approaches, such as Dawid and Skene (1979), model the relationship between observed labels and unobserved true states through observation-specific error probabilities [1]. Related work has further demonstrated that confidence scores can be incorporated into probabilistic models to improve latent-label inference (Oyama et al., 2013) [2]. These studies motivate our use of response-specific reliability to characterize how observed verifier rewards inform latent solvability states.
In our setting, the underlying agreement probability is not directly observable. We therefore use length-normalized sequence probability as a practical proxy. Both sequence confidence and response reliability reflect the strength of evidence associated with an individual response: a response assigned higher probability by the policy is more strongly supported by its generation distribution, and its verifier outcome may therefore provide more informative evidence about latent solvability. Sequence likelihood has also been widely adopted as a confidence measure in language generation [3, 4].
Under our observation model, is interpreted as the agreement probability between the observed reward and the latent state, while its implementation through sequence confidence is an approximation rather than an independently calibrated probability. The consistent improvements over GRPO and other baselines provide empirical support for the usefulness of this proxy in solvability estimation.
We have added the empirical reliability distribution and will further clarify the probabilistic interpretation and motivation for using sequence confidence in the revised manuscript.
[Q3] How sensitive is the method to , , and historical warm starts?
Thank you for your question. Both and are associated with the offline initialization phase, where specifies the initial solvability estimate and denotes the number of EM iterations. In our implementation, we run offline EM until convergence rather than using a fixed number of iterations. To evaluate the effect of offline initialization, we conducted an additional ablation experiment by removing this phase and setting the initial solvability estimate to on Qwen3-1.7B-Base. As shown in the table below, offline initialization consistently improves performance across all six benchmarks, demonstrating the benefit of providing a more informative starting point for subsequent online EM updates.
| Method | MATH500Avg@1 | Minerva MathAvg@1 | OlympiadBenchAvg@1 | AIME24Avg@32 | AIME25Avg@32 | AMC23Avg@32 |
|---|---|---|---|---|---|---|
| RLSI w/o Offline Initialization | 67.0 | 28.3 | 27.9 | 5.0 | 2.1 | 33.2 |
| RLSI | 69.8 | 32.0 | 33.6 | 9.0 | 7.8 | 43.9 |
[W1] Unvalidated interpretation of cij as reliability, especially when cij<0.5.
Thank you for raising this important concern. In our latent-variable model, response reliability characterizes the probability that an observed verifier outcome agrees with the corresponding latent success state, following the general principle of EM-based inference from imperfect observations [1, 2]. Since this agreement probability is not directly observable, we use length-normalized sequence probability as a practical proxy. The underlying intuition is that both confidence and reliability reflect how informative an individual response is about the model's underlying ability to solve a prompt. A response assigned higher probability by the policy is more strongly supported by the model's own generation distribution, making its verifier outcome potentially more informative about latent solvability. Length normalization further reduces the influence of response length, and sequence likelihood has been widely used as a confidence or uncertainty measure in language generation [3, 4]. Although sequence confidence is not necessarily a calibrated agreement probability, our consistent improvements over GRPO and other baselines demonstrate its usefulness for reliability-aware solvability estimation.
Regarding cij<0.5, Theorem 4.4 characterizes the effect of reliability through the following posterior log-odds decomposition:
logit(γij(k))=logit(qi(k))+(2Rij−1)logit(cij)
When cij<0.5, a positive verifier outcome introduces a negative correction to the posterior log-odds, reflecting the signed evidence interpretation of our observation model. However, this correction is relative to the current solvability prior and does not necessarily imply a low posterior probability of latent success. More importantly, the original binary verifier reward remains unchanged in our advantage construction. Correct responses continue to receive positive advantages, while incorrect responses receive negative advantages. Therefore, the signed reliability correction affects the estimation of prompt solvability rather than reversing the reward signal used for policy optimization.
We will further clarify the motivation for using sequence confidence as a reliability proxy, its potential estimation errors, and the interpretation of cij<0.5 in the revised manuscript.
[1] Maximum likelihood from incomplete data via the EM algorithm (Dempster et al., 1977)
[2] Maximum likelihood estimation of observer error rates using the EM algorithm (Dawid & Skene, 1979)
[3] LM-Polygraph: Uncertainty estimation for language models (Fadeeva et al., 2023)
[4] Contextualized sequence likelihood: Enhanced confidence scores for natural language generation (Lin et al., 2024)
[W2&Q2] Missing separate ablation for Bernoulli-variance normalization.
Thank you for this comment. To isolate the contribution of EM-based centering, we conduct an additional EM Center Only ablation, which replaces the empirical group mean with the EM-estimated prompt solvability qi, while retaining GRPO's original variance normalization. Specifically, the advantage is computed as
AijEM-Center=Rˉi(1−Rˉi)+ϵRij−qi,Rˉi=G1∑j=1GRij.
As shown in the table, EM Center Only outperforms GRPO on five of six benchmarks, improving the average score from 27.5 to 28.6 (+1.1 points). Since both methods use the same variance normalization, these results demonstrate that EM-based centering can improve performance independently of the proposed normalization. We have included this additional ablation in Table 6 of the revised manuscript.