acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Group Accuracy: Latent-Reliability Inference for Prompt Solvability in RLVR

Abstract

In reinforcement learning with verifiable rewards (RLVR), group relative policy optimization (GRPO) constructs relative advantages by centering response rewards with the empirical group-average reward, a finite-sample estimate of prompt solvability under the current LLM. However, empirical group accuracy depends only on the success count and treats equally rewarded responses as equally informative. Under heterogeneous response reliability, this can misestimate prompt solvability and miscalibrate relative advantages. Consistent with this concern, we empirically observe substantial reliability variation among prompts with identical group accuracy, revealing that the same success count can mask markedly different response-level reliability patterns. In this work, we formulate prompt-solvability estimation as a latent-variable inference problem and introduce Reliability-Aware Latent Solvability Inference (RLSI). Our approach introduces a latent response-level success state and relates each verifier outcome to this state through its response-specific reliability. Under this latent-variable model, we derive an efficient expectation–maximization (EM) procedure to estimate prompt solvability and use the resulting estimate for advantage construction. Theoretically, we show why reward-only aggregation can fail to recover prompt solvability, while response-level reward–reliability pairings retain information relevant to latent-state inference; we further characterize the resulting reliability-dependent corrections and recover the GRPO group mean in the perfect-reliability limit. Across six mathematical reasoning benchmarks and two model scales, RLSI achieves the strongest average performance among the evaluated methods, outperforming GRPO by 5.2 and 4.8 points on Qwen3-1.7B-Base and Qwen3-4B, respectively. Additional comparisons and ablations further support the effectiveness of latent solvability inference and the value of response-level reliability heterogeneity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_c6FjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reliability-Aware Latent Solvability Inference (RLSI) for group-relative reinforcement learning with verifiable rewards. It models verifier outcomes as noisy observations of latent response success, using response-specific reliability to estimate prompt solvability through EM. The resulting estimate replaces the group reward mean and normalization scale in GRPO-style advantages. Under the proposed observation model, the analysis explains why reward counts alone can miss reliability information. Experiments on mathematical reasoning benchmarks report improvements over GRPO and other evaluated baselines.

Critical strengths
  • Treating prompt solvability as a latent-variable inference problem gives response reliability a direct role in the group baseline rather than using it only to reweight rewards.
  • The EM update is simple, and the analysis shows how reliability changes each response's contribution under the assumed observation model while recovering the GRPO baseline when reliability is perfect.
  • Results improve over the evaluated baselines across two model scales and six math benchmarks. A reliability-weighted control and a heterogeneity ablation help probe the source of the gains.
Critical weaknesses
  • The theoretical observation model interprets the response reliability cijc_{ij} as the probability that the verifier outcome agrees with the latent success state. However, the experiments instantiate cijc_{ij} using length-normalized sequence probability without validating this interpretation. Under the proposed E-step, cij<0.5c_{ij}<0.5 would reverse the evidential meaning of a correct verifier outcome, not merely downweight it.
  • RLSI changes both reward centering and advantage normalization relative to GRPO. However, the paper does not separately quantify the contribution of the EM-derived baseline versus the Bernoulli-variance normalization.
  • The paper does not study sensitivity to rollout group size. With only G=4G=4 responses per prompt, the EM solvability estimate may be noisy.
  • The comparison omits uncertainty-aware GRPO methods such as SEED-GRPO[1] and UARM[2], which also use uncertainty to modify advantages.

[1] Chen M, Chen G, Wang W, et al. SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization[J]. arXiv:2505.12346, 2025.

[2] Pan L, Yang H, Li H, et al. Uncertainty-Aware Reward Modeling for Stable RLHF[J]. arXiv:2606.19818, 2026.

Questions for authors
  • What is the empirical distribution of cijc_{ij}, particularly the fraction of scores below 0.50.5, throughout training? What evidence supports interpreting it as P(Rij=Zij∣cij)P(R_{ij}=Z_{ij}\mid c_{ij}), rather than as a generic confidence score?
  • Can the authors separately ablate the EM baseline and the new advantage normalization under same training settings?
  • Since the online method performs only one EM step initialized from the previous prompt-specific estimate, how sensitive are results to q0q_0, KoffK_{\mathrm{off}}, and historical warm starts?
8 comments

[W1] Unvalidated interpretation of cijc_{ij} as reliability, especially when cij<0.5c_{ij}<0.5.

Thank you for raising this important concern. In our latent-variable model, response reliability characterizes the probability that an observed verifier outcome agrees with the corresponding latent success state, following the general principle of EM-based inference from imperfect observations [1, 2]. Since this agreement probability is not directly observable, we use length-normalized sequence probability as a practical proxy. The underlying intuition is that both confidence and reliability reflect how informative an individual response is about the model's underlying ability to solve a prompt. A response assigned higher probability by the policy is more strongly supported by the model's own generation distribution, making its verifier outcome potentially more informative about latent solvability. Length normalization further reduces the influence of response length, and sequence likelihood has been widely used as a confidence or uncertainty measure in language generation [3, 4]. Although sequence confidence is not necessarily a calibrated agreement probability, our consistent improvements over GRPO and other baselines demonstrate its usefulness for reliability-aware solvability estimation.

Regarding cij<0.5c_{ij}<0.5, Theorem 4.4 characterizes the effect of reliability through the following posterior log-odds decomposition:

logit⁡(γij(k))=logit⁡(qi(k))+(2Rij−1)logit⁡(cij)\operatorname{logit}(\gamma_{ij}^{(k)}) = \operatorname{logit}(q_i^{(k)}) + (2R_{ij}-1)\operatorname{logit}(c_{ij})

When cij<0.5c_{ij}<0.5, a positive verifier outcome introduces a negative correction to the posterior log-odds, reflecting the signed evidence interpretation of our observation model. However, this correction is relative to the current solvability prior and does not necessarily imply a low posterior probability of latent success. More importantly, the original binary verifier reward remains unchanged in our advantage construction. Correct responses continue to receive positive advantages, while incorrect responses receive negative advantages. Therefore, the signed reliability correction affects the estimation of prompt solvability rather than reversing the reward signal used for policy optimization.

We will further clarify the motivation for using sequence confidence as a reliability proxy, its potential estimation errors, and the interpretation of cij<0.5c_{ij}<0.5 in the revised manuscript.

[1] Maximum likelihood from incomplete data via the EM algorithm (Dempster et al., 1977)

[2] Maximum likelihood estimation of observer error rates using the EM algorithm (Dawid & Skene, 1979)

[3] LM-Polygraph: Uncertainty estimation for language models (Fadeeva et al., 2023)

[4] Contextualized sequence likelihood: Enhanced confidence scores for natural language generation (Lin et al., 2024)

[W2&Q2] Missing separate ablation for Bernoulli-variance normalization.

Thank you for this comment. To isolate the contribution of EM-based centering, we conduct an additional EM Center Only ablation, which replaces the empirical group mean with the EM-estimated prompt solvability qiq_i, while retaining GRPO's original variance normalization. Specifically, the advantage is computed as

AijEM-Center=Rij−qiRˉi(1−Rˉi)+ϵ,Rˉi=1G∑j=1GRij.A_{ij}^{\mathrm{EM\text{-}Center}} = \frac{R_{ij}-q_i}{\sqrt{\bar{R}_i(1-\bar{R}_i)+\epsilon}}, \qquad \bar{R}_i = \frac{1}{G}\sum_{j=1}^{G}R_{ij}.

MethodMATH500 (Avg@1)Minerva Math (Avg@1)OlympiadBench (Avg@1)AIME24 (Avg@32)AIME25 (Avg@32)AMC23 (Avg@32)Average
GRPO64.226.827.76.04.236.327.5
EM Center Only65.229.828.17.53.038.028.6

As shown in the table, EM Center Only outperforms GRPO on five of six benchmarks, improving the average score from 27.5 to 28.6 (+1.1 points). Since both methods use the same variance normalization, these results demonstrate that EM-based centering can improve performance independently of the proposed normalization. We have included this additional ablation in Table 6 of the revised manuscript.

[W3] Missing group-size sensitivity experiments (G=4G=4).

Thank you for raising this concern. To investigate the sensitivity of RLSI to rollout group size, we conducted an additional ablation study with G∈{4,8,16}G\in\{4,8,16\} on Qwen3-1.7B-Base, while keeping the remaining training and evaluation configurations unchanged.

MethodMATH500 (Avg@1)Minerva Math (Avg@1)OlympiadBench (Avg@1)AIME24 (Avg@32)AIME25 (Avg@32)AMC23 (Avg@32)Average
RLSI (G=4G=4)69.832.033.69.07.843.932.7
RLSI (G=8G=8)71.233.533.09.65.246.133.1
RLSI (G=16G=16)73.232.736.39.710.247.334.9

As shown in the table, the average performance improves from 32.7 at G=4G=4 to 33.1 at G=8G=8 and 34.9 at G=16G=16, corresponding to an overall improvement of 2.2 percentage points. The overall trend indicates that RLSI benefits from additional rollouts, which provide more response-level information for solvability estimation.

These results demonstrate that RLSI performs effectively with a relatively small group size of G=4G=4 and achieves further improvements as the number of rollouts increases. We have included this additional ablation study in the revised manuscript.

[W4] Missing uncertainty-aware baselines (SEED-GRPO and UARM).

We sincerely appreciate your comments. Following your suggestion, we conducted additional experiments comparing our method with SEED-GRPO on six mathematical reasoning benchmarks using Qwen3-1.7B-Base and Qwen3-4B. As shown in the table below, RLSI consistently outperforms SEED-GRPO across all six benchmarks, further demonstrating the effectiveness of our reliability-aware solvability estimation.

The improvement can be attributed to the different ways the two methods incorporate response information into policy optimization. SEED-GRPO uses semantic entropy to measure prompt-level uncertainty and adjusts the magnitude of policy updates accordingly, while retaining the empirical reward mean as the baseline. In contrast, RLSI uses response-specific reliability to infer latent prompt solvability through EM, providing a more informative baseline for advantage construction. This allows RLSI to distinguish response groups with identical empirical accuracy but different reliability patterns, which cannot be captured by uncertainty-based advantage scaling alone. We have also added discussions of SEED-GRPO and UARM in the Related Work section of the revised paper.

Qwen3-1.7B-Base

MethodMATH500 (Avg@1)Minerva Math (Avg@1)OlympiadBench (Avg@1)AIME24 (Avg@32)AIME25 (Avg@32)AMC23 (Avg@32)
SEED-GRPO68.628.730.17.95.638.6
RLSI69.832.033.69.07.843.9

Qwen3-4B

MethodMATH500 (Avg@1)Minerva Math (Avg@1)OlympiadBench (Avg@1)AIME24 (Avg@32)AIME25 (Avg@32)AMC23 (Avg@32)
SEED-GRPO86.845.655.329.027.873.9
RLSI88.246.358.832.929.479.5

[Q1] What is the distribution of cijc_{ij}, especially the proportion below 0.50.5? What evidence supports interpreting it as P(Rij=Zij∣cij)P(R_{ij}=Z_{ij}|c_{ij})?

Thank you for raising these questions. We have added a new Figure 3 to the revised manuscript to illustrate the empirical distribution of response reliability scores cijc_{ij} during training. As shown in Figure 3, 11.6% of the scores fall below 0.5, while the remaining 88.4% are above or equal to 0.5. Furthermore, 80.1% of the scores fall within the range [0.6,1][0.6,1], indicating that the distribution is concentrated toward higher confidence values. For cij<0.5c_{ij}<0.5, the observation model introduces a signed correction to the latent-state posterior, as characterized in Theorem 4.4, rather than changing the original verifier reward.

Regarding the interpretation of response reliability, our formulation follows the general principle of latent-variable inference from imperfect observations. Classical EM-based approaches, such as Dawid and Skene (1979), model the relationship between observed labels and unobserved true states through observation-specific error probabilities [1]. Related work has further demonstrated that confidence scores can be incorporated into probabilistic models to improve latent-label inference (Oyama et al., 2013) [2]. These studies motivate our use of response-specific reliability to characterize how observed verifier rewards inform latent solvability states.

In our setting, the underlying agreement probability is not directly observable. We therefore use length-normalized sequence probability as a practical proxy. Both sequence confidence and response reliability reflect the strength of evidence associated with an individual response: a response assigned higher probability by the policy is more strongly supported by its generation distribution, and its verifier outcome may therefore provide more informative evidence about latent solvability. Sequence likelihood has also been widely adopted as a confidence measure in language generation [3, 4].

Under our observation model, cijc_{ij} is interpreted as the agreement probability between the observed reward and the latent state, while its implementation through sequence confidence is an approximation rather than an independently calibrated probability. The consistent improvements over GRPO and other baselines provide empirical support for the usefulness of this proxy in solvability estimation.

We have added the empirical reliability distribution and will further clarify the probabilistic interpretation and motivation for using sequence confidence in the revised manuscript.

[Q3] How sensitive is the method to q0q_0, KoffK_{\mathrm{off}}, and historical warm starts?

Thank you for your question. Both q0q_0 and KoffK_{\mathrm{off}} are associated with the offline initialization phase, where q0q_0 specifies the initial solvability estimate and KoffK_{\mathrm{off}} denotes the number of EM iterations. In our implementation, we run offline EM until convergence rather than using a fixed number of iterations. To evaluate the effect of offline initialization, we conducted an additional ablation experiment by removing this phase and setting the initial solvability estimate to qi(0)=0.5q_i^{(0)}=0.5 on Qwen3-1.7B-Base. As shown in the table below, offline initialization consistently improves performance across all six benchmarks, demonstrating the benefit of providing a more informative starting point for subsequent online EM updates.

MethodMATH500Avg@1Minerva MathAvg@1OlympiadBenchAvg@1AIME24Avg@32AIME25Avg@32AMC23Avg@32
RLSI w/o Offline Initialization67.028.327.95.02.133.2
RLSI69.832.033.69.07.843.9
Reviewer_WjkbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new way of using empirical estimates of group mean in algorithms such as GRPO. They argue that the group mean doesn't accurately determine 'prompt solvability' and to improve this estimation, they use a latent variable model which is updated using the rewards for the generated rollouts, using an EM algorithm. The authors conduct experiments on Qwen3-1.7B-Base and Qwen3-4B and demonstrate improvements over DAPO, PPO and GRPO.

Critical strengths
  1. Implementation is simple: The algorithm is easy to implement and doesn't impose any significant overheads to the training process. The EM update is cheap to run and only requires the log-probs of the generated rollouts and the collected rewards.
  2. Novel framing: The framing of the EM algorithm to update the latent model in the context of RLVR is interesting and novel in the reviewer's experience and the theoretical analysis is interesting.
Critical weaknesses

I think the paper has a couple of fixable issues:

  1. Improved presentation: The paper uses 'prompt solvability' and 'response reliability' but these terms are not very well defined explicitly in the paper and not part of the common vocabulary in the field. Therefore, the authors should define such terms before going on to make mathematical claims about them or using them in the introduction. It would also be good if the authors could make a small figure explicitly drawing the latent model including the variables and how they are sampled.
  2. Discussion around why the method improves: The method proposes to use qi(t)q_i^{(t)} in GRPO instead of the empirical mean. The authors do not have a discussion on why the method improves over just using the empirical mean. Is it because qi(t)q_i^{(t)} is lower variance than the empirical mean? What about the bias introduced by the EM update? There needs to be more discussion around this subject because it looks like this is the core of the paper's results. This is the primary weakness of the paper.
  3. Some experimental design weaknesses: The authors use the Qwen3-4B model, is this the thinking version of the model? If so, the performance numbers for datasets like MATH500 and AIME are too lower compared to the officially reported scores. Could the authors elaborate why that is the case? Also, avg@1 evaluations should not be computed at temperature 0 as done for the MATH500 or Minerva dataset. Ideally, one should still do more rollouts at a higher temperature=0.6 with top_p=0.95 and then estimate the mean from those rollouts since the variance at temperature=0 is high and has artifacts like repetitions in the response.
Questions for authors

Could the authors elaborate more on why the method actually improves over using the empirical mean? Also how do they forecast the method's performance when the number of rollouts increase and the variance of mean estimation goes down.

5 comments

[W1] Unclear definitions of prompt solvability and response reliability; a graphical model is suggested.

We sincerely appreciate your suggestion. We have added Figure 2 to the revised paper to illustrate the latent-variable model and clarify the definitions of prompt solvability and response reliability.

Specifically, prompt solvability refers to the probability that a response sampled from the current policy is in a latent solvable state for a given prompt. All responses generated for the same prompt share this prompt-level solvability, while each response has its own latent solvability state. Response reliability characterizes how faithfully the observed verifier outcome reflects the corresponding latent state. Under our observation model, it specifies the probability that the observed reward agrees with the latent state. Therefore, responses with different reliability scores provide different strengths of evidence about latent solvability, even when they receive the same verifier reward.

Let's use math reasoning as an example. Consider two math reasoning prompts, each with two correct responses out of four rollouts. Although both have the same empirical group accuracy of 0.5, the reliability of their observed outcomes may differ. For one prompt, the correct outcomes may provide stronger evidence about the underlying latent states than the incorrect outcomes, while the opposite may hold for the other prompt. Our method accounts for these differences when estimating prompt-level solvability, allowing the two prompts to receive different estimates despite identical group accuracy.

Following your suggestion, we have further clarified these concepts and their relationships in the revised paper.

[W2] Insufficient explanation of why EM improves over the empirical mean, including bias and variance.

We thank the reviewer for this insightful comment! The key limitation of empirical reward averaging is that it treats responses with the same verifier outcome as equally informative, regardless of their reliability. Our method addresses this limitation by incorporating response-specific reliability into posterior inference. We clarify its benefits in terms of bias, information retention, and variance.

First, empirical reward averaging introduces systematic bias under the latent observation model. Proposition 4.1 establishes that

=(1-\bar c_i)(1-2q_i),$$ where $\bar c_i=\frac{1}{G}\sum_{j=1}^{G}c_{ij}$. This motivates inferring latent solvability from reward–reliability pairs rather than directly averaging observed rewards. Second, EM preserves response-specific information through posterior inference. Theorem 4.2 establishes that $$P(\mathbf Z_i=\mathbf z\mid\mathbf R_i,\mathbf c_i,q_i^{(k)}) =\prod_{j=1}^{G} (\gamma_{ij}^{(k)})^{z_j} (1-\gamma_{ij}^{(k)})^{1-z_j}.$$ Furthermore, Corollary 4.3 shows that $$I(S_i;\boldsymbol\gamma_i^{(k)}\mid\mathbf c_i,q_i^{(k)}) =I(S_i;\mathbf R_i\mid\mathbf c_i,q_i^{(k)}) \geq I(S_i;\bar R_i\mid\mathbf c_i,q_i^{(k)}),$$ where $S_i=\sum_{j=1}^{G}Z_{ij}$. Thus, EM retains reliability-dependent information about latent success that can be lost through empirical reward averaging. Third, posterior aggregation also has favorable bias–variance properties. Let $\widetilde q_i=\frac{1}{G}\sum_{j=1}^{G}\gamma_{ij}(q_i)$ denote the posterior mean evaluated at the true solvability prior. By the law of total expectation, $$\mathbb{E}[\widetilde q_i\mid q_i,\mathbf c_i]=q_i.$$ Moreover, since $\gamma_{ij}(q_i)$ is an affine function of the binary reward $R_{ij}$ with slope bounded in absolute value by one, conditional independence yields $$\operatorname{Var}\left(\widetilde{q}_i \mid q_i,\mathbf{c}_i\right) \leq \operatorname{Var}\left(\bar{R}_i \mid q_i,\mathbf{c}_i\right)$$ According to the above results, we can find that posterior aggregation at the true prior is conditionally unbiased and has a variance no greater than that of empirical reward averaging. These results provide theoretical motivation for using EM to estimate prompt solvability. By incorporating response-specific reliability, RLSI can retain information that is lost when only the empirical group mean is used. The experimental improvements over GRPO further support the effectiveness of this approach.

[W3] Qwen3-4B thinking mode, low benchmark scores, and concerns about evaluation at T=0T=0.

Thank you for this comment. We clarify that we use the Qwen3-4B model in thinking mode, rather than the non-thinking version.

Regarding the lower absolute performance, a key difference from the official evaluation is that we limit the maximum generated response length to 3K tokens, which is substantially shorter than the generation budgets used in the official Qwen3 evaluations.

Regarding evaluation fairness, our goal is to evaluate the effectiveness of the proposed solvability estimation method under a controlled RLVR setting, rather than maximize the absolute performance of Qwen3. All methods use identical model initialization, training configurations, generation budgets, and evaluation protocols. Therefore, the reported improvements reflect comparisons under the same experimental conditions.

Furthermore, following your suggestion, we conducted additional evaluations using Qwen3-4B on MATH500, Minerva Math, and OlympiadBench with temperature = 0.6 and top_p = 0.95, sampling four responses per problem (Avg@4). All methods were evaluated under the same settings, with the results shown below.

MethodMATH500 (Avg@4)Minerva Math (Avg@4)OlympiadBench (Avg@4)
Qwen3-4B73.335.837.1
PPO78.538.741.1
GRPO88.943.555.3
DAPO88.344.359.2
RW-GRPO84.341.049.6
RLSI90.047.362.1

As shown in the table, RLSI still achieves the highest performance across all three benchmarks among the methods with reported results. These additional evaluations further support our original findings under the suggested sampling configuration. We agree that longer generation budgets and additional rollouts could provide a more comprehensive assessment of absolute model performance. Given our computational budget, we prioritize controlled comparisons across methods. Nevertheless, the consistent improvements under both evaluation settings support the effectiveness of RLSI.

[Q1] Why does EM improve over the empirical mean?

Thank you for this question. The key motivation of our method is to estimate the underlying prompt solvability rather than directly using the empirical reward frequency. While GRPO estimates prompt solvability by averaging binary rewards, our method incorporates response-specific reliability through EM to infer the latent solvability states.

[Q2] How does performance change as the number of rollouts increases?

Thank you for this question. To investigate how RLSI performs as the number of rollouts increases, we conduct an additional ablation study with group sizes G∈4,8,16G\in{4,8,16} on Qwen3-1.7B-Base.

As shown in the table below, the average performance improves from 32.7 at G=4G=4 to 33.1 at G=8G=8 and 34.9 at G=16G=16, corresponding to an improvement of 2.2 percentage points. These results demonstrate that RLSI continues to benefit from larger rollout groups.

From a theoretical perspective, increasing the group size reduces the sampling variance of empirical group accuracy. However, as established in Proposition 4.1, empirical reward averaging remains generally biased under heterogeneous response reliability. In contrast, RLSI incorporates additional response-level reward–reliability information through posterior inference before aggregation. Larger groups therefore provide more observations for estimating latent prompt solvability.

Overall, these results support the scalability of RLSI with increasing rollout group size and demonstrate its ability to leverage additional response-level information for improved policy optimization. We have updated the results in manuscript Table 5.

MethodMATH500 (Avg@1)Minerva Math (Avg@1)OlympiadBench (Avg@1)AIME24 (Avg@32)AIME25 (Avg@32)AMC23 (Avg@32)Average
RLSI (G=4)69.832.033.69.07.843.932.7
RLSI (G=8)71.233.533.09.65.246.133.1
RLSI (G=16)73.232.736.39.710.247.334.9

Thanks for the response.

'First, empirical reward averaging introduces systematic bias under the latent observation model.'

This doesn't address what I asked for. What I'm saying is: imagine you could estimate the true success probability with a large number of rollouts (say 1024). Then, does your estimator have better bias/variance tradeoffs compared to the empirical mean (using 4 rollouts)? If not, then it is unclear what the gains are coming from.

In general, the paper could benefit from a round of revisions, particularly in terms of presentation and giving some intuition about why the latent model is helping. I will retain my score.

Reviewer_sDerOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Reliability-Aware Latent Solvability Inference (RLSI) to incorporate response-specific reliability into GRPO’s baseline estimation. The method introduces a latent success state for each response and uses reliability to connect this state to the observed verifier reward. An EM procedure computes posterior success probabilities and averages them to update prompt solvability, which determines the shared baseline and advantage normalization. Across six mathematical reasoning benchmarks, RLSI achieves the highest average performance among the evaluated methods. Comparisons with reliability-weighted GRPO and a homogeneous-reliability ablation further support the benefits of latent inference and retaining reliability differences across responses.

Critical strengths
  1. The paper addresses GRPO’s reliance on success counts for baseline estimation by incorporating response-specific reliability through a latent-variable model. This allows advantage construction to account for information beyond binary rewards.

  2. RLSI consistently improves over GRPO across six mathematical reasoning benchmarks and two model scales. Comparisons with RW-GRPO and the homogeneous-reliability ablation further support the benefits of latent inference and retaining reliability differences across responses.

Critical weaknesses

Only a few points require further clarification, as outlined in the questions below.

Questions for authors
  1. Could the authors compare RLSI with and without offline initialization, starting from (qi=0.5q_i=0.5) in the latter case, to clarify its contribution to performance?

  2. Could the authors explain the rationale for using length-normalized sequence probability as a proxy for the agreement probability in the observation model, and discuss how inaccuracies in this proxy might affect solvability estimation?

3 comments

[Q1] Compare with and without offline initialization (qi=0.5q_i=0.5).

Thank you for raising this question. To isolate the effect of offline initialization, we conducted an additional ablation experiment by removing the offline initialization phase and starting from qi=0.5q_i=0.5. As shown in the table, RLSI with offline initialization consistently outperforms the variant without it across all six mathematical reasoning benchmarks. These results demonstrate the effectiveness of offline initialization in providing a more informative initial solvability estimate for subsequent online EM updates. We have updated this in the manuscript, Table 4.

MethodMATH500 Avg@1Minerva Math Avg@1OlympiadBench Avg@1AIME24 Avg@32*AIME25 Avg@32AMC23 Avg@32
RLSI w/o Offline Initialization67.028.327.95.02.133.2
RLSI69.832.033.69.07.843.9

[Q2] Justify using sequence confidence as reliability and discuss the effect of estimation errors.

Thank you for pointing this out. Theoretically, our latent-variable model requires response-specific reliability to characterize the agreement between observed rewards and latent states, following the general principle of EM-based inference from imperfect observations [1, 2]. In practice, since this agreement probability is not directly observable, we use length-normalized sequence probability as a readily available proxy for response reliability. This metric reduces the influence of response length and has been widely adopted as a confidence or uncertainty measure in language generation [3, 4].

Inaccuracies in this proxy may affect posterior inference and introduce bias into solvability estimation. Nevertheless, the consistent improvements over several baselines suggest that length-normalized sequence probability captures useful response-level information for solvability inference, even without exact probability calibration.

[1] Maximum likelihood from incomplete data via the EM algorithm (Dempster et al., 1977)

[2] Maximum likelihood estimation of observer error rates using the EM algorithm (Dawid & Skene, 1979)

[3] LM-Polygraph: Uncertainty estimation for language models (Fadeeva et al., 2023)

[4] Contextualized sequence likelihood: Enhanced confidence scores for natural language generation (Lin et al., 2024)

Thanks for the patient response and the additional ablation on offline initialization, which clarifies where the gains come from. My concerns have been addressed.

We sincerely thank you for your positive feedback and support for our work. We are glad that our responses have addressed all of your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.