acceptodds
Under review as a conference paper at ICLR 2027

DIA-GRPO: Guiding Policy Optimization with Cross-Prompt Diagnosis

Abstract

Group Relative Policy Optimization (GRPO) improves reasoning in large language models through efficient, group-based reinforcement learning. However, its prompt-local advantage construction leaves a key source of learning information untapped: how successes and failures across prompts jointly reveal the model's evolving capabilities and inform how individual outcomes should contribute to its overall improvement. To address this limitation, we introduce Diagnosis-Informed Advantage for GRPO (DIA-GRPO), which translates diagnosis of the LLM's evolving capabilities into training feedback. By jointly modeling training-sample attributes and the LLM's evolving capabilities, DIA-GRPO translates cross-prompt observations and a population improvement objective into response-level advantages. Specifically, DIA-GRPO evaluates each outcome against the model's expected performance and uses the overall improvement objective to determine how that outcome contributes to policy updates. DIA-GRPO is lightweight and plug-and-play: online diagnosis operates in a low-dimensional capability space, and the resulting advantages are mixed with native GRPO advantages within the existing policy optimization objective, requiring no additional verifier or policy-scale critic. Theoretically, under explicit local conditions, we show that the expected unclipped diagnostic update recovers the policy gradient of a population log-success objective. Experiments on mathematical reasoning show that DIA-GRPO improves overall mean correctness by 2.96 and 2.05 percentage points on Qwen3-1.7B and Qwen3-8B, respectively. The diagnostic module also delivers consistent gains across different policy optimization methods with less than 2% additional training time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Group Relative Policy Optimization, Cognitive Diagnosis, Item Response Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_whNMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DIA-GRPO, which augments GRPO with feedback derived from cross-prompt capability diagnosis. A multidimensional item-response model tracks model capabilities and constructs advantages targeting average log success, with a policy-gradient correspondence established under explicit local assumptions. Experiments on five mathematical reasoning benchmarks show mean correctness gains of 2.95 and 2.05 percentage points for Qwen3-1.7B and Qwen3-8B, respectively. The method also improves other policy optimizers with less than 2% additional online training time.

Critical strengths

1.The feedback construction has a clear mathematical rationale. The diagnostic weights follow from matching an expected weighted likelihood gradient to an explicitly chosen utility, rather than introducing an arbitrary weighting heuristic. The authors also distinguish the ideal response-level result from regularization and model mismatch. This gives the method a useful conceptual foundation while keeping the additional model small. 2.The empirical gains extend beyond one model or optimizer. Table 1 shows higher avg@8 on every benchmark at both model sizes, and Table 2 reports improvements when DIA is added to DAPO, Dr. GRPO, and GMPO. The scalar and direction-shuffle controls match the diagnostic advantage’s expected second moment, making them more informative than uncontrolled ablations. The longer-generation evaluation also provides useful evidence that the gain is not confined to the main evaluation’s 768-token limit.

Critical weaknesses

1.The diagnostic representation is insufficiently specified and validated. Appendix B.1 provides the calibration objective, but does not clearly describe the calibration cohort, observations per item, or latent-scale convention. These details matter because the item attributes are subsequently frozen and reused for the 8B model. More importantly, no predictive validation is reported on independent responses from later policy states. Proposition 1 assumes agreement of both success probabilities and policy derivatives with the diagnostic model; the synthetic experiment assumes this structure rather than establishing it for an actual LLM. This limits the evidence supporting the proposed interpretation of the gains. 2.The experiments do not fully isolate the contribution of multidimensional diagnosis. Equation (41) interprets the diagnostic weights as a regularized projection of inverse success probabilities, making direct inverse-success weighting a particularly relevant missing baseline. The scalar and shuffled-direction controls are useful, but do not establish an advantage over a sensible difficulty-dependent alternative. In addition, DIA reserves half the rollout groups for tracking, changing the effective policy-update batch size relative to native GRPO. Without a comparison controlling for this change or results across independent training runs, the attribution and reliability of the reported gains remain uncertain.

Questions for authors

1.What is the composition of the calibration cohort, and how many observations are available per training item? Please clarify how latent scale is fixed and report predictive accuracy on independent responses from subsequent policy states, including the 8B model. 2.How does DIA compare with direct inverse-success weighting using the same predicted probabilities, tracking allocation, filters, and comparable advantage scaling and clipping? This comparison would help determine whether the learned multidimensional structure contributes beyond success-probability-based weighting. 3.Do the main gains persist against GRPO with the same number of policy-update groups, and across independent training seeds? The bootstrap intervals in Table 5 quantify evaluation-input uncertainty for fixed policies, but do not address variability introduced by training.

2 comments
AuthorsOpenReview ↗

Response to Reviewer whNM (Part 1/2)

Thank you for your careful reading of the derivation and for recognizing the value of the scale-matched ablations.

W1 / Q1. What is the composition of the calibration cohort, and how many observations are available per training item? Please clarify how latent scale is fixed and report predictive accuracy on independent responses from subsequent policy states, including the 8B model.

Calibration cohort. Our starting point is the diagnostic structure of GRPO’s grouped rollouts. Repeated answers to the same problem indicate how difficult that problem is for the current policy. When successive policies answer common problems, the resulting patterns also reveal which problems are more sensitive to changes in the learner’s capabilities. DIA-GRPO uses these patterns to estimate problem attributes and the current capability state, then feeds this information back into policy learning through GRPO’s existing advantage term.

Our calibration cohort comprised 20 Qwen3-1.7B checkpoints from a single GRPO warm-up trajectory, saved every 20 updates from step 20 through step 400. We collected one verified response per checkpoint–problem pair on 16,893 common training problems, giving 20 observations per problem and 337,860 responses in total. We jointly fitted the item parameters and checkpoint capabilities from these records. During subsequent training, the item parameters remained fixed. Verified responses from a subset of rollout groups updated the current capability estimate and, through it, the predicted success probabilities for all calibrated problems. This uses the same verifier and requires no policy-scale critic.

Latent scale. We set the origin of each capability coordinate to its mean across the 20 calibration checkpoints and adjust the item intercepts to preserve predicted probabilities. Item discrimination coefficients are constrained to be positive, and the fit uses AdamW weight decay. We impose no unit-variance constraint, so the units of each coordinate are specific to the fit. The frozen item parameters provide a common reference for later policies. For Qwen3-8B, we reuse these parameters and estimate the 8B policy’s capability state from its own responses.

Predictive accuracy on later policies. We evaluated GRPO checkpoints at 1,000, 3,000, and 6,000 post-warm-up updates for both model sizes. At each checkpoint, we estimated the capability state using two responses on each of 256 tracking problems, then predicted success on 512 separate problems. Each of these received 16 fresh responses. We assigned a positive prediction to problems with predicted success probability at least 0.5. Classification accuracy (ACC) is the fraction of the 8,192 individual responses for which this prediction agreed with the verifier.

PolicyUpdatesMIRT predicted positive rateObserved correct response rateMIRT classification ACC
Qwen3-1.7B1,0008.01%10.61%85.07%
Qwen3-1.7B3,0009.18%11.27%84.02%
Qwen3-1.7B6,0007.42%11.88%84.14%
Qwen3-8B1,00010.55%22.35%74.40%
Qwen3-8B3,00011.72%22.67%73.82%
Qwen3-8B6,00011.33%22.92%74.73%

Across the three checkpoints, MIRT achieved 84.02%–85.07% classification accuracy on independent responses from Qwen3-1.7B and 73.82%–74.73% on those from Qwen3-8B.

AuthorsOpenReview ↗

Response to Reviewer whNM (Part 2/2)

W2 / Q2. How does DIA compare with direct inverse-success weighting using the same predicted probabilities, tracking allocation, filters, and comparable advantage scaling and clipping? This comparison would help determine whether the learned multidimensional structure contributes beyond success-probability-based weighting.

We tested the reviewer’s proposed weight, ωit=1/max⁡(pit,0.01)\omega_{it}=1/\max(p_{it},0.01), against the Full DIA weight, ωit=ai⊤mt\omega_{it}=a_i^\top m_t. The former weights responses by predicted difficulty. The latter also accounts for how sensitive each problem is to the shared improvement direction. Within each comparison, both branches started from the same warm-up policy and used the same probability estimator, tracking allocation, response filters, mixing coefficient (α=0.5\alpha=0.5), and clipping thresholds. On the same warm-up responses, the root-mean-square magnitude of the clipped diagnostic advantage was 4.4% higher under inverse weighting.

Policy performance. We evaluated both methods on the same 450 problems with eight responses per problem.

ModelUpdatesInverse avg@8Full DIA avg@8Inverse pass@8Full DIA pass@8
Qwen3-1.7B1,00033.42%34.42%58.44%58.67%
Qwen3-1.7B1,00034.14%34.25%57.78%59.78%
Qwen3-8B3,00038.83%40.75%61.78%63.56%
Qwen3-8B3,00038.14%39.42%60.00%64.22%

Full DIA achieved higher avg@8 in all four comparisons. The gains were 1.00 and 0.11 percentage points on Qwen3-1.7B at 1,000 updates and 1.92 and 1.28 points on Qwen3-8B at 3,000 updates. In eight attempts, corresponding to pass@8 gains of 1.78 and 4.22 points.

How the weights change training credit. We applied both rules to the same 15,357 Qwen3-1.7B training responses. An incorrect answer can earn a format bonus. When predicted success is low, that bonus can produce a positive reward residual, which inverse weighting amplifies. For the 3,266 responses with pit<0.05p_{it}<0.05, we measured the positive part of the final advantage, max⁡(Afinal,0)\max(A^{\mathrm{final}},0), averaged over all responses in each outcome group.

Verified outcomeResponsesInverse weightingFull DIA
Incorrect2,6470.770.26
Correct6192.772.74

On these difficult problems, Full DIA reduced the positive credit assigned to incorrect answers by about 66%, while correct answers retained nearly all their positive credit. Across all 15,357 responses, the share of positive credit going to verified correct answers rose from 76.12% to 89.40%. This identifies a concrete contribution of the learned directions: among problems with low predicted success, they substantially reduce reinforcement of incorrect answers while preserving reinforcement of correct ones.

W2 / Q3. Do the main gains persist against GRPO with the same number of policy-update groups, and across independent training seeds? The bootstrap intervals in Table 5 quantify evaluation-input uncertainty for fixed policies, but do not address variability introduced by training.

Figure 3(a) holds the number of policy-update groups fixed across diagnostic variants. Full DIA-GRPO gained 3.83 percentage points in avg@8 over the scale-matched uniform-weight variant and 6.81 points over the variant with shuffled item directions.

We also compared DIA-GRPO directly with Matched GRPO across three training seeds. Both methods started from the same Qwen3-1.7B warm-up checkpoint, used one group of eight responses per policy update, and were evaluated on the same 450 problems. Entries are avg@8 / pass@8 (%).

Policy updatesTraining seedMatched GRPODIA-GRPO
1,000202633.58 / 59.7834.67 / 61.11
1,000202733.19 / 60.4435.17 / 62.44
1,000202833.83 / 60.0035.42 / 62.89
3,000202634.61 / 62.4437.97 / 64.00
3,000202734.39 / 61.3337.28 / 65.56
3,000202833.97 / 61.7836.50 / 65.11

DIA-GRPO improved both avg@8 and pass@8 over Matched GRPO for all three seeds at both checkpoints. At 1,000 updates, the gains averaged 1.55 and 2.07 percentage points, respectively. At 3,000 updates, they reached 2.93 points in avg@8 and 3.04 points in pass@8.

Reviewer_hnr5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DIA-GRPO, which incorporates cross-prompt capability diagnosis into GRPO to improve advantage estimation. The method uses multidimensional item response theory to estimate the model's evolving capabilities and predict success probabilities across training samples. These predictions are then used to construct diagnostic advantages, which are combined with standard GRPO advantages. The authors also provide a theoretical connection between the diagnostic update and a population log-success objective. Experiments on five mathematical reasoning benchmarks show consistent improvements over GRPO on Qwen3-1.7B and Qwen3-8B. Additional experiments demonstrate that the method can be applied to DAPO, Dr.GRPO, and GMPO with relatively small computational overhead.

Critical strengths
  1. Interesting approach to advantage estimation. Most GRPO variants focus on modifying within-group normalization or reward weighting. This paper instead uses information across prompts to guide policy optimization. I find this direction interesting, as the performance of a model on different problems is not necessarily independent.

  2. A reasonably well-motivated formulation. Rather than introducing another heuristic weighting rule, the authors derive diagnostic advantages from a population log-success objective. The connection between the capability model, Fisher information, and advantage construction is a useful part of the paper.

  3. Good experimental improvements. DIA-GRPO improves avg@8 across all five benchmarks on both Qwen3-1.7B and Qwen3-8B. The gains are particularly noticeable on GSM8K and MATH-500. The experiments with DAPO, Dr.GRPO, and GMPO also suggest that the proposed diagnostic feedback is not restricted to a particular optimizer.

  4. The ablations are informative. The comparisons with scalar weighting, shuffled directions, and frozen diagnosis provide evidence that both the diagnostic structure and its online updates contribute to performance. The reported additional training cost is also relatively small.

Critical weaknesses
  1. The assumptions behind the theoretical result are quite strong. Proposition 1 assumes that the policy's true success probabilities can be locally represented by the multidimensional item response model, including their derivatives. It also requires exact capability estimation and population moments. These conditions are unlikely to hold exactly in practice. Although Appendix A discusses approximation errors, it remains unclear how accurately the diagnostic model captures the behavior of the actual policy during training.

  2. The role of the capability representation could be studied more carefully. The method relies on a ten-dimensional MIRT model, but the paper provides limited evidence for why this representation is appropriate. In particular, I would like to see how the prediction quality and downstream performance change with the number of capability dimensions. It would also help to report whether the diagnostic model remains well calibrated as the policy moves away from the warm-up checkpoints.

  3. The comparison with simpler advantage estimators is insufficient. DIA-GRPO uses predicted success probabilities as baselines and introduces sample-dependent weights. While the existing ablations are useful, they do not fully establish whether modeling shared capabilities is necessary. A comparison with simpler per-question success-rate estimates or difficulty-based advantage weighting would better demonstrate the benefit of cross-prompt diagnosis.

  4. There is a gap between the theoretical formulation and the implemented training procedure. The theoretical analysis considers Bernoulli correctness rewards and an unclipped policy gradient. However, the actual training uses additional format rewards, clipped diagnostic advantages, and response filtering. Appendix C also indicates that zero-reward-variance groups are filtered. These choices may change the expected update, so the extent to which the theoretical result explains the practical algorithm is not entirely clear.

  5. The reliability of the experimental gains could be better established. The results cover two model scales and several benchmarks, but the main tables do not report variation across independent training runs. Since some improvements are relatively modest, reporting multiple training seeds would make the empirical conclusions more convincing. It would also be interesting to know whether the method works beyond mathematical reasoning tasks.

Questions for authors
  1. How well does the MIRT model predict the actual success probabilities of the policy at different training stages? Could the authors report calibration errors or prediction correlations over training?

  2. How sensitive is DIA-GRPO to the number of capability dimensions? Would a simpler one-dimensional model or a higher-dimensional representation produce similar results?

  3. Have the authors compared DIA-GRPO with simpler methods that estimate per-question success probabilities or use difficulty-aware advantage weighting? I think this comparison is important for understanding the contribution of cross-prompt diagnosis.

  4. Since the implementation uses shaped rewards, advantage clipping, and zero-variance group filtering, how much of the theoretical objective-consistency result carries over to the actual training updates? In particular, does zero-variance filtering discard some of the diagnostic feedback that GRPO would otherwise miss?

  5. Are the reported improvements consistent across different training seeds? It would be helpful to include standard deviations or confidence intervals for the main results.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hnr5 (Part 1/3)

Thank you for your careful review and for highlighting the link between cross-prompt feedback and the population objective.

W1 / Q1. How well does the MIRT model predict the actual success probabilities of the policy at different training stages? Could the authors report calibration errors or prediction correlations over training?

We evaluated MIRT at 1,000, 3,000, and 6,000 post-warm-up GRPO updates on both Qwen3-1.7B and Qwen3-8B. At each checkpoint, we estimated the policy’s capability from two responses per problem on 256 tracking problems. We then predicted success on a separate set of 512 problems, held fixed across checkpoints, and sampled 16 fresh responses per problem to estimate its observed success rate.

We compute expected calibration error (ECE) by grouping predictions into ten equal-width probability bins and averaging the absolute gap between predicted and observed success rates in each bin, weighted by its number of problems. Spearman’s ρ measures the rank correlation between predicted and observed per-problem success rates. MIRT classification accuracy (ACC) measures how often a prediction at the 0.5 threshold agrees with the verifier across the 8,192 responses per checkpoint.

PolicyUpdates after warm-upMIRT classification ACC (%)ECE (pp) ↓Spearman ρ ↑
Qwen3-1.7B1,00085.0712.140.18
Qwen3-1.7B3,00084.0213.500.23
Qwen3-1.7B6,00084.1413.440.23
Qwen3-8B1,00074.4019.300.15
Qwen3-8B3,00073.8222.370.14
Qwen3-8B6,00074.7319.400.18

On Qwen3-1.7B, ECE stayed within 12.14–13.50 percentage points across the three checkpoints. On Qwen3-8B, it was higher at every checkpoint, rising from 19.30 to 22.37 points and returning to 19.40 points at 6,000 updates. Item-level rank correlations were positive throughout (Spearman ρ = 0.14–0.23). Incorrect responses predominated, and predicting every response as incorrect yielded higher ACC at all six checkpoints. At each checkpoint, however, problems assigned a success probability of at least 0.5 had higher observed correctness than the remaining problems.

W2 / Q2. How sensitive is DIA-GRPO to the number of capability dimensions? Would a simpler one-dimensional model or a higher-dimensional representation produce similar results?

Following the dimension-selection approach in tinyBenchmarks [1], we treated the number of capability dimensions as a hyperparameter and ran a small sweep over d∈{1,2,5,10,20}d\in\{1,2,5,10,20\}. We selected d=10d=10 for our mathematical reasoning task because it gave the highest avg@8 and pass@8. For each setting, we trained Qwen3-1.7B for 1,000 post-warm-up updates and evaluated it on the same 450 problems with eight responses per problem.

Capability dimensionsavg@8 (%)pass@8 (%)
133.8158.22
233.7558.44
533.9258.89
1035.9760.22
2034.7259.78

Performance varied modestly across the tested range. From one to twenty dimensions, the gap between the highest and lowest scores was 2.22 percentage points in avg@8 and 2.00 points in pass@8. Increasing dd from ten to twenty yielded no further gain on either metric. For a new task, the same small sweep can be used to select a suitable dimension. Earlier work also used task-specific dimensions, choosing ten for SQuAD and six for MRQA [2].

References

[1] F. Maia Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML, 2024, pp. 34303–34326.

[2] P. Rodriguez, J. Barrow, A. M. Hoyle, J. P. Lalor, R. Jia, and J. Boyd-Graber. Evaluation Examples Are Not Equally Informative: How Should That Change NLP Leaderboards? ACL-IJCNLP, pp. 4486–4503, 2021.

AuthorsOpenReview ↗

Response to Reviewer hnr5 (Part 2/3)

W3 / Q3. Have the authors compared DIA-GRPO with simpler methods that estimate per-question success probabilities or use difficulty-aware advantage weighting? I think this comparison is important for understanding the contribution of cross-prompt diagnosis.

Following the reviewer’s suggestion, we compared DIA-GRPO with policies trained under two simpler credit assignment rules. The first comparison tests whether a question’s own response history provides sufficient training feedback. Per-question success rate estimates p^i\hat p_i from prior responses to question ii and mixes Rig−p^iR_{ig}-\hat p_i with the native advantage, where RigR_{ig} is the training reward. The second comparison tests what the learned directions contribute once cross-prompt predictions are available. MIRT + inverse uses the same success estimator as Full DIA and replaces its directional weight ai⊤mta_i^\top m_t with the difficulty weight 1/max⁡(pit,0.01)1/\max(p_{it},0.01). The two MIRT variants share the starting policy, tracking allocation, response filters, mixing coefficient (α=0.5\alpha=0.5), and clipping thresholds.

Policy performance. Each entry reports avg@8 / pass@8 (%) on the same 450 problems, with eight responses per problem.

ModelUpdatesPer-question success rateMIRT + inverseFull DIA
Qwen3-1.7B1,00031.64 / 57.1133.42 / 58.4434.42 / 58.67
Qwen3-8B3,00036.72 / 60.0038.83 / 61.7840.75 / 63.56

Full DIA achieved higher avg@8 and pass@8 than both alternatives in all comparisons. The advantage over MIRT + inverse persisted at both model scales even though both methods already used cross-prompt estimates of success.

Directional weighting concentrates training credit on correct answers. For two questions with the same predicted success rate, inverse weighting applies the same multiplier to their reward residuals. DIA can weight them differently according to how sensitive each question is to the shared improvement direction. This matters on difficult questions, where a format bonus can give an incorrect answer a positive residual that inverse weighting then amplifies. We applied both rules to the same 15,357 Qwen3-1.7B training responses. For the 3,266 responses with pit<0.05p_{it}<0.05, the table reports the mean of max⁡(Afinal,0)\max(A^{\mathrm{final}},0) separately for correct and incorrect answers.

Verified outcomeResponsesInverse weightingFull DIA
Incorrect2,6470.770.26
Correct6192.772.74

On these difficult questions, DIA reduced mean positive credit for incorrect answers by 66% (0.77 to 0.26) while retaining 99% of the credit for correct answers (2.77 to 2.74). Across all 15,357 responses, verified correct answers received 89.40% of total positive credit under DIA, up from 76.12% under inverse weighting. Together, the training and credit comparisons show what the learned directions add to a difficulty estimate: they preserve reinforcement for successful solutions to hard questions while reducing reinforcement for errors, and the resulting policies achieve higher correctness.

AuthorsOpenReview ↗

Response to Reviewer hnr5 (Part 3/3)

W4 / Q4. how much of the theoretical objective-consistency result carries over to the actual training updates?

Under the assumptions of Proposition 1, the expected, unclipped diagnostic update with binary correctness rewards equals the policy gradient of mean log success. The implementation uses the weights derived from this identity. It fits the diagnostic model to shaped rewards clipped to [0,1][0,1], computes the diagnostic advantage using the shaped reward before clipping, and then applies advantage clipping and loss masks. The exact gradient identity is specific to the theoretical diagnostic component.

Zero-variance filtering does discard diagnostic feedback. We examined the group assigned to policy learning at each of 200 Qwen3-1.7B updates. In 37 groups, all responses received the same shaped reward. GRPO assigned zero advantage to every response in those groups. DIA produced nonzero advantages in all 37 groups, but the filter removed all 37 before the policy update.

Shaped rewards within groupGroupsNonzero advantage before masking (groups)Nonzero advantage in the loss (groups)
Equal37370
Unequal163163159

Under the main training recipe, DIA therefore affects the update by reweighting responses in the retained groups. Of the 759 responses that entered the loss, 163 (21.5%) exceeded the clipping threshold for the raw diagnostic advantage. None reached the final clipping threshold after mixing with the GRPO advantage.

Binary rewards with equal-reward groups retained. We trained GRPO and DIA-GRPO from the same Qwen3-1.7B warm-up policy using binary correctness rewards and disabling the zero-variance filter for both methods. We kept the advantage clipping thresholds and truncation mask unchanged. At each checkpoint, we evaluated both policies on the same 450 problems with eight responses per problem.

Post-warm-up updatesGRPO avg@8 (%)DIA-GRPO avg@8 (%)GRPO pass@8 (%)DIA-GRPO pass@8 (%)
1,00032.8133.5057.5661.33
3,00033.8935.0659.1162.67

With binary rewards and equal-reward groups retained, the clearest gain was in the fraction of problems solved within eight attempts. DIA-GRPO improved pass@8 by 3.78 percentage points at 1,000 updates and 3.56 points at 3,000 updates. The corresponding avg@8 gains were 0.69 and 1.17 points.

W5 / Q5. Are the reported improvements consistent across different training seeds? It would be helpful to include standard deviations or confidence intervals for the main results.

To address the reviewer’s question, we tested whether DIA-GRPO’s advantage over GRPO persists when we change the training seed. We repeated the Qwen3-1.7B comparison with three independent post-warm-up seeds. Within each pair, both methods started from the same warm-up checkpoint, used one group of eight responses per policy update, and were evaluated on the same 450 problems with eight responses per problem. The table reports the gain over Matched GRPO for each seed and the mean ± sample standard deviation of those gains, in percentage points.

Policy updatesMetricSeed 2026Seed 2027Seed 2028Mean gain ± SD (pp)
1,000avg@8+1.09+1.98+1.59+1.55 ± 0.45
1,000pass@8+1.33+2.00+2.89+2.07 ± 0.78
3,000avg@8+3.36+2.89+2.53+2.93 ± 0.42
3,000pass@8+1.56+4.23+3.33+3.04 ± 1.36

A separate short-run comparison showed smaller or mixed effects. At 300 updates, the two runs averaged −0.28 points in avg@8 and −1.78 points in pass@8. One 1,000-update run gained 0.39 points in avg@8 with unchanged pass@8.

In the three-seed comparison above, even the smallest gains at 3,000 updates were positive on both metrics (+2.53 points in avg@8 and +1.56 points in pass@8). The size of the improvement varied across seeds, particularly for pass@8. What reproduced across all three seeds was that DIA-GRPO generated correct responses more often and solved a larger fraction of problems within eight attempts, at both 1,000 and 3,000 updates.

Confidence intervals across evaluation problems. To quantify uncertainty across evaluation problems, we compared fixed Qwen3-1.7B GRPO and DIA-GRPO checkpoints on the same 450 problems. At each generation limit, both policies produced eight responses per problem. We resampled the same problem IDs for both policies to obtain a 95% confidence interval for the difference in avg@8.

Generation limit (tokens)Gain in avg@8 (pp)Paired 95% CI (pp)
768+2.08[0.61, 3.58]
1,536+2.31[0.78, 3.86]
2,048+3.33[1.72, 4.94]

All three intervals lie above zero. On this evaluation set, the gain in per-response correctness persisted across all three generation limits.

Reviewer_eCVvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method that uses model capabilities and problem characteristics estimated by an item response model to guide advantage estimation in GRPO and improve performance on mathematical reasoning tasks.

Critical strengths

The method requires little additional computation.

Critical weaknesses
  1. The novelty is limited. The method relies on similar reasoning patterns, answer formats, and semantic information shared by prompts within the same domain. It is unclear whether it can generalize across domains, so the scope of its cross-prompt generalization is limited.

  2. The GSM8K baseline scores are much lower than those reported in the official Qwen3 technical report (https://arxiv.org/abs/2505.09388). The authors should clarify the evaluation settings and explain this discrepancy.

  3. The motivation for introducing this module is unclear. The paper does not clearly explain why it is needed.

  4. The paper does not clearly explain why this particular item response model is chosen to predict success probabilities.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer eCVv (Part 1/3)

Thank you for reviewing our paper and for noting the low computational overhead of DIA-GRPO.

W1. The novelty is limited. The method relies on similar reasoning patterns, answer formats, and semantic information shared by prompts within the same domain. It is unclear whether it can generalize across domains, so the scope of its cross-prompt generalization is limited.

W3. The motivation for introducing this module is unclear. The paper does not clearly explain why it is needed.

Domain-specific training is an established setting for studying policy optimization in LLMs. GRPO itself was introduced and evaluated on mathematical reasoning in DeepSeekMath [1]. DAPO trained on competition mathematics [2], and CodeRL used unit-test feedback to train on programming problems [3]. Ma et al. [4] likewise observe that much of the work on reinforcement learning for LLM reasoning has concentrated on mathematics and coding. Our five-benchmark study follows this setting.

There is empirical evidence that a policy’s performance on one prompt is informative about its performance on others. Qu et al. [6] found that pooling training history across prompts predicted their difficulty for the current policy more accurately than tracking each prompt’s outcomes independently. They used these predictions to select which prompts to train on. We use the same observation to address a different decision in policy optimization: once a response has been generated and verified, how should evidence from other prompts affect its contribution to the update?

Consider two problems on which the policy earns the same sequence of rewards. GRPO assigns the same within-group advantages to both, even if progress along a shared capability direction would improve success on one problem much more than on the other. This is why we introduce diagnosis: it lets us weight each response by how its problem relates to the improvement sought across the training set.

Our starting observation is that GRPO’s verification records have a diagnostic structure. GRPO samples several answers to one problem and uses their rewards to estimate a baseline [1]. SPO tracks a problem’s success rate over time to refine that baseline [5]. When we evaluate successive policy checkpoints on common problems, each checkpoint acts as a respondent and each problem as a fixed test item. The resulting response patterns allow us to estimate a shared capability state from multiple problems and to learn how sensitive each problem is to changes in that state. These relationships are inferred from observed successes and failures, without requiring a measure of similarity between prompt texts, answer formats, or reasoning traces.

Our algorithmic contribution is to turn this diagnostic structure into a policy update. Let pitp_{it} denote the predicted success probability for problem ii, aia_i its capability sensitivity vector, and mtm_t the capability direction derived from the objective of improving mean log success across training problems. Under the binary response model, the rate at which an observed outcome’s log likelihood changes along this direction gives its diagnostic advantage:

AigDIA=ddϵlog⁡Pϑ^t+ϵmt(rig∣i)∣ϵ=0=(rig−pit),ai⊤mt.A_{ig}^{\mathrm{DIA}} = \left.\frac{\mathrm{d}}{\mathrm{d}\epsilon}\log P_{\hat\vartheta_t+\epsilon m_t}(r_{ig}\mid i)\right\vert_{\epsilon=0} = (r_{ig}-p_{it})\\,a_i^\top m_t.

The same model thus supplies a baseline for interpreting the outcome and a weight for its contribution to the update. Two problems can have the same predicted success probability yet receive different weights because their success probabilities respond differently to the desired capability change. Figure 3(a) tests this distinction directly: shuffling the correspondence between problems and their capability directions preserves the success predictions but lowers avg@8 by 6.81 percentage points.

References

[1] Z. Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.

[2] Q. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS, 2025.

[3] H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. NeurIPS, 2022.

[4] X. Ma, Q. Liu, D. Jiang, G. Zhang, Z. Ma, and W. Chen. General-Reasoner: Advancing LLM Reasoning Across All Domains. NeurIPS, 2025.

[5] Z. Xu and Z. Ding. Single-stream Policy Optimization. ICLR, 2026.

[6] Y. Qu et al. Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, pp. 102651–102677, 2026.

AuthorsOpenReview ↗

Response to Reviewer eCVv (Part 2/3)

W2. The GSM8K baseline scores are much lower than those reported in the official Qwen3 technical report (https://arxiv.org/abs/2505.09388). The authors should clarify the evaluation settings and explain this discrepancy.

The Qwen3 technical report evaluates GSM8K using four-shot chain-of-thought (CoT) prompts [7]. Table 1 in our paper uses zero-shot chat prompts with sampling. To quantify how prompting and decoding affect the scores, we randomly selected 256 of the 1,319 GSM8K test problems and used the same subset and grader across all four settings. Each base model generated one response per problem, giving 256 responses per model per setting. The four worked examples came from the GSM8K training set. All sampling runs used temperature 0.7 and top-p=0.95p=0.95.

PromptDecodingQwen3-1.7B-BaseQwen3-8B-Base
Zero-shot, chat templateSampling48.44%49.22%
Zero-shot, plain question–answer templateSampling44.92%41.80%
Four-shot CoT, plain question–answer templateSampling50.00%75.78%
Same four-shot CoT promptGreedy71.48%87.89%

Each successive row changes one factor. Switching from chat to a plain question–answer template alone lowered accuracy at both scales. For 8B, adding four worked examples under the same template and sampling settings raised accuracy by 33.98 percentage points (paired 95% CI [26.56, 41.41]). For 1.7B, the largest gain came from decoding. With the four-shot prompt held fixed, switching from sampling to greedy decoding raised accuracy by 21.48 points (paired 95% CI [14.06, 28.52]).

These protocol changes account for much of the observed gap. With four-shot CoT and greedy decoding, the same base checkpoints reached 71.48% and 87.89% on the 256-problem subset, close to the 75.44% and 89.84% reported on the full test set [7]. Table 1 compares GRPO and DIA-GRPO under the same zero-shot sampling protocol.

References

[7] Qwen Team. Qwen3 Technical Report. arXiv:2505.09388, 2025, Section 3.3, Tables 6 and 8.

AuthorsOpenReview ↗

Response to Reviewer eCVv (Part 3/3)

W4. The paper does not clearly explain why this particular item response model is chosen to predict success probabilities.

IRT has an established role in measuring language-model capabilities from item-level responses. Rodriguez et al. [8] used it to estimate question difficulty and QA system ability. tinyBenchmarks [9] used the same two-parameter multidimensional logistic model we adopt to predict whether an LLM would answer a given item correctly. ATLAS [12] used IRT for adaptive LLM evaluation, and Truong et al. [13] applied an IRT-based model across 6,612 language-model checkpoints. We chose MIRT because it provides a more expressive model of item responses. Unidimensional IRT explains performance across all problems through a single ability coordinate. MIRT includes that case and also allows different problems to be sensitive to different directions of capability change.

Classical skill-diagnosis models require additional structure. DINA [10] specifies which skills each problem requires and uses a conjunctive rule for mastery. The original NeuralCDM [11] also uses a predefined problem–concept map and imposes monotonicity in concept proficiency. Our verification records contain checkpoint–problem outcomes without skill annotations. MIRT can estimate both checkpoint capabilities and item sensitivities directly from those records.

This flexibility matters for policy optimization. In our model, logit⁡pi(ϑ)=ai⊤ϑ−bi\operatorname{logit}p_i(\vartheta)=a_i^\top\vartheta-b_i, so aia_i gives the sensitivity of problem ii’s log-odds of success to a change in capability. Its projection onto the current improvement direction, ωit=ai⊤mt\omega_{it}=a_i^\top m_t, is the weight used in diagnostic credit assignment. In one dimension, changing mtm_t rescales all directional weights by the same factor. In multiple dimensions, the improvement direction can align more closely with one problem and less closely with another. This allows the relative credit assigned to the two problems to change as the policy evolves, even when their predicted success probabilities are equal.

References

[8] P. Rodriguez, J. Barrow, A. Hoyle, J. P. Lalor, R. Jia, and J. Boyd-Graber. Evaluation Examples Are Not Equally Informative: How Should That Change NLP Leaderboards? ACL-IJCNLP, pp. 4486–4503, 2021.

[9] F. Maia Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML, PMLR 235, pp. 34303–34326, 2024.

[10] J. de la Torre. DINA Model and Parameter Estimation: A Didactic. Journal of Educational and Behavioral Statistics, 34(1), pp. 115–130, 2009.

[11] F. Wang, Q. Liu, E. Chen, Z. Huang, Y. Chen, Y. Yin, Z. Huang, and S. Wang. Neural Cognitive Diagnosis for Intelligent Education Systems. AAAI, 34(4), pp. 6153–6161, 2020.

[12] P. Li, X. Tang, S. Chen, Y. Cheng, R. Metoyer, T. Hua, and N. V. Chawla. Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks. ICML, PMLR 306, pp. 70246–70269, 2026.

[13] S. T. Truong, Y. Tu, R. Schaeffer, and S. Koyejo. Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation. ICML, PMLR 306, pp. 122297–122315, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.