- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DIA-GRPO, which augments GRPO with feedback derived from cross-prompt capability diagnosis. A multidimensional item-response model tracks model capabilities and constructs advantages targeting average log success, with a policy-gradient correspondence established under explicit local assumptions. Experiments on five mathematical reasoning benchmarks show mean correctness gains of 2.95 and 2.05 percentage points for Qwen3-1.7B and Qwen3-8B, respectively. The method also improves other policy optimizers with less than 2% additional online training time.
Critical strengths
1.The feedback construction has a clear mathematical rationale. The diagnostic weights follow from matching an expected weighted likelihood gradient to an explicitly chosen utility, rather than introducing an arbitrary weighting heuristic. The authors also distinguish the ideal response-level result from regularization and model mismatch. This gives the method a useful conceptual foundation while keeping the additional model small. 2.The empirical gains extend beyond one model or optimizer. Table 1 shows higher avg@8 on every benchmark at both model sizes, and Table 2 reports improvements when DIA is added to DAPO, Dr. GRPO, and GMPO. The scalar and direction-shuffle controls match the diagnostic advantage’s expected second moment, making them more informative than uncontrolled ablations. The longer-generation evaluation also provides useful evidence that the gain is not confined to the main evaluation’s 768-token limit.
Critical weaknesses
1.The diagnostic representation is insufficiently specified and validated. Appendix B.1 provides the calibration objective, but does not clearly describe the calibration cohort, observations per item, or latent-scale convention. These details matter because the item attributes are subsequently frozen and reused for the 8B model. More importantly, no predictive validation is reported on independent responses from later policy states. Proposition 1 assumes agreement of both success probabilities and policy derivatives with the diagnostic model; the synthetic experiment assumes this structure rather than establishing it for an actual LLM. This limits the evidence supporting the proposed interpretation of the gains. 2.The experiments do not fully isolate the contribution of multidimensional diagnosis. Equation (41) interprets the diagnostic weights as a regularized projection of inverse success probabilities, making direct inverse-success weighting a particularly relevant missing baseline. The scalar and shuffled-direction controls are useful, but do not establish an advantage over a sensible difficulty-dependent alternative. In addition, DIA reserves half the rollout groups for tracking, changing the effective policy-update batch size relative to native GRPO. Without a comparison controlling for this change or results across independent training runs, the attribution and reliability of the reported gains remain uncertain.
Questions for authors
1.What is the composition of the calibration cohort, and how many observations are available per training item? Please clarify how latent scale is fixed and report predictive accuracy on independent responses from subsequent policy states, including the 8B model. 2.How does DIA compare with direct inverse-success weighting using the same predicted probabilities, tracking allocation, filters, and comparable advantage scaling and clipping? This comparison would help determine whether the learned multidimensional structure contributes beyond success-probability-based weighting. 3.Do the main gains persist against GRPO with the same number of policy-update groups, and across independent training seeds? The bootstrap intervals in Table 5 quantify evaluation-input uncertainty for fixed policies, but do not address variability introduced by training.
2 comments
Response to Reviewer whNM (Part 2/2)
W2 / Q2. How does DIA compare with direct inverse-success weighting using the same predicted probabilities, tracking allocation, filters, and comparable advantage scaling and clipping? This comparison would help determine whether the learned multidimensional structure contributes beyond success-probability-based weighting.
We tested the reviewer’s proposed weight, , against the Full DIA weight, . The former weights responses by predicted difficulty. The latter also accounts for how sensitive each problem is to the shared improvement direction. Within each comparison, both branches started from the same warm-up policy and used the same probability estimator, tracking allocation, response filters, mixing coefficient (), and clipping thresholds. On the same warm-up responses, the root-mean-square magnitude of the clipped diagnostic advantage was 4.4% higher under inverse weighting.
Policy performance. We evaluated both methods on the same 450 problems with eight responses per problem.
| Model | Updates | Inverse avg@8 | Full DIA avg@8 | Inverse pass@8 | Full DIA pass@8 |
|---|---|---|---|---|---|
| Qwen3-1.7B | 1,000 | 33.42% | 34.42% | 58.44% | 58.67% |
| Qwen3-1.7B | 1,000 | 34.14% | 34.25% | 57.78% | 59.78% |
| Qwen3-8B | 3,000 | 38.83% | 40.75% | 61.78% | 63.56% |
| Qwen3-8B | 3,000 | 38.14% | 39.42% | 60.00% | 64.22% |
Full DIA achieved higher avg@8 in all four comparisons. The gains were 1.00 and 0.11 percentage points on Qwen3-1.7B at 1,000 updates and 1.92 and 1.28 points on Qwen3-8B at 3,000 updates. In eight attempts, corresponding to pass@8 gains of 1.78 and 4.22 points.
How the weights change training credit. We applied both rules to the same 15,357 Qwen3-1.7B training responses. An incorrect answer can earn a format bonus. When predicted success is low, that bonus can produce a positive reward residual, which inverse weighting amplifies. For the 3,266 responses with , we measured the positive part of the final advantage, , averaged over all responses in each outcome group.
| Verified outcome | Responses | Inverse weighting | Full DIA |
|---|---|---|---|
| Incorrect | 2,647 | 0.77 | 0.26 |
| Correct | 619 | 2.77 | 2.74 |
On these difficult problems, Full DIA reduced the positive credit assigned to incorrect answers by about 66%, while correct answers retained nearly all their positive credit. Across all 15,357 responses, the share of positive credit going to verified correct answers rose from 76.12% to 89.40%. This identifies a concrete contribution of the learned directions: among problems with low predicted success, they substantially reduce reinforcement of incorrect answers while preserving reinforcement of correct ones.
W2 / Q3. Do the main gains persist against GRPO with the same number of policy-update groups, and across independent training seeds? The bootstrap intervals in Table 5 quantify evaluation-input uncertainty for fixed policies, but do not address variability introduced by training.
Figure 3(a) holds the number of policy-update groups fixed across diagnostic variants. Full DIA-GRPO gained 3.83 percentage points in avg@8 over the scale-matched uniform-weight variant and 6.81 points over the variant with shuffled item directions.
We also compared DIA-GRPO directly with Matched GRPO across three training seeds. Both methods started from the same Qwen3-1.7B warm-up checkpoint, used one group of eight responses per policy update, and were evaluated on the same 450 problems. Entries are avg@8 / pass@8 (%).
| Policy updates | Training seed | Matched GRPO | DIA-GRPO |
|---|---|---|---|
| 1,000 | 2026 | 33.58 / 59.78 | 34.67 / 61.11 |
| 1,000 | 2027 | 33.19 / 60.44 | 35.17 / 62.44 |
| 1,000 | 2028 | 33.83 / 60.00 | 35.42 / 62.89 |
| 3,000 | 2026 | 34.61 / 62.44 | 37.97 / 64.00 |
| 3,000 | 2027 | 34.39 / 61.33 | 37.28 / 65.56 |
| 3,000 | 2028 | 33.97 / 61.78 | 36.50 / 65.11 |
DIA-GRPO improved both avg@8 and pass@8 over Matched GRPO for all three seeds at both checkpoints. At 1,000 updates, the gains averaged 1.55 and 2.07 percentage points, respectively. At 3,000 updates, they reached 2.93 points in avg@8 and 3.04 points in pass@8.
Response to Reviewer whNM (Part 1/2)
Thank you for your careful reading of the derivation and for recognizing the value of the scale-matched ablations.
Calibration cohort. Our starting point is the diagnostic structure of GRPO’s grouped rollouts. Repeated answers to the same problem indicate how difficult that problem is for the current policy. When successive policies answer common problems, the resulting patterns also reveal which problems are more sensitive to changes in the learner’s capabilities. DIA-GRPO uses these patterns to estimate problem attributes and the current capability state, then feeds this information back into policy learning through GRPO’s existing advantage term.
Our calibration cohort comprised 20 Qwen3-1.7B checkpoints from a single GRPO warm-up trajectory, saved every 20 updates from step 20 through step 400. We collected one verified response per checkpoint–problem pair on 16,893 common training problems, giving 20 observations per problem and 337,860 responses in total. We jointly fitted the item parameters and checkpoint capabilities from these records. During subsequent training, the item parameters remained fixed. Verified responses from a subset of rollout groups updated the current capability estimate and, through it, the predicted success probabilities for all calibrated problems. This uses the same verifier and requires no policy-scale critic.
Latent scale. We set the origin of each capability coordinate to its mean across the 20 calibration checkpoints and adjust the item intercepts to preserve predicted probabilities. Item discrimination coefficients are constrained to be positive, and the fit uses AdamW weight decay. We impose no unit-variance constraint, so the units of each coordinate are specific to the fit. The frozen item parameters provide a common reference for later policies. For Qwen3-8B, we reuse these parameters and estimate the 8B policy’s capability state from its own responses.
Predictive accuracy on later policies. We evaluated GRPO checkpoints at 1,000, 3,000, and 6,000 post-warm-up updates for both model sizes. At each checkpoint, we estimated the capability state using two responses on each of 256 tracking problems, then predicted success on 512 separate problems. Each of these received 16 fresh responses. We assigned a positive prediction to problems with predicted success probability at least 0.5. Classification accuracy (ACC) is the fraction of the 8,192 individual responses for which this prediction agreed with the verifier.
Across the three checkpoints, MIRT achieved 84.02%–85.07% classification accuracy on independent responses from Qwen3-1.7B and 73.82%–74.73% on those from Qwen3-8B.