- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes KRPO, a variant of GRPO for training language models on math problems. It replaces the average reward of sampled answers with a Kalman-filtered estimate and also changes how the advantages are scaled. Experiments with Llama and Qwen models report higher rewards and scores across several math datasets.
Critical strengths
S1. Simple and cheap. KRPO changes only the advantage computation, adds no parameters, and costs about the same training time as GRPO. If the benefit held up, it would be easy to adopt.
S2. Consistent gains over GRPO in the reported experiments. The improvement appears across several base models, datasets, group sizes, and seeds.
Critical weaknesses
W1. The filter does not do what the paper claims. With Q and R fixed and the filter reset for each prompt, the filter's variance never depends on the rewards. The "uncertainty" used to scale the advantage is just a constant determined by a rollout's position in the group. With such a small Q, the baseline is also essentially the average of the rewards seen so far, which uses fewer samples than the GRPO group mean rather than giving a better estimate. This runs against the paper's main motivation, and the paper does not explain where the gains actually come from.
W2. The evaluation is non-standard. Test accuracy gives half credit whenever the ground-truth answer appears anywhere in the output. The base Llama-3.2-1B-Instruct scores only 3.1% on MATH500, far below the 30.6% that Meta reports on MATH for the same model. Results under a standard evaluation are needed to judge the real improvement.
W3. The experimental scope is narrow. All accuracy results come from models of at most 1.5B parameters, mostly on arithmetic and filtered short OpenMath problems; the larger models are shown only through training reward curves. KRPO is also compared only with GRPO and older baselines, not with other GRPO variants that target the same variance problem, such as the leave-one-out baseline (RLOO).
Questions for authors
-
Is the KRPO baseline actually closer to a prompt's expected reward than the group mean, for example when measured against an estimate from many rollouts?
-
How does GRPO perform if its advantages are rescaled to the same magnitude as KRPO's?
-
Can the main comparisons be reported under standard evaluation, that is, greedy decoding, equivalence-based answer matching, and no partial credit?