acceptodds
Withdrawn from ICLR 2027

Kalman Filter Enhanced Group Relative Policy Optimization for Language Model Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The advantage function is a central concept in RL that helps reduce variance in policy gradient estimates. For language modeling, Group Relative Policy Optimization (GRPO) was proposed to use the within-group sample mean as a baseline for advantage normalization. This estimator can be sensitive to small group size and rollout-level stochasticity, which may lead to unstable or less informative advantage estimates in some settings. In this paper, we propose Kalman Filter Enhanced Group Relative Policy Optimization (KRPO), a lightweight variant that treats per-group rewards as noisy observations of a latent prompt-level reward baseline and uses a 1D Kalman filter to estimate both the baseline and its uncertainty. KRPO introduces no additional learned parameters and can be integrated into GRPO with minimal computational overhead. On mathematical reasoning benchmarks, KRPO consistently improves training reward curves and final accuracy over GRPO. These results suggest that adaptive advantage estimation is a promising direction for critic-free reinforcement learning in language model reasoning. The code is available at https://anonymous.4open.science/r/KRPO-E1D3.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Reasoning, Chain-of-thought, Kalman Filter, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XJGvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes KRPO, a variant of GRPO for training language models on math problems. It replaces the average reward of sampled answers with a Kalman-filtered estimate and also changes how the advantages are scaled. Experiments with Llama and Qwen models report higher rewards and scores across several math datasets.

Critical strengths

S1. Simple and cheap. KRPO changes only the advantage computation, adds no parameters, and costs about the same training time as GRPO. If the benefit held up, it would be easy to adopt.

S2. Consistent gains over GRPO in the reported experiments. The improvement appears across several base models, datasets, group sizes, and seeds.

Critical weaknesses

W1. The filter does not do what the paper claims. With Q and R fixed and the filter reset for each prompt, the filter's variance never depends on the rewards. The "uncertainty" used to scale the advantage is just a constant determined by a rollout's position in the group. With such a small Q, the baseline is also essentially the average of the rewards seen so far, which uses fewer samples than the GRPO group mean rather than giving a better estimate. This runs against the paper's main motivation, and the paper does not explain where the gains actually come from.

W2. The evaluation is non-standard. Test accuracy gives half credit whenever the ground-truth answer appears anywhere in the output. The base Llama-3.2-1B-Instruct scores only 3.1% on MATH500, far below the 30.6% that Meta reports on MATH for the same model. Results under a standard evaluation are needed to judge the real improvement.

W3. The experimental scope is narrow. All accuracy results come from models of at most 1.5B parameters, mostly on arithmetic and filtered short OpenMath problems; the larger models are shown only through training reward curves. KRPO is also compared only with GRPO and older baselines, not with other GRPO variants that target the same variance problem, such as the leave-one-out baseline (RLOO).

Questions for authors
  1. Is the KRPO baseline actually closer to a prompt's expected reward than the group mean, for example when measured against an estimate from many rollouts?

  2. How does GRPO perform if its advantages are rescaled to the same magnitude as KRPO's?

  3. Can the main comparisons be reported under standard evaluation, that is, greedy decoding, equivalence-based answer matching, and no partial credit?

Reviewer_asLvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes Kalman Filter Enhanced Group Relative Policy Optimization (KRPO), which introduces a new way of computing advantage in the LLM RL algorithms like GRPO. Specifically, GRPO uses the average reward of a group of sampled responses as the baseline, and computes each response’s advantage by subtracting this baseline from its reward and dividing by the group’s standard deviation. The authors argue that this way can be unreliable when the group is small. KRPO instead reads the rewards in a group one by one with a Kalman filter. KRPO keeps an estimate of the baseline and its uncertainty. It computes each response’s advantage via making the reward minus the estimated baseline, divided by the square root of the estimated uncertainty. In terms of the training and evaluation results on Llama-3.2-1B-Instruct and Qwen-0.5/1.5B models shown in the manuscript, KRPO clearly outperforms GRPO on the math tasks.

Critical strengths

This method is simple and easy to integrate in the existing RL training frameworks by only replacing the computation of advantage, so the community can easily test this method on different LLM RL algorithms.

Critical weaknesses

(1) I find that the advantage in KRPO depends on the order of responses, which is actually random. The responses for a prompt have no natural order, and the filter is reset for every prompt, so it has no more information than the plain group mean. But KRPO compares each response only with the ones before it, and then divides by an uncertainty term that does not depend on the rewards. In the released code, the same set of rewards can lead to very different advantages just by changing the order. The first response in each group also gets almost no signal, and a group where all answers are correct still gets positive advantages. The manuscript says the method is not sensitive to the order, but I do not see any experiment supports this claim. I also believe the gains come from these side effects, such as much larger advantages and a weaker KL penalty in effect.

(2) I have to say the experiments shown are very limited, and the code does not match the paper as well. The main results use one 1B model and one seed, and mostly on simple arithmetic and short math questions. MATH500 and AIME are split into train and test sets by the authors instead of being used as held-out benchmarks. There is also no comparison with other methods of computing the baseline. More importantly, in terms of the released code, the wrong answers with the right format get a reward of 0.01, not 0 as stated in the paper. The evaluation script is also problematic, which reports the average reward with sampling at temperature 1.0 rather than accuracy, and it gives half credit whenever the correct number appears in the output. Therefore, the results shown in Table 1 is also not convincing.

Questions for authors

Could the authors compare KRPO with GRPO with a smaller KL weight and with advantages scaled to a similar size? We wanna know whether the gain brought by KRPO shown in the manuscript is from the baseline rather than the scale.

Reviewer_21qFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Kalman Filter Enhanced Group Relative Policy Optimization (KRPO), which replaces GRPO's group-mean-based advantage computation with a one-dimensional Kalman filter. The filter recursively estimates a latent reward baseline and its posterior uncertainty, and the resulting normalized residual is used as the advantage in a clipped policy optimization objective. The authors evaluate KRPO on arithmetic and mathematical reasoning datasets, using Llama 3.2 and Qwen2.5 models, and report improved training reward curves and accuracy compared with GRPO and several additional baselines.

Critical strengths
  1. The method is simple and requires neither a learned critic nor additional trainable parameters.

  2. The experiments cover multiple model sizes and mathematical reasoning settings.

Critical weaknesses
  1. The comparisons focus on PPO and vanilla GRPO, omitting directly relevant alternatives such as Dr. GRPO, DAPO, RLOO, and REINFORCE++. Appendix D’s DPO and fixed-baseline controls do not fill this gap. Moreover, with fixed initialization, QQ, and RR, Pi∣iP_{i|i} depends on sample position rather than observed rewards. KRPO changes normalization, scale, and order-dependent weighting alongside the baseline, without isolating their effects. Relevant method comparisons and matched normalization/scale ablations are needed to attribute the gains specifically to uncertainty-aware filtering.

  2. Improved training stability is not directly quantified. The evidence mainly consists of smoothed training-return curves and illustrative analyses. Figure 7’s larger KL divergence and pre-clipping gradient norms do not establish reduced training fluctuations or more stable optimization. Faster convergence is also assessed qualitatively, without a defined convergence criterion.

  3. The evaluation has limited scope and comparability with prior work. Test-accuracy results cover only 0.5B–1.5B models; the 3B/7B experiments report training returns but no held-out accuracy. Arithmetic and OpenMath-Instruct questions are filtered to at most 128 characters, without reporting retained sample counts or assessing representativeness. MATH500 is repartitioned into an 80/20 train–test split, and the half-credit string-matching metric differs from conventional answer-correctness scoring. These choices limit direct comparison with published benchmark results and leave effectiveness under standard evaluation protocols unestablished.

Questions for authors
  1. How does KRPO compare with Dr. GRPO, RLOO, REINFORCE++, and DAPO under matched training budgets? How were hyperparameters selected for each method?

  2. Does the GRPO baseline use standard-deviation normalization? Can normalization and scale-matched controls, together with a permutation test, isolate the contribution of Kalman filtering?

  3. What observations were used for Table 1’s significance tests? What test-set sizes, generations per question, and aggregation procedure produced Table 3’s accuracies?

  4. How many examples remain after the 128-character filter? Do the gains persist on full held-out MATH500 with conventional answer-correctness scoring, and what are the held-out accuracies for the 3B/7B models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.