acceptodds
Withdrawn from ICLR 2027

SGRPO: Optimizing State-Space Language Models with State-Isolated Group-Relative Policy

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

article iclr2027_conference,times math_commands.tex amsmath,amssymb graphicx booktabs hyperref url SGRPO: Optimizing State-Space Language Models with State-Isolated Group-Relative Policy Anonymous Author(s) Affiliation withheld for double-blind review %\iclrfinalcopy % document \maketitle abstract Group-relative policy optimization (\GRPO) assigns each sampled response an advantage by standardizing its reward against a group of rollouts generated from the same prompt. The derivation of this estimator requires the rollouts in a group to be independent samples from the current policy—a condition Transformer policies satisfy, since each rollout re-encodes the prompt and retains no state between generations. State-space policies such as Mamba break this condition: because they carry state recurrently, decoding a group's rollouts sequentially against one shared generation cache, an efficiency choice that avoids re-encoding the prompt, lets each layer's recurrent state and causal-convolution buffer carry forward from one rollout to the next. Each rollout after the first therefore begins not from the clean post-prompt state produced by the prompt alone, but from the state left behind by the preceding rollout. Under a non-degenerate reward map, this carryover gives the estimator an order-dependent bias that, through the shared group baseline, reaches even the uncontaminated first rollout. The correction, state isolation, restores each layer's recurrent state and convolution buffer to their post-prompt values before every rollout, reinstating exact independence at zero additional FLOPs and leaving the group-relative objective unchanged. packages this correction with token-level normalization and future-KL token weighting. The correction is worth adopting for its exactness and negligible cost alone, regardless of its effect on downstream accuracy. abstract %references %iclr2027_conference document

Primary area:
reinforcement learning
Keywords:
reinforcement learning, GRPO, group-relative policy optimization, state-space models, Mamba, RL post-training, policy optimization, advantage estimation, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_dq7rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that when an SSM policy’s GRPO rollouts are decoded one after another against a shared cache, each rollout inherits the previous rollout’s recurrent state. This breaks the i.i.d. assumption and biases every advantage in the group. The proposed fix, state isolation, snapshots the post-prompt cache and restores it before each rollout. SGRPO combines this fix with DAPO- and FIPO-style components. Experiments on mamba-130m-hf measure the state drift and the cost of isolation, and no downstream accuracy is reported.

Critical strengths

The problem is important. As RL post-training extends to SSM and hybrid models, a shared cache can silently break GRPO’s i.i.d. assumption. However, given the thin related work, I can’t tell whether this issue has been raised before.

Critical weaknesses
  • The writing and figures look AI-generated. Key terms are undefined or used before they are defined. For example, the abstract’s “non-degenerate reward map” is never defined, and “readout injectivity” appears in the Introduction but is only defined in Section 4.2. Overall, the paper is poorly written and does not meet academic standards at all.
  • In experiments, nothing shows that the problem affects RL results or that the fix helps. No downstream accuracy is reported, and SGRPO itself is never evaluated.
Questions for authors
  • What does “updated in place rather than recomputed” mean? A Transformer’s KV cache isn’t recomputed either, so which property of the SSM cache causes the leakage?
  • Figure 2 is designed rather than showcasing the performance improvement. With state isolation, the similarity is always 1 by design, since the method restores the same starting state before every rollout.
Reviewer_htUkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates cross-rollout state contamination when group-relative policy optimization is applied to state-space language models using a shared, mutable generation cache. It proposes restoring both the recurrent states and convolution buffers to their post-prompt values before each rollout. The resulting procedure, SGRPO, combines this state isolation with token-level normalization and future-KL weighting. Experiments on Mamba-130M demonstrate changes in cached states and early-termination behavior, while theoretical analysis characterizes the loss of independent sampling under additional assumptions. Downstream task performance and a comprehensive evaluation of SGRPO are not reported.

Critical strengths
  1. The paper identifies a potential state-contamination issue in shared-cache generation and provides a simple correction.
  2. The authors clearly acknowledge the limitations of their experiments and theoretical assumptions.
Critical weaknesses
  1. The paper has limited novelty. The main contribution is restoring the post-prompt cache before each rollout, which appears to be a straightforward implementation correction. The other components, token-level normalization and future-KL weighting, are adopted from existing methods. The paper does not clearly establish a substantial methodological contribution beyond correct cache management.

  2. The paper presents SGRPO as a training method but lacks a complete evaluation, baseline comparisons, and ablations.

  3. The presentation and formatting are poor. For instance, the PDF is missing the standard header, “Under review as a conference paper at ICLR 2027”, and the abstract on the submission page contains LaTeX package commands.

I think that this submission appears preliminary and incomplete.

Questions for authors

See Weaknesses.

Reviewer_HyiTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies cross-rollout state contamination when multiple responses to the same prompt are generated sequentially using a shared, mutable cache in Mamba and other state-space models. It proposes restoring the post-prompt cache before each rollout and incorporates this procedure into a GRPO-based training objective.

Critical strengths

N/A

Critical weaknesses

My main concern is that the paper does not sufficiently establish the practical relevance or architectural specificity of the problem.

First, GRPO requires multiple rollouts, but it does not require those rollouts to inherit one another’s generation state. Independent rollouts can be generated in parallel, or sequentially with separate or appropriately restored caches. Why should the shared-cache procedure studied here be considered a practically relevant baseline? The paper explicitly states that it has not identified a public framework that uses this procedure. Without such evidence, the experiments appear to demonstrate the consequences of an incorrectly implemented sampler rather than an established limitation of applying GRPO to SSMs.

Second, this issue is not unique to SSMs. A Transformer would also condition later rollouts on earlier responses if their tokens remained in the KV cache. Conversely, both architectures can generate independent rollouts when their caches are managed correctly. The paper’s claim that Transformers satisfy the independence requirement by construction, whereas SSMs violate it, therefore conflates an architectural distinction with a cache-management choice.

These concerns do not invalidate Section 4’s analysis under its stated shared-cache assumptions. They do, however, leave unclear whether that analysis addresses a meaningful problem in existing training systems.

Finally, the proposed correction is straightforward: save the state after processing the prompt and restore it before generating each response. This ensures that each rollout starts from the intended context, but the paper does not establish why this constitutes a substantive methodological advance beyond correct cache management. Combining the correction with existing loss-normalization and token-weighting techniques does not resolve this novelty concern.

To strengthen the contribution, the authors should identify a real training pipeline affected by this issue, explain why existing independent-rollout implementations are insufficient, and demonstrate benefits against a correctly implemented GRPO baseline.

Questions for authors

See Critical Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.