- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that when an SSM policy’s GRPO rollouts are decoded one after another against a shared cache, each rollout inherits the previous rollout’s recurrent state. This breaks the i.i.d. assumption and biases every advantage in the group. The proposed fix, state isolation, snapshots the post-prompt cache and restores it before each rollout. SGRPO combines this fix with DAPO- and FIPO-style components. Experiments on mamba-130m-hf measure the state drift and the cost of isolation, and no downstream accuracy is reported.
Critical strengths
The problem is important. As RL post-training extends to SSM and hybrid models, a shared cache can silently break GRPO’s i.i.d. assumption. However, given the thin related work, I can’t tell whether this issue has been raised before.
Critical weaknesses
- The writing and figures look AI-generated. Key terms are undefined or used before they are defined. For example, the abstract’s “non-degenerate reward map” is never defined, and “readout injectivity” appears in the Introduction but is only defined in Section 4.2. Overall, the paper is poorly written and does not meet academic standards at all.
- In experiments, nothing shows that the problem affects RL results or that the fix helps. No downstream accuracy is reported, and SGRPO itself is never evaluated.
Questions for authors
- What does “updated in place rather than recomputed” mean? A Transformer’s KV cache isn’t recomputed either, so which property of the SSM cache causes the leakage?
- Figure 2 is designed rather than showcasing the performance improvement. With state isolation, the similarity is always 1 by design, since the method restores the same starting state before every rollout.