- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper argues that GRPO lets the policy's representation of its answer drift away from the gold-answer semantics because neither the policy loss nor the token-level KL term acts on hidden states. It adds a JEPA-style signal, an advantage-weighted InfoNCE loss in which a policy embedding of the query (read at appended predictor tokens) is pulled toward a frozen-reference embedding of the gold answer and pushed away from reference embeddings of the sampled completions. The signal is used either as an extra loss term (Method A) or as a z-scored controller of the KL coefficient (Method B). Experiments on Spider, GSM8K, HellaSwag, and Yelp with five models from four families report gains over GRPO on all 20 pairs, and gains over SAPO, BNPO, and DAPO on Qwen2.5-3B.
Critical strengths
Using GRPO's own group samples, graded by their advantages, as negatives for an embedding-space objective is a reasonable way to carry an LLM-JEPA style term from supervised fine-tuning into RL post-training, and the Method A gains are consistent across model families.
Critical weaknesses
- The method is an incremental adaptation of LLM-JEPA, and the experiments do not show that the new element, rather than the inherited JEPA term or the use of gold answers, produces the gains. Predictor tokens with the same grid, last-token cosine embeddings, text and code views, the HellaSwag continuation target, and the monitored-loss, singular-value, and t-SNE analyses all come from LLM-JEPA, which leaves the advantage-weighted InfoNCE as the main new component. Its only support is a single-run Spider ablation (Table 5) against uniform InfoNCE. The gold-answer controls are also missing, because GRPO-JEPA introduces the gold answer into training while every baseline (GRPO, SAPO, BNPO, DAPO) sees the reward alone, yet the paper neither cites nor compares with mixed-policy RL such as LUFFY [1] or the SFT, RL with SFT loss, and SFT followed by RL controls, even though all four gold answers are already valid completions. Without these comparisons no gain can be credited to the paper's own contribution, and closing the gap requires new training runs.
- The title claim of mitigating semantic drift is not supported by the evaluated regime. The only behavioral evidence is the Spider shortcut rate (Figure 1a), which is not operationally defined and comes from the paper's own Jaccard partial-credit reward that pays a query for overlapping rows (Section 1, Appendix B.2), so it reflects reward hacking induced by the reward design rather than a property of GRPO, and no drift measure is reported on the other three datasets. The representation-side evidence (Figures 3d to 6) measures , the quantity the loss optimizes, computed from the raw query without the Spider schema and with predictor tokens that never appear at generation (Section 3.2, Appendix B.2), so it does not show that the hidden states used for generation drift less.
[1] Yan et al., Learning to Reason under Off-Policy Guidance
Questions for authors
- How does GRPO-JEPA compare with methods that use the same gold answers, namely LUFFY with the gold answer as the off-policy trajectory, SFT, RL with an SFT loss, and SFT followed by RL on the same 2,000 cases?
- Is semantic drift observed under a standard binary execution-match reward on Spider, or on any of the other three datasets, and how exactly is the shortcut rate in Figure 1(a) computed?
- Method B's stays within to (Section 4.2), and since is not in its objective, its driving signal appears untrained, while is tuned per dataset (Table 8) and the baseline is not stated. Does any gradient reach the predictor tokens in Method B, and how does the GRPO baseline perform at Method B's selected ?