acceptodds
Under review as a conference paper at ICLR 2027

GRPO-JEPA: Mitigating Semantic Drift in GRPO via Joint Embedding Predictive Architecture

Abstract

Group Relative Policy Optimization (GRPO) is a standard method for RL post-training of large language models, yet its objective stays in the output space: neither the policy gradient nor the token-level KL term observes representation-space dynamics. This blind spot lets semantic drift occur while the loss stays low. We present GRPO-JEPA, which brings the Joint-Embedding Predictive Architecture (JEPA) into LLM RL post-training as explicit representation-space supervision: an advantage-weighted InfoNCE signal turns GRPO's group-sampled completions into multi-view data, propagating scalar reward signals into representation-space alignment. This JEPA supervision signal enters training in two ways: (i) Method A (direct regularization) adds it to the GRPO objective, and (ii) Method B (soft modulation) converts it into a dynamic controller of the KL coefficient. Across four benchmarks (Spider, GSM8K, HellaSwag, and Yelp) and five models from four families (Qwen, OLMo, SmolLM, and Gemma), GRPO-JEPA improves on standard GRPO on all model–dataset pairs: average gains are points for Method A and for Method B. It also surpasses recent GRPO variants (SAPO, BNPO, and DAPO) on every benchmark for Method A and on average for Method B. Representation-space predictive consistency offers a usable JEPA supervision signal for RL post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, joint-embedding predictive architecture, GRPO, reinforcement learning, representation learning, semantic drift, RL post-training

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_pvYSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that GRPO lets the policy's representation of its answer drift away from the gold-answer semantics because neither the policy loss nor the token-level KL term acts on hidden states. It adds a JEPA-style signal, an advantage-weighted InfoNCE loss in which a policy embedding of the query (read at appended predictor tokens) is pulled toward a frozen-reference embedding of the gold answer and pushed away from reference embeddings of the sampled completions. The signal is used either as an extra loss term (Method A) or as a z-scored controller of the KL coefficient (Method B). Experiments on Spider, GSM8K, HellaSwag, and Yelp with five models from four families report gains over GRPO on all 20 pairs, and gains over SAPO, BNPO, and DAPO on Qwen2.5-3B.

Critical strengths

Using GRPO's own group samples, graded by their advantages, as negatives for an embedding-space objective is a reasonable way to carry an LLM-JEPA style term from supervised fine-tuning into RL post-training, and the Method A gains are consistent across model families.

Critical weaknesses
  1. The method is an incremental adaptation of LLM-JEPA, and the experiments do not show that the new element, rather than the inherited JEPA term or the use of gold answers, produces the gains. Predictor tokens with the same k∈{0,…,4}k \in \{0,\dots,4\} grid, last-token cosine embeddings, text and code views, the HellaSwag continuation target, and the monitored-loss, singular-value, and t-SNE analyses all come from LLM-JEPA, which leaves the advantage-weighted InfoNCE as the main new component. Its only support is a single-run Spider ablation (Table 5) against uniform InfoNCE. The gold-answer controls are also missing, because GRPO-JEPA introduces the gold answer into training while every baseline (GRPO, SAPO, BNPO, DAPO) sees the reward alone, yet the paper neither cites nor compares with mixed-policy RL such as LUFFY [1] or the SFT, RL with SFT loss, and SFT followed by RL controls, even though all four gold answers are already valid completions. Without these comparisons no gain can be credited to the paper's own contribution, and closing the gap requires new training runs.
  2. The title claim of mitigating semantic drift is not supported by the evaluated regime. The only behavioral evidence is the Spider shortcut rate (Figure 1a), which is not operationally defined and comes from the paper's own Jaccard partial-credit reward that pays a SELECT *\texttt{SELECT *} query for overlapping rows (Section 1, Appendix B.2), so it reflects reward hacking induced by the reward design rather than a property of GRPO, and no drift measure is reported on the other three datasets. The representation-side evidence (Figures 3d to 6) measures EpredE_{\text{pred}}, the quantity the loss optimizes, computed from the raw query without the Spider schema and with predictor tokens that never appear at generation (Section 3.2, Appendix B.2), so it does not show that the hidden states used for generation drift less.

[1] Yan et al., Learning to Reason under Off-Policy Guidance

Questions for authors
  1. How does GRPO-JEPA compare with methods that use the same gold answers, namely LUFFY with the gold answer as the off-policy trajectory, SFT, RL with an SFT loss, and SFT followed by RL on the same 2,000 cases?
  2. Is semantic drift observed under a standard binary execution-match reward on Spider, or on any of the other three datasets, and how exactly is the shortcut rate in Figure 1(a) computed?
  3. Method B's βdyn\beta_{\text{dyn}} stays within 0.890.89 to 1.16 β01.16\,\beta_0 (Section 4.2), and since LJEPA\mathcal{L}_{\text{JEPA}} is not in its objective, its driving signal appears untrained, while β0\beta_0 is tuned per dataset (Table 8) and the baseline β\beta is not stated. Does any gradient reach the predictor tokens in Method B, and how does the GRPO baseline perform at Method B's selected β0\beta_0?
Reviewer_Ba6bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GRPO-JEPA, which brings the Joint-Embedding Predictive Architecture (JEPA) into LLM RL post-training as explicit representation-space supervision. An advantage-weighted InfoNCE signal turns GRPO’s group-sampled completions into multi-view data, propagating scalar reward signals into representation-space alignment. GRPO-JEPA improves on standard GRPO on all 20 model–dataset pairs.

Critical strengths
  1. The paper is overall well written and easy to follow.
  2. The experiment results show that GRPO-JEPA improves on standard GRPO on all 20 model–dataset pairs. The analysis of the JEPA loss, KL loss, cosine similarity of embedding, and the gradient norm provides insightful evidence.
Critical weaknesses
  1. The attribution of performance gains is not fully clear. Method A uses gold answer representations as auxiliary supervision. A GRPO + SFT baseline is needed to distinguish the benefit of the proposed JEPA objective from the additional dense supervised learning. Also, there are several different papers trying to utilize expert data in GRPO, such as On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting, Learning to Reason under Off-Policy Guidance.

  2. The paper uses both uppercase and lowercase for “table,” “figure,” “section,” etc. Please standardize their capitalization throughout the manuscript.

Questions for authors

I have no other questions, please see the weakness.

Reviewer_21eXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GRPO-JEPA, introducing JEPA-style representation space supervision into GRPO post-training: using advantage-weighted InfoNCE signals, it treats in-group sampling completion as multiple views and uses the gold standard answer encoded by the frozen reference model as the sole positive anchor, thus bringing the predicted embedding closer to the answer embedding and pushing away low-advantage completions. The signal is injected in two ways—directly added to the loss (Method A) or dynamically adjusted KL coefficients (Method B).

Critical strengths
  1. The problem motivation is clear: the objective function of GRPO and token-level KL mapping do not observe representation dynamics, making "semantic drift" a valuable perspective.

  2. The method is simple and low-cost: it reuses existing reference models, adding only one forward propagation, requiring no additional models or a second set of parameters.

  3. The experiments have broad coverage: 20 model-dataset combinations, with the main results reporting standard deviations in 5 seed reports.

Critical weaknesses
  1. Unfair comparison puts the gains in doubt. The GRPO baseline is tuned only on Spider (two models) and then frozen across all datasets and families, while JEPA's λ, pooling, and k are selected per-dataset at their best settings. The Gemma2-2B baseline (16.67% on GSM8K) looks undertuned, so the +11.7 there may mostly reflect fixing a broken baseline rather than JEPA. The claim that "the gain comes from the architecture rather than tuning" is unsupported.

  2. No causal control for the mechanism. All evidence that "representation alignment drives the gain" is correlational (monitoring curves, geometry, effective rank, coupling correlations). Without a wrong-anchor / anchor-shuffle ablation (e.g., random or incorrect-answer anchor), the authors cannot rule out that any well-behaved auxiliary contrastive loss would give similar gains — which is exactly the paper's novelty claim.

  3. The abstract overclaims. "Surpasses SAPO, BNPO, and DAPO on every benchmark" is based on a single model (Qwen2.5-3B) with the variants run at baseline hyperparameters, which §A itself calls a "consistency check." The claim's scope must be narrowed accordingly.

Questions for authors

please refer the weaknesses.

Reviewer_j5cSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes predicting an answer representation from the question and aligning it with the ground-truth answer representation produced by a frozen reference model. Responses sampled from the policy serve as negative examples, with weights determined by their within-group advantages. The paper argues that explicit semantic supervision can mitigate semantic drift that the standard GRPO objective does not sufficiently constrain.

Critical strengths

This paper proposes to use the advantages to weight responses sampled during training is reasonable, and this method decreases the alignment with the lower-quality responses. This allows the ranking via the reward from GRPO to guide the representation learning, Table 5 provides the effectiveness of this method, exponential weighting achieves 70.3% accuracy, compared with 62.1% for uniform weighting and 65.4% for linear weighting. The version without predictor tokens (k=0) still achieves 67.2%, suggesting that the gains do not depend entirely on the added tokens.

Critical weaknesses

1.The reliability of the frozen reference encoder as a semantic anchor is insufficiently validated. Freezing the encoder keeps the target representations stable, but does not ensure that they capture the task-relevant distinctions needed to measure semantic drift. If these representations are inadequate, continued alignment may provide misleading supervision or discourage useful representation changes as the policy improves. The paper does not examine how sensitive the method is to the quality of the reference encoder or whether the alignment constraint remains beneficial in later stages of training.

2.Several passages use formulaic questions and responses. For example, in Section 4.1: “Does explicit semantic supervision help across models and datasets, and does it beat tuned GRPO variants? Table 1 answers both.” This question-and-answer pattern makes the discussion sound scripted and generic. The authors should reduce such AI-like phrasing and revise the manuscript into more natural, concise academic prose, replacing stock transitions and repetitive summaries with specific discussion of the findings and their limitations.

Questions for authors

1.The experiments do not sufficiently isolate the proposed mechanism from additional gold-answer supervision. Method A introduces direct supervision from gold answers, but lacks comparisons with GRPO + SFT or simple positive-only alignment. Since the baselines already use gold answers through rewards, matched controls for the supervision objective are needed. Table 5 only evaluates JEPA variants and does not demonstrate an advantage over simpler auxiliary objectives.

2.It is recommended to compare the gains against GRPO + SFT or positive-only alignment to gold-answer embeddings under matched supervision, rewards, and tuning and training budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.