acceptodds
Under review as a conference paper at ICLR 2027

JEPA-Guided Soft Grouping for Step-Level Credit Assignment in Long-Horizon LLM Agents

Abstract

Long-horizon LLM agents trained with outcome rewards face a central credit-assignment challenge: final success or failure does not reveal which intermediate decisions contributed to the outcome. Step-level grouping provides finer-grained supervision by comparing the subsequent returns of intermediate steps across trajectories, making group construction central to effective credit assignment. However, constructing groups through exact observation text matching excludes comparisons between states that share task-relevant context, such as holding the same object for the same goal, but differ in their descriptions of the surroundings. We introduce JEPA-guided soft grouping to recover such comparisons in a learned feature space while retaining the original exact text groups. A task-conditioned JEPA state encoder is trained through future-latent prediction, and current-state feature similarity is used to retrieve neighbors from other trajectories of the same task, forming soft comparison sets. Differences between each step’s observed discounted return and those of its neighbors yield supplementary step-level credit. Because feature similarity alone does not guarantee reliable comparisons, a selective gate controls the admission of this credit and a bounded residual limits its magnitude, while preserving the original episode-level and exact step-level advantages. On ALFWorld with Qwen2.5-1.5B-Instruct, our method achieves 92.97% success—the highest among the configurations in our main comparison—versus 87.76% for the reproduced exact-grouping baseline (+5.21 percentage points). These results support supplementing exact text grouping with learned feature space comparisons for step-level credit assignment.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, credit assignment, LLM agents, long-horizon decision making, predictive representation learning, joint-embedding predictive architecture (JEPA), soft grouping

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ppjhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JEPA-guided soft grouping for step-level credit assignment in long-horizon LLM agents. Building upon GiGPO, which constructs step-level comparison groups using exact observation matching, the proposed method learns task-conditioned state representations through an action-conditioned, multi-horizon JEPA objective. These representations are used to retrieve semantically similar states from different trajectories, enabling additional return-based comparisons beyond exact textual matches.

To mitigate potentially unreliable comparisons, the method introduces a selective gate and a bounded residual mechanism that supplement the original GiGPO advantages without replacing them. Experiments on ALFWorld using Qwen2.5-1.5B-Instruct report a success rate of 92.97%, compared with 87.76% for the reproduced GiGPO baseline. The paper also includes component studies, representation-learning comparisons, and analyses of soft-group coverage.

Critical strengths
  1. Well-motivated problem formulation and a practical extension of existing credit assignment methods. The paper identifies a meaningful limitation of exact observation matching in GiGPO: textually different observations may represent similar task-relevant states, yet cannot contribute to the same comparison group. The proposed soft-grouping mechanism addresses this limitation while preserving the original episode-level and exact step-level advantages. The selective gate and bounded residual provide a reasonable safeguard against unreliable approximate comparisons.
  2. Encouraging empirical improvements with useful implementation and diagnostic details. The proposed method improves the reported ALFWorld success rate from 87.76% to 92.97% over reproduced GiGPO, with particularly notable gains on two-object placement tasks. The paper also provides analyses of retrieval coverage, credit participation, and several design components. These experiments offer useful evidence that the additional grouping mechanism can influence policy optimization and improve performance in the evaluated setting.
Critical weaknesses
  1. The advantage of using JEPA over simpler methods is not sufficiently demonstrated. The main contribution of this paper is using JEPA to identify similar states for step-level credit assignment. However, it is unclear whether JEPA is actually necessary. Simpler methods, such as pretrained text embeddings, might achieve similar results without training an additional JEPA model. Although the paper compares JEPA with SimCSE and CPC, it does not include a direct comparison with these simpler alternatives. Therefore, the experiments do not convincingly show that JEPA itself is responsible for the performance improvement.
  2. The experimental evidence is not strong enough to establish the reliability of the improvements. The final method is tested using only one training run, so it is unclear whether the reported improvement would hold across different training seeds. In addition, all experiments are conducted on ALFWorld, using the valid-seen split that was also used during method development. Without evaluation on unseen tasks or additional training runs, it is difficult to determine how robust and generalizable the method is.
Questions for authors
  1. Why is JEPA necessary for the proposed soft grouping method? The paper shows that JEPA-guided grouping improves performance over GiGPO, but it remains unclear whether the improvement comes from JEPA itself or simply from allowing comparisons between non-identical observations. Could the authors compare JEPA with a simpler alternative, such as pretrained text embeddings or basic semantic similarity? This would help establish whether JEPA provides a meaningful advantage over simpler grouping methods.
  2. How reliable are the reported performance improvements? The final method is evaluated using only one independent training run, and the evaluation uses the ALFWorld valid-seen split, which was also visible during method development. Could the authors provide results from additional training seeds or the valid-unseen split? This would help determine whether the reported improvements are robust and generalize beyond the development setting.
1 comment
AuthorsOpenReview ↗

Response to Reviewer ppjh: retrieval baselines and additional WebShop results

Thank you for recognizing the motivation of soft grouping and for raising the questions of representation choice and empirical reliability. We have conducted additional experiments addressing both questions. Below we highlight supplementary results; complete results for all tested seeds and full experimental protocols will be included in the revised manuscript.

  1. Pretrained text embeddings and simple semantic similarity.

We tested two alternatives to JEPA for neighbor retrieval: (i) a frozen sentence-transformers/all-mpnet-base-v2 encoder and (ii) cosine similarity over masked-mean-pooled final-layer hidden states of the current Qwen policy, without an additional learned encoder. Similarity cutoffs were calibrated on training replay before policy training to approximately match valid-neighbor coverage. Policies were evaluated at the fixed final checkpoint after 150 updates.

WebShop results: Retrieval representation Training seed success (%) JEPA 72.14 Frozen MPNet 70.31 Current-policy hidden-state cosine 67.97

  1. Additional environment and training-seed evidence.

We additionally evaluated Qwen2.5-1.5B-Instruct on WebShop. The tables report baseline → baseline+JEPA successes out of 384 matched evaluation episodes per training seed (evaluation seeds 123/456/789, 128 episodes each), at update 150.

GiGPO with direct JEPA residual fusion: Training seed Successes / 384 Change (percentage points) 0 262 → 279 +4.43 1 274 → 277 +0.78 2 268 → 281 +3.39

These three reported GiGPO pairs have higher endpoint success with the JEPA configuration; their mean success is 69.79% → 72.66%.

results for integration with other grouping methods: Configuration Training seed 0 Training seed 1 HGPO + JEPA, taskwise RMS cap 283 → 286 (+0.78 pp) 283 → 294 (+2.86 pp) Adapted GraphGPO + JEPA, taskwise RMS cap 264 → 274 (+2.60 pp) 255 → 275 (+5.21 pp)

The HGPO/GraphGPO configurations use RMS-budgeted additive residuals, rather than the original ALFWorld gate/cap configuration. The GraphGPO row uses our adapted implementation. The WebShop experiments provide complementary evidence in another environment and across training seeds.

  1. ALFWorld evaluation protocol.

Our ALFWorld comparison uses valid_seen (in-distribution), consistent with GiGPO's released evaluation setup. GraphGPO's released evaluation configuration also selects in-distribution evaluation. HGPO Table 1 reports both in- and out-of-distribution results, including the seen setting.This clarifies comparability with prior work, while leaving your generalization concern valid. We acknowledge that valid_seen was also used during development.

Reviewer_6Ex9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work extends GiGPO with a "soft" grouping step. A small JEPA encoder learns from the policy's own rollouts and retrieves similar states from other trajectories for the same task. Return differences relative to these neighbours provide additional step-level credit, subject to a gate and a cap. With Qwen2.5-1.5B-Instruct on ALFWorld, the reported success rate is 92.97%; the reproduced GiGPO result is 87.76%.

Critical strengths

Relaxing GiGPO's requirement for exact text matching makes sense, and the original GiGPO credit stays unchanged.

The method adds no cost at inference time.

Critical weaknesses

My main reasons for leaning towards rejection are the limited contribution and the lack of experiments explaining the gain.

  1. The method involves substantial engineering for a limited core contribution. It is complicated by many hand-set thresholds and schedules, most of which have no ablation. Appendix C says these "were engineering choices frozen through policy-training pilots".

  2. The writing is very defensive. Qualifications such as "not a semantic action parser" and "they do not guarantee policy improvement" make it tiring to read. Both the abstract and the conclusion claim that the method is "the highest among the configurations in our main comparison". Which configurations does this cover? HGPO and GraphGPO settings in Appendix B.1 already reach or exceed 92.97%. Without the ablations mentioned in point 1, this ranking tells us little.

  3. According to Appendix C, the benchmark split was visible during development. The authors chose the final configuration after inspecting exploratory results and describe the split as "not an untouched test set". The code keeps the test split out of training, but uses it for validation every five iterations as well as for the final evaluation.

  4. The reference code resumes at step 50 with a fresh optimizer and uses two GPUs. The reference run described in the paper did neither. Given this mismatch, the paper does not provide enough information to reproduce Tables 2-4.

Questions for authors

Could you repeat training with random embeddings in place of JEPA, keeping the rest of the method unchanged? Please match the number of retrieved neighbours approximately to the JEPA run. If performance stays about the same, the gain does not come from JEPA.

I find the method heavily engineered and its novelty unclear. Please test sensitivity to the main engineering parameters. The JEPA design also lacks ablations for its 1/4/8-step prediction, action conditioning, EMA target and variance regularizer. Which of these choices matter for the results?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6Ex9: random retrieval, horizon diagnostics, and additional results

Thank you for the concrete suggestions. We have completed a random-retrieval control and an offline horizon study, and will revise the presentation. Below we highlight selected supplementary results; complete tested-seed results and protocols will be included in the revised manuscript.

  1. Random embeddings with matched retrieval support.

We completed a 150-update WebShop HGPO experiment replacing learned retrieval with the frozen, randomly initialized encoder of the same architecture and serializer. Return labels, credit normalization, schedules, and the RMS budget were retained. A learned shadow supplied only support counts; random embeddings determined neighbor identities and ranking. Valid-query counts matched the shadow (29,025 each), without borrowing its donor identities.

Selected training seed 1, fixed update 150, same 384 ordered evaluation episodes: Retrieval representation Successes / 384 Success (%) Frozen random 277 72.14 Learned JEPA 294 76.56

Learned JEPA is higher by 17 successes (+4.43 percentage points).

  1. Horizon study and design choices.

We completed a fixed-replay ALFWorld study of horizons 1/4/8. Each single-horizon encoder received 600 updates from a shared initialization per auxiliary seed, using three auxiliary initialization seeds and common anchor states/actions. Encoder learning and probe testing used instruction-disjoint partitions. Frozen-encoder probes measured discounted-return prediction and trajectory-success discrimination: Horizon Return MAE (lower is better) Success-probe AUROC 1 0.1649 ± 0.0011 0.6260 4 0.1656 ± 0.0006 0.6401 8 0.1663 ± 0.0014 0.6300

MAE is mean ± sample SD across auxiliary seeds. Horizon 4 has the highest observed probe AUROC, while horizon 1 has the lowest MAE. These are offline representation diagnostics, not agent success rates or policy-training ablations. For action conditioning, EMA targets, and variance regularization, we will clarify that these are adopted representation-learning ingredients, not separate innovations. The EMA teacher follows I-JEPA; action conditioning has precedent in V-JEPA 2-AC; the explicit variance penalty is a VICReg-style anti-collapse choice. They are not all unchanged components of original I-JEPA.

For engineering sensitivity, existing Table 3 reports mean success of 85.94/92.45/91.67% for beta_max=0.10/0.25/0.50. This supports 0.25 among the tested settings, not a universal optimum. We will describe the gate/cap as engineering safeguards and narrow necessity claims.

  1. Writing and supplementary WebShop results.

We agree that repeated defensive qualifications obscure the method. We will simplify the abstract, method, and conclusion; consolidate limitations; and replace the ambiguous highest-performing claim with an explicit comparison scope that distinguishes Appendix baselines.

Additional Qwen2.5-1.5B-Instruct WebShop results below are baseline → baseline+JEPA successes out of 384 matched evaluation episodes at update 150 (evaluation seeds 123/456/789, 128 each).

GiGPO with direct JEPA residual fusion: Training seed Successes / 384 Change (pp) 0 262 → 279 +4.43 1 274 → 277 +0.78 2 268 → 281 +3.39

Mean success is 69.79% → 72.66%. Integrations with other grouping methods: Configuration Training seed 0 Training seed 1 HGPO + JEPA, taskwise RMS cap 283 → 286 (+0.78 pp) 283 → 294 (+2.86 pp) Adapted GraphGPO + JEPA, taskwise RMS cap 264 → 274 (+2.60 pp) 255 → 275 (+5.21 pp)

These use direct GiGPO fusion or taskwise RMS budgets, rather than the original ALFWorld gate/cap stack. GraphGPO uses our adapted implementation.

I thank the authors for the new WebShop runs and the random-embedding control. The control passes the test I proposed, and WebShop adds evidence from a second environment. However, Q2 and W4 remain unresolved. The PDF on OpenReview is still the earlier version, so I cannot check the promised changes to the abstract, conclusion, or comparison scope, and I continue to lean towards rejection.

Reviewer_LcurOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets a loophole of GiGPO's exact text grouping: only textually identical states are grouped for comparison, so semantically equivalent states with different wording (same stage, same held objects) are missed, and step-level credit assignment loses a large number of comparable samples. On top of GiGPO, the authors add a complementary path: representations learned by JEPA retrieve similar states to compute soft credit for states that have no exact grouping partners, and the credit is injected only after a four-condition gate and a magnitude cap, so that it neither overrides nor reverses the original exact-grouping signal. Evaluated with Qwen2.5-1.5B on the valid_seen split of ALFWorld, the final configuration reaches 92.97 success, 5.21 points above the authors' own reproduced GiGPO.

Critical strengths
  • The problem is precisely and concretely located: Section 1 points out that exact text grouping misses comparable states at the same stage with the same held objects but different wording, and the three retrieval examples in Figure 4 show that such states do exist.
  • The propose/admit separation is conservative and credible: the four-condition gate in Eq. 16 plus the residual cap bounded by |Ei| in Eq. 17/18 guarantee that soft credit neither overrides nor reverses non-zero exact credit.
Critical weaknesses
  1. The validation is too narrow: a single environment and a single small model, with no evidence of generality. All training-level evidence comes from ALFWorld valid_seen with Qwen2.5-1.5B; ALFWorld is already an overly simple environment at this point, and conclusions drawn on a 1.5B model do not extrapolate to larger models or to web and SWE environments. There is no evidence for the method's general capability.

  2. Limited novelty. JEPA representation retrieval is an off-the-shelf component, soft credit is a patch-style extension of the GiGPO framework, and the gate-and-cap is engineering hardening rather than a new mechanism; the combination yields no insight beyond the intuition of "retrieve similar states to complete the grouping" itself.

  3. The gate-and-stack design is under-motivated, with limited mathematical guarantees. Why is the stacking of a four-condition gate plus a residual cap needed? The paper only proves that soft credit neither overrides nor reverses non-zero exact credit (Eq. 16-18), a conservativeness guarantee that answers neither why the design is necessary nor whether it is optimal; the individual contributions of the stacked components are never isolated.

Questions for authors
  1. All training-level evidence comes from ALFWorld valid_seen with Qwen2.5-1.5B: can the authors add training-level validation with a larger model (e.g., 7B) and with environments beyond ALFWorld? If not, how should the extrapolation scope of the conclusions be delimited?

  2. For the stacked design of the four-condition gate plus the residual cap, what is the necessity of each layer, and can the authors provide per-component isolated ablations showing their respective contributions? Given that the gate ablation gaps are small and within noise, how can the possibility that the gate itself has no substantial effect be ruled out?

  3. Compared with an off-the-shelf retrieval representation that does not rely on JEPA (e.g., cosine similarity over generic sentence embeddings), how much does the JEPA component add? If a simpler similarity metric performs comparably, how should the paper's technical contribution be positioned?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Lcur: WebShop validation, engineering safeguards, and retrieval baselines

Thank you for identifying the limitation of exact grouping and for your constructive questions. We respond below with selected positive supplementary results. Complete results for all tested seeds and full protocols will be included in the revised manuscript.

Q1. Validation beyond ALFWorld and scope of generalization.

We have added full policy-training experiments on WebShop with Qwen2.5-1.5B-Instruct. Results below are baseline → baseline+JEPA successes out of 384 matched evaluation episodes per training seed, evaluated at update 150 (evaluation seeds 123/456/789, 128 episodes each).

GiGPO with direct JEPA residual fusion: Training seed Successes / 384 Change (percentage points) 0 262 → 279 +4.43 1 274 → 277 +0.78 2 268 → 281 +3.39

The JEPA configurations have higher endpoint success in these three pairs; mean success is 69.79% → 72.66%.

Integrations with other grouping methods: Configuration Training seed 0 Training seed 1 HGPO + JEPA, taskwise RMS cap 283 → 286 (+0.78 pp) 283 → 294 (+2.86 pp) Adapted GraphGPO + JEPA, taskwise RMS cap 264 → 274 (+2.60 pp) 255 → 275 (+5.21 pp)

Q2. Role and necessity of the gate/cap design.

We agree that the four-condition gate and residual cap are engineering safeguards, rather than a standalone methodological innovation. The gate controls when an approximate comparison is admitted; the cap controls the magnitude of its contribution. The associated guarantees concern preservation of the original exact-credit signal, not policy improvement or optimality of this design.

Our existing ablations compare bundles of safeguards and do not isolate every gate condition. Their small performance differences do not establish that each condition is necessary or rule out a limited effect of the gate. We will weaken these claims accordingly.

We also do not claim that this exact stack is required in other environments. The direct-fusion WebShop configurations above do not use the original ALFWorld gate/cap stack, while the HGPO/GraphGPO integrations use a different, taskwise RMS budget. Thus the fusion safeguards should be treated as environment-dependent implementation choices. We will position the contribution around extending step-level comparisons beyond exact textual matches, with JEPA as the studied predictive representation, rather than the gate/cap engineering itself.

Q3. Simpler retrieval representations and JEPA's contribution.

We have tested both alternatives you suggested: (i) frozen pretrained text embeddings from sentence-transformers/all-mpnet-base-v2, and (ii) cosine similarity over masked-mean-pooled final-layer hidden states of the current Qwen policy, without an additional learned encoder. Cutoffs were calibrated on training replay before policy training to approximately match valid-neighbor coverage.

Selected WebShop results at update 150, on the same 384-episode evaluation panel: Retrieval representation Training seed 1 success (%) JEPA 72.14 Frozen MPNet 70.31 Current-policy hidden-state cosine 67.97

In this selected comparison, JEPA has higher observed success by 1.83 and 4.17 percentage points, respectively.

Thank you for your detailed response. The additional WebShop experiments, comparisons with non-JEPA representations, and clarification of the gate/cap design address some of my concerns. However, the evidence is still based on a single small model and a limited set of results, so I have decided to keep my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.