- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes JEPA-guided soft grouping for step-level credit assignment in long-horizon LLM agents. Building upon GiGPO, which constructs step-level comparison groups using exact observation matching, the proposed method learns task-conditioned state representations through an action-conditioned, multi-horizon JEPA objective. These representations are used to retrieve semantically similar states from different trajectories, enabling additional return-based comparisons beyond exact textual matches.
To mitigate potentially unreliable comparisons, the method introduces a selective gate and a bounded residual mechanism that supplement the original GiGPO advantages without replacing them. Experiments on ALFWorld using Qwen2.5-1.5B-Instruct report a success rate of 92.97%, compared with 87.76% for the reproduced GiGPO baseline. The paper also includes component studies, representation-learning comparisons, and analyses of soft-group coverage.
Critical strengths
- Well-motivated problem formulation and a practical extension of existing credit assignment methods. The paper identifies a meaningful limitation of exact observation matching in GiGPO: textually different observations may represent similar task-relevant states, yet cannot contribute to the same comparison group. The proposed soft-grouping mechanism addresses this limitation while preserving the original episode-level and exact step-level advantages. The selective gate and bounded residual provide a reasonable safeguard against unreliable approximate comparisons.
- Encouraging empirical improvements with useful implementation and diagnostic details. The proposed method improves the reported ALFWorld success rate from 87.76% to 92.97% over reproduced GiGPO, with particularly notable gains on two-object placement tasks. The paper also provides analyses of retrieval coverage, credit participation, and several design components. These experiments offer useful evidence that the additional grouping mechanism can influence policy optimization and improve performance in the evaluated setting.
Critical weaknesses
- The advantage of using JEPA over simpler methods is not sufficiently demonstrated. The main contribution of this paper is using JEPA to identify similar states for step-level credit assignment. However, it is unclear whether JEPA is actually necessary. Simpler methods, such as pretrained text embeddings, might achieve similar results without training an additional JEPA model. Although the paper compares JEPA with SimCSE and CPC, it does not include a direct comparison with these simpler alternatives. Therefore, the experiments do not convincingly show that JEPA itself is responsible for the performance improvement.
- The experimental evidence is not strong enough to establish the reliability of the improvements. The final method is tested using only one training run, so it is unclear whether the reported improvement would hold across different training seeds. In addition, all experiments are conducted on ALFWorld, using the valid-seen split that was also used during method development. Without evaluation on unseen tasks or additional training runs, it is difficult to determine how robust and generalizable the method is.
Questions for authors
- Why is JEPA necessary for the proposed soft grouping method? The paper shows that JEPA-guided grouping improves performance over GiGPO, but it remains unclear whether the improvement comes from JEPA itself or simply from allowing comparisons between non-identical observations. Could the authors compare JEPA with a simpler alternative, such as pretrained text embeddings or basic semantic similarity? This would help establish whether JEPA provides a meaningful advantage over simpler grouping methods.
- How reliable are the reported performance improvements? The final method is evaluated using only one independent training run, and the evaluation uses the ALFWorld valid-seen split, which was also visible during method development. Could the authors provide results from additional training seeds or the valid-unseen split? This would help determine whether the reported improvements are robust and generalize beyond the development setting.
Response to Reviewer ppjh: retrieval baselines and additional WebShop results
Thank you for recognizing the motivation of soft grouping and for raising the questions of representation choice and empirical reliability. We have conducted additional experiments addressing both questions. Below we highlight supplementary results; complete results for all tested seeds and full experimental protocols will be included in the revised manuscript.
We tested two alternatives to JEPA for neighbor retrieval: (i) a frozen sentence-transformers/all-mpnet-base-v2 encoder and (ii) cosine similarity over masked-mean-pooled final-layer hidden states of the current Qwen policy, without an additional learned encoder. Similarity cutoffs were calibrated on training replay before policy training to approximately match valid-neighbor coverage. Policies were evaluated at the fixed final checkpoint after 150 updates.
WebShop results: Retrieval representation Training seed success (%) JEPA 72.14 Frozen MPNet 70.31 Current-policy hidden-state cosine 67.97
We additionally evaluated Qwen2.5-1.5B-Instruct on WebShop. The tables report baseline → baseline+JEPA successes out of 384 matched evaluation episodes per training seed (evaluation seeds 123/456/789, 128 episodes each), at update 150.
GiGPO with direct JEPA residual fusion: Training seed Successes / 384 Change (percentage points) 0 262 → 279 +4.43 1 274 → 277 +0.78 2 268 → 281 +3.39
These three reported GiGPO pairs have higher endpoint success with the JEPA configuration; their mean success is 69.79% → 72.66%.
results for integration with other grouping methods: Configuration Training seed 0 Training seed 1 HGPO + JEPA, taskwise RMS cap 283 → 286 (+0.78 pp) 283 → 294 (+2.86 pp) Adapted GraphGPO + JEPA, taskwise RMS cap 264 → 274 (+2.60 pp) 255 → 275 (+5.21 pp)
The HGPO/GraphGPO configurations use RMS-budgeted additive residuals, rather than the original ALFWorld gate/cap configuration. The GraphGPO row uses our adapted implementation. The WebShop experiments provide complementary evidence in another environment and across training seeds.
Our ALFWorld comparison uses valid_seen (in-distribution), consistent with GiGPO's released evaluation setup. GraphGPO's released evaluation configuration also selects in-distribution evaluation. HGPO Table 1 reports both in- and out-of-distribution results, including the seen setting.This clarifies comparability with prior work, while leaving your generalization concern valid. We acknowledge that valid_seen was also used during development.