acceptodds
Under review as a conference paper at ICLR 2027

RACE: Recovery-Aware Credit Estimation for Long-Horizon LLM Agents

Abstract

Group-based reinforcement learning (RL) enables critic-free post-training of large language model (LLM) agents for long-horizon tasks. However, uniform trajectory-level credit conflates useful steps with unproductive actions, while GiGPO-style exact-state grouping misses outcome evidence from semantically comparable contexts with different observations. In this work, we introduce Recovery-Aware Credit Estimation (RACE), a plug-in method that assigns action-level credit through changes in contextual recoverability. Using a frozen text encoder, RACE nonparametrically estimates contextual recoverability from outcome-labeled neighboring contexts retrieved from previously collected rollouts, without training a critic or process reward model. A leave-one-trajectory-out retrieval scheme excludes the query trajectory from its retrieved neighborhood, and the clipped change in estimated recoverability across each transition serves as an action-level credit signal added to the base group-relative advantage. The method requires no additional environment interactions and supports both binary and graded outcomes. Experiments with Qwen3-4B and Qwen3-8B on ALFWorld, WebShop, and ScienceWorld show improvements over matched GiGPO runs across all three benchmarks. With Qwen3-8B, RACE achieves absolute success-rate gains of 7.85% and 7.59% on ALFWorld’s in-distribution and out-of-distribution splits, respectively, and improves the ScienceWorld graded score by 6.04 points and exact success rate by an absolute 14.75%. Further experiments show higher validation-curve AUC across all six benchmark–model settings and improved zero-shot transfer, measured by macro-averaged success and progress across BabyAI, PDDL, and TextCraft.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, reinforcement learning, credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VNN6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RACE, a method that credits actions of long-horizon LLM agents according to changes in recoverability. Particularly, the recoverability is estimated from the outcomes of semantically similar, task-compatible contexts in other trajectories within one training batch. It uses frozen embeddings, nearest-neighbor retrieval, and regularization toward the batch outcome average. The clipped change across an action adjusts the base advantage, optionally targeting steps with weak base credit.

Results show performance gains against GiGPO and several other baselines (from public results) using Qwen3-4B and Qwen3-8B on ALFWorld, ScienceWorld, and WebShop, with improved zero-shot transfer to BabyAI, PDDL, and TextCraft on average.

Critical strengths

The method combines nearest-neighbor value estimation with action-level advantage correction for group-relative RL. In particular, when all sampled trajectories within a group fail and within-group comparisons provide little learning signal, similar experiences from other task instances may help distinguish the contributions of individual actions. This is a well-motivated and targeted direction for improvement.

Critical weaknesses
  1. The advance over existing credit-assignment methods is insufficiently established.
  • RACE motivates its contribution through GiGPO’s reliance on exact-state recurrence, but GiGPO (section 5.1 in [1]) already includes similarity-based anchor grouping. A controlled comparison against this variant is necessary to establish the added value of recovery-aware estimation beyond relaxing exact-state matching.
  • Only GiGPO is reproduced under matched conditions; other published baselines retain their original training & evaluation protocols, making the comparison unreliable.
  • Additional representative step-level credit methods or related work (such as [2-4] which also avoids additional rollouts and annotated process rewards) should also be compared both in performance and cost.
  1. The central mechanism and claimed plug-in capability lack validation.
  • The support rate of over 98% in table 3 measures only retrieval coverage, so multiple records may come from the same trajectory. The paper should also report the number of distinct neighboring trajectories and whether the retrieval weights are overly concentrated. More directly, evaluation in a resettable environment should compare RACE’s action rankings with subsequent success rates estimated through repeated continuation rollouts from alternative actions at the same state (Monte-Carlo rollouts) to validate the credit accuracy of the step-level signals.
  • Experiments integrate RACE with several group-based RL algorithms should be added to substantiate the plug-in claim.
  • ALFWorld, WebShop, and ScienceWorld provide structured tasks and constrained interaction interfaces; an open-ended interaction benchmark (e.g., a realistic browser or tool-use env) would more directly test the motivation of learning from semantically comparable contexts when exact recurrence is rare.

References:

  1. Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408.
  2. Liu, Xiaoqian, et al. "Agentic reinforcement learning with implicit step rewards." International Conference on Learning Representations. 2026.
  3. He, Shuo, et al. "Hierarchy-of-groups policy optimization for long-horizon agentic tasks." International Conference on Learning Representations. 2026.
  4. Fang, Yangyi, et al. "Proximity-based multi-turn optimization: Practical credit assignment for llm agent training." ACL 2026.
Questions for authors

Could we first identify transitions whose pre- and post-action contexts both have sufficient eligible neighbors, then compute recoverability differences only for those transitions? Would this be equivalent to applying the support mask in the advantage update (Eq.8), and are there computational or implementation considerations that motivate the chosen formulation?

Reviewer_iHeaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose RACE, a method that use the embedding similarity between two serialized context to weighted average the final reward for credit assignment. This method serializes agent's context, uses a frozen text encoder to calculate the embedding as representation, and weighted average the corresponding final outcome to calculate recoverability. Then, it uses the difference between recoverability as recovery increment. Finally, it adds the recovery increment into advantage of GRPO to perform policy optimization. This method achieves improvement in three popular agent benchmark compare with method like GiGPO.

Critical strengths
  1. This method is intuitive and easy to understand.
  2. The performance improvement is significant.
Critical weaknesses
  1. Lack of Novelty: The key methodology, use text encoder to transform the context into representation and weighted average the final outcome based on similarity, has been proposed by several previous works, like "Proximity-Based Multi-Turn Optimization: Practical Credit Assignment for LLM Agent Training" and "Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning". The "recovery increment" seems like a rephrasing of Temporal Difference.
Questions for authors
  1. The process of embedding computation, store, and retrieval can cause latency in the training process (and the paper says the pipeline runs on cpu). What is the time for your method to complete one step compared with others? Will long reasoning amplify this cost?
  2. Why use a small frozen encoder instead of larger and powerful one, or even LLM itself’s hidden state? In complex task, for example Software Engineering, will the small encoder too weak to distinguish different context and state? Is it possible that small frozen encoder hack by the trained policy?
  3. What is the difference between RACE and two previous works mentioned in Weakness?
Reviewer_UnXrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RACE, a fine-grained credit assignment method for long-horizon LLM agents. RACE retrieves semantically similar states from collected rollouts using a frozen text encoder and estimates state recoverability based on the corresponding trajectories' terminal rewards. It then uses the difference in recoverability between consecutive states to construct action-level credit corrections, which are incorporated into existing group-relative policy optimization. Experiments on ALFWorld, ScienceWorld, and WebShop show improvements over GiGPO, along with some evidence of cross-environment generalization.

Critical strengths

Simple and easily integrable design. RACE leverages existing rollout data to provide fine-grained credit signals without requiring additional environment interactions or training a separate critic or reward model. It can be integrated into existing RL frameworks as a plug-in module.

Critical weaknesses
  1. Lack of fair and comprehensive baseline comparisons. The baselines across different domains and model sizes are not consistently aligned in terms of backbone models, training budgets, and evaluation protocols. Moreover, direct comparisons with other step-level credit assignment methods are missing, making it difficult to fully assess RACE's advantages over existing approaches.
  2. Concerns about the reliability of credit assignment. As the policy evolves during training, its capabilities and success rates change over time. Estimating the recoverability of current states using outcomes from historical rollouts may introduce bias, potentially affecting the accuracy of action-level credit assignment.
  3. Insufficient analysis of the weak-signal gate. Removing the gate causes ALFWorld OOD success to drop from 81.34% to 59.70%. However, the paper does not adequately explain this substantial degradation or justify why different gating strategies are adopted across environments.
Questions for authors
  1. Generalizability of RACE. Could the authors provide a controlled comparison among GRPO, GRPO + RACE, GiGPO, and GiGPO + RACE to investigate whether RACE depends on GiGPO and validate its effectiveness as a general plug-in method?
  2. Potential double-counting in KNN retrieval. Multiple retrieved states may originate from the same trajectory and share the same terminal reward, potentially causing certain trajectories to be overrepresented in recoverability estimation. Have the authors considered this issue? Could they compare state-level KNN with trajectory-deduplicated KNN to evaluate its impact?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.