acceptodds
Withdrawn from ICLR 2027

Consumption Is Unnecessary: Learning a Value Estimate Is Enough for Group-Relative Agent RL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A common pattern in training LLM agents is to let the policy estimate its own progress or its chance of success and then use that estimate — as an advantage, a reward, or a rule for choosing actions. Whether the gain comes from learning the estimate or from using it has not been separated. We separate the two. Our method, RAISE, trains the estimate with a listwise rank loss whose only label is the trajectory outcome, and then never reads it: not in the reward, not in the advantage, not at inference. The loss reaches the policy only through the shared parameters. On ALFWorld, Sokoban and WebShop, with three training seeds each, this objective alone beats the published GiGPO baselines by , and points at the same budget, and reaches their final success within – of it. Where we test consumption — a forward difference of the estimate injected into the advantage at two doses, and three rules for choosing the action with it at inference — it adds nothing we can detect: the injection is flat at both doses and the selection rules do not help. Randomising the targets removes the gain, so what matters is that the targets are correct. What the loss teaches is which states are promising, not which action to take: inside a state, the order it gives the candidate actions is no better than one constant per state. The policy's hidden state already carries the outcome before the loss is added. The practical result is a recipe that needs no demonstrations and no extra forward pass.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, LLM agents, credit assignment, auxiliary objectives, self-assessment, value estimation, group-relative policy optimization, representation probing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HCQROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether self-assessment signals need to be explicitly used during agent reinforcement learning. The authors propose RAISE, which augments GRPO/GiGPO with an auxiliary ranking loss that trains action-level success estimates using trajectory outcomes. These estimates are not directly used for action selection, reward computation, or advantage estimation. Experiments on ALFWorld, Sokoban, and WebShop show improved success rates over the reported baselines, while explicitly injecting the estimates into the advantage or using them for action selection provides no additional benefit. The authors further argue that the auxiliary objective primarily learns state-level outcome information by aligning an existing readout.

Critical strengths

The paper investigates a potentially useful distinction between training self-assessment signals and explicitly using them in policy optimization. The reported performance improvements across multiple benchmarks are encouraging, and the experiments also exclude the effect of the scaffold, making the improvement from adding the ranking loss more convincing.

Critical weaknesses
  1. The central claim is insufficiently supported by the experimental evidence. While the paper separates the explicit consumption of self-assessment estimates from their auxiliary training, it does not establish whether the observed performance improvements specifically arise from learning self-assessment or simply from additional outcome-supervised gradients on the shared policy. Moreover, the negative consumption results are based on a limited set of mechanisms, whose suitability for the learned estimates is not sufficiently justified. Therefore, the conclusion that consumption is unnecessary appears broader than what the experiments demonstrate.
  2. Section 4 should include more information about the experimental setup instead of directly diving into the results, which will make the overall delivery of the results more understandable.
Questions for authors
  1. The ranking loss supervises relative action logits, yet Section 5.1 concludes that the learned estimates primarily capture state values rather than action values. What explains this discrepancy?
  2. It is mentioned in Section 5.2 that the outcome is already decodable before the auxiliary loss exists. What additional capability does the ranking loss provide? More importantly, how does improving the self-assessment readout lead to better policy decisions, especially when Section 5.3 shows limited changes in the underlying representations?
  3. Is the consumption mechanism in Table 2 an appropriate way to evaluate the usefulness of learned self-assessment estimates? Given that Section 5.1 suggests these estimates primarily represent state values, why is their temporal difference directly injected into the action advantage? Would more principled value-based credit assignment methods lead to different conclusions? For instance, it might be more aligned with your claims in Section 5 to compare the existing approach with TD-style credit assignment using γV(st+1)−V(st)\gamma V(s_{t+1})-V(s_t), while controlling for the magnitude of the injected signal.
Reviewer_Hs9jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether supervising an agent's success estimates can improve policy learning even when the estimates are not used to choose actions or modify rewards and advantages. The proposed method adds an outcome-supervised ranking loss to estimates read from the policy's existing language head. The experiments compare this objective with a matched no-loss control and test the additional effect of using the estimates during training or action selection. Results show success-rate gains from auxiliary supervision and no clear additional benefit from the tested ways of using the estimates.

Critical strengths
  1. The matched no-loss control preserves the prompt format, value slots, and backbone, helping isolate the contribution of auxiliary supervision. These comparisons support the value of informative outcome supervision beyond the effect of changing the prompt or adding an arbitrary auxiliary target.

  2. The paper explicitly separates the benefit of learning success estimates from the benefit of using them. The training-time comparisons retain auxiliary supervision and control the scale of the injected signal.

Critical weaknesses
  1. The training experiments are limited to Qwen2.5 backbones. Varying model size and including a vision-language setting does not establish whether the main findings extend to newer open-weight models. Repeating the matched comparisons on newer backbones is needed to assess whether both the benefits of auxiliary supervision and the limited additional benefit of using the estimates persist.

  2. The evaluation lacks performance references from strong current API models, making the practical competitiveness of the trained agents difficult to assess. Improvements over the reported GRPO and GiGPO baselines do not establish how the resulting agents compare with available API models on the same tasks. Comparisons with comparable tool access and interaction budgets, accompanied by inference costs, would help contextualize the achieved performance. These would complement the training ablations rather than replace them.

Questions for authors
  1. Could the authors clarify how Table 3 constructs positive–negative pairs and weights states, and why the per-state-constant baseline obtains an AUROC above 50%? Are comparisons restricted to samples from the same state?

  2. Could the authors repeat the no-loss, auxiliary-supervision-only, and explicit-use comparisons on newer open-weight models, reporting uncertainty for the differences? Could they also include strong current API models as inference baselines on the same tasks, with comparable tool access and interaction budgets and reported inference costs, to contextualize the achieved performance?

Reviewer_42HdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a method RAISE, which trains an implicit value estimate using a listwise ranking loss. The authors show that learning this signal is enough to improve model performance compared to GiGPO and standard GRPO. The authors further show that it incorporating this value function does not provide any further signal during training.

Critical strengths
  • I actually find this idea to be quite interesting as this may be able to yield the same benefit as PPO without requiring an explicit value function.
Critical weaknesses
  • The writing of this work is quite hard to follow, and would benefit significantly from polishing. At it's current state, I don't find the writing to be up to ICLR standard.
  • The paper does not meaningfully cover related works in this area, especially much of the work on value function training.
  • The paper reproduces numbers from the GiGPO work, but it's unclear if the environment is exactly the same, making them directly comparable.
Questions for authors
  • Can you compare this to PPO which learns an explicit value function.
  • More broadly, can this work be contextualized with respect to value function training in general?
  • Can you measure explained variance of this value estimate? how does this estimate compare to learned value functions in PPO?
  • There is another interpretation that policies are indeed implicit Q-functions (https://arxiv.org/abs/2404.12358). How does this differ from the explicit loss that is being used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.