- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether self-assessment signals need to be explicitly used during agent reinforcement learning. The authors propose RAISE, which augments GRPO/GiGPO with an auxiliary ranking loss that trains action-level success estimates using trajectory outcomes. These estimates are not directly used for action selection, reward computation, or advantage estimation. Experiments on ALFWorld, Sokoban, and WebShop show improved success rates over the reported baselines, while explicitly injecting the estimates into the advantage or using them for action selection provides no additional benefit. The authors further argue that the auxiliary objective primarily learns state-level outcome information by aligning an existing readout.
Critical strengths
The paper investigates a potentially useful distinction between training self-assessment signals and explicitly using them in policy optimization. The reported performance improvements across multiple benchmarks are encouraging, and the experiments also exclude the effect of the scaffold, making the improvement from adding the ranking loss more convincing.
Critical weaknesses
- The central claim is insufficiently supported by the experimental evidence. While the paper separates the explicit consumption of self-assessment estimates from their auxiliary training, it does not establish whether the observed performance improvements specifically arise from learning self-assessment or simply from additional outcome-supervised gradients on the shared policy. Moreover, the negative consumption results are based on a limited set of mechanisms, whose suitability for the learned estimates is not sufficiently justified. Therefore, the conclusion that consumption is unnecessary appears broader than what the experiments demonstrate.
- Section 4 should include more information about the experimental setup instead of directly diving into the results, which will make the overall delivery of the results more understandable.
Questions for authors
- The ranking loss supervises relative action logits, yet Section 5.1 concludes that the learned estimates primarily capture state values rather than action values. What explains this discrepancy?
- It is mentioned in Section 5.2 that the outcome is already decodable before the auxiliary loss exists. What additional capability does the ranking loss provide? More importantly, how does improving the self-assessment readout lead to better policy decisions, especially when Section 5.3 shows limited changes in the underlying representations?
- Is the consumption mechanism in Table 2 an appropriate way to evaluate the usefulness of learned self-assessment estimates? Given that Section 5.1 suggests these estimates primarily represent state values, why is their temporal difference directly injected into the action advantage? Would more principled value-based credit assignment methods lead to different conclusions? For instance, it might be more aligned with your claims in Section 5 to compare the existing approach with TD-style credit assignment using , while controlling for the magnitude of the injected signal.