- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Credit Assignment with State Transitions (CAST ), a step-level credit assignment method for LLM agents. The authors identify a limitation of existing hidden-state-based approaches, where similar hidden states and identical actions may lead to different state transitions but receive the same credit. CAST addresses this issue by introducing transition-aware reference selection within latent neighborhoods, filtering same-action records with inconsistent transitions while retaining other-action records for local comparison. The resulting step-level advantages are combined with episode-level advantages for policy optimization. Experiments across four interactive environments and multiple model families demonstrate consistent improvements over GiGPO and BiPACE, with limited additional computational overhead.
Critical strengths
- Clear motivation and well-designed method. The paper addresses a meaningful limitation of hidden-state-based credit assignment, where similar hidden states and identical actions may lead to different state transitions. The authors provide statistical evidence supporting this problem. CAST introduces a simple transition-aware reference selection strategy that naturally aligns with the motivation.
- Practical design with low computational overhead. CAST builds upon the existing latent neighborhood credit estimation framework without requiring additional environment rollouts or a learned value model. The transition filtering mechanism is straightforward and introduces limited computational overhead.
- Consistent experimental improvements. CAST consistently outperforms GiGPO and BiPACE across four interactive environments and different model scales. Additional experiments with Llama further demonstrate its effectiveness across model families.
Critical weaknesses
- Limited methodological novelty. CAST mainly introduces transition-aware reference filtering into the existing latent neighborhood credit estimation framework, making the methodological contribution relatively incremental.
- Reliance on exact string matching. CAST relies on exact string matching for successor-observation consistency, which may incorrectly distinguish semantically equivalent transitions. This partially reintroduces the limitations of explicit state matching that hidden-state-based approaches aim to address.
- Insufficient analysis of reference reduction. In Table 5, count-matched random selection already improves performance over BiPACE by 19.0 percentage points, while CAST provides only an additional 3.0-point gain. This suggests that reference reduction itself may account for a substantial portion of CAST's improvements, weakening the attributed contribution of transition-aware selection.
- Potential limitations of observation-only state representations. The authors extract hidden states using only current observations without incorporating interaction history. However, identical observations may correspond to different underlying environment states due to different histories. It remains unclear whether CAST would provide similar improvements when using full-context hidden states.
Questions for authors
- Local baseline computation. Why are same-action records with different transitions excluded from both the local action-value estimation and the local baseline? While excluding them from the action-value estimation is understandable, retaining them in the baseline seems reasonable. Could the authors provide an ablation comparing Q-only filtering with the current strategy?
- Necessity of successor-observation consistency. In Table 6, removing successor-observation consistency reduces performance by only 0.4 points on ScienceWorld. Given the limitations of exact string matching, could the authors further justify the necessity of this component?