- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TreeHCA, a framework that jointly addresses exploration and credit assignment in multi-turn LLM agents trained with terminal rewards. The method uses changes in reference-answer likelihood to identify consequential intermediate decisions and guide the sampling of alternative continuations. It then applies inverse-hindsight weighting in a tree-based backup to propagate terminal rewards into turn-level advantages. The authors derive importance-weighting identities to motivate the connection between hindsight-guided sampling and credit estimation. Experiments across seven question-answering benchmarks and three Qwen backbones demonstrate improvements in aggregate exact match over existing chain-based and tree-based methods, while component ablations support the benefits of combining hindsight-guided exploration with credit assignment.
Critical strengths
- TreeHCA connects hindsight-guided exploration with inverse-weighted credit assignment through a shared change-of-measure perspective. The finite-pool importance-weighting identity provides a principled motivation for linking sampling decisions to turn-level credit, representing a valuable methodological contribution.
- The paper demonstrates improvements in aggregate exact match across seven QA benchmarks and three Qwen backbones. The component ablation further shows that combining hindsight-guided exploration with the proposed credit assignment achieves better EM and F1 than alternative combinations, supporting the effectiveness of the paired design.
Critical weaknesses
- Proposition 1 establishes an exact importance correction under specific sampling assumptions, but the practical tree backup uses self-normalized inverse scores over adaptively selected children. The theoretical guarantees do not directly extend to this estimator, weakening the claimed principled connection between exploration and credit assignment.
- The proposed method uses changes in reference-answer likelihood as a proxy for the conditional probability of terminal success. However, the paper provides limited evidence that these scores reliably identify consequential decisions or accurately reflect their contributions to successful outcomes. This uncertainty affects the validity of both the exploration strategy and the resulting credit assignment.
Questions for authors
- Proposition 1 establishes an exact importance correction for finite-pool sampling, whereas Eq. (14) applies self-normalized inverse weights to adaptively retained tree children. Could the authors clarify which theoretical properties extend to the practical estimator and how the differences between these sampling settings affect the validity of credit assignment?
- The practical method uses changes in reference-answer likelihood as a proxy for conditional terminal success probabilities. What evidence supports the reliability of this approximation in identifying consequential decisions? Are there existing analyses showing that lower hindsight scores correspond to reduced success probabilities or that alternative continuations improve outcomes?