- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PivoARL for long-horizon LLM-agent reinforcement learning, where failed trajectories are expensive to retry and sparse rewards often assign negative credit to otherwise correct prefixes. PivoARL uses structured self-reflection to identify a pivotal erroneous turn, retries locally from that state while reusing the prefix, and treats the pivotal state as a credit boundary. Successful retry returns reinforce the reused prefix while isolating the erroneous suffix; the subsequent retry return also trains reflection quality. The paper evaluates the method on four agent environments and seven search-based QA benchmarks, reporting gains in Pass@k/success rates, ablations, cross-model results, stability across seeds, and training/test-time scaling analyses.
Critical strengths
- The pivotal-aware retry design reuses the prefix while isolating the erroneous suffix in credit propagation. This directly targets a concrete failure mode of long-horizon sparse-reward training and is consequential to the paper’s main methodological contribution.
- The evaluation covers both interactive agent environments and search-based QA, with component ablations, cross-model results, and multi-seed analysis. This breadth supports the claim that the observed gains are not confined to a single task formulation or model scale.
Critical weaknesses
- The return construction is sensitive to signed and length-dependent rewards. A large negative return from a newly regenerated suffix is propagated to all actions before the pivot in Eq. (7), and the episode-level advantage further affects the whole failed episode. Conversely, Eq. (8) can assign different reflection rewards to equally successful reflections solely because their successful retry suffixes have different lengths or per-step costs. Thus, a correct prefix or reflection may be penalized because of a later error or path-length effect. The current formulation does not state the reward assumptions needed to rule out this behavior.
Questions for authors
- Can you report an end-to-end runtime comparison between PivoARL and full-retry baselines such as MetaRL, including wall-clock time, GPU hours, generated/input tokens, and environment interaction time at matched training performance? In particular, how much of the prefix-reuse benefit remains under parallel rollout, where full retries may be executed concurrently while PivoARL may introduce sequential retry dependencies and state-restoration overhead?
Deepest thanks for your relentless dedication!
Dear Area Chair,
Thank you for overseeing the review of Submission 9709. We would like to highlight the main updates from our rebuttal and the strengths identified across all three reviews.
(1) Main concerns and rebuttal updates
(2) Reviewer 9Qr7 (Overall assessment: 3 / Confidence: 2), Reviewer LoiB (Overall assessment: 3 / Confidence: 2), and Reviewer 5MVE (Overall assessment: 3 / Confidence: 3)
All three reviewers rated the paper Weak Acceptance and highlighted two shared strengths.
We appreciate your consideration of the reviews and supplemental results in the final assessment.
Sincerely, The authors of Submission 9709