acceptodds
Desk rejected at ICLR 2027

Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals. To address this, we propose PivoARL, a self-feedback retry framework for experience exploitation in LLM agents. PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions. From an information-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state-agnostic experience utilization. Based on this insight, we design a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns. We conduct a systematic evaluation on 4 agent tasks and 7 search-based QA benchmarks. Results show that \ourapproach achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5% over MetaRL. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also improves Pass@1 over the default RL baseline on over 80% of the evaluated tasks. On Minesweeper environment, PivoARL improves over GiGPO by more than 45% and reduces interaction turns by about 42% on average compared with full-retry methods.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Credit Assignment, Self-Feedback, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5MVEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PivoARL for long-horizon LLM-agent reinforcement learning, where failed trajectories are expensive to retry and sparse rewards often assign negative credit to otherwise correct prefixes. PivoARL uses structured self-reflection to identify a pivotal erroneous turn, retries locally from that state while reusing the prefix, and treats the pivotal state as a credit boundary. Successful retry returns reinforce the reused prefix while isolating the erroneous suffix; the subsequent retry return also trains reflection quality. The paper evaluates the method on four agent environments and seven search-based QA benchmarks, reporting gains in Pass@k/success rates, ablations, cross-model results, stability across seeds, and training/test-time scaling analyses.

Critical strengths
  1. The pivotal-aware retry design reuses the prefix while isolating the erroneous suffix in credit propagation. This directly targets a concrete failure mode of long-horizon sparse-reward training and is consequential to the paper’s main methodological contribution.
  2. The evaluation covers both interactive agent environments and search-based QA, with component ablations, cross-model results, and multi-seed analysis. This breadth supports the claim that the observed gains are not confined to a single task formulation or model scale.
Critical weaknesses
  1. The return construction is sensitive to signed and length-dependent rewards. A large negative return from a newly regenerated suffix is propagated to all actions before the pivot in Eq. (7), and the episode-level advantage further affects the whole failed episode. Conversely, Eq. (8) can assign different reflection rewards to equally successful reflections solely because their successful retry suffixes have different lengths or per-step costs. Thus, a correct prefix or reflection may be penalized because of a later error or path-length effect. The current formulation does not state the reward assumptions needed to rule out this behavior.
Questions for authors
  1. Can you report an end-to-end runtime comparison between PivoARL and full-retry baselines such as MetaRL, including wall-clock time, GPU hours, generated/input tokens, and environment interaction time at matched training performance? In particular, how much of the prefix-reuse benefit remains under parallel rollout, where full retries may be executed concurrently while PivoARL may introduce sequential retry dependencies and state-restoration overhead?
Reviewer_LoiBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how LLM agents can learn from failed trajectories in long-horizon interactive tasks. It proposes PivoARL, a reinforcement learning framework in which the agent reflects on a failed attempt, identifies the earliest pivotal error, restores the corresponding environment state, and retries only from that point. During training, the pivotal turn also serves as a credit-assignment boundary: later successful retry signals are propagated to the reused prefix while being isolated from the erroneous suffix. The framework further uses the outcome of the subsequent retry to optimize reflection generation. Experiments across interactive agent environments and search-augmented QA tasks show improvements in task success rates, especially when multiple attempts are allowed, while reducing redundant environment interactions.

Critical strengths

Important problem: The paper addresses the practical challenge of extracting useful learning signals from partially correct but ultimately failed agent trajectories.

Clear methodological idea: Treating the pivotal turn as both the retry boundary and the credit-routing boundary is intuitive and well aligned with the identified failure mode.

Broad evaluation: The experiments cover four benchmark agent environments and seven single-hop and multi-hop search-QA datasets, with comparisons against several categories of RL and agent-learning methods.

Useful supplementary analyses: The paper includes component ablations, pivotal-error localization analysis, multi-seed results on two representative settings, and environment-interaction measurements, providing useful supporting evidence for the framework.

Critical weaknesses
  1. The mismatch in policy-conditioning contexts weakens the same-state preference interpretation. PivoARL groups the original failed action and the retry-corrected action according to the underlying environment state, although they are generated under different policy inputs: the original action is sampled from π(a∣s)\pi(a\mid s), whereas the retry action is sampled from π(a∣s,remark)\pi(a\mid s,\mathrm{remark}) with explicit reflective guidance. Their return difference therefore conflates action quality with the effect of additional reflection information and cannot be strictly interpreted as an action preference under identical conditions. This issue not only limits the analogy to step-level preference optimization in Appendix E.3, but also weakens the explanation that pivotal preference signals lead to improved Pass@1 performance. The existing experiments still demonstrate the effectiveness of the complete reflective-retry system, but they do not establish how the retry signal transfers to the first-attempt policy without a remark. The authors should consider ablating or shuffling the remark, or directly measuring the probability of the corrected action under the no-remark context, to verify that the signal is genuinely internalized by the base policy.

  2. The method's dependence on pivot localization and environment-state restoration limits its scope of applicability. The paper evaluates pivotal-error localization and implements state caching or reconstruction for the studied benchmarks, but it does not systematically examine how localization errors, multiple valid pivots, or failures without a clear single error boundary affect performance and credit assignment. Moreover, local retry requires restoration of a historical environment state, which may be infeasible in dynamic environments or real-world tool-use settings involving irreversible operations. The authors should provide a pivot-perturbation analysis and more clearly characterize the class of recoverable environments to which the method applies.

  3. The implementation of the core credit assignment and its effect on the final gradients require further clarification. For t<kt<k, Eq. (7) truncates the erroneous suffix of the original episode and discounts the subsequent retry return by γstepk−t\gamma_{\mathrm{step}}^{k-t}. In contrast, Line 21 of Algorithm 1 adds the future return directly to the full gtg_t and omits this step discount; the two definitions are not evidently equivalent. More importantly, Appendix C.3 defines the final advantage as A^t=AE(τ)+ωAS(at)\hat A_t=A_E(\tau)+\omega A_S(a_t). The negative AEA_E of a failed episode therefore still applies to its correct prefix, so positive prefix credit may not be sufficient to make the final advantage positive. The authors should clarify the actual implementation and report the final advantage or gradient distributions for correct prefixes and erroneous suffixes to verify the central claim that the method protects correct prefixes.

Questions for authors

See above weaknesses.

Reviewer_9Qr7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PivoARL for training LLM agents through reflection and local retry after failure. It uses subsequent retry outcomes to credit the reused trajectory prefix without rewarding the original failed suffix. Experiments evaluate the method on interactive environments and search-based question answering.

Critical strengths
  • Cross-episode prefix crediting is a concrete extension beyond local retry alone and offers a useful way to learn from failures.
  • The evaluation covers varied tasks and includes component ablations and repeated runs.
Critical weaknesses
  • The novelty is relatively incremental. R3L [1] already uses reflection-guided local retry, while LaMer [2] uses cross-episode returns. PivoARL primarily extends these ideas through pivot-conditioned credit assignment.
  • Credit assignment depends on predicted error locations, but robustness to localization errors is not established. LLM-based diagnostics do not test how incorrect boundaries affect learning, and experiments assume restorable environment states.
  • The paper reports savings in environment interactions, but not total training cost including reflection and state restoration.

References:
[1] R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification
[2] Meta-RL Induces Exploration in Language Agents

Questions for authors

See the weaknesses above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Deepest thanks for your relentless dedication!

Dear Area Chair,

Thank you for overseeing the review of Submission 9709. We would like to highlight the main updates from our rebuttal and the strengths identified across all three reviews.

(1) Main concerns and rebuttal updates

  • Pivot robustness. With training-time pivot perturbations, PivoARL still outperforms GiGPO on Minesweeper and ALFWorld.
  • Credit assignment. We clarified how reflection changes the retry policy’s input and how the implementation computes the reused prefix’s advantage. We will correct Algorithm 1 and Eq. (15) accordingly.
  • Reward assumptions and training cost. We clarified when the reflection reward favors shorter retries. Under the same concurrent rollout schedule, PivoARL reduced end-to-end runtime relative to MetaRL by 36.9% on Minesweeper and 16.1% on ALFWorld, including reflection and state restoration.

(2) Reviewer 9Qr7 (Overall assessment: 3 / Confidence: 2), Reviewer LoiB (Overall assessment: 3 / Confidence: 2), and Reviewer 5MVE (Overall assessment: 3 / Confidence: 3)

All three reviewers rated the paper Weak Acceptance and highlighted two shared strengths.

  • Pivot-based credit assignment. The reviewers recognized the value of using retry outcomes to credit the reused prefix while isolating the replaced suffix.
  • Breadth of evaluation. The reviewers noted the coverage of interactive environments and search-based QA, supported by component ablations and repeated runs.

We appreciate your consideration of the reviews and supplemental results in the final assessment.

Sincerely, The authors of Submission 9709

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.