- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the self-evolution and preference alignment of LLM-based agents in long-horizon tasks. The authors propose a "Co-Evolving" framework to address the issue where failed trajectories generated by agents fail to provide effective supervision. Specifically, the method generates hard negative samples by training a Failure Agent exclusively on failed trajectories, encouraging it to produce higher-quality, higher-reward failures. These generated failures are then utilized as informative hard negatives to train the Target Agent via preference optimization
Critical strengths
The paper clearly identifies a critical bottleneck in self-evolving agents: when applying DPO to an agent's own failures, the negative samples are often too trivial, leading to rapid saturation and diminishing returns of the supervisory signal.
Critical weaknesses
-
The framework strongly relies on intermediate or continuous reward signals within to distinguish failure quality. In many real-world environments with purely sparse terminal feedback, the Failure Agent cannot obtain preference signals due to pervasive ties, which may render the method ineffective.
-
The empirical evaluation includes relatively few baselines and lacks comparisons against recent state-of-the-art self-evolving and agent alignment methods.
Questions for authors
Could the authors provide experimental comparisons with more recent state-of-the-art baselines to better demonstrate the empirical advantage of the proposed framework?
3 comments
Response by Reviewer ZX1D
Thank you for the additional experiments and clarifications. The comparisons with HPL and Early Experience address my concern about the limited baselines, and I will raise my score.
We appreciate the time the reviewer spent on our paper. We are glad that our responses addressed the concerns and that the reviewer raised the score.
Response to Reviewer ZX1D (1/2)
We appreciate the reviewer taking the time to review our paper.
Q1 / W2 · Comparisons with recent agent-learning methods
We have evaluated HPL [1] and Early Experience [2]. HPL combines preference learning at the trajectory, action-group, and step levels with a curriculum over task complexity and reward gaps. Early Experience has agents try alternative actions at states from expert demonstrations and learn from the resulting observations through world modeling or self-reflection, without reward supervision.
As shown in the table below, our method achieves higher reward across both models and all environments. The gains are consistent with the benefit of explicitly training a failure agent to generate informative negatives as the target improves. We have added both baselines to Table 2 and updated the related work and baseline descriptions in Sections 2 and 5.1.
[1] Gao et al., “Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents,” ICLR ’26.
[2] Zhang et al., “Agent Learning via Early Experience,” ICML ’26.
W1 · Learning without continuous environment rewards
Our method needs a signal that distinguishes better failures from worse ones, whether continuous or discrete. We agree that learning with only binary environment feedback is important. We have therefore tested self-verification to rank failures without using continuous environment rewards for training.
We have evaluated each failed trajectory and its observations with Qwen3.5-9B using two prompts based on direct scoring and checklist feedback [1, 2]:
We have checked whether these scores rank failures in the same order as environment rewards. The table reports agreement on same-task failure pairs with a reward gap of at least 10 points (0–100 scale; ties count as half correct).
We have trained for three iterations using self-verification scores to rank failures. We selected direct scoring for WebShop and checklist scoring for ScienceWorld and InterCodeSQL based on the agreement analysis above. ETO used its original training procedure without self-verification.
Self-verification remains below environment-reward training, but exceeds ETO on WebShop and both ScienceWorld splits, with comparable performance on InterCodeSQL. It provides a fallback for ranking failures when graded environment rewards are unavailable. We have added the scoring protocol to Section 5.8, with ranking agreement in Table 6(a) and target performance in Table 6(b).
[1] Viswanathan et al., “Checklists Are Better Than Reward Models For Aligning Language Models,” NeurIPS ’25.
[2] Gunjal et al., “Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains,” ICLR ’26.