acceptodds
Under review as a conference paper at ICLR 2027

Co-Evolving Agents: Learning from Failures as Hard Negatives

Abstract

Self-evolving agents improve their performance on long-horizon tasks by learning from their own interactions with an environment. A common approach uses the resulting failed trajectories as negatives for preference training. However, collecting an agent’s own failures does not ensure that they provide informative supervision for further improvement. Obvious failures may be easy to reject without learning to identify errors in more plausible attempts. These plausible but incorrect trajectories can serve as *hard negatives*, helping agents learn to distinguish successful behavior from failures with less obvious errors. To enable agents to generate and learn from such informative negatives, we propose a co-evolving framework that couples a target agent’s self-improvement with a failure agent trained to generate hard negatives. The failure agent learns exclusively from both agents’ failed trajectories to generate higher-reward failures, which the target agent uses as negatives for preference training. As the target policy improves, its failures provide new training data for the failure agent. The updated failure agent, in turn, provides new negatives for target training, allowing negative generation to adapt alongside the target policy. Across online shopping, scientific reasoning, and interactive SQL querying, our framework improves average task reward by 5.7 over the baseline. These results demonstrate the value of learning to generate and use informative failures for agent self-improvement. We will make our code publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving Agents, LLM Agents, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZX1DOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the self-evolution and preference alignment of LLM-based agents in long-horizon tasks. The authors propose a "Co-Evolving" framework to address the issue where failed trajectories generated by agents fail to provide effective supervision. Specifically, the method generates hard negative samples by training a Failure Agent exclusively on failed trajectories, encouraging it to produce higher-quality, higher-reward failures. These generated failures are then utilized as informative hard negatives to train the Target Agent via preference optimization

Critical strengths

The paper clearly identifies a critical bottleneck in self-evolving agents: when applying DPO to an agent's own failures, the negative samples are often too trivial, leading to rapid saturation and diminishing returns of the supervisory signal.

Critical weaknesses
  1. The framework strongly relies on intermediate or continuous reward signals within [0,1][0, 1] to distinguish failure quality. In many real-world environments with purely sparse terminal feedback, the Failure Agent cannot obtain preference signals due to pervasive ties, which may render the method ineffective.

  2. The empirical evaluation includes relatively few baselines and lacks comparisons against recent state-of-the-art self-evolving and agent alignment methods.

Questions for authors

Could the authors provide experimental comparisons with more recent state-of-the-art baselines to better demonstrate the empirical advantage of the proposed framework?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZX1D (1/2)

We appreciate the reviewer taking the time to review our paper.


Q1 / W2 · Comparisons with recent agent-learning methods

We have evaluated HPL [1] and Early Experience [2]. HPL combines preference learning at the trajectory, action-group, and step levels with a curriculum over task complexity and reward gaps. Early Experience has agents try alternative actions at states from expert demonstrations and learn from the resulting observations through world modeling or self-reflection, without reward supervision.

As shown in the table below, our method achieves higher reward across both models and all environments. The gains are consistent with the benefit of explicitly training a failure agent to generate informative negatives as the target improves. We have added both baselines to Table 2 and updated the related work and baseline descriptions in Sections 2 and 5.1.

ModelMethodWebShopScienceWorld seenScienceWorld unseenInterCodeSQL
QwenHPL69.4683.4676.7869.59
Early Experience57.2466.8764.1662.14
Ours72.2587.5882.9469.76
GemmaHPL73.2281.3972.5370.97
Early Experience65.2674.7065.3466.13
Ours73.7585.7581.5172.49

[1] Gao et al., “Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents,” ICLR ’26.

[2] Zhang et al., “Agent Learning via Early Experience,” ICML ’26.


W1 · Learning without continuous environment rewards

Our method needs a signal that distinguishes better failures from worse ones, whether continuous or discrete. We agree that learning with only binary environment feedback is important. We have therefore tested self-verification to rank failures without using continuous environment rewards for training.

We have evaluated each failed trajectory and its observations with Qwen3.5-9B using two prompts based on direct scoring and checklist feedback [1, 2]:

  • Direct scoring assigns each trajectory a score from 0 to 10.
  • Checklist scoring extracts the task requirements, rates each as Yes (1), Partially (0.5), or No (0), and averages the item scores.

We have checked whether these scores rank failures in the same order as environment rewards. The table reports agreement on same-task failure pairs with a reward gap of at least 10 points (0–100 scale; ties count as half correct).

MethodWebShopScienceWorldInterCodeSQL
Checklist prompt38.7%68.1%59.5%
Direct score prompt55.9%53.8%56.3%

We have trained for three iterations using self-verification scores to rank failures. We selected direct scoring for WebShop and checklist scoring for ScienceWorld and InterCodeSQL based on the agreement analysis above. ETO used its original training procedure without self-verification.

Self-verification remains below environment-reward training, but exceeds ETO on WebShop and both ScienceWorld splits, with comparable performance on InterCodeSQL. It provides a fallback for ranking failures when graded environment rewards are unavailable. We have added the scoring protocol to Section 5.8, with ranking agreement in Table 6(a) and target performance in Table 6(b).

MethodWebShopScienceWorld seenScienceWorld unseenInterCodeSQL
Ours (environment reward)72.2587.5882.9469.76
Ours (self-verification)69.1086.1179.4866.98
ETO68.3579.6174.9367.01

[1] Viswanathan et al., “Checklists Are Better Than Reward Models For Aligning Language Models,” NeurIPS ’25.

[2] Gunjal et al., “Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains,” ICLR ’26.


Reviewer_ZX1DOpenReview ↗

Response by Reviewer ZX1D

Thank you for the additional experiments and clarifications. The comparisons with HPL and Early Experience address my concern about the limited baselines, and I will raise my score.

We appreciate the time the reviewer spent on our paper. We are glad that our responses addressed the concerns and that the reviewer raised the score.

Reviewer_RU8BOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a co-evolving framework for improving interactive language agents through learned negative generation. A target agent and a failure agent start from the same supervised checkpoint. The failure agent is optimized using DPO on pairs of unsuccessful trajectories, preferring the trajectory with higher environment reward. The target agent learns to prefer expert demonstrations over failures from either agent, with an auxiliary SFT loss. Repeated collection and optimization allow both agents to influence each other's subsequent training data. Experiments on WebShop, ScienceWorld, and InterCodeSQL report improvements over ETO across several model families, together with sampling comparisons and ablations of the failure-agent objective and continued co-evolution.

Critical strengths
  1. A clear and useful approach to negative generation. Training a separate failure policy on reward-ranked failures is a concrete way to make negative generation part of an iterative agent-training pipeline. The distinction between the two agents' objectives is straightforward, and the failure agent is needed only during training.

  2. Encouraging results across tasks and models. The reported gains span shopping, scientific interaction, and SQL querying. The positive-agent replacement and frozen-failure-agent ablations are relevant attempts to distinguish the proposed objective and continued adaptation from simply maintaining another model. These results make the approach worth investigating, although they do not resolve the concerns below.

Critical weaknesses
  1. The comparisons do not isolate learned negative generation from additional data and training resources. Table 3 specifies one rollout per instruction per generating agent, so Ours has two generating policies while ETO has one. Equation (6) pools both agents' failures for target training, and Section 5.1 explicitly states that total training costs are not matched. Instruction-level weighting in Appendix B.2 controls the total weight per retained instruction, but does not equalize negative diversity, instruction coverage, or optimization cost. The Best-of-N comparison does not fully address this issue: Section 5.3 explains that selecting the highest-reward trajectory can select a success and discard the instruction's preference pair, even when other candidates may be useful failures. An ETO baseline with two rollouts, a comparison with matched retained pairs/updates, and selection of the highest-reward failed candidate would provide more direct controls. Without them, the results establish an advantage for the full pipeline more clearly than an advantage specifically attributable to training a hard-negative generator.

  2. The proposed explanation in terms of hard negatives is not directly established. Section 4.2 appropriately acknowledges that environment reward measures partial task completion rather than target-relative negative difficulty. Nevertheless, the main interpretation relies on high-reward failures being especially useful for target preference training. Figure 2 reports absolute failure counts, which are confounded by the additional generating policy; Section 5.5 also notes that the high-reward failure proportion decreases on InterCodeSQL despite performance gains. These observations do not invalidate the method, but leave its mechanism unresolved. A controlled comparison using equal-sized reward-stratified negative sets, with the same expert demonstrations and training budget, would test the hypothesis more directly. Target-relative diagnostics, such as reference-adjusted preference margins and DPO losses with trajectory-length effects accounted for, would further clarify whether the generated negatives are difficult for the target and whether that predicts their utility.

  3. Incomplete experimental reporting limits confidence in the results and their reproducibility. I could not locate a specified total number of co-evolution rounds, the cross-round reference-policy update rule, or per-round retained preference-pair counts. The failure agent excludes successes and tied failure pairs, making its success rate, tie rate, and usable-data yield important to interpreting the training process. Main results also lack reported run-to-run uncertainty. The Qwen3.5 positive-agent replacement in Table 4 falls to 67.82 average, below SFT (69.20) and ETO (73.22), without analysis of this substantial deterioration. This is not evidence of an implementation error, but makes the control difficult to interpret. Finally, Table 6 reports Llama-2-7B WebShop performance of 70.3, whereas Table 8 gives 66.70 at the main default SFT weight of 0.1 and at most 67.98 over the tested weights. Appendix C mentions separate configurations, so these values need not be contradictory, but the differences are not specified well enough to reconcile them. Exact configurations, baseline provenance, checkpoint selection, and uncertainty estimates are needed.

Questions for authors
  1. Can you provide ETO with two rollouts per instruction and a highest-reward-failure sampling baseline, reporting retained pair counts, instruction coverage, optimizer updates, and total compute? Please also clarify whether ETO and the sampling baselines receive the same auxiliary SFT regularization and pair-weighting scheme as Ours.

  2. Does the advantage persist when low-, medium-, and high-reward negatives are matched in number and trained against the same expert trajectories? How do negatives from each source compare in reference-adjusted DPO margin/loss, accounting for length?

  3. How many co-evolution rounds are used, how are reference policies initialized or refreshed across rounds, and how are final checkpoints selected? Please report per-round target performance, failure-agent success rates, tie rates, and sizes of both preference datasets, and explain the Qwen3.5 positive-agent ablation's deterioration.

  4. What exact settings account for the Table 6/Table 8 difference and the separate RFT experiment in C.2? For Tables 6–7, which baseline results were rerun and which were taken from prior work, and were the data splits, ScienceWorld filtering, and interaction limits identical? What uncertainty estimates support the reported gains?

7 comments
AuthorsOpenReview ↗

Response to Reviewer RU8B (1/3)

We thank the reviewer for the thoughtful review and constructive feedback.


Q1 / W1 · Learned negative generation versus additional sampling and compute

Q1-1 · Multiple rollouts and highest-reward failure selection

We apologize for the unclear description. Our paper already included the requested comparison using the highest-reward failed candidate. For Best-of-N, we sampled four trajectories per instruction, excluded successes, and selected the highest-reward failure for training. We have corrected the description in Section 5.3.

Ours achieves higher task reward than ETO and Best-of-N across both models and all environments, with the largest gains on ScienceWorld. Results are mean ± SD over three runs, except Best-of-N on Gemma–WebShop, which has two runs.

ModelMethodWebShopScienceWorld seenScienceWorld unseenInterCodeSQL
QwenETO68.35 ± 0.7679.61 ± 0.8974.93 ± 1.1067.01 ± 1.96
Best-of-N70.53 ± 0.9083.62 ± 0.8575.79 ± 1.4766.89 ± 1.82
Ours72.25 ± 0.1587.58 ± 1.1882.94 ± 1.7869.76 ± 0.79
GemmaETO71.47 ± 0.2076.94 ± 0.2069.77 ± 0.7571.93 ± 0.45
Best-of-N70.72 ± 0.3780.86 ± 0.3275.18 ± 2.2771.31 ± 0.61
Ours73.75 ± 0.3485.75 ± 2.0681.51 ± 3.4072.49 ± 0.84

Q1-2 · Training data and computational cost

Following the reviewer’s request, we have reported training data and computational cost over three iterations on Gemma–ScienceWorld (Appendix G, Table 12). Retained target pairs are the expert–failure pairs used for target training, summed across iterations. Instruction coverage is the mean percentage of the 1,483 instructions with at least one target pair per iteration, counting each instruction once across generators. Failure-agent training pairs are excluded from these two metrics. Optimizer updates include both agents where applicable; FLOPs include rollout generation and DPO training.

The positive-agent control in Table 4 trained a separate agent with ETO and supplied its failures for target training. It used more updates and compute than Ours but achieved lower target reward, despite similar retained pair counts. These runs were not exactly matched in pair counts or updates.

MethodRetained target pairsInstruction coverageOptimizer updatesTotal compute (10¹⁸ FLOPs)
ETO1,45632.7%1412.95
Best-of-N2,48655.9%2378.69
Positive-agent control4,36049.0%5888.72
Ours4,56651.3%4957.56

Q1-3 · SFT regularization and pair weighting

ETO and the sampling baselines originally used DPO without auxiliary SFT. We have evaluated ETO with the same SFT term and coefficient as Ours and observed little change in task reward (Appendix G, Table 13). ETO learns from its own failures, whereas our target also learns from off-policy failures generated by the failure agent. With off-policy data, DPO can reduce the likelihood of expert trajectories along with that of failures [1, 2]. We therefore used SFT to reinforce expert demonstrations and stabilize target training.

We weighted each pair by 1/n, where n is the number of retained pairs for its instruction. ETO and the sampling baselines retained at most one pair per instruction, so the same rule assigns each retained pair a weight of one.

ModelMethodWebShopScienceWorld seenScienceWorld unseenInterCodeSQL
GemmaETO71.4776.9469.7771.93
ETO + SFT71.3677.0069.3371.51
Ours73.7585.7581.5172.49

[1] Yan et al. “3D-Properties: Identifying Challenges in DPO and Charting a Path Forward,” ICLR ’25.

[2] Razin et al. “Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization,” ICLR ’25.


Q2 / W2 · Testing the hard-negative explanation

Q2-1 · High-reward failures with matched negative counts

On WebShop, we have compared low-, medium-, and high-reward negatives while matching negative counts, instructions, expert trajectories, SFT initialization, and DPO updates. The reward range is divided into thirds, using only instructions represented in all three groups. Each group received as many updates as one ETO iteration, requiring 26 epochs on Qwen–WebShop and 16 on Gemma–WebShop.

With pair counts and optimizer updates fixed, high-reward negatives give the highest target reward for both models on WebShop.

Training negativesQwen · WebShop (116 instructions)Gemma · WebShop (131 instructions)
Low reward65.0370.10
Medium reward62.0870.61
High reward67.9770.82
AuthorsOpenReview ↗

Response to Reviewer RU8B (2/3)

Q2-2 · Negative difficulty for the current target

Following the reviewer’s request, we have computed length-normalized DPO loss for failures from both agents using Qwen (Appendix G, Table 14). Both sources share a reference checkpoint within each iteration, and each pair is evaluated at its first training update. Failure-agent negatives yield lower losses, especially in iterations 2–3.

A possible explanation is that the target readily suppresses these off-policy trajectories despite their higher environment rewards. Prior work shows that off-policy DPO can be dominated by suppression of rejected responses [1], while both preferred and rejected likelihoods can decrease even as their relative margin increases [2]. Our SFT term directly reinforces expert likelihood to limit this behavior. These policy-dependent effects make DPO loss alone difficult to interpret as negative difficulty for the current target.

Environment reward provides a consistent measure of task progress across sources and remains our proxy for informative failures, rather than a direct measure of target-relative difficulty. We thank the reviewer for motivating further work on target-relative measures that can guide which failures self-evolving agents learn from.

Failure sourceIterationWebShopScienceWorldInterCodeSQL
Target agent10.3240.3340.550
Failure agent10.3220.3180.541
Target agent20.6020.4960.668
Failure agent20.3500.3660.496
Target agent30.5730.5900.627
Failure agent30.3340.3270.476

[1] Yan et al. “3D-Properties: Identifying Challenges in DPO and Charting a Path Forward,” ICLR ’25.

[2] Razin et al. “Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization,” ICLR ’25.

W2 · Explaining the performance gains on InterCodeSQL

On InterCodeSQL, Figure 2 shows comparable numbers of high-reward failures but more low- and medium-reward failures than ETO. The lower high-reward proportion therefore reflects a larger pool of failures in the other bins. These additional negatives from the failure agent can provide useful comparisons with expert demonstrations beyond those generated by the target’s own policy. The performance gains are consistent with this broader training experience, even without an increase in the high-reward proportion.


Q3 / W3 · Training details, data yield, and positive-agent performance

Q3-1 · Iterations, reference policy, and checkpoint selection

We used three training iterations. At the start of each iteration, the reference was set to that agent’s starting checkpoint and held fixed until the iteration ended. For every method, including all baselines, we selected the checkpoint with the highest test reward in each run. ScienceWorld selection used the mean of seen and unseen rewards.

Q3-2 · Per-iteration performance and usable preference pairs

We have reported target performance and usable training data at each iteration for Gemma on ScienceWorld (Appendix G, Table 15). The failure agent’s success and tie rates remain similar, but fewer failure–failure pairs are retained. A usable pair requires both agents to fail on the same instruction with different rewards, so a stable failure rate does not imply a stable training-set size. Target reward continues to improve despite this reduction.

MetricIteration 1Iteration 2Iteration 3
Target reward (seen)80.2683.8387.55
Target reward (unseen)75.1781.4484.42
Failure-agent success rate55.4%54.3%54.8%
Failure-pair tie rate30%28%30%
Target preference pairs1,6621,4861,418
Failure-agent preference pairs276202140

The table below gives the range across three iterations for one run in each model–environment setting (Appendix G, Table 16). InterCodeSQL has the highest tie rates and the fewest failure-agent pairs. Many failures receive the same reward and cannot provide a preference ordering, limiting the data available to train the failure agent.

ModelEnvironmentIterationsFailure-agent successFailure-pair tiesFailure-agent pairsTarget pairs
QwenWebShop334.6–45.1%47–69%230–7071730–2209
QwenScienceWorld373.7–75.5%45–54%82–109829–1001
QwenInterCodeSQL378.3–78.6%80–83%27–32872–924
GemmaWebShop338.5–40.5%49–53%355–3912122–2250
GemmaScienceWorld354.3–55.4%28–30%140–2761418–1662
GemmaInterCodeSQL367.5–69.3%82–83%43–531196–1303
AuthorsOpenReview ↗

Response to Reviewer RU8B (3/3)

Q3-3 · Performance drop in the Qwen3.5 positive-agent ablation

In the positive-agent ablation, the Qwen3.5 target often repeats the wait action instead of completing the ScienceWorld task. At iteration 1, 21% of the inspected unseen episodes end at the step limit, compared with 5% for SFT.

This repeated waiting helps explain the lower reward, which persists through iteration 3. In this comparison, training the additional agent to generate failures gives higher target reward than training it to solve tasks.


Q4 / W3 · Experimental configurations, baseline sources, and uncertainty

Q4-1 · Reconciling original Tables 6 and 8

Original Table 6 used a learning rate of 1 × 10⁻⁶ and reported a WebShop reward of 70.30. Table 8 used 5 × 10⁻⁷ and reported 66.70 at an SFT weight of 0.1. These correspond to Tables 7 and 9 in the revised manuscript.

Q4-2 · Setup of the separate RFT experiment

The RFT experiment in Appendix C.2 used the same Llama-2-7B-Chat setup as original Table 6 (Table 7 in the revised manuscript), with additional DART-style sampling. Five initial rollouts estimated instruction difficulty, and ten candidates were allocated per instruction. Successful trajectories augmented the SFT data, increasing WebShop reward from 60.89 to 61.90.

Q4-3 · Baseline sources and evaluation settings

The ScienceWorld PPO and RFT results in original Table 6 were taken from ETO [1]. We reran all other baselines in original Tables 6–7 (Tables 7–8 in the revised manuscript) with identical data splits, ScienceWorld filtering, and interaction limits.

[1] Song et al. “Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents,” ACL ’24.

Q4-4 · Variation across independent runs

We have evaluated ETO and Ours over three independent runs and report mean task reward ± SD (Appendix G, Table 11). Ours has higher mean reward in every reported model–environment setting.

ModelMethodWebShopScienceWorld seenScienceWorld unseenInterCodeSQL
QwenETO68.35 ± 0.7679.61 ± 0.8974.93 ± 1.1067.01 ± 1.96
Ours72.25 ± 0.1587.58 ± 1.1882.94 ± 1.7869.76 ± 0.79
GemmaETO71.47 ± 0.2076.94 ± 0.2069.77 ± 0.7571.93 ± 0.45
Ours73.75 ± 0.3485.75 ± 2.0681.51 ± 3.4072.49 ± 0.84
Reviewer_SY5dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an iterative training framework in which a target agent learns from negatives generated by a separate failure agent. Both agents start from an expert-supervised checkpoint. The failure agent is trained to prefer higher-reward unsuccessful trajectories, and the target agent learns to prefer expert demonstrations over failures produced by either agent. The agents exchange trajectories as training proceeds. Experiments report improvements over trajectory-preference baselines.

Critical strengths
  1. Updating the failure generator using the target agent's latest failures allows the source of negative trajectories to change as the target improves. The ablation supports continued updating over freezing the generator, even when the frozen generator continues to produce fresh negatives.

  2. The experiments cover different interactive tasks and model families. These results provide evidence that the empirical benefits extend beyond a single task or backbone.

Critical weaknesses
  1. The methodological advance over prior work appears incremental. MACPO already introduces dedicated negative agents within iterative preference learning. The proposed method extends these ideas by training the failure generator to prefer higher-reward failed trajectories and updating it as the target agent improves.

  2. The quality and stability of the failure generator are insufficiently evaluated. The reward-bin analysis measures partial task completion, which does not directly establish how difficult the generated negatives are for the current target. The qualitative examples and downstream gains provide useful evidence, but systematic comparisons of generated-negative quality with the target agent's own failures are limited. The paper also does not track how this quality evolves across training iterations or varies across independent runs. Such measurements would help establish whether the generator consistently supplies useful hard negatives.

Questions for authors

Could the authors directly evaluate generator-produced and target-produced failures on matched tasks, using metrics such as valid-failure yield and negative difficulty for the current target, and report their variation across training iterations and independent runs? Human judgments of error subtlety could provide complementary validation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer SY5d

We thank the reviewer for the careful review and constructive suggestions.


Q1 / W2 · Generated failures across training iterations and independent runs

Following the reviewer’s request, we have compared failures generated by the two agents on the same training instructions across iterations and independent runs. We report valid-negative yield as the percentage of instructions for which each agent generates at least one failure retained for target training. Values are mean ± SD across three runs.

Please note that iteration 1 is the initial state before co-evolution. At this stage, the failure agent has not yet learned from an updated target agent. In iterations 2 and 3, it generates valid negatives for more instructions than the target agent in every setting. As the agents co-evolve across iterations, the failure agent continues to generate valid negatives while the target agent’s own failures become less frequent. Our framework achieves higher task reward than ETO and Best-of-N across both models and all environments (Tables 2–3). We have added the corresponding plots to Appendix F (Figure 3).

ModelEnvironmentMethodIter 1Iter 2Iter 3
Qwen3.5-9BWebShopTarget agent55.9 ± 0.221.3 ± 0.320.3 ± 0.7
Failure agent38.1 ± 2.534.8 ± 3.030.7 ± 1.8
ScienceWorldTarget agent24.8 ± 0.216.3 ± 0.612.9 ± 0.9
Failure agent24.9 ± 1.625.5 ± 1.023.7 ± 1.0
InterCodeSQLTarget agent22.0 ± 0.118.6 ± 1.019.7 ± 0.4
Failure agent21.8 ± 0.321.5 ± 0.521.2 ± 0.6
ModelEnvironmentMethodIter 1Iter 2Iter 3
Gemma-4-12B-itWebShopTarget agent30.0 ± 0.226.4 ± 0.726.6 ± 0.8
Failure agent44.9 ± 0.845.6 ± 1.143.5 ± 1.3
ScienceWorldTarget agent36.5 ± 0.023.3 ± 2.218.2 ± 1.9
Failure agent44.8 ± 1.044.4 ± 1.144.6 ± 0.6
InterCodeSQLTarget agent33.8 ± 0.026.0 ± 0.724.0 ± 0.4
Failure agent31.5 ± 0.931.1 ± 0.332.6 ± 0.8

Additionally, to test the learning value of higher-reward failures for the target agent, we have trained the target from the same SFT checkpoint using equal-sized negative sets from three equal-width reward bins on WebShop. The three conditions use the same task instructions, expert trajectories, and number of DPO updates (116 instructions for Qwen; 131 for Gemma). High-reward negatives lead to the highest target reward for both models, with a small gain over medium-reward negatives for Gemma. This provides complementary evidence for their learning value (Appendix F, Table 10).

Training negativesQwen · WebShopGemma · WebShop
Low reward65.0370.10
Medium reward62.0870.61
High reward67.9770.82

W1 · Difference from MACPO

Our contribution is a self-evolving agent framework that jointly learns task execution and hard-negative generation. Although MACPO also uses the term “negative agents”, it initializes them from dispreferred responses paired with preference labels before the main training phase begins and keeps them frozen throughout that phase. Our failure agent instead learns from both agents' own failures throughout training, using environment rewards to form preference pairs without a separate dataset of labeled dispreferred responses. We have clarified this distinction in Section 2.

Thank you for taking the time to review our paper.

We believe that our responses and additional experiments have addressed the concerns you raised. We would appreciate it if you could review these updates and let us know if any concerns remain unaddressed or if you have further questions.

We would be grateful if you could take these updates into account in your final assessment.

Thanks for the additional results.

The results across iterations and runs help address my question about the consistency of negative generation. The reward bin comparison also supports the learning value of higher-reward failures. However, I still have a question about failures produced by the generator. Are these failures harder for the target agent to distinguish from successful trajectories than its own failures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.