acceptodds
Desk rejected at ICLR 2027

ACSE: Agentic Collaborative Self-Evolution for Embodied Intelligence

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Robot policies are typically trained on fixed task and data distributions, with improvements primarily driven by data scaling. Although data scaling has improved generalization to complex tasks, existing approaches largely rely on passive learning from predefined tasks, limiting proactive challenge discovery and capability expansion. We propose Agentic Collaborative Self-Evolution (ACSE), a framework that couples task generation with agent learning in a closed loop, enabling embodied agents to discover and overcome their limitations. ACSE comprises a Scene Generator and a Robotic Solver. The Robotic Solver integrates high-level task reasoning, world modeling, and action execution, while the Scene Generator explores the Solver’s current capability boundary to generate challenging, informative scenarios. The Solver improves through reinforcement learning using closed-loop rollouts, and newly exposed capability gaps guide the Generator’s evolution, allowing scenario difficulty and agent capabilities to co-evolve. Unlike data scaling under fixed task distributions or curricula with predefined difficulty schedules, ACSE uses interaction feedback to adapt the training distribution to the agent’s evolving capabilities. Experiments show that ACSE continually generates novel, informative challenges and substantially improves generalization across complex long-horizon manipulation tasks and out-of-distribution settings. These results suggest a promising path toward autonomous challenge discovery and continual self-improvement in embodied intelligence.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_93gqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ACSE generates task variants from solver failures, validates physical feasibility, filters by completion probability, and weights training through a population meta-game. Its AWAM implementation updates experience memory, action-head LoRA, and a supervised physical head while freezing Qwen3.5 and the future predictor. Evaluation covers RoboDojo, BiCoord, and two physical manipulation tasks.

Critical strengths
  • Practical task construction. Asset, constraint, and reference-execution checks distinguish physical feasibility from difficulty for the current solver. This is a useful design for generating robotic training tasks.

  • Evidence of adaptation gains. BiCoord success improves from 38.2% to 46.0%, exceeding random sampling (40.1%) and failure-weighted sampling (41.6%). Table 5 also supports combining memory and parameter updates. The physical evaluation shares randomized layouts and fixes model parameters during testing.

Critical weaknesses
  1. The population-game formulation is not connected to the generator algorithm. Eq. 6 defines a generator response against a mixture of solvers. Algorithm 1 instead generates and filters tasks using the incumbent solver before computing that mixture; it does not show how the solver mixture affects subsequent generation or selection.

    The concrete meaning of a generator strategy and its population update is also insufficiently specified. Calling generation a heuristic response does not establish its relationship to Eq. 6. Consequently, the distinctive algorithmic contribution beyond failure-guided generation and population-based task weighting remains unclear.

  2. The capability-boundary and generalization claims are not isolated experimentally. Table 4 compares sampling weights, and Table 5 tests solver updates. Neither separates failure-conditioned proposals or capability filtering from data expansion and reweighting under matched interaction and RL budgets. Nor is the simulation task/scene separation between generation, adaptation, and final evaluation specified. The results support adaptation, but do not establish that boundary-targeted generation drives improvement on genuinely unseen conditions.

Questions for authors
  1. What concretely defines a generator strategy, and where does the solver mixture enter its proposal or selection procedure?
  2. Which existing controls isolate capability filtering and targeted generation? Please clarify matched budgets and the held-out simulation tasks/scenes.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 93gq

Thank you for carefully reviewing our manuscript. We appreciate your feedback and respond to each comment below.

Clarification of our core contribution. We thank the reviewer for the thoughtful feedback. Our core contribution is ACSE, an agentic framework that enables the training task distribution and the solver to co-evolve in long-horizon embodied manipulation. ACSE addresses three questions:

  1. What should the solver practice after a failure? The Capability Analyzer uses execution traces and observed object states to identify the first unmet subgoal. The generator then creates physically feasible task variants that allow the solver to practice the operation that failed.

  2. How should a limited training budget be allocated? For each generation rule, ACSE estimates each solver checkpoint’s mean return on tasks sampled from that rule. It uses the resulting payoff matrix to compute sampling weights. Two rules with the same failure rate for the current solver can therefore receive different weights if earlier checkpoints achieved different returns on their tasks.

  3. How can the solver revisit earlier skills while acquiring new ones? ACSE retains historical generation rules so the solver can continue practicing earlier tasks. Failures observed after each solver update guide the construction of new task variants that target difficulties the solver still encounters.

This feedback loop extends the agent’s autonomy from executing assigned tasks to shaping its own training curriculum through experience. It represents an initial exploration of recursive self-improvement in embodied systems.

AuthorsOpenReview ↗

Response to Reviewer 93gq

Q1: The population-game formulation is not connected to the generator algorithm. Eq. 6 defines a generator response against a mixture of solvers. Algorithm 1 instead generates and filters tasks using the incumbent solver before computing that mixture; it does not show how the solver mixture affects subsequent generation or selection.

The concrete meaning of a generator strategy and its population update is also insufficiently specified. Calling generation a heuristic response does not establish its relationship to Eq. 6. Consequently, the distinctive algorithmic contribution beyond failure-guided generation and population-based task weighting remains unclear.

R1: Thank you for raising this question. The solver mixture first enters the generator response in line 8 of Algorithm 1. In lines 3–5, the generator uses failure evidence from the incumbent solver to propose candidate tasks and adds physically feasible candidates to the training pool. Lines 6–7 estimate the payoff matrix and compute the solver and generator mixtures. In lines 8–9, the generator samples solver checkpoints according to the solver mixture and uses unmet subgoals in their execution traces to propose a new generation rule.

Generator strategy. A generator strategy is a reusable rule that defines a distribution over task instances. For example, a bimanual handover rule can preserve the handover goal while specifying how the handover position and the relative timing of the two arms may vary. Scenes generated by this rule are instances of the same strategy. Each rule corresponds to one column of the payoff matrix. Each entry in that column estimates one solver checkpoint’s mean return on tasks sampled from the rule.

Connection to Eq. (6). Equation (6) seeks an admissible rule that minimizes the expected return of the solver mixture. In practice, the generator samples solver checkpoints according to the mixture weights in Eq. (6), identifies unmet subgoals in their execution traces, and constructs a new rule targeting those subgoals. Checkpoints with higher weights are sampled more often in expectation, so the difficulties they encounter are more likely to guide the proposal. This procedure provides a heuristic response to Eq. (6).

Population update. In lines 10–11, the solver samples training tasks according to the generator mixture and updates its policy. Lines 12–13 consolidate the Harness memory and form the new solver checkpoint. Line 14 adds the checkpoint and the new generation rule to their respective populations. In the next round, the new rule corresponds to a new column in the payoff matrix. We have revised Algorithm 1 and the main text to make both population updates explicit.

AuthorsOpenReview ↗

Response to Reviewer 93gq

Q2: The capability-boundary and generalization claims are not isolated experimentally. Table 4 compares sampling weights, and Table 5 tests solver updates. Neither separates failure-conditioned proposals or capability filtering from data expansion and reweighting under matched interaction and RL budgets. Nor is the simulation task/scene separation between generation, adaptation, and final evaluation specified. The results support adaptation, but do not establish that boundary-targeted generation drives improvement on genuinely unseen conditions.

R2: Thank you for raising this point. We added a uniformly sampled, failure-guided baseline to Table 4. Comparing it with Random tests whether failure diagnoses improve scene generation when the number of new scenes and the training budget are held fixed. In our formulation, the capability interval describes the target difficulty range. Algorithm 1 admits physically feasible candidates to the training pool without applying a fixed completion-rate threshold, so there is no separate filtering step to ablate.

Extended Table 4. Scene generation and task sampling on BiCoord. SR denotes full-task success rate, and SSR denotes stage-wise success rate. We have updated Table 4 of the manuscript accordingly.

Table 1. Scene generation and task sampling on BiCoord

MethodFailure-guided generationSamplingSR (%) ↑SSR (%) ↑TL ↓
Base——38.253.8491
RandomNoUniform40.155.4470
Random + failure guidanceYesUniform40.955.0452
Failure-WeightedYesFailure-weighted41.656.3438
ACSEYesPayoff-weighted46.061.1410

1. Random scene expansion and additional training. Continued training on randomly generated scenes increased SR from 38.2% to 40.1% and SSR from 53.8% to 55.4%, gains of 1.9 and 1.6 percentage points, respectively.

2. Failure-guided generation (Random vs. Random + failure guidance). Both methods generate new scenes and sample uniformly from their respective scene pools. The added control uses unmet subgoals identified in execution traces to guide scene modifications. It outperforms Random on the same held-out instances, indicating that failure-guided generation improves performance on unseen instances when the number of new scenes and the training budget are matched.

3. Adaptive sampling and co-evolution (Failure-Weighted vs. ACSE). Failure-Weighted allocates training rollouts according to the current solver’s empirical failure rates. ACSE uses returns across solver checkpoints to weight generator strategies and samples checkpoints from the solver mixture to guide subsequent scene generation. With the same number of solver-training rollouts, ACSE outperforms Failure-Weighted by 4.4 percentage points in SR (41.6% to 46.0%) and 4.8 percentage points in SSR (56.3% to 61.1%). These gains quantify the benefit of the full ACSE configuration over the failure-weighted baseline.

Budgets and evaluation split. All adaptation methods used the same initial solver, number of solver-training rollouts, and solver-update rules. Random and Failure-Random also used the same mutation operators, number of new scenes, total simulation rollouts, and RL update steps. We counted each rollout once, including rollouts reused for payoff estimation. Scene generation and solver adaptation used training data only. We selected checkpoints on validation data and evaluated all methods on the same held-out BiCoord instances. These results therefore measure generalization to new instances within the 18 known task categories.

Reviewer_Aim3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper presents ACSE, a framework in which a capability-aware scene generator uses execution feedback to construct physically feasible tasks near the solver’s current capability boundary. The solver improves through Harness memory updates and action model adaptation, and its updated performance guides subsequent task generation. Experiments on manipulation benchmarks and two real robot tasks show improvements over the initial solver.

Critical strengths
  1. The paper addresses an interesting problem of adapting training environments to a robot’s changing capabilities. Using execution failures to guide future training is a useful direction for improving long-horizon manipulation.

  2. The experiments include simulation benchmarks, real robot tasks, and ablations on sampling strategies and solver updates, providing encouraging evidence for the framework.

Critical weaknesses
  1. Training on tasks near the current capability boundary is not a new idea. The paper selects tasks based on the solver’s current success rate, but does not establish a direct relationship between this selection criterion and subsequent capability gains. Comparisons with additional task scheduling strategies would help justify this choice.

  2. The framework of generator and solver co-evolution is not particularly novel on its own. Here, the generator relies on a Capability Analyzer to extract failure information. Although the paper describes a failure attribution mechanism, it provides limited independent evidence that the analyzer can identify the actual causes of failure. A separate evaluation of attribution accuracy, together with an ablation on how the diagnoses affect task generation, would make this component more convincing.

  3. The evidence for sustained capability expansion is limited. Longer evolution runs, evaluation on a fixed set of unseen tasks, and measurements of forgetting would strengthen this claim.

  4. It remains unclear how much the gains come from collaborative self-evolution rather than adaptive task sampling and additional training. Clearer comparisons under the same computational budget, along with an analysis of generation costs and training efficiency, would help assess the benefit of the full framework.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Aim3

Thank you for carefully reviewing our manuscript. We appreciate your feedback and respond to each comment below.

Clarification of our core contribution. We thank the reviewer for the thoughtful feedback. Our core contribution is ACSE, an agentic framework that enables the training task distribution and the solver to co-evolve in long-horizon embodied manipulation. ACSE addresses three questions:

  1. What should the solver practice after a failure? The Capability Analyzer uses execution traces and observed object states to identify the first unmet subgoal. The generator then creates physically feasible task variants that allow the solver to practice the operation that failed.

  2. How should a limited training budget be allocated? For each generation rule, ACSE estimates each solver checkpoint’s mean return on tasks sampled from that rule. It uses the resulting payoff matrix to compute sampling weights. Two rules with the same failure rate for the current solver can therefore receive different weights if earlier checkpoints achieved different returns on their tasks.

  3. How can the solver revisit earlier skills while acquiring new ones? ACSE retains historical generation rules so the solver can continue practicing earlier tasks. Failures observed after each solver update guide the construction of new task variants that target difficulties the solver still encounters.

This feedback loop extends the agent’s autonomy from executing assigned tasks to shaping its own training curriculum through experience. It represents an initial exploration of recursive self-improvement in embodied systems.

AuthorsOpenReview ↗

Response to Reviewer Aim3

Q1: Training on tasks near the current capability boundary is not a new idea. The paper selects tasks based on the solver’s current success rate, but does not establish a direct relationship between this selection criterion and subsequent capability gains. Comparisons with additional task scheduling strategies would help justify this choice.

R1: Thank you for raising this point. Training on tasks near a learner’s capability boundary has a long history in curriculum learning. We have revised Section 3.2 to clarify how ACSE uses completion probability to characterize task difficulty relative to the current solver. In practice, physically feasible task variants enter the training pool without a fixed success-rate cutoff. ACSE links two decisions: which failed operation to target when generating new scenes, and how often to sample each generation rule based on performance across solver checkpoints.

Two recent studies illustrate related approaches in robotic manipulation. DynaForge [1] trains a residual policy on groups of rollouts that contain both successes and failures under matched conditions. FIND [2] selects feasible tasks from a predefined library and prioritizes those with low recent success rates. In long-horizon manipulation, a task-level success rate alone does not identify which subgoal requires practice. For example, a robot may grasp an object reliably yet struggle with the subsequent bimanual handover. ACSE uses execution traces to identify the failed step and can generate scenes that vary the handover position or the relative timing of the two arms. It weights generation rules using the returns that solver checkpoints achieve on tasks drawn from each rule. Two rules with the same failure rate for the current solver can therefore receive different weights if earlier checkpoints perform differently on them.

Comparison with existing scheduling strategies. In Table 4, random sampling, failure-weighted sampling, and ACSE achieve full-task success rates of 40.1%, 41.6%, and 46.0% on BiCoord, respectively. Their corresponding stage-wise success rates are 55.4%, 56.3%, and 61.1%. ACSE improves on failure-weighted sampling by 4.4 percentage points in full-task success and 4.8 percentage points in stage-wise success. Both preprints appeared around the time of our ICLR submission. At the time of evaluation, we could not locate an official implementation of DynaForge, and FIND’s public real-robot training pipeline required adaptation to BiCoord. These constraints made it difficult to complete and validate a matched-budget comparison during the rebuttal period.

Comparison with failure-guided trajectory perturbation. We designed an additional comparison to evaluate ACSE against a baseline that perturbs trajectories locally at diagnosed failure points. Both methods use failure diagnoses. The baseline perturbs the trajectory segment corresponding to the current solver’s failed subgoal and samples uniformly from the resulting feasible variants. ACSE samples checkpoints according to the solver mixture and uses their failure traces to construct new generation rules. For solver training, it samples rules according to the generator mixture and then draws tasks generated by those rules.

Table 1. Comparison of failure-guided trajectory perturbation and ACSE on BiCoord.

MethodScene generationTraining task samplingSR (%) ↑SSR (%) ↑
Failure-guided trajectory perturbationPerturb failed segmentsUniform40.355.6
ACSE (ours)Derive rules from failures across checkpointsGenerator mixture46.061.1

The comparison protocol specifies the same initial solver checkpoint, physical feasibility checks, and held-out test instances for both methods. It also matches the number of new training scenes, total simulation interactions, and RL update steps. The comparison is designed to measure the combined contribution of ACSE’s generation and sampling mechanisms relative to local failure-guided perturbation.

[1] Yiyang Jin, Yu Zheng, Xiao He, and Hesheng Wang. “DynaForge: Planning-Guided Residual Learning for Dynamic Manipulation Demonstration Generation.” arXiv preprint, 2026. arXiv:2609.25631

[2] Yuan Fang, Zechu Li, Haolei Tong, Puze Liu, and Georgia Chalvatzaki. “Find Something You Can’t Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models.” arXiv preprint, 2026. [arXiv:2609.32069]

AuthorsOpenReview ↗

Response to Reviewer Aim3

Q2: The framework of generator and solver co-evolution is not particularly novel on its own. Here, the generator relies on a Capability Analyzer to extract failure information. Although the paper describes a failure attribution mechanism, it provides limited independent evidence that the analyzer can identify the actual causes of failure. A separate evaluation of attribution accuracy, together with an ablation on how the diagnoses affect task generation, would make this component more convincing.

R2: Thank you for raising this point. Generator–solver co-evolution has precedents in open-ended learning. ACSE contributes an agentic mechanism for long-horizon manipulation that turns execution failures into targeted training data. The Capability Analyzer identifies an unmet subgoal and the completion condition that remains unsatisfied. The generator uses this diagnosis to propose physically feasible task variants, and the solver updates its policy through interactions on those variants. The updated solver’s execution traces then guide the next round of generation. We view this feedback loop as an initial exploration of recursive self-improvement in embodied systems.

Independent evaluation of attribution accuracy. We evaluated the analyzer on 300 held-out failure traces, with 100 traces from each of three BiCoord tasks. We selected Cook for its dependencies between successive steps, Place Plate and Cup for its repeated grasping and placement steps, and Clean Table for its multi-object clearing sequence. Without seeing the analyzer’s output, annotators reviewed the execution videos, simulator states, and task completion criteria to identify the first unmet subgoal and the specific condition that remained unsatisfied. We counted a diagnosis as correct only if it matched the reference annotation on both elements. The analyzer achieved 93.0% accuracy (279/300) across the three tasks.

Table 2. Diagnostic accuracy of the Capability Analyzer on 300 held-out failure traces from three BiCoord tasks

TaskCorrect diagnoses / failure tracesAccuracy (%)
Cook91 / 10091.0
Place Plate and Cup95 / 10095.0
Clean Table93 / 10093.0
Overall279 / 30093.0

Figure 13 illustrates the two elements of a diagnosis. In the Cook rollout, the solver proceeds to the next action before the food reaches the plate. The failed subgoal is the food transfer, and the unmet condition is that the food should be on the plate when the transfer ends. This diagnosis gives the generator a concrete basis for varying the object layout or the timing of subsequent actions (Section 3.2 and Appendix A.1).

Effect of diagnostic guidance on task generation. We added Random + failure guidance to Table 4. This variant and Random use the same base tasks, mutation operators, and physical feasibility checks. Both generate the same number of new scenes and sample uniformly from their respective scene pools. The only difference is whether the generator receives the diagnosed subgoal and its unmet condition when proposing a scene. With the same number of simulation rollouts and RL update steps, Random + failure guidance outperforms Random on the same held-out BiCoord instances. This comparison measures the downstream effect of providing the diagnosis to the generator while controlling for data expansion and sampling.

Table 3. Failure-guided generation and sampling on BiCoord. SR denotes full-task success rate, and SSR denotes stage-wise success rate.

MethodFailure-guided generationSamplingSR (%) ↑SSR (%) ↑TL ↓
Base——38.253.8491
RandomNoUniform40.155.4470
Random + failure guidanceYesUniform40.955.9452
Failure-WeightedYesFailure-weighted41.656.3438
ACSEYesPayoff-weighted46.061.1410
Reviewer_CuMDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets at the problems of proactive challenge discovery and continual capability expansion for robot controll policies, and propose a co-evolving framework, ACSE, with one task generator trained along side with the learning policy. The role of the task generator is to identify the capability boundaries of the learning policy at each training iteration, while the policy is optimized using RL algorithms on these identified task distribution. This capability learning framework is compatible with a wide range of embodied model architectures, including VLA, WAM, Agent-WAM. The experiment results show that ACSE is outperforming a selective set of baseline methods including imitation learning, fixed-distribution RL, and failure-guided RL on three manipulation benchmarks.

Critical strengths
  1. This paper is clearly written with a strong motivation for the problem identified.
  2. The experimental setup is reasonably broad in baselines and tasks.
Critical weaknesses
  1. A mismtach between the name of framework and the underlying mechanism. The underlying framework can be characterized as a co-evolving adversarial learning framework, way off from a self-evolving framework. As in the end, it is the task generator generates the learning task distribution, while the reward is provided by the environment.
  2. There is a conceptual gap regarding the continual learning/capability. It is totally sensible to include a discussion on how well the policy is learning new and unseen tasks as it incrementally expands the capability frontier. However, such a claim on continual learning/capability requires an explicit trade-off analysis between stability and plasticity. The improving benchmark results showing in Table 3 is insufficient.
  3. The capability expansion claim requires further evidence. Like the illustrations in Figure 3, capability expansion describes a particular learning pattern that the model masters a task at which its initialization consistently fails to complete. A general increase in the successful rate after training may not indicate capability expansion.
  4. The insights of this work is limited in its current form, as effectively it proposes an adaptive task scheduling. It can be substantially strengthened from the two aspects: 1) why a static or simple heuristic-guided task planning fails to learning the optimal? 2) provide intuitive explanation or empirical evidence on how your method ACSE manage to close the gap.
  5. Citation format needs revisions. For example, the introduction to baselines (starting from Line 410).
Questions for authors
  1. How ACSE is guaranteed to learn all possible tasks in the task space? Given that those tasks remaining under the threshold c_min are consistently ignored throughout training?
  2. Can ACSE mitigate the catastrophic forgetting isssue widely studied in the continual learning literature?
  3. Adversarial learning frameworks are known for learning stability issues, how is ACSE implemented in this regard?
7 comments
AuthorsOpenReview ↗

Response to Reviewer CuMD

Thank you for carefully reviewing our manuscript. We appreciate your feedback and respond to each comment below.

Q1: A mismtach between the name of framework and the underlying mechanism. The underlying framework can be characterized as a co-evolving adversarial learning framework, way off from a self-evolving framework. As in the end, it is the task generator generates the learning task distribution, while the reward is provided by the environment.

R1: Thank you for pointing out the ambiguity in the framework's name. We agree that ACSE implements capability-constrained adversarial co-evolution between the task generator and the robotic solver. The term self-evolution refers to the coupled system as a whole. The environment supplies rewards according to externally specified rules, and feasibility constraints determine which tasks are admissible. Within these constraints, the system uses its own interaction history to generate tasks, allocate training effort, and update the solver.

The interaction is adversarial within each round because the generator seeks feasible tasks that expose weaknesses near the solver's current capability boundary, while the solver learns to improve its return on those tasks. The term collaborative describes the reciprocal updates across rounds. The solver's failures guide the generation of new tasks, and training on those tasks changes the solver's capabilities, which in turn shape the next task distribution. We have clarified this system-level meaning of collaborative self-evolution at its first use in the manuscript.

Q2: There is a conceptual gap regarding the continual learning/capability. It is totally sensible to include a discussion on how well the policy is learning new and unseen tasks as it incrementally expands the capability frontier. However, such a claim on continual learning/capability requires an explicit trade-off analysis between stability and plasticity. The improving benchmark results showing in Table 3 is insufficient.

Can ACSE mitigate the catastrophic forgetting isssue widely studied in the continual learning literature?

R2: Thank you for raising the question of how ACSE balances learning new skills with retaining existing ones. In each round, the generator samples solver checkpoints from the solver mixture and uses unmet subgoals in their execution traces to propose a new generation rule. The new rule joins the generator population at the end of the round. Historical rules remain in the population, and the generator mixture determines how often each rule supplies training tasks. This allows the solver to revisit earlier task distributions while practicing newly generated variants.

Learning on held-out instances. Table 2 in the main paper compares the initial solver with the Round 3 checkpoint on the same held-out instances across 18 BiCoord task categories. Full-task success increased from 18% to 46% on Clean Table and from 40% to 58% on Place Plate and Cup. Success increased on 17 of the 18 tasks and remained unchanged on one. Mean success rose from 38.2% to 46.0%. These results measure learning on unseen instances within known task categories.

Retention of existing skills. We selected the three tasks with the highest initial success rates using only performance before evolution. We evaluated all five checkpoints on the same held-out instances under identical initial conditions. For each task, we measured forgetting as the difference between its highest observed success rate across the five checkpoints and its Round 4 success rate. We averaged these task-level differences so that a gain on one task could not offset a decline on another.

Table 1. Skill retention across four evolution rounds. Success rates are percentages. Forgetting is the decline from each task’s peak to Round 4, in percentage points (pp).

TaskInitialRound 1Round 2Round 3Round 4Forgetting (pp)
Balance Roller78.079.080.078.078.02.0
Sweep Block72.074.077.076.075.02.0
Exchange Mics63.065.064.067.066.01.0
Mean71.072.773.773.773.01.7

By Round 4, mean success across all 18 tasks was 7.3 percentage points above initialization (38.2% to 45.5%). At that same checkpoint, all three retention tasks matched or exceeded their initial success rates, with a mean decline of 1.7 percentage points from their individual peaks. We therefore observed limited forgetting on these three tasks over four rounds. The extent to which revisiting historical rules contributed to retention remains an open question.

AuthorsOpenReview ↗

Response to Reviewer CuMD

Q3: The capability expansion claim requires further evidence. Like the illustrations in Figure 3, capability expansion describes a particular learning pattern that the model masters a task at which its initialization consistently fails to complete. A general increase in the successful rate after training may not indicate capability expansion.

R3: Thank you for raising this point. We examine capability expansion by identifying a specific operation that the initial solver failed to complete and testing whether the evolved solver succeeds under matched conditions. Figure 13 illustrates the food-transfer step in the Cook task. In the rollout shown, the initial solver moves on to the next action before the food reaches the plate and ultimately fails to complete the task. The evolved solver coordinates pan shaking and pouring to transfer the food onto the plate and completes the cooking sequence. In ACSE, an unmet subgoal of this kind provides a concrete training target. The Capability Analyzer identifies the failed step from execution traces and observed object states. The generator then proposes scene variants that target that step, and physically feasible variants enter the training pool.

To test whether the improvement illustrated in Figure 13 recurs, we selected five held-out Cook scenes in which the initial solver had failed to transfer the food. We selected the scenes using only failure records collected before evolution. For each scene, we evaluated both solvers on the same 10 new seeds, with matched initial layouts and execution budgets. This yielded 50 paired trials.

Table 2. Paired evaluation on five challenging Cook scenes.

OutcomeInitial solverEvolved solver
Food transferred onto the plate3/50 (6%)12/50 (24%)
Full Cook task completed1/50 (2%)9/50 (18%)

The initial solver completed the full task only once, recording no successes in four of the five scenes. Across the 50 matched scene–seed pairs, the evolved solver completed the food transfer in at least nine cases in which the initial solver did not. It also completed the full task in at least eight cases in which the initial solver did not. These results show that the evolved solver completed the operation illustrated in Figure 13 under multiple test conditions in which the initial solver was unsuccessful.

AuthorsOpenReview ↗

Response to Reviewer CuMD

Q4: The insights of this work is limited in its current form, as effectively it proposes an adaptive task scheduling. It can be substantially strengthened from the two aspects: 1) why a static or simple heuristic-guided task planning fails to learning the optimal? 2) provide intuitive explanation or empirical evidence on how your method ACSE manage to close the gap.

R4: Thank you for raising this point. A fixed curriculum sets training priorities in advance. Failure-weighted sampling adjusts these priorities using the current solver’s empirical failure rates. Its limitation is that two task distributions with the same current failure rate receive the same priority, even if earlier checkpoints performed differently on them. For example, the current solver may struggle equally with handover position and arm timing, although only the timing difficulty has persisted across checkpoints.

ACSE uses this history to make two decisions in each round. First, how often should the solver practice each existing challenge? Each entry in the payoff matrix estimates one solver checkpoint’s mean return on tasks drawn from one generation rule. Multiplicative-weights updates use this matrix to compute the solver and generator mixtures. The solver samples rules according to the generator mixture and draws training tasks associated with those rules. Rules with the same failure rate for the current solver can therefore receive different weights when their returns differ across earlier checkpoints. Historical rules remain in the population and can continue to supply training tasks.

Second, which new challenges should the generator introduce? The generator samples checkpoints according to the solver mixture, identifies unmet subgoals in their execution traces, and proposes a new rule targeting those difficulties. The rule joins the population at the end of the round and can generate task variants in subsequent rounds. Physically feasible variants enter the training pool without a fixed completion-rate cutoff. The solver mixture thus guides the construction of future training tasks, while the generator mixture allocates practice across existing rules.

Equation (9) characterizes the optimization within a round. For fixed populations and a fixed payoff matrix with bounded returns, full-information multiplicative-weights updates with suitable learning rates drive the restricted duality gap of the averaged mixtures toward zero. This result concerns the current population game. We assess the effect on the solver’s manipulation performance empirically.

Table 4 provides the comparison on BiCoord. Under uniform sampling, full-task success was 40.1% for Random and 40.9% for Failure-Random, which uses failure diagnoses to guide scene generation. Failure-Weighted also uses failure-guided generation and achieved 41.6% by sampling according to the current solver’s failure rates. Full ACSE achieved 46.0%. Relative to Failure-Weighted, ACSE improved full-task success by 4.4 percentage points and stage-wise success by 4.8 percentage points (56.3% to 61.1%). This comparison measures the combined benefit of using population payoffs to allocate training and the solver mixture to guide subsequent generation.

Clarification of our core contribution. We thank the reviewer for the thoughtful feedback. Our core contribution is ACSE, an agentic framework that enables the training task distribution and the solver to co-evolve in long-horizon embodied manipulation. ACSE addresses three questions:

  1. What should the solver practice after a failure? The Capability Analyzer uses execution traces and observed object states to identify the first unmet subgoal. The generator then creates physically feasible task variants that allow the solver to practice the operation that failed.

  2. How should a limited training budget be allocated? For each generation rule, ACSE estimates each solver checkpoint’s mean return on tasks sampled from that rule. It uses the resulting payoff matrix to compute sampling weights. Two rules with the same failure rate for the current solver can therefore receive different weights if earlier checkpoints achieved different returns on their tasks.

  3. How can the solver revisit earlier skills while acquiring new ones? ACSE retains historical generation rules so the solver can continue practicing earlier tasks. Failures observed after each solver update guide the construction of new task variants that target difficulties the solver still encounters.

Reviewer_aRNROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ACSE uses execution failures to generate feasible training tasks, selects tasks using capability estimates, and adapts an agentic robot solver through population-based sampling, memory updates, and parameter updates. Experiments on RoboDojo and BiCoord, sampling and solver-update ablations, and real-robot results evaluate the framework.

Critical strengths
  1. The paper addresses a useful training problem: selecting feasible experiences that respond to a robot’s current failures. Connecting task construction and solver adaptation is a meaningful systems contribution.
  2. The results include relevant controls. BiCoord success improves from 38.2% to 46.0%; random and failure-weighted sampling reached 40.1% and 41.6%. Removing either memory evolution or parameter adaptation also reduces performance. These comparisons provide evidence for the combined approach.
Critical weaknesses

The main limitation is that the implemented population-sampling procedure and the controls in Table 4 are not described concretely enough to determine what explains the gain over simpler sampling. This limits assessment of the distinctive mechanism, but does not establish that the comparison is unfair or negate the reported system-level improvement. Clarification of the existing setup may resolve the concern without new experiments.

Questions for authors
  1. Could you describe one actual round, including capability thresholds, generator strategies, population construction, and weight updates? For Table 4, which task-generation, filtering, interaction-budget, and solver-update settings are shared, and how are adaptation, checkpoint selection, and final evaluation separated? This is a clarification request, not a request for additional benchmarks or broad ablation sweeps.
  2. Could you reconcile the reported summaries with the underlying records? Table 3 peaks in round three, contrary to the fourth-round claim in the text. Table 2’s initial-solver TL entries average 463.56 rather than 491 unless another weighting is intended. Table 7’s percentages do not all match the stated 30 trials, and its evolved rates average 86.8 rather than 86.6. Corrected summaries and success counts would resolve these reporting concerns.
2 comments
AuthorsOpenReview ↗

Response to Reviewer aRNR

Thank you for carefully reviewing our manuscript. We appreciate your feedback and respond to each comment below.

Q1. Could you describe one evolution round, including the capability thresholds, generator strategies, population construction, and weight updates? Which settings are shared across the Table 4 comparisons, and how are adaptation, checkpoint selection, and final evaluation separated?

R1: Thank you for requesting clarification. We use Round 2 to illustrate how ACSE constructs its strategy populations, updates their weights, and uses the resulting mixtures for task generation and solver training.

Task generation and capability thresholds. Round 2 starts from the solver checkpoint saved at the end of Round 1. The Capability Analyzer identifies unmet subgoals in its execution traces. The generator uses this evidence and the existing generation rules to propose task variants. Physically feasible variants enter the training pool, and each variant is associated with the rule that generated it. In our formulation, cθ(T)c_{\theta}(T) denotes the solver’s completion probability on task TT, and c^θ(T)\hat{c}_{\theta}(T) is its empirical estimate. The interval [cmin⁡,cmax⁡][c_{\min}, c_{\max}] characterizes the target difficulty range. The implementation admits physically feasible candidates without applying a fixed completion-rate cutoff.

Populations and weight updates. At the start of Round 2, the solver population contains the initial checkpoint and the Round 1 checkpoint. Each checkpoint includes the model parameters and Harness memory. The generator population contains reusable generation rules accumulated through Round 1. For example, a handover rule can preserve the transfer goal while varying the handover position. The resulting scenes are distinct instances of the same rule.

Each payoff-matrix entry estimates one solver checkpoint’s mean return on tasks drawn from one generation rule. With both populations and the matrix fixed, multiplicative-weights updates favor solver checkpoints with higher returns against the current generator mixture and generation rules that yield lower solver returns against the current solver mixture. ACSE then uses the two iteration-averaged mixtures. The generator samples checkpoints from the solver mixture and uses unmet subgoals in their traces to propose a new rule. The solver samples rules from the generator mixture and trains on tasks associated with those rules, holding its Harness memory fixed during RL updates. At the end of the round, ACSE consolidates the training experience into memory and adds the updated solver checkpoint and the new rule to their respective populations. BiCoord full-task success increased from 41.5% in Round 1 to 44.0% in Round 2.

Table 1 The four adaptation methods start from the same solver checkpoint and share the physical feasibility checks and solver-update rules. The Base row reports the initial solver’s performance before adaptation. The table below summarizes how the four methods generate and sample tasks.

MethodTask generationSamplingBiCoord SR (%)
RandomUnguided variantsUniform40.1
Failure-RandomFailure-guided variantsUniform40.9
Failure-WeightedFailure-guided variantsFailure-weighted41.6
ACSENew rules from solver-mixture failuresPayoff-weighted46.0

We account separately for solver-training rollouts, candidate-evaluation rollouts, and additional rollouts for payoff estimation. Table 4 in revised manuscript reports task performance, and the four-round cost estimates are reported separately.

Adaptation, checkpoint selection, and final evaluation. Task generation and solver adaptation use training data only. At the end of each round, we save the model parameters and Harness memory together. We apply the same validation-based checkpoint-selection rule to all four adaptation methods and evaluate the selected checkpoints on the same held-out test seeds. Model parameters and persistent memory remain fixed throughout testing. Execution history is retained within each episode and cleared before the next episode begins.

AuthorsOpenReview ↗

Response to Reviewer aRNR

Q2. Could you reconcile the reported summaries with the underlying records? Table 3 peaks in Round 3, the initial-solver TL entries in Table 2 average 463.56 rather than 491, and several Table 7 percentages are inconsistent with 30 trials per task. The printed evolved rates also average 86.8% rather than 86.6%.

R2. Thank you for checking these results carefully. We have corrected the per-task TL entries in Table 2, the description of peak performance in Table 3, and the real-world success rates in Table 7.

Table 2 — Task length. TL is the mean number of environment steps taken to complete a task, averaged over successful trials. It includes all action-execution steps across successive interactions within a trial. Lower values indicate shorter successful trajectories. The reviewer correctly noted that the 18 initial-solver TL values in the original table averaged 463.56. We traced the discrepancy to a transcription error for Handover Block With Bowls, whose TL was recorded as 840 steps in the evaluation log but printed as 336 in Table 2. Correcting this entry brings the mean across the 18 tasks to 491 steps.

Table 3 — Evolution rounds. Round 3 achieved the highest SR (46.0%) and SSR (61.1%). In Round 4, SR and SSR were 45.5% and 60.6%, respectively, while mean TL reached its lowest value of 401 steps. We have revised the text to distinguish peak success in Round 3 from the shortest successful trajectories in Round 4.

Table 7 — Real-world success rates. For the initial solver, 25/30 and 23/30 successes were incorrectly reported as 83.4% and 76.6%. Rounded to one decimal place, the correct rates are 83.3% and 76.7%. The evolved solver’s Number Sorting rate of 83.6% was a transcription error. It succeeded in 25/30 trials (83.3%). Across the two original tasks, the evolved solver succeeded in 52/60 trials (86.7%).

We have expanded the real world task to include Arithmetic and Logic, Chessboard Setup, and Letter Formation. The original two-task average is reported separately. The five-task average gives equal weight to each task and is rounded once, after averaging the unrounded rates. We have revised Table 6 and Figure 7 in the main manuscript accordingly.

Table 2. Success rates of π0.5\pi_{0.5} and the initial and evolved ACSE solvers on five real-world manipulation tasks. Each method was evaluated on 30 trials per task. Avg. denotes the unweighted mean across tasks.

MethodBlock Stacking (%)Number Sorting (%)Arithmetic and Logic (%)Chessboard Setup (%)Letter Formation (%)Avg. (%)
π0.5\pi_{0.5}(22/30) 73.3(18/30) 60.0(13/30) 43.3(21/30) 70.0(18/30) 60.061.3
ACSE (initial)(25/30) 83.3(23/30) 76.7(26/30) 86.7(25/30) 83.3(23/30) 76.781.3
ACSE (evolved)(27/30) 90.0(25/30) 83.3(28/30) 93.3(27/30) 90.0(26/30) 86.788.7

Comments on the paper

1 comment
AuthorsOpenReview ↗

Kind Request to Review Our Responses and Continue the Discussion

Dear Reviewers,

Thank you for your time and thoughtful feedback. We would appreciate it if you could take a look at our responses and the updates to the manuscript when you have a chance. We would be very happy to discuss any remaining concerns and provide further clarification. Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.