acceptodds
Under review as a conference paper at ICLR 2027

InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation

Abstract

We study test-time evolution for humanoid loco-manipulation: solving tasks that a controller was never trained for by repurposing its existing skills, improving from its own attempts, and retaining what it learns, without retraining. Our key insight is that a broad controller already holds much of the competence a new task needs, and that this competence becomes accessible through an interface between planning and control that is expressive enough to specify contact-rich, multi-stage interactions, yet executable and measurable enough that execution feedback can guide planning from experience. InterEvolve realizes this interface with two components. First, we develop an object-aware forward-backward (FB) behavioral foundation model, whose object residuals on a frozen body prior turn a new reward about the body or objects into loco-manipulation behavior at test time. Second, we specify tasks as reward programs: staged rewards with completion conditions and tunable constants. A large language model (LLM) agent revises the program structure in context, drawing on execution feedback and a skill library of verified programs, while a numerical optimizer tunes its constants. With every candidate verified across parallel simulation scenarios, the program explores new ways to induce, repurpose, and compose the controller's existing motor competence for the task at hand, and thus improves over iterations. Experiments show that human-designed rewards leave much of the FB model's loco-manipulation competence untapped, whereas the programs InterEvolve evolves release it, sometimes through novel strategies. It further produces behaviors for diverse tasks, complex scenes, and long-horizon compositions in simulation, and evolved skills run autonomously on a physical Unitree G1 from egocentric onboard perception.

Primary area:
applications to robotics, autonomy, planning
Keywords:
human-object interaction, humanoid loco-manipulation, test-time adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MgfcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies test-time evolution for new humanoid loco-manipulation tasks by repurposing existing skills without retraining the underlying controller. The core system, InterEvolve, represents a task as an editable, staged reward program: an LLM modifies the stage structure and reward terms based on the task, scene, and execution feedback, while CMA-ES tunes numerical constants. The resulting program is executed by a forward-backward behavioral foundation model augmented with an object-aware residual. Verified programs are stored in a skill library to support reuse and composition on subsequent tasks. Object-aware residual branches are trained for the actor and the forward and backward maps while the pretrained body branches remain frozen. Stage rewards weight the cached backward embeddings of a fixed state bank, and their projected sum supplies the latent prompt without per-task RL training. The experiments cover reference motion tracking, eight families of goal-conditioned tasks, composite long-horizon tasks, and deployment on a Unitree G1. The reported reward evolution increases the macro-average success rate from 34.6% for the fixed program to 86.5%, and raises reference-tracking success from 60% to 72%; the skill library achieves 8/10, 4/10, and 5/10 on three composition tasks. Overall, the paper presents a clear and appealing paradigm of “searching over task specifications while keeping the controller fixed.” However, the current evidence comes primarily from privileged-state simulation search, the real-robot experiments are small, and the treatment of search stochasticity and comparisons with general test-time planning methods remain insufficient.

Critical strengths
  1. The paper decomposes multi-stage contact tasks into rewards, completion conditions, and tunable constants, allowing LLMs and CMA-ES to handle structural editing and numerical calibration respectively. The program semantics, feature namespace, and code constraints enforced by the validator and documented in the appendix make the method relatively easy to inspect, and make it easier to express stages such as contact, transport, and release than directly generating joint trajectories.

  2. Table 1 shows that the body-only BFM-Zero performs substantially worse in terms of object error and success rate, whereas adding object states and link geometry gives the controller reusable object-interaction capabilities. Tables 2, 3, and 13 further show that the multi-stage structure and constant calibration jointly provide significant gains. The drift-correction, interaction-input, and residual-capacity ablations also indicate that the gains do not come solely from changing the network size.

  3. Beyond reference tracking, the paper evaluates carrying, tipping, kicking, and obstacle-navigation tasks, and reports a task-level verifier, earned tiers, search cost, and the reported search/evaluation scene splits. The program and lessons ablations for the skill library further suggest that the reuse gains come from adapting existing programs rather than simply copying them. The appendix provides search budgets, scene splits, feature interfaces, and the real-robot perception pipeline, making the overall reproducibility information fairly complete.

Critical weaknesses
  1. The staged design of the reward program, the object-aware residual, LLM-based structural editing, CMA-ES constant tuning, and skill-library reuse each have clear similarities to hierarchical reward shaping, residual control, program search, and skill-library methods. Section 2 relates the design to Language to Rewards and FB reward inference. The object-aware controller and staged program interface are meaningful differences, but stronger evidence against alternative reward-search procedures would clarify how much additional value their joint test-time evolution provides.

  2. The reward-inference bank is built from body-object states and surface geometry in pretraining replay, and the verifier directly reads simulator states. Within the reported search and confirmation grids, the object bearing varies by ±12° and the distance by ±0.15 m around nominal initial states. The real G1 experiments show only a few qualitative kicking and pushing sequences, without success rates across trials, perception dropouts or errors, different objects, or substantial scene changes. It is therefore still unclear whether program evolution in simulation is robust to real-world perception noise.

  3. Reference tracking is compared with ULTRA and BFM-Zero, but the goal-conditioned tasks are mainly compared with human/initial fixed programs using the same controller; there is no compute-matched comparison with non-LLM test-time planners, continuous reward search, or other task-specification optimization methods. A full search uses 6.7–14.2M environment steps per task family, averaging 9.5M, plus LLM calls. The budget-matched component ablations provide useful evidence for the design choices, but do not establish an advantage over alternative search procedures under matched total simulation and compute budgets.

Questions for authors
  1. Within the reported three-stage limit and feature vocabulary, how can failures due to limited expressiveness be distinguished from those caused by insufficient search?

  2. How sensitive are bank-based reward-to-prompt projection and the object-aware residuals to unseen geometries, pose errors, and poorly represented contact configurations?

  3. Do final test scenarios influence program selection? Please reconcile Section C.3's separate evaluation set with Section D.1's use of test scenarios for confirmation.

  4. How are incompatible contact assumptions resolved during skill-library composition, and what timing or trajectory deviations can the frozen controller tolerate during physical latent-prompt replay?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Mgfc (1/2)

We thank the reviewer for the detailed review, and for recognizing the inspectable program interface, the gains from object-aware control and staged optimization, and the breadth of the evaluation and reproducibility details. In short, evolution beats matched-budget search baselines (W3), programs transfer to the real G1 without tuning (W2), and test scenarios never influence selection (Q3).

W1: Why do the object-aware controller and the staged program interface matter beyond similar methods? They are what make test-time evolution for loco-manipulation possible. The comparison with alternative searches, requested here and in W3, is answered in W3.

W1.1: Our contribution. To our knowledge, InterEvolve is the first framework for test-time evolution of humanoid loco-manipulation. Some of its components resemble established methods, but their interface is new: staged reward programs run at test time rather than through per-candidate policy training as in Eureka, on a frozen object-aware controller rather than a body-only one such as BFM-Zero. Test-time execution also makes CMA-ES calibration affordable, since each constant setting costs only rollouts, whereas Eureka relies on the LLM to set the constants.

W1.2: Object-aware controller supplies the interaction competence that programs repurpose: the body-only BFM-Zero succeeds on 8% of tracking clips against 72% for ours (Table 1, p. 7, L325–337), and without object inputs the controller follows the body but rarely completes the task (Table 6, p. 17, L865–871).

W1.3: Staged program interface versus hierarchical reward shaping. Hierarchical reward shaping usually fixes phase rewards before training, whereas our stages, rewards, and completion conditions stay editable at test time. Staging is the largest single contributor to evolution: allowing only one stage drops success from 86.5% to 44.7% (Table 3, p. 8, L393–403).

W2: Is program evolution in simulation robust to real-world perception noise? We clarify that perception noise does not change what a reward program specifies: the program defines the task, not how it is sensed. For example, a carrying program rewards reducing the distance from the box to the marker. It describes what the box and the body should do, whether the box pose is observed exactly or with noise, so programs are developed and verified in simulation by design. The sim-to-real gap lies in execution: whether the frozen controller, given the program, tolerates real observation noise.

W2.1: Training for observation noise. The controller is trained with domain randomization over object properties, robot dynamics, actuation, control delay, and observation noise (Table G2 in the general response).

W2.2: New videos. The updated supplementary webpage shows real-world executions from shifted starts ("10 / Deploy") and recovery from disturbances ("Real G1 / Recovery").

W2.3: New quantitative evaluation. Our controller succeeded in 27 of 35 real-world trials across pushing, kicking, and carrying, including shifted starts and a disturbance during carrying (Table G1 in the general response).

W2.4: Do the ±12° and ±0.15 m grids limit the tested variation? They bound only the scenarios used during evolution. Execution is tested under larger variances, e.g., real-world trials add shifted starts of up to 0.5 m ("10 / Deploy").

W3: How does evolution compare with search baselines under matched budgets? Evolution with LLM feedback is best or tied on all three task families tested (Table R2).

W3.1: Which baselines, and under what controls? All four methods share the frozen controller, task definitions, verifier, and scenarios, and each task gives every method the same simulation budget: 9.7 M steps for pushing, 7.6 M for kicking, and 9.6 M for lifting. The alternatives cover the reviewer's request:

  • Search only hyperparameter: keeps the agent's initial program and spends the whole budget on CMA-ES over its constants.
  • Search reward without LLM feedback: has the LLM iterate programs from the task and scene description alone, without any execution feedback, and tunes each one with CMA-ES.
  • Search latent: search stage-conditioned latent prompts using CEM directly with the cross-entropy method, without reward programs.

W3.2: What does the comparison show? Execution feedback matters most: Search reward without LLM feedback never solves kicking, which requires dedicated body use, although it matches evolution on pushing and comes close on lifting.

Table R2. Success rate (%) under matched simulation budgets (revised manuscript: Table 19, p. 32, L1721–1727)

Task familyEvolution (ours)Search only hyperparameterSearch reward without LLM feedbackSearch latent (CEM)
Push100.089.1100.042.2
Kick71.17.80.017.1
Lift to support85.429.782.861.5
AuthorsOpenReview ↗

Response to Reviewer Mgfc (2/2)

Q1: Do the remaining failures come from limited expressiveness or from insufficient evolution? Success improves during the initial evolution rounds and remains unchanged over three additional rounds (Table R3), supporting convergence within the original budget on these composite tasks.

Table R3. Success rate (%) on the composite tasks across evolution rounds (revised manuscript: Table 21, p. 33, L1763–1768)

Composite taskRound 1Round 2Round 3Round 4 (new)Round 5 (new)Round 6 (new)
Relocate between supports31.343.871.971.971.971.9
Stack two boxes4.728.145.345.345.345.3
Carry, place, and kick50.084.490.690.690.690.6

Q2: How sensitive are projection and the object-aware residuals to unseen geometry, pose errors, and contact coverage? Pose errors are covered by training-time randomization (Table G2 in the general response), a controller trained on one box achieves higher tracking success than ULTRA on unseen suitcase and plastic-box geometries (Table R4), and projection works when the bank holds states relevant to the task.

Q2.1: Unseen geometry? We use suitcase and plastic-box interaction sequences from OMOMO, retargeted to the Unitree G1 using the pipeline described in Sec 4.1 (p. 6, L309–319), together with their corresponding object meshes. These object categories are unseen during training of the large-box-only controller. Without retraining, it achieves 32% and 30% tracking success on 50 reference motions per category, respectively, compared with 30% and 22% for tracking baselines (ULTRA).

Table R4. Reference tracking on OMOMO suitcase and plastic-box objects, unseen by the large-box-only controller and ULTRA baseline controllers (revised manuscript: Table 22, p. 34, L1783–1790)

ObjectControllerSR (%)E_h (cm)E_o (cm)
SuitcaseOurs (Large-box only)32.027.2642.51
SuitcaseULTRA30.027.8639.74
Plastic boxOurs (Large-box only)30.032.7553.21
Plastic boxULTRA22.032.4549.09

Q2.2: Poorly represented contacts? Projection handles them when the reward focuses on the few relevant bank states. On the feet-only task, whose foot contacts are rare in the replay data, uniform weighting gives 0% success, a moderate focus gives 59.4%, and banks of 12.5k to 100k states give 53.1–65.6%, with drops only at the extremes of bank size or focus (Table 9 in Sec. B.3, pp. 19–20, L1010–1042).

Q3: Do the final test scenarios influence program selection? No. Selection uses only the search and confirmation scenarios, and the final test set is used once, after the program is fixed (Sec. C.3, p. 23, L1199–1206).

Q3.1: Which scenario sets are used? Three disjoint sets:

  • Search grid, 16 scenarios: the agent proposes candidates, and CMA-ES tunes their constants.
  • Confirmation grid, 64 scenarios with 3 repeats: a candidate replaces the current best only if it wins by more than the evaluation noise, so lucky edits are rejected.
  • Final test set, 64 scenarios: evaluates the fixed program; evolution never sees these scenarios.

Q3.2: Sec. D.1 (p. 26, L1382–1383) has a typo. Thanks for catching this. Its phrase "confirmation re-evaluates candidates on the test scenarios" is a typo for "on the confirmation scenarios", which the revision corrects. Scenario seeds in our logs confirm that the confirmation and final test sets share no scenario.

Q4.1: How are incompatible contact assumptions resolved during composition? A reward only encourages a behavior within its own stage; it is not a hard constraint and not applied in all stages. When carry, place, and kick switches from the carrying reward, which favors hand contact, to a kicking reward such as no_hand · (push + θ0 · strike) from Fig. 2(a) (p. 4, L178–183), the kicking reward is zero at the switching point. The controller is then prompted toward the states this reward favors, so the robot releases the box and kicks it. Execution feedback tells the agent when a contact term needs revision.

Q4.2: What timing and trajectory deviations can execution tolerate? We now run both latent-prompt replay and closed-loop stage transitions on the real G1 (Table G1 in the general response). Replay succeeds in 11 of 15 trials with start shifts of up to 0.5 m and in 4 of 5 with a disturbance during carrying, as the controller is trained with deviations. Closed-loop stage transitions also re-time each stage switch to the observed state and recover from failures by returning to an earlier stage. Please see “10 / Deploy" and "Real G1 / Recovery" on the supplementary webpage.

We hope the physical trials, the budget-matched comparisons, and the protocol clarification address the reviewer's main concerns and we would be glad to answer further questions.

Reviewer_1W5KOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework for adapting a frozen base humanoid controller to new loco-manipulation tasks through test-time reward-program optimization. The method starts from the base BFM-Zero controller and trains residue forward, backward, and policy networks on human-object interaction data. For new task adaptation, a LLM generates and iteratively revises multi-stage reward programs, and CMA-ES is used to optimize the weights and state transition parameters. The authors evaluate the method on reference motion tracking and 8 goal conditioned locomanip tasks. InterEvolve achieves 86.5% average success compared with 34.6% for tuned initial LLM-generated rewards, requiring on average 2.1 GPU hours per task family.

Critical strengths
  1. Interesting approach to task adaptation

    Combining LLMs and reward optimization for online humanoid policy adaptation for loco-manipulation is quite interesting. This approach avoids retraining the network but does require some rollouts for tuning the parameters.

  2. Strong empirical results and experiment design

    The proposed methods show quite high SR (86.5%) on the tasks, and the ablations show each component’s effect on the overall success rate.

Critical weaknesses
  1. Limited physical validation of the complete framework

    The real-world demonstrations replay latent prompts generated in simulation rather than computing reward-conditioned latents and stage transitions from live observations. Thus, these demonstrate transfer of the motor policy under scheduled latent conditioning, but not deployment of the complete reward-program execution mechanism. Quantitative physical success rates are also missing.

Questions for authors
  1. How does this approach compare to policy fine-tuning? I understand the motivation of this work is not to retrain/finetune a policy, but I am curious about how difficult or how much of a compute is needed to adapt say a pre-trained BFM-Zero to a new task. For the proposed method, training the residues + rollout for the tuning still requires some GPU hours and simulator rollouts.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1W5K

We appreciate the reviewer for finding the approach that avoids retraining the network interesting and the results and ablations strong.

W1: Does the framework run on the real robot, with quantitative success rates? Yes. Per reviewer’s request, we now report repeated real-world trials for two execution modes, latent-prompt replay and the new closed-loop stage transitions, which evaluate the program's stage conditions on live observations. Both modes tolerate shifted starts and carrying disturbances. Please refer to the general response.

W1.1: What runs on the robot? In latent-prompt replay, the robot replays latent prompts computed in a simulated rollout while the controller closes the loop on current proprioception and the estimated box pose at every step (Sec. D.8, p. 30, L1604–1607). 

W1.2: Can stage transitions come from live observations? Yes. With the closed-loop stage transitions, the robot evaluates the program's stage-completion conditions on the observed state and switches stages accordingly, which also lets it recover from failures by returning to an earlier stage. Please refer to the general response and videos under "Real G1 / Recovery" on the updated supplementary webpage.

W1.3: What are the physical success rates? Success rates range from 60–100% across the tested task–condition pairs, with results for each execution mode reported separately in Table G1 in the general response. The table reports 5 trials per mode and condition for pushing, kicking, and carrying under nominal and shifted starts, plus a disturbance during carrying. The revision states the execution protocol and reports these trials in Sec. 4.3 (p. 9, L459–463).

Q1: How much compute does adaptation need, compared with policy fine-tuning? A new task costs one evolution run of about 2.1 GPU-hours with no weight updates, after a one-time cost of about 20 GPU-hours for controller fine-tuning that builds the object-aware controller. Unlike per-task fine-tuning, every task then shares one controller.

Q1.1: What is the one-time cost for fine-tuning? About 20 hours on one A100 and about 200 M environment steps. We add object-state inputs to the pretrained BFM-Zero and train object-aware residuals for the forward map, the backward map, and the actor, while the pretrained body branches stay frozen. The resulting controller serves every task in the paper without further weight updates, and the revision reports this cost in Sec. A.2 (p. 16, L830–831).

Q1.2: What does each new task cost? A full evolution run costs 2.1 GPU-hours, 6.7 to 14.2 M environment steps with an average of 9.5 M, and 0.23 M LLM tokens, and it updates no weights (which was originally reported on Table 2, p. 7, L325–337, Sec. C.4, p. 23, L1215–1229).

We hope these responses address the reviewer's concerns, and we would be glad to answer further questions.

Dear Reviewer 1W5K, thank you again for your review. As the discussion period ends soon, we would like to check whether our responses have addressed your concerns. We would be glad to answer any further questions.

Thank you for your response. It has addressed most of my questions. I will maintain my score.

Reviewer_L4eeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes InterEvolve, a framework for evolving a pretrained reusable humanoid controller to new loco-manipulation tasks through test-time search over reward programs. For this, the authors propose an object-aware forward-backward behavioral foundation model that translates rewards into latent control prompts without task-specific policy training. In particular, an LLM proposes and fine-tunes reward programs using execution feedback of a verifier, while CMA-ES tunes their numerical parameters. Over time, successful tasks are stored as library to supports subsequent tasks by being used as input for the reward shaping LLM agent. The experimental evaluation covers reference tracking, eight goal-conditioned task families, component ablations, search-budget scaling, skill-library reuse, long-horizon composition, and physical deployment on a Unitree G1. Together, these experiments demonstrate that reward-program evolution can improve the behaviors accessible from a frozen controller and add new skills after only a few iterations.

Critical strengths

A major strength of the paper is that it addresses the problem of adapting a pretrained controller to new tasks without retraining it for each objective, which is an important capability of generalist robots. The extension of BFM-Zero to an object-aware forward-backward model and integration with multi-stage reward programs and execution-guided refinement is a good idea that is implemented well. In particular, including object information allows the controller to respond to objectives involving both the robot and its interactions with the environment, making it more generalizable to new tasks involving the object. The multi-stage rewards then provide a way to organize these objectives into sequences, such as approaching an object, lifting it, and placing it on a support. As contact-rich tasks often require different objectives at different points during execution, and a single fixed reward can be difficult to formulate, the proposed system allows these choices to be refined using feedback from the robot’s attempts, while numerical optimization adjusts the corresponding parameters.

Another strength of the paper is the experimental evaluation itself. It shows transfer across multiple manipulation tasks. In particular, tasks such as carrying at chest height and tipping improve from no successful executions to high success rates. This indicates that task formulation can be a major bottleneck in accessing pretrained motor control. The ablation experiments also help identify important parts of the approach. Restricting programs to a single reward stage reduces success, as does removing parameter tuning. These results support the idea that both dividing a task into appropriate phases and adjusting the rewards within those phases are important for successful execution.

Critical weaknesses

A weakness of the paper is that the hardware experiments do not show an execution of the complete reward mechanism on the real hardware. Appendix D.8 explains that the robot replays latent prompts generated during a simulated rollout, while the controller continues to respond to proprioception and estimated object pose. This is a useful demonstration of transfer, but it leaves open how reliably the reward programs and their transition conditions would work when evaluated live during physical interaction. Consequently, it raises the question of how strongly successful execution depends on a carefully calibrated simulation. Differences in object properties, friction, or contact timing could cause the physical interaction to progress differently from the simulated rollout that produced the latent sequence. This distinction should be stated more clearly in the main text and not only in the appendix. Reporting success rates over repeated trials and testing changes in initial conditions or execution timing would make the hardware evaluation even more convincing.

Another weakness is that the paper does not address tasks that require entirely new motor skills beyond those acquired during pretraining. The library experiments mainly evaluate new combinations of familiar interactions, and performance drops when relevant skills are missing from memory. This implies that although the system can search for reward programs without a matching library entry, it remains limited by the capabilities of the frozen controller and the states available in its reward-inference bank. This limitation is acknowledged by the authors, but deserves more discussion. In particular, the evaluation does not systematically examine tasks far outside the available experience, whether irrelevant library entries interfere with search, or whether the system can recognize that a task exceeds its capabilities.

Questions for authors

How are task progression and stage transitions handled during hardware execution? Appendix D.8 describes replaying latent prompts from a simulated rollout. Can execution adjust when physical contact or object movement occurs earlier or later than in simulation? Please also clarify what simulation calibration or task-specific tuning was required and report the number of hardware attempts and their outcomes.

What distinguishes the target tasks in the experiments from the experience available during pretraining and in the skill library? The library experiments mainly examine combinations of familiar interaction modes. Please clarify which evaluated tasks require (how many) substantially different interactions and which primarily recombine existing skills.

The paper reports averages over three independent evaluation runs, but Appendix D.1 states that each variant is searched only once per task family. How sensitive are the results to repeating the complete search? Results from independent searches would help assess whether the improvements are consistent across different generated programs and optimization runs.

The paper I-BFM (https://arxiv.org/pdf/2610.06129v1) got published on arXiv after the submission deadline of this conference. It would be useful to acknowledge it in the related work, given its similar focus on object-aware forward-backward representations.

4 comments
AuthorsOpenReview ↗

Response to Reviewer L4ee (1/2)

We thank the reviewer for the constructive review, and for recognizing task adaptation without retraining, the object-aware FB controller with staged programs and execution-guided refinement, and the evidence that stage structure and calibration both matter.

W1: How do reward programs and their stage transitions run on the real robot, and how robust are they? Stage transitions can run live from observations, and repeated physical trials show that both execution modes tolerate shifted starts and carrying disturbances, with performance varying across tasks and conditions (please refer to the general response).

W1.1: How are stage transitions executed on the robot? With closed-loop stage transitions, they follow the observed state: the robot evaluates each stage's completion condition on the live robot and box states and switches when it holds. In latent-prompt replay, they follow the timing of the simulated rollout, while the controller's feedback on proprioception and the estimated box pose absorbs earlier or later contact (Sec. D.8, p. 30, L1604–1607). The revision describes both modes and reports the trials in the main text (Sec. 4.3, p. 9, L459–463).

W1.2: Was simulation calibration or task-specific tuning required? No. The physical trials use the reward programs and controller parameters selected in simulation, without additional calibration or task-specific tuning. The controller is instead trained with the domain randomization in Table G2 in the general response, and the new trials measure the box pose with motion capture.

W1.3: How reliable is execution under changed starts and disturbances? Both modes succeed in at least 3 of 5 trials in every task and condition. Table G1 in the general response reports 5 trials per mode and conditions for pushing, kicking, and carrying under nominal and shifted starts, plus a disturbance during carrying which changes both initial conditions and execution timing.

W2: Can InterEvolve go beyond the pretraining motions and the stored skills? Yes, because the controller holds competence rather than a set of motions. Pretrained on diverse whole-body interaction, the FB controller learns how to reach body and box states and turns a new reward over these states into a policy that pursues it. A reward program thus specifies outcomes, not motions, so evolution can induce, repurpose, and compose this competence rather than only recombine stored skills. For example, the controller has no demonstration of tipping a box onto another face, yet evolution reaches 95.3% on tipping where calibrated written programs reach 0% (Table 13, p. 27, L1444–1456). 

W2.1: Do evolved behaviors differ from the pretraining motions? Yes, for four of the eight task families. In Fig. 10 (p. 30, L1587–1590) (Sec. D.7, pp. 29–30, L1549–1590), rollouts of lifting onto a support, carrying at chest height, kicking, and tipping form clusters separate from the 729 OMOMO large-box training clips, whereas pushing to a mark and pushing around an obstacle stay close to them. 

W2.2: What do the composite tasks show? That atomic skills compose: the library stores atomic skills, and their combinations cover a large compositional space of long-horizon tasks. None of the three composite tasks, or any program for them, is stored (Sec. D.6, p. 28, L1499–1504), yet evolution builds each one from stored reward components. The revision states this role of composition in Sec. 4.4 (p. 9, L475–477).

  • “Relocate between supports” combines the lift-to-table and carry rewards, then adds a turn toward the rear support, release, and settling.
  • “Stack two boxes” combines lift and carry rewards to clear the lower box, then rewards matching the support height, releasing, and damping motion.
  • “Carry, place, and kick” chains the ground-carry reward with the kick reward, then settles near the mark.

W2.3: Can evolution handle a task with no matching library program? Yes. Following the reviewer's suggestion, we add a new task, throwing a box to a marker (Fig. 12, p. 33, L1753), which no library program covers. After three rounds, the selected program reaches 48.4% success from scratch and 54.7% with the library on the final test scenarios. The updated Fig. 10 (p. 30, L1587–1590) (Sec. D.7, pp. 29–30, L1549–1590) includes the throwing behavior, which extends beyond the main concentration of training clips and library existing skills. We would welcome a specific interaction mode that the reviewer considers a useful stress test.

AuthorsOpenReview ↗

Response to Reviewer L4ee (2/2)

W2.4: Do irrelevant library entries interfere with evolution? No, since the irrelevant library will reach a low score compared to other candidates and not be selected during the evolution. No library variant in Tables 5 (p. 9, L447–457) and 15 (p. 29, L1532–1545) does worse than an empty library, including libraries that lack the skill the task needs, and on throwing, where no stored program matches, the library still raises success from 48.4% to 54.7%. For example, a library with only the three carrying programs, which contain no kick, reaches 23.4% on carry, place, and kick, against 12.5% with an empty library (Table 15, p. 29, L1532–1545).

W2.5: Can the system tell when a task exceeds its capabilities? Not in advance, as the revised Sec. E (p. 34, L1794–1801) states, but after evolution: the verifier's per-criterion results show which criteria a task still fails, which helps identify tasks beyond the current controller.

Q1: How are stage transitions handled on hardware, and can execution adjust to earlier or later contact? Yes, with closed-loop stage transitions: each stage switch is triggered by the observed state, so earlier or later contact shifts the switch (W1.1). In latent-prompt replay, stage transitions follow the simulated timing, and the controller's own robustness absorbs the deviation. Please refer to videos under "Real G1 / Recovery" on the updated supplementary webpage.

Q2: Which tasks need substantially different interactions, and which recombine skills? Four of the eight families produce motions that differ clearly from the training clips (W2.1), the three composite tasks recombine stored atomic skills (W2.2), and the new throwing task has no matching library program (W2.3).

Q3: How sensitive are the results to repeating the whole evolution? The results are stable: per reviewer’s request, three independent evolution runs per family average 91.2% success over the eight families, consistent with the single run in the paper (86.5%) and far above the best calibrated fixed program (34.6%, Table 2, p. 7, L325–337).

Q3.1: What did we repeat? The whole evolution, from scratch, three times per family: program generation, feedback-driven revision, and CMA-ES calibration, with each run scored on the 64 test scenarios. Table R1 therefore measures the variability of the whole evolution rather than repeated evaluation of one program.

Q3.2: How large is the spread across runs? At most 11.8 points, on pushing through a gate, and three families reach 100% in every run (Table R1).

Table R1. Success rate (%) on the 64 test scenarios. New runs give the mean ± standard deviation over three complete evolution runs (revised manuscript: Table 18, p. 32, L1696–1707).

Task familyPaper, one runThree new runs
Push to mark100.0100.0 ± 0.0
Carry at mid height100.0100.0 ± 0.0
Carry at chest height100.0100.0 ± 0.0
Tip onto a new face95.393.2 ± 6.2
Push through a gate73.487.5 ± 11.8
Push around an obstacle71.992.2 ± 6.6
Kicking to mark59.471.1 ± 3.7
Lift to support92.285.4 ± 10.2
Average86.591.2

NOTE: The DeepSeek API now serves DeepSeek-V4.1-Flash-1017, whereas the paper used DeepSeek-V4-Flash. The shift in means relative to the paper may partly reflect this update, while the spread across runs reflects the randomness of evolution.

Q4: How does InterEvolve relate to the concurrent I-BFM? The two are complementary, and the revised Sec. 2 (p. 3, L134–135) cites I-BFM (Han et al., arXiv:2610.06129) as concurrent work, posted after the ICLR submission deadline. Both build object-aware forward-backward controllers. I-BFM learns a robust reward-conditioned controller for humanoid-object interaction and executes a given task reward, whereas InterEvolve keeps such a controller frozen and evolves the staged reward programs it executes, reusing verified programs on later tasks. An I-BFM controller could therefore serve as the frozen controller in InterEvolve.

We hope these responses address the reviewer's concerns, and we would be glad to answer further questions.

Dear Reviewer L4ee, thank you again for your review. As the discussion period ends soon, we would like to check whether our responses have addressed your concerns. We would be glad to answer any further questions.

Thank you very much for your response. Although it addressed most of my questions, I agree with Reviewer Mgfc that many questions remain about the approach’s ability to generalize to settings without access to perfect information, such as that provided by motion capture or simulation. Therefore, I will maintain my rating.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/2): what runs on the robot

We thank all reviewers for their careful and constructive reviews. The reviewers found adapting a frozen controller with agentic reward programs important and appealing (L4ee, 1W5K, Mgfc), the staged program interface easy to inspect (Mgfc), and the ablations informative (all). We answer the remaining points under each review. All page and line references below refer to the revised paper, with modified parts highlighted in blue.

All three reviewers asked how reward programs run on the physical robot. We added repeated real-world experiments comparing latent-prompt replay and closed-loop stage transitions under nominal starts, shifted starts, and disturbances during carrying. Table G1 reports their success rates separately for each task and condition. 

What runs on the robot? 

  • Latent-prompt replay: the G1 replays the latent prompts of a simulated rollout, one per control step, while the frozen controller closes the loop on current proprioception and the estimated box pose (Sec. D.8, p. 30, L1604–1607). Programs are evolved and verified only in simulation. As L4ee suggested, the revision now states this in the main text (Sec. 4.3, p. 9, L459–463).
  • Closed-loop stage transitions run the program's stage logic live: the robot evaluates each stage's completion condition on the observed robot and box states and moves to the next stage when it holds (Sec. D.9, p. 30, L1613–1616). Because the stage follows the observed state, this mode also recovers from failures, for example by returning to an earlier stage and regrasping when the box is disturbed during carrying (videos under "Real G1 / Recovery" on the updated supplementary webpage).

How robust is real-world execution? We find both modes are robust and can recover from failures: each succeeds in 27 of 35 trials, with at least 3 of 5 successes in every task and condition (Table G1). We ran 5 trials per mode and condition for pushing, kicking, and carrying a box to a marked target. Shifted starts move the box away from the nominal start (offsets under "10 / Deploy" on the supplementary webpage), and the disturbance test pushes the box during carrying. All trials use the programs selected in simulation without further tuning, and the box pose is measured with motion capture.

Table G1. Successes out of 5 real-world trials on the Unitree G1 (revised manuscript: Table 16, p. 31, L1626–1635).

TaskStartLatent-prompt replayClosed-loop stage transitions (new)
PushNominal3/54/5
PushShifted3/53/5
KickNominal4/55/5
KickShifted3/53/5
CarryNominal5/55/5
CarryShifted5/53/5
CarryDisturbed during carrying4/54/5

Why are both modes robust? Each latent prompt encodes the goal of its stage, not a motion to imitate, and the frozen controller pursues that goal from whatever state the robot is in. A prompt therefore covers neighboring sub-behaviors: for example, if the contact is delayed to lifting prompt, the lifting prompt also brings the robot to the box when contact comes late, because the lift reward is a product with hand contact (Sec. C.3, p. 23, L1207–1213) and so favors states with the box in hand. This is why replay tolerates shifted starts and disturbances even though it follows simulated latent trajectories. Closed-loop stage transitions add this robustness explicitly: they switch on the observed state and can return to an earlier stage.

Which perception does each real-world result use? The evolved programs run on the physical G1 with both perception sources, and the revision now states which one each result uses (abstract, Sec. 1, and Sec. 4.3). The repeated trials above use motion-capture box poses, and the kicking and pushing runs in Fig. 4 (p. 9, L441–445) execute autonomously from the robot's egocentric camera. Because a reward program specifies the task rather than how the box is sensed, a program evolved in simulation runs unchanged with either source, so advances in onboard pose estimation carry over without re-evolving programs. With egocentric poses in MuJoCo, closed-loop stage transitions succeed in 10 of 15 trials against 6 of 15 for replay (Table R5 in the follow-up reply to Mgfc).

AuthorsOpenReview ↗

General response (2/2): training randomization, new experiments, and revisions

How is the controller randomized during training? The controller is trained with the domain randomization in Table G2, so that programs selected in simulation tolerate differences in object properties, robot dynamics, actuation, and sensing. U(a, b) denotes a uniform distribution, and scale factors multiply nominal values. Object-pose biases are sampled at reset and held for the episode, while observation noise is resampled at every step and applied before observation scaling. Object and actuator randomization ramp up to these ranges during training. The revision includes Table G2 in Sec D.9 (p. 31, L1638–1661).

Table G2. Domain randomization during controller training (revised manuscript: Table 17, p. 31, L1644–1661).

ParameterDistribution
Object mass scaleU(5/7, 10/7)
Object static and dynamic frictionU(0.6, 1.2), one shared sample
Foot frictionU(0.4, 1.25)
Robot link mass scaleU(0.95, 1.05)
Torso CoM offset per axis (m)U(-0.02, 0.02)
PD gain scales Kp and KdU(0.75, 1.25)
Joint position offset (rad)U(-0.02, 0.02)
Additive torque disturbanceU(-0.1, 0.1) × max torque
Control delay (control steps)uniform over {0, 1}
Object position bias per axis (m)U(-0.02, 0.02)
Object orientation bias per Euler axis (deg)U(-2, 2)
Joint position noise (rad)U(-0.01, 0.01)
Joint velocity noise (rad/s)U(-0.5, 0.5)
Base angular velocity noise (rad/s)U(-0.2, 0.2)
Projected gravity noiseU(-0.05, 0.05)

What other experiments are added? Five, each answered in full under the review that asked for it.

  • Repeatability of evolution (L4ee, Q3). Three independent evolution runs per task family average 91.2% success, consistent with the single run in the paper (86.5%) and far above the best calibrated fixed program (34.6%) (Sec. D.10, p. 32, L1686–1707).
  • Search baselines (Mgfc, W3). Under matched simulation budgets, evolution with LLM feedback is best or tied against CMA-ES on a fixed program, CMA-ES on programs generated without feedback, and CEM over latent prompts (Sec. D.11, p. 32, L1710–1727).
  • A new throwing-to-mark task (L4ee, W2.3). Evolution reaches 54.7% success on throwing a box to a marker, a task no library program covers, and 48.4% without the library (Fig. 12, p. 33, L1753).
  • Evolution budget (Mgfc, Q1.1). Three extra evolution rounds leave success unchanged on all three composite tasks, so evolution converges within the reported budget (Sec. D.13, p. 33, L1756–1768).
  • Unseen object geometry (Mgfc, Q2.2). Without retraining, a controller trained only on the large box tracks 32% of suitcase clips and 30% of plastic-box clips, compared with 30% and 22% for tracking baselines (ULTRA), respectively (Sec. D.14, pp. 33–34, L1771–1790).

What changes in the revision? The revised PDF includes every change below, highlighted in blue.

  • ****Sec. 4.3 (p. 9, L459–463): the two execution modes and Table G1 (L4ee W1, 1W5K W1, Mgfc W2).
  • Sec. 4.4 (p. 9, L475–477): the composite tasks as compositions of atomic skills (L4ee W2.2).
  • Sec. 2 (p. 3, L134–135): the concurrent I-BFM (L4ee Q4).
  • Sec. D.1 (p. 26, L1382–1383): the updated description of the confirmation scenarios (Mgfc Q3).
  • Sec. D.7 (pp. 29–30, L1549–1590) and Fig. 12 (p. 33, L1753): the throwing task, added to the UMAP in Fig. 10 (p. 30, L1587–1590) and shown in the new Fig. 12 (p. 33, L1753) (L4ee W2.3).
  • Secs. A.2, D.9–D.11, and D.13–D.14: the one-time controller training cost (A.2, p. 16, L830–831), training-time domain randomization (D.9, p. 31, L1638–1661), independent evolution runs (D.10, p. 32, L1686–1707), budget-matched search baselines (D.11, p. 32, L1710–1727), convergence across evolution rounds (D.13, p. 33, L1756–1768), and tracking on unseen object categories (D.14, pp. 33–34, L1771–1790) (1W5K Q1, L4ee Q3; Mgfc W3, Q1, Q2).
  • Sec. E (p. 34, L1794–1801): that the system cannot tell in advance whether a task exceeds the frozen controller, and that search and the verifier use privileged simulator state (L4ee W2.5, Mgfc W2.3).
  • Supplementary webpage: more real-world videos ("10 / Deploy", "Real G1 / Recovery").
AuthorsOpenReview ↗

Follow-up response to Reviewer Mgfc (1/2)

We thank the reviewer for the follow-up and are glad that the matched-budget comparisons, repeated evolution runs, and physical trials strengthen the paper. We address each remaining question below with new results, which we have also added to the revised PDF, highlighted in blue.

F1: Is the benefit of execution feedback limited to kicking? No. We thank for the comment and we extend the evaluation to all eight task families, evolution averages 91.2% against at most 31.8% for any alternative under matched budgets (Table R2, extended). Program search without execution feedback is competitive only on pushing and lifting; on the other six families it reaches at most 43.8%, including 0.0% on carrying at tipping, a task unseen from training, where evolution reaches 93.2%.

Table R2 (extended). Success rate (%) under matched simulation budgets. Every method receives the realized search-plus-confirmation steps of the corresponding evolution run (7.6–11.3 M steps) and is scored on the same 64 final-test scenarios.

Task familyEvolution (ours)Search only hyperparameterSearch reward without LLM feedbackSearch latent (CEM)
Push to mark100.089.1100.042.2
Carry at mid height (new)100.00.028.10.0
Carry at chest height (new)100.00.00.00.0
Tip onto a new face (new)93.20.00.04.7
Push through a gate (new)87.50.043.855.9
Push around an obstacle (new)92.20.00.04.7
Kicking to mark71.17.80.017.1
Lift to support85.429.782.861.5
Average​​91.215.831.823.3

F2: Remain robust with onboard perception instead of motion capture? 

  • Evolution does not depend on the source of perception. A reward program defines the task, not how the object is sensed, so a program evolved in simulation runs unchanged with either pose source. The paper's scope is test-time evolution of the task specification, which we demonstrate across diverse task families (F1), unseen objects (F3), and long-horizon composite tasks (Table 5, p. 9, L447–456). 
  • Noise enters only at execution, where the frozen controller closes the loop on the estimated box pose at every step and is trained with observation noise and object-pose biases (Table G2 in the general response). Closed-loop stage transitions add robustness to object lost, since they switch stages on the observed state and can return to an earlier stage: with egocentric camera-estimated poses in MuJoCo, they succeed in 10 of 15 trials against 6 of 15 for replay (Table R5).
  • Onboard execution already succeeds on kicking and pushing. Extending this success to more tasks depends more on the off-the-shelf pose estimator. This is because: the stock G1 camera has a small field of view, and FoundationPose loses the box when the motion is fast, object leaves the view, or when the arms occlude it during manipulation (We include more evidence in Sec. E). Improving pose estimation lies outside the scope of this work.
  • The robustness this paper centers on is how the policy handles unseen states, such as the shifted starts and disturbances, and unseen tasks and objects (F1, F3). The new quantitative trials therefore use motion capture, which factors out pose-estimation error and so isolates how evolved programs transfer to the real robot. With these poses, both execution modes succeed with disturbances and out of distributional states (supplementary videos). To our knowledge, existing real-world FB controllers for loco-manipulation also rely on motion capture, including the most recent concurrent work, I-BFM, on the same G1 (Han et al., arXiv:2610.06129). Using motion capture is thus common practice and does not diminish our contribution.

Table R5. Successes out of 5 with box poses estimated from the head camera.

TaskLatent-prompt replayClosed-loop stage transitions
Push to mark3/54/5
Kick to mark2/52/5
Carry to mark1/54/5

Broader egocentric evaluation, together with pose estimation that is more robust to arm occlusion and a small camera field of view, is an interesting direction for future research beyond the scope of this paper. Its core contribution, test-time evolution, is validated in simulation (F1, F3), and the evolved programs run on the physical G1 (Please refer to our updated supplementary webpage). We revised the abstract, introduction, Sec. 4.3, and Sec. E to reflect this scope. This separates the paper's contribution, test-time evolution, from the perception module and MoCap, and we summarize the findings for each.

AuthorsOpenReview ↗

Follow-up response to Reviewer Mgfc (2/2)

F3: Is success on unseen object categories modest?

  • For reference tracking, our absolute success, though modest, notably exceeds the dedicated tracker ULTRA (Table R4); The modest success rate is mainly because we directly call the controller without test-time evolution during the evaluation.
  • With test-time evolution, we test the same controller averages 78.5% on the plastic box, 88.7% on the small box, and 84.6% on the suitcase across all task families.

F4: Do the repeated runs depend on the updated LLM backbone? No, our gain is not tied to one LLM version. Every new comparison is model-matched.. We did not switch models by choice: The DeepSeek API now serves DeepSeek-V4.1-Flash-1017 in place of DeepSeek-V4-Flash after the submission, so every new comparison uses the updated model for evolution and the LLM-based baselines, and the spread across the three runs measures search randomness under one model. 

The updated model is available to every LLM-based method, yet the margin of evolution over the strongest baseline is significant, growing from 51.9 points with the original model (86.5% against 34.6%, Table 2, p. 7, L325–337) to 59.4 points with the update (91.2% against 31.8%, Table R2).

 

Together, these results show that the advantage of evolution is significant to more task families, and that evolution generalizes to unseen objects. We hope they address the remaining concerns, would be grateful if the reviewer would take them into account in the final assessment, and are happy to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.