acceptodds
Withdrawn from ICLR 2027

From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models achieve strong performance across embodied manipulation tasks by unifying visual understanding, language conditioning, and action generation. However, long-horizon tasks can involve non-Markovian dependencies: successful decisions may require retaining task progress and past interaction outcomes, while partial observability and execution errors compound over time. Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution. We organize the high-level VLM as a robotic agent through an editable harness to maintain task context and improve planning through execution experience. We instantiate this approach as SERA (Self-Evolving Robotic Agent), with a fixed VLA executor providing low-level control. The harness maintains and updates task context using observations and execution feedback to guide subsequent planning. Self-evolution uses environment rollouts to propose and evaluate harness revisions, with the selected configuration fixed before held-out testing. On RoboMME, SERA achieves 51.38% overall success with autonomous selection, compared with 44.38% without self-evolution. Our exploration suggests that agentic planning is a promising direction for history-dependent robotic manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Long-Horizon Robotic Manipulation, Agentic Planning, Agent Memory, Self-Evolving Agents

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_ACALOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

strengths

  • The paper shows a considerable improvement on reference tasks and a good improvement on permeance tasks in the Robo MME benchmark.

weaknesses

  • This work presents a single data point: One specific VLM and one specific VLA where this specific harness design leads to an improvement on one specific benchmark. There are no generalization tests over multiple VLMs, multiple VLAs, and there is no valiation of these improvements on additional benchmarks or real-world trials. There are also no ablation studies which would provide useful insight on the design decisions of the harness.
  • The above weakness unfortunately leads me to the conclusion that there is no scientific insight to draw from this publication in its current form. It is an interesting reference implementation, and the improvement on the reference-class tasks is motivates to look more deeply into this agentic memory system, but for a new VLM and a new VLA one can only take this proposal as reference and either hope it transfers or figure out all the design decisions again.
  • I cannot find any info which VLA is actually used in the experiments. Is it the same pi0.5 that is reported as baseline result? How is this model pretrained? I would strongly assume that the base pi0.5 does not work well in robomme without task and scene specific finetuning.
  • Scientific standard of reproducibility would require to also report results with an open-weights model, because results just with the codex API cannot be reproduced or validated. Even for the same model name and reasoning level, the underlying models behind the API change e.g. in their quantization.
  • The description of the method itself is hard to follow and a lot of details on the harness are missing. E.g. equation 3 simply summarizes "VLM call and surrounding harness operations", but there is no list of definition of what these harness operations are and how they are defined. There is no file structure given or prompt skeletion. I cannot distinguish whether the authors use a lot of overblown structure around some prompt engineering or if there is a considerable skeleton with a lot of operations outside of the VLM call.
Critical strengths
  • The paper shows a considerable improvement on reference tasks and a good improvement on permeance tasks in the Robo MME benchmark.
Critical weaknesses
  • This work presents a single data point: One specific VLM and one specific VLA where this specific harness design leads to an improvement on one specific benchmark. There are no generalization tests over multiple VLMs, multiple VLAs, and there is no valiation of these improvements on additional benchmarks or real-world trials. There are also no ablation studies which would provide useful insight on the design decisions of the harness.
  • The above weakness unfortunately leads me to the conclusion that there is no scientific insight to draw from this publication in its current form. It is an interesting reference implementation, and the improvement on the reference-class tasks is motivates to look more deeply into this agentic memory system, but for a new VLM and a new VLA one can only take this proposal as reference and either hope it transfers or figure out all the design decisions again.
Questions for authors
  • Which VLA is used in the experiments?
  • Can the same harness be used with different VLMs?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ACAL

Question 1

Which VLA is used in the experiments?

Thank you for asking us to clarify the executor. We use RoboMME’s released π0.5 checkpoint fine-tuned for grounded-subtask execution, mme_vla_suite/symbolic-grounded-subgoal/79999. Its weights remain fixed during harness self-evolution and evaluation. This also addresses your concern in Weakness 3 about task-specific adaptation: our executor is already fine-tuned for RoboMME. It receives grounded subtask commands together with visual and robot-state inputs, as described in Appendix B.1. The standalone π0.5 baseline in Table 1 is an externally reported configuration with different conditioning and should not be interpreted as an identical-checkpoint comparison. In our framework, the upstream planner issues subtasks to the downstream VLA executor; other VLAs could potentially be used, but we retain RoboMME’s fine-tuned π0.5 to stay consistent with the baseline executor family and support fair comparison within our resource budget.

Question 2

Can the same harness be used with different VLMs?

Thank you for raising this question. Yes, the same harness framework can be used with different VLM planners through its structured planning and command interface. We added experiments with two fine-tuned open-weight planners of different sizes, Qwen3.5-4B and Qwen3.8-27B. SERA (w/o self-evolution) achieves average success rates of 39.13% and 42.75%, respectively, compared with 32.70% for GroundSG+QwenVL. These experiments also address your concerns in Weaknesses 1 and 4 about evaluation with multiple VLMs and the lack of open-weight model results. Due to limited computational resources, self-evolution results for these two planners are not yet available; we will add them as they become available.

MethodCountingPermanenceReferenceImitationAverage ↑
GroundSG+QwenVL38.0039.3431.5621.8932.70
SERA + Qwen3.5-4B w/o SE39.0051.5053.0013.0039.13
SERA + Qwen3.8-27B w/o SE38.5053.5062.0017.0042.75
SERA + GPT-5.6-sol high w/o SE34.0056.0069.0018.5044.38
SERA (Ours)49.0057.0069.5030.0051.38

The GPT-5.6-sol high results with and without self-evolution are from our paper; SERA is the full self-evolved configuration.

Response to Weaknesses 1 and 2: harness design and ablations

Thank you for highlighting the need to examine the harness design. During development, we observed three recurring difficulties: mismatches between the planner’s interpretation and the executor’s command semantics, mistaking apparent visual progress for actual subtask completion, and inconsistent grounding outputs. Together with effective repairs found through open-ended model exploration, these observations motivated Planning Guidance (G), Assessment Criteria (A), and Control Rules (C) as a structured, revisable basis for harness updates.

Planning Guidance clarifies how task concepts map to executor commands. In PatternLock, it explains the mapping from the robot’s reference frame to the front camera’s image frame: moving left corresponds to image-right, and moving backward corresponds to image-up. Assessment Criteria specify the evidence needed to judge completion using visual observations, remembered context, and available robot state, without access to simulator-internal success signals. For example, in MoveCube, the peg should move with the gripper before pickup is considered complete. Control Rules address grounding mismatches through programmatic checks that reject spatially inconsistent outputs before dispatch. In SwingXtimes, the physical right-side target appears in the image-left half; a command to that target paired with a point in the image-right half is rejected before it reaches the executor.

Given the available time and model-call budget, we added ablations on six tasks with ten episodes each. We keep the planner, executor, initialization, test episodes, and evaluation budgets fixed. We remove the corresponding G or C revisions; w/o A replaces task-specific assessment refinements with generic completion checks while retaining the controller's required interfaces.

TaskFullw/o Gw/o Aw/o C
MoveCube6/106/104/106/10
PatternLock5/101/105/10†6/10
SwingXtimes10/1010/10†6/105/10
PickXtimes10/1010/10†7/103/10
VideoRepick2/101/105/102/10
ButtonUnmask6/105/106/10†5/10

† No revision of the ablated type is present, so Full is reused.

The aggregate results are:

ConfigurationSuccesses / evaluated episodesSuccess rateDecrease from Full
Full39/6065.0%—
w/o G33/6055.0%10.0 pp
w/o A33/6055.0%10.0 pp
w/o C27/6045.0%20.0 pp
Reviewer_gUm8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes SERA, a self‑evolving robotic agent powered by an editable external harness framework. On one hand, it explicitly maintains history‑dependent task contexts during task execution to guide planning by a frozen VLM and low‑level execution by a VLA. On the other hand, it automatically revises the harness using environment rollout trajectories at the development stage without fine‑tuning large models, achieving performance improvements on the RoboMME benchmark.

Critical strengths
  1. The proposed method adopts a harness to maintain structured explicit context, which can effectively handle non‑Markovian, history‑dependent long‑horizon robotic manipulation tasks.
  2. Self‑evolution only iterates external harness components without fine‑tuning large models or constructing SFT datasets, reducing the cost of task adaptation.
  3. Experiments demonstrate significant performance gains on the RoboMME benchmark.
Critical weaknesses
  1. It lacks direct comparisons with some representative memory‑augmented VLA methods such as MemoryVLA, leading to limited coverage of baseline evaluations.
  2. Validation is only performed within the RoboMME simulated environment, and the effectiveness of the algorithm has not been tested on real‑world robotic platforms.
  3. It does not quantitatively report runtime‑efficiency metrics including planning latency and computational overhead introduced by VLM inference.
Questions for authors

See above Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Weaknesses

Weakness 1

Thank you for pointing out the missing comparison with MemoryVLA. As of November 14, 2026 (AoE), we have not found RoboMME evaluation results on the MemoryVLA authors’ official project page or in their repository README. Due to our limited computational resources, we use the reproduction results provided by the RoboMME team, which report an overall average success rate of 7.13%. The detailed results are shown below.

Reported success rates on RoboMME (%, higher is better):

MethodCountingPermanenceReferenceImitationAverage ↑
MemoryVLA7.178.336.176.837.13

Category scores are averaged across the four tasks and three seeds in each category, using the reported per-seed results before rounding. You can check this link for their detailed per-task results and implementation details.

Weakness 2

Thank you for highlighting the need for real-world validation. We have initiated real-robot experiments to complement our simulation results. These experiments are not yet complete due to limited computational resources, and we will add the results here once they are available.

Weakness 3

Thank you for this suggestion. We use gpt-5.6-sol with high reasoning effort as the planner and acknowledge that this hierarchical architecture introduces additional planning overhead. We access gpt-5.6 sol via an API and deploy MemER on a single NVIDIA A800 GPU with 80 GB memory. The table below reports mean planner-call latency, VLA-inference latency, and total time per planning–execution round. The total includes both calls, execution of the resulting action chunk, and within-round overhead. These averages are computed over all recorded rounds across the full 800 RoboMME episodes per method, including successes, failures, and timeouts.

ConfigurationMean planner-call latency (s)Mean total round time (s)
gpt-5.6 sol26.6027.33
MemER32.3133.97
AuthorsOpenReview ↗

Response to W2 real-world validation

For real-world validation, We have now evaluated SERA on four physical robot tasks, with 12 trials per task and evaluated configuration:

  • SwingXtimes: move a grasped cube back and forth a specified number of times, tracking completed repetitions.
  • VideoRepick: pick the cube selected in an earlier demonstration, including trials with subsequent position swaps.
  • VideoUnmask: lift the cup hiding a cube, tracking its location through covering and possible cup swaps.
  • PatternLock: visit markers in the order shown in a demonstration.

We compare against π0.5, which directly predicts actions from the task name, current front/wrist images, and robot state without historical context, and Keyframe+GroundSG, which uses a fine-tuned Qwen3.5-4B planner to predict grounded subgoals from historical keyframes and recent images for a subtask-trained π0.5 executor. SERA w/o SE uses the initial harness without self-evolution. The table reports full-task successes:

MethodSwingXtimesVideoRepickVideoUnmaskPatternLock
π0.50/123/121/120/12
Keyframe+GroundSG2/127/125/124/12
SERA w/o SE7/1210/129/122/12
SERA†10/1210/12‡9/12‡12/12

SERA w/o SE achieves 58.3% average success, compared with 37.5% for Keyframe+GroundSG and 8.3% for π0.5. SERA† applies one round of harness corrections proposed from failure analysis, followed directly by testing, improving SwingXtimes from 7/12 to 10/12 and PatternLock from 2/12 to 12/12. These results provide evidence of SERA's effectiveness on the evaluated real-world tasks.

† Single-round correction, without the iterative validation used in simulation. ‡ Initial results carried over without re-testing.

Comments on the paper

3 comments
Area_Chair_iGhQOpenReview ↗

Additional Review from Area Chair

Paper Summary

This paper proposes SERA to address history-dependent reasoning and execution recovery in long-horizon robotic manipulation tasks. Specifically, the authors introduce a VLM-based agentic harness built around a frozen VLA executor. The harness maintains and updates task context, enabling the high-level planner to make decisions based on historical information and execution feedback. Furthermore, the authors propose an execution-driven self-evolution mechanism that analyzes execution failures, revises the harness, and validates improvements through environment rollouts without updating model parameters. On the RoboMME benchmark, SERA achieves an overall success rate of 51.38%, representing a 7% improvement over the version without self-evolution.

Strengths

  1. This paper addresses history-dependent long-horizon robotic manipulation, focusing on important challenges such as task progress tracking, the use of historical information, and execution feedback.

  2. SERA enables task context management and execution-driven optimization through an external agentic harness without updating the parameters of the high-level VLM or low-level VLA.

  3. SERA achieves an overall success rate of 51.38% on RoboMME, improving upon the initial harness by 7%.

Weaknesses

  1. The proposed method uses GPT-5.6-sol with high reasoning effort as its high-level planner, while the baselines employ different foundation models. Therefore, it is difficult to determine whether the performance improvements stem from the stronger VLM or the proposed agentic harness.

  2. The proposed self-evolution mechanism involves several components, including Planning Guidance, Assessment Criteria, and Control Rules. However, the current experiments mainly compare the complete self-evolution framework with the initial harness, without evaluating the individual contributions of these components.

  3. The experiments are primarily conducted on the RoboMME benchmark, where harness revisions can leverage task-specific information and execution feedback. It remains unclear whether the improvements achieved through self-evolution can generalize to unseen tasks.

Rating: 2: Weak rejection

AuthorsOpenReview ↗

Reply to Weakness 2

Thank you for pointing out the missing component-level evaluation. During the early development of our approach, we observed recurring problems: the agent’s interpretation of task concepts could be misaligned with the downstream subtask executor; apparent visual progress could be mistaken for actual subtask completion; and predicted grounding points could be inconsistent with the observed scene. By examining these problems alongside effective repairs proposed during open-ended model exploration, we organized harness revisions into Planning Guidance (G), Assessment Criteria (A), and Control Rules (C). These categories provide a structured reference for harness revision and remain open to refinement.

A G revision clarifies semantic misunderstandings identified during self-evolution to guide subsequent planning. For example, in PatternLock, it clarifies how directions transform from the robot’s reference frame to the front camera’s image frame: moving left corresponds to image-right, and moving backward corresponds to image-up. This helps the planner select executor commands that move toward the intended visual target. An A revision addresses errors caused by inadequate visual completion criteria by specifying the evidence the planner should check, such as requiring the peg to move with the gripper before judging pickup complete in MoveCube. A C revision addresses grounding mismatches through programmatic checks that reject spatially inconsistent outputs before dispatch. For instance, in SwingXtimes, the physical right-side target appears in the image-left half; a command to that target paired with a point in the image-right half is therefore rejected before it reaches the executor.

To evaluate the utility of these three revision categories, we added ablations on six tasks with ten episodes each, given the rebuttal timeline and model-call quotas. We keep the planner, executor, initialization, test episodes, and evaluation budgets fixed. We remove the corresponding G or C revisions; w/o A replaces task-specific assessment refinements with generic completion checks while retaining the controller's required interfaces.

The task-level results are shown below:

TaskFullw/o Gw/o Aw/o C
MoveCube6/106/104/106/10
PatternLock5/101/105/10†6/10
SwingXtimes10/1010/10†6/105/10
PickXtimes10/1010/10†7/103/10
VideoRepick2/101/105/102/10
ButtonUnmask6/105/106/10†5/10

† No revision of the ablated type is present, so the configuration is unchanged and Full is reused.

Aggregating across these tasks gives:

ConfigurationSuccesses / evaluated episodesSuccess rateDecrease from Full
Full39/6065.0%—
w/o G33/6055.0%10.0 pp
w/o A33/6055.0%10.0 pp
w/o C27/6045.0%20.0 pp
AuthorsOpenReview ↗

Response to W1

Thank you for raising this concern. To examine the contribution of the harness, we designed an experiment comparing SERA's harness-based memory management with the GroundSG baseline. We evaluate both methods with Qwen3.5-4B and Qwen3.8-27B, matching planner model families and sizes across the comparisons.

MethodCountingPermanenceReferenceImitationAverage ↑
GroundSG + Qwen3.5-4B28.0041.0030.0026.5031.38
GroundSG + Qwen3.8-27B49.5038.5049.0028.5041.38
SERA harness + Qwen3.5-4B39.0051.5053.0013.0039.13
SERA harness + Qwen3.8-27B38.5053.5062.0017.0042.75

All entries are success rates (%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.