- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
strengths
- The paper shows a considerable improvement on reference tasks and a good improvement on permeance tasks in the Robo MME benchmark.
weaknesses
- This work presents a single data point: One specific VLM and one specific VLA where this specific harness design leads to an improvement on one specific benchmark. There are no generalization tests over multiple VLMs, multiple VLAs, and there is no valiation of these improvements on additional benchmarks or real-world trials. There are also no ablation studies which would provide useful insight on the design decisions of the harness.
- The above weakness unfortunately leads me to the conclusion that there is no scientific insight to draw from this publication in its current form. It is an interesting reference implementation, and the improvement on the reference-class tasks is motivates to look more deeply into this agentic memory system, but for a new VLM and a new VLA one can only take this proposal as reference and either hope it transfers or figure out all the design decisions again.
- I cannot find any info which VLA is actually used in the experiments. Is it the same pi0.5 that is reported as baseline result? How is this model pretrained? I would strongly assume that the base pi0.5 does not work well in robomme without task and scene specific finetuning.
- Scientific standard of reproducibility would require to also report results with an open-weights model, because results just with the codex API cannot be reproduced or validated. Even for the same model name and reasoning level, the underlying models behind the API change e.g. in their quantization.
- The description of the method itself is hard to follow and a lot of details on the harness are missing. E.g. equation 3 simply summarizes "VLM call and surrounding harness operations", but there is no list of definition of what these harness operations are and how they are defined. There is no file structure given or prompt skeletion. I cannot distinguish whether the authors use a lot of overblown structure around some prompt engineering or if there is a considerable skeleton with a lot of operations outside of the VLM call.
Critical strengths
- The paper shows a considerable improvement on reference tasks and a good improvement on permeance tasks in the Robo MME benchmark.
Critical weaknesses
- This work presents a single data point: One specific VLM and one specific VLA where this specific harness design leads to an improvement on one specific benchmark. There are no generalization tests over multiple VLMs, multiple VLAs, and there is no valiation of these improvements on additional benchmarks or real-world trials. There are also no ablation studies which would provide useful insight on the design decisions of the harness.
- The above weakness unfortunately leads me to the conclusion that there is no scientific insight to draw from this publication in its current form. It is an interesting reference implementation, and the improvement on the reference-class tasks is motivates to look more deeply into this agentic memory system, but for a new VLM and a new VLA one can only take this proposal as reference and either hope it transfers or figure out all the design decisions again.
Questions for authors
- Which VLA is used in the experiments?
- Can the same harness be used with different VLMs?
Response to Reviewer ACAL
Question 1
Which VLA is used in the experiments?
Thank you for asking us to clarify the executor. We use RoboMME’s released π0.5 checkpoint fine-tuned for grounded-subtask execution,
mme_vla_suite/symbolic-grounded-subgoal/79999. Its weights remain fixed during harness self-evolution and evaluation. This also addresses your concern in Weakness 3 about task-specific adaptation: our executor is already fine-tuned for RoboMME. It receives grounded subtask commands together with visual and robot-state inputs, as described in Appendix B.1. The standalone π0.5 baseline in Table 1 is an externally reported configuration with different conditioning and should not be interpreted as an identical-checkpoint comparison. In our framework, the upstream planner issues subtasks to the downstream VLA executor; other VLAs could potentially be used, but we retain RoboMME’s fine-tuned π0.5 to stay consistent with the baseline executor family and support fair comparison within our resource budget.Question 2
Can the same harness be used with different VLMs?
Thank you for raising this question. Yes, the same harness framework can be used with different VLM planners through its structured planning and command interface. We added experiments with two fine-tuned open-weight planners of different sizes, Qwen3.5-4B and Qwen3.8-27B. SERA (w/o self-evolution) achieves average success rates of 39.13% and 42.75%, respectively, compared with 32.70% for GroundSG+QwenVL. These experiments also address your concerns in Weaknesses 1 and 4 about evaluation with multiple VLMs and the lack of open-weight model results. Due to limited computational resources, self-evolution results for these two planners are not yet available; we will add them as they become available.
The GPT-5.6-sol high results with and without self-evolution are from our paper; SERA is the full self-evolved configuration.
Response to Weaknesses 1 and 2: harness design and ablations
Thank you for highlighting the need to examine the harness design. During development, we observed three recurring difficulties: mismatches between the planner’s interpretation and the executor’s command semantics, mistaking apparent visual progress for actual subtask completion, and inconsistent grounding outputs. Together with effective repairs found through open-ended model exploration, these observations motivated Planning Guidance (G), Assessment Criteria (A), and Control Rules (C) as a structured, revisable basis for harness updates.
Planning Guidance clarifies how task concepts map to executor commands. In PatternLock, it explains the mapping from the robot’s reference frame to the front camera’s image frame: moving left corresponds to image-right, and moving backward corresponds to image-up. Assessment Criteria specify the evidence needed to judge completion using visual observations, remembered context, and available robot state, without access to simulator-internal success signals. For example, in MoveCube, the peg should move with the gripper before pickup is considered complete. Control Rules address grounding mismatches through programmatic checks that reject spatially inconsistent outputs before dispatch. In SwingXtimes, the physical right-side target appears in the image-left half; a command to that target paired with a point in the image-right half is rejected before it reaches the executor.
Given the available time and model-call budget, we added ablations on six tasks with ten episodes each. We keep the planner, executor, initialization, test episodes, and evaluation budgets fixed. We remove the corresponding G or C revisions; w/o A replaces task-specific assessment refinements with generic completion checks while retaining the controller's required interfaces.
† No revision of the ablated type is present, so Full is reused.
The aggregate results are: