acceptodds
Under review as a conference paper at ICLR 2027

HiSTEVE: Harnessing Human-Interface VLA Agents for Open-World Games

Abstract

Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA. To our knowledge, HiSTEVE is the first agent to complete long-horizon tasks such as crafting an iron pickaxe from scratch under the same interaction constraints as human players.

Primary area:
foundation or frontier models, including LLMs
Keywords:
open-world games agents, vision-language-action models, agent harnesses, long-horizon autonomy, native-interface control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_T4caOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HiSTEVE, a hierarchical VLA agent for long horizon tasks in Minecraft using only visual observations and native keyboard and mouse controls. The method combines a capability aware agent harness with a VLA policy. The authors introduce regime specific RMSNorm and Dual Mouse representations to handle GUI and non GUI interactions, along with a task completion detector for better execution control. Experiments on MineStudio, OpenHA, and LongBench show improvements in both atomic task execution and long horizon performance.

Critical strengths
  1. The proposed RS-Norm and Dual-Mouse designs effectively address the interference between GUI and non GUI interactions. The design is lightweight and well motivated by the gradient analysis. The ablation results also demonstrate the benefits of both components.
  2. The capability aware harness integrates planning, structured memory, task completion detection, and recovery into a coherent framework. The long horizon results are encouraging, especially the improvements achieved with the same underlying VLA policy.
Critical weaknesses
  1. Limited evaluation environments. All experiments are conducted in Minecraft. While Minecraft is a useful testbed, it remains unclear whether the proposed architecture and harness can generalize to other games or interactive environments.
  2. Limited long horizon reliability. Despite the improvements, the success rates on several LongBench task categories remain relatively low. This suggests that robust long horizon execution is still challenging, particularly for recovery and open ended tasks.
Questions for authors
  1. Can the authors provide additional evidence that RS-Norm and Dual-Mouse generalize beyond Minecraft, particularly to other environments involving both GUI and non GUI interactions?
  2. What are the main failure cases in LongBench, and which components of the harness contribute most to the remaining failures?
4 comments
AuthorsOpenReview ↗

Response to Reviewer T4ca (Part 1/2)

We sincerely thank you for your thoughtful comments on our paper and for recognizing the effectiveness of RS-Norm and Dual-Mouse, as well as the design of our capability-aware harness. We respond below to your questions about applicability beyond Minecraft and the main failure cases and component contributions on LongBench.

1. Applicability beyond Minecraft (Weakness 1 and Question 1).

To examine the applicability of our proposed RS-Norm and Dual-Mouse beyond Minecraft, we conducted an additional visual-to-action prediction experiment in Raft. Raft includes both GUI operations, such as inventory and crafting interactions, and non-GUI operations, such as first-person view control, with mouse displacement carrying different semantics in the two types of interaction. Using human gameplay trajectories from D2E (Choi et al., 2026) and Qwen2.5-VL-7B-Instruct as the common backbone, we compared four configurations under the same training data, image inputs, and optimization settings: a baseline without either RS-Norm or Dual-Mouse, a model with RS-Norm only, a model with Dual-Mouse only, and a model with both RS-Norm and Dual-Mouse. We evaluated the final checkpoints using greedy decoding on a total of 10,000 test samples balanced between GUI and non-GUI interactions, reporting full-action accuracy (the proportion of samples with a valid generated sequence whose keyboard key set, mouse button states, scroll action, and quantized horizontal and vertical mouse displacements all match the ground-truth action) and mouse-displacement accuracy (the proportion of samples for which both quantized horizontal and vertical mouse displacements match the ground truth).

MetricBaselineRS-onlyDM-onlyBoth
Full-action accuracy11.0%12.8%12.7%14.6%
Mouse-displacement accuracy29.5%31.2%30.9%33.0%

As shown in the table, RS-Norm and Dual-Mouse each improve both metrics when used individually, and their combination achieves the best results. Compared with the baseline, the combined configuration improves full-action accuracy and mouse-displacement accuracy by 3.6 and 3.5 percentage points, respectively. These results show that distinguishing interaction regimes at the levels of feature normalization and mouse-action representation can improve action prediction, providing evidence for the applicability of both designs to mixed GUI/non-GUI interaction settings beyond Minecraft.

AuthorsOpenReview ↗

Response to Reviewer T4ca (Part 2/2)

2. LongBench failures and component contributions (Weakness 2 and Question 2).

HiSTEVE's failures on LongBench are mainly concentrated in recovery from unexpected situations, completion of open-ended goals, use of information over long horizons, and VLA execution reliability. Recovery and Open both achieve only 14.3%, making them the two lowest-performing task families. This indicates that continuing execution after tool failure, retrieving missed resources, and sustained planning and execution under weakly structured goals are the most prominent current capability limitations. The E&M score of 28.6% reflects difficulties in exploration, location revisiting, and using environmental information across stages. The R&C success rate of 42.9% indicates that more clearly structured resource acquisition and crafting chains are relatively easier to complete; their failures are primarily caused by limited VLA execution reliability, which leads to failures when the VLA step budget is exhausted. The above is a task-level analysis. We additionally conducted ablation experiments on the harness components, with the following results:

ConfigurationR&CE&MRecoveryOpenAvg.
Full HiSTEVE42.928.614.314.325.0
w/o state detector (fixed-horizon handoff)19.812.65.45.110.7
w/o candidate-completion verification31.620.69.27.317.2
w/o local recovery26.621.94.95.214.7
w/o retrieved memory39.319.111.211.920.4

The ablations show that, among the harness components examined, execution handoff and local error correction have the greatest impact on final success rates: removing the state detector or local recovery reduces the macro-average from 25.0% to 10.7% and 14.7%, respectively, indicating that timely handoff and prompt correction of execution deviations are crucial for long-horizon completion. Removing candidate-completion verification reduces the macro-average to 17.2%, demonstrating the importance of verifying completion states and avoiding erroneous advancement. The impact of retrieved memory is more concentrated in exploration and memory tasks: disabling retrieval reduces E&M from 28.6% to 19.1%, while R&C remains at 39.3%, supporting the role of retrieving previously observed landmarks and past experience in using information across stages and revisiting locations. These results reveal key weaknesses spanning handoff, verification, error correction, and memory use.

We will include the Raft experiment, harness ablations, and failure analysis in the revised manuscript.

Reference

Choi et al. (2026). D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI. International Conference on Learning Representations.

We sincerely thank you again for your constructive suggestions. We hope our responses have addressed your concerns, and we look forward to your feedback and further discussion.

Thank you for your rebuttal, which has resolved my concerns. The Raft game has addressed my doubts about the generalization of the method and appears to support the effectiveness of RS-Norm and Dual-Mouse. The analysis of the LongBench failures is also fairly convincing, and I hope the authors will incorporate these revisions in a subsequent version. Therefore, I will maintain my original score of 3 (weak acceptance).

We sincerely thank the reviewer for your positive feedback. Your comments have been invaluable in improving our work. We will make sure to incorporate these revisions in the subsequent version. We truly appreciate the time and support you have devoted to reviewing our paper!

Reviewer_5WeHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents HiSTEVE, a hierarchical Minecraft agent that pairs a capability-aware harness with a native-interface VLA. A GPT-4o planner picks parameterized atomic subtasks from a schema library restricted to what the VLA can actually execute; a structured memory tracks verified progress; a policy-coupled completion detector (1.4B, conditioned on the VLA's hidden state) signals candidate handoff, and the harness then verifies completion visually before advancing, recovering, or replanning. The VLA (Qwen2.5-VL-7B) acts through keyboard and mouse only, with two regime-specific mechanisms, RS-Norm branches and Dual-Mouse tokens, to separate GUI from non-GUI interference, trained in three stages (world knowledge injection, imitation learning, environment-feedback RL). Results: 85.0% on MineStudio Bench (vs 65.0% for the strongest baseline), 37.4% on OpenHA (vs 36.8%), and LongBench goal success of 25.0% versus about 6.4% for the best baseline. Ablations show the harness more than doubles long-horizon success with the VLA frozen, while RL improves atomic execution without improving long-horizon success at all. The detector beats GPT-5.5 by 8.7 F1 points on a balanced validation split.

Critical strengths
  • The paper proves its central claim with the right experiment. With the VLA frozen, swapping fixed-horizon handoff for the full harness more than doubles LongBench goal success (10.7% to 25.0%) while atomic performance barely moves (81.2% to 85.0%), so the gain is orchestration, not a stronger controller. The training ablation makes the same point from the other side: RL lifts MineStudio from 65.1% to 80.7% yet leaves LongBench goal success exactly at 14.3%. Together these isolate a lesson the field keeps relearning the hard way: local execution reliability and long-horizon capability are different problems.
Critical weaknesses
  • Attribution to the planner is unmeasured, and the generality claim outruns the evidence. All planning runs through GPT-4o, and there is no ablation swapping in a weaker or open planner with the harness and VLA fixed, so we cannot tell how much of the 25.0% LongBench success is harness design versus frontier-model planning. Meanwhile the title says "games" but every component is Minecraft-specific: the schema library is hand-authored (an admitted limitation), the detector is trained on VPT human gameplay, and all three benchmarks are one game. Nothing here shows the harness transfers.
Questions for authors
  • What do you think is the novelty of this paper? The harness pattern has close precedent in Voyager, JARVIS-1, DoReMi, which makes the contribution look more like a well-executed combination than a new mechanism.
  • Can you ablate the planner, keeping the harness and VLA fixed, with a weaker or open-weight VLM planner? If LongBench goal success collapses without GPT-4o, the contribution is partly a frontier-planner result; if it holds, the harness-design claim is much stronger. Either answer materially changes my assessment.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5WeH (Part 1/2)

We sincerely thank you for your thoughtful comments on our paper and for recognizing the evidence from our controlled experiments for the role of harness orchestration in long-horizon performance. We respond below to your questions about novelty, planner dependence, and generality.

1. Novelty (Question 1).

(1) Capability constraints and handoff mechanisms for VLAs. Voyager, JARVIS-1, and DoReMi indeed include planning, verification, or recovery mechanisms. Compared with these works, our key mechanism extension is to introduce the low-level VLA's policy context into completion detection. This policy-coupled design makes a lightweight detector possible, reducing system overhead. Please see our response to Reviewer pHQr for the corresponding experimental results. Meanwhile, our harness predefines the VLA's capability boundaries and filters subtasks based on their applicability before planning to reduce calls to subtasks that exceed those capabilities or do not meet the applicability conditions.

(2) Control representations for heterogeneous interactions. At the VLA keyboard-and-mouse control level, our novel RS-Norm distinguishes interaction regimes at the normalization layers to mitigate interference in feature learning; Dual-Mouse separates action representations for camera and cursor movement to address the semantic ambiguity of the same mouse input. Gradient analysis and individual and joint ablations support their design motivations and complementary benefits.

2. Planner replacement experiment (Weakness and Question 2).

We conducted an additional planner replacement experiment, replacing GPT-4o with Qwen3.5-9B and Qwen3.5-35B-A3B while keeping the VLA checkpoint, state detection, completion verification, memory, and recovery mechanisms fixed. All configurations use the same LongBench tasks and evaluation budgets: three environment seeds, with 100 runs per task per seed. We report goal success rates for the four task families and their macro-average. The results are as follows:

ConfigurationR&CE&MRecoveryOpenAvg.
Full HiSTEVE42.928.614.314.325.0
GPT-4o → Qwen3.5-9B17.610.45.45.29.7
GPT-4o → Qwen3.5-35B-A3B38.624.912.814.122.6

The results show that replacing GPT-4o with Qwen3.5-9B reduces the macro-average success rate from 25.0% to 9.7%, with substantial degradation across all four task families. In contrast, Qwen3.5-35B-A3B achieves 22.6%, a decrease of only 2.4 percentage points, retaining approximately 90.4% of the GPT-4o configuration's macro-average success rate. This shows that HiSTEVE can accommodate open-weight planners and retain most of its long-horizon task performance with a stronger open model, while still requiring sufficient high-level planning capability. The smaller model's limitations in goal decomposition, state understanding, and sustained decision-making limit the effectiveness of the orchestration framework, leading to reduced long-horizon execution performance.

We will include this planner comparison in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 5WeH (Part 2/2)

3. Scope of generality (Weakness).

We agree that the current validation is confined to Minecraft and that the existing evidence is insufficient to support a conclusion of cross-game effectiveness. In the revised manuscript, we will explicitly specify Minecraft World in the title and correspondingly adjust the abstract, contribution summary, and conclusion, limiting our empirical contribution to unified control and long-horizon orchestration through native keyboard-and-mouse interaction in Minecraft. As Reviewer T4ca raised the question of cross-game generalization, we also conducted additional cross-game experiments on RS-Norm and Dual-Mouse: to examine the applicability of our proposed RS-Norm and Dual-Mouse beyond Minecraft, we conducted an additional visual-to-action prediction experiment in Raft. Raft includes both GUI operations, such as inventory and crafting interactions, and non-GUI operations, such as first-person view control, with mouse displacement carrying different semantics in the two types of interaction. Using human gameplay trajectories from D2E (Choi et al., 2026) and Qwen2.5-VL-7B-Instruct as the common backbone, we compared four configurations under the same training data, image inputs, and optimization settings: a baseline without either RS-Norm or Dual-Mouse, a model with RS-Norm only, a model with Dual-Mouse only, and a model with both RS-Norm and Dual-Mouse. We evaluated the final checkpoints using greedy decoding on a total of 10,000 test samples balanced between GUI and non-GUI interactions, reporting full-action accuracy (the proportion of samples with a valid generated sequence whose keyboard key set, mouse button states, scroll action, and quantized horizontal and vertical mouse displacements all match the ground-truth action) and mouse-displacement accuracy (the proportion of samples for which both quantized horizontal and vertical mouse displacements match the ground truth).

MetricBaselineRS-onlyDM-onlyBoth
Full-action accuracy11.0%12.8%12.7%14.6%
Mouse-displacement accuracy29.5%31.2%30.9%33.0%

As shown in the table, RS-Norm and Dual-Mouse each improve both metrics when used individually, and their combination achieves the best results. Compared with the baseline, the combined configuration improves full-action accuracy and mouse-displacement accuracy by 3.6 and 3.5 percentage points, respectively. These results show that distinguishing interaction regimes at the levels of feature normalization and mouse-action representation can improve action prediction, providing evidence for the applicability of both designs to mixed GUI/non-GUI interaction settings beyond Minecraft.

Regarding the harness transferability you mentioned, we would like to offer a brief discussion here. Our analysis distinguishes orchestration mechanisms from environment-specific instantiation. The former includes capability-constrained subtask selection, visual verification following candidate completion, memory updates based on verified progress, and recovery and replanning, which provide a design basis for reuse across environments in embodied tasks. The latter includes Minecraft schemas, executor capabilities, and detector training data, which need to be reconfigured for the target environment.

Reference

Choi et al. (2026). D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI. International Conference on Learning Representations.

We sincerely thank you again for your constructive suggestions. We hope our responses have addressed your concerns, and we look forward to your feedback and further discussion.

The paper proposes a VLA-combined harness package for minecraft, getting superior performance compared to prior work especially in long horizon tasks. The experiments (fixed controller harness comparison, environment feedback RL, RS-Norm and Dual-Mouse) in the paper are well designed to show the effectiveness of the author's proposed method. There is a question on the novelty and significance of this work, especially since there were lots of papers Voyager, JARVIS-1, and DoReMi that tried doing a similar orchestration mechanic, and the author's introduced mechanics has some questioning on the generalizability when it's not proved with experiments. In the rebuttal the authors defensed the novelty by positioning the paper's contribution as introducing the low-level VLA's policy context into completion detection, and RS-Norm + Dual-Mouse, also with experiments that show generalization capability of RS-Norm + Dual-Mouse and the role of the planner. I decide to raise my score.

We sincerely thank the reviewer for your positive feedback. Your comments have been invaluable in improving our work. We truly appreciate the time and support you have devoted to reviewing our paper!

Reviewer_pHQrOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

HiSTEVE is a hierarchical Minecraft agent that combines a capability-constrained VLM planner with a single VLA executor using first-person images and native keyboard-and-mouse actions. Its harness maintains structured memory and coordinates completion verification, recovery, and replanning. A policy-coupled completion detector signals when execution should return to the harness. For low-level control, the paper introduces regime-specific RMSNorm parameters and separate mouse-action tokens for GUI and non-GUI interactions, followed by knowledge injection, imitation learning, and environment-feedback post-training. Evaluation reports 85.0% success on MineStudio, 37.4% on OpenHA, and improved long-horizon performance on the introduced LongBench suite. Ablations examine policy training, regime-specific representations, detector architecture, and harness orchestration.

Critical strengths
  1. The system meaningfully integrates native-interface execution with long-horizon coordination. High-level planning is substantially easier when an agent can delegate execution to scripted skills or privileged APIs. HiSTEVE instead performs physical interactions through a learned keyboard-and-mouse policy and uses visual evidence during deployment. The fixed-policy harness comparison—raising LongBench goal success from 10.7% to 25.0%—provides useful evidence that organizing existing capabilities matters beyond simply strengthening the executor.

  2. The regime-specific control design is lightweight and empirically motivated. GUI cursor movement and first-person camera movement have different semantics despite sharing physical mouse inputs. The gradient analysis motivates specialization at normalization layers, while separate mouse tokens address action-representation ambiguity. Their individual and combined ablations show substantial gains, particularly on MineStudio, making this a concrete and potentially reusable contribution to heterogeneous visuomotor control.

Critical weaknesses
  1. The long-horizon evaluation leaves comparison fairness and robustness insufficiently established. In procedurally generated worlds, success depends strongly on initialization, resource availability, and execution budgets. However, LongBench contains only 14 tasks, and the paper does not clearly report per-task evaluation denominators, seed coverage, confidence intervals, or how competing agents receive comparable planning support and budgets. The fixed-policy harness ablation is useful, but simultaneously changes handoff, verification, memory, and recovery, so it cannot isolate their individual contributions. These limitations weaken the evidence for a general long-horizon advantage, particularly since the OpenHA improvement is only 0.6 percentage points.

  2. The policy-coupled completion detector lacks decisive validation under deployment conditions. Premature or delayed handoffs can disrupt execution and consume the budget needed for subsequent subtasks. Yet detection is evaluated on balanced human-gameplay samples rather than deployed-policy trajectories, and the actual error rates and costs of the two-query handoff rule are not reported. Comparisons against prompted VLMs also conflate specialized training with policy coupling; a comparably trained detector without policy-context injection is missing. Finally, no runtime measurements establish whether the 7B VLA and detector can sustain the stated 20 Hz loop. Thus, both the detector’s distinctive benefit and its operational feasibility remain uncertain.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer pHQr (Part 1/2)

We sincerely thank you for your thoughtful comments on our paper and for recognizing the integration of native-interface execution with long-horizon coordination, as well as our regime-specific control designs. We respond below to your questions about evaluation fairness and robustness, harness component contributions, completion detection, and runtime feasibility.

1. Fairness and robustness of long-horizon evaluation (Weakness 1).

LongBench contains 14 tasks across four families. Each task is run 100 times under each of three fixed environment seeds, yielding 300 runs per task. Each episode is recorded as a success or failure according to whether the final task conditions are satisfied; overall Goal SR is the equally weighted macro-average of the four family success rates. All methods use the same environment seeds and task initial states to ensure a fair comparison. Methods in the main table retain their original planning architectures without any modifications and are compared under the same initial conditions, environment action-step limits, and final success criteria. All methods receive only the final goal and execute through their native observation and action interfaces. We will add these evaluation details to the revised manuscript.

2. Fine-grained harness ablations (Weakness 1).

We conducted three additional fine-grained harness ablations, separately disabling candidate-completion verification, local recovery, and retrieved memory while keeping the VLA, GPT-4o planner, tasks, environment seeds, and action-step budgets fixed. In addition, the w/o state detector configuration in the table is the fixed-horizon handoff configuration from the original paper: only detector-driven, state-dependent handoff is replaced with handoff after a predetermined execution horizon, while completion verification, memory, local recovery, and replanning are all retained.

ConfigurationR&CE&MRecoveryOpenAvg.
Full HiSTEVE42.928.614.314.325.0
w/o state detector (fixed-horizon handoff)19.812.65.45.110.7
w/o candidate-completion verification31.620.69.27.317.2
w/o local recovery26.621.94.95.214.7
w/o retrieved memory39.319.111.211.920.4

The ablations show that, among the harness components examined, execution handoff and local error correction have the greatest impact on final success rates: removing the state detector or local recovery reduces the macro-average from 25.0% to 10.7% and 14.7%, respectively, indicating that timely handoff and prompt correction of execution deviations are crucial for long-horizon completion. Removing candidate-completion verification reduces the macro-average to 17.2%, demonstrating the importance of verifying completion states and avoiding erroneous advancement. The impact of retrieved memory is more concentrated in exploration and memory tasks: disabling retrieval reduces E&M from 28.6% to 19.1%, while R&C remains at 39.3%, supporting the role of retrieving historical landmarks and experiences in using information across stages and revisiting locations.

AuthorsOpenReview ↗

Response to Reviewer pHQr (Part 2/2)

3. Completion detection on deployed-policy trajectories and trained controls (Weakness 2).

We conducted an additional completion detection evaluation on trajectories generated by policies in actual deployment. We sampled trajectories actually generated by HiSTEVE and multiple baselines on MineStudio Bench, OpenHA, and LongBench to construct a test set of 2,306 samples, with 1,153 complete and 1,153 incomplete samples. We added completion detectors trained with comparable training procedures but without policy-context injection as controls, using Qwen2.5-VL-3B and 7B with either Text SFT to produce textual completion judgments or a Binary head for binary classification. The control models use only visual observations and task instructions, whereas our detector additionally incorporates the VLA's policy context. All detectors are evaluated on both the VPT validation set and trajectories actually generated by deployed policies, reporting accuracy and F1 metrics. The results are as follows:

ModelMethodVPT validation set Acc (%) ↑VPT validation set F1 (%) ↑Policy rollout Acc (%) ↑Policy rollout F1 (%) ↑
Qwen2.5-VL-3BText SFT82.283.276.469.6
Qwen2.5-VL-3BBinary head81.081.675.267.5
Qwen2.5-VL-7BText SFT87.787.879.073.8
Qwen2.5-VL-7BBinary head87.688.078.974.1
Ours (1.4B)Policy-coupled86.386.379.574.3

In the additional evaluation on trajectories generated by deployed policies, all models score lower than on the VPT validation set but retain good completion-state discrimination. Our detector achieves 79.5% accuracy and 74.3% F1, supporting its ability to provide candidate completion signals for subsequent verification during actual policy execution. Against the two types of trained controls without policy-context injection, our detector outperforms both 3B controls with only 1.4B parameters. Although it scores slightly below the 7B controls on the VPT validation set, the accuracy and F1 gaps are only 1.3–1.7 percentage points, while both metrics are slightly higher than those of the 7B controls on Policy rollout. The detector achieves comparable or better performance with approximately 20% of the parameters of a 7B model, demonstrating the parameter-efficiency advantage provided by policy context.

We further measured handoff behavior and its processing overhead in trajectories generated by deployed policies. The state detector triggers a completion signal when two consecutive queries both exceed the completion threshold; 13.1% of these signals were not confirmed as complete by the subsequent verification pipeline. Among these unconfirmed handoffs, the first subsequent processing branch was local recovery in 58.7% of cases and another call to the high-level planner in 39.6%. From the candidate handoff trigger to the execution of the next VLA action, the median latency was 3.2 seconds for the local recovery path and 5.9 seconds for the high-level replanning path.

4. Runtime measurements and clarification of 20 Hz (Weakness 2).

We conducted additional component-level runtime measurements of the 7B VLA and completion detector, each using a single NVIDIA H200, batch size 1, and BF16 precision. The VLA's mean inference latency is 59.6 ms, with P50 and P95 of 59.9 ms and 65.1 ms, respectively; the completion detector's mean latency is 29.7 ms with the image-encoder features of historical frames cached in advance. We also clarify the meaning of 20 Hz in the original paper: it refers to Minecraft's nominal simulation timescale, rather than the actual frequency of end-to-end model control. The system uses synchronous stepping between the model and environment: the environment advances after receiving an action and returns the next observation, while its progression is paused during model inference. Thus, VLA and detector computation need not fit within each 50 ms real-time window. In the revised manuscript, we will explicitly distinguish simulation time from wall-clock time and correct wording that may imply the system runs at 20 Hz in real time.

We will also include the additional harness ablations, detector evaluations, handoff statistics, and runtime measurements in the revised manuscript.

We sincerely thank you again for your constructive suggestions. We hope our responses have addressed your concerns, and we look forward to your feedback and further discussion.

We sincerely thank the reviewer for your positive assessment. Your comments have been invaluable in improving our work. We hope our responses have addressed your concerns, and we truly appreciate the time and support you have devoted to reviewing our paper!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Response to All Reviewers and Area Chair

Dear Area Chair and Reviewers,

We sincerely thank you for your time and constructive feedback. We appreciate the reviewers’ recognition of our controlled evidence for harness orchestration and the effectiveness of RS-Norm and Dual-Mouse designs. Your suggestions helped us strengthen the evidence and clarify our contributions and scope.

During the rebuttal, we provided additional experiments and clarifications on five key aspects:

  1. Harness Contributions and Failure Analysis: With the VLA fixed, HiSTEVE achieves 25.0% LongBench goal success versus 10.7% using fixed-horizon handoff. We added ablations of verification, local recovery, and retrieved memory, alongside failure analysis covering recovery, open-ended tasks, memory use, and VLA execution reliability.

  2. Planner Dependence and Open-Weight Compatibility: With the VLA and harness fixed, Qwen3.5-35B-A3B achieves 22.6% LongBench goal success versus GPT-4o’s 25.0%, while Qwen3.5-9B achieves 9.7%. These results demonstrate compatibility with open-weight planners while highlighting the need for sufficient planning capability.

  3. Policy-Coupled Completion Detection: On 2,306 balanced samples from deployed-policy trajectories, our 1.4B detector achieves 79.5% accuracy and 74.3% F1, competitive with comparably trained 7B controls without policy context and supporting its parameter efficiency. We also reported candidate-handoff verification outcomes and subsequent processing overhead.

  4. RS-Norm and Dual-Mouse Beyond Minecraft: Controlled action-prediction experiments in Raft show benefits from each design, with their combination improving full-action accuracy from 11.0% to 14.6% and mouse-displacement accuracy from 29.5% to 33.0%. These results support the cross-game generalizability of RS-Norm and Dual-Mouse.

  5. Evaluation Protocol and Runtime: We detailed the LongBench protocol: 300 runs per task across three fixed seeds, with matched initial conditions, action-step budgets, and success criteria. We also reported component runtimes and distinguished nominal 20 Hz simulation time from wall-clock execution under synchronous stepping.

We will incorporate these results and clarifications into the revised manuscript. We hope HiSTEVE advances native-interface agents in Minecraft and offers useful insights into establishing a reliable closed loop between planning and execution.

Once again, we sincerely thank all reviewers and the Area Chair for your thoughtful feedback and careful consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.