- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents HiSTEVE, a hierarchical VLA agent for long horizon tasks in Minecraft using only visual observations and native keyboard and mouse controls. The method combines a capability aware agent harness with a VLA policy. The authors introduce regime specific RMSNorm and Dual Mouse representations to handle GUI and non GUI interactions, along with a task completion detector for better execution control. Experiments on MineStudio, OpenHA, and LongBench show improvements in both atomic task execution and long horizon performance.
Critical strengths
- The proposed RS-Norm and Dual-Mouse designs effectively address the interference between GUI and non GUI interactions. The design is lightweight and well motivated by the gradient analysis. The ablation results also demonstrate the benefits of both components.
- The capability aware harness integrates planning, structured memory, task completion detection, and recovery into a coherent framework. The long horizon results are encouraging, especially the improvements achieved with the same underlying VLA policy.
Critical weaknesses
- Limited evaluation environments. All experiments are conducted in Minecraft. While Minecraft is a useful testbed, it remains unclear whether the proposed architecture and harness can generalize to other games or interactive environments.
- Limited long horizon reliability. Despite the improvements, the success rates on several LongBench task categories remain relatively low. This suggests that robust long horizon execution is still challenging, particularly for recovery and open ended tasks.
Questions for authors
- Can the authors provide additional evidence that RS-Norm and Dual-Mouse generalize beyond Minecraft, particularly to other environments involving both GUI and non GUI interactions?
- What are the main failure cases in LongBench, and which components of the harness contribute most to the remaining failures?
4 comments
Response to Reviewer T4ca (Part 2/2)
2. LongBench failures and component contributions (Weakness 2 and Question 2).
HiSTEVE's failures on LongBench are mainly concentrated in recovery from unexpected situations, completion of open-ended goals, use of information over long horizons, and VLA execution reliability. Recovery and Open both achieve only 14.3%, making them the two lowest-performing task families. This indicates that continuing execution after tool failure, retrieving missed resources, and sustained planning and execution under weakly structured goals are the most prominent current capability limitations. The E&M score of 28.6% reflects difficulties in exploration, location revisiting, and using environmental information across stages. The R&C success rate of 42.9% indicates that more clearly structured resource acquisition and crafting chains are relatively easier to complete; their failures are primarily caused by limited VLA execution reliability, which leads to failures when the VLA step budget is exhausted. The above is a task-level analysis. We additionally conducted ablation experiments on the harness components, with the following results:
| Configuration | R&C | E&M | Recovery | Open | Avg. |
|---|---|---|---|---|---|
| Full HiSTEVE | 42.9 | 28.6 | 14.3 | 14.3 | 25.0 |
| w/o state detector (fixed-horizon handoff) | 19.8 | 12.6 | 5.4 | 5.1 | 10.7 |
| w/o candidate-completion verification | 31.6 | 20.6 | 9.2 | 7.3 | 17.2 |
| w/o local recovery | 26.6 | 21.9 | 4.9 | 5.2 | 14.7 |
| w/o retrieved memory | 39.3 | 19.1 | 11.2 | 11.9 | 20.4 |
The ablations show that, among the harness components examined, execution handoff and local error correction have the greatest impact on final success rates: removing the state detector or local recovery reduces the macro-average from 25.0% to 10.7% and 14.7%, respectively, indicating that timely handoff and prompt correction of execution deviations are crucial for long-horizon completion. Removing candidate-completion verification reduces the macro-average to 17.2%, demonstrating the importance of verifying completion states and avoiding erroneous advancement. The impact of retrieved memory is more concentrated in exploration and memory tasks: disabling retrieval reduces E&M from 28.6% to 19.1%, while R&C remains at 39.3%, supporting the role of retrieving previously observed landmarks and past experience in using information across stages and revisiting locations. These results reveal key weaknesses spanning handoff, verification, error correction, and memory use.
We will include the Raft experiment, harness ablations, and failure analysis in the revised manuscript.
Reference
Choi et al. (2026). D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI. International Conference on Learning Representations.
We sincerely thank you again for your constructive suggestions. We hope our responses have addressed your concerns, and we look forward to your feedback and further discussion.
Thank you for your rebuttal, which has resolved my concerns. The Raft game has addressed my doubts about the generalization of the method and appears to support the effectiveness of RS-Norm and Dual-Mouse. The analysis of the LongBench failures is also fairly convincing, and I hope the authors will incorporate these revisions in a subsequent version. Therefore, I will maintain my original score of 3 (weak acceptance).
We sincerely thank the reviewer for your positive feedback. Your comments have been invaluable in improving our work. We will make sure to incorporate these revisions in the subsequent version. We truly appreciate the time and support you have devoted to reviewing our paper!
Response to Reviewer T4ca (Part 1/2)
We sincerely thank you for your thoughtful comments on our paper and for recognizing the effectiveness of RS-Norm and Dual-Mouse, as well as the design of our capability-aware harness. We respond below to your questions about applicability beyond Minecraft and the main failure cases and component contributions on LongBench.
1. Applicability beyond Minecraft (Weakness 1 and Question 1).
To examine the applicability of our proposed RS-Norm and Dual-Mouse beyond Minecraft, we conducted an additional visual-to-action prediction experiment in Raft. Raft includes both GUI operations, such as inventory and crafting interactions, and non-GUI operations, such as first-person view control, with mouse displacement carrying different semantics in the two types of interaction. Using human gameplay trajectories from D2E (Choi et al., 2026) and Qwen2.5-VL-7B-Instruct as the common backbone, we compared four configurations under the same training data, image inputs, and optimization settings: a baseline without either RS-Norm or Dual-Mouse, a model with RS-Norm only, a model with Dual-Mouse only, and a model with both RS-Norm and Dual-Mouse. We evaluated the final checkpoints using greedy decoding on a total of 10,000 test samples balanced between GUI and non-GUI interactions, reporting full-action accuracy (the proportion of samples with a valid generated sequence whose keyboard key set, mouse button states, scroll action, and quantized horizontal and vertical mouse displacements all match the ground-truth action) and mouse-displacement accuracy (the proportion of samples for which both quantized horizontal and vertical mouse displacements match the ground truth).
As shown in the table, RS-Norm and Dual-Mouse each improve both metrics when used individually, and their combination achieves the best results. Compared with the baseline, the combined configuration improves full-action accuracy and mouse-displacement accuracy by 3.6 and 3.5 percentage points, respectively. These results show that distinguishing interaction regimes at the levels of feature normalization and mouse-action representation can improve action prediction, providing evidence for the applicability of both designs to mixed GUI/non-GUI interaction settings beyond Minecraft.