acceptodds
Withdrawn from ICLR 2027

HIVE: History-Initialized Visuomotor Evolution with a Compact Visual Bottleneck for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World-action models jointly predict future actions and visual outcomes, but noise-initialized generation can require costly iterative refinement. While some policies initialize generation from semantically informative priors, they predict actions alone without jointly forecasting visual outcomes. We present HIVE (History-Initialized Visuomotor Evolution), a flow-matching framework that uses historical actions and observed visual outcomes as the starting state for joint future prediction. This structured prior starts closer to the target than Gaussian noise, enabling more accurate generation with fewer integration steps while retaining the predictive visual supervision of world-action models. To keep joint modeling efficient, we introduce an Action-Centric Visual Bottleneck (ACVB), a lightweight 28.5M-parameter compressor that reduces 196 frozen DINOv3 patch tokens to very few compact, manipulation-informative tokens per frame. Dense-feature and policy supervision preserve local interaction information in a shared representation for history and future targets. Causal temporal attention captures action–outcome dependencies. Across 18 RoboTwin tasks, six-step HIVE improves mean success by 12.06 percentage points over its action-only prediction variant, and by 13.84 and 9.34 points over six- and ten-step noise-initialized variants, respectively. Compared with the ten-step variant, it reduces mean inference latency by 56.3% per eight-action chunk. Using ACVB instead of global CLS features improves mean success by 17.89 points over HIVE-CLS. It achieves 71% mean success on five real-world tasks, supporting effective and efficient history-initialized visuomotor prediction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, flow matching, robotic manipulation, visual representation, robot policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KTQfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper combines the world-action model paradigm with informed initialization for flow-matching policies. Instead of generating from Gaussian noise, the proposed HIVE framework initializes the flow from the interaction history: historical actions interleaved with their observed visual outcomes are transported toward the joint future.

  2. Specifically, the authors design an Action-Centric Visual Bottleneck (ACVB) to compress frozen DINOv3 patch features into a few compact tokens per frame, and use a causal flow-matching Transformer to learn the transition from historical video features and actions to future actions and visual states.

  3. The authors validate the method on 18 RoboTwin simulation tasks and a self-built real-robot benchmark of five tasks. HIVE achieves 57.06% mean success in simulation and 71% on the real-world tasks, improving over prior methods such as VITA (28.83%) and A2A (35.33%), as well as over noise-initialized variants.

Critical strengths

S1. An effective combination of the world-action-model paradigm with informed initialization for flow-matching policies.

Generation starts from the interaction history (past actions paired with their observed visual outcomes) instead of Gaussian noise, and the idea is realized through a coherent design: an ACVB module that compresses frozen DINOv3 patch features into a few compact tokens per frame, and a causal Transformer that preserves the action→outcome ordering. The paper substantiates each design choice with controlled ablations (visual supervision: +12.06; history vs. noise initialization at a matched 6-step budget: +13.84; causal vs. bidirectional attention: +10.23).

S2. Strong empirical gains at a favorable parameter and latency budget, in simulation and on real robots.

HIVE reaches 57.06% mean success on 18 RoboTwin tasks, improving over VITA (28.83%) and A2A (35.33%), and averages 71% on five real-world tasks (A2A: 32%). The HIVE-small variant uses fewer trainable parameters than both VITA and A2A (22.69M vs 31.86M/34.75M) while still outperforming them (52.11%).

Critical weaknesses

W1. Limited methodological novelty.

The main contribution is a combination of two existing ideas—world-action models (joint action and visual prediction) and informed initialization of flow-matching policies (A2A)—rather than a substantial methodological advance over either field. As a result, the paper reads more like a moderate improvement over the WAM/A2A baselines than a conceptual contribution. The authors would need to better articulate what the joint history-initialized transport fundamentally enables that neither paradigm alone provides, in order to raise the contribution above an effective combination.

W2. The attribution of the main gain to history initialization is not precisely investigated. The key comparison, HIVE vs. HIVE-N6/N10, differs in more than the initialization: HIVE places history inside the transported sequence under causal attention, while HIVE-N conditions on historical actions and visual tokens via adaLN/cross-attention. The paper does not provide further controlled experiments that precisely isolate the effect of initialization from that of the history-conditioning mechanism (e.g., changing only the source from noise to history while holding the conditioning pathway fixed), so the +13.84-point gain cannot be cleanly attributed to history initialization itself.

Questions for authors

Q1. Have you considered validating the history-initialized paradigm on larger, pretrained world-action models instead of training from scratch at the current scale? If history initialization transfers to pretrained WAMs, it would substantially strengthen the claim that this is a general paradigm.

Q2. Have you ablated the individual loss terms and their weights? The current ablations remove entire design components (visual supervision as a group, the attention mask, the representation), but it is unclear which of the loss terms in Eq. (7) (flow matching, action reconstruction, visual alignment, target-side compression, inference consistency) actually drive the gains, and how sensitive the results are to the chosen weights (0.1 vs. 1.0).

Reviewer_V5MDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HIVE proposes generating future actions and visual representations by continuously transforming observed interaction history. HIVE initializes flow matching with eight historical action–outcome pairs. Its Action-Centric Visual Bottleneck (ACVB) compresses each frame’s 196 frozen DINOv3 patch features into three learned tokens, supervised by dense visual features and policy objectives. A causal Transformer jointly predicts the next eight actions and visual outcomes. The intended contribution is a compact, informative starting state that reduces refinement cost while preserving predictive visual supervision.

Critical strengths
  1. HIVE achieves 57.06% success across 18 simulated tasks and 71% across five physical tasks, supporting practical usefulness across robot configurations.

  2. Three-token visual compression retains strong control performance, and predictive supervision improves both ACVB and frozen-CLS variants.

Critical weaknesses
  1. As described, the causal mask prevents every predicted action from accessing the latest observation V_t. This undermines the current-feedback rationale unless an additional information path exists.

  2. The ablations confound initialization with conditioning architecture and predictive supervision with compressor training. Consequently, the reported gains do not cleanly establish the proposed mechanisms.

Questions for authors
  1. Clarification: How does V_t influence predicted actions under the specified causal mask? Is an additional conditioning path omitted from the description?

  2. Evidence request: Can you isolate initialization while holding conditioning and representation fixed, and isolate predictive supervision while retaining compressor supervision?

Reviewer_BbCwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HIVE is a flow-matching visuomotor policy that builds on Action-to-Action (A2A) flow matching and has the policy predict future actions and future visual states together. Most flow policies start from Gaussian noise. HIVE instead starts from the recent history: the flow’s starting point is the past 8 executed actions, interleaved with compact visual tokens of the frame observed after each action. The endpoint is the next 8 actions and their visual tokens, so both sequences have the same length. The model has no separate conditioning input and uses causal attention over this interleaved sequence.

The visual tokens come from an Action-Centric Visual Bottleneck (ACVB). The ACVB is a 4-layer Transformer on top of frozen DINOv3 patch features. It takes the outputs at the last three patch positions as 3 tokens per frame and is trained with a patch-coverage alignment loss. History and future share the same ACVB.

Results: on 18 RoboTwin tasks (12 single-arm Franka, 6 dual-arm ARX-X5), HIVE reaches an average success rate of 57.06%, against 35.33% for the strongest external baseline, A2A. The ablations attribute the gains to three parts:

visual supervision (+12.06) history initialization (+13.84 over noise initialization at the same 6 steps, with lower latency than the 10-step noise variant) causal attention (+10.23)

On 5 real-robot tasks, with 20 trials per task, HIVE’s average success rate is 71%.

Critical strengths
  • History-initialized flow generation is well motivated and backed by a direct controlled comparison. The robot’s recent actions and their visual outcomes are a natural starting point for predicting what comes next. The paper puts the visual history into the state being transported, rather than using it only as conditioning. It also isolates the effect of initialization with noise-initialized variants on the same backbone (HIVE-N6/N10): at the same 6 steps, HIVE is 13.84 points higher, and its latency is lower than the 10-step noise variant. I weight this point most heavily in my assessment because it supports the paper’s central claim.

  • The experiments are broad and the component ablations are complete. The paper covers 18 simulated tasks and 5 real-robot tasks. It ablates visual supervision, initialization scheme, causal masking, token count and visual backbone separately, and it reports per-task results. The frozen-CLS-feature pair (HIVE-CLS-VS 28.89% → HIVE-CLS 39.17%) shows fairly cleanly that predicting future frames helps as a supervision signal. The smaller HIVE-small (22.69M trainable parameters) still reaches 52.11%, which suggests the gains do not come only from model scale.

Critical weaknesses
  1. The causal mask implementation needs clarification. Appendix A (describing HIVE-CLS, which uses the same causal attention) states that each token attends only to itself and preceding positions. Section 4.3 states that the flow starts directly from the history, with no separate conditioning input. Read literally, the first predicted action â(t) would depend only on a(t−8), and the current frame V(t) at the end of the sequence could not influence any predicted action. This seems hard to reconcile with the paper's strong results. Since the causal-attention ablation (57.06 vs. 46.83) relies on this mechanism, I ask the authors to clarify the exact attention mask used in training and inference, and to explain how V(t) influences â(t).

  2. The visual-supervision ablation is confounded. HIVE-VS removes all visual losses, including the target-side ACVB alignment loss. The +12.06 gain therefore also reflects the removal of ACVB's own visual supervision.

  3. The comparison with A2A is confounded. The 21.72-point gap combines three changes: the visual backbone (frozen DINOv3 vs. trainable ResNet-18), the number of trainable parameters (96.31M vs. 34.75M) and the architecture.

  4. Novelty is limited. History-initialized flow, H = F = 8, 6-step Euler integration and the FM/AE/IC losses all come from A2A. A2A's frames-to-frames extension already performs flow from past frames to future frames and lists integrating predicted video into the policy as future work. The interleaving scheme, with each action followed by its outcome under a causal mask, is also standard: it already appears in Trajectory Transformer, GR-1, Diffusion Forcing and WorldVLA. Diffusion Forcing additionally refines all positions of such a causal sequence in parallel. HIVE's main novelty therefore lies in merging A2A's two branches into a single unconditioned interleaved sequence.

  5. ACVB's key design choices are not justified. ACVB is built from common components. It uses the outputs at the last three patch positions as tokens instead of dedicated learnable queries, and the paper neither explains nor ablates this choice. The alignment loss ℓ_align has the same form as a per-frame spherical k-means objective with three centroids. The "object / arm / background" partition in Figure 7 may therefore largely reflect this loss rather than the readout design.

  6. Stochasticity and multimodality are not discussed. A2A adds small noise to the history to keep some multimodality, and it notes that without this noise the policy collapses to a single mode. HIVE appears to start from noise-free history, which makes the whole ODE a deterministic map. The paper does not discuss multimodality and does not include a direct regression baseline with the same architecture.

  7. There are no external WAM baselines. The paper positions itself as an improvement over noise-initialized world-action models. However, all external baselines (DP, VITA, Patch Policy, A2A) predict only actions, and FM-N6/N10 are also action-only. The only joint-prediction comparisons are the authors' own variants. It is therefore unclear whether HIVE outperforms published WAMs such as UVA or UWM.

  8. Typo. Fig. 1(c) labels the module "ABCV", whereas Fig. 2 and the main text use "ACVB" (Action-Centric Visual Bottleneck); please correct this typo for consistency.

Questions for authors

Please see the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.