acceptodds
Under review as a conference paper at ICLR 2027

HIPPOVLA: Integrating Short-Term Visual Memory into Vision-Language-Action Policies

Abstract

Vision-language-action (VLA) policies conditioned primarily on current observations face temporal ambiguity: similar observations may require different actions depending on prior interactions. We introduce HIPPOVLA, which incorporates short-term visual history through the backbone's existing intermediate visual injection pathways. Its Short-Term Memory Bank (STMB) organizes sparsely sampled historical features into sets corresponding to selected visual levels and camera views, retrieves historical context through cross-attention, and fuses it with current features through learnable gates. The enhanced representations contribute to action prediction without appending historical observation tokens, keeping the language-model input sequence length unchanged. On CALVIN , HIPPOVLA improves average completed sequence length from 4.085 to 4.619 and the success rate for completing five consecutive instructions from 71.2% to 83.0% over the no-STMB baseline with the same backbone and training protocol. Across the four LIBERO suites, it achieves a mean success rate of 98.5%. In real-world repeated pick-and-place manipulation, success rates for completing two and three full cycles both improve over the no-STMB baseline by 30 percentage points.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Short-Term Visual Memory, Multi-Level Visual Feature Fusion, Memory Retrieval, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DmoXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses ambiguous decisions that arise because current vision-language-action (VLA) policies cannot use historical information. Through the vision-language model’s (VLM) native interface, it injects intermediate historical features, so the VLA policy can use history when predicting actions. Contributions: (1) It proposes a new way to extract information from historical frames. Ablations show that higher-level abstract historical information is more useful for the current decision. (2) It proposes a way to inject historical information that is compatible with the VLM architecture. Its computation is lower than explicit history-token input.

Critical strengths

1.The ablations on injecting historical information are thorough. Authors show that higher-level abstract historical features, once extracted and fused with the current observation, are more useful for the current decision. This is a main reason I find the paper informative and reasonably sound.

2.Authors propose Implicitly injecting historical information into the VLM, rather than appending historical tokens. The module keeps the language-model input length fixed.

3.On CALVIN ABCD, the no-STMB baseline shares the backbone, initialization, action expert, and training budget, so the comparison isolates STMB. Avg.Len rises from 4.085 to 4.619 and SR5 from 71.2% to 83.0% (4.607 ± 0.012 across six evaluation seeds).

Critical weaknesses
  1. The whole paper consisting of quite a lot AI writings / figures, authors should clearly spend more time on polishing them.

  2. LIBERO does not directly show that the proposed method can use historical information, because policies without any history already approach the ceiling on it. The submission's own no-STMB baseline reaches 97.8%. OpenVLA-OFT (Kim et al., 2025), which uses single-step inputs, reaches 97.1%. SimpleVLA-RL (Li et al., ICLR 2026) fine-tunes OpenVLA-OFT with RL using only a single third-view image and the instruction, and reaches 99.1% on average (98.5% on LIBERO-Long), higher than HIPPOVLA's 98.5%. A simulation benchmark built for that ability, such as RMBench (Chen et al., 2026), would be a more appropriate benchmark.

  3. There is only one real-world manipulation task, and it is not compared with other methods. The task is too simple, yet the success rate , although higher, stays far from 100%, which is questionable. Failure cases are not analyzed.

  4. After temporal encoding is added, the success rate drops sharply, and the paper does not analyze why. This weakens the soundness of that claim.

  5. The method is compared with only one baseline, and that baseline is already strong. The paper therefore does not show that the gain holds more generally.

  6. Against history-token input, the paper reports inference cost but not training cost.

  7. Table 1, OpenVLA row. The CALVIN numbers (Avg.Len 3.27) are attributed to Kim et al. (2024), but the OpenVLA paper reports no CALVIN results. The same numbers appear in later works cited here, e.g., HiF-VLA (Table 2) and DeFI (Table 1), presumably from a third-party re-implementation. These sources disagree on the input setting: HiF-VLA lists it as third-view, DeFI as multi-view. Please cite the actual source and its evaluation setting, so readers can judge whether the comparison with HIPPOVLA's two-view setting is fair

Questions for authors
  1. Does the policy use the history at test time? If the history of the trained HIPPOVLA checkpoint is masked at inference (so that the retrieved context is zero) or replaced with frames from another episode, how much of the gain over the no-STMB baseline remains on CALVIN and LIBERO-Long?

  2. After the first cycle, the cube rests on the target at the start and end of every cycle, so the scene alone does not reveal how many cycles remain. Is a trial counted as fully successful only if the robot stops after the third cycle, and how often does each policy continue beyond it? Please also classify the failures of both policies into stage errors and manipulation errors. With 10 trials per method, the differences in Table 3 are not statistically significant (two-sided Fisher exact test, p ≈ 0.35 for two cycles and p ≈ 0.37 for three cycles). Could the authors run more trials?

  3. Why does explicit temporal encoding reduce SR5 by 7.2 and 10.2 points (Table 5)? Appendix A.5 adds fixed sinusoidal vectors to the current features as well (e(t) for absolute and e(0) for relative encoding), and these features are injected into the language model. What is the norm of these vectors relative to the projected features, and does the drop persist with a learned, scaled, or key-only temporal embedding?

  4. Apart from internal ablations, the only trained comparison is the no-STMB baseline. The history-token configuration in Section 4.6 is profiled for cost but not trained (Appendix A.6). Could the authors report the task performance of a trained history-token variant on CALVIN with the same backbone and training budget?

  5. Please report training time and peak GPU memory for HIPPOVLA and the no-STMB baseline, and for the history-token variant if it is trained. At the default five-step history, HIPPOVLA is also slower and uses more memory than history-token input (144.92 vs. 132.74 ms; 18.585 vs. 18.479 GB, Table 6), and all historical images are re-encoded at every call (Appendix A.6). What latency does HIPPOVLA reach with cached historical features, and what efficiency benefit remains at the default setting?

  6. The AI use statement notes that AI tools assisted with identifying relevant literature. Given this attribution issue, could the authors confirm that every baseline number in Tables 1 and 2 has been checked against its original source?

6 comments

Thank you for your detailed questions about whether HIPPOVLA relies on history at inference time, how we define success and stopping behavior and classify failures on the real robot, why explicit temporal encoding reduces performance, how HIPPOVLA compares with a history-token baseline under a matched training budget, and what training and inference costs each approach incurs. We also appreciate your careful scrutiny of the sources and evaluation protocols behind the CALVIN baseline results and your suggestion to evaluate on RMBench. Below, we address your six questions and then discuss how RMBench could complement our evaluation of visual memory.

1. Does HIPPOVLA use history at test time?

Does the policy use the history at test time? If the history of the trained HIPPOVLA checkpoint is masked at inference (so that the retrieved context is zero) or replaced with frames from another episode, how much of the gain over the no-STMB baseline remains on CALVIN and LIBERO-Long?

Thank you for suggesting this test. HIPPOVLA does use historical information at inference time. For each benchmark, we evaluated the same trained HIPPOVLA checkpoint, without retraining, under three inference conditions:

  1. Normal history. The policy receives the original historical observations from the current rollout.
  2. Zero retrieved context. We set the retrieval output to zero at all three fusion levels and for both camera views. The learned gates, normalization, and current-observation pathway remain active.
  3. Cross-episode history. We replace valid historical observations with observations from another rollout of the same task. On CALVIN, the donor rollout executes the same subtask but belongs to a different instruction sequence. We exclude the entire current sequence. On LIBERO-Long, the donor rollout belongs to a different episode of the same task. We preserve the five-frame history window, ten-step sampling interval, camera views, and the current rollout’s history-validity mask.

All three conditions use evaluation seed 42 and the same benchmark protocols: 1,000 instruction sequences on CALVIN ABC→D and 50 trials for each of the ten LIBERO-Long tasks.

Inference conditionCALVIN Avg.LenCALVIN SR5LIBERO-Long success
Normal history4.61983.0%97.8%
Zero retrieved context1.48310.0%0.0%
Cross-episode history0.3320.0%21.4%
Separately trained no-STMB baseline4.08571.2%96.6%

Neither intervention preserves the gain over the no-STMB baseline. The trained policy’s performance drops sharply both when retrieved context is removed and when the supplied history comes from another rollout of the same task. These results show that HIPPOVLA uses its historical retrieval pathway at inference time and is sensitive to whether the history corresponds to the current interaction. The intervention rows measure the sensitivity of a fixed checkpoint. The reference row measures a policy trained from the outset without STMB.

2. How is full real-robot success defined, and what kinds of failures occur?

Is a trial counted as fully successful only if the robot stops after the third cycle, and how often does each policy continue beyond it? Please also classify the failures of both policies into stage errors and manipulation errors. Could the authors run more trials?

Thank you for asking us to clarify the stopping criterion and expand the evaluation. Full success requires three complete pick-and-place cycles, followed by a return to the preset home pose and a stop. We increased the evaluation from 10 to 30 trials per method. Neither policy initiated a fourth cycle: 0/30 trials with STMB and 0/30 without STMB.

We classify each failed trial by the event that terminated it. A stage error occurs if the robot completes a cycle but returns home and stops while another cycle is required, or if it completes the third cycle but fails to return home and stop as required. A manipulation error occurs if the robot fails to complete the current pick-and-place operation and the trial cannot progress.

Policy or statisticComplete cycle 1 and enter cycle 2(Stage errors/Manipulation errors)Complete cycle 2 and enter cycle 3(Stage errors/Manipulation errors)Complete cycle 3, return home, and stop(Stage errors/Manipulation errors)
With STMB30/30 (0/0)23/30 (1/6)16/30 (0/7)
Without STMB29/30 (1/0)14/30 (3/12)8/30 (0/6)
Two-sided Fisher’s exact test pp1.00000.03260.0641

Success counts are cumulative and use all 30 trials as the denominator. Error counts are specific to the stage at which the trial ended. With STMB, the proportion of trials completing the second cycle and entering the third increased from 46.7% to 76.7%, full-task success increased from 26.7% to 53.3%. Most failures in both groups were manipulation errors: 13 of 14 with STMB and 18 of 22 without STMB. We will include the stopping criterion, failure breakdown, and expanded results in the revision.

3. Why does explicit temporal encoding reduce SR5?

Why does explicit temporal encoding reduce SR5 by 7.2 and 10.2 points (Table 5)? What is the norm of these vectors relative to the projected features, and does the drop persist with a learned, scaled, or key-only temporal embedding?

Thank you for pointing out that the effect of explicit temporal encoding on task performance warrants further experimental investigation. One likely contributor to the drop is the magnitude of the unscaled sinusoidal encoding. It is comparable in norm to the projected visual features, so adding it directly may interfere with the visual representations used for history retrieval and gated fusion. In Table 5, HIPPOVLA achieves an SR5 of 83.0% without explicit temporal encoding. Absolute and relative encoding reduce SR5 to 75.8% and 72.8%, corresponding to drops of 7.2 and 10.2 percentage points, respectively.

To examine this explanation, we measured the ratio of the L2 norm of the temporal encoding to that of each projected visual token immediately before addition, using the same 303 observation timesteps from CALVIN D validation demonstrations. For current-frame tokens, the median ratios at L0, L1, and L2 were 1.05, 0.83, and 1.04 for absolute encoding and 1.14, 0.83, and 1.00 for relative encoding. Valid historical frames showed similar ratios. In particular, even at relative time zero, the cosine components of e(0)e(0) equal one, so the current frame also receives an encoding comparable in norm to its visual features.

We then retrained the model with the relative encoding scaled to 0.1e(Δt)0.1e(\Delta t). The median current-frame ratios fell to 0.285, 0.200, and 0.239 across the three levels, and SR5 rose from 72.8% to 80.9%. The gap to the no-encoding configuration thus narrowed from 10.2 to 2.1 percentage points. This remaining but substantially smaller gap supports encoding magnitude as a contributing factor in the original drop for relative encoding.

Temporal encodingMedian norm ratio at L0Median norm ratio at L1Median norm ratio at L2SR5 (%) ↑SR5 change vs. no encoding
No explicit encoding———83.00.0
Absolute sinusoidal (α=1\alpha=1)1.0530.8291.04475.8−7.2
Relative sinusoidal (α=1\alpha=1)1.1440.8331.00172.8−10.2
Relative sinusoidal (α=0.1\alpha=0.1)0.2850.2000.23980.9−2.1

For each current-frame visual token, we computed the norm ratio ∥αe(τ)∥2/∥xproj∥2\lVert\alpha e(\tau)\rVert_2 / \lVert x_{\mathrm{proj}}\rVert_2 immediately before adding the temporal encoding to the projected feature. Each entry is the median for that level over the same 303 observation timesteps from CALVIN D validation demonstrations.

4. CALVIN performance of a trained history-token baseline

Apart from internal ablations, the only trained comparison is the no-STMB baseline. Could the authors report the task performance of a trained history-token variant on CALVIN with the same backbone and training budget?

Thank you for pointing out that the cost measurements in Section 4.6 leave open how well a trained history-token policy performs on CALVIN. To address this question, we trained a history-token baseline that encodes the same five sampled historical observations as visual tokens and appends them to the language-model input. HIPPOVLA uses the same historical observations but retrieves and fuses their features through the backbone’s existing intermediate visual pathways.

We used the same backbone and CALVIN training data and matched the training budget at 100,000 optimizer updates with a global batch size of 32 (8 GPUs × 4 examples per GPU, with one gradient accumulation step). We used ZeRO-3 to fit the history-token model at this batch size. On CALVIN ABC→D, the 100k-step history-token checkpoint achieved 4.470 average completed instructions per sequence and 78.3% success on all five instructions. Under the same evaluation protocol, **HIPPOVLA ** achieved 4.608 and 82.6%, respectively. We will add these results alongside the inference-cost measurements in Section 4.6 so that readers can compare both task performance and computational cost.

MethodHistory integrationTraining setupGlobal batch sizeTraining stepsAvg.Len ↑SR5 (%) ↑
HIPPOVLA (original paper)Intermediate visual fusionZeRO-232100,0004.61983.0
HIPPOVLA (matched training run)Intermediate visual fusionZeRO-332100,0004.60882.6
History-token baselineHistorical visual tokens appended to the LM inputZeRO-332100,0004.47078.3

5. Training cost and inference efficiency with cached historical features

Please report training time and peak GPU memory for HIPPOVLA and the no-STMB baseline, and for the history-token variant if it is trained. What latency does HIPPOVLA reach with cached historical features, and what efficiency benefit remains at the default setting?

Thank you for asking us to quantify the training cost and to test whether caching historical features improves inference efficiency.

Training cost. We reported the three configurations on CALVIN using eight H100 GPUs, a per-GPU batch size of 4, one gradient accumulation step, and a global batch size of 32.

ShardingModelTime to 100k updates (h:mm:ss)Highest sampled GPU memory (GiB/device)
ZeRO-2Without STMB16:32:1561.62
ZeRO-2HIPPOVLA25:18:3879.03
ZeRO-2History-tokenOOM79.17 before OOM
ZeRO-3HIPPOVLA53:27:4767.36
ZeRO-3History-token60:08:0579.18

Cached inference. As described in Section 4.6 and Appendix A.6, the original comparison uses the same trained HIPPOVLA checkpoint and 20 paired observations. The History-token configuration removes STMB and supplies historical images through the VLM’s native multi-image interface. At the default five-frame history, the original results are:

MethodLLM input tokensFLOPs (T)Measured latency (ms)GPU memory (GB)
History-token input857–86516.391132.7418.479
HIPPOVLA without caching197–2056.162144.9218.585

We replayed 1,011 instruction instances, producing 8,360 calls at an eight-step interval. History and caches are cleared at each new instruction. We retain five historical frames sampled at ten-step intervals and report latency over the 1,582 calls with complete history.

Our implementation caches visual features by observation timestep within each instruction. Only historical frames missing from the feature cache undergo image-processor preprocessing, pixel transfer to the GPU, and visual encoding; cache hits directly reuse stored features.

MethodMean latency (ms) ↓P95 latency (ms) ↓GPU memory (GB) ↓
HIPPOVLA without caching150.01151.1918.585
HIPPOVLA with optimized caching140.30146.7818.696
Trained History-token baseline151.10155.4618.479

Cached HIPPOVLA achieves 140.30 ms mean latency and 146.78 ms P95. Its recorded mean latency is 6.5% lower than uncached HIPPOVLA and 7.1% lower than the trained History-token baseline, at the cost of slightly higher inference memory.

At the default setting, HIPPOVLA therefore retains its shorter language-model input and lower FLOPs, offers lower training time and sampled training memory than History-token under ZeRO-3, and achieves lower measured inference latency with caching in the evaluated replay setting.

Reviewer_xRvVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HIPPOVLA, a vision-language-action framework that incorporates short-term visual history through intermediate visual feature injection. The proposed Short-Term Memory Bank (STMB) employs cross-attention and learnable gating to integrate historical information without increasing the language-model input sequence length. Experiments on CALVIN, LIBERO, and real-world robotic manipulation demonstrate improved performance over a memory-free baseline. However, the methodological novelty and experimental evidence are insufficient to fully support the claimed advantages.

Critical strengths
  1. The paper addresses an important problem of temporal ambiguity in vision-language-action policies.
  2. The integration of historical visual features into intermediate backbone layers provides a practical alternative to directly appending historical tokens.
  3. Experiments on both simulation benchmarks and a real robotic platform demonstrate the potential benefits of short-term visual memory.
Critical weaknesses
  1. Limited methodological novelty. The proposed STMB primarily combines standard cross-attention, historical feature aggregation, and learnable gating. These components are well established, and the distinction from existing memory-based VLA approaches, particularly MemoryVLA and ReMem-VLA, is insufficiently substantiated.
  2. Ablation results undermine the proposed multilevel design. Table 4 shows that single-level L2 fusion achieves better performance (85.5% SR5) than the proposed three-level fusion (83.0% SR5). This raises concerns about the necessity and effectiveness of the central multilevel architecture.
  3. Insufficient experimental validation. Although HIPPOVLA improves CALVIN performance, the improvement on LIBERO is only 0.7 percentage points over the baseline. Furthermore, the real-world evaluation includes only 10 trials per method on a single repeated pick-and-place task, limiting the reliability and generalizability of the conclusions.
  4. Unconvincing practical efficiency. While HIPPOVLA preserves the language-model input length, Table 6 shows higher inference latency than history-token input with three and five historical observations. The need to repeatedly encode historical frames introduces substantial computational overhead, weakening the claimed efficiency advantages.
Questions for authors
  1. What is the fundamental methodological novelty of STMB compared with existing memory retrieval and gated fusion mechanisms in MemoryVLA and ReMem-VLA?
  2. Why does single-level L2 fusion outperform the proposed three-level architecture? Would a simpler single-level design provide a better accuracy-efficiency trade-off?
  3. Can the authors provide more comprehensive real-world experiments across different manipulation tasks, objects, and environments, together with statistical significance analysis?
  4. Can the authors compare HIPPOVLA against alternative history integration methods under matched backbone, training budget, history length, and inference latency?
4 comments

Thank you for the detailed comments on methodological novelty, fusion-level design, real-world evaluation, and comparisons with alternative history-integration methods. We have expanded the original real-robot evaluation to 30 trials per method and added four manipulation tasks on the SO-101 robot. We address these questions individually below.

1. Methodological distinction from MemoryVLA and ReMem-VLA

What is the fundamental methodological novelty of STMB compared with existing memory retrieval and gated fusion mechanisms in MemoryVLA and ReMem-VLA?

Thank you for raising this question. Cross-attention and learnable gating are established mechanisms, and we do not claim these operators themselves as novel. Our contribution concerns how short-term visual history is represented and through which architectural interface it enters a VLA policy.

In MemoryVLA, visual encoder features are compressed into perceptual tokens. Visual features and language instructions are processed by the language model, with the output at the EOS position serving as a cognitive token. These two types of tokens retrieve their corresponding historical memories through cross-attention and are fused with the retrieved context through gating. The enhanced perceptual and cognitive representations condition the diffusion action expert, rather than being reinjected through the language model’s intermediate visual interfaces.

ReMem-VLA introduces frame-level and chunk-level memory queries into the frozen VLM’s input to extract information from the current observation. Their output representations undergo exponential moving average (EMA) updates at different temporal frequencies, forming query states that are fed back into the VLM at subsequent timesteps to propagate historical information recurrently. The output memory queries interact with action queries through an additional bidirectional-attention Connector, allowing historical context to inform diffusion-based action prediction.

In contrast, HIPPOVLA organizes recent historical features by selected visual level and camera view. Retrieval and gated fusion enrich the current visual representations, which convey historical context through the backbone’s existing intermediate visual injection pathways. It introduces no additional dedicated memory tokens into the VLM input. The VLM’s output action representations directly condition DiT action decoding, without an additional post-VLM Connector for fusing memory and action representations.

2. Fusion-level design and the accuracy–efficiency trade-off

Ablation results undermine the proposed multilevel design. Table 4 shows that single-level L2 fusion achieves better performance (85.5% SR5) than the proposed three-level fusion (83.0% SR5). This raises concerns about the necessity and effectiveness of the central multilevel architecture.

Thank you for raising this question about the location of history fusion and the accuracy–efficiency trade-off. The central focus of this work is Integrating Short-Term Visual Memory into Vision-Language-Action Policies, incorporating short-term visual memory through the backbone’s existing intermediate visual interfaces. We use three-level fusion as a consistent default across tasks, without assuming that any single level is suitable for every task. The design allows the location of history injection to be selected for a specific task; it does not require three-level fusion in every setting.

The CALVIN results demonstrate the value of a task-specific configuration. L2-only achieves an Avg.Len of 4.665 and an SR5 of 85.5%, compared with 4.619 and 83.0% for three-level fusion. On CALVIN, L2-only therefore offers a simpler configuration that improves task performance while using fewer history-retrieval and fusion modules. Importantly, this configuration restricts only the locations of history fusion; all multilevel visual pathways for the current observation remain intact. One possible explanation is that L2 already provides useful historical context, while additional levels offer no further benefit.

To examine whether this advantage extends to other tasks, we additionally trained L2-only on LIBERO. Its average success rate is 98.3%, slightly below 98.5% for three-level fusion. On LIBERO-Long, L2-only achieves 96.8%, compared with 97.8% for three-level fusion. Thus, the advantage of single-level fusion on CALVIN does not consistently carry over to LIBERO, suggesting that the appropriate fusion configuration may depend on the task. Three-level fusion provides a consistent default without selecting different fusion levels for each benchmark, while L2-only offers a simpler alternative for specific tasks.

For applications that prioritize efficiency, L2-only remains worth considering even when it incurs some task-performance loss. To examine its computational impact, we conducted separate stage-level timing measurements on CALVIN and LIBERO-Long. Within each benchmark, both models used the same 20 paired observation sets, each containing the current observation and five historical frames from both cameras. The models ran sequentially on the same NVIDIA H100 GPU, using BF16, five action-decoding steps, and no historical-feature caching. Each model received 25 warm-up calls, followed by 100 measured calls.

BenchmarkConfigurationModel parameters (B) ↓Historical-memory module parameters (M) ↓Peak GPU memory (GB) ↓
CALVINFull HIPPOVLA9.159302.0418.585
CALVINL2-only8.957100.6818.182
LIBERO-LongFull HIPPOVLA9.159302.0418.588
LIBERO-LongL2-only8.957100.6818.185
Benchmark and modelInput processor and transferCurrent-image encodingHistorical-image encodingHistory retrieval and fusionRemaining VLM computationAction decodingOther unprofiled overheadTotal with stage synchronization
CALVIN, full HIPPOVLA14.8520.8125.432.8649.7324.649.11147.44
CALVIN, L2-only15.0521.8626.461.2851.3325.8610.00151.83
LIBERO-Long, full HIPPOVLA12.8320.3225.012.7948.7224.051.65135.36
LIBERO-Long, L2-only12.7220.1624.871.1647.9824.111.63132.64

L2-only consistently reduces the cost of history retrieval and fusion, from 2.86 ms to 1.28 ms on CALVIN and from 2.79 ms to 1.16 ms on LIBERO-Long, saving approximately 1.6 ms in both cases. However, this stage accounts for only approximately 2% of the full model’s total latency, while visual encoding, language-model computation, and action decoding remain. The resulting savings are small and can be offset by runtime variability. Fusion levels should primarily be selected based on task performance, rather than inferring efficiency from the number of fusion modules alone.

3. More real-world evaluation and statistical analysis

Can the authors provide more comprehensive real-world experiments across different manipulation tasks, objects, and environments, together with statistical significance analysis?

Thank you for suggesting broader real-world evaluation and statistical analysis. In addition to the original repeated pick-and-place experiment on Piper, we have added experiments on an SO101 robot. The evaluation covers four tasks involving different cube colors, destination cups, and manipulation orders: two single-object placement tasks and two sequential three-object placement tasks.

The SO101 policy receives RGB observations from an external camera and a wrist-mounted camera. The demonstration dataset contains 368 trajectories and 248,370 observation frames, recorded at 30 Hz. We split complete trajectories with task stratification into 350 training trajectories (237,110 frames) and 18 validation trajectories (11,260 frames), with no trajectory overlap. HIPPOVLA and the no-STMB baseline use the same training data and action normalization configuration. Both are trained on eight NVIDIA H100 GPUs, using BF16 and ZeRO-2 with CPU optimizer offloading, a per-device batch size of 4, and a gradient accumulation factor of 1, giving a global batch size of 32. We evaluate both methods using their 22,500-update checkpoints. The no-STMB baseline disables historical inputs and memory fusion while retaining the current-observation pathways and action expert.

Each method is evaluated for 15 trials per task, totaling 60 trials per method across the four common tasks. The task instructions, success rates, and statistical comparisons are reported below.

TaskInstructionHIPPOVLANo STMBTwo-sided Fisher’s exact test pp
S1Put the green cube into the right cup.15/15 (100.0%)0/15 (0.0%)1.29×10−81.29\times10^{-8}
S2Put the blue cube into the left cup.15/15 (100.0%)3/15 (20.0%)1.05×10−51.05\times10^{-5}
M1Put the green, gray, and black cubes into the right cup, in that order.12/15 (80.0%)0/15 (0.0%)1.05\times10^{-5}\
M2Put the blue, gray, and green cubes into the right cup, in that order.9/15 (60.0%)0/15 (0.0%)6.997\times10^{-4}\
OverallFour common tasks51/60 (85.0%)3/60 (5.0%)—

HIPPOVLA achieves an overall success rate of 85.0%, compared with 5.0% for the no-STMB baseline, an improvement of 80.0 percentage points. We also analyze manually recorded failures separately for single-object tasks (S) and sequential three-object tasks (M). Each failed trial is counted once. Re-grasping refers to grasping an already placed object again and thereby causing task failure.

Task categoryMethodSuccesses / trialsGrasping failureSlippageRe-graspingStalled after graspSlippage followed by failed placement retryTotal failures
S: single-objectHIPPOVLA30/30000000
S: single-objectNo STMB3/30170100027
M: sequential three-objectHIPPOVLA21/30070119
M: sequential three-objectNo STMB0/3030000030

In single-object tasks, an object may remain visible after placement, so the current observation alone may not distinguish between yet to be manipulated and already completed. The no-STMB baseline exhibits 10 cases in which it re-grasps the object after placement and thereby causes task failure, whereas no such failures are observed in HIPPOVLA’s 30 trials. In sequential three-object tasks, HIPPOVLA completes the instructed tasks in its successful trials, and no failures caused by re-grasping previously placed objects are observed.

We also expanded the original Piper repeated pick-and-place evaluation from 10 to 30 trials per method and report stage-wise success rates, failure categories, and two-sided Fisher’s exact tests below.

Policy or statisticComplete cycle 1 and enter cycle 2(Stage errors/Manipulation errors)Complete cycle 2 and enter cycle 3(Stage errors/Manipulation errors)Complete cycle 3, return home, and stop(Stage errors/Manipulation errors)
With STMB30/30 (0/0)23/30 (1/6)16/30 (0/7)
Without STMB29/30 (1/0)14/30 (3/12)8/30 (0/6)
Two-sided Fisher’s exact test pp1.00000.03260.0641

4. CALVIN performance of a trained history-token baseline

Can the authors compare HIPPOVLA against alternative history integration methods under matched backbone, training budget, history length, and inference latency?

Thank you for suggesting a controlled comparison with alternative history-integration methods. We have added a trained History-token baseline to examine how the location and form of history integration affect performance and inference cost. The history-token baseline encodes the same five sampled historical observations as visual tokens and appends them to the language-model input. HIPPOVLA uses the same historical observations but retrieves and fuses their features through the backbone’s existing intermediate visual pathways.

We used the same backbone and CALVIN training data and matched the training budget at 100,000 optimizer updates with a global batch size of 32 (8 GPUs × 4 examples per GPU, with one gradient accumulation step). We used ZeRO-3 to fit the history-token model at this batch size. On CALVIN ABC→D, the 100k-step history-token checkpoint achieved 4.470 average completed instructions per sequence and 78.3% success on all five instructions. Under the same evaluation protocol, **HIPPOVLA ** achieved 4.608 and 82.6%.

MethodHistory integrationTraining setupGlobal batch sizeTraining stepsAvg.Len ↑SR5 (%) ↑
HIPPOVLA (original paper)Intermediate visual fusionZeRO-232100,0004.61983.0
HIPPOVLA (matched training run)Intermediate visual fusionZeRO-332100,0004.60882.6
History-token baselineHistorical visual tokens appended to the LM inputZeRO-332100,0004.47078.3
Reviewer_Ee9XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HIPPOVLA adds a Short-Term Memory Bank to a vision-language-action backbone. It samples a few recent frames, retrieves their features with cross-attention at selected visual levels and camera views, and gates those features into the current visual representation. The proposed method achieves good performance on both simulation and real tasks.

Critical strengths

The proposed method achieves good performance on both simulation and real tasks. On CALVIN ABC→D, average completed length rises from 4.085 to 4.619 and five-instruction success from 71.2% to 83.0% against a matched model without the bank. LIBERO reaches 98.5%, up 0.7 points. On a real robot, two-cycle and three-cycle success both rise by 30 points, from 5/10 to 8/10 and from 3/10 to 6/10.

Critical weaknesses

None

Questions for authors
  1. On CALVIN tasks, the history is cleared at every new instruction, so the five-instruction gain is better execution inside one subtask, not memory across the chain. LIBERO is already at 97.8% without the bank, and the real result is one repeated pick-and-place with 10 trials. The author does not discuss these senerios specificly.

  2. The real-robot difference is three trials on one repeated pick-and-place. Does it hold when the current view is actually ambiguous, for example under occlusion? Can report the results on more challenge cases? Such as Libero-plus.

  3. The idea, "incorporates short-term visual history through the backbone's existing intermediate" is similar with AVA-VLA (Ava-vla: improving vision-language-action models with active visual attention, CVPR 2025). The author can add discussion on this work.

3 comments

Thank you for the specific comments. For Question 1, we clarify the scope of the CALVIN and LIBERO results and report an expanded evaluation on the original real-world repeated pick-and-place task, with 30 trials per method. For Question 2, we have selected the SO-101 robot arm for a follow-up experiment and will provide a full response once the results are available. For Question 3, we will add a discussion of AVA-VLA to Related Work, acknowledging its use of historical context to guide visual attention and clarifying how it relates to visual history retrieval in HIPPOVLA.

1. Interpretation of the CALVIN, LIBERO, and real-robot results

On CALVIN tasks, the history is cleared at every new instruction, so the five-instruction gain is better execution inside one subtask, not memory across the chain. LIBERO is already at 97.8% without the bank, and the real result is one repeated pick-and-place with 10 trials. The author does not discuss these senerios specificly.

CALVIN. Thank you for highlighting the distinction between using short-term visual history within a subtask and retaining memory across subtasks. As described in Section 4.2, HIPPOVLA resets its visual history at the start of each new instruction and can therefore access only the visual history accumulated within the current subtask, while the effects of earlier actions on the environment persist into subsequent subtasks. The CALVIN results thus reflect improved execution of consecutive instructions when short-term visual history is available within each subtask. They do not establish that the policy retains memory across instructions. Because the environment state carries over when instructions change, later subtasks may still depend on the physical state established by earlier actions. The evaluation therefore still measures the policy’s ability to execute a sequence of tasks.

LIBERO. Thank you for noting the high success rate of the no-STMB baseline. We agree that the overall gain is modest: mean success across the four suites increases from 97.8% to 98.5%. Given our focus on how short-term visual history can support long-horizon manipulation, we will expand our discussion of LIBERO-Long. With the same backbone and training protocol, adding STMB raises success on this suite from 96.6% to 97.8%, a gain of 1.2 percentage points. This result shows that HIPPOVLA achieves a higher success rate on the long-horizon manipulation tasks evaluated in this suite.

Real-robot evaluation. Thank you for pointing out that the original evaluation covered a single repeated pick-and-place task with only 10 trials per method. We have expanded the evaluation to 30 trials per method on the same task. With STMB, the proportion of trials in which the robot completed the second cycle and entered the third increased from 46.7% (14/30) to 76.7% (23/30). Full-task success required the robot to complete the third cycle, return to its home pose, and stop. Under this criterion, success increased from 26.7% (8/30) to 53.3% (16/30). These additional trials provide further evidence about performance on this repeated manipulation task. Generalization to other real-world settings remains open. We will update the results in the revised manuscript and discuss the observed gains and their limitations in the corresponding experimental sections.

Policy or statisticComplete cycle 1 and enter cycle 2(Stage errors/Manipulation errors)Complete cycle 2 and enter cycle 3(Stage errors/Manipulation errors)Complete cycle 3, return home, and stop(Stage errors/Manipulation errors)
With STMB30/30 (0/0)23/30 (1/6)16/30 (0/7)
Without STMB29/30 (1/0)14/30 (3/12)8/30 (0/6)
Two-sided Fisher’s exact test pp1.00000.03260.0641

2. Evaluation on LIBERO-Plus

Can report the results on more challenge cases? Such as Libero-plus.

Thank you for suggesting an evaluation under more challenging conditions. We have added experiments on LIBERO-Plus, which introduces perturbations to camera viewpoints, robot initial states, language instructions, lighting, background textures, sensor noise, and object layouts.

Evaluation protocol. We directly evaluate the original LIBERO-trained HIPPOVLA checkpoint, without additional training or fine-tuning on LIBERO-Plus. A single checkpoint is used across all four suites, corresponding to the “one policy for all 4 suites” setting in AVA-VLA. We use evaluation seed 42 and run one rollout per perturbation task variant, yielding 10,030 rollouts: 2,402 on Spatial, 2,518 on Object, 2,591 on Goal, and 2,519 on Long. HIPPOVLA uses five historical observations sampled at ten-step intervals.

Following the aggregation described in AVA-VLA’s Table 6, we first compute the success rate for each perturbation category within each suite, then average equally across the four suites. The final Average column is the arithmetic mean of the seven category-level scores. The following table compares HIPPOVLA with the published results in AVA-VLA under the single-policy setting. All values are success rates (%).

MethodCameraRobotLanguageLightBackgroundNoiseLayoutAverage
OpenVLA-OFT, reported in AVA-VLA55.621.781.092.791.078.668.767.9
AVA-VLA, reported results55.525.985.695.588.978.074.170.1
HIPPOVLA, our evaluation44.258.490.398.693.977.678.377.3

HIPPOVLA achieves an average success rate of 77.3%, compared with the reported 70.1% of AVA-VLA under the same single-policy grouping. HIPPOVLA obtains higher scores in five categories: robot initial states, language, lighting, background, and layout. However, its camera-viewpoint score is lower (44.2% versus 55.5%), and its sensor-noise score is slightly lower (77.6% versus 78.0%).

3. Discussion of AVA-VLA

The idea, "incorporates short-term visual history through the backbone's existing intermediate" is similar with AVA-VLA (Ava-vla: improving vision-language-action models with active visual attention, CVPR 2025). The author can add discussion on this work.

Thank you for pointing out the connection to AVA-VLA. We agree that both methods use historical context to guide visual processing for action prediction. AVA-VLA projects the previous timestep’s action-related hidden states into a recurrent state. This state initializes the current action-token embeddings and, together with instruction-conditioned image features, determines an importance weight for each current visual token. These weights modulate attention to visual tokens across the language-model layers.

HIPPOVLA approaches the problem from the perspective of visual retrieval. At each selected visual level and camera view, STMB uses features from the most recently sampled historical frame as queries to retrieve context from the corresponding historical feature set. A learned gate fuses the retrieved context with current visual features, and the enriched representations enter the language model through the backbone’s existing intermediate visual injection pathways. We will cite AVA-VLA in Related Work and discuss how its use of history to guide attention over the current image relates to our use of past visual content to enrich the current representation.

2*. Expanded real-robot evaluation and a qualitative occlusion example

The real-robot difference is three trials on one repeated pick-and-place. Does it hold when the current view is actually ambiguous, for example under occlusion?

Thank you for raising concerns about the limited number of real-robot trials and asking about behavior under ambiguous observations. We have expanded the Piper evaluation, added experiments on SO101, and included a qualitative example with temporary occlusion.

We increased the original Piper repeated pick-and-place evaluation from 10 to 30 trials per method. Both methods are evaluated under the same set of 30 initial conditions. Full-task success requires completing three pick-and-place cycles, returning to the predefined home pose, and stopping without starting a fourth cycle. The updated results are as follows.

Stage criterionHIPPOVLA: successes (stage/manipulation errors)No STMB: successes (stage/manipulation errors)Two-sided Fisher’s exact test, pp
Complete cycle 1 and enter cycle 230/30 (0/0)29/30 (1/0)1.0000
Complete cycle 2 and enter cycle 323/30 (1/6)14/30 (3/12)0.0326
Complete cycle 3, return home, and stop16/30 (0/7)8/30 (0/6)0.0641

HIPPOVLA improves Stage 2 success from 46.7% to 76.7% and full-task success from 26.7% to 53.3%. Neither method starts a fourth cycle.

We additionally evaluate an SO101 robot on four tasks involving different cube colors, destination cups, and placement orders: two single-object tasks and two sequential three-object tasks. The policy receives RGB observations from an external camera and a wrist-mounted camera. The demonstration dataset contains 368 trajectories and 248,370 frames recorded at 30 Hz. Using seed 42, we split complete trajectories with task stratification into 350 training trajectories (237,110 frames) and 18 validation trajectories (11,260 frames), with no trajectory overlap.

Both methods use the same training data and action normalization configuration. Training uses eight NVIDIA H100 GPUs, BF16, and ZeRO-2 with CPU optimizer offloading, with a per-device batch size of 4 and a gradient accumulation factor of 1, giving a global batch size of 32. Both methods are evaluated using their 22,500-update checkpoints. The no-STMB baseline disables historical inputs and memory fusion while retaining the current-observation pathways and action expert. We conduct 15 trials per task and method, totaling 60 trials per method. SO101 success is assessed by completion of the instructed placement task, including the required order for multi-object tasks; returning home is not an additional success requirement.

TaskInstructionHIPPOVLANo STMBTwo-sided Fisher’s exact test, pp
S1Put the green cube into the right cup.15/15 (100.0%)0/15 (0.0%)1.29×10−81.29\times10^{-8}
S2Put the blue cube into the left cup.15/15 (100.0%)3/15 (20.0%)1.05×10−51.05\times10^{-5}
M1Put the green, gray, and black cubes into the right cup, in that order.12/15 (80.0%)0/15 (0.0%)1.05×10−51.05\times10^{-5}
M2Put the blue, gray, and green cubes into the right cup, in that order.9/15 (60.0%)0/15 (0.0%)7.00×10−47.00\times10^{-4}
OverallFour tasks51/60 (85.0%)3/60 (5.0%)—

In the single-object tasks, the no-STMB baseline exhibits 10 failures involving re-grasping an already placed object, whereas no such failures are observed in HIPPOVLA’s 30 trials. The task results and failure analysis are provided in Appendix A.5.

To observe behavior under occlusion, we additionally provide a qualitative SO101 rollout with the instruction Put the blue cube into the left cup. A sheet of white paper temporarily occludes part of the workspace. After the paper is removed, HIPPOVLA grasps the blue cube and places it into the left cup. Appendix A.5, Figure 4 presents 16 timestamped frames of this execution. This example illustrates successful task completion following temporary occlusion.

Comments on the paper

2 comments

Dear All Reviewers,

As the rebuttal discussion period is drawing to a close, we kindly encourage you to review the authors' responses to your previous concerns and provide any further feedback or clarification as appropriate.

Thank you for your time and cooperation.

Best regards,

Your Area Chairs

We thank all reviewers for their constructive feedback. We have incorporated the results of the discussion into the revised paper. We also thank Reviewer DmoX for the follow-up and for noting that the additional experiments addressed most of the concerns raised in the review.

Strengths noted by the reviewers

  • Ee9X and DmoX highlighted the gains on CALVIN against the matched no-STMB baseline. Average completed sequence length improved from 4.085 to 4.619, and five-instruction success increased from 71.2% to 83.0%.
  • xRvV and DmoX noted that integrating history through the backbone’s existing visual pathways preserves the language model’s input length. xRvV also highlighted the importance of temporal ambiguity in VLA policies, and DmoX highlighted the fusion-level ablations.

Main concerns and our responses

  1. Use of history (Ee9X, DmoX). We clarified that CALVIN resets history at each new instruction. Removing the retrieved context or replacing history with observations from another rollout substantially reduced performance on CALVIN and LIBERO-Long (Appendix A.7).
  2. Method and fusion design (Ee9X, xRvV). We clarified how HIPPOVLA differs from AVA-VLA, MemoryVLA, and ReMem-VLA. The additional ablations show that the best fusion configuration depends on the task.
  3. Real-world evaluation and robustness (Ee9X, xRvV, DmoX). We expanded the Piper evaluation to 30 trials per method, added four SO101 tasks and an example with temporary occlusion, and reported a separate LIBERO-Plus evaluation.
  4. History-token comparison and efficiency (xRvV, DmoX). HIPPOVLA performed better than a separately trained History-token baseline under matched training conditions. We also reported training costs and found that caching reduced mean inference latency on CALVIN replay (Appendix A.9).
  5. Temporal encoding (DmoX). We measured the encoding norms and found that scaling the relative encoding improved CALVIN SR5 from 72.8% to 80.9% (Appendix A.8).
  6. Baseline sources and RMBench (DmoX). We checked the baseline results against their sources, updated the OpenVLA comparison, and added RMBench as a direction for future evaluation.

We thank the reviewers and the Area Chair again for their time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.