- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses ambiguous decisions that arise because current vision-language-action (VLA) policies cannot use historical information. Through the vision-language model’s (VLM) native interface, it injects intermediate historical features, so the VLA policy can use history when predicting actions. Contributions: (1) It proposes a new way to extract information from historical frames. Ablations show that higher-level abstract historical information is more useful for the current decision. (2) It proposes a way to inject historical information that is compatible with the VLM architecture. Its computation is lower than explicit history-token input.
Critical strengths
1.The ablations on injecting historical information are thorough. Authors show that higher-level abstract historical features, once extracted and fused with the current observation, are more useful for the current decision. This is a main reason I find the paper informative and reasonably sound.
2.Authors propose Implicitly injecting historical information into the VLM, rather than appending historical tokens. The module keeps the language-model input length fixed.
3.On CALVIN ABCD, the no-STMB baseline shares the backbone, initialization, action expert, and training budget, so the comparison isolates STMB. Avg.Len rises from 4.085 to 4.619 and SR5 from 71.2% to 83.0% (4.607 ± 0.012 across six evaluation seeds).
Critical weaknesses
-
The whole paper consisting of quite a lot AI writings / figures, authors should clearly spend more time on polishing them.
-
LIBERO does not directly show that the proposed method can use historical information, because policies without any history already approach the ceiling on it. The submission's own no-STMB baseline reaches 97.8%. OpenVLA-OFT (Kim et al., 2025), which uses single-step inputs, reaches 97.1%. SimpleVLA-RL (Li et al., ICLR 2026) fine-tunes OpenVLA-OFT with RL using only a single third-view image and the instruction, and reaches 99.1% on average (98.5% on LIBERO-Long), higher than HIPPOVLA's 98.5%. A simulation benchmark built for that ability, such as RMBench (Chen et al., 2026), would be a more appropriate benchmark.
-
There is only one real-world manipulation task, and it is not compared with other methods. The task is too simple, yet the success rate , although higher, stays far from 100%, which is questionable. Failure cases are not analyzed.
-
After temporal encoding is added, the success rate drops sharply, and the paper does not analyze why. This weakens the soundness of that claim.
-
The method is compared with only one baseline, and that baseline is already strong. The paper therefore does not show that the gain holds more generally.
-
Against history-token input, the paper reports inference cost but not training cost.
-
Table 1, OpenVLA row. The CALVIN numbers (Avg.Len 3.27) are attributed to Kim et al. (2024), but the OpenVLA paper reports no CALVIN results. The same numbers appear in later works cited here, e.g., HiF-VLA (Table 2) and DeFI (Table 1), presumably from a third-party re-implementation. These sources disagree on the input setting: HiF-VLA lists it as third-view, DeFI as multi-view. Please cite the actual source and its evaluation setting, so readers can judge whether the comparison with HIPPOVLA's two-view setting is fair
Questions for authors
-
Does the policy use the history at test time? If the history of the trained HIPPOVLA checkpoint is masked at inference (so that the retrieved context is zero) or replaced with frames from another episode, how much of the gain over the no-STMB baseline remains on CALVIN and LIBERO-Long?
-
After the first cycle, the cube rests on the target at the start and end of every cycle, so the scene alone does not reveal how many cycles remain. Is a trial counted as fully successful only if the robot stops after the third cycle, and how often does each policy continue beyond it? Please also classify the failures of both policies into stage errors and manipulation errors. With 10 trials per method, the differences in Table 3 are not statistically significant (two-sided Fisher exact test, p ≈ 0.35 for two cycles and p ≈ 0.37 for three cycles). Could the authors run more trials?
-
Why does explicit temporal encoding reduce SR5 by 7.2 and 10.2 points (Table 5)? Appendix A.5 adds fixed sinusoidal vectors to the current features as well (e(t) for absolute and e(0) for relative encoding), and these features are injected into the language model. What is the norm of these vectors relative to the projected features, and does the drop persist with a learned, scaled, or key-only temporal embedding?
-
Apart from internal ablations, the only trained comparison is the no-STMB baseline. The history-token configuration in Section 4.6 is profiled for cost but not trained (Appendix A.6). Could the authors report the task performance of a trained history-token variant on CALVIN with the same backbone and training budget?
-
Please report training time and peak GPU memory for HIPPOVLA and the no-STMB baseline, and for the history-token variant if it is trained. At the default five-step history, HIPPOVLA is also slower and uses more memory than history-token input (144.92 vs. 132.74 ms; 18.585 vs. 18.479 GB, Table 6), and all historical images are re-encoded at every call (Appendix A.6). What latency does HIPPOVLA reach with cached historical features, and what efficiency benefit remains at the default setting?
-
The AI use statement notes that AI tools assisted with identifying relevant literature. Given this attribution issue, could the authors confirm that every baseline number in Tables 1 and 2 has been checked against its original source?
6 comments
2. How is full real-robot success defined, and what kinds of failures occur?
Is a trial counted as fully successful only if the robot stops after the third cycle, and how often does each policy continue beyond it? Please also classify the failures of both policies into stage errors and manipulation errors. Could the authors run more trials?
Thank you for asking us to clarify the stopping criterion and expand the evaluation. Full success requires three complete pick-and-place cycles, followed by a return to the preset home pose and a stop. We increased the evaluation from 10 to 30 trials per method. Neither policy initiated a fourth cycle: 0/30 trials with STMB and 0/30 without STMB.
We classify each failed trial by the event that terminated it. A stage error occurs if the robot completes a cycle but returns home and stops while another cycle is required, or if it completes the third cycle but fails to return home and stop as required. A manipulation error occurs if the robot fails to complete the current pick-and-place operation and the trial cannot progress.
| Policy or statistic | Complete cycle 1 and enter cycle 2(Stage errors/Manipulation errors) | Complete cycle 2 and enter cycle 3(Stage errors/Manipulation errors) | Complete cycle 3, return home, and stop(Stage errors/Manipulation errors) |
|---|---|---|---|
| With STMB | 30/30 (0/0) | 23/30 (1/6) | 16/30 (0/7) |
| Without STMB | 29/30 (1/0) | 14/30 (3/12) | 8/30 (0/6) |
| Two-sided Fisher’s exact test | 1.0000 | 0.0326 | 0.0641 |
Success counts are cumulative and use all 30 trials as the denominator. Error counts are specific to the stage at which the trial ended. With STMB, the proportion of trials completing the second cycle and entering the third increased from 46.7% to 76.7%, full-task success increased from 26.7% to 53.3%. Most failures in both groups were manipulation errors: 13 of 14 with STMB and 18 of 22 without STMB. We will include the stopping criterion, failure breakdown, and expanded results in the revision.
3. Why does explicit temporal encoding reduce SR5?
Why does explicit temporal encoding reduce SR5 by 7.2 and 10.2 points (Table 5)? What is the norm of these vectors relative to the projected features, and does the drop persist with a learned, scaled, or key-only temporal embedding?
Thank you for pointing out that the effect of explicit temporal encoding on task performance warrants further experimental investigation. One likely contributor to the drop is the magnitude of the unscaled sinusoidal encoding. It is comparable in norm to the projected visual features, so adding it directly may interfere with the visual representations used for history retrieval and gated fusion. In Table 5, HIPPOVLA achieves an SR5 of 83.0% without explicit temporal encoding. Absolute and relative encoding reduce SR5 to 75.8% and 72.8%, corresponding to drops of 7.2 and 10.2 percentage points, respectively.
To examine this explanation, we measured the ratio of the L2 norm of the temporal encoding to that of each projected visual token immediately before addition, using the same 303 observation timesteps from CALVIN D validation demonstrations. For current-frame tokens, the median ratios at L0, L1, and L2 were 1.05, 0.83, and 1.04 for absolute encoding and 1.14, 0.83, and 1.00 for relative encoding. Valid historical frames showed similar ratios. In particular, even at relative time zero, the cosine components of equal one, so the current frame also receives an encoding comparable in norm to its visual features.
We then retrained the model with the relative encoding scaled to . The median current-frame ratios fell to 0.285, 0.200, and 0.239 across the three levels, and SR5 rose from 72.8% to 80.9%. The gap to the no-encoding configuration thus narrowed from 10.2 to 2.1 percentage points. This remaining but substantially smaller gap supports encoding magnitude as a contributing factor in the original drop for relative encoding.
| Temporal encoding | Median norm ratio at L0 | Median norm ratio at L1 | Median norm ratio at L2 | SR5 (%) ↑ | SR5 change vs. no encoding |
|---|---|---|---|---|---|
| No explicit encoding | — | — | — | 83.0 | 0.0 |
| Absolute sinusoidal () | 1.053 | 0.829 | 1.044 | 75.8 | −7.2 |
| Relative sinusoidal () | 1.144 | 0.833 | 1.001 | 72.8 | −10.2 |
| Relative sinusoidal () | 0.285 | 0.200 | 0.239 | 80.9 | −2.1 |
For each current-frame visual token, we computed the norm ratio immediately before adding the temporal encoding to the projected feature. Each entry is the median for that level over the same 303 observation timesteps from CALVIN D validation demonstrations.
4. CALVIN performance of a trained history-token baseline
Apart from internal ablations, the only trained comparison is the no-STMB baseline. Could the authors report the task performance of a trained history-token variant on CALVIN with the same backbone and training budget?
Thank you for pointing out that the cost measurements in Section 4.6 leave open how well a trained history-token policy performs on CALVIN. To address this question, we trained a history-token baseline that encodes the same five sampled historical observations as visual tokens and appends them to the language-model input. HIPPOVLA uses the same historical observations but retrieves and fuses their features through the backbone’s existing intermediate visual pathways.
We used the same backbone and CALVIN training data and matched the training budget at 100,000 optimizer updates with a global batch size of 32 (8 GPUs × 4 examples per GPU, with one gradient accumulation step). We used ZeRO-3 to fit the history-token model at this batch size. On CALVIN ABC→D, the 100k-step history-token checkpoint achieved 4.470 average completed instructions per sequence and 78.3% success on all five instructions. Under the same evaluation protocol, **HIPPOVLA ** achieved 4.608 and 82.6%, respectively. We will add these results alongside the inference-cost measurements in Section 4.6 so that readers can compare both task performance and computational cost.
| Method | History integration | Training setup | Global batch size | Training steps | Avg.Len ↑ | SR5 (%) ↑ |
|---|---|---|---|---|---|---|
| HIPPOVLA (original paper) | Intermediate visual fusion | ZeRO-2 | 32 | 100,000 | 4.619 | 83.0 |
| HIPPOVLA (matched training run) | Intermediate visual fusion | ZeRO-3 | 32 | 100,000 | 4.608 | 82.6 |
| History-token baseline | Historical visual tokens appended to the LM input | ZeRO-3 | 32 | 100,000 | 4.470 | 78.3 |
5. Training cost and inference efficiency with cached historical features
Please report training time and peak GPU memory for HIPPOVLA and the no-STMB baseline, and for the history-token variant if it is trained. What latency does HIPPOVLA reach with cached historical features, and what efficiency benefit remains at the default setting?
Thank you for asking us to quantify the training cost and to test whether caching historical features improves inference efficiency.
Training cost. We reported the three configurations on CALVIN using eight H100 GPUs, a per-GPU batch size of 4, one gradient accumulation step, and a global batch size of 32.
| Sharding | Model | Time to 100k updates (h:mm:ss) | Highest sampled GPU memory (GiB/device) |
|---|---|---|---|
| ZeRO-2 | Without STMB | 16:32:15 | 61.62 |
| ZeRO-2 | HIPPOVLA | 25:18:38 | 79.03 |
| ZeRO-2 | History-token | OOM | 79.17 before OOM |
| ZeRO-3 | HIPPOVLA | 53:27:47 | 67.36 |
| ZeRO-3 | History-token | 60:08:05 | 79.18 |
Cached inference. As described in Section 4.6 and Appendix A.6, the original comparison uses the same trained HIPPOVLA checkpoint and 20 paired observations. The History-token configuration removes STMB and supplies historical images through the VLM’s native multi-image interface. At the default five-frame history, the original results are:
| Method | LLM input tokens | FLOPs (T) | Measured latency (ms) | GPU memory (GB) |
|---|---|---|---|---|
| History-token input | 857–865 | 16.391 | 132.74 | 18.479 |
| HIPPOVLA without caching | 197–205 | 6.162 | 144.92 | 18.585 |
We replayed 1,011 instruction instances, producing 8,360 calls at an eight-step interval. History and caches are cleared at each new instruction. We retain five historical frames sampled at ten-step intervals and report latency over the 1,582 calls with complete history.
Our implementation caches visual features by observation timestep within each instruction. Only historical frames missing from the feature cache undergo image-processor preprocessing, pixel transfer to the GPU, and visual encoding; cache hits directly reuse stored features.
| Method | Mean latency (ms) ↓ | P95 latency (ms) ↓ | GPU memory (GB) ↓ |
|---|---|---|---|
| HIPPOVLA without caching | 150.01 | 151.19 | 18.585 |
| HIPPOVLA with optimized caching | 140.30 | 146.78 | 18.696 |
| Trained History-token baseline | 151.10 | 155.46 | 18.479 |
Cached HIPPOVLA achieves 140.30 ms mean latency and 146.78 ms P95. Its recorded mean latency is 6.5% lower than uncached HIPPOVLA and 7.1% lower than the trained History-token baseline, at the cost of slightly higher inference memory.
At the default setting, HIPPOVLA therefore retains its shorter language-model input and lower FLOPs, offers lower training time and sampled training memory than History-token under ZeRO-3, and achieves lower measured inference latency with caching in the evaluated replay setting.
Thank you for your detailed questions about whether HIPPOVLA relies on history at inference time, how we define success and stopping behavior and classify failures on the real robot, why explicit temporal encoding reduces performance, how HIPPOVLA compares with a history-token baseline under a matched training budget, and what training and inference costs each approach incurs. We also appreciate your careful scrutiny of the sources and evaluation protocols behind the CALVIN baseline results and your suggestion to evaluate on RMBench. Below, we address your six questions and then discuss how RMBench could complement our evaluation of visual memory.
1. Does HIPPOVLA use history at test time?
Thank you for suggesting this test. HIPPOVLA does use historical information at inference time. For each benchmark, we evaluated the same trained HIPPOVLA checkpoint, without retraining, under three inference conditions:
All three conditions use evaluation seed 42 and the same benchmark protocols: 1,000 instruction sequences on CALVIN ABC→D and 50 trials for each of the ten LIBERO-Long tasks.
Neither intervention preserves the gain over the no-STMB baseline. The trained policy’s performance drops sharply both when retrieved context is removed and when the supplied history comes from another rollout of the same task. These results show that HIPPOVLA uses its historical retrieval pathway at inference time and is sensitive to whether the history corresponds to the current interaction. The intervention rows measure the sensitivity of a fixed checkpoint. The reference row measures a policy trained from the outset without STMB.