acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Current Observation: History-Informed Adaptation for Vision-Language-Action Models

Abstract

In robotic manipulation, the task-relevant scene state is often shaped by prior interactions. A robot must infer prior events, retain relevant evidence, and use it to guide subsequent actions. Most existing Vision-Language-Action (VLA) models, however, implicitly assume that the current observation alone provides sufficient information for action prediction. We introduce HI-Adapter, a lightweight and transferable history-informed adaptation framework that equips pretrained VLA policies to understand and utilize recent interaction history without robot-video pretraining. HI-Adapter comprises three components: a spatiotemporal vision adapter that aggregates multi-view observations across time to contextualize visual evidence; a reconstruction-guided memory adapter that distills task-relevant historical information into compact memory tokens; and an action expert adapter that selectively incorporates relevant memory representations into the action-generation process. Together, they improve temporal grounding and memory-guided manipulation in a 1B-parameter VLA policy while training only around 10% of its parameters. Our method achieves high success rates on RMBench and RoboMME, outperforming the baselines. It further reaches high performance on real-world tasks requiring recent historical context. Its consistent improvements when integrated into demonstrate transferability across VLA architectures. These results establish history-informed adaptation as an effective and parameter-efficient approach for overcoming the reliance of existing VLA policies on the current observation alone.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model, Memory, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_oQ5WOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes HI-Adapter, a framework that adapts pretrained VLAs for history-dependent manipulation while keeping their backbones frozen. It combines a spatiotemporal vision adapter for encoding recent multi-view observations, recurrent memory for consolidating historical features, and policy adapters that condition action generation on this memory. Part of the memory is supervised through SigLIP feature reconstruction and relative temporal-order prediction, alongside action supervision. The method is evaluated on RoboMME and six real-world tasks using RDT-1B and π0.5\pi_\text{0.5}.

Critical strengths
  • The consistent adapter design offers a practical way to augment pretrained VLAs with memory while training only a small fraction of their parameters. In particular, the vision adapter could be useful for policies that need to process multi-view, multi-frame observations.
  • The paper presents six real-world manipulation tasks that collectively test multiple memory requirements, including counting and temporal reference.
Critical weaknesses
  • The paper's positioning relative to existing memory-augmented policies is unclear, and its efficiency claims lack comparisons of task performance, training cost, and inference latency against such methods. Additionally, the Related Work discussion frames prior PEFT applications as limited in scope, but does not explain why this constitutes a limitation motivating the proposed method, where adapters serve as a technical design choice for memory augmentation.
  • Inconsistent terminology for the vision and memory components, together with notation mismatches (e.g., zz in Figure 3(a) versus XX in Equation (2)), makes the methodology difficult to follow. Figure 2 further obscures the visual/language-conditioning and memory-injection paths, as well as the distinction between proprioceptive inputs and noisy action tokens.
  • The component ablations do not isolate the contributions of recurrent aggregation, reconstruction/order supervision, or the memory slot partition (while the spatiotemporal-only variant for π0.5\pi_\text{0.5} is also omitted from Table 1). Targeted ablations and analyses of memory content would better establish the benefits of these design choices and whether the action-oriented slots retain additional control-relevant information.
  • Several quantitative claims conflict with the reported tables. For example, Table 1 labels π0.5\pi_\text{0.5}'s Permanence improvement as 19%p rather than 29%p, while Section 4.2 describes the final real-world success rates (14.16% and 19.16%) as gains instead of the corresponding 8.33%p and 7.5%p improvements.
  • The paper lacks sufficient implementation detail for reproducibility, particularly regarding SigLIP target selection (anchor view, selection rule, target count, and candidate history window). The history-buffer sampling schedule and auxiliary-loss weights are also not specified.
Questions for authors

Please see the weaknesses above and provide the corresponding clarifications and supporting analyses.

Reviewer_JzBKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper adds history awareness to frame-based VLAs through adapters, keeping the backbones frozen. There are three parts. (1) An AIM-style spatiotemporal adapter that reuses the frozen image encoder's spatial attention along the time axis, with Perceiver-style query read/write adapters. (2) A slot-based gated recurrent memory. Half of the slots are trained to reconstruct SigLIP features of past frames selected by instruction-image similarity (Hungarian matching), and a margin ranking loss supervises their temporal order. (3) Cross-attention adapters that inject the memory into the frozen action head. The method is applied to RDT-1B and pi0.5 and evaluated on the four RoboMME suites and six real-world history-dependent tasks with 20 trials each. Reported gains: RoboMME 1.3 -> 14.8 (RDT-1B) and 18.9 -> 37.6 (pi0.5); real world 5.8 -> 14.2 and 11.7 -> 19.2.

Critical strengths
  • The real-world tasks are built so that the current frame is actually ambiguous (identical tubes, swapped plates, stirring counts). This is the right kind of test for memory.
  • The method is applied to two backbones with different action heads (RDT diffusion transformer, pi0.5 flow-matching expert).
Critical weaknesses
  1. Limited novelty. Every component is an existing technique moved to a new place. The spatiotemporal adapter is AIM (Yang et al., 2023), as the paper acknowledges. The query adapter is Perceiver-style read/write. The memory is a gated recurrent slot state, close to the recurrent queries of ReMem-VLA and the memory module of HAMLET. The auxiliary losses are feature distillation with Hungarian matching plus a ranking loss. The problem setting, making a pretrained single-frame VLA history-aware with lightweight added modules, is the same as HAMLET's. The paper does not show what this design can do that these methods cannot.

  2. No memory baselines. ContextVLA, HAMLET, MemoryVLA and ReMem-VLA are discussed in Sec. 2, but none of them appears in any table. The only comparison is a memoryless backbone against the same backbone given history. On tasks that are constructed to be unsolvable without history, beating a memoryless policy says very little. Even the simplest controls are missing: multi-frame stacking, and LoRA or full fine-tuning with the same history input. Without the latter, the "parameter-efficient" claim is also untested. The RoboMME paper itself evaluates several memory designs on pi0.5 (the authors cite this at L433), so a direct comparison was readily available.

  3. The ablations do not cover the claimed contributions. Sec. 4.3 repeats the "+Spatiotemporal Adaptation" row of Table 1, and only for RDT-1B, although the text says two backbones. Nothing ablates L_rec, L_ord, the task/action slot split, the SigLIP target selection, or the memory readout, so contribution 2 has no supporting evidence. On RDT-1B the spatiotemporal adapter alone accounts for 7.6 of the 13.5-point gain, which leaves the contribution of the memory module unclear.

  4. The real-world evidence is weak. With 20 trials per task, most per-task differences (5-15 points) are 1-3 episodes. For pi0.5, HI-Adaptation gives no gain on Stir Memory and is worse on Plug Restoration (5% -> 0%). The best average is 19%. There are no seeds or confidence intervals, and failures are not broken down into wrong-target and manipulation failures. That breakdown is what would show that the memory, rather than noise or extra fine-tuning, causes the gain.

  5. The memory supervision targets are questionable. Frames are selected by SigLIP cosine similarity between the instruction and a single image. For instructions like "the beaker stirred the most times" or "the second contaminated tube", relevance depends on an event or a count across frames, not on how similar one frame is to the text. The paper does not analyze which frames are actually selected.

  6. The manuscript is unfinished and its numbers are inconsistent.

    • L435: "an average gain of [X] percentage points" is a leftover placeholder.
    • L426: "improves RDT-1B by 14.5 points", but Table 1 gives 1.3 -> 14.8 (13.5).
    • L447-448: "improves ... by 14.16 and 19.16 percentage points", but these are the final success rates. The gains in Table 2 are 8.33 and 7.5.
    • Fig. 2 shows a "Semantic Matching Loss" with a teacher text encoder, which does not match the losses in Sec. 3.4.
    • Naming is inconsistent: Plug Restoration vs. Peg Restoration, and "HI Adapters" vs. "H.I. Adaptation". Figures are low resolution and hard to read. Per-task RoboMME results are not reported.
Questions for authors
  1. How are the baselines trained? Same data and steps? Full fine-tuning or frozen? Do they see any history?
  2. How does the method compare with HAMLET / ContextVLA / MemoryVLA and with the memory variants reported in the RoboMME paper, under the same backbone?
  3. Can you provide per-component ablations (L_rec, L_ord, slot split, memory readout) on pi0.5?
  4. How are the 16 history frames sampled from episodes of ~430 steps? In the real-world tasks, does the key evidence ever fall outside this buffer? If not, the recurrent memory is never needed beyond what the spatiotemporal adapter already sees.
  5. What are the trainable parameter count for pi0.5, the training cost, and the inference latency (3 views x 16 frames through the vision encoder with temporal attention at every control step)?
Reviewer_aRqcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes History-informed Adaptation, which is a parameter-efficient framework that allows pretrained frame-based VLAs to have spatiotemporal modeling and history-conditioned action generation. The goal is to be lightweight and keep the pretrained backbones frozen. The paper also introduces recurrent interaction memory with task-oriented and action-oriented slots to capture task relevant features. The paper evaluates RoboMME and six real world interaction tasks by adapting pi 0.5 and RDT-1B.

Critical strengths

The simulated benchmark testing, RoboMME, is relevant to the literature and the real-world tasks are well done.

Critical weaknesses

The clarity of the paper can generally be improved. This methodology relies on have a quick parameter efficient adaption of different modalities that is not task-specific. Generally, robotic systems should tackle memory without requiring adaption of existing VLAs, rather the VLA should be trained to have longer context in pretraining. What is the benefit of tacking on memory modules to the VLAs, other than convenience?

Questions for authors

Why do VLAs have to be adapted in a lightweight manner, and why can they not just be trained to understand context?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.